Reading Spark DAGs - DZone Java
See how to effectively read Directed Acyclic Graphs (DAGs) in Spark to better understand the steps a program takes to complete a computation.
Spark
Deep Dive
Dynamic Partition Pruning in Spark 3.0 - DZone Big Data
This blog will give you a deep insight on Dynamic Partition Pruning used in Apache Spark and how this works in the newer version of Spark released.
Therefore, we don’t need to actually scan the full fact table as we are only interested in two filtering partitions that result from the dimension table.
To avoid this, a simple approach is to take the filter from the dimension table incorporated into a sub query. Then run that sub query below the scan on the fact table.
pyspark.SparkConf — PySpark 3.2.1 documentation
Most of the time, you would create a SparkConf object with SparkConf(), which will load values from spark.* Java system properties as well. In this case, any parameters you set directly on the SparkConf object take priority over system properties.
Spark Window Functions with Examples - Spark by {Examples}
Spark Window functions are used to calculate results such as the rank, row number e.t.c over a range of input rows and these are available to you by importing org.apache.spark.sql.functions._, this article explains the concept of window functions, it’s usage, syntax and finally how to use them with Spark SQL and Spark’s DataFrame API. These […]
Populate current date and current timestamp in pyspark - DataScience Made Simple
In order to populate current date and current timestamp in pyspark we will be using current_date() and current_timestamp() function respectively - example
Tutorials