Spark
Deep Dive
spark createOrReplaceTempView vs createGlobalTempView
Spark Dataset 2.0 provides two functions createOrReplaceTempView and createGlobalTempView. I am not able to understand the basic difference between both functions.
According to API documents:
createOrReplaceTempView() creates or replaces a local temporary view with this dataframe df. Lifetime of this view is dependent to SparkSession class
createGlobalTempView() creates a global temporary view with this dataframe df. life time of this view is dependent to spark application itself
Spark Broadcast Variables - Spark by {Examples}
In Spark RDD and DataFrame, Broadcast variables are read-only shared variables that are cached and available on all nodes in a cluster in-order to access or use by the tasks. Instead of sending this data along with every task, spark distributes broadcast variables to the machine using efficient broadcast algorithms to reduce communication costs. Use
In Spark RDD and DataFrame, Broadcast variables are read-only shared variables that are cached and available on all nodes in a cluster in-order to access or use by the tasks. Instead of sending this data along with every task, spark distributes broadcast variables to the machine using efficient broadcast algorithms to reduce communication costs.
Dynamic Partition Pruning in Spark 3.0 - DZone Big Data
This blog will give you a deep insight on Dynamic Partition Pruning used in Apache Spark and how this works in the newer version of Spark released.
Therefore, we don’t need to actually scan the full fact table as we are only interested in two filtering partitions that result from the dimension table.
To avoid this, a simple approach is to take the filter from the dimension table incorporated into a sub query. Then run that sub query below the scan on the fact table.
Spark Window Functions with Examples - Spark by {Examples}
Spark Window functions are used to calculate results such as the rank, row number e.t.c over a range of input rows and these are available to you by importing org.apache.spark.sql.functions._, this article explains the concept of window functions, it’s usage, syntax and finally how to use them with Spark SQL and Spark’s DataFrame API. These […]
pyspark.SparkConf — PySpark 3.2.1 documentation
Most of the time, you would create a SparkConf object with SparkConf(), which will load values from spark.* Java system properties as well. In this case, any parameters you set directly on the SparkConf object take priority over system properties.
Reading Spark DAGs - DZone Java
See how to effectively read Directed Acyclic Graphs (DAGs) in Spark to better understand the steps a program takes to complete a computation.
Tuning - Spark 3.3.0 Documentation
Tuning and performance optimization guide for Spark 3.3.0
The main point to remember here is
that the cost of garbage collection is proportional to the number of Java objects, so using data
structures with fewer objects (e.g. an array of Ints instead of a LinkedList) greatly lowers
this cost.
Spark Architecture: Shuffle
This is my second article about Apache Spark architecture and today I will be more specific and tell you about the shuffle, one of the most interesting topics in the overall Spark design. The previous part was mostly about general Spark architecture and its memory management. It can be accessed here. The next one is [...]
Spark Repartition & Coalesce - Explained
DataNoon - Making Big Data and Analytics simple!
Structured Streaming Programming Guide - Spark 3.5.1 Documentation
Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine.
Finally, the system ensures end-to-end exactly-once fault-tolerance guarantees through checkpointing and Write-Ahead Logs.
Structured Streaming queries are processed using a micro-batch processing engine, which processes data streams as a series of small batch jobs thereby achieving end-to-end latencies as low as 100 milliseconds and exactly-once fault-tolerance guarantees
However, since Spark 2.3, we have introduced a new low-latency processing mode called Continuous Processing, which can achieve end-to-end latencies as low as 1 millisecond with at-least-once guarantees.
The output can be defined in a different mode
Complete Mode
Append Mode
Update Mode
The engine uses checkpointing and write-ahead logs to record the offset range of the data being processed in each trigger.
sliding event-time
We can
easily define watermarking on the previous example using withWatermark() as shown below.
In other words, late data within the threshold will be aggregated,
but data later than the threshold will start getting dropped
(see later
in the section for the exact guarantees). Let’s understand this with an example
This means the system needs to know when an old
aggregate can be dropped from the in-memory state because the application is not going to receive
late data for that aggregate any more.
we have introduced
watermarking, which lets the engine automatically track the current event time in the data
and attempt to clean up old state accordingly. You can define the watermark of a query by
specifying the event time column and the threshold on how late the data is expected to be in terms of
event time
Note that after every trigger,
the updated counts (i.e. purple rows) are written to sink as the trigger output, as dictated by
the Update mode.
Understanding Spark Connect API - Part 1: Shortcomings of Spark Driver Architecture
Thoughts on technology, life and everything else.
Spark Optimization : Reducing Shuffle
“Shuffling is the only thing which Nature cannot undo.” — Arthur Eddington
Optimization
The SwirlAI Data Engineering Project Master Template: The Collector (Part 1).
And how to run it on Kubernetes.
Create Your Very Own Apache Spark/Hadoop Cluster....then do something with it? - Confessions of a Data Guy
I’ve never seen so many posts about Apache Spark before, not sure if it’s 3.0, or because the world is burning down. I’ve written about Spark a few times, even 2 years ago, but it still seems to be steadily increasing in popularity, albeit still missing from many companies tech stacks. With the continued rise […]
Data-Labpanovvv/hadoop-hive-spark-docker: Base Docker image with just essentials: Hadoop, Hive and Spark.
Base Docker image with just essentials: Hadoop, Hive and Spark. - GitHub - panovvv/hadoop-hive-spark-docker: Base Docker image with just essentials: Hadoop, Hive and Spark.
spark_hive_test/src/main/scala/tests/SparkHiveTest.scala at master · arempter/spark_hive_test · GitHub
Example for article Running Spark 3 with standalone Hive Metastore 3.0
pyspark connect to aws s3a filesystem
jar dependencies are very finicky
Reading and Writing Data from/to MinIO using Spark
MinIO is a cloud object storage that offers high-performance, S3 compatible. Native to Kubernetes, MinIO is the only object storage suite…
java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
I'm trying to run a simple spark to s3 app from a server but I keep getting the below error because the server has hadoop 2.7.3 installed and it looks like it doesn't include the GlobalStorageStati...
cookbook/docs/apache-spark-with-minio.md at master · nitisht/cookbook · GitHub
Collection of Minio recipes. Contribute to nitisht/cookbook development by creating an account on GitHub.
Add Jar to standalone pyspark
I'm launching a pyspark program:
$ export SPARK_HOME=
$ export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.9-src.zip
$ python
And the py code:
from pyspark import SparkContext,
.config('spark.jars.packages', 'org.apache.spark:spark-sql-kafka-0-10_2.12:3.0.1')
Adding some MinIO to your standalone Apache Spark cluster
Disaggregated compute and storage for the apprentice Data Engineer
DataOps 02: Spawn up Apache Spark infrastructure by using Docker
When working on real data products, we will register an account on cloud providers such as Amazon, Azure, or Google so that we are able to…
Populate current date and current timestamp in pyspark - DataScience Made Simple
In order to populate current date and current timestamp in pyspark we will be using current_date() and current_timestamp() function respectively - example
TutorialsHow to connect to remote hive server from spark
I'm running spark locally and want to to access Hive tables, which are located in the remote Hadoop cluster.
I'm able to access the hive tables by lauching beeline under SPARK_HOME
[ml@master spa...
How to read data from HDFS in Pyspark -
This recipe helps you read data from HDFS in Pyspark