Data-Lab

I’ve never seen so many posts about Apache Spark before, not sure if it’s 3.0, or because the world is burning down. I’ve written about Spark a few times, even 2 years ago, but it still seems to be steadily increasing in popularity, albeit still missing from many companies tech stacks. With the continued rise […]

Base Docker image with just essentials: Hadoop, Hive and Spark. - GitHub - panovvv/hadoop-hive-spark-docker: Base Docker image with just essentials: Hadoop, Hive and Spark.

Example for article Running Spark 3 with standalone Hive Metastore 3.0

Intro

jar dependencies are very finicky

MinIO is a cloud object storage that offers high-performance, S3 compatible. Native to Kubernetes, MinIO is the only object storage suite…

I'm trying to run a simple spark to s3 app from a server but I keep getting the below error because the server has hadoop 2.7.3 installed and it looks like it doesn't include the GlobalStorageStati...

Collection of Minio recipes. Contribute to nitisht/cookbook development by creating an account on GitHub.

I'm launching a pyspark program: $ export SPARK_HOME= $ export PYTHONPATH=$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.9-src.zip $ python And the py code: from pyspark import SparkContext,

Disaggregated compute and storage for the apprentice Data Engineer

When working on real data products, we will register an account on cloud providers such as Amazon, Azure, or Google so that we are able to…