Data Engineering

System design interviews are usually vague and depend on you (as the interviewee) to guide the interviewer. If you are thinking: How do I prepare for data engineering system design interviews? I struggle to think of questions you would ask in a system design interview for data engineering; I don't have enough interview experience to know what companies ask. Is data engineering "system design" more than choosing between technologies like Spark and Airflow? This post is for you! Imagine being able to solve any data systems design interviews systematically. You'll be able to showcase your abilities and demonstrate clear thinking to your interviewer. By the end of this post, you will have a list of questions ordered by concepts that you can use to approach any data systems design interview.

B-trees are used by many modern DBMSs. Learn how they work, how databases use them, and how your choice of primary key can affect index performance.

Why the reports of software’s demise are greatly exaggerated.

Imagine working for a company that processes a few GBs of data every day but spends hours configuring/debugging large-scale data processing systems! Whoever set up the data infrastructure copied it from some blog/talk by big tech. Now, the responsibility of managing the data team's expenses has fallen on your shoulders. You're under pressure to scrutinize every system expense, no matter how small, in an effort to save some money for the organization. It can be frustrating when data vendors charge you a lot and will gladly charge you more if you are not careful with usage. Imagine if your data processing costs were dirt cheap! Imagine being able to replicate and debug issues quickly on your laptop! In this post, we will discuss how to use the latest advancements in data processing systems and cheap hardware to enable cheap data processing. We will use DuckDB and Python to demonstrate how to process data quickly while improving developer ergonomics.

Level up your performance reviews and influence. Get clear expectations to the next level.

Learn about the Data Tech Stack used by Netflix to process trillions of events every day.

As data engineers, you might have heard the terms functional data pipeline, factory pattern, singleton pattern, etc. One can quickly look up the implementation, but it can be tricky to understand what they are precisely and when to (& when not to) use them. Blindly following a pattern can help in some cases, but not knowing the caveats of a design will lead to hard-to-maintain and brittle code! While writing clean and easy-to-read code takes years of experience, you can accelerate that by understanding the nuances and reasoning behind each pattern. Imagine being able to design an implementation that provides the best extensibility and maintainability! Your colleagues (& future self) will be extremely grateful, your feature delivery speed will increase, and your boss will highly value your opinion. In this post, we will go over the specific code design patterns used for data pipelines, when and why to use them, and when not to use them, and we will also go over a few python specific techniques to help you write better pipelines. By the end of this post, you will be able to identify patterns in your data pipelines and apply the appropriate code design patterns. You will also be able to take advantage of pythonic features to write bug-free, maintainable code that is a joy to work on!

A primer on GPU architecture and computing

This is a single SVG file created purely through D2 text:

Data Engineering Practice Problems. Contribute to danielbeach/data-engineering-practice development by creating an account on GitHub.

When I talk to people who are not deep into the Power BI world, I often get the impression that they think of Power BI as a visualization tool exclusively. While that is true to a certain extent, it seems to me that they are not seeing the bigger picture – or maybe it’s better to say – they see just

Course schedule with slides, lecture notes, and videos.

Course schedule with slides, lecture notes, and videos.

Discover the most used messaging patterns and how data movement affects operational complexity, costs, and sustainability.

Read writing about Data Architecture in Data Engineer Things. Things learned in our data engineering journey and ideas on data and engineering.

bytewax - Stream Processing Python framework

We are excited to ship the first version of pgroll, a command line tool that offers safe and reversible schema migrations for PostgreSQL

(empty introductory line to avoid a formatting issue with Medium editor)

And how to run it on Kubernetes.

In recent years, I’ve done dozens of training on various data platform topics, for all kinds of audiences. When teaching various data platform concepts and techniques, I find one of the concepts particularly intimidating for many business analysts, especially those who are just starting their journe

There are 300+ databases; what do they all do?

YTsaurus is a scalable and fault-tolerant open-source big data platform. - ytsaurus/ytsaurus: YTsaurus is a scalable and fault-tolerant open-source big data platform.

Mock interviews with engineers from FAANG and other top companies. Get actionable feedback, get awesome, get fast-tracked.

Metadata, Statistics on Row Groups, Partitions discovery, and Repartitioning

How to build a Recoverable & Reproducible data pipeline