Data Engineering

Data Lake and the Data Warehouse. They seemed similar, but there are differences.

When I talk to people who are not deep into the Power BI world, I often get the impression that they think of Power BI as a visualization tool exclusively. While that is true to a certain extent, it seems to me that they are not seeing the bigger picture – or maybe it’s better to say – they see just

Course schedule with slides, lecture notes, and videos.

Course schedule with slides, lecture notes, and videos.

Discover the most used messaging patterns and how data movement affects operational complexity, costs, and sustainability.

Read writing about Data Architecture in Data Engineer Things. Things learned in our data engineering journey and ideas on data and engineering.

bytewax - Stream Processing Python framework

The rise of tools and processes to manage and control data. By Assaf Araki and Ben Lorica. Data has emerged as an imperative foundational asset for all organizations. Data fuels significant initiatives such as digital transformation and the adoption of analytics, machine learning, and AI. Organizations that are able to tame, manage, and unlock theirContinue reading "What is DataOps?"

Proven approach to get usable experience and land a data engineering job

Frustrated with handling data type conversion issues in python? Then this post is for you. In this post, we go over a reusable data type conversion pattern using Pydantic. We will also go over the caveats involved in using this library.

Ten steps to ensure your data monitoring solution is effective.

System design questions have increasingly become an integral part of software engineering interviews. For senior engineers, the discussion around system design is considered even more important than solving a coding question. In a system design interview, you can show your real design skills and show how they will work with designing complex systems. It is a given that a good performance in system design interviews will get you a senior position and result in higher salaries. This course presents the architectural review of famous distributed systems. The main goal is to extract out important design details that are relevant to system design interviews. The course also presents a list of system design patterns that constitute the common design problems and their solutions that different distributed systems have developed over time.

A pragmatic approach to data contracts

Batch data processing — historically known as ETL — is extremely challenging. It’s time-consuming, brittle, and often unrewarding. Not only…

Data pipelines built (and added on to) without a solid foundation will suffer from poor efficiency, slow development speed, long times to triage production issues, and hard testability. What if your data pipelines are elegant and enable you to deliver features quickly? An easy-to-maintain and extendable data pipeline significantly increase developer morale, stakeholder trust, and the business bottom line! Using the correct design pattern will increase feature delivery speed and developer value (allowing devs to do more in less time), decrease toil during pipeline failures, and build trust with stakeholders. This post goes over the most commonly used data flow design patterns, what they do, when to use them, and, more importantly, when not to use them. By the end of this post, you will have an overview of the typical data flow patterns and be able to choose the right one for your use case.

How to build a Recoverable & Reproducible data pipeline

(empty introductory line to avoid a formatting issue with Medium editor)

And how to run it on Kubernetes.

What is Multidimensional schemas? Multidimensional schema is especially designed to model data warehouse systems. The schemas are designed to address the unique needs of very large databases designed

In recent years, I’ve done dozens of training on various data platform topics, for all kinds of audiences. When teaching various data platform concepts and techniques, I find one of the concepts particularly intimidating for many business analysts, especially those who are just starting their journe

YTsaurus is a scalable and fault-tolerant open-source big data platform. - ytsaurus/ytsaurus: YTsaurus is a scalable and fault-tolerant open-source big data platform.

Concise screencasts for data engineers.

System Design is an important topic to understand if you want to advance further in your career as a software engineer. Even if you are just beginning your coding journey, it's a good idea to get a head start on learning about system design. Early in your career you will

The weekly podcast about data engineering. Learn and stay current on modern data management, featuring weekly deep dives with the engineers, innovators, and entrepreneurs who are shaping the industry. Your best resource for big data, ETL, databases, data lakes, and running machine learning in production.

But almost identical to a less known profession

Mock interviews with engineers from FAANG and other top companies. Get actionable feedback, get awesome, get fast-tracked.

Metadata, Statistics on Row Groups, Partitions discovery, and Repartitioning

Preparing for interviews is a stressful task. There is an enormous amount of resources available in the internet, multiple repositories, there are even companies that help students prepare for interviews at the Big Tech companies. The idea here is to create an accessible version of these resources so people on this journey can benefit from it.

Get the sample models for MongoDB, Neo4j, Cassandra, Swagger, Avro, Parquet, Glue, and more! After download, open the models using Hackolade, and learn through the examples how to leverage the modeling power of the software.