Data Engineering

Learn the key steps of deploying databases and stateful workloads in Kubernetes and meet cloud-native technologies that can streamline Apache Cassandra for K8s.

Data Lake and the Data Warehouse. They seemed similar, but there are differences.

Everything you ever wanted for your Python project continuous integration pipeline — up-and-running in matter of minutes

Explore top 15 real-world data engineering projects ideas for beginners with source code to gain hands-on experience on diverse data engineering skills.

System design questions have increasingly become an integral part of software engineering interviews. For senior engineers, the discussion around system design is considered even more important than solving a coding question. In a system design interview, you can show your real design skills and show how they will work with designing complex systems. It is a given that a good performance in system design interviews will get you a senior position and result in higher salaries. This course presents the architectural review of famous distributed systems. The main goal is to extract out important design details that are relevant to system design interviews. The course also presents a list of system design patterns that constitute the common design problems and their solutions that different distributed systems have developed over time.

In this article, we will go through the Microsoft Azure Data Factory service, that can be used to ingest, copy and transform data generated from various data sources

Resources and tips to help you nail down the DP-203 exam to earn the Azure Data Engineer Associate Certificate

Preparing for a data engineering interview and are overwhelmed by all the tools and concepts?. Then this post is for you, in this post we go over the most common tools and concepts you need to know to ace your data engineering interviews.

What is data versioning? When is data versioning appropriate? We review the various tools and use-cases needed for the best implementation.

This post goes over what the ETL and ELT data pipeline paradigms are. It tries to address the inconsistency in naming conventions and how to understand what they really mean. Finally ends with a comparison of the 2 paradigms and how to use these concepts to build efficient and scalable data pipelines.

When you watch the blueprint I also use in my cookbook you see the different phases: Connect, Processing Framework, Store and Buffer. At…

Spark Data processing in an easy, cost-effective and serverless way

A step-by-step guide on how to start with Airflow: from your local set-up to creating simple tasks.

Are you disappointed with online SQL tutorials that aren't deep enough? Are you frustrated knowing that you are missing SQL skills, but can't quite put your finger on it? This post is for you. In this post, we go over a few topics that can take your SQL skills to the next level and help you be a better data engineer.

In this post, we go over 6 key concepts to help you master window functions. Window functions are one the most powerful features of SQL, they are very useful in analytics and performing operations that cannot be done easily with the standard group by, subquery and filters. Despite this, window functions are not used frequently. If you have ever thought 'window functions are confusing', then this post is for you.

You have heard of Common Table Expressions(CTEs), but are not be sure what they are and when to use them. What if you knew exactly what Common Table Expressions(CTEs) were and when to use them? In this post, we go over what CTEs are, and their performance comparisons against subqueries, derived tables, and temp tables to help decide when to use them.

Moving to Azure and implementing Databricks and Delta Lake for managing your data pipelines is recommended by Microsoft for the Modern Data…

A step by step guide on how to build an Event-driven architecture in Azure.

The transformations to mold the data from an application in a form that is better suited for data analysis is done in a data warehouse.

Helsingin yliopiston kaikille avoin ja ilmainen ohjelmoinnin perusteet opettava verkkokurssi. Kurssilla perehdytään nykyaikaisen ohjelmoinnin perusideoihin sekä ohjelmoinnissa käytettävien työvälineiden lisäksi algoritmien laatimiseen. Kurssille osallistuminen ei vaadi ennakkotietoja ohjelmoinnista.

Learning Resources

Frustrated with handling data type conversion issues in python? Then this post is for you. In this post, we go over a reusable data type conversion pattern using Pydantic. We will also go over the caveats involved in using this library.

There are lots of data-storage options available today. You have to choose between managed or unmanaged, relational or NoSQL, write- or read-optimized, proprietary or open-source — and it doesn't end there. Once you begin your search, you will end up in the universe that is database marketing. All of the vendors

Frustrated that hiring managers are not reading your Github projects? then this post is for you. In this post, we discuss a way to impress hiring managers by hosting a live dashboard with near real-time data. We will also go over coding best practices such as project structure, automated formatting, and testing to make your code professional. By the end of this post, you will have deployed a live dashboard that you can link to your resume and LinkedIn.

The weekly podcast about data engineering. Learn and stay current on modern data management, featuring weekly deep dives with the engineers, innovators, and entrepreneurs who are shaping the industry. Your best resource for big data, ETL, databases, data lakes, and running machine learning in production.

Concise screencasts for data engineers.

Roadmap to becoming a data engineer in 2021. Contribute to datastacktv/data-engineer-roadmap development by creating an account on GitHub.

To connect to Azure SQL Database using MFA (which is in SSMS as "Active Directory - Universal") Microsoft recommends and currently only has a tutorial on connecting with C# using Microsoft.Identity...

This is the first of a series of posts about Azure Data Studio. If you have not used Data Studio or only tried it when it first came out a few years ago, time to take another look. Especially if yo…