The missing piece of the modern data stack
Our cool new house needs one more plank in its foundation.
Data Engineering
Kicking the tires on dbt Metrics
Daily Active YAML is up and to the right
The modern data experience (w/ Benn Stancil)
For most people, the modern data stack isn’t a collection of architectural diagrams; it’s an experience. It’s rushing to answer the CEO’s urgent question bef...
Engineers Shouldn’t Write ETL: A Guide to Building a High Functioning Data Science Department | Stitch Fix Technology – Multithreaded
“What is the relationship like between your team and the data scientists?” This is, without a doubt, the question I’m most frequently asked when conducting i...
There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume.
Instead, give people end-to-end ownership of the work they produce (autonomy). In the case of data scientists, that means ownership of the ETL.
Viewpoint | dbt Docs
In 2015-2016, a team of folks at RJMetrics had the opportunity to observe, and participate in, a significant evolution of the analytics ecosystem. The seeds of dbt were conceived in this environment, and the viewpoint below was written to reflect what we had learned and how we believed the world should be different. dbt is our attempt to address the workflow challenges we observed, and as such, this viewpoint is the most foundational statement of the dbt project's goals.
Upgrading Data Warehouse Infrastructure at Airbnb
This blog aims to introduce Airbnb’s experience upgrading Data Warehouse infrastructure to Spark and Iceberg
We the purple people
The data world needs more purple people — generalists who can navigate both the business context and the modern data stack. Let's put aside skillset dichotomies, and learn to feel comfortable in the space between.
The end of Big Data
Databricks, Snowflake, and the end of an overhyped era.
Take real-time products, for example. Most businesses have little use for true real-time experiences. But, all else being equal, real-time data is better than latent data. We all have dashboards that update a little too slowly, or marketing emails we wish we could send a little sooner. While these annoyances don’t justify the effort currently required to build real-time pipelines, they do cause small headaches. But if someone came along and offered me a streaming Fivetran, or a reactive version of dbt, I’d take it. If the cost of a real-time architecture was low enough, regardless of the shoehorned use-cases, there’d be no reason to turn it down. And just as we came to rely on Snowflake after we chose it as a better Postgres, I’m certain we’d come to rely on streaming pipelines if they replaced our current batch ones. We’d start doing more real-time marketing outreach, or build customer success workflows around live customer behavior. Over the next five years, I’d guess that real-time data tools follow this exact path: They’ll finally go mainstream, not because we all discover we need them, but because there will be no reason not to have them. And once we do, we’ll find ways to push it to their limits, just as we did with fast internet connections and powerful browsers.
Ep 30: The Personal Data Warehouse (w/ Jordan Tigani of MotherDuck)
Flipping the vision for "data apps" on its head: what if, instead of having data run round trips to a cloud data warehouse, we just bring the user's data to their machine?
Microsoft, Google, and the original purple people
And, of course, Pokémon.
A Thought of Stream...
...and a three-horse race
HTAP Databases
Do we actually need so many different databases? Or can we shove them all into a single cloud infrastructure and behind the same SQL API?
dbt Tests – Create Staging Environments for Flawless Data CI/CD
Introduce new data with confidence by using lakeFS branches and dbt testing to create perfect staging environments to validate data.
Modern Data Analytics Reference Architecture on AWS
Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics
Meetup: Comparison of Data Lakehouse Table Formats
This presentation covers the three major data lake table formats – Apache Iceberg, Apache Hudi, and Delta Lake – how they work, their features, and their limitations so you can make an informed decision when architecting your data lakehouse.
Iceberg Case Studies
This talk will introduce the use cases for Apache Iceberg tables that we didn’t expect when we created Iceberg and will explain the details so you can use Iceberg for similar cases.
Implementation of SCD-2 (Slowly Changing Dimension) with Apache Hudi
Authored & Contributed by : Jayasheel Kalgal, Esha Dhing, Prashant Mishra
The Modern Data Stack: Past, Present, and Future
My thoughts on where our space has been and where it might be going.
Comparison of Data Lake Table Formats (Iceberg, Hudi and Delta Lake)
Comparison of Data Lake Table Formats (Iceberg, Hudi and Delta Lake).
Why You Shouldn’t Care About Iceberg | Tabular
Slides: https://www.datacouncil.ai/talks/why-you-shouldnt-care-about-iceberg
ABOUT THE TALK:
Ryan Blue, co-creator of the Apache Iceberg project will try to convince you not to care about Iceberg: if you’re thinking about your table format, then it isn’t doing a good enough job.
This session will show how Iceberg solves real-world problems that used to take hours or days of time from data engineers and analysts:
Safe schema changes — no more zombie data columns
Layout evolution — update table partitioning without rewriting any queries
Hidden partitioning — safe and fast queries without being a DBA
Future work — current frustrations and how we’re making them disappear
ABOUT THE SPEAKER:
Ryan is the co-creator of Apache Iceberg and spent the last decade working on big data formats and infrastructure at Netflix, Cloudera, and now Tabular. He is an ASF member and a committer in the Apache Parquet, Avro, and Spark communities.
ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.
FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai/
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520
What Is dbt and Why Are Companies Using It?
And What Do Data Engineers Do On A Daily Basis - Community Update #24
Red Hot: The 2021 Machine Learning, AI and Data (MAD) Landscape
Full resolution version of the landscape image here
It’s been a hot, hot year in the world of data, machine learning and AI.
Just when you thought it couldn’t grow any more explosively, the data/AI landscape just did: rapid pace of company creation, exciting new product and project launch
The Baseline Data Stack - Going Beyond The Modern Data Stack - Part 1
Billions of dollars have been put into investing into companies that fall under the concept of “Modern Data Stack. Fivetran nearly has one billion dollars funding them, DBT has 150 million(and is looking to raise more), Starburst has 100 million(Not considered part of the MDS
Rethinking the Modern Data Stack
A version of this post was originally published as a byline for DEVOPSdigest on November 8,…
Hiflylabs/awesome-dbt: A curated list of awesome dbt resources
A curated list of awesome dbt resources . Contribute to Hiflylabs/awesome-dbt development by creating an account on GitHub.