Data Engineering

45 bookmarks
Custom sorting
🫡🐳 pedramdb🫡🐳 on Twitter
🫡🐳 pedramdb🫡🐳 on Twitter
“Does anyone here (not vendors) work with CDPs, either traditional or unbundled as part of a data team? What’s the experience been like? How much input did you have in the process ?”
·twitter.com·
🫡🐳 pedramdb🫡🐳 on Twitter
Data Systems Tend Towards Production
Data Systems Tend Towards Production
Data teams have substantially larger influence than a decade ago. The surface area of what can go wrong has grown just as fast.
·ian-macomber.medium.com·
Data Systems Tend Towards Production
Airbyte Monitoring with dbt and Metabase - Part I | Airbyte
Airbyte Monitoring with dbt and Metabase - Part I | Airbyte
How to implement an Airbyte Monitoring Dashboard with dbt and Metabase on a locally deployed instance to get an operational view and high-level overview.
·airbyte.com·
Airbyte Monitoring with dbt and Metabase - Part I | Airbyte
Building a Data Engineering Project in 20 Minutes
Building a Data Engineering Project in 20 Minutes
You'll learn web-scraping with real-estates, uploading them to S3, Spark and Delta Lake, adding Data Science with Jupyter, ingesting into Druid, visualising with Superset and managing everything with Dagster.
·sspaeti.com·
Building a Data Engineering Project in 20 Minutes
The Contract-Powered Data Platform | Buz
The Contract-Powered Data Platform | Buz
The contract-powered data platform is a step towards improving data quality, reducing organizational friction, and automating the toil data teams face. Here's what it looks like and how it works.
·buz.dev·
The Contract-Powered Data Platform | Buz
Engineers Shouldn’t Write ETL: A Guide to Building a High Functioning Data Science Department | Stitch Fix Technology – Multithreaded
Engineers Shouldn’t Write ETL: A Guide to Building a High Functioning Data Science Department | Stitch Fix Technology – Multithreaded
“What is the relationship like between your team and the data scientists?” This is, without a doubt, the question I’m most frequently asked when conducting i...
There is nothing more soul sucking than writing, maintaining, modifying, and supporting ETL to produce data that you yourself never get to use or consume. Instead, give people end-to-end ownership of the work they produce (autonomy). In the case of data scientists, that means ownership of the ETL.
Mediocre engineers really excel at building enormously over complicated, awful-to-work-with messes they call “solutions”. Messes tend to necessitate specialization.
most technologies have evolved to a point where they can trivially scale to your needs.
·multithreaded.stitchfix.com·
Engineers Shouldn’t Write ETL: A Guide to Building a High Functioning Data Science Department | Stitch Fix Technology – Multithreaded
Viewpoint | dbt Docs
Viewpoint | dbt Docs
In 2015-2016, a team of folks at RJMetrics had the opportunity to observe, and participate in, a significant evolution of the analytics ecosystem. The seeds of dbt were conceived in this environment, and the viewpoint below was written to reflect what we had learned and how we believed the world should be different. dbt is our attempt to address the workflow challenges we observed, and as such, this viewpoint is the most foundational statement of the dbt project's goals.
·docs.getdbt.com·
Viewpoint | dbt Docs
We the purple people
We the purple people
The data world needs more purple people — generalists who can navigate both the business context and the modern data stack. Let's put aside skillset dichotomies, and learn to feel comfortable in the space between.
·getdbt.com·
We the purple people
The end of Big Data
The end of Big Data
Databricks, Snowflake, and the end of an overhyped era.
Take real-time products, for example. Most businesses have little use for true real-time experiences. But, all else being equal, real-time data is better than latent data. We all have dashboards that update a little too slowly, or marketing emails we wish we could send a little sooner. While these annoyances don’t justify the effort currently required to build real-time pipelines, they do cause small headaches. But if someone came along and offered me a streaming Fivetran, or a reactive version of dbt, I’d take it. If the cost of a real-time architecture was low enough, regardless of the shoehorned use-cases, there’d be no reason to turn it down. And just as we came to rely on Snowflake after we chose it as a better Postgres, I’m certain we’d come to rely on streaming pipelines if they replaced our current batch ones. We’d start doing more real-time marketing outreach, or build customer success workflows around live customer behavior. Over the next five years, I’d guess that real-time data tools follow this exact path: They’ll finally go mainstream, not because we all discover we need them, but because there will be no reason not to have them. And once we do, we’ll find ways to push it to their limits, just as we did with fast internet connections and powerful browsers.
·benn.substack.com·
The end of Big Data