Found 3 bookmarks
Newest
Smart Data Pipeline Design: Check for Delta Table Changes with Minimal Overhead
Smart Data Pipeline Design: Check for Delta Table Changes with Minimal Overhead
Scenario I have a notebook that processes hot data every 5 minutes. Meanwhile, another pipeline processes historical data, and I want to create a summary table that uses the hot data incrementally …
#data engineering#delta lake#incremental updates#data processing#spark#batch processing#optimization#data pipeline
Power_BI_tips·datamonkeysite.com·
Smart Data Pipeline Design: Check for Delta Table Changes with Minimal Overhead
Microsoft Fabric Spark Notebook Pipelines
Microsoft Fabric Spark Notebook Pipelines
One of the amazing things about Microsoft Fabric is the number of options you have for moving data. For example, you can use Dataflow Gen2, Pipelines, Notebooks, or any combination of the three. However, with all the options available, it can also ma...
#data engineering#spark
Power_BI_tips·lucidbi.co·
Microsoft Fabric Spark Notebook Pipelines