Found 2 bookmarks
Custom sorting
Smart Data Pipeline Design: Check for Delta Table Changes with Minimal Overhead
Smart Data Pipeline Design: Check for Delta Table Changes with Minimal Overhead
Scenario I have a notebook that processes hot data every 5 minutes. Meanwhile, another pipeline processes historical data, and I want to create a summary table that uses the hot data incrementally …
fabricfabric#data engineering#delta lake#incremental updates#data processing#spark#batch processing#optimization#data pipeline
Power_BI_tips·datamonkeysite.com·
Smart Data Pipeline Design: Check for Delta Table Changes with Minimal Overhead