Learning series · 6 episodes · Senior / Staff / Principal
The Batch Finished. What Did We Actually Produce?
Six technical episodes on shuffle, skew, joins, data layout, atomic snapshots and trustworthy analytics.
Start series →Follow the decisions in order
- Part 1 of 6
Senior
Ten terabytes went into the job. Why is the shuffle taking longer than the scan?Explain the all-to-all exchange and distinguish input volume from shuffled bytes, with safe combinable pre-aggregation.
- Part 2 of 6
Staff
Nine hundred ninety-nine tasks finished. Why is the last one still running?Diagnose a slow executor versus one oversized key and select speculation or workload restructuring accordingly.
- Part 3 of 6
Staff
The lookup table was tiny. Why did the join move four terabytes?Choose distributed broadcast versus shuffle join from post-filter size, per-worker memory and source cardinality.
- Part 4 of 6
Staff
The dataset is only 200 GB. Why does opening it take longer than reading it?Explain why high file counts hurt planning and reads, and how snapshot-safe compaction and writer layout repair it.
- Part 5 of 6
Staff
Five hundred files were uploaded. Why can readers still see yesterday's table?Describe multi-file atomic visibility through immutable data, manifests and catalog compare-and-swap, including unknown commits.
- Part 6 of 6
Principal
Every pipeline task succeeded. Why is a week of revenue missing?Separate job success from semantic and business data correctness, gate publication, and recover through versioned lineage.