Every pipeline task succeeded. Why is a week of revenue missing?

Successful computation does not imply correct meaning

A task can process every input record and faithfully compute the wrong expression. Neither the scheduler nor storage engine knows when a payment amount changes units or moves to a new field.

We need to distinguish several meanings of success. Operational success means the scheduled computation ran without infrastructure errors. Structural correctness means records match expected types, required fields and relationships. Business correctness means the resulting amounts and classifications preserve the intended meaning. Completeness means the output covers the agreed input population and time range.

The finance incident is not a timeout, a partial object-storage commit or a missing stream watermark. The data is there. The semantic interpretation failed.

Begin with the source contract

Ask what the partner actually promised. The payment event should have a stable transaction ID, currency, amount representation, payment state, event timestamp and agreed version. If amount_minor_units appears, the consumer must not continue reading amount_major_units and treat null as zero.

Preserve raw input or a durable canonical representation, schema version, source ID and batch. If conversion discarded original facts, a full correction may be impossible.

Validate required fields, accepted states, currencies, uniqueness and type constraints. But don't stop there. A schema can remain syntactically valid while the meaning of a field changes from dollars to cents. Versioned semantics and explicit unit conventions belong in the contract as much as JSON field names do.

Tools such as dbt support generic tests for not-null values, uniqueness, accepted values and relationships, while allowing custom assertions. These do not replace the business contract. dbt data tests

The missing week should be detectable in numbers

I would monitor volume and completeness by partner, day, region and event type. For example, if one partner normally sends 2 million successful payment events per day and yesterday contained 1.9 million, volume alone might look plausible. But if 95 percent have null amounts, a partner-specific null-rate check should fail immediately.

A dashboard can also mislead when totals look plausible but distribution changes. Imagine the largest partner drops out just as another partner has a promotion. The global revenue figure might remain close to its usual value. Checks at only the grand-total level would miss the problem.

I would therefore validate both row-level invariants and group-level expectations. Row-level tests include valid amounts and unique transaction IDs. Group-level checks include counts, total value, null rates, unknown payment states and differences against an independent settlement feed when available.

Thresholds need thought. An alert saying revenue must equal yesterday's revenue would fail every real holiday or campaign. A zero-tolerance null rule may be appropriate for a required field, while a volume anomaly should use expected business variation and a declared delay window. Separate a source that is late from a source that is semantically invalid. Both need attention, but the repair paths differ.

Don't publish merely because the job turned green

For this system I would load data into a candidate snapshot or staging table and run validations against that candidate. The promotion controller should publish it only when the required checks pass or a clearly authorized exception is recorded.

If one partner's feed is invalid, the team may block the table or publish an explicitly marked partial result. Which option is acceptable depends on whether finance uses the number for settlement, a live marketing estimate or a retrospective report. Silently dropping the partner and reporting the output as complete is not acceptable.

Record input ranges, source versions, transformation version, validation results and output snapshot ID to identify affected dates and revisions. It also lets downstream consumers distinguish a corrected historical value from a new sale happening today.

Correct the week without inventing extra revenue

Once the partner restores the field contract or we implement a version-aware parser, I would reprocess the retained raw records for the affected interval into a new output generation. The corrected generation should replace the affected derived values, not append a second copy of those revenue totals.

Validate the new result by comparing partner totals, transaction counts, independent settlement evidence and a sample of affected events. Then publish the corrected table version atomically using the lakehouse commit protocol from the previous lesson. Readers can see one coherent updated version rather than a mix of repaired and unrepaired partitions.

There may be downstream reports, exports and materialized views that already incorporated the wrong numbers. Updating the source table does not automatically repair every downstream copy. The recovery plan should identify consumers of the incorrect versions, trigger their recomputation where necessary, and communicate that prior reports were revised.

In an interview, this is the point where I would challenge the product requirement. Does the dashboard need to be fresh at all costs, or accurate enough to support settlement? Those are different contracts, and a reliable platform must be willing to show incomplete instead of publishing a confident wrong number.

To extend the reasoning, sketch an analytics service whose ingestion, validation, serving, cost and recovery contracts remain visible end to end. Keep its semantic-quality gate separate from the scheduler and storage commit mechanisms.