Skip to content
ArchCrux

Learning series · 6 episodes · Staff / Senior / Principal

The Dashboard Was Green. The System Was Not.

Six connected production investigations into async traces, sampling bias, silent data loss, SLO burn rates, telemetry outages, and incident experiments.

Start series →

Follow the decisions in order

  1. Part 1 of 6

    Staff

    The order completed in 200 ms. Why did the customer wait 40 seconds?

    Find a customer-visible 40-second gap that is absent from a fast synchronous HTTP trace, using messaging context, event timestamps and population SLIs.

  2. Part 2 of 6

    Staff

    p99 doubled after deployment. Why do the sampled traces look fine?

    Diagnose a release-specific tail regression with comparable cohorts, unbiased histograms and tracing aware of head and tail sampling.

  3. Part 3 of 6

    Staff

    Every dashboard is green. Where did 12,000 account updates go?

    Detect silently dropped source updates with freshness, completeness and reconciled source-to-projection version checks.

  4. Part 4 of 6

    Senior

    Checkout is 99.9% reliable. Why did one bad hour consume so much budget?

    Derive actionable multi-window burn-rate alerts from a service-specific request SLI rather than thresholds on average success rate.

  5. Part 5 of 6

    Principal

    The outage began. Why did every graph stop updating at the same time?

    Design independent service checks, bounded collector queues and degraded diagnostic paths without making customer requests depend on exporters.

  6. Part 6 of 6

    Principal

    Five dashboards disagree. What do you do in the first ten minutes?

    Lead the first ten minutes of an ambiguous production incident with customer evidence, controlled changes, safe mitigation and documented hypotheses.