It ran every day. It stopped working in March.
The scary failure is the one that pages you at 2am. The expensive one is the job that keeps running, keeps exiting zero, keeps producing output — and quietly stopped doing the thing it was for. Below are ninety days of a scheduled job's life, four different ways. Every day passes the check the job actually has. Pick the day you think it died, then see how long it went on reporting success.
Honest-AI note. No model here. The ninety days are simulated live from the rules in the engine — deterministic, so every reader sees the same timeline and the page's own tests can assert against it. The claim being demonstrated is computed, not written down: for all four rots, the health check almost everyone actually has (did it run, did it exit cleanly) never fires — not once in ninety days. This one is closest to the ops doctrine in Edition 009, from the other side: 009 is about alerting on failure, this is about failure that never produces one.
The rot
Plain-language key (rot, health check, backfill, backlog age, deterministic simulation)
- Rot
- A failure that leaves the job running and reporting success. It has no moment of alarm, so it is found by accident, usually much later.
- Health check
- Whatever the monitoring actually asserts. "Ran and exited zero" is the common one and is true of every dead job here.
- Backfill
- The pass that applies a schema change to rows that already exist. Skipping it is invisible until something aggregates across the change.
- Backlog age
- How old the oldest unresolved item is. A daily success rate can look perfect while this grows without limit.
- Deterministic simulation
- The timeline is generated from a fixed seed, so it is identical for every reader and can be asserted against in tests.