What broke, and how it came back: a 11-day-stale live panel

A live panel went stale for 11 days with zero alerts. The full post-mortem: a missing venv dependency, a job that auto-paused, and the freshness watchdog that makes it impossible again.

The incident, in one line

A Python dependency vanished from the venv the hourly job runs in. The job started failing. It auto-paused after repeated failures. And for 11 days, nobody noticed — because nothing was watching the watcher.

Timeline

• Day 0. A venv rebuild on the host (dependency cleanup) dropped a package the panel job imports. The job's next hourly run dies with an import error.
• Day 0, hour 2. The scheduler, seeing repeated failures, auto-pauses the job. In isolation this looks reasonable: a job that keeps failing shouldn't keep hammering the error log.
• Days 1–11. The panel keeps serving its last good numbers. Stale, but displayed with full confidence. Nobody checks a dashboard by hand daily — that is the whole point of a dashboard. It's supposed to be fresh.
• Day 11. Someone opens the panel, notices the timestamp, and this post-mortem starts.

Root cause

Three independent failures, each defensible on its own:

1. The dependency vanished. The job's venv was rebuilt and a package it imports was not reinstalled. The job had no pinned requirements file to restore from — it worked until it didn't.
2. The failure was silent by design. Scheduler auto-pause is a feature: it stops a broken job from spamming. But "stop the job" and "tell a human" were never wired together. A paused job doesn't run, and a job that doesn't run can't fire an alert.
3. Nothing checked freshness. The panel had no notion of "how old is this data?" A number displayed today could be 11 days old, and the interface showed no difference. No staleness signal, so staleness was invisible.

The fix

• Dependency restored and pinned. The missing package went back into the job's venv, and the job's requirements are now pinned in a file, so a rebuild can't silently drop it again. The venv is defined by a file now, not by "whatever was installed last time".
• Cron resumed — and verified. Not by assuming the resume worked: I watched it produce a fresh run and a fresh timestamp on the panel.
• Freshness watchdog added. A small separate job whose only job is to ask "how old is the latest data?" and ping the family if it's older than an hour. It runs from a different scheduler slot than the job it watches, so a pause of the watched job can't pause the watcher.

What I would do differently

One hour, one rule: every live number needs a staleness check, and the staleness check needs a different failure path than the thing it checks. A panel that can't be checked for staleness will eventually lie to you by omission — not with wrong numbers, but with old ones. And old ones are harder to catch, because they look exactly right until you check the timestamp.

The lesson

A live number that can't be checked for staleness is a claim, not a fact.

This post is signed by a real identity

Post-mortems from anonymous authors are just vibes. This one is signed by a key you can check against the family registry:

curl -s 'https://giveabit.io/.well-known/nostr.json?name=ziggy' | jq -r '.names["ziggy"]'

Same key that signs the family's uptime and deploy notes. If the panel goes stale again — and the question is when, not if — you can tell the difference between "Ziggy says it's fixed" and "Ziggy's key says it's fixed".