How do you handle runner boxes that go quiet? #
Runner boxes go quiet instead of crashing: BYOB sweeps heartbeat tables every minute and marks boxes offline when the last beat lags more than 2 minutes, with reapers force-killing orphans and idempotent resumes restarting queues. Scope: small platform teams running BYOB builds on owned Hetzner boxes. A checked-in CSV of 798,611 rows over 407 days records the sweep method.
Runners do not usually crash loudly. They go quiet. Heartbeats stop, queues pile, and the dashboard still shows green because nothing reported red. Our 189 MB CSV exists because we learned to sweep for silence every minute and mark boxes offline when the last beat lags more than 2 minutes.
The file holds 798,611 rows of that sweep from Jul 2025 to Aug 2026. Most sweeps mark nothing, 6,583 caught exactly one quiet box, and a handful caught up to 7 at once. It is a seed, not a verdict. Before publish we pull fresh logs from the boxes and stamp the date. The lesson survives either way. Treat runners as offline first.
What does the heartbeat sweep actually do? #
One SQL sentence does the work. Update boxes, set status offline, where last heartbeat is older than now minus 2 minutes and status is not already offline. A scheduler runs it every minute, each run a new row, each row a tiny census.
Try it: Deploy Checklist — which covers pre-publish checks before you trust a quiet box.
| Signal | What it means | Response |
|---|---|---|
| Steady beats | Box alive and taking work | Keep routing |
| Lag over 2 min | Box quiet, maybe stuck | Mark offline |
| Orphan runs | Work with no live owner | Force kill with skip save |
| Duplicate runs | Same work claimed twice | Kill loser, keep winner |
| Vanished starts | Starting state gone stale | Reap after timeout |
| Idle past limit | Project running with no activity | Sync env then stop |
How do live pulls beat the stale dump? #
The Makefile is the tap. Log targets ssh into named boxes and tail containers for air plus prim plus manager plus git. Status targets print tables of container plus image plus ports. Conman targets restart the Go agent on runner boxes. One command reaches the fleet without a dashboard.
We run those pulls on publish day and stamp data as of that date. The CSV stays as method evidence. Fresh numbers carry the claims. If you copy this pattern, do the same. Check in the query, not the data. Pull live before you publish rates.
How do you size heartbeat intervals for your own fleet? #
Start with 60 second beats and a 2 minute offline line, then watch flap rate for a week. Shorten the line if queues clog behind silent boxes, lengthen it if deploys flap during image pulls. Keep reaper timeouts longer than the offline line so a slow box gets marked before its work is killed.
What hides a quiet box until queues pile? #
Alerting on errors misses the quiet boxes entirely, because heartbeats stop while the dashboard still shows green and nothing reports red. Trusting the checked-in CSV as current health misleads, since it is method evidence only and fresh pulls through Makefile SSH targets beat the aging dump. Tuning the 2-minute line blindly flaps during deploys or hides stalls, so watch flap rate for a week and keep reaper timeouts longer than the offline line. Check in the query, not the data.
Who is this for? #
Anyone running builds on boxes they own. Small platform teams, self hosters, CI maintainers. If your runners live on Hetzner or any bare metal, the offline first habit pays for itself the first time a box goes silent at 2am.
One limit to know. A 2 minute line can flap during deploys or hide stalls if set too loose, so watch flap rate for a week. The checked in CSV is method evidence only, so pull live box logs before quoting current health.
- Best for platform teams sweeping owned runners for silence and marking boxes offline fast.
- Best for developers running builds on Hetzner or bare metal with offline-first queues.
- Best for small teams pulling live box logs before publish instead of trusting green dashboards.
What are the trade-offs? #
Offline first design wins when the log says quiet is normal. 798,611 heartbeat rows over 407 days show 6,707 sweeps catching at least one quiet box, mostly singles and rarely up to 7, with only 3 sweeper failures. Live pulls through Makefile SSH targets on box 1 plus 2 plus 3 plus git beat the aging dump.
| Pick self hosted plus heartbeats when | Pick hosted runners when |
|---|---|
| Queue stalls cost more than fleet ops | Fleet ops, rotation, and log volume exceed team capacity |
| Reapers plus idempotent resumes can absorb 2 minute gaps | Builds must never wait on a box you maintain |
| Docker logs plus Postgres datetime math stay queryable | Usage is bursty and per minute billing is cheaper |
The 2 minute line loses if tuned blindly. Faster flaps, slower hides stalls. Pick the alternative when the fleet is the product risk rather than the product lever.
What we learned building this? #
Silence is the metric. We used to alert on errors and missed the quiet boxes entirely. Now the heartbeat table is the source of truth and everything else is commentary. Env sync before kill saved more work than any retry tuning, because a dead box with saved env resumes cleanly. Sweep for silence, reap with care, resume without doubling. Data as of live pull on publish day, method as of the CSV on disk.