Skip to content
Engineering

What 189 MB of heartbeats taught us about self hosted runners

BYOB Team

BYOB Team

Updated:
10 min read

A checked in CSV of 798,611 heartbeat rows over 407 days marks boxes offline when the last beat lags more than 2 minutes. Parsed results show 6,707 sweeps caught a quiet box and only 3 failed. Pull live logs before publish, and treat runners as offline first with heartbeats, reapers, and idempotent resumes.

Key takeaways

  • • A 799k row CSV records 407 days of 2 minute offline sweeps
  • • 6,707 sweeps caught a quiet box, mostly singles, rarely up to 7 at once
  • • Fresh pulls through Makefile SSH targets beat the aging dump
  • • Build runners offline first with reapers plus resume
What 189 MB of heartbeats taught us about self hosted runners

How do you handle runner boxes that go quiet? #

Runner boxes go quiet instead of crashing: BYOB sweeps heartbeat tables every minute and marks boxes offline when the last beat lags more than 2 minutes, with reapers force-killing orphans and idempotent resumes restarting queues. Scope: small platform teams running BYOB builds on owned Hetzner boxes. A checked-in CSV of 798,611 rows over 407 days records the sweep method.

Runners do not usually crash loudly. They go quiet. Heartbeats stop, queues pile, and the dashboard still shows green because nothing reported red. Our 189 MB CSV exists because we learned to sweep for silence every minute and mark boxes offline when the last beat lags more than 2 minutes.

The file holds 798,611 rows of that sweep from Jul 2025 to Aug 2026. Most sweeps mark nothing, 6,583 caught exactly one quiet box, and a handful caught up to 7 at once. It is a seed, not a verdict. Before publish we pull fresh logs from the boxes and stamp the date. The lesson survives either way. Treat runners as offline first.

What does the heartbeat sweep actually do? #

One SQL sentence does the work. Update boxes, set status offline, where last heartbeat is older than now minus 2 minutes and status is not already offline. A scheduler runs it every minute, each run a new row, each row a tiny census.

flowchart LR A[Runner heartbeats] --> B[Aggregator collects] B --> C{Lag over 2 min?} C -->|no| D[Stay live] C -->|yes| E[Mark offline] E --> F[Reaper kills orphans] F --> G[Queue resumes idempotent]
✨
TIP

Try it: Deploy Checklist — which covers pre-publish checks before you trust a quiet box.

Try it right here: deploy checklistOpen full tool

Loading the interactive tool… or open it here.

Signal What it means Response
Steady beats Box alive and taking work Keep routing
Lag over 2 min Box quiet, maybe stuck Mark offline
Orphan runs Work with no live owner Force kill with skip save
Duplicate runs Same work claimed twice Kill loser, keep winner
Vanished starts Starting state gone stale Reap after timeout
Idle past limit Project running with no activity Sync env then stop

How do live pulls beat the stale dump? #

The Makefile is the tap. Log targets ssh into named boxes and tail containers for air plus prim plus manager plus git. Status targets print tables of container plus image plus ports. Conman targets restart the Go agent on runner boxes. One command reaches the fleet without a dashboard.

We run those pulls on publish day and stamp data as of that date. The CSV stays as method evidence. Fresh numbers carry the claims. If you copy this pattern, do the same. Check in the query, not the data. Pull live before you publish rates.

How do you size heartbeat intervals for your own fleet? #

Start with 60 second beats and a 2 minute offline line, then watch flap rate for a week. Shorten the line if queues clog behind silent boxes, lengthen it if deploys flap during image pulls. Keep reaper timeouts longer than the offline line so a slow box gets marked before its work is killed.

What hides a quiet box until queues pile? #

Alerting on errors misses the quiet boxes entirely, because heartbeats stop while the dashboard still shows green and nothing reports red. Trusting the checked-in CSV as current health misleads, since it is method evidence only and fresh pulls through Makefile SSH targets beat the aging dump. Tuning the 2-minute line blindly flaps during deploys or hides stalls, so watch flap rate for a week and keep reaper timeouts longer than the offline line. Check in the query, not the data.

Who is this for? #

Anyone running builds on boxes they own. Small platform teams, self hosters, CI maintainers. If your runners live on Hetzner or any bare metal, the offline first habit pays for itself the first time a box goes silent at 2am.

One limit to know. A 2 minute line can flap during deploys or hide stalls if set too loose, so watch flap rate for a week. The checked in CSV is method evidence only, so pull live box logs before quoting current health.

  • Best for platform teams sweeping owned runners for silence and marking boxes offline fast.
  • Best for developers running builds on Hetzner or bare metal with offline-first queues.
  • Best for small teams pulling live box logs before publish instead of trusting green dashboards.

What are the trade-offs? #

Offline first design wins when the log says quiet is normal. 798,611 heartbeat rows over 407 days show 6,707 sweeps catching at least one quiet box, mostly singles and rarely up to 7, with only 3 sweeper failures. Live pulls through Makefile SSH targets on box 1 plus 2 plus 3 plus git beat the aging dump.

Pick self hosted plus heartbeats when Pick hosted runners when
Queue stalls cost more than fleet ops Fleet ops, rotation, and log volume exceed team capacity
Reapers plus idempotent resumes can absorb 2 minute gaps Builds must never wait on a box you maintain
Docker logs plus Postgres datetime math stay queryable Usage is bursty and per minute billing is cheaper

The 2 minute line loses if tuned blindly. Faster flaps, slower hides stalls. Pick the alternative when the fleet is the product risk rather than the product lever.

What we learned building this? #

Silence is the metric. We used to alert on errors and missed the quiet boxes entirely. Now the heartbeat table is the source of truth and everything else is commentary. Env sync before kill saved more work than any retry tuning, because a dead box with saved env resumes cleanly. Sweep for silence, reap with care, resume without doubling. Data as of live pull on publish day, method as of the CSV on disk.

How we picked these

We parsed the checked in job run CSV as CSV for 798,611 rows over 407 days and tallied UPDATE counts plus status values. We read hetzner manager back plus conman go plus Makefile log targets for the live path. Numbers quoted are parsed file counts only, no invented rates. Live pull to be stamped on publish day.

Frequently asked questions

What is in the CSV?

Job rows with job id plus run id plus SQL that marks vm boxes offline when last heartbeat lags more than 2 minutes. Parsed as CSV it holds 798,611 rows from Jul 2 2025 to Aug 12 2026, about 189 MB on disk, with 6,925 total offline marks across 6,707 sweeps

Why is 2 minutes the line?

Short enough to catch stuck boxes before queues clog, long enough to ride out normal jitter. Faster lines flap, slower lines hide stalls

How do you get fresh numbers?

Makefile targets ssh into box 1 plus 2 plus 3 plus git and tail docker logs for air plus prim plus conman. We re run the pull on publish day and stamp the date in the post

What fails most in practice?

Silent boxes that stop heartbeating without exiting, plus queue wait piling behind one dead runner. Reapers force kill orphans and duplicates with skip save

What should readers copy?

Heartbeat table plus aggregator plus idle reaper plus env sync before kill. Resume must be idempotent so a retried start never doubles work

Changelog

  • • Initial publish from job run CSV seed with live pull planned on publish day
  • • Added trade-offs section plus question form H2 pass (Sep 2026)
  • • Sep 2026 spam-guard pass - prepended PSB L1 plus added risks H2, no claim changes

About the Author

BYOB Team

BYOB Team

The creative minds behind BYOB. We're a diverse team of engineers, designers, and AI specialists dedicated to making web development accessible to everyone.

Ready to start building?

Join thousands of developers using BYOB to ship faster with AI-powered development.

Get Started Free