Status

backmesh status [--json]

It shows:

• Pools: each pool’s state, VM count and limits, whether it’s privileged, and its last error, such as no spot capacity or a quota limit.
• Queue, per label: GitHub’s assigned and running jobs, backmesh’s runners queued, starting and running, the oldest queued runner’s age, and the reasons runners are short.
• Jobs, per label, over 24 hours and 7 days: how many completed, the p50 and p95 time from queued to started with the share of it spent at its slowest stage (at GitHub, in the listener, waiting for capacity or starting the runner), the p50 and p95 run time, and how many were reclaimed or failed to start.
• Cost: the installation’s month-to-date cost, by cloud.
• Checks: the GitHub App key’s age, whether bm-control runs anything but the listener, and other warnings.

  backmesh-4: 5 assigned, 2 running; runner Jobs 2 queued, 0 starting, 1 running
    24 h: 42 jobs; queued to started p50 21 s, p95 1.6 min, 61 % waiting for capacity; run p50 4.2 min, p95 11.0 min; 1 reclaimed, 0 startup failures
    7 d: 288 jobs; queued to started p50 19 s, p95 1.4 min, 58 % waiting for capacity; run p50 4.0 min, p95 10.2 min; 3 reclaimed, 0 startup failures

The figures are illustrative. The listener keeps the job rows in memory, so a restart or upgrade empties them. Each completed job also writes one job completed JSON line to the listener’s log.

In each job’s log

When a job finishes, the last lines of its log give the resources it used: peak memory against the size’s limit, OOM kills, average and peak CPU, and the work directory’s peak disk. Use them to pick a size.

Spot reclaims

When a cloud reclaims a spot VM, the jobs on it fail. When you have to rerun them, backmesh posts one neutral check run per workflow run, named reclaims in run <id>, on the run’s commit. It lists every reclaimed job with its VM and pool, and says to rerun the failed jobs from the Actions tab once the run finishes. backmesh doesn’t rerun them itself yet.