Build benchmarks against GitHub's ubuntu-latest
We built three pinned open-source projects 5 times each on GitHub’s
ubuntu-latest and on backmesh’s backmesh-2 and backmesh-4 spot VMs on
AWS, on 2026-11-18. The figures are medians of the 5 rounds, and every run is
linked below.
backmesh-4 ran the builds 1.4 to 2.1× faster than ubuntu-latest, at 2.8 to
5.5× less per build. At the same 2 vCPUs, backmesh-2 was faster for Docker
(1.5×) and TypeScript (1.3×) but slower for Rust (0.8×), partly because
backmesh’s image installs cargo on first use, inside the measured time
(backmesh#22). It was cheaper
on all three, 3.8 to 6.9× less per build. Each backmesh job waited longer to
start, for its VM to boot.
Results
Section titled “Results”Each cell gives the job’s runtime, its cost a build and its cold start, all
medians. On backmesh, the × after the runtime is how many times faster than
ubuntu-latest it was (under 1 is slower), and after the cost how many times
cheaper.
| Workload | ubuntu-latest2 vCPUs |
backmesh-22 vCPUs |
backmesh-44 vCPUs |
|---|---|---|---|
| Docker k6 v0.54.0 |
148 s $0.018 2 s start |
102 s · 1.5× $0.0026 · 6.9× 40 s start |
71 s · 2.1× $0.0032 · 5.5× 48 s start |
| Rust ripgrep 14.1.1 |
70 s $0.012 2 s start |
83 s · 0.8× $0.0024 · 5.1× 46 s start |
49 s · 1.4× $0.0027 · 4.4× 50 s start |
| TypeScript microsoft/TypeScript v5.6.3 |
56 s $0.006 2 s start |
44 s · 1.3× $0.0016 · 3.8× 42 s start |
35 s · 1.6× $0.0021 · 2.8× 43 s start |
Method
Section titled “Method”- Runtime is a job’s start to its finish, as GitHub records them.
- Cold start is a job’s queueing to its start. backmesh’s runners started
without a cue, so each job waited for its VM to boot. With
backmesh cue, the runner boots before the push, and the median wait was 17 s in our cue benchmark. - Size. The repository was private when the runs were made, so
ubuntu-latesthad 2 vCPUs.backmesh-2has 2 vCPUs and 8 GiB, andbackmesh-44 vCPUs and 16 GiB. - GitHub’s cost is each job’s runtime rounded up to whole minutes, at $0.006 a minute.
- backmesh’s cost is the median runtime plus the median cold start, at its
VM’s spot price and its disk’s. The VM boots during the cold start, so
counting all of it slightly overstates the cost. The spot price is the middle
of the range of the label’s AWS types in us-east-2, each in its cheapest zone:
$0.024 to $0.0308 an hour for
backmesh-2, $0.0497 to $0.07 forbackmesh-4. The prices are from the anycloud catalog on 2026-11-13, the same snapshot as thebackmesh-4prices on the cost page. The disk is 100 GB of gp3 at 4,000 IOPS and 500 MiB/s, about $0.0384 an hour. It leaves out backmesh’s control plane, about $86 a month on AWS. - Ratios compare the unrounded runtimes and costs, so a ratio can differ slightly from one worked out from the rounded figures shown.
- Caches. Neither side used a build cache, except setup-node’s default npm cache, on both.
- Toolchains.
ubuntu-latesthas Rust preinstalled. backmesh’s image doesn’t: its first-use handler installs Ubuntu’scargopackage when the job first calls it, inside the measured time.
The workflow
Section titled “The workflow”benchmarks/builds/bench.yml
has a job for each workload. The repository variable BENCH_RUNNER picks the
runner, and a push to the bench branch starts a round.
name: benchon: push: branches: [bench]permissions: contents: readjobs: typescript: runs-on: ${{ vars.BENCH_RUNNER }} steps: - uses: actions/checkout@v5 with: { repository: microsoft/TypeScript, ref: v5.6.3 } - uses: actions/setup-node@v5 with: { node-version: 22 } - run: npm ci --no-audit --no-fund - run: npx hereby local rust: runs-on: ${{ vars.BENCH_RUNNER }} steps: - uses: actions/checkout@v5 with: { repository: BurntSushi/ripgrep, ref: 14.1.1 } - run: cargo build --release --locked docker: runs-on: ${{ vars.BENCH_RUNNER }} steps: - uses: actions/checkout@v5 with: { repository: grafana/k6, ref: v0.54.0 } - run: docker build -t bench .Each round is one run of the workflow, its three jobs in parallel, in backmesh/benchmarks.
| Runner | Round 1 | Round 2 | Round 3 | Round 4 | Round 5 |
|---|---|---|---|---|---|
ubuntu-latest |
44364487787 | 44365905066 | 44367215179 | 44368389022 | 44369659562 |
backmesh-2 |
44379996886 | 44380630467 | 44381256875 | 44381793669 | 44382248237 |
backmesh-4 |
44365142077 | 44366607804 | 44367828921 | 44369100505 | 44370343686 |