--- name: backmesh description: Set up backmesh, self-hosted GitHub Actions runners on spot VMs in the user's own AWS, GCP or Azure accounts, then use it. Use when asked to install, use, check or remove backmesh, to move GitHub Actions jobs onto backmesh runners, or at least a minute ahead of every `git push`, when the pre-push checks start, in a repo whose workflows use `runs-on: backmesh-*`. --- # backmesh backmesh runs each GitHub Actions job on its own spot or on-demand VM in the user's own clouds. One `backmesh install` sets it up for one GitHub org. The docs at https://backmesh.com/install/ say more; when they and this file disagree with the CLI, follow the CLI's messages and `backmesh --help`. ## Rules - Stop and ask the human at every point marked **Human**. Never answer a prompt for them, pipe input into one, or work around one. - Never pass `--yes` unless the human has read the plan it agrees to and told you to. - Never ask for, paste, print or commit secrets: cloud keys, GitHub tokens, or what is in `~/.backmesh`. backmesh needs none of them copied; its sign-ins are saved for it. - In a job's steps, never prune or remove all containers, as with `docker system prune`, `docker rm -f $(docker ps -aq)` or `docker kill $(docker ps -q)`. Each job gets its own VM and controls that VM's Docker, and the runner's own container is on it. Remove only the containers the job started. ## 1. Check the prerequisites 1. **Human:** confirm who owns the GitHub org. The org owner opens one link during install to sign in with GitHub and install backmesh's GitHub App on the org. Only the org's private repositories can use the runners. An org that restricts OAuth apps must approve backmesh first, as https://backmesh.com/install/#before-you-start says. 2. Check the cloud CLI for each cloud the human wants, and ask the human to sign in with the fix when one fails: | Cloud | Check | Fix | | ------- | -------------------------------- | ------------------- | | `aws` | `aws sts get-caller-identity` | `aws sso login` | | `gcp` | `gcloud auth print-access-token` | `gcloud auth login` | | `azure` | `az account show` | `az login` | For `aws`, also check that `aws configure get region` or `$AWS_REGION` gives a region, or ask for `aws configure set region `. For `gcp`, check that `gcloud config get-value project` prints a project, or ask for `gcloud config set project `. Install creates the control plane and each cloud's role through these CLIs, as the human, so the signed-in identity needs the rights https://backmesh.com/install/#before-you-start lists. 3. Get the `backmesh` CLI for macOS or Linux as https://backmesh.com/install/#before-you-start says. Download the platform's `backmesh-.tar.gz` and its release's `checksums.txt` into one directory, and check the archive before you unpack it: ```sh sha256sum --ignore-missing -c checksums.txt # Linux shasum -a 256 --ignore-missing -c checksums.txt # macOS ``` Stop if the check fails. Then run `backmesh --version`. ## 2. Install Ask the human for the org, the control plane's cloud and region, and the runners' clouds. Show them the plan first; it signs in to nothing and changes nothing: ```sh backmesh install --org --control-plane aws:us-east-2 --runners aws --runners gcp --plan ``` `--control-plane :` is where the management VM and the listeners run. `--runners ` repeats, one per cloud: alone it uses that cloud's default regions, and `--runners aws:us-east-2,us-west-2` uses exactly those. Without either flag, both go on the signed-in cloud in US East. The plan prints each cloud's runner regions, what a `backmesh-4` VM costs an hour on spot and on demand, how many jobs the vCPU quotas hold at once, what the control plane costs a month, and the install command with every choice spelled out. **Human:** the human reads the plan and runs the install, without `--plan`, in a terminal they can see and type into. If they tell you to run the printed `--yes --plan-id` command instead, run it and show them every line it prints. On the way, install stops for the human: 1. **Human:** at a terminal, install asks `Proceed?` before anything is billed. When adding an org to a control plane that already runs other orgs, it asks whether they all have the same owner; orgs with different owners need separate installations. Only the human answers. 2. **Human:** install prints a `https://backmesh.com/connect/` link and its code. The org owner opens it on any device, checks that the page shows the same code, signs in with GitHub and installs backmesh's GitHub App on the org. There is nothing to copy back. Install ends by printing the label to use, then runs `backmesh status`. If it stops partway, fix the cause it names and run the same command again: it reuses what it already created. ## 3. Use the labels Set a job's `runs-on` to a label. Ask the human before changing workflows that other people depend on. ```yaml jobs: test: runs-on: backmesh-4 ``` - Sizes: `backmesh-2`, `backmesh-4` (the default), `backmesh-8` and `backmesh-16`, at 4 GiB of memory per vCPU. - The control plane's cloud has the plain labels. Each other cloud's labels name it: `backmesh-gcp-4`, `backmesh-azure-4`. A job never moves to another cloud. - Spot is the default. Use the on-demand labels, such as `backmesh-od-4`, for long jobs on the critical path; they cost about 2 to 5 times spot. When spot reclaims were a run's only failures, backmesh reruns its failed jobs on demand by itself, at most twice; GitHub may still put a rerun on spot. `runs-on: [backmesh-4, backmesh-noretry]` turns that off for a job. - Jobs that need `/dev/kvm`, for microVMs or emulators, use `backmesh-kvm-2` to `backmesh-kvm-16`, or `backmesh-kvm-od-4` and so on on demand. A cloud whose regions lack nested virtualization gets no KVM labels. - A container a job starts runs on the job's VM: mount files from `$GITHUB_WORKSPACE` or `$RUNNER_TEMP`. The full list and the VM types are at https://backmesh.com/labels/. ## 4. Cue each push In a repo whose workflows use backmesh labels, run `backmesh cue --in ` at least a minute ahead of every `git push`, when your pre-push checks, such as the tests or the build, start. Pass the seconds until you expect to push: ```sh backmesh cue --in 120 ``` It warns the org's listener that a push is coming, so the run's first jobs start sooner. `--in` takes 30 to 600. Right before a push is too late: under 30 seconds, it sends nothing. It takes the repo from `origin` and the current branch; pass `--branch ` when you'll push another branch, and `--repo ` when `origin` isn't the GitHub repo. If you won't push soon after all, run `backmesh cue --cancel`. It gives up after 3 seconds and never fails the push: when it can't send, it says why, and the run's jobs start as they would without it. Push even if `backmesh cue` fails. When it says it isn't signed in, run `backmesh cue --sign-in --in `. **Human:** a GitHub user who can push to the repo opens `https://github.com/login/device` on any device, enters the printed code, and signs in. In a sandbox or cloud environment that starts fresh each time, don't sign in there. **Human:** copy the sign-in from a machine where backmesh is signed in (`~/.backmesh/anycloud/.token`) into the environment's secrets as `BACKMESH_CUE_TOKEN`; `backmesh cue` uses it before any saved sign-in. It is the human's full backmesh sign-in, not a cue-only key, and a later sign-in on that machine can revoke it. Treat it as a secret: never print it, paste it into chat, or commit it. So that later sessions and other agents keep doing this, ask the human whether to add this line to the repo's `AGENTS.md` or `CLAUDE.md`, and add it only if they agree: ```md Run `backmesh cue --in ` (60 to 600) at least a minute ahead of every `git push`, when your pre-push checks start; push even if it fails. ``` ## 5. Check and troubleshoot - `backmesh status` shows the control plane, each org's runners, queue, job times, cost and checks, and why runners are short. Add `--org ` for one org, or `--json`. - `backmesh logs --org ` shows the org's listener's log, and `backmesh logs ` a runner's; `-f` follows it. - Pointers: https://backmesh.com/status/ for what status shows, https://backmesh.com/install/#quotas when jobs wait on quota (on GCP, raise `CPUS_ALL_REGIONS` above its default 32 before big parallel bursts), https://backmesh.com/labels/ for sizes, and https://backmesh.com/faq/. - `backmesh upgrade` moves the installation to the CLI's release and restarts each org's listener. **Human:** run it only when the human asks; see https://backmesh.com/upgrade/. ## 6. Uninstall **Human:** only when the human asks. Give them the command to run in a terminal: ```sh backmesh uninstall ``` It asks before deleting anything and says how many jobs are running, which fail. An org admin enters a printed code at `https://github.com/login/device`. `--org ` removes one org and keeps the control plane for the others. Never add `--yes`, `--delete-checkpoint-bucket` or `--confirm` unless the human tells you to: the checkpoint bucket holds the control plane's backups. If it stops, run the same command again. See https://backmesh.com/uninstall/.