Faster Rust builds: shared builders and smarter caching

1. The goal

• One way to build. Developer builds and CI builds go through the same path onto the same fleet of builder machines.
• Send each build to a warm cache. Each builder keeps its cache on its own fast disk. Builds go to the machine that recently built the same project, so only the changed parts recompile. No shared network cache.
• A personal "preferred builder". A developer can name their own strong machine as first choice, with one simple rule for spilling over to the shared pool when it gets busy.
• Build less. Skip work whose inputs did not change.
• Keep the "rolling release" model. Projects deliberately track the latest version of each other, so breakage shows up fast. Caching has to work within that.

2. Where we are today

• CI starts cold every time. Each run happens in a fresh container with no cache, and compiles everything 3–4 times (lint check, tests, then two release targets).
• The builder service exists and already supports remote builds, sending uncommitted code, and reporting its load and which projects it has warm. But it puts every project in one build directory, which makes parallel builds wait on each other.
• Routing exists in a basic form, but ignores how recently a machine built the project and how long its queue is.
• The compiler cache (sccache) is set up in a way that defeats it: one environment variable makes every build directory look different, so nothing is shared between them, and incremental compilation is switched off.

3. What is on the table

Benchmarked on four representative projects on one 28-core machine, plus 30 days of CI history. The first two charts are gains we get whichever cache tool we use.

Other measured gains

Cache tool: sccache vs Mr. Boxington (mbx)

mbx is a new Rust build cache aimed at many agents and worktrees on one machine. Same code, same dependency versions, each tool with its own empty cache, sccache with the fixes above.

Caveat: about 11 of our 63 build scripts write generated files into the source tree rather than Cargo's official output folder. mbx caches build scripts, so with its defaults those projects fail to build in a fresh checkout; with that one feature switched off they work, and the gains above still hold.

4. The design

Choosing a builder

• The local code service keeps a background snapshot of every builder (load, queue, which projects it built and when), so choosing takes milliseconds.
• Pick the machine expected to finish first: its queue wait plus a warm or cold build time learned from history. A warm idle machine wins; a warm machine with a long queue can lose to an idle cold one.
• Projects get a stable "home" machine through a simple hashing rule every client computes the same way, so they settle across the fleet without a central coordinator. Busy projects spill to a second home.
• Your own preferred builder comes first until its queue passes a number you choose.

On each builder

• One build directory per project, so different projects build in parallel.
• Admit builds by available memory, not a fixed count.
• Evict whole projects, least recently used first, when disk runs low.

Dependencies (rolling release)

• The code service already checks repositories against their remotes on a schedule. Extend it to keep a cache of upstream branch heads.
• Before a build, compare against that cache: if nothing moved, skip the update step; if something moved, update only that dependency. Releases always check live.

CI

• CI runners live on the builder machines and become thin clients that hand the build to the warmest builder.
• Skip runs where neither the code nor the dependency versions changed.
• Lint, tests and release share one warm build directory.

When a build fails: who caused it?

Lock files (the exact dependency versions a checkout resolved) are not committed, by design, so every checkout drifts on its own. A real incident: after a merge one service would not build, because its local lock still pointed at an old version of a dependency. A fresh checkout built fine, so the fix looked like "no change needed". A partial update of that one dependency then broke the build in a new way; only a full fresh resolve fixed it.

• Update dependencies only when needed. Not on every build: only when an upstream moved, or when a build fails inside one of our own shared libraries (a missing field or module, a package that cannot be found). Then the builder does a full fresh resolve, never a single-package update, and retries once. If the retry passes, it was a stale lock and nobody needs to look at it.
• Attribute every failure without extra builds. Each build is recorded by its code commit and a fingerprint of its lock. Compared with the branch's last green build: only the lock changed means the update caused it; only the code changed means the change caused it; if both changed, and only then, one extra (mostly cached) build with the old lock and the new code tells them apart.
• Report, don't fix. A build of the branch tip after every merge classifies the failure (stale lock, caused by an update, caused by the change, ambiguous, infrastructure) and reports it, with the log and the list of changed dependencies, to whoever asked for the build. The builder runs no repair agents; deciding to start one stays with the person or agent who asked. Further automation is a later concern, once this is rolled out.

5. Steps and status

6. Risks

• risk Changing the agreed CI plan. The current approved plan builds CI inside each fresh CI container, which stays cold. Routing CI to the shared fleet changes that and needs the plan owner's agreement.
• risk Cache tool choice. We stay on sccache (with the fixes) for now; mbx is faster for fresh checkouts but young and fast-changing, so it waits for team consensus.
• risk Build scripts that write into the source tree break under build-script caching. Mitigation: fix them (they break Cargo's rules anyway) or keep that feature off.
• risk Disk. A warm project takes 4–40 GB per machine, so no builder can keep everything warm; routing to home machines is what makes this work.
• risk Rolling dependencies. A day of upstream changes still costs a large rebuild; caching avoids paying it twice, not paying it at all.

7. Decisions so far

• CI goes to the builders. CI runners become coordinators; they keep running only work that does not go through the builder (one-off projects, other languages).
• Dependencies refresh when an upstream moved, as a full fresh resolve, plus a daily safety net: if a project's lock has not been fully resolved in 24 hours, its next build re-resolves it. That covers projects with few or no moving dependencies, so registry updates still arrive.
• ARM builds per project: CI builds what each project's CI declares; developer builds build what is requested.
• Fleet later. Right now the work is code. The main development machine is its owner's preferred builder, not part of the shared pool.
• Cache tool: sccache with the fixes now; build scripts get fixed regardless; mbx later if the team agrees.
• No central failure-outcome collection for now; the builder detects, classifies and reports only.

Still open

• How stale a cached upstream head may be before checking live (suggestion: 5 minutes)?
• Keep full nightly release builds, or switch to a warm check-and-test when only dependencies moved?
• Aligning the CI change with the owner of the current build plan.

8. Done vs pending

Numbers are single runs on one machine and should be read as rough. CI durations include queue time. The "warm builder" CI figure is an estimate: two release targets rebuilt after a day of upstream changes, excluding queueing. Contains no credentials or internal addresses.