Rethinking the merge queue

At Reason Machines, engineers work through coding agents. In six weeks, merges into our monorepo went from about 17 a day to over 600.

At that volume, the merge queue became our bottleneck. On October 12, 84 pull requests were waiting at once. The slowest 5% waited two and a half hours.

Some teams at this scale have decided a merge queue can't keep up. They merge first and let agents fix what breaks. We wanted to keep the guarantee that main is always green. So before changing anything, we measured where the time went.

Most of the wait wasn't CI

We ran Mergify, and we had tuned it hard. On its final settings it tested up to twelve pull requests in parallel, each on top of the ones ahead of it. It can also batch pull requests and bisect a failing batch. On a busy afternoon with those settings (October 12, 133 pull requests), a ready pull request took 3.4 minutes to merge at the median.

CI was 79 seconds of that.

The rest was coordination, and most of it came before CI even started. Mergify took 37 seconds to notice a pull request was ready, then 26 more to create its test branch and draft pull request. After CI, merging took another half a minute. Each step is a round trip through GitHub, and each waits for the one before.

Failures make it worse. When one candidate fails, every candidate stacked on top of it is tested again. When an event went missing, a pull request could sit in "Preparing checks" until someone refreshed it.

None of this is a Mergify bug. It is what queueing through a remote API costs at 600 pull requests a day. Super Merge spends about 16 seconds around CI where our Mergify setup spent 124.

Build the next main inside git

Super Merge does the coordination in one process, with its own copy of the repository.

Ready pull requests are squashed one after another onto main with git merge-tree. This happens inside git's object database, with no checkout and no draft pull request. Building a candidate takes about three seconds; Super Merge then pushes one branch so CI can run on it.

CI runs once, on the tip of the batch. If the tip is green, every pull request below it is green too. While one batch is in CI, the next is already built on top of it.

If a batch fails, Super Merge splits it to find the pull request that broke it, and lands the rest. A flaky failure gets one retry. Pull requests land as merge commits, squashes or rebases, whichever your repository uses.

Reverting stays cheap too. Every pull request still lands as its own commit, so undoing one is a single revert. If main breaks after a land, Super Merge watches each new commit for 20 minutes, points at the pull request that caused it, and can revert just that one. On our monorepo, a person makes that call.

Batching is the reserve

On day one, batches averaged 1.1 pull requests. At our current volume, the speedup does not come from sharing CI. It comes from the coordination that is gone.

Batching matters when the queue gets deep. We replayed our busiest day at ten times its volume. Mergify as we ran it fell two hours behind; Super Merge stayed under two minutes, because building a batch costs seconds. Mergify's own batching would narrow that gap, but at agent volume a queue has to batch, and batching only stays cheap when building a candidate is cheap.

Prove it on your own history

A merge queue sits in front of every change, so Super Merge did not touch main until it had earned it.

First, we replayed 30 days of our real history through it. Every pull request arrived when it really did. Every CI run took as long as it really took, and passed or failed as it really did. Main stayed green in every run.

We also replayed five public repositories that already use GitHub's merge queue, among them flutter, servo and nixpkgs. In the simulator, the typical pull request barely changed: no queue beats one CI run, and GitHub's is already close. The slowest 5% landed two to five times sooner, and main stayed green in every run.

Then we ran it in shadow mode, read-only, next to Mergify on live traffic. On the first day it wanted to revert a good change, because it read a cancelled CI run as a failure. It was wrong somewhere harmless, and we fixed it before it touched main.

Three days in

We switched on October 13 with one config file, and kept a one-command way back.

Measured the same way on both sides, from the merge label to merged:

• Mergify, the week before the switch (2,203 pull requests): 7.9 minutes at the median, 114 minutes for the slowest 5%.
• Super Merge, since the switch (904 pull requests): 2.9 minutes and 7.2 minutes.

On days with similar volume, 250 to 400 pull requests, Mergify's slowest 5% waited anywhere from 20 minutes to two hours, about 50 minutes on a typical such day. On each full day so far, Super Merge's waited 6.5. 99.8% of its batches passed CI on the first try, and main only ever moved to commits that had passed CI.

The guarantee was never the expensive part of a merge queue. The waiting was. Super Merge is plain git and GitHub, with no new version control system. Today it runs on one repository: ours.