Inside llama.cpp PR #26802: +13% from graphs that were switched off for no reason

PR #26802 changes 38 lines in llama.cpp's CUDA backend and returns 13% more tokens per second on a quantised mixture-of-experts model in parallel decode, and 5% on bf16 MoE at batch one. It was merged on 11 August 2026. It is the third of three llama.cpp pull requests from Rafail Giavrimis on our team in four days.

The repo

ggml-org/llama.cpp is how a very large number of people actually run language models on their own hardware — around 123,000 stars, and the engine underneath a long list of desktop and server tools. Its CUDA backend is heavily worked on by people who know it far better than we do. That is the interesting part: the headroom below had survived all of them, because it was not a missing optimisation. It was a guard that was too broad.

The problem

llama.cpp captures CUDA graphs to cut per-launch overhead, which matters most exactly when the model is small enough that launch overhead is a real fraction of the work — MoE decode, in other words.

Graph capture cannot survive a mid-capture stream synchronisation. One path in mul_mat_id — the expert-routing matmul — can perform one. So the backend took the safe route: if a model contains a MUL_MAT_ID node that is unquantized, or over the MMVQ batch limit, disable CUDA graphs entirely.

The guard is correct. It is also far wider than the hazard. The dispatch underneath sends most of those cases to MMQ or MMF, and neither of those synchronises. So two large groups of models lost graph capture for a sync that would never happen:

• quantised MoE in parallel decode, once concurrency pushes past the MMVQ batch limit and the experts take the sync-free MMQ path
• bf16 and f16 MoE everywhere, including single-batch decode

The solution

The PR narrows the guard to the hazard. It adds a predicate that mirrors the mul_mat_id dispatch exactly, so graphs are only disabled when the synchronising fallback is genuinely the path that will run. The fallback then asserts that same predicate — resolving a TODO that was already sitting in the file — so if the two ever drift apart in future, the build aborts loudly instead of silently corrupting a graph capture.

It also adds bf16 MUL_MAT_ID cases to test-backend-ops, since bf16 experts now run under captured graphs via MMF and deserve coverage there.

How Discovery found it

The objective was ordinary: more tokens per second on a MoE model, on one RTX 5090, without changing what the model returns.

Discovery does not read the code looking for smells. It proposes candidate changes, builds each one, and runs it against the benchmark. What makes a guard like this findable is that the search does not share the assumption that produced it. A person reading if (unquantized || over_batch_limit) disable_graphs() sees a safety check and moves on, because it is a safety check. A search that has been told only make this faster will happily try removing it, discover the result is faster and still correct on the cases that matter, and hand back a candidate that forces someone to ask the sharper question: which cases actually need this?

The answer to that question — the predicate mirroring the dispatch — is the human part, and it is why the pull request is 38 lines rather than one.

The receipts

Parallel decode, llama-batched-bench on Qwen3-30B-A3B-Q4_K_M, RTX 5090, CUDA 13.3:

The npl=4 row is a control: both builds already have graphs there, so both are the same. The gain appears exactly where the theory says it should — above the MMVQ batch limit, where the experts take the sync-free path.

bf16 decode on LFM2-8B-A1B tells the same story from the other side:

That third row is the one worth pausing on. Turning graphs back off inside the patched build reproduces master exactly — which is how you know the whole gain is the restored graph capture and not something else that happened to change.

And the control that matters most, a quantised model at batch one, where nothing should move: GLM-5.2 753B UD-IQ1_S, 50.52 t/s on master against 50.45 t/s on the PR. Unchanged, as intended.

Every number above is in the pull request, with the commands and the hardware, so you can run it yourself rather than take ours.