Making OpenAI Whisper 19% Faster by Fixing a Silent Kernel Fallback

August 20, 2026

A three-line guard put Whisper's cross-attention back on PyTorch's fused kernel during beam search, 4.9x faster on the attention call itself. Merged upstream in openai/whisper#2812.

_{Note: the headline numbers above (4.9x faster kernel, 19% faster transcription end to end) are from OpenAI's own merged PR, measured on an RTX 3090 with FlashAttention available. The rest of this post reports our own benchmark on an RTX 4060 without FlashAttention, 5.4x on the attention call, about 13% end to end. See "Run it on your own hardware" below for why the hardware changes the ratio.}
‍

Silent performance fallbacks

PyTorch's scaled_dot_product_attention doesn't run one attention kernel. It picks one for you: FlashAttention, a memory-efficient kernel, a cuDNN path, or a plain unfused "math" version, chosen from your dtype, head dimensions, and tensor shapes.If your tensors don't meet a fast kernel's requirements, nothing breaks. You get the same numbers back. You just wait longer for them. The API promises correct results, not fast ones, so it isn't doing anything wrong by quietly downgrading. It's just easy to miss, because everything still works. The same pattern shows up in quantization paths that hit an unexpected memory layout, or autocast regions that upcast one op and drop you off the tensor cores. Nothing tells you. Whisper had one of these in its beam search.

What was happening

Whisper's decoder does something sensible: the audio encoder runs once per 30-second window, so the cross-attention keys and values are computed once and reused across every beam. Only the queries get duplicated per beam.At beam_size=5, the attention call looks like this:

q     [5, 20,    1, 64]     ← one token, duplicated across 5 beams
k, v  [1, 20, 1500, 64]     ← 1500 encoder frames, computed once and shared

PyTorch is happy to broadcast that automatically. The fused kernels aren't: they require q, k, and v to share the same batch size. So every cross-attention call, in every layer, on every decode step, fell back to the slow, unfused path.

Why it stayed hidden

The output is correct, so every test passes.
There's no warning unless you turn on backend diagnostics explicitly.
Greedy decoding is unaffected. At beam_size=1 the shapes already match, so a quick benchmark shows nothing wrong.
The profiler shows the dispatcher, not the kernel. You see aten::scaled_dot_product_attention, the fast-sounding name, unless you specifically look for aten::_scaled_dot_product_attention_math.

The fallback arrived with PR #2359 in September 2024, which routed Whisper's attention through scaled_dot_product_attention in the first place. That was a good change, and greedy decoding got faster immediately. But from that day until this fix landed, close to two years, every beam-search user on a GPU was quietly paying for the unfused kernel, in the reference implementation of one of the most widely deployed speech models.

How we found it

Neither this fallback nor its best fix shows up from reading code or a quick benchmark. Both take systematically comparing candidates against measured cost. We automated that with Artemis, our platform for improving production AI and software systems.

‍1. Profile the baseline. We profiled Whisper large-v3 end to end to see where inference time actually went.
‍2. Run Artemis Discovery. We gave Artemis the goal, the benckmark and our target hardware. Artemis scanned the codebase and proposed optimization candidates worth testing, each scored for its potential impact.
‍3. Iterate toward the solution. Three Discovery runs produced 17 experiments and 51 candidates. Artemis discarded the weakest candidates each round and fed what worked into the next run's experiments, converging on one final version.
‍4. Artemis found the fix. It identified the cross-attention fallback in that final version and generated the patch, a three-line change.

What mattered wasn't just the final version. Nobody had to spend hours hand-tune the search, every dropped candidate had a clear reason attached to it, and the winner still had to match the original transcripts, not just outrun them.

The fix

 if SDPA_AVAILABLE and MultiHeadAttention.use_sdpa:
+     if k.shape[0] == 1 and q.shape[0] != 1:
+         # Cross-attention K/V have batch 1 and broadcast against the
+         # beam-expanded query; the fused SDPA kernels reject the batch
+         # mismatch and fall back to math, so expand K/V to a stride-0
+         # view (no copy) to keep them on the fast path.
+         k = k.expand(q.shape[0], *k.shape[1:])
+         v = v.expand(q.shape[0], *v.shape[1:])
      a = scaled_dot_product_attention(
          q, k, v, is_causal=mask is not None and n_ctx > 1
      )

.expand() is PyTorch's way of repeating a tensor along a size-1 dimension without copying it. It returns a view with stride zero on that dimension, so all five beams read the same memory, but the tensor now reports a batch of 5 and satisfies the fused kernel's precondition.The guard is narrow on purpose. It only fires when k's batch is 1 and q's isn't, which is the specific shape beam search produces. Self-attention, greedy decoding, and anything where the batch sizes already agree pass straight through, so where the fix doesn't apply, it costs one shape comparison.

Proving nothing regressed

Two questions, two checks.
‍Did the transcripts change? We ran 200 LibriSpeech test-clean samples through both builds and compared word error rate sample by sample, not just on average. The merged PR reports 2.070% on its own 200-sample draw, a different subset and normalization; the before/after equality is the same either way.
‍Does it hold up outside clean speech? Artemis ASR Bench ran nine scenarios over HTTP, from clean and long-form audio to noise, phone-quality audio, and overlapping speakers, stressing different parts of the decoder. All nine passed the exact-match validity gate.

Above beam_size=1, the two paths differ only at the last digit fp16 can represent, the kind of difference you get from adding the same numbers in a different order, not from computing something different. The two builds were isolated as git worktrees one commit apart, diffed to confirm those lines were the only change.

The numbers

At the operator level, the cross-attention call went from 802.5 µs to 147.8 µs, about 5.4x faster. End to end, the gain ranges from roughly 10 percent on short clean audio up to 25 percent on overlapping speech, where harder audio pushes more of total time into the decode loop this fix lives in.
The gain requires an expanded decode batch: beam_size above 1, or best_of above 1 for the same reason. At beam_size=1 the guard doesn't fire, the tensors are untouched, and the cost is a single shape comparison. Above 1, the penalty it removes grows with the beam, from about 5x at beam size 2 up to roughly 7x at beam size 8.

Config. Whisper large-v3, fp16, beam_size=5, temperature=0.0, language en, NVIDIA RTX 4060 Laptop GPU 8GB, driver 555.97, CUDA 12.5, PyTorch 2.5.1+cu121, Python 3.12.4, Windows 11. Operator benchmark: n_head=20, head_dim=64, ctx_kv=1500, mean of 300 timed calls after 20 untimed warmup calls.

‍

Run it on your own hardware

None of this is specific to one card or backend. The fix was originally measured on an RTX 3090 with FlashAttention available: the merged PR reports the cross-attention call going from 424 µs to 87 µs there, about 16 percent off real-time factor. The numbers in this post came from an RTX 4060 on a Windows build without FlashAttention, so the fix lands on the memory-efficient kernel instead, a platform difference, not a card limitation. Any fused backend picks up the gain once batch dimensions agree, so measure yours rather than take ours:

from torch.profiler import profile, ProfilerActivity

with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA]) as prof:
    model.transcribe(audio, beam_size=5, fp16=True)

for e in prof.key_averages():
    if "_scaled_dot_product_attention_math" in e.key:
        print(f"{e.key}: {e.device_time_total/1000:.1f} ms across {e.count} calls")

Run that on your GPU with beam_size=5. Anything it prints is time the fused kernel could have had. The fix is on Whisper's main branch, not yet in a release, so install from source:

pip install git+https://github.com/openai/whisper.git

Profile again afterward and the math-kernel entry should be gone. The same check works on any encoder-decoder that caches cross-attention keys and values and expands queries for beam search. This pattern isn't unique to Whisper, and if your model does it, it may well be paying the same tax.

Merged upstream in openai/whisper#2812.