Running large models like Whisper in production brings a familiar set of problems: slow inference, high compute costs, and infrastructure that has to be over-provisioned to cope. The pain is sharpest when you are on CPUs, or trying to scale cost-effectively in the cloud.
Why CPU inference is the hard case
GPU capacity is expensive and contended; a lot of real transcription work runs on CPU because that is what is available and affordable. But most published optimisation work assumes a GPU, so CPU deployments inherit whatever the default code path happens to do.
The numbers
We applied Artemis to Whisper on Intel's Tiber Cloud and 3rd Gen Xeon processors. No model changes — profiling, code optimisation, and validation only:
- 24.96% faster runtime on GPU
- 14.65% acceleration on CPU
Tuning starts with the code
The lesson we keep relearning: before you buy more hardware or accept a smaller model, check whether the code you are running is actually using the hardware you already have.
