Improving vLLM CPU Inference: Rebuilding a Dequantisation-GEMM Kernel

Over the past several months, we have been working with customers to improve their AI inference stacks using Artemis. Two challenges come up repeatedly: how can we compress language models to get better performance per byte, and how can we ensure that a compressed model still runs efficiently in the inference engine?

These questions matter even more for agentic workloads, where long contexts and multiple workers increase inference demand. For many workloads, a forward pass is limited less by arithmetic throughput than by how quickly model weights can be moved into the processor. Reducing their size can improve output throughput, but only if unpacking them does not become the next bottleneck.

This is the problem we explored with Artemis Discovery: how to make fused dequantisation and matrix multiplication more efficient in vLLM on Intel CPUs. More than 100 experiments later, the rebuilt kernel returned 10-30% more output throughput. The largest gain, 31%, came on the smallest model we tested. The reason it lands there is the interesting part.

Quantisation saves bandwidth but it is not free

Quantisation compresses a model by storing its weights in a lower-precision format. Instead of storing every weight in 16 bits, for example, we can group weights under a shared scale and represent each one with a smaller 8-bit or 4-bit code.To use the weight in a matrix multiplication, the inference engine reconstructs an approximation of its original value:

W_dequantised = scale * (W_quantised - zero_point)

Moving from 16-bit to 4-bit storage reduces the raw weight data by a factor of four. In a strongly bandwidth-bound workload, that creates substantial headroom for higher throughput. It does not guarantee a fourfold end-to-end speedup, however: the engine must unpack, rescale and convert those weights before the hardware can multiply them.That creates two trade-offs: quantised weights approximate the originals, so accuracy must be checked, and dequantisation adds computation to every forward pass. Compression solves one bottleneck, but a poor dequantisation path can create another.

What a dequantisation-GEMM kernel does

The compute units in a modern CPU cannot feed packed 4-bit codes directly into a BF16 matrix multiplication. Before the GEMM can run, the kernel has to:

1. extract each 4-bit code from its packed machine word;
2. subtract the zero-point and apply the group's scale;
3. convert the result into the format expected by the matrix multiplication; and
4. arrange the values for efficient use by Intel AMX.

Each operation is inexpensive on its own. Together, though, they are work that an unquantised model does not perform. The aim is to preserve the bandwidth benefit of 4-bit storage while minimising the cost of reconstructing the weights.

Why decode exposes the cost

The relative cost of dequantisation depends on how many tokens reuse each reconstructed block of weights.During prefill, the model processes many prompt tokens together. The matrix multiplication has a large amount of work to do, and the dequantisation cost is amortised across all of those tokens. GEMM performance tends to dominate.During decode, the model usually generates one token per sequence at each step. Each step still touches the model's weights, but there is much less arithmetic over which to spread the unpacking and rescaling cost. A dequantisation path that was almost invisible during prefill can become a significant share of decode time.

This made the fused dequantisation-GEMM path a useful optimisation target: improve it once, and every AWQ model using that kernel can benefit.

Artemis Discovery

There is no obvious one-shot change. Unpacking, vectorisation, data layout, conversion and instruction scheduling interact, and a change that helps one shape can hurt another.Artemis Discovery helped generate hypotheses, turn them into candidate implementations, run the benchmark workflow and record each outcome. Ideas, experiments and results remained visible, so failures became evidence for the next iteration rather than work to rediscover.For this investigation, we used Artemis to coordinate and track more than 100 experiments. The result was an optimised fused kernel that unpacks the quantised weights, applies their scales and hands them directly to the matrix multiplication.

The results

We benchmarked the optimised kernel across a range of AWQ and W4A16 models on a two-socket Intel Xeon Granite Rapids system, using 48 physical cores.

The optimised path increased output throughput across the sweep by approximately 10–30%. The largest gain was 31% on Qwen3-4B. Even the 70-billion-parameter models improved by 17–21%.The metric here is output throughput, which is dominated by decode—the phase where our analysis predicted dequantisation overhead would matter most.The kernel applies to a class of models rather than one checkpoint: AWQ models such as Qwen 3 AWQ and Llama 3 AWQ use this path. It should not be confused with a separate Qwen optimisation discussed with Intel; these percentages come from our controlled benchmark sweep, not an Intel production deployment.

What this means in production

Higher decode throughput can translate into different benefits depending on the deployment: more concurrent users on the same hardware, fewer compute resources for the same demand, lower cost per generated token, or more tokens per watt.Quantisation is not only a model-format decision. Its production performance also depends on the path that turns compact weights into values the hardware can consume. When decode throughput falls short of what the memory savings suggest, profile unpacking and conversion as well as the GEMM.To evaluate a similar change, compare prefill and decode separately, cover the model sizes and batch regimes you serve, and place an accuracy gate alongside the throughput benchmark. Optimisation is multi-objective and many attempts will fail. With a persistent experimental record, every result can still move the system towards a production-ready solution.