TensorRT-LLM INT8 Quantization for Non-Gated MoE: +40% Nemotron Throughput on A100

How we got 40 percent more throughput out of an A100, and why the speedup was already sitting in the kernels.

‍Quantization support in an inference engine is written one architecture family at a time. Somebody adds a path for the models that exist, tests it against those models, and moves on. Then the architecture landscape shifts. A new variant appears that the plumbing was never shaped for, and the fast path quietly does not cover it. Not because the kernels cannot do the work, but because the layer above was written for the shapes that were in circulation at the time.That gap between what the kernels can do and what the plumbing exposes is where a surprising amount of free performance lives. Here is one case, worth about 40 percent output throughput. The fix is now merged in NVIDIA/TensorRT-LLM #15550.

What was not covered yet

Mixture-of-Experts models come in two flavours. Gated MoEs, like Mixtral or DeepSeek, use activations such as SwiGLU, where the first expert projection produces two halves, a gate and an up projection. Its intermediate dimension is twice that of the second projection. Non-gated MoEs, like NVIDIA's Nemotron-3-Nano-30B-A3B with its squared-ReLU experts, produce one half. Their fc1 and fc2 intermediate dimensions are equal.TensorRT-LLM supported INT8 weight-only quantization (W8A16) for MoE layers. That matters most on hardware without native FP8, an A100 for instance, where INT8 is the natural way to compress a model. The support was written around the gated shape: host-side validation in moeOp.cpp checked for the 2x relationship, and weight loading in quantization.py looked for a gate projection to concatenate.It stayed unexercised for a straightforward reason. Gated MoEs dominate the open-model landscape, so the path was mostly driven by models the shape assumption held for. And when a non-gated model did arrive, the outcome read as "this combination is not supported yet" rather than as an invitation to look underneath. Which is a shame, because underneath, the CUTLASS grouped GEMM kernels handled the non-gated layout perfectly well. The kernels were ahead of the plumbing.

How we found it

This came out of Artemis, our platform for automatically optimising code. We pointed it at the Nemotron serving stack on A100 with one goal: more output throughput at equal accuracy. Artemis ran the search, 15 experiments across 21 candidate changes, scoring each against that baseline over 3 times. Among the things it surfaced was that the INT8 weight-only MoE path was declining the model on layout grounds rather than capability grounds.That division of labour is the part worth describing. The automation meant nobody spent a week hand-bisecting an inference stack looking for reachable-but-unreached performance. The scoring meant we knew the size of the prize before writing the fix. Our job was the last mile: work out why the shape check was written the way it was, confirm the kernels genuinely supported the alternative layout, and turn the finding into a clean native change worth sending upstream rather than a local patch.

The change

Two files. In moeOp.cpp, the shape validation becomes conditional on activation type, mirroring the logic the non-quantized path already used:

if (isGatedActivation(base_activation_type))
{
    TORCH_CHECK(fc1_expert_weights.sizes()[2] == fc2_expert_weights.sizes()[1] * mInnerDimMultiplier * 2,
        "fc1_expert_weights inter size must be 2 times fc2_expert_weights inter size.");
}
else
{
    TORCH_CHECK(fc1_expert_weights.sizes()[2] == fc2_expert_weights.sizes()[1] * mInnerDimMultiplier,
        "fc1_expert_weights inter size must be equal to fc2_expert_weights inter size.");
}

In quantization.py, weight loading learns to load a single up projection for non-gated models instead of always concatenating gate and up. No new kernels, no new configuration surface, nothing behind a flag. It is reachable through the standard QuantAlgo.W8A16 path, which is exactly what made it worth upstreaming rather than keeping.

How we proved it

Three layers. Unit tests first: we ported non-gated coverage into TensorRT-LLM's unified MoE test framework, which validates kernel output against a dequantized reference rather than only checking shapes. Forty W8A16 cases pass, twenty gated and twenty non-gated, across float16 and bfloat16,  run on the same A100-SXM4-80GB.

Then a control. We ran the same quantization on a dense, non-MoE Nemotron model, where the change should do nothing at all, and measured exactly that: zero delta.

Then accuracy at the serving level. GSM8K strict-match on the quantized model lands inside evaluation noise of the BF16 baseline of 0.8393 ± 0.0101. Lossless, as weight-only quantization should be.

After that, NVIDIA's engineers reviewed it, ran it through their full CI across their hardware pools, and merged it.

The numbers

Single A100-SXM4-80GB, 1 GPU, TensorRT-LLM 1.3.0rc19 PyTorch backend, Nemotron-3-Nano-30B-A3B, concurrency 32, BF16 baseline versus W8A16

As a cross check, the same quantization on Qwen3.6-35B-A3B, a gated MoE the existing path already handled, gave +33 percent and confirmed the change leaves the gated path undisturbed. The gain is conditional and the condition is worth stating plainly: non-gated MoE models, on hardware where INT8 weight-only is the sensible compression choice. On a gated MoE or a dense model it is a no-op by construction, which is what the control run confirms.

Who benefits today, and who benefits next

The change branches on the activation type, not the model, so it covers the whole non-gated family: squared ReLU, plain ReLU, GELU, SiLU. One wider family of LLMs currently used in TensorRT-LLM passes a non-gated activation into the fused MoE path: NVIDIA's hybrid Mamba-Transformer MoE line, the Nemotron 3 family, which TensorRT-LLM serves through its Nemotron-H architecture class and which includes the Nano checkpoint we benchmarked. Every other MoE in the tree, from Mixtral to DeepSeek to Qwen, is gated. There is a pattern in that. The open ecosystem standardised on SwiGLU, while NVIDIA's own architectures kept choosing squared ReLU: the hybrid Nemotron lines, the dense Nemotron Nano, the Parakeet speech model, etc. The INT8 MoE path simply had not caught up with the house style yet. Essentially, TensorRT-LLM's INT8 weight-only MoE path did not support NVIDIA's own preferred activation.

Nothing outside language models is affected for now, since the diffusion and encoder decoder paths never touch this op, but any future multimodal model built on a non-gated MoE backbone inherits the fix automatically, and so does the next non-gated architecture anyone adds. The wall is gone for whoever arrives next.

Try it against your own stack

If you are serving a non-gated MoE on Ampere-class hardware, the route is now the ordinary one: quantize with QuantAlgo.W8A16 and serve through the PyTorch backend. The PR is at https://github.com/NVIDIA/TensorRT-LLM/pull/15550.The wider habit is the takeaway. When a quantization path declines your model on shape grounds, it is worth reading the check before accepting the answer. Sometimes what looks like an unsupported configuration is a coverage gap sitting one layer above kernels that were ready the whole time.