Red Hat’s llm-d tuning study cuts first-token latency 71% without changing hardware

Red Hat has published a detailed inference-tuning study that reduced 90th-percentile time to first token (TTFT) for Qwen3-32B from 995 milliseconds to 287 milliseconds—a 71% drop—without changing the model, workload or GPU fleet. The work used llm-d from Red Hat AI Inference 3.4, vLLM 0.18.0 and inference-aware endpoint picking on Kubernetes. Red Hat Developer describes the complete test setup and results.

What changed

The test bed comprised 16 NVIDIA H200 GPUs across two CoreWeave Kubernetes nodes connected by RDMA/InfiniBand. Its workload used 2,000-token inputs, 100-token outputs, 100 concurrent users and a 50% prefix-cache hit rate—an enterprise RAG-shaped profile rather than a generic short-prompt benchmark. Red Hat tested both aggregated serving and prefill/decode-disaggregated layouts across 95 configurations. Those boundaries matter when interpreting the headline result.

Four vLLM settings drove much of the gain: gpu_memory_utilization, max_num_seqs, block_size and max_num_batched_tokens. For Qwen3-32B, raising GPU memory utilization and tuning concurrency and cache blocks moved the best aggregated configuration from the 995 ms default to 304 ms while sustaining 23.6 requests per second. Adjusting llm-d endpoint-picker weights for a prefix-heavy workload then lowered TTFT to 287 ms. The study reports the selected values and explains their tradeoffs.

One template does not fit every model

The most useful result is not a single recommended number. At tensor parallelism eight, the calculated max_num_seqs was 192 for Qwen3-32B but 1,433 for the FP8 Llama 3.1 70B model because their activation and KV-cache constraints differed. Reusing the smaller value would leave Llama capacity idle; applying the larger value to Qwen risked out-of-memory failures. Red Hat argues that model-specific sizing is therefore operationally necessary.

The version boundary also mattered. The Llama configuration regressed from 831 ms on the earlier stack to 1,292 ms with Red Hat AI Inference 3.4 defaults after routing behavior changed, then improved to 463 ms once retuned. That makes an inference-stack upgrade a performance-validation event, not merely a software rollout. The published comparison ties the regression and recovery to the changed routing and tuning assumptions.

What platform teams should do

Teams should reproduce the workload shape that matters to them, sweep feasible tensor-parallel and prefill/decode layouts, then tune memory, concurrency, batching and endpoint-picker weights together. Prefill/decode disaggregation still requires fast KV-cache transfer over RDMA, while the study shows that a tuned aggregated deployment can outperform an untuned disaggregated one. Red Hat also released the Kubernetes-based ServeIt Studio pipeline used to automate calibration, architecture search, parameter tuning and validation. The article positions it as a way to rerun that process after models, hardware or stack versions change.

The 71% figure should not be treated as a universal promise. It is a measured result for two models on a specific 16-H200 cluster and RAG-style workload. The transferable lesson is narrower and more useful: safe defaults preserve compatibility, but production inference performance depends on continuously matching configuration to model architecture, traffic shape and the routing behavior of the installed release.