vLLM-backed kernel alignment stabilizes asynchronous reinforcement learning

Asynchronous reinforcement learning promises better hardware utilization by letting model rollout and optimization run at the same time. Its weakness is that the learner trains on samples produced by older model weights—and, in practice, by kernels that can calculate slightly different token probabilities from the training stack.

A new engineering report from the vLLM community says those are separate problems and should be treated separately. REINFORCE Zero, developed by NVIDIA Research and Inferact contributors, aligns rollout and learner kernels so they produce bitwise-identical token probabilities at the same weights. It then uses ordinary, unclipped token-level importance sampling to correct the remaining lag between the older rollout policy and the current learner.

What changed

The approach removes sample-rejection gates and clipping from the core recipe. Rollout records the probability of each sampled token; the learner divides its current probability by that recorded value and uses the ratio directly. With the numerical mismatch removed first, the authors report that the remaining weight-lag correction changes smoothly enough for training to remain stable.

The paper and open repository report stable runs through 20,000 policy updates. On 16 GPUs over 12,000 updates, asynchronous training completed in 68.55 training-step hours versus 164.77 hours for synchronous training with aligned kernels—a reported 2.40× throughput improvement with comparable reasoning quality. A Qwen3-4B tool-use run continued learning with policy lag 64, while a Qwen3-30B-A3B mixture-of-experts run raised four-benchmark macro pass@1 from 31.29% to 71.55% over 3,000 updates.

The comparison with standard kernels is the sharper result. The authors report that standard-kernel asynchronous importance sampling produced an 83.33× rise in rollout-to-learner KL divergence and a peak-to-median importance-weight deviation ratio of 442.54. With aligned kernels, KL stayed flat and that ratio fell to 1.50.

Why it matters for vLLM users

The work extends an earlier vLLM effort to make rollout and training agree exactly. The new report applies that consistency to asynchronous training and validates the path across reasoning, Python tool use and visual computer-use workloads.

It also broadens the hardware and model coverage. The team reports zero-mismatch execution for DeepSeek-V4.1-Flash and day-zero support for Kimi K3.1, including LoRA training. It validated DeepSeek-V4-Flash and Kimi K3.1 paths on AMD MI355X accelerators as well as NVIDIA systems. The implementation materials, model-specific recipes and an agent-executable kernel-alignment skill are published as open source.

The caveat

Kernel alignment is not free. The report says aligned execution increased the mean end-to-end training-step time by 6.8% on an OSWorld workload compared with the standard-kernel baseline. For some AMD paths, the team still lists performance optimization as ongoing.

That trade is the point: accept a modest per-step cost to remove numerical disagreement, then recover more time by overlapping rollout and learning without destabilizing the run. The evidence is promising, but the flagship-model section currently reports alignment validation rather than long asynchronous training runs; those are explicitly listed as next work.