vLLM Co-Introduces REINFORCE Zero: Zero-Mismatch Asynchronous Reinforcement Learning via Operator Alignment
Zero Mismatch, Simple Async RL: REINFORCE Zero, DeepSeek-V4.1 and Kimi K3.1Nov 6, 2026·11 min readAsynchronous RL trains on stale weights. Align rollout and training kernels bit for bit, and unclipped importance sampling corrects that lag. Plus zero-mismatch RL for DeepSeek-V4.1-Flash, Kimi K3.1 and AMD MI355X.
In collaboration with NVIDIA Research and other teams, vLLM has introduced REINFORCE Zero. By fully aligning the operator computational precision between the sampling engine and the trainer, it resolves the operator mismatch challenge in asynchronous reinforcement learning.
The study analyzes how operator numerical errors affect training stability in asynchronous RL and demonstrates a method that enables unclipped importance sampling to achieve stable convergence even under long delays via operator alignment.
Source:vLLM 官方博客（网页） · vllm.ai