![\"\"](\"https://cdn-uploads.huggingface.co/production/uploads/61f750055596aa721ce68efe/2Q2pkNtiYPrlLVLoH51dO.png\")
lm_head. The final V1 run returns close to the V0 trajectory across clip rate, KL, entropy, and reward.PipelineRL uses vLLM as the inference engine for rollout generation. The inference engine samples tokens and returns token logprobs; the trainer uses those logprobs to compute policy ratios, KL, clip rate, entropy, and reward. Any discrepancy in how those logprobs are computed can change the training dynamics. This is the train-inference mismatch we needed to eliminate during the vLLM V0 to V1 migration.
\nTL;DR. vLLM V1 matched our vLLM V0 reference after we fixed four things: processed rollout logprobs, V1-specific runtime defaults, the inflight weight-update path, and the fp32 lm_head used for the final projection. We fixed the backend behavior before changing the RL objective.
The reference run used vLLM 0.8.5; the V1 runs used vLLM 0.18.1. Figure 1 shows the final result. The red run is the initial V1 attempt, and the green run is the final V1 run after the fixes described below.
lm_head. The final V1 run returns close to the V0 trajectory across clip rate, KL, entropy, and reward.vLLM V1 is a substantial rewrite of the V0 engine. Our migration target was therefore deliberately narrow:
\nThe first visible symptoms appeared in:
\nclamp_log_ratio_new_old_indicatorkl_new_oldentropyrewardThose metrics came from a GSPO training run, the objective used for this experiment. The same class of mismatch can surface in PPO, GRPO, or any online RL system that treats rollout-side logprobs as part of the optimization target.
\nThe initial V1 run showed the problem clearly. The trainer-side logprobs and reward moved away from the V0 reference early in training.
\n![\"\"](\"https://cdn-uploads.huggingface.co/production/uploads/61f750055596aa721ce68efe/YhRfKIEKndf4LGG9g7Ck9.png\")
The same pattern appears in the trainer metrics. Clip rate is the easiest signal to read in the initial comparison.
\n![\"\"](\"https://cdn-uploads.huggingface.co/production/uploads/61f750055596aa721ce68efe/6fQ4PVHFb_bbYMzducj1o.png\")
We separated the possible causes into three layers:
\nWe initially suspected the third category too early. The useful diagnosis came from treating the first two as backend behavior problems and ruling them out first.
\nThe first issue was semantic. vLLM V1 returns logprobs from the raw model outputs by default, before logits post-processing such as temperature scaling, penalties, and top-k/top-p filtering. PipelineRL expected logprobs from the processed distribution used by the sampler.
\nThe required setting was:
\nlogprobs-mode=processed_logprobsThis removed the obvious mean offset in rollout logprobs. The training curves still showed a gap relative to the known-good reference, so the next issue had to be in the inference path.
\nThe policy-ratio plot shows this directly. Once processed_logprobs is on for V1, the mean policy ratio stays centered extremely close to 1.0 across all three runs. That establishes the mean-bias fix. The remaining mismatch shows up in clip rate, KL, entropy, and downstream training behavior.
![\"\"](\"https://cdn-uploads.huggingface.co/production/uploads/61f750055596aa721ce68efe/K0PQyxwunQUipAXv9Iy1n.png\")
The early V1 run mixed the engine version with V1 runtime defaults:
\n0.18.1 default applied0.18.1 default applieddisable-cascade-attn override that was set through launch-time kwarg passthrough and sits outside the parity recipe in committed configFor the parity run, we made these choices explicit:
\nvllm_config:\n use_v1: true\n vllm_kwargs:\n logprobs-mode: processed_logprobs\n enable-prefix-caching: false\n async-scheduling: false\nPrefix caching deserves a separate note. It is normally a correctness-preserving inference optimization for a fixed model state. In this online RL setup, it was a V1-only difference in cache lifetime and reuse relative to the V0 reference path. The actor was also handling repeated prefixes, concurrent requests, async scheduling, and inflight weight updates.
\nA prefix-cache hit can reuse state computed before a weight update when the cache policy ignores the weight-update boundary. Disabling prefix caching removed one V1-only degree of freedom from the parity comparison.
\nWeight synchronization also had to match the online-RL update model. One option was to make V1 stricter than V0 by draining requests and clearing caches at every update. That would answer a separate question. We first needed to verify that V1 could match the existing V0 behavior.
\nWhat V0 effectively did was closer to:
\nThe nearest V1 analogue was:
\nawait \nawait \nawait \nTwo details matter:
\nmode="keep" matches the old inflight update model more closely than wait or abortclear_cache=False matches the V0 wrapper behavior, which left cached state intact on updateLag was a useful runtime diagnostic. The initial V1 path carries more persistent lag later in training than the corrected V1 run.
\n![\"\"](\"https://cdn-uploads.huggingface.co/production/uploads/680ba1729f7688275d2ce0f4/XWXONkgJtznG09PVLFCAl.png\")
The V1 backend fixes above removed the obvious migration issues, but final parity still required matching the numerical path used to compute logits. The trainer used an fp32 lm_head for the final projection. The rollout backend had to match that behavior.
A closely related issue appears in the MiniMax-M1 technical report: their RL run showed a training/inference token-probability mismatch that they traced to the LM output head and fixed by computing the head in fp32.
\nThis matters because the RL update consumes token logprobs directly. Small changes in logits can become visible in policy ratios, KL, and clipping. The final projection precision is therefore part of the correctness surface for online RL. The ScaleRL paper later includes fp32 logits/head computation as part of its RL recipe and ablates it as a useful design choice for large-scale RL.
\nWith the fp32 lm_head path included, reward gives a compact view of the final parity result. In Figure 6, the final V1 run tracks the V0 reference; the initial V1 attempt produces a clearly different reward curve.
![\"\"](\"https://cdn-uploads.huggingface.co/production/uploads/680ba1729f7688275d2ce0f4/oCuXYiTLSl6-jPJLBzWoU.png\")
lm_head path (green). With the fp32 head included, the final V1 run tracks the V0 reference.The negative results are important because they rule out common explanations.
\nprocessed_logprobs alone: fixed the semantic logprob bug; the training mismatch remained.Objective-side corrections such as truncated importance sampling, importance-ratio reweighting, and related methods are useful tools. If rollouts are intentionally stale, generated asynchronously, or produced by a backend where equivalence to the trainer-side policy is unavailable, then some form of correction is often the right thing to add.
\nThe first problem here was inference correctness. After moving to V1, the rollout backend returned logprobs and runtime behavior that broke the trainer assumption. Adding an objective-side correction at that point would have mixed two questions:
\nThose questions need to be separated. Otherwise an objective-side correction can compensate for broken inference-backend behavior, which makes the training curve harder to interpret.
\nThe current objective can still improve. After inference parity is restored, the next improvement is the usual async/off-policy cleanup:
\nThe main lesson from this migration is narrower: fix backend correctness first, then add corrections for the mismatch that remains.
\n", "content_markdown": "[PipelineRL](https://huggingface.co/blog/ServiceNow-AI/github.com/ServiceNow/PipelineRL/) uses vLLM as the inference engine for rollout generation. The inference engine samples tokens and returns token logprobs; the trainer uses those logprobs to compute policy ratios, KL, clip rate, entropy, and reward. Any discrepancy in how those logprobs are computed can change the training dynamics. This is the train-inference mismatch we needed to eliminate during the vLLM V0 to V1 migration.\n\n**TL;DR.** vLLM V1 matched our vLLM V0 reference after we fixed four things: processed rollout logprobs, V1-specific runtime defaults, the inflight weight-update path, and the fp32 `lm_head` used for the final projection. We fixed the backend behavior before changing the RL objective.\n\nThe reference run used vLLM `0.8.5`; the V1 runs used vLLM `0.18.1`. Figure 1 shows the final result. The red run is the initial V1 attempt, and the green run is the final V1 run after the fixes described below.\n\n\\\nFigure 1. Trainer-side metrics for the vLLM V0 reference (blue), the initial vLLM V1 attempt (red), and the final vLLM V1 run after our fixes (green), including the fp32 `lm_head`. The final V1 run returns close to the V0 trajectory across clip rate, KL, entropy, and reward.\n\n## Migration Objective\n\nvLLM V1 is a substantial rewrite of the V0 engine. Our migration target was therefore deliberately narrow:\n\n1. verify that V1 returned rollout logprobs in the form the trainer expected\n2. rerun the same workload against the V0 reference\n3. evaluate objective-level changes only after backend parity was restored\n\nThe first visible symptoms appeared in:\n\n- `clamp_log_ratio_new_old_indicator`\n- `kl_new_old`\n- `entropy`\n- `reward`\n\nThose metrics came from a GSPO training run, the objective used for this experiment. The same class of mismatch can surface in PPO, GRPO, or any online RL system that treats rollout-side logprobs as part of the optimization target.\n\nThe initial V1 run showed the problem clearly. The trainer-side logprobs and reward moved away from the V0 reference early in training.\n\n\\\nFigure 2. Current-policy logprobs computed by the trainer during updates (left) and reward (right). The initial vLLM V1 run (red) separates from the vLLM V0 reference (blue).\n\nThe same pattern appears in the trainer metrics. Clip rate is the easiest signal to read in the initial comparison.\n\n\\\nFigure 3. Trainer-side metrics for the vLLM V0 reference (blue) and the initial vLLM V1 attempt (red). Clip rate tracks the rollout/trainer policy gap; entropy and reward show how that gap propagates into training.\n\n## Failure Modes\n\nWe separated the possible causes into three layers:\n\n1. **Semantic mismatch**: the backend returns logprobs with different meaning relative to what the trainer expects.\n2. **Inference-path mismatch**: the backend uses different runtime defaults for caching, scheduling, or request handling, so the same prompts follow a different execution path.\n3. **Objective mismatch**: the RL objective needs correction for the amount of staleness or backend mismatch that remains.\n\nWe initially suspected the third category too early. The useful diagnosis came from treating the first two as backend behavior problems and ruling them out first.\n\n## V1 Backend Fixes\n\n### Logprob Semantics\n\nThe first issue was semantic. vLLM V1 returns logprobs from the raw model outputs by default, before logits post-processing such as temperature scaling, penalties, and top-k/top-p filtering. PipelineRL expected logprobs from the processed distribution used by the sampler.\n\nThe required setting was:\n\n- `logprobs-mode=processed_logprobs`\n\nThis removed the obvious mean offset in rollout logprobs. The training curves still showed a gap relative to the known-good reference, so the next issue had to be in the inference path.\n\nThe policy-ratio plot shows this directly. Once `processed_logprobs` is on for V1, the mean policy ratio stays centered extremely close to `1.0` across all three runs. That establishes the mean-bias fix. The remaining mismatch shows up in clip rate, KL, entropy, and downstream training behavior.\n\n\\\nFigure 4. Per-step deviation of the rollout/trainer policy ratio from 1.0, scaled by 10,000, for the vLLM V0 reference (blue), the initial vLLM V1 run (red), and the corrected vLLM V1 run (green).\n\n### Runtime Defaults\n\nThe early V1 run mixed the engine version with V1 runtime defaults:\n\n- prefix caching, left unset in the early run so the vLLM `0.18.1` default applied\n- async scheduling, left unset in the early run so the vLLM `0.18.1` default applied\n- an ad-hoc `disable-cascade-attn` override that was set through launch-time kwarg passthrough and sits outside the parity recipe in committed config\n\nFor the parity run, we made these choices explicit:\n\n```yaml\nvllm_config:\n use_v1: true\n vllm_kwargs:\n logprobs-mode: processed_logprobs\n enable-prefix-caching: false\n async-scheduling: false\n```\n\nPrefix caching deserves a separate note. It is normally a correctness-preserving inference optimization for a fixed model state. In this online RL setup, it was a V1-only difference in cache lifetime and reuse relative to the V0 reference path. The actor was also handling repeated prefixes, concurrent requests, async scheduling, and inflight weight updates.\n\nA prefix-cache hit can reuse state computed before a weight update when the cache policy ignores the weight-update boundary. Disabling prefix caching removed one V1-only degree of freedom from the parity comparison.\n\n### Inflight Weight Updates\n\nWeight synchronization also had to match the online-RL update model. One option was to make V1 stricter than V0 by draining requests and clearing caches at every update. That would answer a separate question. We first needed to verify that V1 could match the existing V0 behavior.\n\nWhat V0 effectively did was closer to:\n\n- block execution at an engine boundary\n- load the new weights\n- resume without an explicit cached-state invalidation\n\nThe nearest V1 analogue was:\n\n```python\nawait engine.pause_generation(mode=\"keep\", clear_cache=False)\nawait engine_client.collective_rpc_async(\n \"receive_weight_update\",\n args=(request.model_dump_json(),),\n)\nawait engine.resume_generation()\n```\n\nTwo details matter:\n\n- `mode=\"keep\"` matches the old inflight update model more closely than `wait` or `abort`\n- `clear_cache=False` matches the V0 wrapper behavior, which left cached state intact on update\n\nLag was a useful runtime diagnostic. The initial V1 path carries more persistent lag later in training than the corrected V1 run.\n\n\\\nFigure 5. Number of steps the weights in the rollout server are behind the trainer policy, for the vLLM V0 reference (blue), the initial vLLM V1 run (red), and the corrected vLLM V1 run (green).\n\n## The Remaining Gap: fp32 lm\\_head\n\nThe V1 backend fixes above removed the obvious migration issues, but final parity still required matching the numerical path used to compute logits. The trainer used an fp32 `lm_head` for the final projection. The rollout backend had to match that behavior.\n\nA closely related issue appears in the [MiniMax-M1 technical report](https://arxiv.org/abs/2506.13585): their RL run showed a training/inference token-probability mismatch that they traced to the LM output head and fixed by computing the head in fp32.\n\nThis matters because the RL update consumes token logprobs directly. Small changes in logits can become visible in policy ratios, KL, and clipping. The final projection precision is therefore part of the correctness surface for online RL. The [ScaleRL paper](https://arxiv.org/abs/2510.13786) later includes fp32 logits/head computation as part of its RL recipe and ablates it as a useful design choice for large-scale RL.\n\nWith the fp32 `lm_head` path included, reward gives a compact view of the final parity result. In Figure 6, the final V1 run tracks the V0 reference; the initial V1 attempt produces a clearly different reward curve.\n\n\\\nFigure 6. Reward for the vLLM V0 reference (blue), the initial vLLM V1 attempt (red), and the final vLLM V1 run with the fp32 `lm_head` path (green). With the fp32 head included, the final V1 run tracks the V0 reference.\n\n## Ablations\n\nThe negative results are important because they rule out common explanations.\n\n- **`processed_logprobs` alone**: fixed the semantic logprob bug; the training mismatch remained.\n- **Batch invariance**: the mismatch remained in a separate test, with higher lag, higher clip rate, and NCCL complications.\n- **Treating the first V1 run as a fair baseline**: the first V1 run had multiple V1-only defaults enabled, so it was a confounded migration comparison.\n\n## Why We Fixed Backend Correctness First\n\nObjective-side corrections such as truncated importance sampling, importance-ratio reweighting, and related methods are useful tools. If rollouts are intentionally stale, generated asynchronously, or produced by a backend where equivalence to the trainer-side policy is unavailable, then some form of correction is often the right thing to add.\n\nThe first problem here was inference correctness. After moving to V1, the rollout backend returned logprobs and runtime behavior that broke the trainer assumption. Adding an objective-side correction at that point would have mixed two questions:\n\n- is the inference backend producing the right logprobs?\n- given correct logprobs, does the objective still need an off-policy or async correction?\n\nThose questions need to be separated. Otherwise an objective-side correction can compensate for broken inference-backend behavior, which makes the training curve harder to interpret.\n\nThe current objective can still improve. After inference parity is restored, the next improvement is the usual async/off-policy cleanup:\n\n- keep explicit behavior-policy logprobs from rollout time\n- recompute trainer-side old-policy logprobs at optimization time\n- separate backend mismatch correction from the policy-update ratio\n- track diagnostics like ESS for the correction term alongside aggregate trainer metrics\n\nThe main lesson from this migration is narrower: fix backend correctness first, then add corrections for the mismatch that remains.\n", "dateCreated": "2026-10-19T01:26:53.998049939+00:00", "date_created": "2026-05-06T19:06:55+00:00", "date_modified": "2026-05-06T19:06:55+00:00", "date_published": "2026-05-06T19:06:55+00:00", "description": "PipelineRL uses vLLM as the inference engine for rollout generation. The inference engine samples tokens and returns token logprobs; the trainer uses those logprobs to compute policy ratios, KL, clip rate, entropy, and reward. Any…", "external_url": "https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/05/2026-05-06-vllm-v0-to-v1-correctness-before-corrections-in-rl.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/05/2026-05-06-vllm-v0-to-v1-correctness-before-corrections-in-rl.md" }, "id": "urn:aggr:item:26c8daa909bffb463d8397a5d53ac347e5788fb8", "isBasedOn": { "@id": "https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections" }, "mainEntity": { "@id": "https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-05-06-vllm-v0-to-v1-correctness-before-corrections-in-rl/#webpage" }, "dateModified": "2026-05-06T19:06:55+00:00", "datePublished": "2026-05-06T19:06:55+00:00", "headline": "vLLM V0 to V1: Correctness Before Corrections in RL", "image": { "@type": "ImageObject", "caption": null, "height": 90, "url": "https://cdn-thumbnails.huggingface.co/social-thumbnails/blog/ServiceNow-AI/correctness-before-corrections.png", "width": 160 }, "timeRequired": "PT7M", "url": "https://huggingface.co/blog/ServiceNow-AI/correctness-before-corrections", "wordCount": 1536 }, "name": "Archived snapshot: vLLM V0 to V1: Correctness Before Corrections in RL", "preview": { "alt": null, "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://cdn-thumbnails.huggingface.co/social-thumbnails/blog/ServiceNow-AI/correctness-before-corrections.png", "width": 160 }, "reading_minutes": 7, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-19T01:26:53.998049939+00:00", "title": "vLLM V0 to V1: Correctness Before Corrections in RL", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-05-06-vllm-v0-to-v1-correctness-before-corrections-in-rl/", "word_count": 1536 }