\n\nTL;DR, because you have models to train and we respect that:
\n\n
\n- Async RL has a dirty secret: every step, the trainer has to ship the whole model to the inference engine. For a 7B in bf16 that is 14 GB. For a frontier 1T model checkpoint that is on the order of a terabyte. Per step.
\n- It turns out you do not have to. Between two consecutive RL optimizer steps, roughly 99% of bf16 weights are bit-identical (and never less than 98% in the worst case). The actual delta is tiny.
\n- We landed a TRL PR that encodes just the changed elements as a sparse safetensors file, uploads it to a Hugging Face Bucket, and tells vLLM to fetch it. On Qwen3-0.6B, the per-step payload drops from 1.2 GB to 20 to 35 MB.
\n- The cherry on top: we ran a full disaggregated training where the trainer was on one box, vLLM lived in a Hugging Face Space, the Wordle environment lived in another Space, and weights flowed through a single Hub bucket. No shared cluster, no RDMA, no VPN.
\nAsync RL just got a lot cheaper. Read on.
\n
Two ways to ship the same weights. Red is wall-clock time during which no tokens are being generated.
\nIf you read our previous post on the landscape of async RL training, you already know the punchline. Every async RL library, regardless of how it spells "actor model" or which color its NCCL backend is painted, eventually trips over the same root: weight synchronization.
\nThe inference engine speaks the policy of step N. The trainer just finished step N+1. The fresh weights have to get from one side to the other before the inference engine starts drifting hopelessly off-policy. This sits on the critical path whether you are running sync or async: a blocking transfer is wasted idle compute of GPUs not generating tokens. With a sparse delta path you collapse that idle time into seconds, and the trainer does not even have to wait for the inference engine to be ready: it just publishes "weights ready" and uploads the weights to the shared bucket the moment its optimizer step finishes, while the inference engine fetches on its own time.
\nFireworks put a very memorable number on this in their post Frontier RL Is Cheaper Than You Think: for a frontier 1T-parameter checkpoint at fp8 (their setting), a full snapshot is 1024 GiB, and that is what conventional wisdom says you have to ship every time you update your rollout fleet. That is the kind of number that gets people to start drawing diagrams with mega-clusters, RDMA fabrics, and dedicated cross-region links. Their measured average delta between adjacent checkpoints lands at 20.3 GiB, or 1.98% of the full model, and "more than 98% of weights in bf16 format remain bit-equivalent between consecutive checkpoints".
\nCursor's Composer 2 report tells a parallel story. They run training and inference in different regions and stitch them together with a shared S3 bucket (their exact words), into which the trainer uploads compressed weight diffs every training step. Each cluster independently downloads and reconstructs from the shared delta chain, "requiring no direct connectivity to the training cluster". The two sides never speak to each other about parameters directly. The bucket is the wire.
\nBoth papers agree on three things, and we want to repeat them slowly, because the rest of this post is essentially a faithful open source translation:
\nThe only thing missing was a version of this story that you can pip install. So we wrote one.
Before we wire anything up, it is worth understanding why this whole game is even winnable. The "98% of weights do not change" claim sounds suspiciously like one of those numbers that works in the demo and falls apart in the wild. It is not. It falls out of how bf16 arithmetic works at the learning rates RL uses.
\nA bf16 number has 7 mantissa bits. Between two consecutive powers of two, there are exactly 27=1282^7 = 128 representable values, so the spacing between adjacent bf16 numbers around ∣w∣|w| is roughly ∣w∣⋅2−7|w| \\cdot 2^{-7}. An update gets absorbed by the bf16 cast whenever it sits below half of that spacing, i.e., when ∣Δw∣<∣w∣/256|\\Delta w| < |w|/256. This is the "bf16 visibility threshold" PULSE plots in their Figure 3.
\nNow look at what Adam does. At an RL learning rate of, say, 3×10−63 \\times 10^{-6}, the update to a single weight is: Δw=−η⋅m^v^+ϵ\\Delta w = -\\eta \\cdot \\frac{\\hat{m}}{\\sqrt{\\hat{v}} + \\epsilon}
\nThe normalized step m^/(v^+ϵ)\\hat{m}/(\\sqrt{\\hat{v}}+\\epsilon) is roughly order one, so ∣Δw∣≈η≈3×10−6|\\Delta w| \\approx \\eta \\approx 3 \\times 10^{-6}. For most weights, ∣w∣|w| sits somewhere around 10−210^{-2} to 10−110^{-1} (PULSE reports a median of 0.019 for representative LLM weights). The threshold ∣w∣/256|w|/256 at that magnitude is around 4×10−54 \\times 10^{-5} to 4×10−44 \\times 10^{-4}, which is bigger than the update.
\nIn other words: the optimizer is whispering, and bf16 cannot hear it. The update gets absorbed by rounding, the byte representation of ww does not change, and from the inference engine's perspective, this weight did not move. Multiply that by a few hundred million parameters, and you get the >99% sparsity number, for free, with zero approximation.
\nThis is exactly the argument made formal in the PULSE paper (Mihai & Belilovsky, 2026). They define two thresholds. The absorption bound 10η10\\eta is the conservative worst case for an Adam update, and the effective bound η\\eta is the regime you actually live in. The bf16 visibility threshold is ∣w∣/256|w|/256. Whenever the update sits below the visibility threshold, it gets absorbed, and the bf16 byte does not change. Their Figure 3 plots both bounds against a cloud of representative LLM weights, and the conclusion is unambiguous: at η=3×10−6\\eta = 3 \\times 10^{-6}, the absorption bound itself already sits below the visibility threshold for almost every weight in the model. They measure this empirically across Qwen2.5 (0.5B/1.5B/7B), Llama-3.2-3B, and Gemma-3-4B, and consistently find a mean per-step sparsity of ~99%, with a standard deviation of 0.2 to 0.4% over 400 training steps. The worst-case step stays above 98%. So <1% changed is not a lucky measurement; it is what the arithmetic guarantees.
\nWe do not have to predict this analytically (and indeed, we tried predicting the change mask from Adam's mm and vv statistics, but recall was a sad 30%, more on that later). We just need to observe which bytes flipped. That is a tiny boolean tensor per parameter, computed right around the optimizer step.
\nDrag the learning rate down to RL territory and watch the cast-back-to-bf16 marker snap to the original tick. The 256-element grid on the bottom left is the aggregate effect across a tiny model.
\nHere is where the second piece of the story comes in, and where this post stops being a translation of Fireworks/Cursor and starts being a Hugging Face thing.
\nA Bucket is a repo type on the Hub designed for high-frequency object storage. No commit ceremony, no PR workflow, no LFS quirks. You add files, you list files, you download files. The Python interface is two functions:
\n\n\n# Trainer side\n\n\n# Inference side\n\nThat is it. Two function calls and your weights are in flight.
\nUnder the hood, buckets are backed by Xet, the Hub's content-defined chunking storage layer. Xet looks at every file you upload, slices it into chunks based on its actual content (not fixed offsets), and deduplicates against everything already in the bucket. The practical upshot, which is delightful in this context, is that even if we were too lazy to write the sparse encoding and just uploaded full anchors every step, Xet would still only transfer the changed chunks. Sparse encoding + Xet stack: we pay for what moved, and we pay for it once.
\nThis is the open source equivalent of the "shared S3 bucket" both Fireworks and Cursor reach for, except the storage layer already knows about content hashing, your existing HF token already has permission, and it composes natively with the rest of the stack (Spaces, datasets, models).
\nThe full architecture has exactly three boxes and one shared substrate:
\nanchors/ for occasional full snapshots and deltas/ for the sparse patches in between. This is the only thing both sides agree on.The property to internalize, the one Cursor's paper sells hard and that holds verbatim here: the trainer and the rollout server never talk to each other about weights. They exchange a tiny POST containing {"repo_id": ..., "filename": ...}, and that is the entire control plane. The actual byte transfer happens between each side and the bucket, in parallel, with no shared network fabric.
Why that matters in practice:
\nThe trainer writes. Replicas read. The Hub does the plumbing.
\nNow we can open the hood. The protocol has four parts: a wire format, a bucket layout, a 30 line vLLM extension, and a trainer side change detector. It is honestly less code than it sounds.
\nWe picked safetensors for the on-disk and on-wire format. It is already the canonical checkpoint format on the Hub, every reasonable framework can read it, and the header carries arbitrary string metadata. That metadata field is where we hide the protocol.
\nThere are two kinds of files in the bucket.
\nAnchors look like a normal checkpoint: one tensor per parameter, full bf16 weights, written every NN syncs (we default to N=10N=10).
\nanchors/step_000010.safetensors\n ├── model.layers.0.self_attn.q_proj.weight (bf16, full)\n ├── model.layers.0.self_attn.k_proj.weight (bf16, full)\n └── ...\nmetadata:\n sparse=False, model_version=10, sparsity=0.0\nDeltas are the interesting bit. For each parameter that actually changed, we store two entries: a flat int32 tensor of element indices, and a bf16 tensor of values at those indices.
\ndeltas/step_000011.safetensors\n ├── model.layers.0.self_attn.q_proj.weight.indices (int32, [num_changed])\n ├── model.layers.0.self_attn.q_proj.weight.values (bf16, [num_changed])\n ├── model.layers.0.mlp.gate_proj.weight.indices\n ├── model.layers.0.mlp.gate_proj.weight.values\n └── ...\nmetadata:\n sparse=True, model_version=11, sparsity=0.9938, changed_params=[...]\nA few nice consequences of this choice:
\nsafe_open(...) in Python and inspect every tensor in it. No proprietary framing, no length prefixes, no version handshake.sparse=True/False and branches. There is no separate manifest.The cadence is straightforward: anchor every Nth step, delta in between. Both end up in the same bucket under anchors/ and deltas/ prefixes. Each new inference replica only needs to grab the most recent anchor and then replay the deltas since.
Ten training steps. Anchor (full snapshot) on step 1 and step 6, sparse delta on every other step. Files land in the bucket as you watch.
\nThe trainer needs to know which bf16 elements actually flipped. We do this with a tiny BF16ChangeDetector that registers a pre-step and post-step hook on the optimizer:
\n\n : = \n : = \n \n \n\n\n \n = \n\n\n \n = \nThe actual code in the PR has a bit more plumbing (matching optimizer param objects to model params via data_ptr(), because Accelerate wraps them as different Python objects), but the idea fits on a napkin: snapshot, step, diff.
This is ground truth. We tried the more elegant path of predicting the mask from Adam's mm and vv statistics, using the bf16 ULP threshold directly. It works in principle. In practice, recall was around 30%, which means we would have shipped a delta missing two thirds of the actual updates. Adam's normalization is messy enough that the analytical threshold is not tight. So we just compare bytes. It costs one bf16 CPU snapshot of the model on the trainer side, which we are willing to pay.
\nThe four phases of the new _sync_weight flow are:
/update_weights. Send the bucket coordinates. vLLM downloads, applies, returns.The log lines tell the story:
\nDelta: 1234567/200000000 elements changed (sparsity=99.38%)\n[delta_engine] uploaded user/wordle-deltas/deltas/step_000042.safetensors (27.4 MB, ...)\nWeight sync: done. Total 9.4s (inference paused 1.1s)\nThe line that matters is the parenthesis. Inference was paused for 1.1 seconds. The remaining 9.4 seconds were spent uploading, which occurred while the rollout server was still generating tokens. With NCCL, we were paying the full sync time as pause time. Here we are paying for it as background time.
\nA single sync, end to end. Switch between delta-over-bucket and NCCL broadcast, and try the replica count toggle to see the fan-out story.
\nvLLM has a clean abstraction for this called WeightTransferEngine. We implement a DeltaWeightTransferEngine whose receive_weights method is, in spirit:
\n \n \n = \n \n # Anchor: feed every tensor and snapshot for future deltas\n \n = \n = \n \n \n # Delta: apply (indices, values) to snapshot, hand full tensor to vLLM\n \n = \n = \n = \n = \n = \n \nWe register it via vLLM's --worker-extension-cls flag, which means no fork of vLLM is required. You install TRL into the same image as vLLM, point the CLI at our class, and you are done.
Worth mentioning: vLLM itself has an in-flight effort to land sparse weight transfer natively, vllm-project/vllm#40096. It adds receive_sparse_weights() and trainer_send_sparse_weights() directly on the WeightTransferEngine base class, with patches encoded as (indices, values) and applied in place via index_copy_(), removing the GPU/CPU validation roundtrip entirely. The PR reports a transfer of 0.16 MB in 0.40 ms for a sparse patch on Qwen3-1.7B versus 942 MB in 192 ms for a full dense send.
One honest caveat in our implementation on the inference side: we keep a CPU bf16 snapshot of the model so we can reconstruct full tensors from sparse (indices, values) patches, because load_weights in vLLM today expects full tensors. Once #40096 (or its successor) lands and exposes an in-place sparse load_weights path, we can apply the indices directly on the GPU and drop the snapshot!
This is the part we are smug about. Everything we have described so far works on your laptop, but the point of routing weights through a Hub bucket is that the trainer and the rollout server do not have to live anywhere near each other. So we ran a fully disaggregated training with three machines, none of which share a network:
\nSetting this up is genuinely a few hf CLI calls. The vLLM Space's Dockerfile is essentially the upstream vLLM image plus pip install trl@... plus the entrypoint:
FROM vllm/vllm-openai:latest\nRUN pip install "trl @ git+https://github.com/huggingface/trl.git@delta-weight-sync"\nENV VLLM_SERVER_DEV_MODE=1\nEXPOSE 7860\nENTRYPOINT ["vllm", "serve", "Qwen/Qwen3-1.7B", \\\n "--host", "0.0.0.0", "--port", "7860", \\\n "--worker-extension-cls", "trl.experimental.async_grpo.delta_engine.DeltaWorkerExtension", \\\n "--weight-transfer-config", "{\\"backend\\":\\"nccl\\"}", \\\n "--max-model-len", "32768", \\\n "--gpu-memory-utilization", "0.8"]\nDeploy it as a Space:
\n\n\nAnd kick off training from anywhere on the planet that can talk HTTPS:
\n\nThe trainer never opens a port. The Space never sees the trainer's IP. The Wordle environment does not know either of them exists. They all talk to the Hub. Training converged on the immediate-EOS sanity check, then on real Wordle rollouts: reward went up, delta payloads stayed in the 20 to 35 MB band, and the inference-paused window per sync stayed around a second. The full run logs are linked in the companion PR.
\nA few things, and we think they are big.
\nAsync RL training without a cluster. If you have one GPU and a Hugging Face account, you can now do real disaggregated training. Your trainer is on the GPU; your rollout fleet lives in Spaces; your environment lives in another Space; weights move through a bucket. This used to require either a colocated setup (with all the throughput compromises that brings) or a real cluster with shared networking. It does not anymore.
\nMulti-replica inference, for free. Stand up two vLLM Spaces, or ten. They all pull from the same bucket. Xet content-addresses storage so consecutive anchors share chunks at rest (which keeps your bucket from blowing up), and the Hub's edge cache makes repeated downloads of the same file cheap to serve. Want a globally distributed rollout fleet? That is now a small DevOps exercise, not a research project.
\nA wire format you can debug with your existing tools. A delta is a safetensors file. You can safe_open it from a notebook, list its keys, inspect the indices, compute the sparsity yourself. We have spent enough hours in tcpdump on opaque NCCL streams to appreciate this.
A path to frontier scale. The 20 to 35 MB number is for Qwen3-0.6B. The interesting question is what the curve looks like once you turn the dial up. Let us do the napkin math.
\nTake Llama-3.1-405B. In bf16 that is 810 GB on disk. PULSE measures ~99% mean per-step sparsity at RL learning rates, so the actual delta sits around 1% of the parameters. Their deployment-measured encoding hits 108 MB on a 7B model, which is the ~130× reduction PULSE reports. Scaled linearly to 405B, the delta lands at roughly 6 GB per step.
\nWhat does that buy you in wall-clock? NCCL is fast inside a cluster, sure. Assume a generous 100 GB/s aggregate broadcast bandwidth (multi-node, RDMA, the works). A full sync is 810 GB / 100 GB/s ≈ 8 seconds of inference pause, every step. With the delta path, the trainer streams 6 GB to a bucket in the background while generation keeps running, and the rollout server's actual paused window is just the apply step, which on this scale lands at a couple of seconds. So even before we leave the cluster, delta cuts the visible pause by 4× and the bytes on the wire by ~130×.
Now leave the cluster. NCCL straight up does not work across clouds. Once you want a rollout fleet in us-east, another in eu-west, maybe one in a Hugging Face Space, the bucket-based path is the only path. At 1 GB/s of usable internet bandwidth, a single full broadcast would take 13 minutes; the delta does it in 6 seconds.
For a 1 TB-class model in the Fireworks framing, their own measured numbers show 20.3 GiB deltas vs the 1024 GiB full snapshot, a ~50× reduction. PULSE's tighter, sparse encoding would push that further (extrapolating ~15 GB per delta, closer to ~65×). Either way, you are in a regime where shipping weights through commodity object storage stops being a hack and starts being the only sensible architecture.
\nWe are not pretending this is finished. Here is the honest list.
\nload_weights). The first one we are stuck with until someone finds a tight analytical mask, which is harder than it looks. The second one goes away when vLLM gains a sparse load_weights API. PR forthcoming.BF16ChangeDetector is built around per-process optimizer hooks. It should generalize cleanly to FSDP2, but we have not measured it at multi-node scale yet. There is a TODO in the PR with our name on it.delta-weight-sync.examples/scripts/openenv/async_wordle.py.examples/scripts/openenv/vllm_space/ and examples/scripts/openenv/wordle_space/.