{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-03-09-ulysses-sequence-parallelism-training-with-million-token/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Training large language models on long sequences has become essential for building capable AI systems. As models are increasingly used for tasks like document analysis, code understanding, complex reasoning, and RAG workloads, the need to process sequences of hundreds of thousands—or even millions—of tokens has grown dramatically. To put this in perspective, an average book is roughly 250k tokens, so training on multi-document contexts or book-length inputs requires handling sequences well beyond what fits on a single GPU. However, training with such long contexts presents significant memory challenges: the attention computation scales quadratically with sequence length, quickly exceeding GPU memory for contexts beyond tens of thousands of tokens.\n\nUlysses Sequence Parallelism (part of the [Arctic Long Sequence Training (ALST) protocol](https://huggingface.co/papers/2506.13996) from Snowflake AI Research) provides an elegant solution by distributing the attention computation across multiple GPUs through attention head parallelism. In this post, we'll explore how Ulysses works and how it's been integrated across the Hugging Face ecosystem—from Accelerate to the Transformers Trainer and TRL's SFTTrainer.\n\n## Contents\n\n- [The Challenge of Long Sequence Training](https://huggingface.co/blog/ulysses-sp#the-challenge-of-long-sequence-training)\n- [How Ulysses Works](https://huggingface.co/blog/ulysses-sp#how-ulysses-works)\n- [Integration with Accelerate](https://huggingface.co/blog/ulysses-sp#integration-with-accelerate)\n- [Integration with Transformers Trainer](https://huggingface.co/blog/ulysses-sp#integration-with-transformers-trainer)\n- [Integration with TRL's SFTTrainer](https://huggingface.co/blog/ulysses-sp#integration-with-trl-sfttrainer)\n- [Comparing Ulysses and Ring Attention](https://huggingface.co/blog/ulysses-sp#comparing-ulysses-and-ring-attention)\n- [Best Practices](https://huggingface.co/blog/ulysses-sp#best-practices)\n- [Benchmarks](https://huggingface.co/blog/ulysses-sp#benchmarks)\n- [Resources](https://huggingface.co/blog/ulysses-sp#resources)\n\n## The Challenge of Long Sequence Training\n\nThe attention mechanism in transformers scales quadratically with sequence length. For a sequence of length n n, standard attention requires O(n2) O(n^2) FLOPs and O(n2) O(n^2) memory to compute and store the attention score matrix. Optimized implementations like [FlashAttention](https://huggingface.co/papers/2205.14135) reduce the **memory** to O(n) O(n) by tiling the computation and never materializing the full attention matrix—but the O(n2) O(n^2) compute remains. For very long sequences (32k+ tokens), even with FlashAttention, training still pushes the limits of single-GPU memory.\n\nConsider these scenarios where long-context training is essential:\n\n- **Document understanding**: Processing entire books, legal documents, or research papers\n- **Code analysis**: Understanding large codebases with multiple interconnected files\n- **Reasoning tasks**: Models that \"think\" step-by-step may generate thousands of tokens during inference\n- **Retrieval-augmented generation**: Incorporating many retrieved passages into the context\n\nTraditional data parallelism doesn't help here—each GPU still needs to process the full sequence inside the attention block. We need a way to split the sequence itself across multiple devices.\n\n## How Ulysses Works\n\nUlysses Sequence Parallelism (SP), introduced in the [DeepSpeed Ulysses paper](https://huggingface.co/papers/2309.14509), takes a clever approach: in addition to splitting on the sequence dimension, it also partitions the attention heads across GPUs.\n\n![Ulysses Sequence Parallelism Overview](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ulysses/ulysses_overview.png)\\\nUlysses splits input sequences along the sequence dimension and uses all-to-all communication to exchange key-value pairs, enabling each GPU to compute a subset of attention heads. (***Source: [Snowflake Engineering Blog](https://www.snowflake.com/en/engineering-blog/ulysses-low-latency-llm-inference/)***)\n\nHere's how it works:\n\n1. **Sequence Sharding**: The input sequence is split along the sequence dimension across P P GPUs. Each GPU i i holds tokens \\[i⋅n/P,(i+1)⋅n/P) \\[i \\\\cdot n/P, (i+1) \\\\cdot n/P).\n\n2. **QKV Projection**: Each GPU computes the query, key, and value projections for its local sequence chunk.\n\n3. **All-to-All Communication**: An all-to-all collective operation redistributes the data so that each GPU holds *all* sequence positions after the projections, but only for a subset of attention heads.\n\n4. **Local Attention**: Each GPU computes attention for its assigned heads using standard attention mechanisms (FlashAttention or SDPA).\n\n5. **All-to-All Communication**: Another all-to-all operation reverses the redistribution, returning to sequence-sharded format.\n\n6. **Output Projection**: Each GPU computes the output projection for its local sequence chunk.\n\nThe key insight is that attention heads are independent—each head can be computed separately. By trading sequence locality for head locality, Ulysses enables efficient parallelization with relatively low communication overhead.\n\n### Communication Complexity\n\nUlysses requires two all-to-all operations per attention layer, with total communication volume of O(n⋅d/P) O(n \\\\cdot d / P) per GPU, where:\n\n- n n is the sequence length\n- d d is the hidden dimension\n- P P is the parallelism degree\n\nRing Attention communicates O(n⋅d) O(n \\\\cdot d) per GPU — a factor of P P more — via P−1 P-1 sequential point-to-point transfers around the ring. Ulysses also benefits from lower latency because all-to-all can exploit full bisectional bandwidth in a single collective step, whereas Ring Attention serializes over P−1 P-1 hops.\n\n## Integration with Accelerate\n\nAccelerate provides the foundation for Ulysses sequence parallelism through its [`ParallelismConfig`](https://huggingface.co/docs/accelerate/concept_guides/sequence_parallelism#alstulysses-sp-backend-configuration) class and DeepSpeed integration.\n\n### Configuration\n\n```python\nfrom accelerate import Accelerator\nfrom accelerate.utils import ParallelismConfig, DeepSpeedSequenceParallelConfig\n\nparallelism_config = ParallelismConfig(\n sp_backend=\"deepspeed\",\n sp_size=4, # Split across 4 GPUs\n dp_shard_size=1, # Must satisfy: dp_replicate × dp_shard × sp_size = num_processes\n sp_handler=DeepSpeedSequenceParallelConfig(\n sp_seq_length=None, # None for variable-length sequences\n sp_seq_length_is_variable=True,\n sp_attn_implementation=\"flash_attention_2\", # or \"sdpa\"\n ),\n)\n\naccelerator = Accelerator(parallelism_config=parallelism_config)\n```\n\n### Key Parameters\n\n| Parameter | Description |\n| --------------------------- | --------------------------------------------------------- |\n| `sp_size` | Number of GPUs for sequence parallelism |\n| `sp_backend` | Must be `\"deepspeed\"` for Ulysses |\n| `sp_seq_length_is_variable` | Set to `True` for varying sequence lengths across batches |\n| `sp_attn_implementation` | `\"flash_attention_2\"`, `\"flash_attention_3\"`, or `\"sdpa\"` |\n\n### Using the Accelerator\n\nWhen you call `accelerator.prepare()`, Ulysses is automatically set up:\n\n```python\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-3.1-8B\")\noptimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)\n\n# This registers the model with Ulysses and wraps the dataloader\nmodel, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)\n```\n\nThe `prepare()` call:\n\n1. Registers the model with DeepSpeed's `UlyssesSPAttentionHF`\n2. Wraps the dataloader with `UlyssesSPDataLoaderAdapter` to handle sequence sharding\n3. Automatically injects `shift_labels` for correct loss computation\n\n### Loss Aggregation\n\nWith Ulysses, each GPU computes loss on different parts of the sequence. The losses must be aggregated properly, weighted by the number of valid tokens per rank. If you're using the Transformers `Trainer` or TRL's `SFTTrainer`, this is handled automatically—the code below is only needed when writing a custom Accelerate training loop:\n\n```python\nsp_size = parallelism_config.sp_size\nif sp_size > 1:\n from deepspeed.utils import groups\n\n sp_group = groups._get_sequence_parallel_group()\n\n # Gather losses and token counts from all SP ranks\n losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)\n good_tokens = (batch[\"shift_labels\"] != -100).view(-1).sum()\n good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)\n\n # Weighted aggregation\n total_loss = sum(\n losses_per_rank[i] * good_tokens_per_rank[i]\n for i in range(sp_size)\n if good_tokens_per_rank[i] > 0\n )\n loss = total_loss / max(sum(good_tokens_per_rank), 1)\n\naccelerator.backward(loss)\n```\n\n> The weighted loss aggregation ensures correct gradients when tokens are unevenly distributed across ranks (e.g., when some ranks contain only padding or masked out prompt tokens).\n\n> Both Ulysses and Ring Attention use `position_ids` instead of `attention_mask` for causal masking during training. A 4D attention mask at these sequence lengths would be just as prohibitive as the attention scores themselves—at 128k tokens, that's another ~1TB tensor. Position IDs achieve the same causal behavior with O(n) O(n) memory instead of O(n2) O(n^2). During evaluation/inference, DeepSpeed's SP attention layer can bypass the SP operations entirely (via `disable_in_eval`) and fall back to the model's default attention implementation.\n\n## Integration with Transformers Trainer\n\nThe Transformers [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) provides seamless Ulysses integration through [`TrainingArguments.parallelism_config`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.parallelism_config). It handles all the SP-specific details automatically—dataloader wrapping, sequence sharding, and loss aggregation—so you don't need to write any of the custom loss code shown above.\n\n### Configuration\n\nJust pass the same `parallelism_config` from above into `TrainingArguments`:\n\n```python\nfrom transformers import TrainingArguments\n\ntraining_args = TrainingArguments(\n parallelism_config=parallelism_config, # same ParallelismConfig as above\n per_device_train_batch_size=1,\n)\n```\n\n### What the Trainer Handles Automatically\n\n1. **Dataloader Wrapping**: After model preparation, the Trainer wraps the dataloader with `UlyssesSPDataLoaderAdapter`\n\n2. **Loss Computation**: The `compute_loss` method detects SP mode and routes to specialized `_deepspeed_sp_compute_loss` which handles:\n\n - Gathering losses across SP ranks\n - Computing valid token counts per rank\n - Weighted loss aggregation\n3. **Batch Size Calculation**: The effective data parallel world size accounts for SP:\n\n ```python\n dp_world_size = world_size // sp_size\n ```\n\n4. **Dataloader Length Adjustment**: Training step calculations are adjusted for SP's effect on iteration count\n\n### Launch Command\n\nUse an accelerate config file or command-line arguments:\n\n```bash\naccelerate launch \\\n --config_file deepspeed_ulysses.yaml \\\n train.py \\\n --per_device_train_batch_size 1\n```\n\n## Integration with TRL SFTTrainer\n\nTRL's [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer) builds on the Transformers Trainer and adds specific optimizations for supervised fine-tuning with long sequences.\n\n### Configuration\n\n```python\nfrom trl import SFTConfig, SFTTrainer\nfrom accelerate.utils import ParallelismConfig, DeepSpeedSequenceParallelConfig\n\nparallelism_config = ParallelismConfig(\n sp_backend=\"deepspeed\",\n sp_size=2,\n dp_shard_size=2, # 2D parallelism: SP × DP = 4 GPUs\n sp_handler=DeepSpeedSequenceParallelConfig(\n sp_seq_length_is_variable=True,\n sp_attn_implementation=\"flash_attention_2\",\n ),\n)\n\ntraining_args = SFTConfig(\n ...,\n parallelism_config=parallelism_config,\n max_length=32768,\n pad_to_multiple_of=2, # Must equal sp_size\n per_device_train_batch_size=1,\n)\n\ntrainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\n### Key SFTConfig Parameters for Ulysses\n\n| Parameter | Description |\n| -------------------- | -------------------------------------------------------------------------------------------- |\n| `pad_to_multiple_of` | Must equal `sp_size` to ensure sequence divisibility |\n| `max_length` | Global sequence length (before splitting across GPUs) |\n| `packing` | Works well with SP — packing reduces padding waste, especially for variable-length sequences |\n\n### Accelerate Config File\n\nCreate `alst_ulysses_4gpu.yaml`:\n\n```yaml\ncompute_environment: LOCAL_MACHINE\ndistributed_type: DEEPSPEED\nmixed_precision: bf16\nnum_processes: 4\ndeepspeed_config:\n zero_stage: 3\n seq_parallel_communication_data_type: bf16\nparallelism_config:\n parallelism_config_sp_size: 2\n parallelism_config_sp_backend: deepspeed\n parallelism_config_dp_shard_size: 2\n parallelism_config_sp_seq_length_is_variable: true\n parallelism_config_sp_attn_implementation: flash_attention_2\n```\n\n### Complete Training Command\n\n```bash\naccelerate launch --config_file alst_ulysses_4gpu.yaml \\\n trl/scripts/sft.py \\\n --model_name_or_path meta-llama/Llama-3.1-8B \\\n --dataset_name trl-lib/Capybara \\\n --max_length 32768 \\\n --packing \\\n --pad_to_multiple_of 2 \\\n --per_device_train_batch_size 1\n```\n\n### Shift Labels Handling\n\nThe SFTTrainer automatically handles pre-shifted labels when Ulysses is enabled:\n\n```python\n# When using SP, labels are pre-shifted by the dataloader adapter\n# The trainer detects this and uses shift_labels directly\nlabels = inputs[\"labels\"] if \"shift_labels\" not in inputs else None\n\n# Loss computation uses the pre-shifted labels\nif \"shift_labels\" in inputs:\n shift_logits = outputs.logits.contiguous()\n shift_labels = inputs[\"shift_labels\"]\nelse:\n shift_logits = outputs.logits[..., :-1, :].contiguous()\n shift_labels = labels[..., 1:].contiguous()\n```\n\n## Comparing Ulysses and Ring Attention\n\nBoth Ulysses and Ring Attention enable long-context training, but they have different characteristics:\n\n| Aspect | Ulysses (DeepSpeed) | Ring Attention (FSDP2) |\n| ------------------------- | --------------------------------- | ---------------------- |\n| **Parallelism Method** | Attention head partitioning | Ring-based KV exchange |\n| **Backend** | DeepSpeed ZeRO | PyTorch FSDP2 |\n| **Attention Support** | FlashAttention 2/3, SDPA | SDPA only |\n| **Communication** | Two `all-to-all`s per layer | P2P ring communication |\n| **Comm volume per GPU** | O(total\\_seq x hidden / sp\\_size) | O(total\\_seq x hidden) |\n| **Sequence Divisibility** | `sp_size` | `cp_size * 2` |\n| **Num Head Constraint** | `num_heads >= sp_size` | None |\n\n### When to Choose Ulysses vs Ring Attention\n\nSince switching between the two only requires changing the accelerate config, we recommend trying both and comparing performance and memory usage on your specific setup. The main constraint is that Ulysses requires `num_heads >= sp_size`, while Ring Attention has no such limitation.\n\n## Best Practices\n\n### 1. Sequence Length Divisibility\n\nAlways ensure your sequence length is divisible by `sp_size`:\n\n```python\ntraining_args = SFTConfig(\n pad_to_multiple_of=4, # For sp_size=4\n max_length=32768, # Must be divisible by 4\n)\n```\n\n### 2. Use Flash Attention\n\nFlash Attention 2 provides cleaner output and better performance than SDPA:\n\n```python\nparallelism_config = ParallelismConfig(\n sp_handler=DeepSpeedSequenceParallelConfig(\n sp_attn_implementation=\"flash_attention_2\",\n ),\n)\n```\n\nUse Flash Attention 3 for Hopper and look out for Flash Attention 4 release for Blackwell (FA2 on Blackwell is quite slow).\n\n### 3. Combine with DeepSpeed ZeRO\n\nFor very large models, combine Ulysses with ZeRO Stage 3:\n\n```yaml\ndeepspeed_config:\n zero_stage: 3\n offload_optimizer:\n device: cpu\n```\n\nIf the model is huge, you can offload the params as well by adding to the above:\n\n```yaml\n offload_param:\n device: cpu\n```\n\n### 5. Use memory fragmentation-friendly PyTorch allocator\n\nThis environment variable will allow for a longer sequence length:\n\n```bash\nexport PYTORCH_ALLOC_CONF=expandable_segments:True\n```\n\n### 6. 2D Parallelism Configuration\n\nBalance SP and DP for your GPU count:\n\n| GPUs | `sp_size` | `dp_shard_size` | Use Case |\n| ---- | --------- | --------------- | ----------------------------------------------- |\n| 4 | 2 | 2 | Balanced throughput and sequence length |\n| 4 | 4 | 1 | Maximum sequence length |\n| 8 | 2 | 4 | Higher throughput with moderate sequence length |\n| 8 | 4 | 2 | Longer sequences with moderate throughput |\n\nRemember: `dp_replicate_size × dp_shard_size × sp_size = num_processes`\n\n### 7. Liger-Kernel\n\nIf your desired model architecture is supported by [Liger-Kernel](https://github.com/linkedin/Liger-Kernel), it is fully compatible with Ulysses SP and can be enabled with a single flag:\n\n```python\ntraining_args = SFTConfig(\n use_liger_kernel=True,\n)\n```\n\nThe main memory saving comes from `FusedLinearCrossEntropy` which avoids materializing the full logits tensor during loss calculation. The savings grow with longer sequences where the logits tensor is larger.\n\nAdditionally, you can enable [`TiledMLP`](https://www.deepspeed.ai/tutorials/ulysses-alst-sequence-parallelism/#tiled-mlp-computation) to further extend sequence length — like `FusedLinearCrossEntropy`, it saves working memory by tiling large matrix operations.\n\n### 8. Token Distribution Across Ranks\n\nYou don't need to worry about manually balancing tokens across SP ranks—the loss aggregation code handles uneven distributions gracefully (including ranks with zero valid tokens). With random batching over a reasonably sized dataset, the distribution evens out statistically over training.\n\n## Benchmarks\n\nTo quantify the benefits of Ulysses SP, we trained [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) on the [Gutenberg English](https://huggingface.co/datasets/sedthh/gutenberg_english) streaming dataset using TRL's SFTTrainer. All experiments ran on H100 80GB GPUs with DeepSpeed ZeRO-3, CPU optimizer offloading, gradient checkpointing, and flash-attn2 as the attention backend.\n\n### Setup\n\n| Config | GPUs | SP | DP | Seq Length | Grad Acc | Global Batch |\n| -------- | ---- | -- | -- | ---------- | -------- | ------------ |\n| Baseline | 1 | 1 | 1 | 8K | 8 | 8 |\n| SP=4 | 4 | 4 | 1 | 8K | 8 | 8 |\n| SP=4 | 4 | 4 | 1 | 32K | 8 | 8 |\n| SP=4 | 4 | 4 | 1 | 64K | 8 | 8 |\n| SP=4 | 4 | 4 | 1 | 96K | 8 | 8 |\n\nThe benchmark runs in the table above use the same global batch size (8 micro-batches), cosine learning-rate schedule, and seed, so those benchmark loss curves are directly comparable.\n\n### Loss Curve Matching Diagnostics (4 GPU)\n\nTo verify SP-vs-DP loss equivalence, we ran controlled 4-GPU A/B experiments with identical seed, model, optimizer, learning-rate schedule, and data order.\n\n#### Methodology for Fair DP vs SP Comparison\n\nCompared setups:\n\n- `DP=4, SP=1, GAS=1` (baseline)\n- `DP=1, SP=4, GAS=4` (Ulysses SP)\n\nFor fair comparison, `GAS` must scale with `SP`:\n\n- Ulysses SP splits the sequence across `SP` ranks, so each SP rank sees roughly `1/SP` of the sequence tokens per micro-step.\n- If `GAS` is unchanged, each optimizer step in SP aggregates fewer total tokens than the DP baseline.\n- Setting `GAS=SP` keeps effective tokens per optimizer step matched:\n - DP tokens/step: `dp_world_size * micro_batch * seq_len * GAS = 4 * B * L * 1`\n - SP tokens/step: `dp_world_size * micro_batch * (L/SP) * GAS * SP_ranks = 1 * B * (L/4) * 4 * 4 = 4 * B * L`\n\n![Canonical loss on Gutenberg for DP=4 vs SP=4](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ulysses/gutenberg_canonical_loss_dp4_vs_sp4.png)\\\nOn Gutenberg text (20 steps), canonical loss matches within logging precision between `DP=4,SP=1,GAS=1` and `DP=1,SP=4,GAS=4`.\n\nMeasured over 20 steps on 4 GPUs in controlled equivalence harnesses:\n\n| Harness | Metric | DP vs SP setting | Mean abs diff | Max abs diff |\n| ------------ | ------------- | ------------------------ | ------------- | ------------ |\n| `Trainer` | `loss` | DP=4, SP=1 vs DP=1, SP=4 | 0.0054 | 0.0131 |\n| `SFTTrainer` | logged `loss` | DP=4, SP=1 vs DP=1, SP=4 | 0.0811 | 0.0812 |\n| `SFTTrainer` | canonical NLL | DP=4, SP=1 vs DP=1, SP=4 | 0.000004 | 0.000005 |\n\nTakeaway: under matched token budget, SP and non-SP match on canonical token-normalized loss. The remaining difference is in trainer-reported logging (`loss`), not in the underlying cross-entropy objective.\n\n### Memory Reduction\n\n![Peak GPU Memory per Rank](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ulysses/memory_usage.png)\\\nSP=4 reduces per-GPU memory by 3.3x at the same sequence length, enabling training at up to 96K tokens on 4× H100 80GB. At 128K, the model OOMs.\n\n| Config | Seq Length | Peak Memory | Notes |\n| ------------ | ---------- | ----------- | --------------------------------- |\n| DP=4 (4 GPU) | 8K | 22.4 GB | Baseline — no SP |\n| SP=4 (4 GPU) | 8K | 22.8 GB | Similar memory at same seq length |\n| SP=4 (4 GPU) | 32K | 35.0 GB | 4x longer than DP baseline |\n| SP=4 (4 GPU) | 64K | 50.5 GB | 8x longer than DP baseline |\n| SP=4 (4 GPU) | 96K | 66.0 GB | **12x longer than DP baseline** |\n| SP=4 (4 GPU) | 128K | OOM | Exceeds 80 GB limit |\n\nAt 8K tokens, DP=4 and SP=4 use nearly the same memory per GPU (~22 GB with ZeRO-3). The advantage of SP is that it enables scaling to much longer sequences: at 96K tokens (12x longer), peak memory is 66 GB — still within the H100's 80 GB capacity. At 128K, the model OOMs, establishing the practical limit for this configuration. DP=4 without SP cannot scale beyond 8K for this model.\n\n### Throughput\n\n![Training Throughput](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ulysses/throughput.png)\\\nLonger sequences with SP process dramatically more tokens per second. SP=4 at 64K achieves 3.7x the throughput of the baseline.\n\n| Config | Seq Length | Tokens/s | vs Baseline |\n| ---------------- | ---------- | -------- | ----------- |\n| Baseline (1 GPU) | 8K | 3,633 | — |\n| SP=4 (4 GPU) | 8K | 3,933 | \\~1x |\n| SP=4 (4 GPU) | 32K | 7,733 | **2.1x** |\n| SP=4 (4 GPU) | 64K | 13,396 | **3.7x** |\n\nAt the same sequence length (8K), SP=4 has comparable throughput to the single-GPU baseline — the all-to-all communication overhead is minimal on NVLink-connected GPUs. The real benefit comes from longer sequences: as sequence length grows, the quadratic attention computation dominates over communication and other overheads, making each training step increasingly compute-efficient. Each step also processes proportionally more tokens, so throughput scales with sequence length. At 64K, SP=4 processes 13,396 tokens/second — 3.7x the baseline.\n\n> These results use only 4 GPUs with SP=4. With 8 GPUs (SP=8), you can push to even longer sequences — up to 256K+ tokens — or use 2D parallelism (SP=4, DP=2) to combine long-context training with data-parallel throughput.\n\n## Requirements\n\n- HF Accelerate: `deepspeed>=0.18.1 accelerate>=1.12`\n- HF Trainer: `deepspeed>=0.18.1 accelerate>=1.12 transformers>=5.0`\n- HF TRL: `deepspeed>=0.18.1 accelerate>=1.12 transformers>=5.0 trl>=0.18.0`\n\nUse `flash_attention_2` for Ampere GPUs, or `flash_attention_3` for Hopper GPUs. Wait for `flash_attention_4` on Blackwell 🕰.\n\n## Resources\n\n### Documentation\n\n- [Accelerate: Context Parallelism Guide](https://huggingface.co/docs/accelerate/concept_guides/context_parallelism)\n- [TRL: Distributing Training](https://huggingface.co/docs/trl/distributing_training)\n- [DeepSpeed Sequence Parallelism](https://www.deepspeed.ai/tutorials/ulysses-alst-sequence-parallelism)\n\n### Examples\n\n- [Accelerate ALST Example](https://github.com/huggingface/accelerate/tree/main/examples/alst_ulysses_sequence_parallelism)\n- [TRL Accelerate Configs](https://github.com/huggingface/trl/tree/main/examples/accelerate_configs)\n\n### Papers\n\n- [Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences](https://huggingface.co/papers/2506.13996)\n- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://huggingface.co/papers/2309.14509)\n\n### Related Blog Posts\n\n- [Accelerate ND-Parallel: A Guide to Efficient Multi-GPU Training](https://huggingface.co/blog/accelerate-nd-parallel)\n- [Understanding Ulysses and Ring Attention](https://huggingface.co/blog/exploding-gradients/ulysses-ring-attention)\n- [Enabling Long-Context Training with Sequence Parallelism in Axolotl](https://huggingface.co/blog/axolotl-ai-co/long-context-with-sequence-parallelism-in-axolotl)\n", "authors": [], "captured_at": "2026-10-04T12:50:30.804479567+00:00", "category": "ai", "content_html": "

Training large language models on long sequences has become essential for building capable AI systems. As models are increasingly used for tasks like document analysis, code understanding, complex reasoning, and RAG workloads, the need to process sequences of hundreds of thousands—or even millions—of tokens has grown dramatically. To put this in perspective, an average book is roughly 250k tokens, so training on multi-document contexts or book-length inputs requires handling sequences well beyond what fits on a single GPU. However, training with such long contexts presents significant memory challenges: the attention computation scales quadratically with sequence length, quickly exceeding GPU memory for contexts beyond tens of thousands of tokens.

\n

Ulysses Sequence Parallelism (part of the Arctic Long Sequence Training (ALST) protocol from Snowflake AI Research) provides an elegant solution by distributing the attention computation across multiple GPUs through attention head parallelism. In this post, we'll explore how Ulysses works and how it's been integrated across the Hugging Face ecosystem—from Accelerate to the Transformers Trainer and TRL's SFTTrainer.

\n

Contents

\n\n

The Challenge of Long Sequence Training

\n

The attention mechanism in transformers scales quadratically with sequence length. For a sequence of length n n, standard attention requires O(n2) O(n^2) FLOPs and O(n2) O(n^2) memory to compute and store the attention score matrix. Optimized implementations like FlashAttention reduce the memory to O(n) O(n) by tiling the computation and never materializing the full attention matrix—but the O(n2) O(n^2) compute remains. For very long sequences (32k+ tokens), even with FlashAttention, training still pushes the limits of single-GPU memory.

\n

Consider these scenarios where long-context training is essential:

\n\n

Traditional data parallelism doesn't help here—each GPU still needs to process the full sequence inside the attention block. We need a way to split the sequence itself across multiple devices.

\n

How Ulysses Works

\n

Ulysses Sequence Parallelism (SP), introduced in the DeepSpeed Ulysses paper, takes a clever approach: in addition to splitting on the sequence dimension, it also partitions the attention heads across GPUs.

\n
\"Ulysses
Ulysses splits input sequences along the sequence dimension and uses all-to-all communication to exchange key-value pairs, enabling each GPU to compute a subset of attention heads. (Source: Snowflake Engineering Blog)
\n

Here's how it works:

\n
    \n
  1. \n

    Sequence Sharding: The input sequence is split along the sequence dimension across P P GPUs. Each GPU i i holds tokens [i⋅n/P,(i+1)⋅n/P) [i \\cdot n/P, (i+1) \\cdot n/P).

    \n
    2. \n
  2. \n

    QKV Projection: Each GPU computes the query, key, and value projections for its local sequence chunk.

    \n
    3. \n
  3. \n

    All-to-All Communication: An all-to-all collective operation redistributes the data so that each GPU holds all sequence positions after the projections, but only for a subset of attention heads.

    \n
    4. \n
  4. \n

    Local Attention: Each GPU computes attention for its assigned heads using standard attention mechanisms (FlashAttention or SDPA).

    \n
    5. \n
  5. \n

    All-to-All Communication: Another all-to-all operation reverses the redistribution, returning to sequence-sharded format.

    \n
    6. \n
  6. \n

    Output Projection: Each GPU computes the output projection for its local sequence chunk.

    \n
    7. \n
\n

The key insight is that attention heads are independent—each head can be computed separately. By trading sequence locality for head locality, Ulysses enables efficient parallelization with relatively low communication overhead.

\n

Communication Complexity

\n

Ulysses requires two all-to-all operations per attention layer, with total communication volume of O(n⋅d/P) O(n \\cdot d / P) per GPU, where:

\n\n

Ring Attention communicates O(n⋅d) O(n \\cdot d) per GPU — a factor of P P more — via P−1 P-1 sequential point-to-point transfers around the ring. Ulysses also benefits from lower latency because all-to-all can exploit full bisectional bandwidth in a single collective step, whereas Ring Attention serializes over P−1 P-1 hops.

\n

Integration with Accelerate

\n

Accelerate provides the foundation for Ulysses sequence parallelism through its ParallelismConfig class and DeepSpeed integration.

\n

Configuration

\n
from accelerate import Accelerator\nfrom accelerate.utils import ParallelismConfig, DeepSpeedSequenceParallelConfig\n\nparallelism_config = ParallelismConfig(\n    sp_backend="deepspeed",\n    sp_size=4,  # Split across 4 GPUs\n    dp_shard_size=1,  # Must satisfy: dp_replicate × dp_shard × sp_size = num_processes\n    sp_handler=DeepSpeedSequenceParallelConfig(\n        sp_seq_length=None,  # None for variable-length sequences\n        sp_seq_length_is_variable=True,\n        sp_attn_implementation="flash_attention_2",  # or "sdpa"\n    ),\n)\n\naccelerator = Accelerator(parallelism_config=parallelism_config)\n
\n

Key Parameters

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ParameterDescription
sp_sizeNumber of GPUs for sequence parallelism
sp_backendMust be "deepspeed" for Ulysses
sp_seq_length_is_variableSet to True for varying sequence lengths across batches
sp_attn_implementation"flash_attention_2", "flash_attention_3", or "sdpa"
\n

Using the Accelerator

\n

When you call accelerator.prepare(), Ulysses is automatically set up:

\n
from transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")\noptimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)\n\n# This registers the model with Ulysses and wraps the dataloader\nmodel, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)\n
\n

The prepare() call:

\n
    \n
  1. Registers the model with DeepSpeed's UlyssesSPAttentionHF
    2. \n
  2. Wraps the dataloader with UlyssesSPDataLoaderAdapter to handle sequence sharding
    3. \n
  3. Automatically injects shift_labels for correct loss computation
    4. \n
\n

Loss Aggregation

\n

With Ulysses, each GPU computes loss on different parts of the sequence. The losses must be aggregated properly, weighted by the number of valid tokens per rank. If you're using the Transformers Trainer or TRL's SFTTrainer, this is handled automatically—the code below is only needed when writing a custom Accelerate training loop:

\n
sp_size = parallelism_config.sp_size\nif sp_size > 1:\n    from deepspeed.utils import groups\n\n    sp_group = groups._get_sequence_parallel_group()\n\n    # Gather losses and token counts from all SP ranks\n    losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)\n    good_tokens = (batch["shift_labels"] != -100).view(-1).sum()\n    good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)\n\n    # Weighted aggregation\n    total_loss = sum(\n        losses_per_rank[i] * good_tokens_per_rank[i]\n        for i in range(sp_size)\n        if good_tokens_per_rank[i] > 0\n    )\n    loss = total_loss / max(sum(good_tokens_per_rank), 1)\n\naccelerator.backward(loss)\n
\n
\n

The weighted loss aggregation ensures correct gradients when tokens are unevenly distributed across ranks (e.g., when some ranks contain only padding or masked out prompt tokens).

\n
\n
\n

Both Ulysses and Ring Attention use position_ids instead of attention_mask for causal masking during training. A 4D attention mask at these sequence lengths would be just as prohibitive as the attention scores themselves—at 128k tokens, that's another ~1TB tensor. Position IDs achieve the same causal behavior with O(n) O(n) memory instead of O(n2) O(n^2). During evaluation/inference, DeepSpeed's SP attention layer can bypass the SP operations entirely (via disable_in_eval) and fall back to the model's default attention implementation.

\n
\n

Integration with Transformers Trainer

\n

The Transformers Trainer provides seamless Ulysses integration through TrainingArguments.parallelism_config. It handles all the SP-specific details automatically—dataloader wrapping, sequence sharding, and loss aggregation—so you don't need to write any of the custom loss code shown above.

\n

Configuration

\n

Just pass the same parallelism_config from above into TrainingArguments:

\n
from transformers import TrainingArguments\n\ntraining_args = TrainingArguments(\n    parallelism_config=parallelism_config,  # same ParallelismConfig as above\n    per_device_train_batch_size=1,\n)\n
\n

What the Trainer Handles Automatically

\n
    \n
  1. \n

    Dataloader Wrapping: After model preparation, the Trainer wraps the dataloader with UlyssesSPDataLoaderAdapter

    \n
    2. \n
  2. \n

    Loss Computation: The compute_loss method detects SP mode and routes to specialized _deepspeed_sp_compute_loss which handles:

    \n\n
    3. \n
  3. \n

    Batch Size Calculation: The effective data parallel world size accounts for SP:

    \n
    dp_world_size = world_size // sp_size\n
    \n
    4. \n
  4. \n

    Dataloader Length Adjustment: Training step calculations are adjusted for SP's effect on iteration count

    \n
    5. \n
\n

Launch Command

\n

Use an accelerate config file or command-line arguments:

\n
accelerate launch \\\n    --config_file deepspeed_ulysses.yaml \\\n    train.py \\\n    --per_device_train_batch_size 1\n
\n

Integration with TRL SFTTrainer

\n

TRL's SFTTrainer builds on the Transformers Trainer and adds specific optimizations for supervised fine-tuning with long sequences.

\n

Configuration

\n
from trl import SFTConfig, SFTTrainer\nfrom accelerate.utils import ParallelismConfig, DeepSpeedSequenceParallelConfig\n\nparallelism_config = ParallelismConfig(\n    sp_backend="deepspeed",\n    sp_size=2,\n    dp_shard_size=2,  # 2D parallelism: SP × DP = 4 GPUs\n    sp_handler=DeepSpeedSequenceParallelConfig(\n        sp_seq_length_is_variable=True,\n        sp_attn_implementation="flash_attention_2",\n    ),\n)\n\ntraining_args = SFTConfig(\n    ...,\n    parallelism_config=parallelism_config,\n    max_length=32768,\n    pad_to_multiple_of=2,  # Must equal sp_size\n    per_device_train_batch_size=1,\n)\n\ntrainer = SFTTrainer(\n    model=model,\n    args=training_args,\n    train_dataset=dataset,\n)\ntrainer.train()\n
\n

Key SFTConfig Parameters for Ulysses

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ParameterDescription
pad_to_multiple_ofMust equal sp_size to ensure sequence divisibility
max_lengthGlobal sequence length (before splitting across GPUs)
packingWorks well with SP — packing reduces padding waste, especially for variable-length sequences
\n

Accelerate Config File

\n

Create alst_ulysses_4gpu.yaml:

\n
compute_environment: LOCAL_MACHINE\ndistributed_type: DEEPSPEED\nmixed_precision: bf16\nnum_processes: 4\ndeepspeed_config:\n  zero_stage: 3\n  seq_parallel_communication_data_type: bf16\nparallelism_config:\n  parallelism_config_sp_size: 2\n  parallelism_config_sp_backend: deepspeed\n  parallelism_config_dp_shard_size: 2\n  parallelism_config_sp_seq_length_is_variable: true\n  parallelism_config_sp_attn_implementation: flash_attention_2\n
\n

Complete Training Command

\n
accelerate launch --config_file alst_ulysses_4gpu.yaml \\\n    trl/scripts/sft.py \\\n    --model_name_or_path meta-llama/Llama-3.1-8B \\\n    --dataset_name trl-lib/Capybara \\\n    --max_length 32768 \\\n    --packing \\\n    --pad_to_multiple_of 2 \\\n    --per_device_train_batch_size 1\n
\n

Shift Labels Handling

\n

The SFTTrainer automatically handles pre-shifted labels when Ulysses is enabled:

\n
# When using SP, labels are pre-shifted by the dataloader adapter\n# The trainer detects this and uses shift_labels directly\nlabels = inputs["labels"] if "shift_labels" not in inputs else None\n\n# Loss computation uses the pre-shifted labels\nif "shift_labels" in inputs:\n    shift_logits = outputs.logits.contiguous()\n    shift_labels = inputs["shift_labels"]\nelse:\n    shift_logits = outputs.logits[..., :-1, :].contiguous()\n    shift_labels = labels[..., 1:].contiguous()\n
\n

Comparing Ulysses and Ring Attention

\n

Both Ulysses and Ring Attention enable long-context training, but they have different characteristics:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
AspectUlysses (DeepSpeed)Ring Attention (FSDP2)
Parallelism MethodAttention head partitioningRing-based KV exchange
BackendDeepSpeed ZeROPyTorch FSDP2
Attention SupportFlashAttention 2/3, SDPASDPA only
CommunicationTwo all-to-alls per layerP2P ring communication
Comm volume per GPUO(total_seq x hidden / sp_size)O(total_seq x hidden)
Sequence Divisibilitysp_sizecp_size * 2
Num Head Constraintnum_heads >= sp_sizeNone
\n

When to Choose Ulysses vs Ring Attention

\n

Since switching between the two only requires changing the accelerate config, we recommend trying both and comparing performance and memory usage on your specific setup. The main constraint is that Ulysses requires num_heads >= sp_size, while Ring Attention has no such limitation.

\n

Best Practices

\n

1. Sequence Length Divisibility

\n

Always ensure your sequence length is divisible by sp_size:

\n
training_args = SFTConfig(\n    pad_to_multiple_of=4,  # For sp_size=4\n    max_length=32768,  # Must be divisible by 4\n)\n
\n

2. Use Flash Attention

\n

Flash Attention 2 provides cleaner output and better performance than SDPA:

\n
parallelism_config = ParallelismConfig(\n    sp_handler=DeepSpeedSequenceParallelConfig(\n        sp_attn_implementation="flash_attention_2",\n    ),\n)\n
\n

Use Flash Attention 3 for Hopper and look out for Flash Attention 4 release for Blackwell (FA2 on Blackwell is quite slow).

\n

3. Combine with DeepSpeed ZeRO

\n

For very large models, combine Ulysses with ZeRO Stage 3:

\n
deepspeed_config:\n  zero_stage: 3\n  offload_optimizer:\n    device: cpu\n
\n

If the model is huge, you can offload the params as well by adding to the above:

\n
  offload_param:\n    device: cpu\n
\n

5. Use memory fragmentation-friendly PyTorch allocator

\n

This environment variable will allow for a longer sequence length:

\n
export PYTORCH_ALLOC_CONF=expandable_segments:True\n
\n

6. 2D Parallelism Configuration

\n

Balance SP and DP for your GPU count:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
GPUssp_sizedp_shard_sizeUse Case
422Balanced throughput and sequence length
441Maximum sequence length
824Higher throughput with moderate sequence length
842Longer sequences with moderate throughput
\n

Remember: dp_replicate_size × dp_shard_size × sp_size = num_processes

\n

7. Liger-Kernel

\n

If your desired model architecture is supported by Liger-Kernel, it is fully compatible with Ulysses SP and can be enabled with a single flag:

\n
training_args = SFTConfig(\n    use_liger_kernel=True,\n)\n
\n

The main memory saving comes from FusedLinearCrossEntropy which avoids materializing the full logits tensor during loss calculation. The savings grow with longer sequences where the logits tensor is larger.

\n

Additionally, you can enable TiledMLP to further extend sequence length — like FusedLinearCrossEntropy, it saves working memory by tiling large matrix operations.

\n

8. Token Distribution Across Ranks

\n

You don't need to worry about manually balancing tokens across SP ranks—the loss aggregation code handles uneven distributions gracefully (including ranks with zero valid tokens). With random batching over a reasonably sized dataset, the distribution evens out statistically over training.

\n

Benchmarks

\n

To quantify the benefits of Ulysses SP, we trained Qwen3-4B on the Gutenberg English streaming dataset using TRL's SFTTrainer. All experiments ran on H100 80GB GPUs with DeepSpeed ZeRO-3, CPU optimizer offloading, gradient checkpointing, and flash-attn2 as the attention backend.

\n

Setup

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ConfigGPUsSPDPSeq LengthGrad AccGlobal Batch
Baseline1118K88
SP=44418K88
SP=444132K88
SP=444164K88
SP=444196K88
\n

The benchmark runs in the table above use the same global batch size (8 micro-batches), cosine learning-rate schedule, and seed, so those benchmark loss curves are directly comparable.

\n

Loss Curve Matching Diagnostics (4 GPU)

\n

To verify SP-vs-DP loss equivalence, we ran controlled 4-GPU A/B experiments with identical seed, model, optimizer, learning-rate schedule, and data order.

\n

Methodology for Fair DP vs SP Comparison

\n

Compared setups:

\n\n

For fair comparison, GAS must scale with SP:

\n\n
\"Canonical
On Gutenberg text (20 steps), canonical loss matches within logging precision between DP=4,SP=1,GAS=1 and DP=1,SP=4,GAS=4.
\n

Measured over 20 steps on 4 GPUs in controlled equivalence harnesses:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
HarnessMetricDP vs SP settingMean abs diffMax abs diff
TrainerlossDP=4, SP=1 vs DP=1, SP=40.00540.0131
SFTTrainerlogged lossDP=4, SP=1 vs DP=1, SP=40.08110.0812
SFTTrainercanonical NLLDP=4, SP=1 vs DP=1, SP=40.0000040.000005
\n

Takeaway: under matched token budget, SP and non-SP match on canonical token-normalized loss. The remaining difference is in trainer-reported logging (loss), not in the underlying cross-entropy objective.

\n

Memory Reduction

\n
\"Peak
SP=4 reduces per-GPU memory by 3.3x at the same sequence length, enabling training at up to 96K tokens on 4× H100 80GB. At 128K, the model OOMs.
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ConfigSeq LengthPeak MemoryNotes
DP=4 (4 GPU)8K22.4 GBBaseline — no SP
SP=4 (4 GPU)8K22.8 GBSimilar memory at same seq length
SP=4 (4 GPU)32K35.0 GB4x longer than DP baseline
SP=4 (4 GPU)64K50.5 GB8x longer than DP baseline
SP=4 (4 GPU)96K66.0 GB12x longer than DP baseline
SP=4 (4 GPU)128KOOMExceeds 80 GB limit
\n

At 8K tokens, DP=4 and SP=4 use nearly the same memory per GPU (~22 GB with ZeRO-3). The advantage of SP is that it enables scaling to much longer sequences: at 96K tokens (12x longer), peak memory is 66 GB — still within the H100's 80 GB capacity. At 128K, the model OOMs, establishing the practical limit for this configuration. DP=4 without SP cannot scale beyond 8K for this model.

\n

Throughput

\n
\"Training
Longer sequences with SP process dramatically more tokens per second. SP=4 at 64K achieves 3.7x the throughput of the baseline.
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ConfigSeq LengthTokens/svs Baseline
Baseline (1 GPU)8K3,633—
SP=4 (4 GPU)8K3,933~1x
SP=4 (4 GPU)32K7,7332.1x
SP=4 (4 GPU)64K13,3963.7x
\n

At the same sequence length (8K), SP=4 has comparable throughput to the single-GPU baseline — the all-to-all communication overhead is minimal on NVLink-connected GPUs. The real benefit comes from longer sequences: as sequence length grows, the quadratic attention computation dominates over communication and other overheads, making each training step increasingly compute-efficient. Each step also processes proportionally more tokens, so throughput scales with sequence length. At 64K, SP=4 processes 13,396 tokens/second — 3.7x the baseline.

\n
\n

These results use only 4 GPUs with SP=4. With 8 GPUs (SP=8), you can push to even longer sequences — up to 256K+ tokens — or use 2D parallelism (SP=4, DP=2) to combine long-context training with data-parallel throughput.

\n
\n

Requirements

\n\n

Use flash_attention_2 for Ampere GPUs, or flash_attention_3 for Hopper GPUs. Wait for flash_attention_4 on Blackwell 🕰.

\n

Resources

\n

Documentation

\n\n

Examples

\n\n

Papers

\n\n

Related Blog Posts

\n\n", "content_markdown": "Training large language models on long sequences has become essential for building capable AI systems. As models are increasingly used for tasks like document analysis, code understanding, complex reasoning, and RAG workloads, the need to process sequences of hundreds of thousands—or even millions—of tokens has grown dramatically. To put this in perspective, an average book is roughly 250k tokens, so training on multi-document contexts or book-length inputs requires handling sequences well beyond what fits on a single GPU. However, training with such long contexts presents significant memory challenges: the attention computation scales quadratically with sequence length, quickly exceeding GPU memory for contexts beyond tens of thousands of tokens.\n\nUlysses Sequence Parallelism (part of the [Arctic Long Sequence Training (ALST) protocol](https://huggingface.co/papers/2506.13996) from Snowflake AI Research) provides an elegant solution by distributing the attention computation across multiple GPUs through attention head parallelism. In this post, we'll explore how Ulysses works and how it's been integrated across the Hugging Face ecosystem—from Accelerate to the Transformers Trainer and TRL's SFTTrainer.\n\n## Contents\n\n- [The Challenge of Long Sequence Training](https://huggingface.co/blog/ulysses-sp#the-challenge-of-long-sequence-training)\n- [How Ulysses Works](https://huggingface.co/blog/ulysses-sp#how-ulysses-works)\n- [Integration with Accelerate](https://huggingface.co/blog/ulysses-sp#integration-with-accelerate)\n- [Integration with Transformers Trainer](https://huggingface.co/blog/ulysses-sp#integration-with-transformers-trainer)\n- [Integration with TRL's SFTTrainer](https://huggingface.co/blog/ulysses-sp#integration-with-trl-sfttrainer)\n- [Comparing Ulysses and Ring Attention](https://huggingface.co/blog/ulysses-sp#comparing-ulysses-and-ring-attention)\n- [Best Practices](https://huggingface.co/blog/ulysses-sp#best-practices)\n- [Benchmarks](https://huggingface.co/blog/ulysses-sp#benchmarks)\n- [Resources](https://huggingface.co/blog/ulysses-sp#resources)\n\n## The Challenge of Long Sequence Training\n\nThe attention mechanism in transformers scales quadratically with sequence length. For a sequence of length n n, standard attention requires O(n2) O(n^2) FLOPs and O(n2) O(n^2) memory to compute and store the attention score matrix. Optimized implementations like [FlashAttention](https://huggingface.co/papers/2205.14135) reduce the **memory** to O(n) O(n) by tiling the computation and never materializing the full attention matrix—but the O(n2) O(n^2) compute remains. For very long sequences (32k+ tokens), even with FlashAttention, training still pushes the limits of single-GPU memory.\n\nConsider these scenarios where long-context training is essential:\n\n- **Document understanding**: Processing entire books, legal documents, or research papers\n- **Code analysis**: Understanding large codebases with multiple interconnected files\n- **Reasoning tasks**: Models that \"think\" step-by-step may generate thousands of tokens during inference\n- **Retrieval-augmented generation**: Incorporating many retrieved passages into the context\n\nTraditional data parallelism doesn't help here—each GPU still needs to process the full sequence inside the attention block. We need a way to split the sequence itself across multiple devices.\n\n## How Ulysses Works\n\nUlysses Sequence Parallelism (SP), introduced in the [DeepSpeed Ulysses paper](https://huggingface.co/papers/2309.14509), takes a clever approach: in addition to splitting on the sequence dimension, it also partitions the attention heads across GPUs.\n\n![Ulysses Sequence Parallelism Overview](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ulysses/ulysses_overview.png)\\\nUlysses splits input sequences along the sequence dimension and uses all-to-all communication to exchange key-value pairs, enabling each GPU to compute a subset of attention heads. (***Source: [Snowflake Engineering Blog](https://www.snowflake.com/en/engineering-blog/ulysses-low-latency-llm-inference/)***)\n\nHere's how it works:\n\n1. **Sequence Sharding**: The input sequence is split along the sequence dimension across P P GPUs. Each GPU i i holds tokens \\[i⋅n/P,(i+1)⋅n/P) \\[i \\\\cdot n/P, (i+1) \\\\cdot n/P).\n\n2. **QKV Projection**: Each GPU computes the query, key, and value projections for its local sequence chunk.\n\n3. **All-to-All Communication**: An all-to-all collective operation redistributes the data so that each GPU holds *all* sequence positions after the projections, but only for a subset of attention heads.\n\n4. **Local Attention**: Each GPU computes attention for its assigned heads using standard attention mechanisms (FlashAttention or SDPA).\n\n5. **All-to-All Communication**: Another all-to-all operation reverses the redistribution, returning to sequence-sharded format.\n\n6. **Output Projection**: Each GPU computes the output projection for its local sequence chunk.\n\nThe key insight is that attention heads are independent—each head can be computed separately. By trading sequence locality for head locality, Ulysses enables efficient parallelization with relatively low communication overhead.\n\n### Communication Complexity\n\nUlysses requires two all-to-all operations per attention layer, with total communication volume of O(n⋅d/P) O(n \\\\cdot d / P) per GPU, where:\n\n- n n is the sequence length\n- d d is the hidden dimension\n- P P is the parallelism degree\n\nRing Attention communicates O(n⋅d) O(n \\\\cdot d) per GPU — a factor of P P more — via P−1 P-1 sequential point-to-point transfers around the ring. Ulysses also benefits from lower latency because all-to-all can exploit full bisectional bandwidth in a single collective step, whereas Ring Attention serializes over P−1 P-1 hops.\n\n## Integration with Accelerate\n\nAccelerate provides the foundation for Ulysses sequence parallelism through its [`ParallelismConfig`](https://huggingface.co/docs/accelerate/concept_guides/sequence_parallelism#alstulysses-sp-backend-configuration) class and DeepSpeed integration.\n\n### Configuration\n\n```python\nfrom accelerate import Accelerator\nfrom accelerate.utils import ParallelismConfig, DeepSpeedSequenceParallelConfig\n\nparallelism_config = ParallelismConfig(\n sp_backend=\"deepspeed\",\n sp_size=4, # Split across 4 GPUs\n dp_shard_size=1, # Must satisfy: dp_replicate × dp_shard × sp_size = num_processes\n sp_handler=DeepSpeedSequenceParallelConfig(\n sp_seq_length=None, # None for variable-length sequences\n sp_seq_length_is_variable=True,\n sp_attn_implementation=\"flash_attention_2\", # or \"sdpa\"\n ),\n)\n\naccelerator = Accelerator(parallelism_config=parallelism_config)\n```\n\n### Key Parameters\n\n| Parameter | Description |\n| --------------------------- | --------------------------------------------------------- |\n| `sp_size` | Number of GPUs for sequence parallelism |\n| `sp_backend` | Must be `\"deepspeed\"` for Ulysses |\n| `sp_seq_length_is_variable` | Set to `True` for varying sequence lengths across batches |\n| `sp_attn_implementation` | `\"flash_attention_2\"`, `\"flash_attention_3\"`, or `\"sdpa\"` |\n\n### Using the Accelerator\n\nWhen you call `accelerator.prepare()`, Ulysses is automatically set up:\n\n```python\nfrom transformers import AutoModelForCausalLM\n\nmodel = AutoModelForCausalLM.from_pretrained(\"meta-llama/Llama-3.1-8B\")\noptimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)\n\n# This registers the model with Ulysses and wraps the dataloader\nmodel, optimizer, dataloader = accelerator.prepare(model, optimizer, dataloader)\n```\n\nThe `prepare()` call:\n\n1. Registers the model with DeepSpeed's `UlyssesSPAttentionHF`\n2. Wraps the dataloader with `UlyssesSPDataLoaderAdapter` to handle sequence sharding\n3. Automatically injects `shift_labels` for correct loss computation\n\n### Loss Aggregation\n\nWith Ulysses, each GPU computes loss on different parts of the sequence. The losses must be aggregated properly, weighted by the number of valid tokens per rank. If you're using the Transformers `Trainer` or TRL's `SFTTrainer`, this is handled automatically—the code below is only needed when writing a custom Accelerate training loop:\n\n```python\nsp_size = parallelism_config.sp_size\nif sp_size > 1:\n from deepspeed.utils import groups\n\n sp_group = groups._get_sequence_parallel_group()\n\n # Gather losses and token counts from all SP ranks\n losses_per_rank = torch.distributed.nn.functional.all_gather(loss, group=sp_group)\n good_tokens = (batch[\"shift_labels\"] != -100).view(-1).sum()\n good_tokens_per_rank = torch.distributed.nn.functional.all_gather(good_tokens, group=sp_group)\n\n # Weighted aggregation\n total_loss = sum(\n losses_per_rank[i] * good_tokens_per_rank[i]\n for i in range(sp_size)\n if good_tokens_per_rank[i] > 0\n )\n loss = total_loss / max(sum(good_tokens_per_rank), 1)\n\naccelerator.backward(loss)\n```\n\n> The weighted loss aggregation ensures correct gradients when tokens are unevenly distributed across ranks (e.g., when some ranks contain only padding or masked out prompt tokens).\n\n> Both Ulysses and Ring Attention use `position_ids` instead of `attention_mask` for causal masking during training. A 4D attention mask at these sequence lengths would be just as prohibitive as the attention scores themselves—at 128k tokens, that's another ~1TB tensor. Position IDs achieve the same causal behavior with O(n) O(n) memory instead of O(n2) O(n^2). During evaluation/inference, DeepSpeed's SP attention layer can bypass the SP operations entirely (via `disable_in_eval`) and fall back to the model's default attention implementation.\n\n## Integration with Transformers Trainer\n\nThe Transformers [`Trainer`](https://huggingface.co/docs/transformers/main_classes/trainer) provides seamless Ulysses integration through [`TrainingArguments.parallelism_config`](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.TrainingArguments.parallelism_config). It handles all the SP-specific details automatically—dataloader wrapping, sequence sharding, and loss aggregation—so you don't need to write any of the custom loss code shown above.\n\n### Configuration\n\nJust pass the same `parallelism_config` from above into `TrainingArguments`:\n\n```python\nfrom transformers import TrainingArguments\n\ntraining_args = TrainingArguments(\n parallelism_config=parallelism_config, # same ParallelismConfig as above\n per_device_train_batch_size=1,\n)\n```\n\n### What the Trainer Handles Automatically\n\n1. **Dataloader Wrapping**: After model preparation, the Trainer wraps the dataloader with `UlyssesSPDataLoaderAdapter`\n\n2. **Loss Computation**: The `compute_loss` method detects SP mode and routes to specialized `_deepspeed_sp_compute_loss` which handles:\n\n - Gathering losses across SP ranks\n - Computing valid token counts per rank\n - Weighted loss aggregation\n3. **Batch Size Calculation**: The effective data parallel world size accounts for SP:\n\n ```python\n dp_world_size = world_size // sp_size\n ```\n\n4. **Dataloader Length Adjustment**: Training step calculations are adjusted for SP's effect on iteration count\n\n### Launch Command\n\nUse an accelerate config file or command-line arguments:\n\n```bash\naccelerate launch \\\n --config_file deepspeed_ulysses.yaml \\\n train.py \\\n --per_device_train_batch_size 1\n```\n\n## Integration with TRL SFTTrainer\n\nTRL's [`SFTTrainer`](https://huggingface.co/docs/trl/sft_trainer) builds on the Transformers Trainer and adds specific optimizations for supervised fine-tuning with long sequences.\n\n### Configuration\n\n```python\nfrom trl import SFTConfig, SFTTrainer\nfrom accelerate.utils import ParallelismConfig, DeepSpeedSequenceParallelConfig\n\nparallelism_config = ParallelismConfig(\n sp_backend=\"deepspeed\",\n sp_size=2,\n dp_shard_size=2, # 2D parallelism: SP × DP = 4 GPUs\n sp_handler=DeepSpeedSequenceParallelConfig(\n sp_seq_length_is_variable=True,\n sp_attn_implementation=\"flash_attention_2\",\n ),\n)\n\ntraining_args = SFTConfig(\n ...,\n parallelism_config=parallelism_config,\n max_length=32768,\n pad_to_multiple_of=2, # Must equal sp_size\n per_device_train_batch_size=1,\n)\n\ntrainer = SFTTrainer(\n model=model,\n args=training_args,\n train_dataset=dataset,\n)\ntrainer.train()\n```\n\n### Key SFTConfig Parameters for Ulysses\n\n| Parameter | Description |\n| -------------------- | -------------------------------------------------------------------------------------------- |\n| `pad_to_multiple_of` | Must equal `sp_size` to ensure sequence divisibility |\n| `max_length` | Global sequence length (before splitting across GPUs) |\n| `packing` | Works well with SP — packing reduces padding waste, especially for variable-length sequences |\n\n### Accelerate Config File\n\nCreate `alst_ulysses_4gpu.yaml`:\n\n```yaml\ncompute_environment: LOCAL_MACHINE\ndistributed_type: DEEPSPEED\nmixed_precision: bf16\nnum_processes: 4\ndeepspeed_config:\n zero_stage: 3\n seq_parallel_communication_data_type: bf16\nparallelism_config:\n parallelism_config_sp_size: 2\n parallelism_config_sp_backend: deepspeed\n parallelism_config_dp_shard_size: 2\n parallelism_config_sp_seq_length_is_variable: true\n parallelism_config_sp_attn_implementation: flash_attention_2\n```\n\n### Complete Training Command\n\n```bash\naccelerate launch --config_file alst_ulysses_4gpu.yaml \\\n trl/scripts/sft.py \\\n --model_name_or_path meta-llama/Llama-3.1-8B \\\n --dataset_name trl-lib/Capybara \\\n --max_length 32768 \\\n --packing \\\n --pad_to_multiple_of 2 \\\n --per_device_train_batch_size 1\n```\n\n### Shift Labels Handling\n\nThe SFTTrainer automatically handles pre-shifted labels when Ulysses is enabled:\n\n```python\n# When using SP, labels are pre-shifted by the dataloader adapter\n# The trainer detects this and uses shift_labels directly\nlabels = inputs[\"labels\"] if \"shift_labels\" not in inputs else None\n\n# Loss computation uses the pre-shifted labels\nif \"shift_labels\" in inputs:\n shift_logits = outputs.logits.contiguous()\n shift_labels = inputs[\"shift_labels\"]\nelse:\n shift_logits = outputs.logits[..., :-1, :].contiguous()\n shift_labels = labels[..., 1:].contiguous()\n```\n\n## Comparing Ulysses and Ring Attention\n\nBoth Ulysses and Ring Attention enable long-context training, but they have different characteristics:\n\n| Aspect | Ulysses (DeepSpeed) | Ring Attention (FSDP2) |\n| ------------------------- | --------------------------------- | ---------------------- |\n| **Parallelism Method** | Attention head partitioning | Ring-based KV exchange |\n| **Backend** | DeepSpeed ZeRO | PyTorch FSDP2 |\n| **Attention Support** | FlashAttention 2/3, SDPA | SDPA only |\n| **Communication** | Two `all-to-all`s per layer | P2P ring communication |\n| **Comm volume per GPU** | O(total\\_seq x hidden / sp\\_size) | O(total\\_seq x hidden) |\n| **Sequence Divisibility** | `sp_size` | `cp_size * 2` |\n| **Num Head Constraint** | `num_heads >= sp_size` | None |\n\n### When to Choose Ulysses vs Ring Attention\n\nSince switching between the two only requires changing the accelerate config, we recommend trying both and comparing performance and memory usage on your specific setup. The main constraint is that Ulysses requires `num_heads >= sp_size`, while Ring Attention has no such limitation.\n\n## Best Practices\n\n### 1. Sequence Length Divisibility\n\nAlways ensure your sequence length is divisible by `sp_size`:\n\n```python\ntraining_args = SFTConfig(\n pad_to_multiple_of=4, # For sp_size=4\n max_length=32768, # Must be divisible by 4\n)\n```\n\n### 2. Use Flash Attention\n\nFlash Attention 2 provides cleaner output and better performance than SDPA:\n\n```python\nparallelism_config = ParallelismConfig(\n sp_handler=DeepSpeedSequenceParallelConfig(\n sp_attn_implementation=\"flash_attention_2\",\n ),\n)\n```\n\nUse Flash Attention 3 for Hopper and look out for Flash Attention 4 release for Blackwell (FA2 on Blackwell is quite slow).\n\n### 3. Combine with DeepSpeed ZeRO\n\nFor very large models, combine Ulysses with ZeRO Stage 3:\n\n```yaml\ndeepspeed_config:\n zero_stage: 3\n offload_optimizer:\n device: cpu\n```\n\nIf the model is huge, you can offload the params as well by adding to the above:\n\n```yaml\n offload_param:\n device: cpu\n```\n\n### 5. Use memory fragmentation-friendly PyTorch allocator\n\nThis environment variable will allow for a longer sequence length:\n\n```bash\nexport PYTORCH_ALLOC_CONF=expandable_segments:True\n```\n\n### 6. 2D Parallelism Configuration\n\nBalance SP and DP for your GPU count:\n\n| GPUs | `sp_size` | `dp_shard_size` | Use Case |\n| ---- | --------- | --------------- | ----------------------------------------------- |\n| 4 | 2 | 2 | Balanced throughput and sequence length |\n| 4 | 4 | 1 | Maximum sequence length |\n| 8 | 2 | 4 | Higher throughput with moderate sequence length |\n| 8 | 4 | 2 | Longer sequences with moderate throughput |\n\nRemember: `dp_replicate_size × dp_shard_size × sp_size = num_processes`\n\n### 7. Liger-Kernel\n\nIf your desired model architecture is supported by [Liger-Kernel](https://github.com/linkedin/Liger-Kernel), it is fully compatible with Ulysses SP and can be enabled with a single flag:\n\n```python\ntraining_args = SFTConfig(\n use_liger_kernel=True,\n)\n```\n\nThe main memory saving comes from `FusedLinearCrossEntropy` which avoids materializing the full logits tensor during loss calculation. The savings grow with longer sequences where the logits tensor is larger.\n\nAdditionally, you can enable [`TiledMLP`](https://www.deepspeed.ai/tutorials/ulysses-alst-sequence-parallelism/#tiled-mlp-computation) to further extend sequence length — like `FusedLinearCrossEntropy`, it saves working memory by tiling large matrix operations.\n\n### 8. Token Distribution Across Ranks\n\nYou don't need to worry about manually balancing tokens across SP ranks—the loss aggregation code handles uneven distributions gracefully (including ranks with zero valid tokens). With random batching over a reasonably sized dataset, the distribution evens out statistically over training.\n\n## Benchmarks\n\nTo quantify the benefits of Ulysses SP, we trained [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) on the [Gutenberg English](https://huggingface.co/datasets/sedthh/gutenberg_english) streaming dataset using TRL's SFTTrainer. All experiments ran on H100 80GB GPUs with DeepSpeed ZeRO-3, CPU optimizer offloading, gradient checkpointing, and flash-attn2 as the attention backend.\n\n### Setup\n\n| Config | GPUs | SP | DP | Seq Length | Grad Acc | Global Batch |\n| -------- | ---- | -- | -- | ---------- | -------- | ------------ |\n| Baseline | 1 | 1 | 1 | 8K | 8 | 8 |\n| SP=4 | 4 | 4 | 1 | 8K | 8 | 8 |\n| SP=4 | 4 | 4 | 1 | 32K | 8 | 8 |\n| SP=4 | 4 | 4 | 1 | 64K | 8 | 8 |\n| SP=4 | 4 | 4 | 1 | 96K | 8 | 8 |\n\nThe benchmark runs in the table above use the same global batch size (8 micro-batches), cosine learning-rate schedule, and seed, so those benchmark loss curves are directly comparable.\n\n### Loss Curve Matching Diagnostics (4 GPU)\n\nTo verify SP-vs-DP loss equivalence, we ran controlled 4-GPU A/B experiments with identical seed, model, optimizer, learning-rate schedule, and data order.\n\n#### Methodology for Fair DP vs SP Comparison\n\nCompared setups:\n\n- `DP=4, SP=1, GAS=1` (baseline)\n- `DP=1, SP=4, GAS=4` (Ulysses SP)\n\nFor fair comparison, `GAS` must scale with `SP`:\n\n- Ulysses SP splits the sequence across `SP` ranks, so each SP rank sees roughly `1/SP` of the sequence tokens per micro-step.\n- If `GAS` is unchanged, each optimizer step in SP aggregates fewer total tokens than the DP baseline.\n- Setting `GAS=SP` keeps effective tokens per optimizer step matched:\n - DP tokens/step: `dp_world_size * micro_batch * seq_len * GAS = 4 * B * L * 1`\n - SP tokens/step: `dp_world_size * micro_batch * (L/SP) * GAS * SP_ranks = 1 * B * (L/4) * 4 * 4 = 4 * B * L`\n\n![Canonical loss on Gutenberg for DP=4 vs SP=4](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ulysses/gutenberg_canonical_loss_dp4_vs_sp4.png)\\\nOn Gutenberg text (20 steps), canonical loss matches within logging precision between `DP=4,SP=1,GAS=1` and `DP=1,SP=4,GAS=4`.\n\nMeasured over 20 steps on 4 GPUs in controlled equivalence harnesses:\n\n| Harness | Metric | DP vs SP setting | Mean abs diff | Max abs diff |\n| ------------ | ------------- | ------------------------ | ------------- | ------------ |\n| `Trainer` | `loss` | DP=4, SP=1 vs DP=1, SP=4 | 0.0054 | 0.0131 |\n| `SFTTrainer` | logged `loss` | DP=4, SP=1 vs DP=1, SP=4 | 0.0811 | 0.0812 |\n| `SFTTrainer` | canonical NLL | DP=4, SP=1 vs DP=1, SP=4 | 0.000004 | 0.000005 |\n\nTakeaway: under matched token budget, SP and non-SP match on canonical token-normalized loss. The remaining difference is in trainer-reported logging (`loss`), not in the underlying cross-entropy objective.\n\n### Memory Reduction\n\n![Peak GPU Memory per Rank](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ulysses/memory_usage.png)\\\nSP=4 reduces per-GPU memory by 3.3x at the same sequence length, enabling training at up to 96K tokens on 4× H100 80GB. At 128K, the model OOMs.\n\n| Config | Seq Length | Peak Memory | Notes |\n| ------------ | ---------- | ----------- | --------------------------------- |\n| DP=4 (4 GPU) | 8K | 22.4 GB | Baseline — no SP |\n| SP=4 (4 GPU) | 8K | 22.8 GB | Similar memory at same seq length |\n| SP=4 (4 GPU) | 32K | 35.0 GB | 4x longer than DP baseline |\n| SP=4 (4 GPU) | 64K | 50.5 GB | 8x longer than DP baseline |\n| SP=4 (4 GPU) | 96K | 66.0 GB | **12x longer than DP baseline** |\n| SP=4 (4 GPU) | 128K | OOM | Exceeds 80 GB limit |\n\nAt 8K tokens, DP=4 and SP=4 use nearly the same memory per GPU (~22 GB with ZeRO-3). The advantage of SP is that it enables scaling to much longer sequences: at 96K tokens (12x longer), peak memory is 66 GB — still within the H100's 80 GB capacity. At 128K, the model OOMs, establishing the practical limit for this configuration. DP=4 without SP cannot scale beyond 8K for this model.\n\n### Throughput\n\n![Training Throughput](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ulysses/throughput.png)\\\nLonger sequences with SP process dramatically more tokens per second. SP=4 at 64K achieves 3.7x the throughput of the baseline.\n\n| Config | Seq Length | Tokens/s | vs Baseline |\n| ---------------- | ---------- | -------- | ----------- |\n| Baseline (1 GPU) | 8K | 3,633 | — |\n| SP=4 (4 GPU) | 8K | 3,933 | \\~1x |\n| SP=4 (4 GPU) | 32K | 7,733 | **2.1x** |\n| SP=4 (4 GPU) | 64K | 13,396 | **3.7x** |\n\nAt the same sequence length (8K), SP=4 has comparable throughput to the single-GPU baseline — the all-to-all communication overhead is minimal on NVLink-connected GPUs. The real benefit comes from longer sequences: as sequence length grows, the quadratic attention computation dominates over communication and other overheads, making each training step increasingly compute-efficient. Each step also processes proportionally more tokens, so throughput scales with sequence length. At 64K, SP=4 processes 13,396 tokens/second — 3.7x the baseline.\n\n> These results use only 4 GPUs with SP=4. With 8 GPUs (SP=8), you can push to even longer sequences — up to 256K+ tokens — or use 2D parallelism (SP=4, DP=2) to combine long-context training with data-parallel throughput.\n\n## Requirements\n\n- HF Accelerate: `deepspeed>=0.18.1 accelerate>=1.12`\n- HF Trainer: `deepspeed>=0.18.1 accelerate>=1.12 transformers>=5.0`\n- HF TRL: `deepspeed>=0.18.1 accelerate>=1.12 transformers>=5.0 trl>=0.18.0`\n\nUse `flash_attention_2` for Ampere GPUs, or `flash_attention_3` for Hopper GPUs. Wait for `flash_attention_4` on Blackwell 🕰.\n\n## Resources\n\n### Documentation\n\n- [Accelerate: Context Parallelism Guide](https://huggingface.co/docs/accelerate/concept_guides/context_parallelism)\n- [TRL: Distributing Training](https://huggingface.co/docs/trl/distributing_training)\n- [DeepSpeed Sequence Parallelism](https://www.deepspeed.ai/tutorials/ulysses-alst-sequence-parallelism)\n\n### Examples\n\n- [Accelerate ALST Example](https://github.com/huggingface/accelerate/tree/main/examples/alst_ulysses_sequence_parallelism)\n- [TRL Accelerate Configs](https://github.com/huggingface/trl/tree/main/examples/accelerate_configs)\n\n### Papers\n\n- [Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences](https://huggingface.co/papers/2506.13996)\n- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://huggingface.co/papers/2309.14509)\n\n### Related Blog Posts\n\n- [Accelerate ND-Parallel: A Guide to Efficient Multi-GPU Training](https://huggingface.co/blog/accelerate-nd-parallel)\n- [Understanding Ulysses and Ring Attention](https://huggingface.co/blog/exploding-gradients/ulysses-ring-attention)\n- [Enabling Long-Context Training with Sequence Parallelism in Axolotl](https://huggingface.co/blog/axolotl-ai-co/long-context-with-sequence-parallelism-in-axolotl)\n", "dateCreated": "2026-10-04T12:50:30.804479567+00:00", "date_created": "2026-03-09T00:00:00+00:00", "date_modified": "2026-03-09T00:00:00+00:00", "date_published": "2026-03-09T00:00:00+00:00", "description": "Training large language models on long sequences has become essential for building capable AI systems. As models are increasingly used for tasks like document analysis, code understanding, complex reasoning, and RAG workloads, the need to…", "external_url": "https://huggingface.co/blog/ulysses-sp", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/03/2026-03-09-ulysses-sequence-parallelism-training-with-million-token.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/03/2026-03-09-ulysses-sequence-parallelism-training-with-million-token.md" }, "id": "urn:aggr:item:ae61e2bf024e6021f20842d262a06bc775d0bdb4", "isBasedOn": { "@id": "https://huggingface.co/blog/ulysses-sp" }, "mainEntity": { "@id": "https://huggingface.co/blog/ulysses-sp", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-03-09-ulysses-sequence-parallelism-training-with-million-token/#webpage" }, "dateModified": "2026-03-09T00:00:00+00:00", "datePublished": "2026-03-09T00:00:00+00:00", "headline": "Ulysses Sequence Parallelism: Training with Million-Token Contexts", "timeRequired": "PT13M", "url": "https://huggingface.co/blog/ulysses-sp", "wordCount": 2814 }, "name": "Archived snapshot: Ulysses Sequence Parallelism: Training with Million-Token Contexts", "preview": null, "reading_minutes": 13, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00", "title": "Ulysses Sequence Parallelism: Training with Million-Token Contexts", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-03-09-ulysses-sequence-parallelism-training-with-million-token/", "word_count": 2814 }