{
  "@context": "https://schema.org",
  "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-03-03-prx-part-3-training-a-text-to-image-model-in-24h/#webpage",
  "@type": [
    "WebPage",
    "ArchiveComponent"
  ],
  "articleBody": "![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/7MUExmr-ed7sgd0D9H6yh.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/wFevAF_ajrLWJxo8qll76.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/2JDkjJN0_FlFab5z6rz0V.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/LZjbLPmMKSTv95T2hHFqs.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/F3uWaHXreakVacTkf6UIN.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/X_XyKCFN1uaFhbz6fDC5f.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/vuxXqk1jaAHjDcsaYfzoW.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/8xP_XO-ovK2V7IPPUl4X9.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/GrVwtipSxQPc_7AdX_Cyf.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/bo3AdZb6PFYste4gNI_hS.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/xSKUZg-4oCkY2709PFHJF.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/B3QtNXI5RtqqKorQiPx75.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/luwmLyhZgs7h2v2Wx4YRZ.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/Q2zqJoyRe0-OtJxddzu42.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/rAMAI5mNE8PPhG0Lg5ln2.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/nkS6ejpXCCdqqxSnbHQBY.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/drJMVDA36Lu7hYWVsrHaR.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/Ze4BuybUnlDu6bcZoVt-Y.png)\n\n## Introduction\n\nWelcome back 👋\n\nIn the last two posts ([Part 1](https://huggingface.co/blog/Photoroom/prx-part1-architectures) and [Part 2](https://huggingface.co/blog/Photoroom/prx-part2)), we explored a wide range of architectural and training tricks for diffusion models. We tried to evaluate each idea in isolation, measuring throughput, convergence speed, and final image quality, and tried to understand what actually moves the needle.\n\nIn this post, we want to answer a much more practical question:\n\n**What happens when we combine all the tricks that worked?**\n\nInstead of optimizing one dimension at a time, we’ll stack the most promising ingredients together and see how far we can push performance under a strict compute budget.\n\nTo make things concrete, we’re doing a **24-hour speedrun**:\n\n- **32 H200**\n- \\~**$1500 total compute budget (2$/hour/GPU)**\n\nThis is very far from the early diffusion days, where training competitive models could cost millions of dollars. The goal here is to demonstrate how much the field has evolved and how far careful engineering can take you in just a single day of training.\n\nThis speedrun is not just a fun experiment. It will likely serve as the foundation for our large-scale training recipe going forward.\n\nAlongside the results, we’re also open-sourcing our code ([Github link](https://github.com/Photoroom/PRX)), which contains:\n\n- The training code used for this speedrun\n- The experimental framework from the previous blog post\n\nSo you can reproduce, modify, and extend everything yourself.\n\n* * *\n\n## The Training Recipe\n\nNow let’s walk through what went into this 24h run.\n\n### X-prediction and Training in the Pixel Space\n\nWe use the x-prediction formulation from *Back to Basics: Let Denoising Generative Models Denoise* \\[[Li and He, 2025](https://arxiv.org/abs/2511.13720)\\]. As seen in [Part 2](https://huggingface.co/blog/Photoroom/prx-part2), this enables **training directly in pixel space and eliminates the need for a VAE altogether**. We use a patch size of 32 and use a 256-dimensional bottleneck in the initial token projection layer. This design keeps the sequence length under control, making pixel-space training computationally manageable even at higher resolutions.\n\nAt 512px, the sequence length is:\n\n(512/32)2=256 (512 / 32)^2 = 256\n\nAt 1024px, the sequence length becomes:\n\n(1024/32)2=1024 (1024 / 32)^2 = 1024\n\nInstead of following the usual 256px → 512px → 1024px schedule, we start directly at 512px and then fine-tune at 1024px.\n\nWith controlled token counts and modern hardware, pixel-space training is no longer prohibitive. It is simply a cleaner and more direct formulation.\n\n* * *\n\n### Perceptual Losses\n\nOne very nice side effect of predicting x0x\\_0 directly in pixel space is that we can reuse a whole toolbox from classical computer vision.\n\nWhen your model outputs latents, perceptual supervision becomes awkward. You either have to decode back to pixels or define losses in a learned latent space that may or may not align with human perception. Once you predict pixels directly, everything becomes straightforward again. You can plug in perceptual losses exactly as they were originally designed.\n\nWe take inspiration from the paper *PixelGen: Pixel Diffusion Beats Latent Diffusion with Perceptual Loss* \\[[Ma et al.](https://arxiv.org/abs/2602.02493)\\], where the authors introduce additional perceptual objectives on top of the diffusion loss. They show that adding perceptual signals can noticeably improve convergence speed and final visual quality.\n\nFor this 24h run, we add two auxiliary losses:\n\n- **LPIPS** (\\[[Zhang et al.](https://arxiv.org/abs/1801.03924)\\])\n- A **DINO-based perceptual loss** (we use DINOv2 \\[[Oquab et al.](https://arxiv.org/abs/2304.07193)\\])\n\nThe idea is simple: In addition to the standard flow matching objective, we encourage the predicted clean image to match the target image in a perceptual feature space. LPIPS captures low-level perceptual similarity, while DINO features provide a stronger semantic signal.\n\nWe keep the same overall idea as the paper, but we tweaked a few details. In our experiments, we empirically found that it worked better to:\n\n- apply the perceptual losses on pooled full images instead of patch-wise features\n- apply them at all noise levels\n\nThese are small implementation details, but in our setting they consistently gave better results.\n\nWe used a weight of **0.1** for the LPIPS loss and **0.01** for the DINO perceptual loss, matching the values recommended in the original paper.\n\nThese losses are lightweight compared to the main transformer forward pass, and in our setup they add only a small overhead while providing a consistent quality boost.\n\n### Token Routing with TREAD\n\nTo make each step cheaper, we use token routing with *TREAD* \\[[Krause et al., 2025](https://openaccess.thecvf.com/content/ICCV2025/papers/Krause_TREAD_Token_Routing_for_Efficient_Architecture-agnostic_Diffusion_Training_ICCV_2025_paper.pdf)\\]), which randomly selects a fraction of tokens and lets them bypass a contiguous chunk of transformer blocks, then re-injects them later so nothing is dropped.\n\nWe picked TREAD over SPRINT ([Park et al., 2025](https://arxiv.org/pdf/2510.21986)) mostly for simplicity, and because the extra complexity of SPRINT did not feel worth the fairly small additional compute savings in our setting (sequence length 64 vs. 128 with TREAD at 512px).\n\nFollowing the TREAD recipe, we route **50% of the tokens** from the **2nd block** to the **penultimate block** of the transformer.\n\nRouted models can look worse under vanilla CFG, especially when undertrained, so we implemented a simple self-guidance scheme inspired by Guiding Token-Sparse Diffusion Models ([Krause et al., 2025](https://arxiv.org/abs/2601.01608)), which guides using a dense vs. routed conditional prediction instead of relying on an unconditional branch.\n\n### Representation Alignment with REPA and DINOv3\n\nWe used **REPA** \\[[Yu et al., 2024](https://arxiv.org/abs/2410.06940)\\] for representation alignment.\n\nFor the teacher, we went with **DINOv3** \\[[Siméoni et al. 2025](https://arxiv.org/abs/2508.10104)\\] since it gave the best quality improvements in our previous experiments.\n\nConcretely, we apply the alignment loss once,at the **8th transformer block** with a loss weight of **0.5**.\n\nSince we combine **REPA** with **TREAD** routing, we only compute the alignment loss on the **non-routed tokens**, meaning the tokens that actually go through the blocks where we apply the loss. This keeps the REPA signal consistent and avoids comparing features for tokens that skipped the computation path.\n\n* * *\n\n### Optimizer: Muon\n\nWe used the **Muon** optimizer, using the FSDP implementation from [muon\\_fsdp\\_2](https://github.com/samsja/muon_fsdp_2), since it showed a clear improvement over Adam in our previous runs.\n\nMuon is only applied to **2D parameters** (basically matrices). Everything else (biases, norms, embeddings, etc.) is optimized with Adam, which is why the config has two parameter groups.\n\n| Group    | What it applies to    | Key params we used                                        |\n| -------- | --------------------- | --------------------------------------------------------- |\n| **Muon** | 2D parameters         | `lr=1e-4`, `momentum=0.95`, `nesterov=true`, `ns_steps=5` |\n| **Adam** | all non-2D parameters | `lr=1e-4`, `betas=(0.9, 0.95)`, `eps=1e-8`                |\n\n## Training Settings\n\nWe trained on three publicly available synthetic datasets:\n\n- **Flux generated** (1.7M), [lehduong/flux\\_generated](https://huggingface.co/datasets/lehduong/flux_generated)\n- **FLUX-Reason-6M** (6M), [LucasFang/FLUX-Reason-6M](https://huggingface.co/datasets/LucasFang/FLUX-Reason-6M)\n- **midjourney-v6-llava** (1M), [brivangl/midjourney-v6-llava](https://huggingface.co/datasets/brivangl/midjourney-v6-llava) which we re-captioned with **Gemini 1.5** to make prompts more consistent and cut down caption noise.\n\nThe schedule is basically: go fast at 512, then sharpen at 1024:\n\n- **512px** for **100k steps** with **batch size 1024**\n- **1024px** for **20k steps** with **batch size 512** without **REPA**.\n\nWe also keep an **EMA** of the weights for sampling and eval:\n\n- `smoothing = 0.999`\n- `update_interval = 10ba`\n- `ema_start = 0ba`\n\n## Results and Closing Thoughts\n\nBelow are the evaluation curves we tracked throughout the run and a few sample grids from the final checkpoint:\n\n[![image](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/s2-rKg3fqtGefcBXmNFHJ.png)](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/s2-rKg3fqtGefcBXmNFHJ.png)\n\n![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/c5iZ9AQQ_wB7UONOiZaWs.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/f_KZ7Xdmx1aqLv1aWtHld.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/TNvYw9hHJ_87TiDYVdLuF.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/wsWk2O4X4SZdEzuqdkxOI.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/RcnVYOwKjSciNdC_Ok4zW.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/D1OUz7HPERstZe6r-x2Ji.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/3SKgSXkcBGnJATwG_wOH_.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/F1lIaKaZKnTGpyNO3rzAX.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/44VXfW9Sw1iBpGFDL7r3x.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/VNQHVh2m1k6IcuxnxkLn-.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/1KwDxBoDcCkFAj2gXMJzU.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/zRiDsNZFnV4zDQopDMNBv.png)\n\nFor a one day training run, this is already a pretty solid place to be. The model is not flawless yet (you can still spot some texture glitches, occasional weird anatomy, and it can get a bit shaky on very hard prompts), but it is clearly usable. Prompt following is strong, the overall aesthetic is consistent, and the 1024 stage mostly does what we want: sharpen details without breaking composition.\n\nThe key takeaway is that we're very close. The remaining issues look more like undertraining artifacts and limited data diversity than signs of a structural flaw in the recipe. The failure modes are consistent with what you’d expect from a model that simply hasn’t seen enough varied data yet. With more compute and broader coverage, this exact setup should continue improving in a fairly predictable way.\n\nZooming out, this speed run also highlights how far diffusion training has come. By combining pixel-space training, efficient routing, representation alignment, and lightweight perceptual guidance, you can now get a meaningful model in about a day on a budget that would have sounded unrealistic not that long ago.\n\n## What’s next?\n\nThis 24h run is just a starting point, not the finish line. Next, we will keep pushing the same recipe with a bit more scale and iterate on the dataset mix and captioning.\n\nAll the code and configs behind this speedrun, as well as the full experimental framework used throughout [Part 1](https://huggingface.co/blog/Photoroom/prx-part1-architectures) and [Part 2](https://huggingface.co/blog/Photoroom/prx-part2), are available in the PRX repository: [https://github.com/Photoroom/PRX](https://github.com/Photoroom/PRX).\n\nWhile we don’t redistribute the exact training datasets used in this run, the pipeline is fully configurable and designed to be easily adapted to your own data. You can plug in different datasets, tweak individual components (TREAD, REPA, perceptual losses, Muon, etc.), and run controlled experiments with minimal friction. Our goal is to make this a practical playground for fast diffusion research, and we hope the community will use it to explore, benchmark, and iterate on these techniques in their own setups.\n\nIf you made it this far, thank you for reading. We would also love to have you join our [Discord](https://discord.gg/ZRwdpBSyyC) community, where we share PRX progress and results, and discuss anything diffusion and text-to-image related.\n\nGoodbye for now, and stay tuned for the next round of experiments! 🚀\n\n## Acknowledgements.\n\nThis speedrun was inspired by several recent efforts exploring fast and low-cost training of diffusion models. If you're interested in speedrunning text-to-image models, we encourage you to check out the following works:\n\n- Haridas, A., Shen, T., Yu, J. *Nitro-T: Training a Text-to-Image Diffusion Model from Scratch in 1 Day.* [https://rocm.blogs.amd.com/artificial-intelligence/nitro-t-diffusion/README.html](https://rocm.blogs.amd.com/artificial-intelligence/nitro-t-diffusion/README.html)\n- Bhanded, S. *Speedrunning ImageNet Diffusion.* [https://arxiv.org/abs/2512.12386](https://arxiv.org/abs/2512.12386)\n- Sehwag, V., Kong, X., Li, J., Spranger, M., Lyu, L. *Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget.* [https://arxiv.org/abs/2407.15811](https://arxiv.org/abs/2407.15811)\n- Yeh, S.-Y. *Home-made Diffusion Model from Scratch to Hatch.* [https://arxiv.org/abs/2509.06068](https://arxiv.org/abs/2509.06068)\n",
  "authors": [],
  "captured_at": "2026-10-04T12:50:30.804479567+00:00",
  "category": "ai",
  "content_html": "<p><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/7MUExmr-ed7sgd0D9H6yh.png\" alt=\"\" loading=\"eager\" decoding=\"async\" fetchpriority=\"high\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/wFevAF_ajrLWJxo8qll76.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/2JDkjJN0_FlFab5z6rz0V.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/LZjbLPmMKSTv95T2hHFqs.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/F3uWaHXreakVacTkf6UIN.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/X_XyKCFN1uaFhbz6fDC5f.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/vuxXqk1jaAHjDcsaYfzoW.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/8xP_XO-ovK2V7IPPUl4X9.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/GrVwtipSxQPc_7AdX_Cyf.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/bo3AdZb6PFYste4gNI_hS.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/xSKUZg-4oCkY2709PFHJF.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/B3QtNXI5RtqqKorQiPx75.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/luwmLyhZgs7h2v2Wx4YRZ.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/Q2zqJoyRe0-OtJxddzu42.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/rAMAI5mNE8PPhG0Lg5ln2.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/nkS6ejpXCCdqqxSnbHQBY.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/drJMVDA36Lu7hYWVsrHaR.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/Ze4BuybUnlDu6bcZoVt-Y.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></p>\n<h2>Introduction</h2>\n<p>Welcome back 👋</p>\n<p>In the last two posts (<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/blog/Photoroom/prx-part1-architectures\">Part 1</a> and <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/blog/Photoroom/prx-part2\">Part 2</a>), we explored a wide range of architectural and training tricks for diffusion models. We tried to evaluate each idea in isolation, measuring throughput, convergence speed, and final image quality, and tried to understand what actually moves the needle.</p>\n<p>In this post, we want to answer a much more practical question:</p>\n<p><strong>What happens when we combine all the tricks that worked?</strong></p>\n<p>Instead of optimizing one dimension at a time, we’ll stack the most promising ingredients together and see how far we can push performance under a strict compute budget.</p>\n<p>To make things concrete, we’re doing a <strong>24-hour speedrun</strong>:</p>\n<ul>\n<li><strong>32 H200</strong></li>\n<li>~<strong>$1500 total compute budget (2$/hour/GPU)</strong></li>\n</ul>\n<p>This is very far from the early diffusion days, where training competitive models could cost millions of dollars. The goal here is to demonstrate how much the field has evolved and how far careful engineering can take you in just a single day of training.</p>\n<p>This speedrun is not just a fun experiment. It will likely serve as the foundation for our large-scale training recipe going forward.</p>\n<p>Alongside the results, we’re also open-sourcing our code (<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/Photoroom/PRX\">Github link</a>), which contains:</p>\n<ul>\n<li>The training code used for this speedrun</li>\n<li>The experimental framework from the previous blog post</li>\n</ul>\n<p>So you can reproduce, modify, and extend everything yourself.</p>\n<hr />\n<h2>The Training Recipe</h2>\n<p>Now let’s walk through what went into this 24h run.</p>\n<h3>X-prediction and Training in the Pixel Space</h3>\n<p>We use the x-prediction formulation from <em>Back to Basics: Let Denoising Generative Models Denoise</em> [<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/2511.13720\">Li and He, 2025</a>]. As seen in <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/blog/Photoroom/prx-part2\">Part 2</a>, this enables <strong>training directly in pixel space and eliminates the need for a VAE altogether</strong>. We use a patch size of 32 and use a 256-dimensional bottleneck in the initial token projection layer. This design keeps the sequence length under control, making pixel-space training computationally manageable even at higher resolutions.</p>\n<p>At 512px, the sequence length is:</p>\n<p>(512/32)2=256 (512 / 32)^2 = 256</p>\n<p>At 1024px, the sequence length becomes:</p>\n<p>(1024/32)2=1024 (1024 / 32)^2 = 1024</p>\n<p>Instead of following the usual 256px → 512px → 1024px schedule, we start directly at 512px and then fine-tune at 1024px.</p>\n<p>With controlled token counts and modern hardware, pixel-space training is no longer prohibitive. It is simply a cleaner and more direct formulation.</p>\n<hr />\n<h3>Perceptual Losses</h3>\n<p>One very nice side effect of predicting x0x_0 directly in pixel space is that we can reuse a whole toolbox from classical computer vision.</p>\n<p>When your model outputs latents, perceptual supervision becomes awkward. You either have to decode back to pixels or define losses in a learned latent space that may or may not align with human perception. Once you predict pixels directly, everything becomes straightforward again. You can plug in perceptual losses exactly as they were originally designed.</p>\n<p>We take inspiration from the paper <em>PixelGen: Pixel Diffusion Beats Latent Diffusion with Perceptual Loss</em> [<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/2602.02493\">Ma et al.</a>], where the authors introduce additional perceptual objectives on top of the diffusion loss. They show that adding perceptual signals can noticeably improve convergence speed and final visual quality.</p>\n<p>For this 24h run, we add two auxiliary losses:</p>\n<ul>\n<li><strong>LPIPS</strong> ([<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/1801.03924\">Zhang et al.</a>])</li>\n<li>A <strong>DINO-based perceptual loss</strong> (we use DINOv2 [<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/2304.07193\">Oquab et al.</a>])</li>\n</ul>\n<p>The idea is simple: In addition to the standard flow matching objective, we encourage the predicted clean image to match the target image in a perceptual feature space. LPIPS captures low-level perceptual similarity, while DINO features provide a stronger semantic signal.</p>\n<p>We keep the same overall idea as the paper, but we tweaked a few details. In our experiments, we empirically found that it worked better to:</p>\n<ul>\n<li>apply the perceptual losses on pooled full images instead of patch-wise features</li>\n<li>apply them at all noise levels</li>\n</ul>\n<p>These are small implementation details, but in our setting they consistently gave better results.</p>\n<p>We used a weight of <strong>0.1</strong> for the LPIPS loss and <strong>0.01</strong> for the DINO perceptual loss, matching the values recommended in the original paper.</p>\n<p>These losses are lightweight compared to the main transformer forward pass, and in our setup they add only a small overhead while providing a consistent quality boost.</p>\n<h3>Token Routing with TREAD</h3>\n<p>To make each step cheaper, we use token routing with <em>TREAD</em> [<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://openaccess.thecvf.com/content/ICCV2025/papers/Krause_TREAD_Token_Routing_for_Efficient_Architecture-agnostic_Diffusion_Training_ICCV_2025_paper.pdf\">Krause et al., 2025</a>]), which randomly selects a fraction of tokens and lets them bypass a contiguous chunk of transformer blocks, then re-injects them later so nothing is dropped.</p>\n<p>We picked TREAD over SPRINT (<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/pdf/2510.21986\">Park et al., 2025</a>) mostly for simplicity, and because the extra complexity of SPRINT did not feel worth the fairly small additional compute savings in our setting (sequence length 64 vs. 128 with TREAD at 512px).</p>\n<p>Following the TREAD recipe, we route <strong>50% of the tokens</strong> from the <strong>2nd block</strong> to the <strong>penultimate block</strong> of the transformer.</p>\n<p>Routed models can look worse under vanilla CFG, especially when undertrained, so we implemented a simple self-guidance scheme inspired by Guiding Token-Sparse Diffusion Models (<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/2601.01608\">Krause et al., 2025</a>), which guides using a dense vs. routed conditional prediction instead of relying on an unconditional branch.</p>\n<h3>Representation Alignment with REPA and DINOv3</h3>\n<p>We used <strong>REPA</strong> [<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/2410.06940\">Yu et al., 2024</a>] for representation alignment.</p>\n<p>For the teacher, we went with <strong>DINOv3</strong> [<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/2508.10104\">Siméoni et al. 2025</a>] since it gave the best quality improvements in our previous experiments.</p>\n<p>Concretely, we apply the alignment loss once,at the <strong>8th transformer block</strong> with a loss weight of <strong>0.5</strong>.</p>\n<p>Since we combine <strong>REPA</strong> with <strong>TREAD</strong> routing, we only compute the alignment loss on the <strong>non-routed tokens</strong>, meaning the tokens that actually go through the blocks where we apply the loss. This keeps the REPA signal consistent and avoids comparing features for tokens that skipped the computation path.</p>\n<hr />\n<h3>Optimizer: Muon</h3>\n<p>We used the <strong>Muon</strong> optimizer, using the FSDP implementation from <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/samsja/muon_fsdp_2\">muon_fsdp_2</a>, since it showed a clear improvement over Adam in our previous runs.</p>\n<p>Muon is only applied to <strong>2D parameters</strong> (basically matrices). Everything else (biases, norms, embeddings, etc.) is optimized with Adam, which is why the config has two parameter groups.</p>\n<table>\n<thead>\n<tr>\n<th>Group</th>\n<th>What it applies to</th>\n<th>Key params we used</th>\n</tr>\n</thead>\n<tbody>\n<tr>\n<td><strong>Muon</strong></td>\n<td>2D parameters</td>\n<td><code>lr=1e-4</code>, <code>momentum=0.95</code>, <code>nesterov=true</code>, <code>ns_steps=5</code></td>\n</tr>\n<tr>\n<td><strong>Adam</strong></td>\n<td>all non-2D parameters</td>\n<td><code>lr=1e-4</code>, <code>betas=(0.9, 0.95)</code>, <code>eps=1e-8</code></td>\n</tr>\n</tbody>\n</table>\n<h2>Training Settings</h2>\n<p>We trained on three publicly available synthetic datasets:</p>\n<ul>\n<li><strong>Flux generated</strong> (1.7M), <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/datasets/lehduong/flux_generated\">lehduong/flux_generated</a></li>\n<li><strong>FLUX-Reason-6M</strong> (6M), <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/datasets/LucasFang/FLUX-Reason-6M\">LucasFang/FLUX-Reason-6M</a></li>\n<li><strong>midjourney-v6-llava</strong> (1M), <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/datasets/brivangl/midjourney-v6-llava\">brivangl/midjourney-v6-llava</a> which we re-captioned with <strong>Gemini 1.5</strong> to make prompts more consistent and cut down caption noise.</li>\n</ul>\n<p>The schedule is basically: go fast at 512, then sharpen at 1024:</p>\n<ul>\n<li><strong>512px</strong> for <strong>100k steps</strong> with <strong>batch size 1024</strong></li>\n<li><strong>1024px</strong> for <strong>20k steps</strong> with <strong>batch size 512</strong> without <strong>REPA</strong>.</li>\n</ul>\n<p>We also keep an <strong>EMA</strong> of the weights for sampling and eval:</p>\n<ul>\n<li><code>smoothing = 0.999</code></li>\n<li><code>update_interval = 10ba</code></li>\n<li><code>ema_start = 0ba</code></li>\n</ul>\n<h2>Results and Closing Thoughts</h2>\n<p>Below are the evaluation curves we tracked throughout the run and a few sample grids from the final checkpoint:</p>\n<p><a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/s2-rKg3fqtGefcBXmNFHJ.png\"><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/s2-rKg3fqtGefcBXmNFHJ.png\" alt=\"image\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></a></p>\n<p><picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/c5iZ9AQQ_wB7UONOiZaWs.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/f_KZ7Xdmx1aqLv1aWtHld.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/TNvYw9hHJ_87TiDYVdLuF.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/wsWk2O4X4SZdEzuqdkxOI.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/RcnVYOwKjSciNdC_Ok4zW.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/D1OUz7HPERstZe6r-x2Ji.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/3SKgSXkcBGnJATwG_wOH_.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/F1lIaKaZKnTGpyNO3rzAX.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/44VXfW9Sw1iBpGFDL7r3x.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/VNQHVh2m1k6IcuxnxkLn-.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/1KwDxBoDcCkFAj2gXMJzU.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture> <picture class=\"article-picture article-picture-fallback\" style=\"--image-ratio:16 / 9\"><img src=\"https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/zRiDsNZFnV4zDQopDMNBv.png\" alt=\"\" loading=\"lazy\" decoding=\"async\" fetchpriority=\"low\" referrerpolicy=\"no-referrer\" class=\"progressive-image\"></picture></p>\n<p>For a one day training run, this is already a pretty solid place to be. The model is not flawless yet (you can still spot some texture glitches, occasional weird anatomy, and it can get a bit shaky on very hard prompts), but it is clearly usable. Prompt following is strong, the overall aesthetic is consistent, and the 1024 stage mostly does what we want: sharpen details without breaking composition.</p>\n<p>The key takeaway is that we're very close. The remaining issues look more like undertraining artifacts and limited data diversity than signs of a structural flaw in the recipe. The failure modes are consistent with what you’d expect from a model that simply hasn’t seen enough varied data yet. With more compute and broader coverage, this exact setup should continue improving in a fairly predictable way.</p>\n<p>Zooming out, this speed run also highlights how far diffusion training has come. By combining pixel-space training, efficient routing, representation alignment, and lightweight perceptual guidance, you can now get a meaningful model in about a day on a budget that would have sounded unrealistic not that long ago.</p>\n<h2>What’s next?</h2>\n<p>This 24h run is just a starting point, not the finish line. Next, we will keep pushing the same recipe with a bit more scale and iterate on the dataset mix and captioning.</p>\n<p>All the code and configs behind this speedrun, as well as the full experimental framework used throughout <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/blog/Photoroom/prx-part1-architectures\">Part 1</a> and <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/blog/Photoroom/prx-part2\">Part 2</a>, are available in the PRX repository: <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://github.com/Photoroom/PRX\">https://github.com/Photoroom/PRX</a>.</p>\n<p>While we don’t redistribute the exact training datasets used in this run, the pipeline is fully configurable and designed to be easily adapted to your own data. You can plug in different datasets, tweak individual components (TREAD, REPA, perceptual losses, Muon, etc.), and run controlled experiments with minimal friction. Our goal is to make this a practical playground for fast diffusion research, and we hope the community will use it to explore, benchmark, and iterate on these techniques in their own setups.</p>\n<p>If you made it this far, thank you for reading. We would also love to have you join our <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://discord.gg/ZRwdpBSyyC\">Discord</a> community, where we share PRX progress and results, and discuss anything diffusion and text-to-image related.</p>\n<p>Goodbye for now, and stay tuned for the next round of experiments! 🚀</p>\n<h2>Acknowledgements.</h2>\n<p>This speedrun was inspired by several recent efforts exploring fast and low-cost training of diffusion models. If you're interested in speedrunning text-to-image models, we encourage you to check out the following works:</p>\n<ul>\n<li>Haridas, A., Shen, T., Yu, J. <em>Nitro-T: Training a Text-to-Image Diffusion Model from Scratch in 1 Day.</em> <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://rocm.blogs.amd.com/artificial-intelligence/nitro-t-diffusion/README.html\">https://rocm.blogs.amd.com/artificial-intelligence/nitro-t-diffusion/README.html</a></li>\n<li>Bhanded, S. <em>Speedrunning ImageNet Diffusion.</em> <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/2512.12386\">https://arxiv.org/abs/2512.12386</a></li>\n<li>Sehwag, V., Kong, X., Li, J., Spranger, M., Lyu, L. <em>Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget.</em> <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/2407.15811\">https://arxiv.org/abs/2407.15811</a></li>\n<li>Yeh, S.-Y. <em>Home-made Diffusion Model from Scratch to Hatch.</em> <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://arxiv.org/abs/2509.06068\">https://arxiv.org/abs/2509.06068</a></li>\n</ul>\n",
  "content_markdown": "![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/7MUExmr-ed7sgd0D9H6yh.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/wFevAF_ajrLWJxo8qll76.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/2JDkjJN0_FlFab5z6rz0V.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/LZjbLPmMKSTv95T2hHFqs.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/F3uWaHXreakVacTkf6UIN.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/X_XyKCFN1uaFhbz6fDC5f.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/vuxXqk1jaAHjDcsaYfzoW.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/8xP_XO-ovK2V7IPPUl4X9.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/GrVwtipSxQPc_7AdX_Cyf.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/bo3AdZb6PFYste4gNI_hS.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/xSKUZg-4oCkY2709PFHJF.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/B3QtNXI5RtqqKorQiPx75.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/luwmLyhZgs7h2v2Wx4YRZ.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/Q2zqJoyRe0-OtJxddzu42.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/rAMAI5mNE8PPhG0Lg5ln2.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/nkS6ejpXCCdqqxSnbHQBY.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/drJMVDA36Lu7hYWVsrHaR.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/Ze4BuybUnlDu6bcZoVt-Y.png)\n\n## Introduction\n\nWelcome back 👋\n\nIn the last two posts ([Part 1](https://huggingface.co/blog/Photoroom/prx-part1-architectures) and [Part 2](https://huggingface.co/blog/Photoroom/prx-part2)), we explored a wide range of architectural and training tricks for diffusion models. We tried to evaluate each idea in isolation, measuring throughput, convergence speed, and final image quality, and tried to understand what actually moves the needle.\n\nIn this post, we want to answer a much more practical question:\n\n**What happens when we combine all the tricks that worked?**\n\nInstead of optimizing one dimension at a time, we’ll stack the most promising ingredients together and see how far we can push performance under a strict compute budget.\n\nTo make things concrete, we’re doing a **24-hour speedrun**:\n\n- **32 H200**\n- \\~**$1500 total compute budget (2$/hour/GPU)**\n\nThis is very far from the early diffusion days, where training competitive models could cost millions of dollars. The goal here is to demonstrate how much the field has evolved and how far careful engineering can take you in just a single day of training.\n\nThis speedrun is not just a fun experiment. It will likely serve as the foundation for our large-scale training recipe going forward.\n\nAlongside the results, we’re also open-sourcing our code ([Github link](https://github.com/Photoroom/PRX)), which contains:\n\n- The training code used for this speedrun\n- The experimental framework from the previous blog post\n\nSo you can reproduce, modify, and extend everything yourself.\n\n* * *\n\n## The Training Recipe\n\nNow let’s walk through what went into this 24h run.\n\n### X-prediction and Training in the Pixel Space\n\nWe use the x-prediction formulation from *Back to Basics: Let Denoising Generative Models Denoise* \\[[Li and He, 2025](https://arxiv.org/abs/2511.13720)\\]. As seen in [Part 2](https://huggingface.co/blog/Photoroom/prx-part2), this enables **training directly in pixel space and eliminates the need for a VAE altogether**. We use a patch size of 32 and use a 256-dimensional bottleneck in the initial token projection layer. This design keeps the sequence length under control, making pixel-space training computationally manageable even at higher resolutions.\n\nAt 512px, the sequence length is:\n\n(512/32)2=256 (512 / 32)^2 = 256\n\nAt 1024px, the sequence length becomes:\n\n(1024/32)2=1024 (1024 / 32)^2 = 1024\n\nInstead of following the usual 256px → 512px → 1024px schedule, we start directly at 512px and then fine-tune at 1024px.\n\nWith controlled token counts and modern hardware, pixel-space training is no longer prohibitive. It is simply a cleaner and more direct formulation.\n\n* * *\n\n### Perceptual Losses\n\nOne very nice side effect of predicting x0x\\_0 directly in pixel space is that we can reuse a whole toolbox from classical computer vision.\n\nWhen your model outputs latents, perceptual supervision becomes awkward. You either have to decode back to pixels or define losses in a learned latent space that may or may not align with human perception. Once you predict pixels directly, everything becomes straightforward again. You can plug in perceptual losses exactly as they were originally designed.\n\nWe take inspiration from the paper *PixelGen: Pixel Diffusion Beats Latent Diffusion with Perceptual Loss* \\[[Ma et al.](https://arxiv.org/abs/2602.02493)\\], where the authors introduce additional perceptual objectives on top of the diffusion loss. They show that adding perceptual signals can noticeably improve convergence speed and final visual quality.\n\nFor this 24h run, we add two auxiliary losses:\n\n- **LPIPS** (\\[[Zhang et al.](https://arxiv.org/abs/1801.03924)\\])\n- A **DINO-based perceptual loss** (we use DINOv2 \\[[Oquab et al.](https://arxiv.org/abs/2304.07193)\\])\n\nThe idea is simple: In addition to the standard flow matching objective, we encourage the predicted clean image to match the target image in a perceptual feature space. LPIPS captures low-level perceptual similarity, while DINO features provide a stronger semantic signal.\n\nWe keep the same overall idea as the paper, but we tweaked a few details. In our experiments, we empirically found that it worked better to:\n\n- apply the perceptual losses on pooled full images instead of patch-wise features\n- apply them at all noise levels\n\nThese are small implementation details, but in our setting they consistently gave better results.\n\nWe used a weight of **0.1** for the LPIPS loss and **0.01** for the DINO perceptual loss, matching the values recommended in the original paper.\n\nThese losses are lightweight compared to the main transformer forward pass, and in our setup they add only a small overhead while providing a consistent quality boost.\n\n### Token Routing with TREAD\n\nTo make each step cheaper, we use token routing with *TREAD* \\[[Krause et al., 2025](https://openaccess.thecvf.com/content/ICCV2025/papers/Krause_TREAD_Token_Routing_for_Efficient_Architecture-agnostic_Diffusion_Training_ICCV_2025_paper.pdf)\\]), which randomly selects a fraction of tokens and lets them bypass a contiguous chunk of transformer blocks, then re-injects them later so nothing is dropped.\n\nWe picked TREAD over SPRINT ([Park et al., 2025](https://arxiv.org/pdf/2510.21986)) mostly for simplicity, and because the extra complexity of SPRINT did not feel worth the fairly small additional compute savings in our setting (sequence length 64 vs. 128 with TREAD at 512px).\n\nFollowing the TREAD recipe, we route **50% of the tokens** from the **2nd block** to the **penultimate block** of the transformer.\n\nRouted models can look worse under vanilla CFG, especially when undertrained, so we implemented a simple self-guidance scheme inspired by Guiding Token-Sparse Diffusion Models ([Krause et al., 2025](https://arxiv.org/abs/2601.01608)), which guides using a dense vs. routed conditional prediction instead of relying on an unconditional branch.\n\n### Representation Alignment with REPA and DINOv3\n\nWe used **REPA** \\[[Yu et al., 2024](https://arxiv.org/abs/2410.06940)\\] for representation alignment.\n\nFor the teacher, we went with **DINOv3** \\[[Siméoni et al. 2025](https://arxiv.org/abs/2508.10104)\\] since it gave the best quality improvements in our previous experiments.\n\nConcretely, we apply the alignment loss once,at the **8th transformer block** with a loss weight of **0.5**.\n\nSince we combine **REPA** with **TREAD** routing, we only compute the alignment loss on the **non-routed tokens**, meaning the tokens that actually go through the blocks where we apply the loss. This keeps the REPA signal consistent and avoids comparing features for tokens that skipped the computation path.\n\n* * *\n\n### Optimizer: Muon\n\nWe used the **Muon** optimizer, using the FSDP implementation from [muon\\_fsdp\\_2](https://github.com/samsja/muon_fsdp_2), since it showed a clear improvement over Adam in our previous runs.\n\nMuon is only applied to **2D parameters** (basically matrices). Everything else (biases, norms, embeddings, etc.) is optimized with Adam, which is why the config has two parameter groups.\n\n| Group    | What it applies to    | Key params we used                                        |\n| -------- | --------------------- | --------------------------------------------------------- |\n| **Muon** | 2D parameters         | `lr=1e-4`, `momentum=0.95`, `nesterov=true`, `ns_steps=5` |\n| **Adam** | all non-2D parameters | `lr=1e-4`, `betas=(0.9, 0.95)`, `eps=1e-8`                |\n\n## Training Settings\n\nWe trained on three publicly available synthetic datasets:\n\n- **Flux generated** (1.7M), [lehduong/flux\\_generated](https://huggingface.co/datasets/lehduong/flux_generated)\n- **FLUX-Reason-6M** (6M), [LucasFang/FLUX-Reason-6M](https://huggingface.co/datasets/LucasFang/FLUX-Reason-6M)\n- **midjourney-v6-llava** (1M), [brivangl/midjourney-v6-llava](https://huggingface.co/datasets/brivangl/midjourney-v6-llava) which we re-captioned with **Gemini 1.5** to make prompts more consistent and cut down caption noise.\n\nThe schedule is basically: go fast at 512, then sharpen at 1024:\n\n- **512px** for **100k steps** with **batch size 1024**\n- **1024px** for **20k steps** with **batch size 512** without **REPA**.\n\nWe also keep an **EMA** of the weights for sampling and eval:\n\n- `smoothing = 0.999`\n- `update_interval = 10ba`\n- `ema_start = 0ba`\n\n## Results and Closing Thoughts\n\nBelow are the evaluation curves we tracked throughout the run and a few sample grids from the final checkpoint:\n\n[![image](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/s2-rKg3fqtGefcBXmNFHJ.png)](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/s2-rKg3fqtGefcBXmNFHJ.png)\n\n![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/c5iZ9AQQ_wB7UONOiZaWs.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/f_KZ7Xdmx1aqLv1aWtHld.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/TNvYw9hHJ_87TiDYVdLuF.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/wsWk2O4X4SZdEzuqdkxOI.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/RcnVYOwKjSciNdC_Ok4zW.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/D1OUz7HPERstZe6r-x2Ji.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/3SKgSXkcBGnJATwG_wOH_.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/F1lIaKaZKnTGpyNO3rzAX.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/44VXfW9Sw1iBpGFDL7r3x.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/VNQHVh2m1k6IcuxnxkLn-.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/1KwDxBoDcCkFAj2gXMJzU.png) ![](https://cdn-uploads.huggingface.co/production/uploads/68e529af513e724edd8702f6/zRiDsNZFnV4zDQopDMNBv.png)\n\nFor a one day training run, this is already a pretty solid place to be. The model is not flawless yet (you can still spot some texture glitches, occasional weird anatomy, and it can get a bit shaky on very hard prompts), but it is clearly usable. Prompt following is strong, the overall aesthetic is consistent, and the 1024 stage mostly does what we want: sharpen details without breaking composition.\n\nThe key takeaway is that we're very close. The remaining issues look more like undertraining artifacts and limited data diversity than signs of a structural flaw in the recipe. The failure modes are consistent with what you’d expect from a model that simply hasn’t seen enough varied data yet. With more compute and broader coverage, this exact setup should continue improving in a fairly predictable way.\n\nZooming out, this speed run also highlights how far diffusion training has come. By combining pixel-space training, efficient routing, representation alignment, and lightweight perceptual guidance, you can now get a meaningful model in about a day on a budget that would have sounded unrealistic not that long ago.\n\n## What’s next?\n\nThis 24h run is just a starting point, not the finish line. Next, we will keep pushing the same recipe with a bit more scale and iterate on the dataset mix and captioning.\n\nAll the code and configs behind this speedrun, as well as the full experimental framework used throughout [Part 1](https://huggingface.co/blog/Photoroom/prx-part1-architectures) and [Part 2](https://huggingface.co/blog/Photoroom/prx-part2), are available in the PRX repository: [https://github.com/Photoroom/PRX](https://github.com/Photoroom/PRX).\n\nWhile we don’t redistribute the exact training datasets used in this run, the pipeline is fully configurable and designed to be easily adapted to your own data. You can plug in different datasets, tweak individual components (TREAD, REPA, perceptual losses, Muon, etc.), and run controlled experiments with minimal friction. Our goal is to make this a practical playground for fast diffusion research, and we hope the community will use it to explore, benchmark, and iterate on these techniques in their own setups.\n\nIf you made it this far, thank you for reading. We would also love to have you join our [Discord](https://discord.gg/ZRwdpBSyyC) community, where we share PRX progress and results, and discuss anything diffusion and text-to-image related.\n\nGoodbye for now, and stay tuned for the next round of experiments! 🚀\n\n## Acknowledgements.\n\nThis speedrun was inspired by several recent efforts exploring fast and low-cost training of diffusion models. If you're interested in speedrunning text-to-image models, we encourage you to check out the following works:\n\n- Haridas, A., Shen, T., Yu, J. *Nitro-T: Training a Text-to-Image Diffusion Model from Scratch in 1 Day.* [https://rocm.blogs.amd.com/artificial-intelligence/nitro-t-diffusion/README.html](https://rocm.blogs.amd.com/artificial-intelligence/nitro-t-diffusion/README.html)\n- Bhanded, S. *Speedrunning ImageNet Diffusion.* [https://arxiv.org/abs/2512.12386](https://arxiv.org/abs/2512.12386)\n- Sehwag, V., Kong, X., Li, J., Spranger, M., Lyu, L. *Stretching Each Dollar: Diffusion Training from Scratch on a Micro-Budget.* [https://arxiv.org/abs/2407.15811](https://arxiv.org/abs/2407.15811)\n- Yeh, S.-Y. *Home-made Diffusion Model from Scratch to Hatch.* [https://arxiv.org/abs/2509.06068](https://arxiv.org/abs/2509.06068)\n",
  "dateCreated": "2026-10-04T12:50:30.804479567+00:00",
  "date_created": "2026-03-03T16:50:49+00:00",
  "date_modified": "2026-03-03T16:50:49+00:00",
  "date_published": "2026-03-03T16:50:49+00:00",
  "description": "Introduction Welcome back 👋 In the last two posts (Part 1 and Part 2), we explored a wide range of architectural and training tricks for diffusion models. We tried to evaluate each idea in isolation, measuring throughput, convergence…",
  "external_url": "https://huggingface.co/blog/Photoroom/prx-part3",
  "git": {
    "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e",
    "path": "items/huggingface-co/2026/03/2026-03-03-prx-part-3-training-a-text-to-image-model-in-24h.md",
    "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/03/2026-03-03-prx-part-3-training-a-text-to-image-model-in-24h.md"
  },
  "id": "urn:aggr:item:64dd77c9a44b127fd03c893c7843380251b14055",
  "isBasedOn": {
    "@id": "https://huggingface.co/blog/Photoroom/prx-part3"
  },
  "mainEntity": {
    "@id": "https://huggingface.co/blog/Photoroom/prx-part3",
    "@type": "CreativeWork",
    "archivedAt": {
      "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-03-03-prx-part-3-training-a-text-to-image-model-in-24h/#webpage"
    },
    "dateModified": "2026-03-03T16:50:49+00:00",
    "datePublished": "2026-03-03T16:50:49+00:00",
    "headline": "PRX Part 3 — Training a Text-to-Image Model in 24h!",
    "timeRequired": "PT8M",
    "url": "https://huggingface.co/blog/Photoroom/prx-part3",
    "wordCount": 1658
  },
  "name": "Archived snapshot: PRX Part 3 — Training a Text-to-Image Model in 24h!",
  "preview": null,
  "reading_minutes": 8,
  "replicated_at": null,
  "source": "huggingface-co",
  "source_name": "Hugging Face - Blog",
  "summary": null,
  "tags": [],
  "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00",
  "title": "PRX Part 3 — Training a Text-to-Image Model in 24h!",
  "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-03-03-prx-part-3-training-a-text-to-image-model-in-24h/",
  "word_count": 1658
}