{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-07-23-bringing-nunchaku-4-bit-diffusion-inference-to-diffusers/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Large diffusion transformers can create stunning images (or even videos, audio snippets, and now text), but loading a modern text-to-image model in BF16 precision often requires 20-30 GB of VRAM, which puts these models out of reach of most consumer GPUs. Quantization is a powerful solution to this problem, and Diffusers already integrates several quantization backends such as bitsandbytes, GGUF, torchao, and Quanto, which we covered in [Exploring Quantization Backends in Diffusers](https://huggingface.co/blog/diffusers-quantization).\n\nMost of these backends are *weight-only*. This means that they store the weights in low precision and dequantize them back to high precision at compute time. This reduces memory usage significantly, but it usually does not make inference faster, and can even add a small latency overhead.\n\n[SVDQuant](https://arxiv.org/abs/2411.05007), the quantization method behind the popular [Nunchaku](https://github.com/nunchaku-tech/nunchaku) inference engine, takes a different approach. It runs the main transformer layers with 4-bit weights and activations (W4A4), reducing memory while also speeding up the denoising loop. The details are covered below, but until now, using these checkpoints required a separate inference library.\n\nWith current Diffusers, loading a Nunchaku checkpoint is as simple as calling `from_pretrained()`, with no local CUDA compilation required thanks to the [`kernels`](https://github.com/huggingface/kernels) package. In addition, the companion [diffuse-compressor](https://github.com/rootonchair/diffuse-compressor) toolkit lets you quantize new architectures yourself and publish them as regular Diffusers repositories.\n\n![Nunchaku Lite image quality and performance comparison](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/nunchaku-diffusers/contact_sheet_top3_metrics_bold.png)\n\n## Table of Contents\n\n- [Getting started with Nunchaku Lite](https://huggingface.co/blog/nunchaku-diffusers#getting-started-with-nunchaku-lite)\n- [Background: SVDQuant and Nunchaku](https://huggingface.co/blog/nunchaku-diffusers#background-svdquant-and-nunchaku)\n- [Introducing Nunchaku Lite](https://huggingface.co/blog/nunchaku-diffusers#introducing-nunchaku-lite)\n- [Native loading in Diffusers](https://huggingface.co/blog/nunchaku-diffusers#native-loading-in-diffusers)\n- [Getting more speed and lower memory](https://huggingface.co/blog/nunchaku-diffusers#getting-more-speed-and-lower-memory)\n- [Benchmarks](https://huggingface.co/blog/nunchaku-diffusers#benchmarks)\n- [Quantizing your own model](https://huggingface.co/blog/nunchaku-diffusers#quantizing-your-own-model)\n- [Ready-to-use checkpoints](https://huggingface.co/blog/nunchaku-diffusers#ready-to-use-checkpoints)\n- [Conclusion](https://huggingface.co/blog/nunchaku-diffusers#conclusion)\n- [Acknowledgements](https://huggingface.co/blog/nunchaku-diffusers#acknowledgements)\n\n## Getting started with Nunchaku Lite\n\nFirst, install the requirements. You need a recent version of Diffusers and the Hugging Face `kernels` package:\n\n```bash\npip install -U diffusers transformers accelerate kernels bitsandbytes\n```\n\nThen load a pre-quantized pipeline like any other Diffusers model:\n\n```python\nimport torch\nfrom diffusers import ErnieImagePipeline\n\npipe = ErnieImagePipeline.from_pretrained(\n \"lite-infer/ERNIE-Image-Turbo-nunchaku-lite-nvfp4_r32-bnb4-text-encoder\",\n torch_dtype=torch.bfloat16,\n).to(\"cuda\")\n\nimage = pipe(\n prompt=\"A cinematic portrait of a red fox in a misty forest at sunrise, \"\n \"detailed fur, volumetric light\",\n height=1024,\n width=1024,\n num_inference_steps=8,\n guidance_scale=1.0,\n generator=torch.Generator(\"cuda\").manual_seed(42),\n).images[0]\nimage.save(\"output.png\")\n```\n\n![BF16 and Nunchaku Lite outputs for a red fox prompt](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/nunchaku-diffusers/fox_bf16_vs_nunchaku_no_metrics.png)\n\nNo custom pipeline class or separate inference engine is needed, and there is nothing to compile locally. The NVFP4 kernels are downloaded from the Hub through the [Nunchaku Lite kernels page](https://huggingface.co/kernels/rootonchair/nunchaku-lite-kernels) the first time they are used. This checkpoint pairs a Nunchaku NVFP4 transformer with a bitsandbytes NF4 text encoder, and generates a 1024x1024 image in about 1.7 seconds on an RTX 5090 with a peak memory usage of about 12 GB, compared with about 24 GB for the BF16 pipeline. You can find more details about the Nunchaku Lite checkpoint format in the [official Diffusers documentation](https://huggingface.co/docs/diffusers/main/en/quantization/nunchaku).\n\n> NVFP4 checkpoints require an NVIDIA Blackwell GPU (RTX 50 series, RTX PRO 6000, B200). For earlier generations, use the INT4 variants. See the [hardware support](https://huggingface.co/blog/nunchaku-diffusers#hardware-support) table below for details.\n\n## Background: SVDQuant and Nunchaku\n\n**SVDQuant** is the quantization method behind **Nunchaku**, its reference CUDA inference engine. Standard 4-bit quantization is difficult for diffusion transformers because both weights and activations contain large outliers. SVDQuant handles this by moving activation outliers into the weights, representing the hardest part of each weight matrix with a small 16-bit low-rank branch, and quantizing the remaining residual to 4 bits. Nunchaku makes this fast with fused kernels for the 4-bit path and the low-rank branch.\n\n![Nunchaku kernel fusion: the low-rank down projection is fused with input quantization, and the low-rank up projection is fused with the 4-bit matmul](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/nunchaku-diffusers/svdquant_kernel_fusion.png)\\\nNunchaku fuses the low-rank down projection with the quantization kernel and the low-rank up projection with the 4-bit compute kernel, eliminating the memory access overhead of the 16-bit branch. Figure from the [SVDQuant paper](https://arxiv.org/abs/2411.05007).\n\n## Introducing Nunchaku Lite\n\nThe original [Nunchaku engine](https://github.com/nunchaku-ai/nunchaku) gets much of its speed from [model-specific fused execution paths](https://huggingface.co/blog/nunchaku-diffusers#quantizing-models-with-structural-rewrites), such as fused QKV projections and fused GELU/MLP kernels. Those optimizations are tied to each architecture's module layout and checkpoint format, so supporting a new model family usually requires model-specific integration work.\n\n**Nunchaku Lite** is the new integration path in Diffusers. With it, Diffusers can load Nunchaku-style checkpoints without a custom pipeline or a separate inference engine. Under the hood, Nunchaku Lite patches the relevant `nn.Linear` modules of a stock Diffusers model with runtime SVDQ/AWQ linear layers before the checkpoint is loaded. The CUDA kernels come from the Hub through the `kernels` package. Two kernel families are used:\n\n- **`svdq_w4a4`**: 4-bit weights and activations with the SVDQuant low-rank correction. This layer is used for the transformer's attention and MLP projections, where nearly all of the compute is spent, and is available in INT4 and NVFP4 variants.\n- **`awq_w4a16`**: 4-bit weights with 16-bit activations, used for adaptive normalization and modulation projections such as FLUX `adanorm_single` / `adanorm_zero` or Qwen-Image modulation layers. These layers are memory-bound and precision-sensitive, making AWQ a good fit to preserve precision while still saving memory and space.\n\nThe trade-off is that, without architecture-specific fused kernels and modules, Nunchaku Lite cannot match the speedup of the original Nunchaku engine. However, the bare-bones implementation still delivers around **30% speedup** while retaining the same level of **VRAM reduction**.\n\n## Native loading in Diffusers\n\nIf you have used bitsandbytes or torchao in Diffusers, the mechanics will feel familiar. A Nunchaku Lite model repository is an ordinary Diffusers repository. The only special part is a `quantization_config` block inside the transformer's `config.json`:\n\n```json\n\"quantization_config\": {\n \"quant_method\": \"nunchaku_lite\",\n \"compute_dtype\": \"bfloat16\",\n \"svdq_w4a4\": {\n \"precision\": \"nvfp4\",\n \"group_size\": 16,\n \"rank\": 32,\n \"targets\": [\n \"layers.0.self_attention.to_q\",\n \"layers.0.self_attention.to_k\",\n \"...\"\n ]\n },\n \"awq_w4a16\": {\n \"precision\": \"int4\",\n \"group_size\": 64,\n \"targets\": [\n \"adaLN_modulation.1\",\n \"...\"\n ]\n }\n}\n```\n\nThis config tells Diffusers which modules were quantized, which scheme they use, and which Nunchaku Lite runtime layer to instantiate (`SVDQW4A4Linear` or `AWQW4A16Linear`).\n\nBecause the quantized model keeps the exact module structure of the dense one, everything downstream (schedulers, LoRA loading hooks, offloading, `torch.compile`) sees a normal Diffusers model.\n\n### Hardware support\n\nNunchaku Lite uses different kernel variants depending on the GPU generation and checkpoint precision:\n\n| Scheme | Precision | Supported GPUs |\n| ----------- | --------- | ------------------------------------------------------ |\n| `svdq_w4a4` | `nvfp4` | Blackwell (RTX 50 series, RTX PRO 6000, B200) |\n| `svdq_w4a4` | `int4` | Turing / Ampere / Ada (RTX 30 & 40 series, A100, L40S) |\n| `awq_w4a16` | `int4` | Turing / Ampere / Ada (RTX 30 & 40 series, A100, L40S) |\n\n> Volta and Hopper GPUs are currently not supported by the 4-bit kernels. The quantizer validates the GPU's CUDA capability at load time and raises a clear error instead of producing incorrect outputs.\n\n## Getting more speed and lower memory\n\nNunchaku Lite can be combined with other Diffusers memory and speed optimizations.\n\n**`torch.compile`.** Compiling the transformer improves the end-to-end speedup from 1.35x to 1.8x:\n\n```python\npipe.transformer.compile(fullgraph=True)\n\n# or compile_repeated_blocks() for faster compilation\n\npipe.transformer.compile_repeated_blocks(fullgraph=True)\n```\n\n**Quantized text encoders.** The transformer is not the only component with a large memory footprint. Text encoders such as T5 or Qwen3 can occupy several gigabytes on their own. Further quantizing the text encoder with bitsandbytes NF4 reduces peak VRAM by about 22% in our benchmark.\n\n**Offloading.** Diffusers offloading helpers such as `enable_model_cpu_offload()` and `enable_sequential_cpu_offload()` work as usual if you need to fit the pipeline onto a smaller GPU.\n\n## Benchmarks\n\nAll numbers below were measured on an NVIDIA RTX PRO 6000 (Blackwell) at 1024x1024 using [rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder](https://huggingface.co/rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder).\n\n### End-to-end latency and memory\n\n| Configuration | Full pipeline | Denoise loop | Peak VRAM | Speedup |\n| -------------------------------------- | ------------- | ------------ | --------- | ------- |\n| BF16 baseline | 3.00 s | 2.86 s | 31.1 GB | 1.0x |\n| Nunchaku Lite NVFP4 | 2.27 s | 2.13 s | 20.6 GB | 1.35x |\n| Nunchaku Lite NVFP4 + `torch.compile` | 1.68 s | 1.53 s | 20.6 GB | 1.8x |\n| Nunchaku Lite NVFP4 + NF4 text encoder | 2.29 s | 2.13 s | 16.0 GB | 1.35x |\n\nAs shown above, Nunchaku reduces peak VRAM by up to 50% while still improving latency by roughly 30%. The remaining overhead comes largely from extra kernel launches, which `torch.compile` can mitigate, bringing the full pipeline down to 1.68 s, or 1.8x faster than the BF16 baseline.\n\n### Image quality\n\n![Quality comparison grid](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/nunchaku-diffusers/quality_grid.png)\\\nBF16 vs 4-bit outputs with identical seeds and settings.\n\n## Quantizing your own model\n\nNunchaku Lite support in Diffusers is architecture-agnostic, and the [diffuse-compressor](https://github.com/rootonchair/diffuse-compressor) toolkit provides an end-to-end SVDQuant workflow for Diffusers models: calibrate, quantize, package, and publish.\n\nBelow, we walk through quantizing FLUX.2 Klein 4B as an example. It covers the main steps: inspect the model, calibrate and quantize the transformer, package the result as a Diffusers pipeline, then verify and push it to the Hub. The [full tutorial](https://github.com/rootonchair/diffuse-compressor/blob/main/docs/quantize_new_hf_model.md) covers every flag in detail.\n\n### 1. Inspect what will be quantized\n\nThe generic scanner walks the model and decides what to target: compatible linears inside the repeated transformer-block stack become SVDQ W4A4 targets, recognized modulation linears become AWQ W4A16 targets, and everything else stays dense.\n\n```bash\npython examples/text_to_image/quantize_hf.py black-forest-labs/FLUX.2-klein-4B \\\n --precision int4 --rank 32 --inspect-config\n```\n\nAlways read this report before quantizing. For FLUX.2 Klein 4B, the expected result is 100 SVDQ targets, 3 AWQ targets, and 6 dense outer linears, with no missing patterns or duplicate names.\n\n### 2. Run quantization\n\nThe following command runs SVDQuant on the transformer and writes the quantized checkpoint to `outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors`:\n\n```bash\npython examples/text_to_image/quantize_hf.py black-forest-labs/FLUX.2-klein-4B \\\n --precision int4 \\\n --output outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors\n```\n\nReplace `--precision int4` with `nvfp4` to build Blackwell-native weights.\n\n### 3. Package a Diffusers pipeline\n\nThe converter combines the quantized transformer with the base pipeline's other components, writes the compact `nunchaku_lite` configuration into `transformer/config.json`, and can optionally convert text encoders to NF4:\n\n```bash\npython examples/convert_nunchaku_lite_diffusers.py \\\n --checkpoint outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors \\\n --model-id black-forest-labs/FLUX.2-klein-4B \\\n --bnb4-text-encoder text_encoder \\\n --compute-dtype bfloat16 \\\n --output-dir outputs/diffusers/FLUX.2-klein-4B-nunchaku-lite-int4-bnb4-text-encoder\n```\n\n### 4. Load, verify, and push to the Hub\n\n```python\nimport torch\nfrom diffusers import DiffusionPipeline\n\npipe = DiffusionPipeline.from_pretrained(\n \"outputs/diffusers/FLUX.2-klein-4B-nunchaku-lite-int4-bnb4-text-encoder\",\n device_map=\"cuda\",\n)\nimage = pipe(\n \"A glass robot in a greenhouse, cinematic lighting\",\n num_inference_steps=4, guidance_scale=1.0,\n generator=torch.Generator(\"cuda\").manual_seed(12345),\n).images[0]\n```\n\nOnce the outputs look good, run `pipe.push_to_hub(\"your-name/your-model-nunchaku-lite-int4\")`. Other users can then load it with the same `from_pretrained()` pattern shown above.\n\n### Quantizing models with structural rewrites\n\nNote that the generic path assumes the architecture can be quantized without structural rewrites. For additional speedup, the original Nunchaku engine rewrites groups of Diffusers layers as fused modules. The generic path cannot infer these changes on its own, such as combining separate Q, K, and V projections into one module or splitting a fused projection across several modules.\n\nFLUX.1-dev's QKV projection is a concrete example. [Diffusers defines three separate modules](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py#L313-L329):\n\n```python\nself.to_q = torch.nn.Linear(query_dim, self.inner_dim, bias=bias)\nself.to_k = torch.nn.Linear(query_dim, self.inner_dim, bias=bias)\nself.to_v = torch.nn.Linear(query_dim, self.inner_dim, bias=bias)\n```\n\nThe [Nunchaku FLUX module combines those layers](https://github.com/nunchaku-ai/nunchaku/blob/main/nunchaku/models/transformers/transformer_flux_v2.py#L63-L79) into one quantized `to_qkv` module:\n\n```python\nto_qkv = fuse_linears([other.to_q, other.to_k, other.to_v])\nself.to_qkv = SVDQW4A4Linear.from_linear(to_qkv, **kwargs)\n```\n\nThis grouped module is required because Nunchaku's fused operator consumes the QKV projection, Q/K normalization, and rotary embeddings together. By comparison, the [default Diffusers path](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py#L45-L116) executes them separately:\n\n```python\nquery = attn.to_q(hidden_states)\nkey = attn.to_k(hidden_states)\nvalue = attn.to_v(hidden_states)\n\nquery = query.unflatten(-1, (attn.heads, -1))\nkey = key.unflatten(-1, (attn.heads, -1))\nvalue = value.unflatten(-1, (attn.heads, -1))\n\nquery = attn.norm_q(query)\nkey = attn.norm_k(key)\n\nif image_rotary_emb is not None:\n query = apply_rotary_emb(query, image_rotary_emb, sequence_dim=1)\n key = apply_rotary_emb(key, image_rotary_emb, sequence_dim=1)\n```\n\nThe [Nunchaku path](https://github.com/nunchaku-ai/nunchaku/blob/main/nunchaku/models/attention_processors/flux.py#L69-L93) supplies the grouped projection, normalization modules, and rotary embeddings to one fused operator:\n\n```python\nqkv = fused_qkv_norm_rottary(\n hidden_states, attn.to_qkv, attn.norm_q, attn.norm_k, image_rotary_emb\n)\n```\n\nThis is the structural rewrite that the generic path cannot infer. Diffusers has three destination modules with `to_q`, `to_k`, and `to_v` parameter prefixes, while Nunchaku has one grouped module under `to_qkv`. A model-specific target config or adapter must state that the Q, K, and V parameters should be concatenated along the output dimension, in that order, and loaded into `to_qkv`.\n\nStructural rewrites like these are described by a model-specific target config during quantization and handled by a small runtime adapter when the checkpoint is loaded. The [FLUX.2 Klein 4B quantization script](https://github.com/rootonchair/diffuse-compressor/blob/main/examples/text_to_image/quantize_flux2_klein_4b.py) provides a concrete target-config example for producing a structurally rewritten checkpoint, while [rootonchair/nunchaku-lite](https://github.com/rootonchair/nunchaku-lite) provides the runtime adapters needed to load grouped QKV tensors, split fused projections, and other fused operations. For the complete workflow, you can check the [Adding A New Model](https://github.com/rootonchair/diffuse-compressor/blob/main/docs/adding_new_model.md) guide.\n\n## Ready-to-use checkpoints\n\nTo get started right away, check out the following repositories:\n\n- [rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder](https://huggingface.co/rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder): INT4 ERNIE-Image-Turbo with a bitsandbytes NF4 text encoder\n- [rootonchair/ERNIE-Image-Turbo-nunchaku-lite-nvfp4-bnb4-text-encoder](https://huggingface.co/rootonchair/ERNIE-Image-Turbo-nunchaku-lite-nvfp4-bnb4-text-encoder): NVFP4 ERNIE-Image-Turbo with a bitsandbytes NF4 text encoder\n- [OzzyGT/Krea\\_2\\_Turbo\\_nunchaku\\_lite\\_nvfp4](https://huggingface.co/OzzyGT/Krea_2_Turbo_nunchaku_lite_nvfp4): NVFP4 Krea 2 Turbo checkpoint\n- [lite-infer](https://huggingface.co/lite-infer): more Nunchaku Lite checkpoints and collections\n\n## Conclusion\n\nNunchaku's SVDQuant kernels are one of the most effective ways to run diffusion transformers efficiently on consumer hardware, and they are now natively supported in Diffusers. Pre-quantized checkpoints load with `from_pretrained()`, and the diffuse-compressor toolkit makes it possible to quantize new architectures without waiting for engine support. By quantizing both weights and activations, the W4A4 path lowers memory use while improving denoising latency, keeping image quality close to the BF16 original.\n\nIf you quantize and publish a new model, we would love to hear about it. Share it on the Hub and let us know! If you have any questions about this feature, feel free to join our [Discord](https://discord.gg/G7tWnz98XR).\n\nTo learn more, check out the following resources:\n\n- [Diffusers Nunchaku documentation](https://huggingface.co/docs/diffusers/quantization/nunchaku)\n- [The integration PR (huggingface/diffusers#14100)](https://github.com/huggingface/diffusers/pull/14100)\n- [SVDQuant paper](https://arxiv.org/abs/2411.05007) and the [Nunchaku engine](https://github.com/nunchaku-tech/nunchaku)\n- [diffuse-compressor](https://github.com/rootonchair/diffuse-compressor)\n- Previous posts: [Exploring Quantization Backends in Diffusers](https://huggingface.co/blog/diffusers-quantization) and [Memory-efficient Diffusion Transformers with Quanto and Diffusers](https://huggingface.co/blog/quanto-diffusers)\n\n## Acknowledgements\n\nThanks to the Diffusers maintainers for reviews and guidance throughout the integration, and to the MIT HAN Lab / Nunchaku team for the original SVDQuant work. Thanks to Marc Sun for providing feedback on the blog post. Thanks to Álvaro Somoza for trying out `nunchaku-lite` and for providing feedback.\n\n`rootonchair` is also grateful to SilverAI for supporting this work and providing the environment in which much of this development took place.\n", "authors": [], "captured_at": "2026-10-19T01:26:53.998049939+00:00", "category": "ai", "content_html": "

Large diffusion transformers can create stunning images (or even videos, audio snippets, and now text), but loading a modern text-to-image model in BF16 precision often requires 20-30 GB of VRAM, which puts these models out of reach of most consumer GPUs. Quantization is a powerful solution to this problem, and Diffusers already integrates several quantization backends such as bitsandbytes, GGUF, torchao, and Quanto, which we covered in Exploring Quantization Backends in Diffusers.

\n

Most of these backends are weight-only. This means that they store the weights in low precision and dequantize them back to high precision at compute time. This reduces memory usage significantly, but it usually does not make inference faster, and can even add a small latency overhead.

\n

SVDQuant, the quantization method behind the popular Nunchaku inference engine, takes a different approach. It runs the main transformer layers with 4-bit weights and activations (W4A4), reducing memory while also speeding up the denoising loop. The details are covered below, but until now, using these checkpoints required a separate inference library.

\n

With current Diffusers, loading a Nunchaku checkpoint is as simple as calling from_pretrained(), with no local CUDA compilation required thanks to the kernels package. In addition, the companion diffuse-compressor toolkit lets you quantize new architectures yourself and publish them as regular Diffusers repositories.

\n
\"Nunchaku

\n

Table of Contents

\n\n

Getting started with Nunchaku Lite

\n

First, install the requirements. You need a recent version of Diffusers and the Hugging Face kernels package:

\n
pip install -U diffusers transformers accelerate kernels bitsandbytes\n
\n

Then load a pre-quantized pipeline like any other Diffusers model:

\n
import torch\nfrom diffusers import ErnieImagePipeline\n\npipe = ErnieImagePipeline.from_pretrained(\n    "lite-infer/ERNIE-Image-Turbo-nunchaku-lite-nvfp4_r32-bnb4-text-encoder",\n    torch_dtype=torch.bfloat16,\n).to("cuda")\n\nimage = pipe(\n    prompt="A cinematic portrait of a red fox in a misty forest at sunrise, "\n           "detailed fur, volumetric light",\n    height=1024,\n    width=1024,\n    num_inference_steps=8,\n    guidance_scale=1.0,\n    generator=torch.Generator("cuda").manual_seed(42),\n).images[0]\nimage.save("output.png")\n
\n

\"BF16

\n

No custom pipeline class or separate inference engine is needed, and there is nothing to compile locally. The NVFP4 kernels are downloaded from the Hub through the Nunchaku Lite kernels page the first time they are used. This checkpoint pairs a Nunchaku NVFP4 transformer with a bitsandbytes NF4 text encoder, and generates a 1024x1024 image in about 1.7 seconds on an RTX 5090 with a peak memory usage of about 12 GB, compared with about 24 GB for the BF16 pipeline. You can find more details about the Nunchaku Lite checkpoint format in the official Diffusers documentation.

\n
\n

NVFP4 checkpoints require an NVIDIA Blackwell GPU (RTX 50 series, RTX PRO 6000, B200). For earlier generations, use the INT4 variants. See the hardware support table below for details.

\n
\n

Background: SVDQuant and Nunchaku

\n

SVDQuant is the quantization method behind Nunchaku, its reference CUDA inference engine. Standard 4-bit quantization is difficult for diffusion transformers because both weights and activations contain large outliers. SVDQuant handles this by moving activation outliers into the weights, representing the hardest part of each weight matrix with a small 16-bit low-rank branch, and quantizing the remaining residual to 4 bits. Nunchaku makes this fast with fused kernels for the 4-bit path and the low-rank branch.

\n

\"Nunchaku

Nunchaku fuses the low-rank down projection with the quantization kernel and the low-rank up projection with the 4-bit compute kernel, eliminating the memory access overhead of the 16-bit branch. Figure from the SVDQuant paper.
\n

Introducing Nunchaku Lite

\n

The original Nunchaku engine gets much of its speed from model-specific fused execution paths, such as fused QKV projections and fused GELU/MLP kernels. Those optimizations are tied to each architecture's module layout and checkpoint format, so supporting a new model family usually requires model-specific integration work.

\n

Nunchaku Lite is the new integration path in Diffusers. With it, Diffusers can load Nunchaku-style checkpoints without a custom pipeline or a separate inference engine. Under the hood, Nunchaku Lite patches the relevant nn.Linear modules of a stock Diffusers model with runtime SVDQ/AWQ linear layers before the checkpoint is loaded. The CUDA kernels come from the Hub through the kernels package. Two kernel families are used:

\n\n

The trade-off is that, without architecture-specific fused kernels and modules, Nunchaku Lite cannot match the speedup of the original Nunchaku engine. However, the bare-bones implementation still delivers around 30% speedup while retaining the same level of VRAM reduction.

\n

Native loading in Diffusers

\n

If you have used bitsandbytes or torchao in Diffusers, the mechanics will feel familiar. A Nunchaku Lite model repository is an ordinary Diffusers repository. The only special part is a quantization_config block inside the transformer's config.json:

\n
"quantization_config": {\n    "quant_method": "nunchaku_lite",\n    "compute_dtype": "bfloat16",\n    "svdq_w4a4": {\n        "precision": "nvfp4",\n        "group_size": 16,\n        "rank": 32,\n        "targets": [\n            "layers.0.self_attention.to_q",\n            "layers.0.self_attention.to_k",\n            "..."\n        ]\n    },\n    "awq_w4a16": {\n        "precision": "int4",\n        "group_size": 64,\n        "targets": [\n            "adaLN_modulation.1",\n            "..."\n        ]\n    }\n}\n
\n

This config tells Diffusers which modules were quantized, which scheme they use, and which Nunchaku Lite runtime layer to instantiate (SVDQW4A4Linear or AWQW4A16Linear).

\n

Because the quantized model keeps the exact module structure of the dense one, everything downstream (schedulers, LoRA loading hooks, offloading, torch.compile) sees a normal Diffusers model.

\n

Hardware support

\n

Nunchaku Lite uses different kernel variants depending on the GPU generation and checkpoint precision:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
SchemePrecisionSupported GPUs
svdq_w4a4nvfp4Blackwell (RTX 50 series, RTX PRO 6000, B200)
svdq_w4a4int4Turing / Ampere / Ada (RTX 30 & 40 series, A100, L40S)
awq_w4a16int4Turing / Ampere / Ada (RTX 30 & 40 series, A100, L40S)
\n
\n

Volta and Hopper GPUs are currently not supported by the 4-bit kernels. The quantizer validates the GPU's CUDA capability at load time and raises a clear error instead of producing incorrect outputs.

\n
\n

Getting more speed and lower memory

\n

Nunchaku Lite can be combined with other Diffusers memory and speed optimizations.

\n

torch.compile. Compiling the transformer improves the end-to-end speedup from 1.35x to 1.8x:

\n
pipe.transformer.compile(fullgraph=True)\n\n# or compile_repeated_blocks() for faster compilation\n\npipe.transformer.compile_repeated_blocks(fullgraph=True)\n
\n

Quantized text encoders. The transformer is not the only component with a large memory footprint. Text encoders such as T5 or Qwen3 can occupy several gigabytes on their own. Further quantizing the text encoder with bitsandbytes NF4 reduces peak VRAM by about 22% in our benchmark.

\n

Offloading. Diffusers offloading helpers such as enable_model_cpu_offload() and enable_sequential_cpu_offload() work as usual if you need to fit the pipeline onto a smaller GPU.

\n

Benchmarks

\n

All numbers below were measured on an NVIDIA RTX PRO 6000 (Blackwell) at 1024x1024 using rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder.

\n

End-to-end latency and memory

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ConfigurationFull pipelineDenoise loopPeak VRAMSpeedup
BF16 baseline3.00 s2.86 s31.1 GB1.0x
Nunchaku Lite NVFP42.27 s2.13 s20.6 GB1.35x
Nunchaku Lite NVFP4 + torch.compile1.68 s1.53 s20.6 GB1.8x
Nunchaku Lite NVFP4 + NF4 text encoder2.29 s2.13 s16.0 GB1.35x
\n

As shown above, Nunchaku reduces peak VRAM by up to 50% while still improving latency by roughly 30%. The remaining overhead comes largely from extra kernel launches, which torch.compile can mitigate, bringing the full pipeline down to 1.68 s, or 1.8x faster than the BF16 baseline.

\n

Image quality

\n
\"Quality
BF16 vs 4-bit outputs with identical seeds and settings.
\n

Quantizing your own model

\n

Nunchaku Lite support in Diffusers is architecture-agnostic, and the diffuse-compressor toolkit provides an end-to-end SVDQuant workflow for Diffusers models: calibrate, quantize, package, and publish.

\n

Below, we walk through quantizing FLUX.2 Klein 4B as an example. It covers the main steps: inspect the model, calibrate and quantize the transformer, package the result as a Diffusers pipeline, then verify and push it to the Hub. The full tutorial covers every flag in detail.

\n

1. Inspect what will be quantized

\n

The generic scanner walks the model and decides what to target: compatible linears inside the repeated transformer-block stack become SVDQ W4A4 targets, recognized modulation linears become AWQ W4A16 targets, and everything else stays dense.

\n
python examples/text_to_image/quantize_hf.py black-forest-labs/FLUX.2-klein-4B \\\n  --precision int4 --rank 32 --inspect-config\n
\n

Always read this report before quantizing. For FLUX.2 Klein 4B, the expected result is 100 SVDQ targets, 3 AWQ targets, and 6 dense outer linears, with no missing patterns or duplicate names.

\n

2. Run quantization

\n

The following command runs SVDQuant on the transformer and writes the quantized checkpoint to outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors:

\n
python examples/text_to_image/quantize_hf.py black-forest-labs/FLUX.2-klein-4B \\\n  --precision int4 \\\n  --output outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors\n
\n

Replace --precision int4 with nvfp4 to build Blackwell-native weights.

\n

3. Package a Diffusers pipeline

\n

The converter combines the quantized transformer with the base pipeline's other components, writes the compact nunchaku_lite configuration into transformer/config.json, and can optionally convert text encoders to NF4:

\n
python examples/convert_nunchaku_lite_diffusers.py \\\n  --checkpoint outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors \\\n  --model-id black-forest-labs/FLUX.2-klein-4B \\\n  --bnb4-text-encoder text_encoder \\\n  --compute-dtype bfloat16 \\\n  --output-dir outputs/diffusers/FLUX.2-klein-4B-nunchaku-lite-int4-bnb4-text-encoder\n
\n

4. Load, verify, and push to the Hub

\n
import torch\nfrom diffusers import DiffusionPipeline\n\npipe = DiffusionPipeline.from_pretrained(\n    "outputs/diffusers/FLUX.2-klein-4B-nunchaku-lite-int4-bnb4-text-encoder",\n    device_map="cuda",\n)\nimage = pipe(\n    "A glass robot in a greenhouse, cinematic lighting",\n    num_inference_steps=4, guidance_scale=1.0,\n    generator=torch.Generator("cuda").manual_seed(12345),\n).images[0]\n
\n

Once the outputs look good, run pipe.push_to_hub("your-name/your-model-nunchaku-lite-int4"). Other users can then load it with the same from_pretrained() pattern shown above.

\n

Quantizing models with structural rewrites

\n

Note that the generic path assumes the architecture can be quantized without structural rewrites. For additional speedup, the original Nunchaku engine rewrites groups of Diffusers layers as fused modules. The generic path cannot infer these changes on its own, such as combining separate Q, K, and V projections into one module or splitting a fused projection across several modules.

\n

FLUX.1-dev's QKV projection is a concrete example. Diffusers defines three separate modules:

\n
self.to_q = torch.nn.Linear(query_dim, self.inner_dim, bias=bias)\nself.to_k = torch.nn.Linear(query_dim, self.inner_dim, bias=bias)\nself.to_v = torch.nn.Linear(query_dim, self.inner_dim, bias=bias)\n
\n

The Nunchaku FLUX module combines those layers into one quantized to_qkv module:

\n
to_qkv = fuse_linears([other.to_q, other.to_k, other.to_v])\nself.to_qkv = SVDQW4A4Linear.from_linear(to_qkv, **kwargs)\n
\n

This grouped module is required because Nunchaku's fused operator consumes the QKV projection, Q/K normalization, and rotary embeddings together. By comparison, the default Diffusers path executes them separately:

\n
query = attn.to_q(hidden_states)\nkey = attn.to_k(hidden_states)\nvalue = attn.to_v(hidden_states)\n\nquery = query.unflatten(-1, (attn.heads, -1))\nkey = key.unflatten(-1, (attn.heads, -1))\nvalue = value.unflatten(-1, (attn.heads, -1))\n\nquery = attn.norm_q(query)\nkey = attn.norm_k(key)\n\nif image_rotary_emb is not None:\n    query = apply_rotary_emb(query, image_rotary_emb, sequence_dim=1)\n    key = apply_rotary_emb(key, image_rotary_emb, sequence_dim=1)\n
\n

The Nunchaku path supplies the grouped projection, normalization modules, and rotary embeddings to one fused operator:

\n
qkv = fused_qkv_norm_rottary(\n    hidden_states, attn.to_qkv, attn.norm_q, attn.norm_k, image_rotary_emb\n)\n
\n

This is the structural rewrite that the generic path cannot infer. Diffusers has three destination modules with to_q, to_k, and to_v parameter prefixes, while Nunchaku has one grouped module under to_qkv. A model-specific target config or adapter must state that the Q, K, and V parameters should be concatenated along the output dimension, in that order, and loaded into to_qkv.

\n

Structural rewrites like these are described by a model-specific target config during quantization and handled by a small runtime adapter when the checkpoint is loaded. The FLUX.2 Klein 4B quantization script provides a concrete target-config example for producing a structurally rewritten checkpoint, while rootonchair/nunchaku-lite provides the runtime adapters needed to load grouped QKV tensors, split fused projections, and other fused operations. For the complete workflow, you can check the Adding A New Model guide.

\n

Ready-to-use checkpoints

\n

To get started right away, check out the following repositories:

\n\n

Conclusion

\n

Nunchaku's SVDQuant kernels are one of the most effective ways to run diffusion transformers efficiently on consumer hardware, and they are now natively supported in Diffusers. Pre-quantized checkpoints load with from_pretrained(), and the diffuse-compressor toolkit makes it possible to quantize new architectures without waiting for engine support. By quantizing both weights and activations, the W4A4 path lowers memory use while improving denoising latency, keeping image quality close to the BF16 original.

\n

If you quantize and publish a new model, we would love to hear about it. Share it on the Hub and let us know! If you have any questions about this feature, feel free to join our Discord.

\n

To learn more, check out the following resources:

\n\n

Acknowledgements

\n

Thanks to the Diffusers maintainers for reviews and guidance throughout the integration, and to the MIT HAN Lab / Nunchaku team for the original SVDQuant work. Thanks to Marc Sun for providing feedback on the blog post. Thanks to Álvaro Somoza for trying out nunchaku-lite and for providing feedback.

\n

rootonchair is also grateful to SilverAI for supporting this work and providing the environment in which much of this development took place.

\n", "content_markdown": "Large diffusion transformers can create stunning images (or even videos, audio snippets, and now text), but loading a modern text-to-image model in BF16 precision often requires 20-30 GB of VRAM, which puts these models out of reach of most consumer GPUs. Quantization is a powerful solution to this problem, and Diffusers already integrates several quantization backends such as bitsandbytes, GGUF, torchao, and Quanto, which we covered in [Exploring Quantization Backends in Diffusers](https://huggingface.co/blog/diffusers-quantization).\n\nMost of these backends are *weight-only*. This means that they store the weights in low precision and dequantize them back to high precision at compute time. This reduces memory usage significantly, but it usually does not make inference faster, and can even add a small latency overhead.\n\n[SVDQuant](https://arxiv.org/abs/2411.05007), the quantization method behind the popular [Nunchaku](https://github.com/nunchaku-tech/nunchaku) inference engine, takes a different approach. It runs the main transformer layers with 4-bit weights and activations (W4A4), reducing memory while also speeding up the denoising loop. The details are covered below, but until now, using these checkpoints required a separate inference library.\n\nWith current Diffusers, loading a Nunchaku checkpoint is as simple as calling `from_pretrained()`, with no local CUDA compilation required thanks to the [`kernels`](https://github.com/huggingface/kernels) package. In addition, the companion [diffuse-compressor](https://github.com/rootonchair/diffuse-compressor) toolkit lets you quantize new architectures yourself and publish them as regular Diffusers repositories.\n\n![Nunchaku Lite image quality and performance comparison](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/nunchaku-diffusers/contact_sheet_top3_metrics_bold.png)\n\n## Table of Contents\n\n- [Getting started with Nunchaku Lite](https://huggingface.co/blog/nunchaku-diffusers#getting-started-with-nunchaku-lite)\n- [Background: SVDQuant and Nunchaku](https://huggingface.co/blog/nunchaku-diffusers#background-svdquant-and-nunchaku)\n- [Introducing Nunchaku Lite](https://huggingface.co/blog/nunchaku-diffusers#introducing-nunchaku-lite)\n- [Native loading in Diffusers](https://huggingface.co/blog/nunchaku-diffusers#native-loading-in-diffusers)\n- [Getting more speed and lower memory](https://huggingface.co/blog/nunchaku-diffusers#getting-more-speed-and-lower-memory)\n- [Benchmarks](https://huggingface.co/blog/nunchaku-diffusers#benchmarks)\n- [Quantizing your own model](https://huggingface.co/blog/nunchaku-diffusers#quantizing-your-own-model)\n- [Ready-to-use checkpoints](https://huggingface.co/blog/nunchaku-diffusers#ready-to-use-checkpoints)\n- [Conclusion](https://huggingface.co/blog/nunchaku-diffusers#conclusion)\n- [Acknowledgements](https://huggingface.co/blog/nunchaku-diffusers#acknowledgements)\n\n## Getting started with Nunchaku Lite\n\nFirst, install the requirements. You need a recent version of Diffusers and the Hugging Face `kernels` package:\n\n```bash\npip install -U diffusers transformers accelerate kernels bitsandbytes\n```\n\nThen load a pre-quantized pipeline like any other Diffusers model:\n\n```python\nimport torch\nfrom diffusers import ErnieImagePipeline\n\npipe = ErnieImagePipeline.from_pretrained(\n \"lite-infer/ERNIE-Image-Turbo-nunchaku-lite-nvfp4_r32-bnb4-text-encoder\",\n torch_dtype=torch.bfloat16,\n).to(\"cuda\")\n\nimage = pipe(\n prompt=\"A cinematic portrait of a red fox in a misty forest at sunrise, \"\n \"detailed fur, volumetric light\",\n height=1024,\n width=1024,\n num_inference_steps=8,\n guidance_scale=1.0,\n generator=torch.Generator(\"cuda\").manual_seed(42),\n).images[0]\nimage.save(\"output.png\")\n```\n\n![BF16 and Nunchaku Lite outputs for a red fox prompt](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/nunchaku-diffusers/fox_bf16_vs_nunchaku_no_metrics.png)\n\nNo custom pipeline class or separate inference engine is needed, and there is nothing to compile locally. The NVFP4 kernels are downloaded from the Hub through the [Nunchaku Lite kernels page](https://huggingface.co/kernels/rootonchair/nunchaku-lite-kernels) the first time they are used. This checkpoint pairs a Nunchaku NVFP4 transformer with a bitsandbytes NF4 text encoder, and generates a 1024x1024 image in about 1.7 seconds on an RTX 5090 with a peak memory usage of about 12 GB, compared with about 24 GB for the BF16 pipeline. You can find more details about the Nunchaku Lite checkpoint format in the [official Diffusers documentation](https://huggingface.co/docs/diffusers/main/en/quantization/nunchaku).\n\n> NVFP4 checkpoints require an NVIDIA Blackwell GPU (RTX 50 series, RTX PRO 6000, B200). For earlier generations, use the INT4 variants. See the [hardware support](https://huggingface.co/blog/nunchaku-diffusers#hardware-support) table below for details.\n\n## Background: SVDQuant and Nunchaku\n\n**SVDQuant** is the quantization method behind **Nunchaku**, its reference CUDA inference engine. Standard 4-bit quantization is difficult for diffusion transformers because both weights and activations contain large outliers. SVDQuant handles this by moving activation outliers into the weights, representing the hardest part of each weight matrix with a small 16-bit low-rank branch, and quantizing the remaining residual to 4 bits. Nunchaku makes this fast with fused kernels for the 4-bit path and the low-rank branch.\n\n![Nunchaku kernel fusion: the low-rank down projection is fused with input quantization, and the low-rank up projection is fused with the 4-bit matmul](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/nunchaku-diffusers/svdquant_kernel_fusion.png)\\\nNunchaku fuses the low-rank down projection with the quantization kernel and the low-rank up projection with the 4-bit compute kernel, eliminating the memory access overhead of the 16-bit branch. Figure from the [SVDQuant paper](https://arxiv.org/abs/2411.05007).\n\n## Introducing Nunchaku Lite\n\nThe original [Nunchaku engine](https://github.com/nunchaku-ai/nunchaku) gets much of its speed from [model-specific fused execution paths](https://huggingface.co/blog/nunchaku-diffusers#quantizing-models-with-structural-rewrites), such as fused QKV projections and fused GELU/MLP kernels. Those optimizations are tied to each architecture's module layout and checkpoint format, so supporting a new model family usually requires model-specific integration work.\n\n**Nunchaku Lite** is the new integration path in Diffusers. With it, Diffusers can load Nunchaku-style checkpoints without a custom pipeline or a separate inference engine. Under the hood, Nunchaku Lite patches the relevant `nn.Linear` modules of a stock Diffusers model with runtime SVDQ/AWQ linear layers before the checkpoint is loaded. The CUDA kernels come from the Hub through the `kernels` package. Two kernel families are used:\n\n- **`svdq_w4a4`**: 4-bit weights and activations with the SVDQuant low-rank correction. This layer is used for the transformer's attention and MLP projections, where nearly all of the compute is spent, and is available in INT4 and NVFP4 variants.\n- **`awq_w4a16`**: 4-bit weights with 16-bit activations, used for adaptive normalization and modulation projections such as FLUX `adanorm_single` / `adanorm_zero` or Qwen-Image modulation layers. These layers are memory-bound and precision-sensitive, making AWQ a good fit to preserve precision while still saving memory and space.\n\nThe trade-off is that, without architecture-specific fused kernels and modules, Nunchaku Lite cannot match the speedup of the original Nunchaku engine. However, the bare-bones implementation still delivers around **30% speedup** while retaining the same level of **VRAM reduction**.\n\n## Native loading in Diffusers\n\nIf you have used bitsandbytes or torchao in Diffusers, the mechanics will feel familiar. A Nunchaku Lite model repository is an ordinary Diffusers repository. The only special part is a `quantization_config` block inside the transformer's `config.json`:\n\n```json\n\"quantization_config\": {\n \"quant_method\": \"nunchaku_lite\",\n \"compute_dtype\": \"bfloat16\",\n \"svdq_w4a4\": {\n \"precision\": \"nvfp4\",\n \"group_size\": 16,\n \"rank\": 32,\n \"targets\": [\n \"layers.0.self_attention.to_q\",\n \"layers.0.self_attention.to_k\",\n \"...\"\n ]\n },\n \"awq_w4a16\": {\n \"precision\": \"int4\",\n \"group_size\": 64,\n \"targets\": [\n \"adaLN_modulation.1\",\n \"...\"\n ]\n }\n}\n```\n\nThis config tells Diffusers which modules were quantized, which scheme they use, and which Nunchaku Lite runtime layer to instantiate (`SVDQW4A4Linear` or `AWQW4A16Linear`).\n\nBecause the quantized model keeps the exact module structure of the dense one, everything downstream (schedulers, LoRA loading hooks, offloading, `torch.compile`) sees a normal Diffusers model.\n\n### Hardware support\n\nNunchaku Lite uses different kernel variants depending on the GPU generation and checkpoint precision:\n\n| Scheme | Precision | Supported GPUs |\n| ----------- | --------- | ------------------------------------------------------ |\n| `svdq_w4a4` | `nvfp4` | Blackwell (RTX 50 series, RTX PRO 6000, B200) |\n| `svdq_w4a4` | `int4` | Turing / Ampere / Ada (RTX 30 & 40 series, A100, L40S) |\n| `awq_w4a16` | `int4` | Turing / Ampere / Ada (RTX 30 & 40 series, A100, L40S) |\n\n> Volta and Hopper GPUs are currently not supported by the 4-bit kernels. The quantizer validates the GPU's CUDA capability at load time and raises a clear error instead of producing incorrect outputs.\n\n## Getting more speed and lower memory\n\nNunchaku Lite can be combined with other Diffusers memory and speed optimizations.\n\n**`torch.compile`.** Compiling the transformer improves the end-to-end speedup from 1.35x to 1.8x:\n\n```python\npipe.transformer.compile(fullgraph=True)\n\n# or compile_repeated_blocks() for faster compilation\n\npipe.transformer.compile_repeated_blocks(fullgraph=True)\n```\n\n**Quantized text encoders.** The transformer is not the only component with a large memory footprint. Text encoders such as T5 or Qwen3 can occupy several gigabytes on their own. Further quantizing the text encoder with bitsandbytes NF4 reduces peak VRAM by about 22% in our benchmark.\n\n**Offloading.** Diffusers offloading helpers such as `enable_model_cpu_offload()` and `enable_sequential_cpu_offload()` work as usual if you need to fit the pipeline onto a smaller GPU.\n\n## Benchmarks\n\nAll numbers below were measured on an NVIDIA RTX PRO 6000 (Blackwell) at 1024x1024 using [rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder](https://huggingface.co/rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder).\n\n### End-to-end latency and memory\n\n| Configuration | Full pipeline | Denoise loop | Peak VRAM | Speedup |\n| -------------------------------------- | ------------- | ------------ | --------- | ------- |\n| BF16 baseline | 3.00 s | 2.86 s | 31.1 GB | 1.0x |\n| Nunchaku Lite NVFP4 | 2.27 s | 2.13 s | 20.6 GB | 1.35x |\n| Nunchaku Lite NVFP4 + `torch.compile` | 1.68 s | 1.53 s | 20.6 GB | 1.8x |\n| Nunchaku Lite NVFP4 + NF4 text encoder | 2.29 s | 2.13 s | 16.0 GB | 1.35x |\n\nAs shown above, Nunchaku reduces peak VRAM by up to 50% while still improving latency by roughly 30%. The remaining overhead comes largely from extra kernel launches, which `torch.compile` can mitigate, bringing the full pipeline down to 1.68 s, or 1.8x faster than the BF16 baseline.\n\n### Image quality\n\n![Quality comparison grid](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/nunchaku-diffusers/quality_grid.png)\\\nBF16 vs 4-bit outputs with identical seeds and settings.\n\n## Quantizing your own model\n\nNunchaku Lite support in Diffusers is architecture-agnostic, and the [diffuse-compressor](https://github.com/rootonchair/diffuse-compressor) toolkit provides an end-to-end SVDQuant workflow for Diffusers models: calibrate, quantize, package, and publish.\n\nBelow, we walk through quantizing FLUX.2 Klein 4B as an example. It covers the main steps: inspect the model, calibrate and quantize the transformer, package the result as a Diffusers pipeline, then verify and push it to the Hub. The [full tutorial](https://github.com/rootonchair/diffuse-compressor/blob/main/docs/quantize_new_hf_model.md) covers every flag in detail.\n\n### 1. Inspect what will be quantized\n\nThe generic scanner walks the model and decides what to target: compatible linears inside the repeated transformer-block stack become SVDQ W4A4 targets, recognized modulation linears become AWQ W4A16 targets, and everything else stays dense.\n\n```bash\npython examples/text_to_image/quantize_hf.py black-forest-labs/FLUX.2-klein-4B \\\n --precision int4 --rank 32 --inspect-config\n```\n\nAlways read this report before quantizing. For FLUX.2 Klein 4B, the expected result is 100 SVDQ targets, 3 AWQ targets, and 6 dense outer linears, with no missing patterns or duplicate names.\n\n### 2. Run quantization\n\nThe following command runs SVDQuant on the transformer and writes the quantized checkpoint to `outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors`:\n\n```bash\npython examples/text_to_image/quantize_hf.py black-forest-labs/FLUX.2-klein-4B \\\n --precision int4 \\\n --output outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors\n```\n\nReplace `--precision int4` with `nvfp4` to build Blackwell-native weights.\n\n### 3. Package a Diffusers pipeline\n\nThe converter combines the quantized transformer with the base pipeline's other components, writes the compact `nunchaku_lite` configuration into `transformer/config.json`, and can optionally convert text encoders to NF4:\n\n```bash\npython examples/convert_nunchaku_lite_diffusers.py \\\n --checkpoint outputs/checkpoints/svdq-int4_r32-flux-2-klein-4b.safetensors \\\n --model-id black-forest-labs/FLUX.2-klein-4B \\\n --bnb4-text-encoder text_encoder \\\n --compute-dtype bfloat16 \\\n --output-dir outputs/diffusers/FLUX.2-klein-4B-nunchaku-lite-int4-bnb4-text-encoder\n```\n\n### 4. Load, verify, and push to the Hub\n\n```python\nimport torch\nfrom diffusers import DiffusionPipeline\n\npipe = DiffusionPipeline.from_pretrained(\n \"outputs/diffusers/FLUX.2-klein-4B-nunchaku-lite-int4-bnb4-text-encoder\",\n device_map=\"cuda\",\n)\nimage = pipe(\n \"A glass robot in a greenhouse, cinematic lighting\",\n num_inference_steps=4, guidance_scale=1.0,\n generator=torch.Generator(\"cuda\").manual_seed(12345),\n).images[0]\n```\n\nOnce the outputs look good, run `pipe.push_to_hub(\"your-name/your-model-nunchaku-lite-int4\")`. Other users can then load it with the same `from_pretrained()` pattern shown above.\n\n### Quantizing models with structural rewrites\n\nNote that the generic path assumes the architecture can be quantized without structural rewrites. For additional speedup, the original Nunchaku engine rewrites groups of Diffusers layers as fused modules. The generic path cannot infer these changes on its own, such as combining separate Q, K, and V projections into one module or splitting a fused projection across several modules.\n\nFLUX.1-dev's QKV projection is a concrete example. [Diffusers defines three separate modules](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py#L313-L329):\n\n```python\nself.to_q = torch.nn.Linear(query_dim, self.inner_dim, bias=bias)\nself.to_k = torch.nn.Linear(query_dim, self.inner_dim, bias=bias)\nself.to_v = torch.nn.Linear(query_dim, self.inner_dim, bias=bias)\n```\n\nThe [Nunchaku FLUX module combines those layers](https://github.com/nunchaku-ai/nunchaku/blob/main/nunchaku/models/transformers/transformer_flux_v2.py#L63-L79) into one quantized `to_qkv` module:\n\n```python\nto_qkv = fuse_linears([other.to_q, other.to_k, other.to_v])\nself.to_qkv = SVDQW4A4Linear.from_linear(to_qkv, **kwargs)\n```\n\nThis grouped module is required because Nunchaku's fused operator consumes the QKV projection, Q/K normalization, and rotary embeddings together. By comparison, the [default Diffusers path](https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/transformers/transformer_flux.py#L45-L116) executes them separately:\n\n```python\nquery = attn.to_q(hidden_states)\nkey = attn.to_k(hidden_states)\nvalue = attn.to_v(hidden_states)\n\nquery = query.unflatten(-1, (attn.heads, -1))\nkey = key.unflatten(-1, (attn.heads, -1))\nvalue = value.unflatten(-1, (attn.heads, -1))\n\nquery = attn.norm_q(query)\nkey = attn.norm_k(key)\n\nif image_rotary_emb is not None:\n query = apply_rotary_emb(query, image_rotary_emb, sequence_dim=1)\n key = apply_rotary_emb(key, image_rotary_emb, sequence_dim=1)\n```\n\nThe [Nunchaku path](https://github.com/nunchaku-ai/nunchaku/blob/main/nunchaku/models/attention_processors/flux.py#L69-L93) supplies the grouped projection, normalization modules, and rotary embeddings to one fused operator:\n\n```python\nqkv = fused_qkv_norm_rottary(\n hidden_states, attn.to_qkv, attn.norm_q, attn.norm_k, image_rotary_emb\n)\n```\n\nThis is the structural rewrite that the generic path cannot infer. Diffusers has three destination modules with `to_q`, `to_k`, and `to_v` parameter prefixes, while Nunchaku has one grouped module under `to_qkv`. A model-specific target config or adapter must state that the Q, K, and V parameters should be concatenated along the output dimension, in that order, and loaded into `to_qkv`.\n\nStructural rewrites like these are described by a model-specific target config during quantization and handled by a small runtime adapter when the checkpoint is loaded. The [FLUX.2 Klein 4B quantization script](https://github.com/rootonchair/diffuse-compressor/blob/main/examples/text_to_image/quantize_flux2_klein_4b.py) provides a concrete target-config example for producing a structurally rewritten checkpoint, while [rootonchair/nunchaku-lite](https://github.com/rootonchair/nunchaku-lite) provides the runtime adapters needed to load grouped QKV tensors, split fused projections, and other fused operations. For the complete workflow, you can check the [Adding A New Model](https://github.com/rootonchair/diffuse-compressor/blob/main/docs/adding_new_model.md) guide.\n\n## Ready-to-use checkpoints\n\nTo get started right away, check out the following repositories:\n\n- [rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder](https://huggingface.co/rootonchair/ERNIE-Image-Turbo-nunchaku-lite-int4-bnb4-text-encoder): INT4 ERNIE-Image-Turbo with a bitsandbytes NF4 text encoder\n- [rootonchair/ERNIE-Image-Turbo-nunchaku-lite-nvfp4-bnb4-text-encoder](https://huggingface.co/rootonchair/ERNIE-Image-Turbo-nunchaku-lite-nvfp4-bnb4-text-encoder): NVFP4 ERNIE-Image-Turbo with a bitsandbytes NF4 text encoder\n- [OzzyGT/Krea\\_2\\_Turbo\\_nunchaku\\_lite\\_nvfp4](https://huggingface.co/OzzyGT/Krea_2_Turbo_nunchaku_lite_nvfp4): NVFP4 Krea 2 Turbo checkpoint\n- [lite-infer](https://huggingface.co/lite-infer): more Nunchaku Lite checkpoints and collections\n\n## Conclusion\n\nNunchaku's SVDQuant kernels are one of the most effective ways to run diffusion transformers efficiently on consumer hardware, and they are now natively supported in Diffusers. Pre-quantized checkpoints load with `from_pretrained()`, and the diffuse-compressor toolkit makes it possible to quantize new architectures without waiting for engine support. By quantizing both weights and activations, the W4A4 path lowers memory use while improving denoising latency, keeping image quality close to the BF16 original.\n\nIf you quantize and publish a new model, we would love to hear about it. Share it on the Hub and let us know! If you have any questions about this feature, feel free to join our [Discord](https://discord.gg/G7tWnz98XR).\n\nTo learn more, check out the following resources:\n\n- [Diffusers Nunchaku documentation](https://huggingface.co/docs/diffusers/quantization/nunchaku)\n- [The integration PR (huggingface/diffusers#14100)](https://github.com/huggingface/diffusers/pull/14100)\n- [SVDQuant paper](https://arxiv.org/abs/2411.05007) and the [Nunchaku engine](https://github.com/nunchaku-tech/nunchaku)\n- [diffuse-compressor](https://github.com/rootonchair/diffuse-compressor)\n- Previous posts: [Exploring Quantization Backends in Diffusers](https://huggingface.co/blog/diffusers-quantization) and [Memory-efficient Diffusion Transformers with Quanto and Diffusers](https://huggingface.co/blog/quanto-diffusers)\n\n## Acknowledgements\n\nThanks to the Diffusers maintainers for reviews and guidance throughout the integration, and to the MIT HAN Lab / Nunchaku team for the original SVDQuant work. Thanks to Marc Sun for providing feedback on the blog post. Thanks to Álvaro Somoza for trying out `nunchaku-lite` and for providing feedback.\n\n`rootonchair` is also grateful to SilverAI for supporting this work and providing the environment in which much of this development took place.\n", "dateCreated": "2026-10-19T01:26:53.998049939+00:00", "date_created": "2026-07-23T00:00:00+00:00", "date_modified": "2026-07-23T00:00:00+00:00", "date_published": "2026-07-23T00:00:00+00:00", "description": "Large diffusion transformers can create stunning images (or even videos, audio snippets, and now text), but loading a modern text-to-image model in BF16 precision often requires 20-30 GB of VRAM, which puts these models out of reach of…", "external_url": "https://huggingface.co/blog/nunchaku-diffusers", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/07/2026-07-23-bringing-nunchaku-4-bit-diffusion-inference-to-diffusers.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/07/2026-07-23-bringing-nunchaku-4-bit-diffusion-inference-to-diffusers.md" }, "id": "urn:aggr:item:aca977f6944f64fdbf535e14c4d2b5a7dbca5103", "isBasedOn": { "@id": "https://huggingface.co/blog/nunchaku-diffusers" }, "mainEntity": { "@id": "https://huggingface.co/blog/nunchaku-diffusers", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-07-23-bringing-nunchaku-4-bit-diffusion-inference-to-diffusers/#webpage" }, "dateModified": "2026-07-23T00:00:00+00:00", "datePublished": "2026-07-23T00:00:00+00:00", "headline": "Bringing Nunchaku 4-bit Diffusion Inference to Diffusers", "image": { "@type": "ImageObject", "caption": null, "height": 90, "url": "https://huggingface.co/blog/assets/nunchaku-diffusers/thumbnail.png", "width": 160 }, "timeRequired": "PT11M", "url": "https://huggingface.co/blog/nunchaku-diffusers", "wordCount": 2336 }, "name": "Archived snapshot: Bringing Nunchaku 4-bit Diffusion Inference to Diffusers", "preview": { "alt": null, "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://huggingface.co/blog/assets/nunchaku-diffusers/thumbnail.png", "width": 160 }, "reading_minutes": 11, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-19T01:26:53.998049939+00:00", "title": "Bringing Nunchaku 4-bit Diffusion Inference to Diffusers", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-07-23-bringing-nunchaku-4-bit-diffusion-inference-to-diffusers/", "word_count": 2336 }