{
  "@context": "https://schema.org",
  "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-11-25-diffusers-welcomes-flux-2/#webpage",
  "@type": [
    "WebPage",
    "ArchiveComponent"
  ],
  "articleBody": "import torch\\\nfrom transformers import Mistral3ForConditionalGeneration\n\nfrom diffusers import Flux2Pipeline, Flux2Transformer2DModel\n\nrepo\\_id = \"diffusers/FLUX.2-dev-bnb-4bit\"\\\ndevice = \"cuda:0\"\\\ntorch\\_dtype = torch.bfloat16\n\ntransformer = Flux2Transformer2DModel.from\\_pretrained(\\\n repo\\_id, subfolder=\"transformer\", torch\\_dtype=torch\\_dtype, device\\_map=\"cpu\"\\\n)\\\ntext\\_encoder = Mistral3ForConditionalGeneration.from\\_pretrained(\\\n repo\\_id, subfolder=\"text\\_encoder\", dtype=torch\\_dtype, device\\_map=\"cpu\"\\\n)\n\npipe = Flux2Pipeline.from\\_pretrained(\\\n repo\\_id, transformer=transformer, text\\_encoder=text\\_encoder, torch\\_dtype=torch\\_dtype\\\n)\\\npipe.enable\\_model\\_cpu\\_offload()\n\nprompt = \"Realistic macro photograph of a hermit crab using a soda can as its shell, partially emerging from the can, captured with sharp detail and natural colors, on a sunlit beach with soft shadows and a shallow depth of field, with blurred ocean waves in the background. The can has the text `BFL Diffusers` on it and it has a color gradient that start with #FF5733 at the top and transitions to #33FF57 at the bottom.\"\n\nimage = pipe(\\\n prompt=prompt,\\\n generator=torch.Generator(device=device).manual\\_seed(42),\\\n num\\_inference\\_steps=50, # 28 is a good trade-off\\\n guidance\\_scale=4,\\\n).images\\[0\\]\n\nimage.save(\"flux2\\_t2i\\_nf4.png\")\n\nFlax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.\\\nFlax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.\\\n/usr/local/lib/python3.12/dist-packages/huggingface\\_hub/utils/\\_validators.py:206: UserWarning: The `local_dir_use_symlinks` argument is deprecated and ignored in `hf_hub_download`. Downloading to a local directory does not use symlinks anymore.\\\n warnings.warn(\\\nDownload complete:  0.00/0.00 \\[00:00\\<?, ?B/s\\]Fetching 2 files: 100% 2/2 \\[00:00<00:00, 98.80it/s\\]Loading checkpoint shards: 100% 2/2 \\[00:01<00:00,  2.03it/s\\]Download complete:  0.00/0.00 \\[00:00\\<?, ?B/s\\]Fetching 4 files: 100% 4/4 \\[00:00<00:00, 167.57it/s\\]Loading weights: 100% 585/585 \\[00:02<00:00, 310.91it/s, Materializing param=model.vision\\_tower.transformer.layers.23.ffn\\_norm.weight\\]The tied weights mapping and config for this model specifies to tie model.language\\_model.embed\\_tokens.weight to lm\\_head.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning\\\nLoading pipeline components...: 100% 5/5 \\[00:03<00:00,  1.14it/s\\]---------------------------------------------------------------------------\\\nOutOfMemoryError Traceback (most recent call last)\\\n/tmp/ipykernel\\_18947/863729753.py in \\<cell line: 0>()\\\n 22 prompt = \"Realistic macro photograph of a hermit crab using a soda can as its shell, partially emerging from the can, captured with sharp detail and natural colors, on a sunlit beach with soft shadows and a shallow depth of field, with blurred ocean waves in the background. The can has the text `BFL Diffusers` on it and it has a color gradient that start with #FF5733 at the top and transitions to #33FF57 at the bottom.\"\\\n 23\\\n---> 24 image = pipe(\\\n 25 prompt=prompt,\\\n 26 generator=torch.Generator(device=device).manual\\_seed(42),\n\n36 frames/usr/local/lib/python3.12/dist-packages/torch/utils/\\_contextlib.py in decorate\\_context(\\*args, \\*\\*kwargs)\\\n 122 # pyrefly: ignore \\[bad-context-manager\\]\\\n 123 with ctx\\_factory():\\\n--> 124 return func(\\*args, \\*\\*kwargs)\\\n 125\\\n 126 return decorate\\_context\n\n/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline\\_flux2.py in **call**(self, image, prompt, height, width, num\\_inference\\_steps, sigmas, guidance\\_scale, num\\_images\\_per\\_prompt, generator, latents, prompt\\_embeds, output\\_type, return\\_dict, attention\\_kwargs, callback\\_on\\_step\\_end, callback\\_on\\_step\\_end\\_tensor\\_inputs, max\\_sequence\\_length, text\\_encoder\\_out\\_layers, caption\\_upsample\\_temperature)\\\n 869 prompt, images=image, temperature=caption\\_upsample\\_temperature, device=device\\\n 870 )\\\n--> 871 prompt\\_embeds, text\\_ids = self.encode\\_prompt(\\\n 872 prompt=prompt,\\\n 873 prompt\\_embeds=prompt\\_embeds,\n\n/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline\\_flux2.py in encode\\_prompt(self, prompt, device, num\\_images\\_per\\_prompt, prompt\\_embeds, max\\_sequence\\_length, text\\_encoder\\_out\\_layers)\\\n 586\\\n 587 if prompt\\_embeds is None:\\\n--> 588 prompt\\_embeds = self.\\_get\\_mistral\\_3\\_small\\_prompt\\_embeds(\\\n 589 text\\_encoder=self.text\\_encoder,\\\n 590 tokenizer=self.tokenizer,\n\n/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline\\_flux2.py in \\_get\\_mistral\\_3\\_small\\_prompt\\_embeds(text\\_encoder, tokenizer, prompt, dtype, device, max\\_sequence\\_length, system\\_message, hidden\\_states\\_layers)\\\n 337\\\n 338 # Forward pass through the model\\\n--> 339 output = text\\_encoder(\\\n 340 input\\_ids=input\\_ids,\\\n 341 attention\\_mask=attention\\_mask,\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py in new\\_forward(module, \\*args, \\*\\*kwargs)\\\n 190 output = module.\\_old\\_forward(\\*args, \\*\\*kwargs)\\\n 191 else:\\\n--> 192 output = module.\\_old\\_forward(\\*args, \\*\\*kwargs)\\\n 193 return module.\\_hf\\_hook.post\\_forward(module, output)\\\n 194\n\n/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, \\*args, \\*\\*kwargs)\\\n 1000 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1001 else:\\\n-> 1002 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1003 except TypeError as original\\_exception:\\\n 1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.\n\n/usr/local/lib/python3.12/dist-packages/transformers/models/mistral3/modeling\\_mistral3.py in forward(self, input\\_ids, pixel\\_values, attention\\_mask, position\\_ids, past\\_key\\_values, inputs\\_embeds, labels, use\\_cache, output\\_attentions, output\\_hidden\\_states, return\\_dict, cache\\_position, logits\\_to\\_keep, image\\_sizes, \\*\\*kwargs)\\\n 444 return\\_dict = return\\_dict if return\\_dict is not None else self.config.use\\_return\\_dict\\\n 445\\\n--> 446 outputs = self.model(\\\n 447 input\\_ids=input\\_ids,\\\n 448 pixel\\_values=pixel\\_values,\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, \\*args, \\*\\*kwargs)\\\n 1000 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1001 else:\\\n-> 1002 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1003 except TypeError as original\\_exception:\\\n 1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.\n\n/usr/local/lib/python3.12/dist-packages/transformers/models/mistral3/modeling\\_mistral3.py in forward(self, input\\_ids, pixel\\_values, attention\\_mask, position\\_ids, past\\_key\\_values, inputs\\_embeds, vision\\_feature\\_layer, use\\_cache, output\\_attentions, output\\_hidden\\_states, return\\_dict, cache\\_position, image\\_sizes, \\*\\*kwargs)\\\n 323 inputs\\_embeds = inputs\\_embeds.masked\\_scatter(special\\_image\\_mask, image\\_features)\\\n 324\\\n--> 325 outputs = self.language\\_model(\\\n 326 attention\\_mask=attention\\_mask,\\\n 327 position\\_ids=position\\_ids,\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, \\*args, \\*\\*kwargs)\\\n 1000 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1001 else:\\\n-> 1002 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1003 except TypeError as original\\_exception:\\\n 1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.\n\n/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling\\_mistral.py in forward(self, input\\_ids, attention\\_mask, position\\_ids, past\\_key\\_values, inputs\\_embeds, use\\_cache, cache\\_position, \\*\\*kwargs)\\\n 395\\\n 396 for decoder\\_layer in self.layers\\[: self.config.num\\_hidden\\_layers\\]:\\\n--> 397 hidden\\_states = decoder\\_layer(\\\n 398 hidden\\_states,\\\n 399 attention\\_mask=causal\\_mask,\n\n/usr/local/lib/python3.12/dist-packages/transformers/modeling\\_layers.py in **call**(self, \\*args, \\*\\*kwargs)\\\n 91\\\n 92 return self.\\_gradient\\_checkpointing\\_func(partial(super().**call**, \\*\\*kwargs), \\*args)\\\n---> 93 return super().**call**(\\*args, \\*\\*kwargs)\\\n 94\\\n 95\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapped\\_forward(\\*args, \\*\\*kwargs)\\\n 953 if key == \"hidden\\_states\" and len(collected\\_outputs\\[key\\]) == 0:\\\n 954 collected\\_outputs\\[key\\] += (args\\[0\\],)\\\n--> 955 output = orig\\_forward(\\*args, \\*\\*kwargs)\\\n 956 if not isinstance(output, tuple):\\\n 957 collected\\_outputs\\[key\\] += (output,)\n\n/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling\\_mistral.py in forward(self, hidden\\_states, attention\\_mask, position\\_ids, past\\_key\\_values, use\\_cache, cache\\_position, position\\_embeddings, \\*\\*kwargs)\\\n 228 hidden\\_states = self.input\\_layernorm(hidden\\_states)\\\n 229 # Self Attention\\\n--> 230 hidden\\_states, \\_ = self.self\\_attn(\\\n 231 hidden\\_states=hidden\\_states,\\\n 232 attention\\_mask=attention\\_mask,\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling\\_mistral.py in forward(self, hidden\\_states, position\\_embeddings, attention\\_mask, past\\_key\\_values, cache\\_position, \\*\\*kwargs)\\\n 151 hidden\\_shape = (\\*input\\_shape, -1, self.head\\_dim)\\\n 152\\\n--> 153 query\\_states = self.q\\_proj(hidden\\_states).view(hidden\\_shape).transpose(1, 2)\\\n 154 key\\_states = self.k\\_proj(hidden\\_states).view(hidden\\_shape).transpose(1, 2)\\\n 155 value\\_states = self.v\\_proj(hidden\\_states).view(hidden\\_shape).transpose(1, 2)\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/bitsandbytes/nn/modules.py in forward(self, x)\\\n 554 weight = self.weight if getattr(quant\\_state, \"packing\\_format\\_for\\_cpu\", False) else self.weight.t()\\\n 555\\\n--> 556 return bnb.matmul\\_4bit(x, weight, bias=bias, quant\\_state=quant\\_state).to(inp\\_dtype)\\\n 557\\\n 558\n\n/usr/local/lib/python3.12/dist-packages/bitsandbytes/autograd/\\_functions.py in matmul\\_4bit(A, B, quant\\_state, out, bias)\\\n 399 return out\\\n 400 else:\\\n--> 401 return MatMul4Bit.apply(A, B, out, bias, quant\\_state)\n\n/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py in apply(cls, \\*args, \\*\\*kwargs)\\\n 581 # See NOTE: \\[functorch vjp and autograd interaction\\]\\\n 582 args = \\_functorch.utils.unwrap\\_dead\\_wrappers(args)\\\n--> 583 return super().apply(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 584\\\n 585 if not is\\_setup\\_ctx\\_defined:\n\n/usr/local/lib/python3.12/dist-packages/bitsandbytes/autograd/\\_functions.py in forward(ctx, A, B, out, bias, quant\\_state)\\\n 313 # 1. Dequantize\\\n 314 # 2. MatmulnN\\\n--> 315 output = torch.nn.functional.linear(A, F.dequantize\\_4bit(B, quant\\_state).to(A.dtype).t(), bias)\\\n 316\\\n 317 # 3. Save state\n\n/usr/local/lib/python3.12/dist-packages/bitsandbytes/functional.py in dequantize\\_4bit(A, quant\\_state, absmax, out, blocksize, quant\\_type)\\\n 1048 )\\\n 1049 else:\\\n-> 1050 out = torch.ops.bitsandbytes.dequantize\\_4bit.default(\\\n 1051 A,\\\n 1052 absmax,\n\n/usr/local/lib/python3.12/dist-packages/torch/\\_ops.py in **call**(self, \\*args, \\*\\*kwargs)\\\n 817 # that are named \"self\". This way, all the aten ops can be called by kwargs.\\\n 818 def **call**(self, /, \\*args: \\_P.args, \\*\\*kwargs: \\_P.kwargs) -> \\_T:\\\n--> 819 return self.\\_op(\\*args, \\*\\*kwargs)\\\n 820\\\n 821 # Use positional-only argument to avoid naming collision with aten ops arguments\n\n/usr/local/lib/python3.12/dist-packages/torch/\\_compile.py in inner(\\*args, \\*\\*kwargs)\\\n 52 fn.\\_\\_dynamo\\_disable = disable\\_fn # type: ignore\\[attr-defined\\]\\\n 53\\\n---> 54 return disable\\_fn(\\*args, \\*\\*kwargs)\\\n 55\\\n 56 return inner\n\n/usr/local/lib/python3.12/dist-packages/torch/\\_dynamo/eval\\_frame.py in \\_fn(\\*args, \\*\\*kwargs)\\\n 1179 ):\\\n 1180 return fn(\\*args, \\*\\*kwargs)\\\n-> 1181 return fn(\\*args, \\*\\*kwargs)\\\n 1182 finally:\\\n 1183 set\\_eval\\_frame(None)\n\n/usr/local/lib/python3.12/dist-packages/torch/library.py in func\\_no\\_dynamo(\\*args, \\*\\*kwargs)\\\n 740 [@torch](https://huggingface.co/torch) .\\_disable\\_dynamo\\\n 741 def func\\_no\\_dynamo(\\*args, \\*\\*kwargs):\\\n--> 742 return func(\\*args, \\*\\*kwargs)\\\n 743\\\n 744 for key in keys:\n\n/usr/local/lib/python3.12/dist-packages/bitsandbytes/backends/cuda/ops.py in \\_(A, absmax, blocksize, quant\\_type, shape, dtype)\\\n 361 dtype: torch.dtype,\\\n 362 ) -> torch.Tensor:\\\n--> 363 out = torch.empty(shape, dtype=dtype, device=A.device)\\\n 364 \\_dequantize\\_4bit\\_impl(A, absmax, blocksize, quant\\_type, dtype, out=out)\\\n 365 return out\n\nOutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 MiB. GPU 0 has a total capacity of 14.56 GiB of which 17.81 MiB is free. Including non-PyTorch memory, this process has 14.54 GiB memory in use. Of the allocated memory 14.40 GiB is allocated by PyTorch, and 15.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH\\_ALLOC\\_CONF=expandable\\_segments:True to avoid fragmentation. See documentation for Memory Management ([https://pytorch.org/docs/stable/notes/cuda.html#environment-variables](https://pytorch.org/docs/stable/notes/cuda.html#environment-variables))\n",
  "authors": [],
  "captured_at": "2026-10-04T12:50:30.804479567+00:00",
  "category": "ai",
  "content_html": "<p>import torch<br />\nfrom transformers import Mistral3ForConditionalGeneration</p>\n<p>from diffusers import Flux2Pipeline, Flux2Transformer2DModel</p>\n<p>repo_id = &quot;diffusers/FLUX.2-dev-bnb-4bit&quot;<br />\ndevice = &quot;cuda:0&quot;<br />\ntorch_dtype = torch.bfloat16</p>\n<p>transformer = Flux2Transformer2DModel.from_pretrained(<br />\nrepo_id, subfolder=&quot;transformer&quot;, torch_dtype=torch_dtype, device_map=&quot;cpu&quot;<br />\n)<br />\ntext_encoder = Mistral3ForConditionalGeneration.from_pretrained(<br />\nrepo_id, subfolder=&quot;text_encoder&quot;, dtype=torch_dtype, device_map=&quot;cpu&quot;<br />\n)</p>\n<p>pipe = Flux2Pipeline.from_pretrained(<br />\nrepo_id, transformer=transformer, text_encoder=text_encoder, torch_dtype=torch_dtype<br />\n)<br />\npipe.enable_model_cpu_offload()</p>\n<p>prompt = &quot;Realistic macro photograph of a hermit crab using a soda can as its shell, partially emerging from the can, captured with sharp detail and natural colors, on a sunlit beach with soft shadows and a shallow depth of field, with blurred ocean waves in the background. The can has the text <code>BFL Diffusers</code> on it and it has a color gradient that start with #FF5733 at the top and transitions to #33FF57 at the bottom.&quot;</p>\n<p>image = pipe(<br />\nprompt=prompt,<br />\ngenerator=torch.Generator(device=device).manual_seed(42),<br />\nnum_inference_steps=50, # 28 is a good trade-off<br />\nguidance_scale=4,<br />\n).images[0]</p>\n<p>image.save(&quot;flux2_t2i_nf4.png&quot;)</p>\n<p>Flax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.<br />\nFlax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.<br />\n/usr/local/lib/python3.12/dist-packages/huggingface_hub/utils/_validators.py:206: UserWarning: The <code>local_dir_use_symlinks</code> argument is deprecated and ignored in <code>hf_hub_download</code>. Downloading to a local directory does not use symlinks anymore.<br />\nwarnings.warn(<br />\nDownload complete:  0.00/0.00 [00:00&lt;?, ?B/s]Fetching 2 files: 100% 2/2 [00:00&lt;00:00, 98.80it/s]Loading checkpoint shards: 100% 2/2 [00:01&lt;00:00,  2.03it/s]Download complete:  0.00/0.00 [00:00&lt;?, ?B/s]Fetching 4 files: 100% 4/4 [00:00&lt;00:00, 167.57it/s]Loading weights: 100% 585/585 [00:02&lt;00:00, 310.91it/s, Materializing param=model.vision_tower.transformer.layers.23.ffn_norm.weight]The tied weights mapping and config for this model specifies to tie model.language_model.embed_tokens.weight to lm_head.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with <code>tie_word_embeddings=False</code> to silence this warning<br />\nLoading pipeline components...: 100% 5/5 [00:03&lt;00:00,  1.14it/s]---------------------------------------------------------------------------<br />\nOutOfMemoryError Traceback (most recent call last)<br />\n/tmp/ipykernel_18947/863729753.py in &lt;cell line: 0&gt;()<br />\n22 prompt = &quot;Realistic macro photograph of a hermit crab using a soda can as its shell, partially emerging from the can, captured with sharp detail and natural colors, on a sunlit beach with soft shadows and a shallow depth of field, with blurred ocean waves in the background. The can has the text <code>BFL Diffusers</code> on it and it has a color gradient that start with #FF5733 at the top and transitions to #33FF57 at the bottom.&quot;<br />\n23<br />\n---&gt; 24 image = pipe(<br />\n25 prompt=prompt,<br />\n26 generator=torch.Generator(device=device).manual_seed(42),</p>\n<p>36 frames/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py in decorate_context(*args, **kwargs)<br />\n122 # pyrefly: ignore [bad-context-manager]<br />\n123 with ctx_factory():<br />\n--&gt; 124 return func(*args, **kwargs)<br />\n125<br />\n126 return decorate_context</p>\n<p>/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline_flux2.py in <strong>call</strong>(self, image, prompt, height, width, num_inference_steps, sigmas, guidance_scale, num_images_per_prompt, generator, latents, prompt_embeds, output_type, return_dict, attention_kwargs, callback_on_step_end, callback_on_step_end_tensor_inputs, max_sequence_length, text_encoder_out_layers, caption_upsample_temperature)<br />\n869 prompt, images=image, temperature=caption_upsample_temperature, device=device<br />\n870 )<br />\n--&gt; 871 prompt_embeds, text_ids = self.encode_prompt(<br />\n872 prompt=prompt,<br />\n873 prompt_embeds=prompt_embeds,</p>\n<p>/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline_flux2.py in encode_prompt(self, prompt, device, num_images_per_prompt, prompt_embeds, max_sequence_length, text_encoder_out_layers)<br />\n586<br />\n587 if prompt_embeds is None:<br />\n--&gt; 588 prompt_embeds = self._get_mistral_3_small_prompt_embeds(<br />\n589 text_encoder=self.text_encoder,<br />\n590 tokenizer=self.tokenizer,</p>\n<p>/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline_flux2.py in _get_mistral_3_small_prompt_embeds(text_encoder, tokenizer, prompt, dtype, device, max_sequence_length, system_message, hidden_states_layers)<br />\n337<br />\n338 # Forward pass through the model<br />\n--&gt; 339 output = text_encoder(<br />\n340 input_ids=input_ids,<br />\n341 attention_mask=attention_mask,</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)<br />\n1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]<br />\n1775 else:<br />\n-&gt; 1776 return self._call_impl(*args, **kwargs)<br />\n1777<br />\n1778 # torchrec tests the code consistency with the following code</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)<br />\n1785 or _global_backward_pre_hooks or _global_backward_hooks<br />\n1786 or _global_forward_hooks or _global_forward_pre_hooks):<br />\n-&gt; 1787 return forward_call(*args, **kwargs)<br />\n1788<br />\n1789 result = None</p>\n<p>/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py in new_forward(module, *args, **kwargs)<br />\n190 output = module._old_forward(*args, **kwargs)<br />\n191 else:<br />\n--&gt; 192 output = module._old_forward(*args, **kwargs)<br />\n193 return module._hf_hook.post_forward(module, output)<br />\n194</p>\n<p>/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, *args, **kwargs)<br />\n1000 outputs = func(self, *args, **kwargs)<br />\n1001 else:<br />\n-&gt; 1002 outputs = func(self, *args, **kwargs)<br />\n1003 except TypeError as original_exception:<br />\n1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.</p>\n<p>/usr/local/lib/python3.12/dist-packages/transformers/models/mistral3/modeling_mistral3.py in forward(self, input_ids, pixel_values, attention_mask, position_ids, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict, cache_position, logits_to_keep, image_sizes, **kwargs)<br />\n444 return_dict = return_dict if return_dict is not None else self.config.use_return_dict<br />\n445<br />\n--&gt; 446 outputs = self.model(<br />\n447 input_ids=input_ids,<br />\n448 pixel_values=pixel_values,</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)<br />\n1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]<br />\n1775 else:<br />\n-&gt; 1776 return self._call_impl(*args, **kwargs)<br />\n1777<br />\n1778 # torchrec tests the code consistency with the following code</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)<br />\n1785 or _global_backward_pre_hooks or _global_backward_hooks<br />\n1786 or _global_forward_hooks or _global_forward_pre_hooks):<br />\n-&gt; 1787 return forward_call(*args, **kwargs)<br />\n1788<br />\n1789 result = None</p>\n<p>/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, *args, **kwargs)<br />\n1000 outputs = func(self, *args, **kwargs)<br />\n1001 else:<br />\n-&gt; 1002 outputs = func(self, *args, **kwargs)<br />\n1003 except TypeError as original_exception:<br />\n1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.</p>\n<p>/usr/local/lib/python3.12/dist-packages/transformers/models/mistral3/modeling_mistral3.py in forward(self, input_ids, pixel_values, attention_mask, position_ids, past_key_values, inputs_embeds, vision_feature_layer, use_cache, output_attentions, output_hidden_states, return_dict, cache_position, image_sizes, **kwargs)<br />\n323 inputs_embeds = inputs_embeds.masked_scatter(special_image_mask, image_features)<br />\n324<br />\n--&gt; 325 outputs = self.language_model(<br />\n326 attention_mask=attention_mask,<br />\n327 position_ids=position_ids,</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)<br />\n1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]<br />\n1775 else:<br />\n-&gt; 1776 return self._call_impl(*args, **kwargs)<br />\n1777<br />\n1778 # torchrec tests the code consistency with the following code</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)<br />\n1785 or _global_backward_pre_hooks or _global_backward_hooks<br />\n1786 or _global_forward_hooks or _global_forward_pre_hooks):<br />\n-&gt; 1787 return forward_call(*args, **kwargs)<br />\n1788<br />\n1789 result = None</p>\n<p>/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, *args, **kwargs)<br />\n1000 outputs = func(self, *args, **kwargs)<br />\n1001 else:<br />\n-&gt; 1002 outputs = func(self, *args, **kwargs)<br />\n1003 except TypeError as original_exception:<br />\n1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.</p>\n<p>/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling_mistral.py in forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, use_cache, cache_position, **kwargs)<br />\n395<br />\n396 for decoder_layer in self.layers[: self.config.num_hidden_layers]:<br />\n--&gt; 397 hidden_states = decoder_layer(<br />\n398 hidden_states,<br />\n399 attention_mask=causal_mask,</p>\n<p>/usr/local/lib/python3.12/dist-packages/transformers/modeling_layers.py in <strong>call</strong>(self, *args, **kwargs)<br />\n91<br />\n92 return self._gradient_checkpointing_func(partial(super().<strong>call</strong>, **kwargs), *args)<br />\n---&gt; 93 return super().<strong>call</strong>(*args, **kwargs)<br />\n94<br />\n95</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)<br />\n1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]<br />\n1775 else:<br />\n-&gt; 1776 return self._call_impl(*args, **kwargs)<br />\n1777<br />\n1778 # torchrec tests the code consistency with the following code</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)<br />\n1785 or _global_backward_pre_hooks or _global_backward_hooks<br />\n1786 or _global_forward_hooks or _global_forward_pre_hooks):<br />\n-&gt; 1787 return forward_call(*args, **kwargs)<br />\n1788<br />\n1789 result = None</p>\n<p>/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapped_forward(*args, **kwargs)<br />\n953 if key == &quot;hidden_states&quot; and len(collected_outputs[key]) == 0:<br />\n954 collected_outputs[key] += (args[0],)<br />\n--&gt; 955 output = orig_forward(*args, **kwargs)<br />\n956 if not isinstance(output, tuple):<br />\n957 collected_outputs[key] += (output,)</p>\n<p>/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling_mistral.py in forward(self, hidden_states, attention_mask, position_ids, past_key_values, use_cache, cache_position, position_embeddings, **kwargs)<br />\n228 hidden_states = self.input_layernorm(hidden_states)<br />\n229 # Self Attention<br />\n--&gt; 230 hidden_states, _ = self.self_attn(<br />\n231 hidden_states=hidden_states,<br />\n232 attention_mask=attention_mask,</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)<br />\n1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]<br />\n1775 else:<br />\n-&gt; 1776 return self._call_impl(*args, **kwargs)<br />\n1777<br />\n1778 # torchrec tests the code consistency with the following code</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)<br />\n1785 or _global_backward_pre_hooks or _global_backward_hooks<br />\n1786 or _global_forward_hooks or _global_forward_pre_hooks):<br />\n-&gt; 1787 return forward_call(*args, **kwargs)<br />\n1788<br />\n1789 result = None</p>\n<p>/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling_mistral.py in forward(self, hidden_states, position_embeddings, attention_mask, past_key_values, cache_position, **kwargs)<br />\n151 hidden_shape = (*input_shape, -1, self.head_dim)<br />\n152<br />\n--&gt; 153 query_states = self.q_proj(hidden_states).view(hidden_shape).transpose(1, 2)<br />\n154 key_states = self.k_proj(hidden_states).view(hidden_shape).transpose(1, 2)<br />\n155 value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _wrapped_call_impl(self, *args, **kwargs)<br />\n1774 return self._compiled_call_impl(*args, **kwargs) # type: ignore[misc]<br />\n1775 else:<br />\n-&gt; 1776 return self._call_impl(*args, **kwargs)<br />\n1777<br />\n1778 # torchrec tests the code consistency with the following code</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in _call_impl(self, *args, **kwargs)<br />\n1785 or _global_backward_pre_hooks or _global_backward_hooks<br />\n1786 or _global_forward_hooks or _global_forward_pre_hooks):<br />\n-&gt; 1787 return forward_call(*args, **kwargs)<br />\n1788<br />\n1789 result = None</p>\n<p>/usr/local/lib/python3.12/dist-packages/bitsandbytes/nn/modules.py in forward(self, x)<br />\n554 weight = self.weight if getattr(quant_state, &quot;packing_format_for_cpu&quot;, False) else self.weight.t()<br />\n555<br />\n--&gt; 556 return bnb.matmul_4bit(x, weight, bias=bias, quant_state=quant_state).to(inp_dtype)<br />\n557<br />\n558</p>\n<p>/usr/local/lib/python3.12/dist-packages/bitsandbytes/autograd/_functions.py in matmul_4bit(A, B, quant_state, out, bias)<br />\n399 return out<br />\n400 else:<br />\n--&gt; 401 return MatMul4Bit.apply(A, B, out, bias, quant_state)</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py in apply(cls, *args, **kwargs)<br />\n581 # See NOTE: [functorch vjp and autograd interaction]<br />\n582 args = _functorch.utils.unwrap_dead_wrappers(args)<br />\n--&gt; 583 return super().apply(*args, **kwargs) # type: ignore[misc]<br />\n584<br />\n585 if not is_setup_ctx_defined:</p>\n<p>/usr/local/lib/python3.12/dist-packages/bitsandbytes/autograd/_functions.py in forward(ctx, A, B, out, bias, quant_state)<br />\n313 # 1. Dequantize<br />\n314 # 2. MatmulnN<br />\n--&gt; 315 output = torch.nn.functional.linear(A, F.dequantize_4bit(B, quant_state).to(A.dtype).t(), bias)<br />\n316<br />\n317 # 3. Save state</p>\n<p>/usr/local/lib/python3.12/dist-packages/bitsandbytes/functional.py in dequantize_4bit(A, quant_state, absmax, out, blocksize, quant_type)<br />\n1048 )<br />\n1049 else:<br />\n-&gt; 1050 out = torch.ops.bitsandbytes.dequantize_4bit.default(<br />\n1051 A,<br />\n1052 absmax,</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/_ops.py in <strong>call</strong>(self, *args, **kwargs)<br />\n817 # that are named &quot;self&quot;. This way, all the aten ops can be called by kwargs.<br />\n818 def <strong>call</strong>(self, /, *args: _P.args, **kwargs: _P.kwargs) -&gt; _T:<br />\n--&gt; 819 return self._op(*args, **kwargs)<br />\n820<br />\n821 # Use positional-only argument to avoid naming collision with aten ops arguments</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/_compile.py in inner(*args, **kwargs)<br />\n52 fn.__dynamo_disable = disable_fn # type: ignore[attr-defined]<br />\n53<br />\n---&gt; 54 return disable_fn(*args, **kwargs)<br />\n55<br />\n56 return inner</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py in _fn(*args, **kwargs)<br />\n1179 ):<br />\n1180 return fn(*args, **kwargs)<br />\n-&gt; 1181 return fn(*args, **kwargs)<br />\n1182 finally:<br />\n1183 set_eval_frame(None)</p>\n<p>/usr/local/lib/python3.12/dist-packages/torch/library.py in func_no_dynamo(*args, **kwargs)<br />\n740 <a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://huggingface.co/torch\">@torch</a> ._disable_dynamo<br />\n741 def func_no_dynamo(*args, **kwargs):<br />\n--&gt; 742 return func(*args, **kwargs)<br />\n743<br />\n744 for key in keys:</p>\n<p>/usr/local/lib/python3.12/dist-packages/bitsandbytes/backends/cuda/ops.py in _(A, absmax, blocksize, quant_type, shape, dtype)<br />\n361 dtype: torch.dtype,<br />\n362 ) -&gt; torch.Tensor:<br />\n--&gt; 363 out = torch.empty(shape, dtype=dtype, device=A.device)<br />\n364 _dequantize_4bit_impl(A, absmax, blocksize, quant_type, dtype, out=out)<br />\n365 return out</p>\n<p>OutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 MiB. GPU 0 has a total capacity of 14.56 GiB of which 17.81 MiB is free. Including non-PyTorch memory, this process has 14.54 GiB memory in use. Of the allocated memory 14.40 GiB is allocated by PyTorch, and 15.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://pytorch.org/docs/stable/notes/cuda.html#environment-variables\">https://pytorch.org/docs/stable/notes/cuda.html#environment-variables</a>)</p>\n",
  "content_markdown": "import torch\\\nfrom transformers import Mistral3ForConditionalGeneration\n\nfrom diffusers import Flux2Pipeline, Flux2Transformer2DModel\n\nrepo\\_id = \"diffusers/FLUX.2-dev-bnb-4bit\"\\\ndevice = \"cuda:0\"\\\ntorch\\_dtype = torch.bfloat16\n\ntransformer = Flux2Transformer2DModel.from\\_pretrained(\\\n repo\\_id, subfolder=\"transformer\", torch\\_dtype=torch\\_dtype, device\\_map=\"cpu\"\\\n)\\\ntext\\_encoder = Mistral3ForConditionalGeneration.from\\_pretrained(\\\n repo\\_id, subfolder=\"text\\_encoder\", dtype=torch\\_dtype, device\\_map=\"cpu\"\\\n)\n\npipe = Flux2Pipeline.from\\_pretrained(\\\n repo\\_id, transformer=transformer, text\\_encoder=text\\_encoder, torch\\_dtype=torch\\_dtype\\\n)\\\npipe.enable\\_model\\_cpu\\_offload()\n\nprompt = \"Realistic macro photograph of a hermit crab using a soda can as its shell, partially emerging from the can, captured with sharp detail and natural colors, on a sunlit beach with soft shadows and a shallow depth of field, with blurred ocean waves in the background. The can has the text `BFL Diffusers` on it and it has a color gradient that start with #FF5733 at the top and transitions to #33FF57 at the bottom.\"\n\nimage = pipe(\\\n prompt=prompt,\\\n generator=torch.Generator(device=device).manual\\_seed(42),\\\n num\\_inference\\_steps=50, # 28 is a good trade-off\\\n guidance\\_scale=4,\\\n).images\\[0\\]\n\nimage.save(\"flux2\\_t2i\\_nf4.png\")\n\nFlax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.\\\nFlax classes are deprecated and will be removed in Diffusers v1.0.0. We recommend migrating to PyTorch classes or pinning your version of Diffusers.\\\n/usr/local/lib/python3.12/dist-packages/huggingface\\_hub/utils/\\_validators.py:206: UserWarning: The `local_dir_use_symlinks` argument is deprecated and ignored in `hf_hub_download`. Downloading to a local directory does not use symlinks anymore.\\\n warnings.warn(\\\nDownload complete:  0.00/0.00 \\[00:00\\<?, ?B/s\\]Fetching 2 files: 100% 2/2 \\[00:00<00:00, 98.80it/s\\]Loading checkpoint shards: 100% 2/2 \\[00:01<00:00,  2.03it/s\\]Download complete:  0.00/0.00 \\[00:00\\<?, ?B/s\\]Fetching 4 files: 100% 4/4 \\[00:00<00:00, 167.57it/s\\]Loading weights: 100% 585/585 \\[00:02<00:00, 310.91it/s, Materializing param=model.vision\\_tower.transformer.layers.23.ffn\\_norm.weight\\]The tied weights mapping and config for this model specifies to tie model.language\\_model.embed\\_tokens.weight to lm\\_head.weight, but both are present in the checkpoints, so we will NOT tie them. You should update the config with `tie_word_embeddings=False` to silence this warning\\\nLoading pipeline components...: 100% 5/5 \\[00:03<00:00,  1.14it/s\\]---------------------------------------------------------------------------\\\nOutOfMemoryError Traceback (most recent call last)\\\n/tmp/ipykernel\\_18947/863729753.py in \\<cell line: 0>()\\\n 22 prompt = \"Realistic macro photograph of a hermit crab using a soda can as its shell, partially emerging from the can, captured with sharp detail and natural colors, on a sunlit beach with soft shadows and a shallow depth of field, with blurred ocean waves in the background. The can has the text `BFL Diffusers` on it and it has a color gradient that start with #FF5733 at the top and transitions to #33FF57 at the bottom.\"\\\n 23\\\n---> 24 image = pipe(\\\n 25 prompt=prompt,\\\n 26 generator=torch.Generator(device=device).manual\\_seed(42),\n\n36 frames/usr/local/lib/python3.12/dist-packages/torch/utils/\\_contextlib.py in decorate\\_context(\\*args, \\*\\*kwargs)\\\n 122 # pyrefly: ignore \\[bad-context-manager\\]\\\n 123 with ctx\\_factory():\\\n--> 124 return func(\\*args, \\*\\*kwargs)\\\n 125\\\n 126 return decorate\\_context\n\n/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline\\_flux2.py in **call**(self, image, prompt, height, width, num\\_inference\\_steps, sigmas, guidance\\_scale, num\\_images\\_per\\_prompt, generator, latents, prompt\\_embeds, output\\_type, return\\_dict, attention\\_kwargs, callback\\_on\\_step\\_end, callback\\_on\\_step\\_end\\_tensor\\_inputs, max\\_sequence\\_length, text\\_encoder\\_out\\_layers, caption\\_upsample\\_temperature)\\\n 869 prompt, images=image, temperature=caption\\_upsample\\_temperature, device=device\\\n 870 )\\\n--> 871 prompt\\_embeds, text\\_ids = self.encode\\_prompt(\\\n 872 prompt=prompt,\\\n 873 prompt\\_embeds=prompt\\_embeds,\n\n/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline\\_flux2.py in encode\\_prompt(self, prompt, device, num\\_images\\_per\\_prompt, prompt\\_embeds, max\\_sequence\\_length, text\\_encoder\\_out\\_layers)\\\n 586\\\n 587 if prompt\\_embeds is None:\\\n--> 588 prompt\\_embeds = self.\\_get\\_mistral\\_3\\_small\\_prompt\\_embeds(\\\n 589 text\\_encoder=self.text\\_encoder,\\\n 590 tokenizer=self.tokenizer,\n\n/usr/local/lib/python3.12/dist-packages/diffusers/pipelines/flux2/pipeline\\_flux2.py in \\_get\\_mistral\\_3\\_small\\_prompt\\_embeds(text\\_encoder, tokenizer, prompt, dtype, device, max\\_sequence\\_length, system\\_message, hidden\\_states\\_layers)\\\n 337\\\n 338 # Forward pass through the model\\\n--> 339 output = text\\_encoder(\\\n 340 input\\_ids=input\\_ids,\\\n 341 attention\\_mask=attention\\_mask,\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/accelerate/hooks.py in new\\_forward(module, \\*args, \\*\\*kwargs)\\\n 190 output = module.\\_old\\_forward(\\*args, \\*\\*kwargs)\\\n 191 else:\\\n--> 192 output = module.\\_old\\_forward(\\*args, \\*\\*kwargs)\\\n 193 return module.\\_hf\\_hook.post\\_forward(module, output)\\\n 194\n\n/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, \\*args, \\*\\*kwargs)\\\n 1000 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1001 else:\\\n-> 1002 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1003 except TypeError as original\\_exception:\\\n 1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.\n\n/usr/local/lib/python3.12/dist-packages/transformers/models/mistral3/modeling\\_mistral3.py in forward(self, input\\_ids, pixel\\_values, attention\\_mask, position\\_ids, past\\_key\\_values, inputs\\_embeds, labels, use\\_cache, output\\_attentions, output\\_hidden\\_states, return\\_dict, cache\\_position, logits\\_to\\_keep, image\\_sizes, \\*\\*kwargs)\\\n 444 return\\_dict = return\\_dict if return\\_dict is not None else self.config.use\\_return\\_dict\\\n 445\\\n--> 446 outputs = self.model(\\\n 447 input\\_ids=input\\_ids,\\\n 448 pixel\\_values=pixel\\_values,\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, \\*args, \\*\\*kwargs)\\\n 1000 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1001 else:\\\n-> 1002 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1003 except TypeError as original\\_exception:\\\n 1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.\n\n/usr/local/lib/python3.12/dist-packages/transformers/models/mistral3/modeling\\_mistral3.py in forward(self, input\\_ids, pixel\\_values, attention\\_mask, position\\_ids, past\\_key\\_values, inputs\\_embeds, vision\\_feature\\_layer, use\\_cache, output\\_attentions, output\\_hidden\\_states, return\\_dict, cache\\_position, image\\_sizes, \\*\\*kwargs)\\\n 323 inputs\\_embeds = inputs\\_embeds.masked\\_scatter(special\\_image\\_mask, image\\_features)\\\n 324\\\n--> 325 outputs = self.language\\_model(\\\n 326 attention\\_mask=attention\\_mask,\\\n 327 position\\_ids=position\\_ids,\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapper(self, \\*args, \\*\\*kwargs)\\\n 1000 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1001 else:\\\n-> 1002 outputs = func(self, \\*args, \\*\\*kwargs)\\\n 1003 except TypeError as original\\_exception:\\\n 1004 # If we get a TypeError, it's possible that the model is not receiving the recordable kwargs correctly.\n\n/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling\\_mistral.py in forward(self, input\\_ids, attention\\_mask, position\\_ids, past\\_key\\_values, inputs\\_embeds, use\\_cache, cache\\_position, \\*\\*kwargs)\\\n 395\\\n 396 for decoder\\_layer in self.layers\\[: self.config.num\\_hidden\\_layers\\]:\\\n--> 397 hidden\\_states = decoder\\_layer(\\\n 398 hidden\\_states,\\\n 399 attention\\_mask=causal\\_mask,\n\n/usr/local/lib/python3.12/dist-packages/transformers/modeling\\_layers.py in **call**(self, \\*args, \\*\\*kwargs)\\\n 91\\\n 92 return self.\\_gradient\\_checkpointing\\_func(partial(super().**call**, \\*\\*kwargs), \\*args)\\\n---> 93 return super().**call**(\\*args, \\*\\*kwargs)\\\n 94\\\n 95\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/transformers/utils/generic.py in wrapped\\_forward(\\*args, \\*\\*kwargs)\\\n 953 if key == \"hidden\\_states\" and len(collected\\_outputs\\[key\\]) == 0:\\\n 954 collected\\_outputs\\[key\\] += (args\\[0\\],)\\\n--> 955 output = orig\\_forward(\\*args, \\*\\*kwargs)\\\n 956 if not isinstance(output, tuple):\\\n 957 collected\\_outputs\\[key\\] += (output,)\n\n/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling\\_mistral.py in forward(self, hidden\\_states, attention\\_mask, position\\_ids, past\\_key\\_values, use\\_cache, cache\\_position, position\\_embeddings, \\*\\*kwargs)\\\n 228 hidden\\_states = self.input\\_layernorm(hidden\\_states)\\\n 229 # Self Attention\\\n--> 230 hidden\\_states, \\_ = self.self\\_attn(\\\n 231 hidden\\_states=hidden\\_states,\\\n 232 attention\\_mask=attention\\_mask,\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/transformers/models/mistral/modeling\\_mistral.py in forward(self, hidden\\_states, position\\_embeddings, attention\\_mask, past\\_key\\_values, cache\\_position, \\*\\*kwargs)\\\n 151 hidden\\_shape = (\\*input\\_shape, -1, self.head\\_dim)\\\n 152\\\n--> 153 query\\_states = self.q\\_proj(hidden\\_states).view(hidden\\_shape).transpose(1, 2)\\\n 154 key\\_states = self.k\\_proj(hidden\\_states).view(hidden\\_shape).transpose(1, 2)\\\n 155 value\\_states = self.v\\_proj(hidden\\_states).view(hidden\\_shape).transpose(1, 2)\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_wrapped\\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1774 return self.\\_compiled\\_call\\_impl(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 1775 else:\\\n-> 1776 return self.\\_call\\_impl(\\*args, \\*\\*kwargs)\\\n 1777\\\n 1778 # torchrec tests the code consistency with the following code\n\n/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py in \\_call\\_impl(self, \\*args, \\*\\*kwargs)\\\n 1785 or \\_global\\_backward\\_pre\\_hooks or \\_global\\_backward\\_hooks\\\n 1786 or \\_global\\_forward\\_hooks or \\_global\\_forward\\_pre\\_hooks):\\\n-> 1787 return forward\\_call(\\*args, \\*\\*kwargs)\\\n 1788\\\n 1789 result = None\n\n/usr/local/lib/python3.12/dist-packages/bitsandbytes/nn/modules.py in forward(self, x)\\\n 554 weight = self.weight if getattr(quant\\_state, \"packing\\_format\\_for\\_cpu\", False) else self.weight.t()\\\n 555\\\n--> 556 return bnb.matmul\\_4bit(x, weight, bias=bias, quant\\_state=quant\\_state).to(inp\\_dtype)\\\n 557\\\n 558\n\n/usr/local/lib/python3.12/dist-packages/bitsandbytes/autograd/\\_functions.py in matmul\\_4bit(A, B, quant\\_state, out, bias)\\\n 399 return out\\\n 400 else:\\\n--> 401 return MatMul4Bit.apply(A, B, out, bias, quant\\_state)\n\n/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py in apply(cls, \\*args, \\*\\*kwargs)\\\n 581 # See NOTE: \\[functorch vjp and autograd interaction\\]\\\n 582 args = \\_functorch.utils.unwrap\\_dead\\_wrappers(args)\\\n--> 583 return super().apply(\\*args, \\*\\*kwargs) # type: ignore\\[misc\\]\\\n 584\\\n 585 if not is\\_setup\\_ctx\\_defined:\n\n/usr/local/lib/python3.12/dist-packages/bitsandbytes/autograd/\\_functions.py in forward(ctx, A, B, out, bias, quant\\_state)\\\n 313 # 1. Dequantize\\\n 314 # 2. MatmulnN\\\n--> 315 output = torch.nn.functional.linear(A, F.dequantize\\_4bit(B, quant\\_state).to(A.dtype).t(), bias)\\\n 316\\\n 317 # 3. Save state\n\n/usr/local/lib/python3.12/dist-packages/bitsandbytes/functional.py in dequantize\\_4bit(A, quant\\_state, absmax, out, blocksize, quant\\_type)\\\n 1048 )\\\n 1049 else:\\\n-> 1050 out = torch.ops.bitsandbytes.dequantize\\_4bit.default(\\\n 1051 A,\\\n 1052 absmax,\n\n/usr/local/lib/python3.12/dist-packages/torch/\\_ops.py in **call**(self, \\*args, \\*\\*kwargs)\\\n 817 # that are named \"self\". This way, all the aten ops can be called by kwargs.\\\n 818 def **call**(self, /, \\*args: \\_P.args, \\*\\*kwargs: \\_P.kwargs) -> \\_T:\\\n--> 819 return self.\\_op(\\*args, \\*\\*kwargs)\\\n 820\\\n 821 # Use positional-only argument to avoid naming collision with aten ops arguments\n\n/usr/local/lib/python3.12/dist-packages/torch/\\_compile.py in inner(\\*args, \\*\\*kwargs)\\\n 52 fn.\\_\\_dynamo\\_disable = disable\\_fn # type: ignore\\[attr-defined\\]\\\n 53\\\n---> 54 return disable\\_fn(\\*args, \\*\\*kwargs)\\\n 55\\\n 56 return inner\n\n/usr/local/lib/python3.12/dist-packages/torch/\\_dynamo/eval\\_frame.py in \\_fn(\\*args, \\*\\*kwargs)\\\n 1179 ):\\\n 1180 return fn(\\*args, \\*\\*kwargs)\\\n-> 1181 return fn(\\*args, \\*\\*kwargs)\\\n 1182 finally:\\\n 1183 set\\_eval\\_frame(None)\n\n/usr/local/lib/python3.12/dist-packages/torch/library.py in func\\_no\\_dynamo(\\*args, \\*\\*kwargs)\\\n 740 [@torch](https://huggingface.co/torch) .\\_disable\\_dynamo\\\n 741 def func\\_no\\_dynamo(\\*args, \\*\\*kwargs):\\\n--> 742 return func(\\*args, \\*\\*kwargs)\\\n 743\\\n 744 for key in keys:\n\n/usr/local/lib/python3.12/dist-packages/bitsandbytes/backends/cuda/ops.py in \\_(A, absmax, blocksize, quant\\_type, shape, dtype)\\\n 361 dtype: torch.dtype,\\\n 362 ) -> torch.Tensor:\\\n--> 363 out = torch.empty(shape, dtype=dtype, device=A.device)\\\n 364 \\_dequantize\\_4bit\\_impl(A, absmax, blocksize, quant\\_type, dtype, out=out)\\\n 365 return out\n\nOutOfMemoryError: CUDA out of memory. Tried to allocate 40.00 MiB. GPU 0 has a total capacity of 14.56 GiB of which 17.81 MiB is free. Including non-PyTorch memory, this process has 14.54 GiB memory in use. Of the allocated memory 14.40 GiB is allocated by PyTorch, and 15.19 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH\\_ALLOC\\_CONF=expandable\\_segments:True to avoid fragmentation. See documentation for Memory Management ([https://pytorch.org/docs/stable/notes/cuda.html#environment-variables](https://pytorch.org/docs/stable/notes/cuda.html#environment-variables))\n",
  "dateCreated": "2026-10-04T12:50:30.804479567+00:00",
  "date_created": "2025-11-25T00:00:00+00:00",
  "date_modified": "2025-11-25T00:00:00+00:00",
  "date_published": "2025-11-25T00:00:00+00:00",
  "description": "import torch from transformers import Mistral3ForConditionalGeneration from diffusers import Flux2Pipeline, Flux2Transformer2DModel repo_id = \"diffusers/FLUX.2-dev-bnb-4bit\" device = \"cuda:0\" torch_dtype = torch.bfloat16 transformer =…",
  "external_url": "https://huggingface.co/blog/flux-2",
  "git": {
    "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e",
    "path": "items/huggingface-co/2025/11/2025-11-25-diffusers-welcomes-flux-2.md",
    "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2025/11/2025-11-25-diffusers-welcomes-flux-2.md"
  },
  "id": "urn:aggr:item:b5e7d12d4a8d374fa5cebf4ed5c39410d074eea0",
  "isBasedOn": {
    "@id": "https://huggingface.co/blog/flux-2"
  },
  "mainEntity": {
    "@id": "https://huggingface.co/blog/flux-2",
    "@type": "CreativeWork",
    "archivedAt": {
      "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-11-25-diffusers-welcomes-flux-2/#webpage"
    },
    "dateModified": "2025-11-25T00:00:00+00:00",
    "datePublished": "2025-11-25T00:00:00+00:00",
    "headline": "Diffusers welcomes FLUX-2",
    "timeRequired": "PT9M",
    "url": "https://huggingface.co/blog/flux-2",
    "wordCount": 1995
  },
  "name": "Archived snapshot: Diffusers welcomes FLUX-2",
  "preview": null,
  "reading_minutes": 9,
  "replicated_at": null,
  "source": "huggingface-co",
  "source_name": "Hugging Face - Blog",
  "summary": null,
  "tags": [],
  "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00",
  "title": "Diffusers welcomes FLUX-2",
  "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-11-25-diffusers-welcomes-flux-2/",
  "word_count": 1995
}