{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-09-29-zml-llmd-20260929-0/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Article\n\nDeepSeek-V4.1-Flash lands on NVIDIA and AMD.\n\nSeptember 29, 2026 inference / serving / NVIDIA / AMD / DeepSeek\n\nWe’re happy to announce the release of `zml/llmd 20260929.0`, which notably brings [deepseek-ai/DeepSeek-V4.1-Flash](https://hf.co/deepseek-ai/DeepSeek-V4.1-Flash) support on NVIDIA and AMD GPUs.\n\n## Models\n\nThis release adds 2 model families:\n\n- [deepseek-ai/DeepSeek-V4.1-Flash](https://hf.co/deepseek-ai/DeepSeek-V4.1-Flash) (with native DSpark support)\n- [meta-models/Muse-Glimmer-30B](https://hf.co/meta-models/Muse-Glimmer-30B)\n\nAs stated, DeepSeek is only available on platforms with enough VRAM to host it natively. We do not plan to support custom quants in the near future.\n\n## DeepSeek-V4.1-Flash\n\nObviously the highlight, this is our first release supporting a frontier model. ZML is now mature enough to support complex frontier models, and we wanted to bring it to you. Bear in mind this is a 763B model, weighing 510GB, so for now we only support it on platforms LLMD supports with enough VRAM, namely NVIDIA and AMD GPUs.\n\nThat said, this is a cool release because it pushes the limits and shows how powerful ZML can be: MXFP4 MoEs, block-FP8, DSpark speculative decoding, DSA, HCA, and more. Nothing was spared. And it all works.\n\nSadly, we couldn’t bring it to TPUs because Google told us there were no TPUv7s available “for at least a few weeks”. LLMD doesn’t support Trainium yet due to an AWS Neuron bug that has since been fixed. We’ll revisit that.\n\nOn performance, we’re about 20% slower than vLLM and 30% slower than SGLang on 4xGB300. We could’ve waited to close the gap, but we wanted to bring it to you now. The roadmap is clear and we’ll get there, so why wait?\n\nToken Generation, deepseek-ai/DeepSeek-V4.1-Flash, bs=1, 4xGB300 (tok/s)\n\nSpeculative decoding is also supported on DeepSeek-V4.1-Flash, and it works well. It can be enabled with the `--speculative-drafter-model=native` flag.\n\n## Cold start\n\nWe’re pretty happy about our cold-start performance. We already have improvements that are not in this release that we can’t wait to share with you soon.\n\nCold Start, deepseek-ai/DeepSeek-V4.1-Flash, 4xGB300\n\n## Bringing frontier IR support to ZML\n\nIt was always possible in [zml/zml](https://github.com/zml/zml) to emit custom kernels as MLIR bytecode using the built-in codegen builder. This allows to leverage ZML’s memory allocation, stream management, concurrency and sharding while going off the trail if needed. We used that feature to emit [Triton](https://github.com/zml/zml/tree/master/kernels/triton) and [Mosaic (Pallas)](https://github.com/zml/zml/tree/master/kernels/mosaic_tpu) kernels.\n\nSince the last release, we have added support for [CuTe IR](https://github.com/zml/zml/tree/master/kernels/cute), [CUDA Tile IR](https://github.com/zml/zml/tree/master/kernels/cuda_tile), and [Fly IR](https://github.com/zml/zml/tree/master/kernels/fly).\n\nWhy does this matter? Well, because Triton cannot get the full performance of the underlying hardware. That’s why there’s [Gluon](https://github.com/triton-lang/triton/blob/main/python/tutorials/gluon/01-intro.py) now (which we’ll add support for soon).\n\nOn NVIDIA, CUDA Tile is faster than Triton, and CuTe is as fast as raw C++ CUDA (while still being a JIT). On AMD, Fly IR is faster than AMD’s preferred Triton path.\n\nUsing these new backends, we replaced the few custom kernels we had (MoE and PagedAttention) using their fastest respective IR. For instance, on NVIDIA, CuTe IR is as fast as C++ CUDA. So here goes.\n\nAs always with us, this was achieved with zero Python. No compromise.\n\nMetal gains MXFP8, MXFP4 and NVFP4 block scaled dot products and block-FP8 MoE paths.\n\nThat means you can run NVFP4 models on Apple Silicon:\n\n```\n$ llmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1\n```\n\nThe performance is nice, too:\n\nApple M3 Max, nvidia/Qwen3.8-27B-NVFP4, 1024/128, bs=1 (tg tok/s)\n\n## Various improvements\n\n## AMD\n\nThis release ships with ROCm 10.\n\n## Intel\n\nThis release brings fixes for four GPU configurations.\n\n## Serving diagnostics\n\nPrometheus metrics (`/metrics`) now cover scheduling, request outcomes, prompt-token counts, and speculative-decoding efficiency.\n\n`ignore_eos` is also now supported for benchmarking.\n\n## Try it today\n\n**NVIDIA**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n```\n\n**AMD**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n```\n\n**Intel**\n\n```bash\ndocker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Google TPU**\n\n```bash\ndocker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Apple Metal**\n\n```bash\nbrew install zml/zml/llmd\nllmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1\n```\n", "authors": [], "captured_at": "2026-09-29T23:14:03.129222144+00:00", "category": "ai", "content_html": "

Article

\n

DeepSeek-V4.1-Flash lands on NVIDIA and AMD.

\n

September 29, 2026 inference / serving / NVIDIA / AMD / DeepSeek

\n

We’re happy to announce the release of zml/llmd 20260929.0, which notably brings deepseek-ai/DeepSeek-V4.1-Flash support on NVIDIA and AMD GPUs.

\n

Models

\n

This release adds 2 model families:

\n\n

As stated, DeepSeek is only available on platforms with enough VRAM to host it natively. We do not plan to support custom quants in the near future.

\n

DeepSeek-V4.1-Flash

\n

Obviously the highlight, this is our first release supporting a frontier model. ZML is now mature enough to support complex frontier models, and we wanted to bring it to you. Bear in mind this is a 763B model, weighing 510GB, so for now we only support it on platforms LLMD supports with enough VRAM, namely NVIDIA and AMD GPUs.

\n

That said, this is a cool release because it pushes the limits and shows how powerful ZML can be: MXFP4 MoEs, block-FP8, DSpark speculative decoding, DSA, HCA, and more. Nothing was spared. And it all works.

\n

Sadly, we couldn’t bring it to TPUs because Google told us there were no TPUv7s available “for at least a few weeks”. LLMD doesn’t support Trainium yet due to an AWS Neuron bug that has since been fixed. We’ll revisit that.

\n

On performance, we’re about 20% slower than vLLM and 30% slower than SGLang on 4xGB300. We could’ve waited to close the gap, but we wanted to bring it to you now. The roadmap is clear and we’ll get there, so why wait?

\n

Token Generation, deepseek-ai/DeepSeek-V4.1-Flash, bs=1, 4xGB300 (tok/s)

\n

Speculative decoding is also supported on DeepSeek-V4.1-Flash, and it works well. It can be enabled with the --speculative-drafter-model=native flag.

\n

Cold start

\n

We’re pretty happy about our cold-start performance. We already have improvements that are not in this release that we can’t wait to share with you soon.

\n

Cold Start, deepseek-ai/DeepSeek-V4.1-Flash, 4xGB300

\n

Bringing frontier IR support to ZML

\n

It was always possible in zml/zml to emit custom kernels as MLIR bytecode using the built-in codegen builder. This allows to leverage ZML’s memory allocation, stream management, concurrency and sharding while going off the trail if needed. We used that feature to emit Triton and Mosaic (Pallas) kernels.

\n

Since the last release, we have added support for CuTe IR, CUDA Tile IR, and Fly IR.

\n

Why does this matter? Well, because Triton cannot get the full performance of the underlying hardware. That’s why there’s Gluon now (which we’ll add support for soon).

\n

On NVIDIA, CUDA Tile is faster than Triton, and CuTe is as fast as raw C++ CUDA (while still being a JIT). On AMD, Fly IR is faster than AMD’s preferred Triton path.

\n

Using these new backends, we replaced the few custom kernels we had (MoE and PagedAttention) using their fastest respective IR. For instance, on NVIDIA, CuTe IR is as fast as C++ CUDA. So here goes.

\n

As always with us, this was achieved with zero Python. No compromise.

\n

Metal gains MXFP8, MXFP4 and NVFP4 block scaled dot products and block-FP8 MoE paths.

\n

That means you can run NVFP4 models on Apple Silicon:

\n
$ llmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1\n
\n

The performance is nice, too:

\n

Apple M3 Max, nvidia/Qwen3.8-27B-NVFP4, 1024/128, bs=1 (tg tok/s)

\n

Various improvements

\n

AMD

\n

This release ships with ROCm 10.

\n

Intel

\n

This release brings fixes for four GPU configurations.

\n

Serving diagnostics

\n

Prometheus metrics (/metrics) now cover scheduling, request outcomes, prompt-token counts, and speculative-decoding efficiency.

\n

ignore_eos is also now supported for benchmarking.

\n

Try it today

\n

NVIDIA

\n
docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n    --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n
\n

AMD

\n
docker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n    --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n
\n

Intel

\n
docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n    --model=hf://Qwen/Qwen3.8-27B\n
\n

Google TPU

\n
docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n    --model=hf://Qwen/Qwen3.8-27B\n
\n

Apple Metal

\n
brew install zml/zml/llmd\nllmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1\n
\n", "content_markdown": "Article\n\nDeepSeek-V4.1-Flash lands on NVIDIA and AMD.\n\nSeptember 29, 2026 inference / serving / NVIDIA / AMD / DeepSeek\n\nWe’re happy to announce the release of `zml/llmd 20260929.0`, which notably brings [deepseek-ai/DeepSeek-V4.1-Flash](https://hf.co/deepseek-ai/DeepSeek-V4.1-Flash) support on NVIDIA and AMD GPUs.\n\n## Models\n\nThis release adds 2 model families:\n\n- [deepseek-ai/DeepSeek-V4.1-Flash](https://hf.co/deepseek-ai/DeepSeek-V4.1-Flash) (with native DSpark support)\n- [meta-models/Muse-Glimmer-30B](https://hf.co/meta-models/Muse-Glimmer-30B)\n\nAs stated, DeepSeek is only available on platforms with enough VRAM to host it natively. We do not plan to support custom quants in the near future.\n\n## DeepSeek-V4.1-Flash\n\nObviously the highlight, this is our first release supporting a frontier model. ZML is now mature enough to support complex frontier models, and we wanted to bring it to you. Bear in mind this is a 763B model, weighing 510GB, so for now we only support it on platforms LLMD supports with enough VRAM, namely NVIDIA and AMD GPUs.\n\nThat said, this is a cool release because it pushes the limits and shows how powerful ZML can be: MXFP4 MoEs, block-FP8, DSpark speculative decoding, DSA, HCA, and more. Nothing was spared. And it all works.\n\nSadly, we couldn’t bring it to TPUs because Google told us there were no TPUv7s available “for at least a few weeks”. LLMD doesn’t support Trainium yet due to an AWS Neuron bug that has since been fixed. We’ll revisit that.\n\nOn performance, we’re about 20% slower than vLLM and 30% slower than SGLang on 4xGB300. We could’ve waited to close the gap, but we wanted to bring it to you now. The roadmap is clear and we’ll get there, so why wait?\n\nToken Generation, deepseek-ai/DeepSeek-V4.1-Flash, bs=1, 4xGB300 (tok/s)\n\nSpeculative decoding is also supported on DeepSeek-V4.1-Flash, and it works well. It can be enabled with the `--speculative-drafter-model=native` flag.\n\n## Cold start\n\nWe’re pretty happy about our cold-start performance. We already have improvements that are not in this release that we can’t wait to share with you soon.\n\nCold Start, deepseek-ai/DeepSeek-V4.1-Flash, 4xGB300\n\n## Bringing frontier IR support to ZML\n\nIt was always possible in [zml/zml](https://github.com/zml/zml) to emit custom kernels as MLIR bytecode using the built-in codegen builder. This allows to leverage ZML’s memory allocation, stream management, concurrency and sharding while going off the trail if needed. We used that feature to emit [Triton](https://github.com/zml/zml/tree/master/kernels/triton) and [Mosaic (Pallas)](https://github.com/zml/zml/tree/master/kernels/mosaic_tpu) kernels.\n\nSince the last release, we have added support for [CuTe IR](https://github.com/zml/zml/tree/master/kernels/cute), [CUDA Tile IR](https://github.com/zml/zml/tree/master/kernels/cuda_tile), and [Fly IR](https://github.com/zml/zml/tree/master/kernels/fly).\n\nWhy does this matter? Well, because Triton cannot get the full performance of the underlying hardware. That’s why there’s [Gluon](https://github.com/triton-lang/triton/blob/main/python/tutorials/gluon/01-intro.py) now (which we’ll add support for soon).\n\nOn NVIDIA, CUDA Tile is faster than Triton, and CuTe is as fast as raw C++ CUDA (while still being a JIT). On AMD, Fly IR is faster than AMD’s preferred Triton path.\n\nUsing these new backends, we replaced the few custom kernels we had (MoE and PagedAttention) using their fastest respective IR. For instance, on NVIDIA, CuTe IR is as fast as C++ CUDA. So here goes.\n\nAs always with us, this was achieved with zero Python. No compromise.\n\nMetal gains MXFP8, MXFP4 and NVFP4 block scaled dot products and block-FP8 MoE paths.\n\nThat means you can run NVFP4 models on Apple Silicon:\n\n```\n$ llmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1\n```\n\nThe performance is nice, too:\n\nApple M3 Max, nvidia/Qwen3.8-27B-NVFP4, 1024/128, bs=1 (tg tok/s)\n\n## Various improvements\n\n## AMD\n\nThis release ships with ROCm 10.\n\n## Intel\n\nThis release brings fixes for four GPU configurations.\n\n## Serving diagnostics\n\nPrometheus metrics (`/metrics`) now cover scheduling, request outcomes, prompt-token counts, and speculative-decoding efficiency.\n\n`ignore_eos` is also now supported for benchmarking.\n\n## Try it today\n\n**NVIDIA**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n```\n\n**AMD**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n```\n\n**Intel**\n\n```bash\ndocker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Google TPU**\n\n```bash\ndocker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Apple Metal**\n\n```bash\nbrew install zml/zml/llmd\nllmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1\n```\n", "dateCreated": "2026-09-29T23:14:03.129222144+00:00", "date_created": "2026-09-29T07:00:00+00:00", "date_modified": "2026-09-29T07:00:00+00:00", "date_published": "2026-09-29T07:00:00+00:00", "description": "We’re happy to announce the release of zml/llmd 20260929.0, which notably brings deepseek-ai/DeepSeek-V4.1-Flash support on NVIDIA and AMD GPUs. Models This release adds 2 model families: deepseek-ai/DeepSeek-V4.1-Flash (with native DSpark…", "external_url": "https://zml.ai/posts/llmd-20260929.0/", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/zml-ai/2026/09/2026-09-29-zml-llmd-20260929-0.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/zml-ai/2026/09/2026-09-29-zml-llmd-20260929-0.md" }, "id": "urn:aggr:item:c66f276cb7cc45b6c1e1925f62e185c79d0f4536", "isBasedOn": { "@id": "https://zml.ai/posts/llmd-20260929.0/" }, "mainEntity": { "@id": "https://zml.ai/posts/llmd-20260929.0/", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-09-29-zml-llmd-20260929-0/#webpage" }, "dateModified": "2026-09-29T07:00:00+00:00", "datePublished": "2026-09-29T07:00:00+00:00", "headline": "ZML/LLMD 20260929.0", "timeRequired": "PT4M", "url": "https://zml.ai/posts/llmd-20260929.0/", "wordCount": 726 }, "name": "Archived snapshot: ZML/LLMD 20260929.0", "preview": null, "reading_minutes": 4, "replicated_at": null, "source": "zml-ai", "source_name": "ZML - Model to Metal", "summary": "We’re happy to announce the release of zml/llmd 20260929.0, which notably brings deepseek-ai/DeepSeek-V4.1-Flash support on NVIDIA and AMD GPUs. Models This release adds 2 model families: deepseek-ai/DeepSeek-V4.1-Flash (with native DSpark support) meta-models/Muse-Glimmer-30B As stated, DeepSeek is only available on platforms with enough VRAM to host it natively. We do not plan to support custom quants in the near future. DeepSeek-V4.1-Flash Obviously the highlight, this is our first release supporting a frontier model. ZML is now mature enough to support complex frontier models, and we wanted to bring it to you. Bear in mind this is a 763B model, weighing 510GB, so for now we only support it on platforms LLMD supports with enough VRAM, namely NVIDIA and AMD GPUs.", "tags": [], "temporalCoverage": "2026-09-29T23:14:03.129222144+00:00", "title": "ZML/LLMD 20260929.0", "url": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-09-29-zml-llmd-20260929-0/", "word_count": 726 }