{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-09-29-zml-llmd-20261009-0/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "[Back to posts](https://zml.ai/posts/)\n\nArticle\n\nPerformance is all around me, and so the feeling grows.\n\nOctober 9, 2026 inference / serving / NVIDIA / AMD / TPU / Metal / Intel\n\nAnother week, another release. We’re happy to announce the release of `zml/llmd 20261009.0`, which brings considerable performance improvements across the board.\n\n## Faster Compilation\n\nIn this release, we have spent a considerable amount of effort improving compilation performance. Notably, in configurations with multiple GPUs, autotuning is now done in parallel across GPUs. This has a significant impact on cold-start performance, as shown below.\n\nThese changes impact NVIDIA, AMD, Intel and Apple Metal backends.\n\nCold Start, deepseek-ai/DeepSeek-V4.1-Flash, 4xGB300, less is better\n\nAnd here is a visual representation of the compilation time improvements (BEFORE / AFTER): ![](https://zml.ai/posts/llmd-20261009.0/compilation_before_after.png)\n\n## Faster Qwen3.8-27B\n\nIn this release, we have decided to focus on some fundamental performance work and apply it to Qwen/Qwen3.8-27B as a testing ground.\n\nWe chose to benchmark it with the ridiculous `bs=1024` pure decode path because we found it tends to highlight inefficiencies better.\n\nToken Generation, Qwen/Qwen3.8-27B, w16a16, bs=1024, 4xGB300, more is better\n\nApple M3 Max, nvidia/Qwen3.8-27B-NVFP4, pp1024/tg128, bs=1, more is better (tg tok/s)\n\nCommand lines\n\n### zml/llmd\n\n```bash\nllmd --model=/var/models/Qwen/Qwen3.8-27B --batch-size=1024\n```\n\n### SGLang\n\n```bash\nuv run sglang serve \\\n --trust-remote-code \\\n --model-path /var/models/Qwen/Qwen3.8-27B \\\n --mem-fraction-static 0.85 \\\n --reasoning-parser qwen3 \\\n --tool-call-parser qwen3_coder \\\n --mamba-full-memory-ratio 3.67 \\\n --host 0.0.0.0 \\\n --port 30000 \\\n --mamba-radix-cache-strategy extra_buffer_lazy \\\n --mamba-ssm-dtype float32 \\\n --max-running-requests 1024 \\\n --tp 4 \\\n --cuda-graph-max-bs-decode 1024 \\\n --enable-torch-compile\n```\n\n## Faster DeepSeek-V4.1-Flash\n\nAs stated in the previous blog post, we chose to release DeepSeek-V4.1-Flash early, even though we were not yet at the performance level we can achieve. This release brings incremental performance improvements to DeepSeek-V4.1-Flash, as shown below.\n\nToken Generation, deepseek-ai/DeepSeek-V4.1-Flash, bs=1, 4xGB300 (tok/s)\n\nWe had missed enabling speculative decoding on Metal in the previous release. This release brings it back, and it works well. DFlash, DFlash2 and DSpark are all supported. If the model natively supports it, you can also pass the `--speculative-drafter-model=native` flag to enable it.\n\n## ZIO is now the default I/O backend\n\nThis release now enables [ZIO](https://github.com/lalinsky/zio) as the default Zig `std.Io` backend.\n\n> ZIO is an asynchronous runtime for Zig, in the same spirit as Go’s runtime or Tokio: it schedules lightweight coroutines onto a pool of OS threads, and gives you blocking-looking network, file, and process I/O that’s actually backed by non-blocking, event-driven OS APIs under the hood.\n\nThanks to it, zml/llmd runs on top of `kqueue` on macOS and `epoll` or `io_uring` on Linux (detected at runtime).\n\nZIO is exactly what we wanted to build ourselves (and had started to prior to `std.Io` in older `zml/zml` versions). So a huge shout-out to [Lukáš Lalinský](https://lalinsky.com/). Great job.\n\nThe old `std.Io.Threaded` backend is still available and can be enabled with the `--io-impl=threaded` flag.\n\n## Try it today\n\n**NVIDIA**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n```\n\n**AMD**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n```\n\n**Intel**\n\n```bash\ndocker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Google TPU**\n\n```bash\ndocker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Apple Metal**\n\n```bash\nbrew install zml/zml/llmd\nllmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1\n```\n", "authors": [], "captured_at": "2026-10-11T20:11:56.661477988+00:00", "category": "ai", "content_html": "

Back to posts

\n

Article

\n

Performance is all around me, and so the feeling grows.

\n

October 9, 2026 inference / serving / NVIDIA / AMD / TPU / Metal / Intel

\n

Another week, another release. We’re happy to announce the release of zml/llmd 20261009.0, which brings considerable performance improvements across the board.

\n

Faster Compilation

\n

In this release, we have spent a considerable amount of effort improving compilation performance. Notably, in configurations with multiple GPUs, autotuning is now done in parallel across GPUs. This has a significant impact on cold-start performance, as shown below.

\n

These changes impact NVIDIA, AMD, Intel and Apple Metal backends.

\n

Cold Start, deepseek-ai/DeepSeek-V4.1-Flash, 4xGB300, less is better

\n

And here is a visual representation of the compilation time improvements (BEFORE / AFTER): \"\"

\n

Faster Qwen3.8-27B

\n

In this release, we have decided to focus on some fundamental performance work and apply it to Qwen/Qwen3.8-27B as a testing ground.

\n

We chose to benchmark it with the ridiculous bs=1024 pure decode path because we found it tends to highlight inefficiencies better.

\n

Token Generation, Qwen/Qwen3.8-27B, w16a16, bs=1024, 4xGB300, more is better

\n

Apple M3 Max, nvidia/Qwen3.8-27B-NVFP4, pp1024/tg128, bs=1, more is better (tg tok/s)

\n

Command lines

\n

zml/llmd

\n
llmd --model=/var/models/Qwen/Qwen3.8-27B --batch-size=1024\n
\n

SGLang

\n
uv run sglang serve \\\n    --trust-remote-code \\\n    --model-path /var/models/Qwen/Qwen3.8-27B \\\n    --mem-fraction-static 0.85 \\\n    --reasoning-parser qwen3 \\\n    --tool-call-parser qwen3_coder \\\n    --mamba-full-memory-ratio 3.67 \\\n    --host 0.0.0.0 \\\n    --port 30000 \\\n    --mamba-radix-cache-strategy extra_buffer_lazy \\\n    --mamba-ssm-dtype float32 \\\n    --max-running-requests 1024 \\\n    --tp 4 \\\n    --cuda-graph-max-bs-decode 1024 \\\n    --enable-torch-compile\n
\n

Faster DeepSeek-V4.1-Flash

\n

As stated in the previous blog post, we chose to release DeepSeek-V4.1-Flash early, even though we were not yet at the performance level we can achieve. This release brings incremental performance improvements to DeepSeek-V4.1-Flash, as shown below.

\n

Token Generation, deepseek-ai/DeepSeek-V4.1-Flash, bs=1, 4xGB300 (tok/s)

\n

We had missed enabling speculative decoding on Metal in the previous release. This release brings it back, and it works well. DFlash, DFlash2 and DSpark are all supported. If the model natively supports it, you can also pass the --speculative-drafter-model=native flag to enable it.

\n

ZIO is now the default I/O backend

\n

This release now enables ZIO as the default Zig std.Io backend.

\n
\n

ZIO is an asynchronous runtime for Zig, in the same spirit as Go’s runtime or Tokio: it schedules lightweight coroutines onto a pool of OS threads, and gives you blocking-looking network, file, and process I/O that’s actually backed by non-blocking, event-driven OS APIs under the hood.

\n
\n

Thanks to it, zml/llmd runs on top of kqueue on macOS and epoll or io_uring on Linux (detected at runtime).

\n

ZIO is exactly what we wanted to build ourselves (and had started to prior to std.Io in older zml/zml versions). So a huge shout-out to Lukáš Lalinský. Great job.

\n

The old std.Io.Threaded backend is still available and can be enabled with the --io-impl=threaded flag.

\n

Try it today

\n

NVIDIA

\n
docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n    --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n
\n

AMD

\n
docker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n    --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n
\n

Intel

\n
docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n    --model=hf://Qwen/Qwen3.8-27B\n
\n

Google TPU

\n
docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n    --model=hf://Qwen/Qwen3.8-27B\n
\n

Apple Metal

\n
brew install zml/zml/llmd\nllmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1\n
\n", "content_markdown": "[Back to posts](https://zml.ai/posts/)\n\nArticle\n\nPerformance is all around me, and so the feeling grows.\n\nOctober 9, 2026 inference / serving / NVIDIA / AMD / TPU / Metal / Intel\n\nAnother week, another release. We’re happy to announce the release of `zml/llmd 20261009.0`, which brings considerable performance improvements across the board.\n\n## Faster Compilation\n\nIn this release, we have spent a considerable amount of effort improving compilation performance. Notably, in configurations with multiple GPUs, autotuning is now done in parallel across GPUs. This has a significant impact on cold-start performance, as shown below.\n\nThese changes impact NVIDIA, AMD, Intel and Apple Metal backends.\n\nCold Start, deepseek-ai/DeepSeek-V4.1-Flash, 4xGB300, less is better\n\nAnd here is a visual representation of the compilation time improvements (BEFORE / AFTER): ![](https://zml.ai/posts/llmd-20261009.0/compilation_before_after.png)\n\n## Faster Qwen3.8-27B\n\nIn this release, we have decided to focus on some fundamental performance work and apply it to Qwen/Qwen3.8-27B as a testing ground.\n\nWe chose to benchmark it with the ridiculous `bs=1024` pure decode path because we found it tends to highlight inefficiencies better.\n\nToken Generation, Qwen/Qwen3.8-27B, w16a16, bs=1024, 4xGB300, more is better\n\nApple M3 Max, nvidia/Qwen3.8-27B-NVFP4, pp1024/tg128, bs=1, more is better (tg tok/s)\n\nCommand lines\n\n### zml/llmd\n\n```bash\nllmd --model=/var/models/Qwen/Qwen3.8-27B --batch-size=1024\n```\n\n### SGLang\n\n```bash\nuv run sglang serve \\\n --trust-remote-code \\\n --model-path /var/models/Qwen/Qwen3.8-27B \\\n --mem-fraction-static 0.85 \\\n --reasoning-parser qwen3 \\\n --tool-call-parser qwen3_coder \\\n --mamba-full-memory-ratio 3.67 \\\n --host 0.0.0.0 \\\n --port 30000 \\\n --mamba-radix-cache-strategy extra_buffer_lazy \\\n --mamba-ssm-dtype float32 \\\n --max-running-requests 1024 \\\n --tp 4 \\\n --cuda-graph-max-bs-decode 1024 \\\n --enable-torch-compile\n```\n\n## Faster DeepSeek-V4.1-Flash\n\nAs stated in the previous blog post, we chose to release DeepSeek-V4.1-Flash early, even though we were not yet at the performance level we can achieve. This release brings incremental performance improvements to DeepSeek-V4.1-Flash, as shown below.\n\nToken Generation, deepseek-ai/DeepSeek-V4.1-Flash, bs=1, 4xGB300 (tok/s)\n\nWe had missed enabling speculative decoding on Metal in the previous release. This release brings it back, and it works well. DFlash, DFlash2 and DSpark are all supported. If the model natively supports it, you can also pass the `--speculative-drafter-model=native` flag to enable it.\n\n## ZIO is now the default I/O backend\n\nThis release now enables [ZIO](https://github.com/lalinsky/zio) as the default Zig `std.Io` backend.\n\n> ZIO is an asynchronous runtime for Zig, in the same spirit as Go’s runtime or Tokio: it schedules lightweight coroutines onto a pool of OS threads, and gives you blocking-looking network, file, and process I/O that’s actually backed by non-blocking, event-driven OS APIs under the hood.\n\nThanks to it, zml/llmd runs on top of `kqueue` on macOS and `epoll` or `io_uring` on Linux (detected at runtime).\n\nZIO is exactly what we wanted to build ourselves (and had started to prior to `std.Io` in older `zml/zml` versions). So a huge shout-out to [Lukáš Lalinský](https://lalinsky.com/). Great job.\n\nThe old `std.Io.Threaded` backend is still available and can be enabled with the `--io-impl=threaded` flag.\n\n## Try it today\n\n**NVIDIA**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n```\n\n**AMD**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n --model=hf://deepseek-ai/DeepSeek-V4.1-Flash\n```\n\n**Intel**\n\n```bash\ndocker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Google TPU**\n\n```bash\ndocker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Apple Metal**\n\n```bash\nbrew install zml/zml/llmd\nllmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1\n```\n", "dateCreated": "2026-10-11T20:11:56.661477988+00:00", "date_created": "2026-09-29T07:00:00+00:00", "date_modified": "2026-09-29T07:00:00+00:00", "date_published": "2026-09-29T07:00:00+00:00", "description": "Another week, another release. We’re happy to announce the release of zml/llmd 20261009.0, which brings considerable performance improvements across the board. Faster Compilation In this release, we have spent a considerable amount of…", "external_url": "https://zml.ai/posts/llmd-20261009.0/", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/zml-ai/2026/09/2026-09-29-zml-llmd-20261009-0.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/zml-ai/2026/09/2026-09-29-zml-llmd-20261009-0.md" }, "id": "urn:aggr:item:4139e97fff52f0315aa04ab31b3c4d51fd8c539a", "isBasedOn": { "@id": "https://zml.ai/posts/llmd-20261009.0/" }, "mainEntity": { "@id": "https://zml.ai/posts/llmd-20261009.0/", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-09-29-zml-llmd-20261009-0/#webpage" }, "dateModified": "2026-09-29T07:00:00+00:00", "datePublished": "2026-09-29T07:00:00+00:00", "headline": "ZML/LLMD 20261009.0", "timeRequired": "PT3M", "url": "https://zml.ai/posts/llmd-20261009.0/", "wordCount": 621 }, "name": "Archived snapshot: ZML/LLMD 20261009.0", "preview": null, "reading_minutes": 3, "replicated_at": null, "source": "zml-ai", "source_name": "ZML - Model to Metal", "summary": "Another week, another release. We’re happy to announce the release of zml/llmd 20261009.0, which brings considerable performance improvements across the board. Faster Compilation In this release, we have spent a considerable amount of effort into improving compilation performance. Notably, when in multiple GPU configurations, auto-tunning is now done in parallel across GPUs. This has a significant impact on cold-start performance, as shown below. This changes impact NVIDIA, AMD, Intel and Apple Metal backends.", "tags": [], "temporalCoverage": "2026-10-11T20:11:56.661477988+00:00", "title": "ZML/LLMD 20261009.0", "url": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-09-29-zml-llmd-20261009-0/", "word_count": 621 }