{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-11-06-zml-llmd-20261106-0/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Article\n\nMore models, less waiting.\n\nNovember 6, 2026 inference / serving / NVIDIA / AMD / TPU / Metal / Intel\n\nWe are pleased to announce the release of `zml/llmd 20261106.0`. This week, we are bringing two new models, asynchronous scheduling in beta, and more performance work across our backends.\n\n## Beam-1.0 and Ripple\n\nThis release brings support for [Reflection’s Beam 1.0](https://huggingface.co/reflection/Beam-1.0-501B-A23B) and Day 1 support for [Ricursive’s Ripple](https://huggingface.co/ricursiveAI/Ripple-65B-Preview) (well, technically Day 0, but we didn’t want to derail the release).\n\nWe’re pretty happy with the performance: ![](https://zml.ai/posts/llmd-20261106.0/ripple.png)\n\nThe list of models supported by zml/llmd is growing, here is a full list of the supported models:\n\n- Qwen 2, 3, 3.5, 3.6 and 3.8 series (dense and MoE)\n- Gemma 3 and 4 series (dense and MoE)\n- LFM2.5 series\n- Mistral 3 and Ministral series\n- Llama 2 and 3 series\n- Poolside Laguna S 2.1\n- DeepSeek 4.1 Flash\n- Muse Glimmer\n- Qwen3.8 Flash Next\n- Xiaomi MiMo V2.6\n- Mistral Large 4\n- GLM 5.2\n- GLM 5.3\n- LFM 3\n- Beam 1.0\n- Ripple 65B\n\n## Asynchronous scheduling beta\n\nWe have added asynchronous scheduling to Llama and Gemma 4. The idea is simple: while the accelerator is running one step, the CPU can prepare the next one. Less time waiting between steps.\n\nYou can enable it with `--async-scheduling`. It is opt-in for now. Speculative decoding still uses the synchronous path, and Llama’s asynchronous path also works with data parallelism. The next release should bring more models on board.\n\n*This is in beta. Once we are satisfied with the performance and stability, we will make it the default.*\n\n## Data parallelism beta\n\nLlama now supports `--data-parallel`. This distributes requests and their KV caches across groups of devices. We also improved weight sharding for this configuration to reduce communication between those groups.\n\nFor example, `--data-parallel=2` splits the work across two groups.\n\nOn 2×RTX 5090, our Llama development benchmark went from 700 to 1,600 tok/s with `--data-parallel=2`, `--batch-size=16` and `--token-batch-size=256`. That’s a 2.3× improvement to the data-parallel path. The same benchmark with `--data-parallel=1` stays at roughly 2,000 tok/s, so the best configuration still depends on the workload.\n\n*This is still in beta.*\n\n## Faster LFM 3\n\nLast week, we brought LFM 3 to zml/llmd. This week, we improved how its embeddings are sharded across devices, which improves throughput when running it on multiple GPUs.\n\n## Faster prefill\n\nThe hill climb continues. This week, we worked on Gated DeltaNet, with new CuTe kernels for NVIDIA and Fly kernels for AMD. The NVIDIA and AMD implementations improve prefill performance over the previous Triton backend.\n\nOn a single MI300X, running Qwen3.8-27B with 32K-token prompts and `bs=16`, time to first token improves as follows (using `z-lab/Qwen3.8-27B-DFlash2` for speculative decoding):\n\nTime to first token, Qwen3.8-27B, bs=16, 32K-token prompts, 1xMI300X, less is better (s)\n\nThat’s about 12.5% less waiting for ordinary prefill and 14.5% less with speculative decoding.\n\nDeepSeek-V4.1 also gets optimized mHC kernels for NVIDIA Blackwell and AMD MI300X.\n\nAt 8,192 tokens, our DeepSeek prefill benchmark goes from 35,016 to 40,115 tok/s, a 14.6% improvement from the combined kernel and sharding changes.\n\nThe mHC kernel benchmarks also show where some of that improvement comes from.\n\nAt 32,768 tokens:\n\nmHC execution time, 32768 tokens, less is better (µs)\n\nWe also removed unnecessary waits in the Qwen 3.8 Flash Next and MiMo-V2.6 forward passes.\n\n## Serving improvements\n\nA few fixes also made it into this release:\n\n- Cache hits are now reported in API usage responses through `prompt_tokens_details.cached_tokens`.\n- Prompts that exceed the context limit return HTTP `400` with `context_length_exceeded`. Unsupported `response_format` values also return HTTP `400`.\n- When `enable_thinking` is omitted, we now respect the model’s default. This fixes Gemma 4’s default behaviour.\n- We fixed KV-cache matching in LFM 2 and an issue with grouped-weight packing.\n- [Tini](https://github.com/krallin/tini) is now the default entrypoint for all zml/llmd container images, which should improve signal handling and shutdown behaviour.\n\n## Try it today\n\n**NVIDIA**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://Qwen/Qwen3.8-Flash-Next\n```\n\n**AMD**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n --model=hf://Qwen/Qwen3.8-Flash-Next\n```\n\n**Intel**\n\n```bash\ndocker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Google TPU**\n\n```bash\ndocker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Apple Metal**\n\n```bash\nbrew install zml/zml/llmd\nllmd --model=hf://LiquidAI/LFM3-1.7B-FP8 --batch-size=4\n```\n", "authors": [], "captured_at": "2026-11-06T18:21:10.909146084+00:00", "category": "ai", "content_html": "

Article

\n

More models, less waiting.

\n

November 6, 2026 inference / serving / NVIDIA / AMD / TPU / Metal / Intel

\n

We are pleased to announce the release of zml/llmd 20261106.0. This week, we are bringing two new models, asynchronous scheduling in beta, and more performance work across our backends.

\n

Beam-1.0 and Ripple

\n

This release brings support for Reflection’s Beam 1.0 and Day 1 support for Ricursive’s Ripple (well, technically Day 0, but we didn’t want to derail the release).

\n

We’re pretty happy with the performance: \"\"

\n

The list of models supported by zml/llmd is growing, here is a full list of the supported models:

\n\n

Asynchronous scheduling beta

\n

We have added asynchronous scheduling to Llama and Gemma 4. The idea is simple: while the accelerator is running one step, the CPU can prepare the next one. Less time waiting between steps.

\n

You can enable it with --async-scheduling. It is opt-in for now. Speculative decoding still uses the synchronous path, and Llama’s asynchronous path also works with data parallelism. The next release should bring more models on board.

\n

This is in beta. Once we are satisfied with the performance and stability, we will make it the default.

\n

Data parallelism beta

\n

Llama now supports --data-parallel. This distributes requests and their KV caches across groups of devices. We also improved weight sharding for this configuration to reduce communication between those groups.

\n

For example, --data-parallel=2 splits the work across two groups.

\n

On 2×RTX 5090, our Llama development benchmark went from 700 to 1,600 tok/s with --data-parallel=2, --batch-size=16 and --token-batch-size=256. That’s a 2.3× improvement to the data-parallel path. The same benchmark with --data-parallel=1 stays at roughly 2,000 tok/s, so the best configuration still depends on the workload.

\n

This is still in beta.

\n

Faster LFM 3

\n

Last week, we brought LFM 3 to zml/llmd. This week, we improved how its embeddings are sharded across devices, which improves throughput when running it on multiple GPUs.

\n

Faster prefill

\n

The hill climb continues. This week, we worked on Gated DeltaNet, with new CuTe kernels for NVIDIA and Fly kernels for AMD. The NVIDIA and AMD implementations improve prefill performance over the previous Triton backend.

\n

On a single MI300X, running Qwen3.8-27B with 32K-token prompts and bs=16, time to first token improves as follows (using z-lab/Qwen3.8-27B-DFlash2 for speculative decoding):

\n

Time to first token, Qwen3.8-27B, bs=16, 32K-token prompts, 1xMI300X, less is better (s)

\n

That’s about 12.5% less waiting for ordinary prefill and 14.5% less with speculative decoding.

\n

DeepSeek-V4.1 also gets optimized mHC kernels for NVIDIA Blackwell and AMD MI300X.

\n

At 8,192 tokens, our DeepSeek prefill benchmark goes from 35,016 to 40,115 tok/s, a 14.6% improvement from the combined kernel and sharding changes.

\n

The mHC kernel benchmarks also show where some of that improvement comes from.

\n

At 32,768 tokens:

\n

mHC execution time, 32768 tokens, less is better (µs)

\n

We also removed unnecessary waits in the Qwen 3.8 Flash Next and MiMo-V2.6 forward passes.

\n

Serving improvements

\n

A few fixes also made it into this release:

\n\n

Try it today

\n

NVIDIA

\n
docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n    --model=hf://Qwen/Qwen3.8-Flash-Next\n
\n

AMD

\n
docker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n    --model=hf://Qwen/Qwen3.8-Flash-Next\n
\n

Intel

\n
docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n    --model=hf://Qwen/Qwen3.8-27B\n
\n

Google TPU

\n
docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n    --model=hf://Qwen/Qwen3.8-27B\n
\n

Apple Metal

\n
brew install zml/zml/llmd\nllmd --model=hf://LiquidAI/LFM3-1.7B-FP8 --batch-size=4\n
\n", "content_markdown": "Article\n\nMore models, less waiting.\n\nNovember 6, 2026 inference / serving / NVIDIA / AMD / TPU / Metal / Intel\n\nWe are pleased to announce the release of `zml/llmd 20261106.0`. This week, we are bringing two new models, asynchronous scheduling in beta, and more performance work across our backends.\n\n## Beam-1.0 and Ripple\n\nThis release brings support for [Reflection’s Beam 1.0](https://huggingface.co/reflection/Beam-1.0-501B-A23B) and Day 1 support for [Ricursive’s Ripple](https://huggingface.co/ricursiveAI/Ripple-65B-Preview) (well, technically Day 0, but we didn’t want to derail the release).\n\nWe’re pretty happy with the performance: ![](https://zml.ai/posts/llmd-20261106.0/ripple.png)\n\nThe list of models supported by zml/llmd is growing, here is a full list of the supported models:\n\n- Qwen 2, 3, 3.5, 3.6 and 3.8 series (dense and MoE)\n- Gemma 3 and 4 series (dense and MoE)\n- LFM2.5 series\n- Mistral 3 and Ministral series\n- Llama 2 and 3 series\n- Poolside Laguna S 2.1\n- DeepSeek 4.1 Flash\n- Muse Glimmer\n- Qwen3.8 Flash Next\n- Xiaomi MiMo V2.6\n- Mistral Large 4\n- GLM 5.2\n- GLM 5.3\n- LFM 3\n- Beam 1.0\n- Ripple 65B\n\n## Asynchronous scheduling beta\n\nWe have added asynchronous scheduling to Llama and Gemma 4. The idea is simple: while the accelerator is running one step, the CPU can prepare the next one. Less time waiting between steps.\n\nYou can enable it with `--async-scheduling`. It is opt-in for now. Speculative decoding still uses the synchronous path, and Llama’s asynchronous path also works with data parallelism. The next release should bring more models on board.\n\n*This is in beta. Once we are satisfied with the performance and stability, we will make it the default.*\n\n## Data parallelism beta\n\nLlama now supports `--data-parallel`. This distributes requests and their KV caches across groups of devices. We also improved weight sharding for this configuration to reduce communication between those groups.\n\nFor example, `--data-parallel=2` splits the work across two groups.\n\nOn 2×RTX 5090, our Llama development benchmark went from 700 to 1,600 tok/s with `--data-parallel=2`, `--batch-size=16` and `--token-batch-size=256`. That’s a 2.3× improvement to the data-parallel path. The same benchmark with `--data-parallel=1` stays at roughly 2,000 tok/s, so the best configuration still depends on the workload.\n\n*This is still in beta.*\n\n## Faster LFM 3\n\nLast week, we brought LFM 3 to zml/llmd. This week, we improved how its embeddings are sharded across devices, which improves throughput when running it on multiple GPUs.\n\n## Faster prefill\n\nThe hill climb continues. This week, we worked on Gated DeltaNet, with new CuTe kernels for NVIDIA and Fly kernels for AMD. The NVIDIA and AMD implementations improve prefill performance over the previous Triton backend.\n\nOn a single MI300X, running Qwen3.8-27B with 32K-token prompts and `bs=16`, time to first token improves as follows (using `z-lab/Qwen3.8-27B-DFlash2` for speculative decoding):\n\nTime to first token, Qwen3.8-27B, bs=16, 32K-token prompts, 1xMI300X, less is better (s)\n\nThat’s about 12.5% less waiting for ordinary prefill and 14.5% less with speculative decoding.\n\nDeepSeek-V4.1 also gets optimized mHC kernels for NVIDIA Blackwell and AMD MI300X.\n\nAt 8,192 tokens, our DeepSeek prefill benchmark goes from 35,016 to 40,115 tok/s, a 14.6% improvement from the combined kernel and sharding changes.\n\nThe mHC kernel benchmarks also show where some of that improvement comes from.\n\nAt 32,768 tokens:\n\nmHC execution time, 32768 tokens, less is better (µs)\n\nWe also removed unnecessary waits in the Qwen 3.8 Flash Next and MiMo-V2.6 forward passes.\n\n## Serving improvements\n\nA few fixes also made it into this release:\n\n- Cache hits are now reported in API usage responses through `prompt_tokens_details.cached_tokens`.\n- Prompts that exceed the context limit return HTTP `400` with `context_length_exceeded`. Unsupported `response_format` values also return HTTP `400`.\n- When `enable_thinking` is omitted, we now respect the model’s default. This fixes Gemma 4’s default behaviour.\n- We fixed KV-cache matching in LFM 2 and an issue with grouped-weight packing.\n- [Tini](https://github.com/krallin/tini) is now the default entrypoint for all zml/llmd container images, which should improve signal handling and shutdown behaviour.\n\n## Try it today\n\n**NVIDIA**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://Qwen/Qwen3.8-Flash-Next\n```\n\n**AMD**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n --model=hf://Qwen/Qwen3.8-Flash-Next\n```\n\n**Intel**\n\n```bash\ndocker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Google TPU**\n\n```bash\ndocker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n --model=hf://Qwen/Qwen3.8-27B\n```\n\n**Apple Metal**\n\n```bash\nbrew install zml/zml/llmd\nllmd --model=hf://LiquidAI/LFM3-1.7B-FP8 --batch-size=4\n```\n", "dateCreated": "2026-11-06T18:21:10.909146084+00:00", "date_created": "2026-11-06T08:00:00+00:00", "date_modified": "2026-11-06T08:00:00+00:00", "date_published": "2026-11-06T08:00:00+00:00", "description": "We are pleased to announce the release of zml/llmd 20261106.0. This week, we are bringing two new models, asynchronous scheduling in beta, and more performance work across our backends. Beam-1.0 and Ripple This release brings support for…", "external_url": "https://zml.ai/posts/llmd-20261106.0/", "git": { "commit": "20068632b65d2830a2d54f25ca1d90d697bea2c0", "path": "items/zml-ai/2026/11/2026-11-06-zml-llmd-20261106-0.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/20068632b65d2830a2d54f25ca1d90d697bea2c0/items/zml-ai/2026/11/2026-11-06-zml-llmd-20261106-0.md" }, "id": "urn:aggr:item:3bef9771c2ae2ed4319e40d627e81e027edd4b87", "isBasedOn": { "@id": "https://zml.ai/posts/llmd-20261106.0/" }, "mainEntity": { "@id": "https://zml.ai/posts/llmd-20261106.0/", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-11-06-zml-llmd-20261106-0/#webpage" }, "dateModified": "2026-11-06T08:00:00+00:00", "datePublished": "2026-11-06T08:00:00+00:00", "headline": "zml/llmd 20261106.0", "image": { "@type": "ImageObject", "caption": "zml/llmd 20261106.0 — More models, less waiting.", "height": 90, "url": "https://zml.ai/og/background_hu_fc70c6554a0e3c8e.png", "width": 160 }, "timeRequired": "PT4M", "url": "https://zml.ai/posts/llmd-20261106.0/", "wordCount": 763 }, "name": "Archived snapshot: zml/llmd 20261106.0", "preview": { "alt": "zml/llmd 20261106.0 — More models, less waiting.", "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://zml.ai/og/background_hu_fc70c6554a0e3c8e.png", "width": 160 }, "reading_minutes": 4, "replicated_at": null, "source": "zml-ai", "source_name": "ZML - Model to Metal", "summary": "We are pleased to announce the release of zml/llmd 20261106.0. This week, we are bringing two new models, asynchronous scheduling in beta, and more performance work across our backends. Beam-1.0 and Ripple This release brings support for Reflection’s Beam 1.0 and Day 1 support for Ricursive’s Ripple (well, technically Day 0, but we didn’t want to derail the release). We’re pretty happy with the performance: The list of models supported by zml/llmd is growing, here is a full list of the supported models:", "tags": [], "temporalCoverage": "2026-11-06T18:21:10.909146084+00:00", "title": "zml/llmd 20261106.0", "url": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-11-06-zml-llmd-20261106-0/", "word_count": 763 }