ZML/LLMD 20261009.0

Back to posts Article Performance is all around me, and so the feeling grows. October 9, 2026 inference / serving / NVIDIA / AMD / TPU / Metal / Intel Another week, another release. We’re happy to announce the release of zml/llmd 20261009.0, which brings considerable performance improvements across the board. Faster Compilation In this release, we have spent a considerable amount of effort improving compilation performance. Notably, in configurations with multiple GPUs, autotuning is now done in parallel across GPUs. This has a significant impact on cold-start performance, as shown below. These changes impact NVIDIA, AMD, Intel and Apple Metal backends. Cold Start, deepseek-ai/DeepSeek-V4.1-Flash, 4xGB300, less is better And here is a visual representation of the compilation time improvements (BEFORE / AFTER): Faster Qwen3.8-27B In this release, we have decided to focus on some fundamental performance work and apply it to Qwen/Qwen3.8-27B as a testing ground. We chose to benchmark it with the ridiculous bs=1024 pure decode path because we found it tends to highlight inefficiencies better. Token Generation, Qwen/Qwen3.8-27B, w16a16, bs=1024, 4xGB300, more is better Apple M3 Max, nvidia/Qwen3.8-27B-NVFP4, pp1024/tg128, bs=1, more is better (tg tok/s) Command lines zml/llmd llmd --model=/var/models/Qwen/Qwen3.8-27B --batch-size=1024 SGLang uv run sglang serve \ --trust-remote-code \ --model-path /var/models/Qwen/Qwen3.8-27B \ --mem-fraction-static 0.85 \ --reasoning-parser qwen3 \ --tool-call-parser qwen3_coder \ --mamba-full-memory-ratio 3.67 \ --host 0.0.0.0 \ --port 30000 \ --mamba-radix-cache-strategy extra_buffer_lazy \ --mamba-ssm-dtype float32 \ --max-running-requests 1024 \ --tp 4 \ --cuda-graph-max-bs-decode 1024 \ --enable-torch-compile Faster DeepSeek-V4.1-Flash As stated in the previous blog post, we chose to release DeepSeek-V4.1-Flash early, even though we were not yet at the performance level we can achieve. This release brings incremental performance improvements to DeepSeek-V4.1-Flash, as shown below. Token Generation, deepseek-ai/DeepSeek-V4.1-Flash, bs=1, 4xGB300 (tok/s) We had missed enabling speculative decoding on Metal in the previous release. This release brings it back, and it works well. DFlash, DFlash2 and DSpark are all supported. If the model natively supports it, you can also pass the --speculative-drafter-model=native flag to enable it. ZIO is now the default I/O backend This release now enables ZIO as the default Zig std.Io backend. ZIO is an asynchronous runtime for Zig, in the same spirit as Go’s runtime or Tokio: it schedules lightweight coroutines onto a pool of OS threads, and gives you blocking-looking network, file, and process I/O that’s actually backed by non-blocking, event-driven OS APIs under the hood. Thanks to it, zml/llmd runs on top of kqueue on macOS and epoll or io_uring on Linux (detected at runtime). ZIO is exactly what we wanted to build ourselves (and had started to prior to std.Io in older zml/zml versions). So a huge shout-out to Lukáš Lalinský. Great job. The old std.Io.Threaded backend is still available and can be enabled with the --io-impl=threaded flag. Try it today NVIDIA docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \ --model=hf://deepseek-ai/DeepSeek-V4.1-Flash AMD docker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \ --model=hf://deepseek-ai/DeepSeek-V4.1-Flash Intel docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \ --model=hf://Qwen/Qwen3.8-27B Google TPU docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \ --model=hf://Qwen/Qwen3.8-27B Apple Metal brew install zml/zml/llmd llmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1