{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-07-08-zml-llmd-alpha/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs transparently on **5 architectures**: **NVIDIA CUDA**, **AMD ROCm**, **Google TPU**, **Intel oneAPI** and **Apple Metal**.\n\n![](https://zml.ai/posts/llmd/llmd1.png)\n\n## Modern serving features\n\nZML/LLMD supports modern serving features: **continuous batching**, **paged attention**, **tensor parallel sharding**, **prefix caching**, **tool calling** and does so on **all platforms**.\n\nMetrics are also exposed in the Prometheus format via the `/metrics` endpoint.\n\n## Supported models\n\nZML/LLMD alpha ships with support for the following models:\n\n- Qwen 2, 3, 3.5 and 3.6 series (dense and MoE)\n- Gemma 3 and 4 series (dense and MoE)\n- LFM2.5 series\n- Mistral 3 and Ministral series\n- LLaMa 2 and 3 series\n\nDeepSeek, Kimi, GLM, MiniMax and StepFun models are coming soon.\n\n## Native HuggingFace, S3 and GCS support\n\nBuilt on ZML, ZML/LLMD has native support for ZML’s VFS subsystem, which allows for zero-copy loading of models from HuggingFace (`hf://`), S3 (`s3://`), and GCS (`gs://`). This means that you can load models directly from these sources without having to download them first, saving time and storage space. Authentication is handled with the standard environment or paths for each backend.\n\nFor instance, loading from HuggingFace is as simple as using the `hf://` prefix on any flag that expects a path:\n\n```bash\n$ docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://Qwen/Qwen3-8B\n```\n\nNotice we’re also passing the `HF_TOKEN` environment variable to authenticate.\n\n## DFlash speculative decoding\n\nZML/LLMD alpha ships with native support for DFlash on Gemma 4 series and soon Qwen series models. DFlash is a new speculative decoding algorithm that can speed up tok/s/user by **up to 10x** on supported models.\n\nTo use it, just pass the `--dflash-model` flag when launching LLMD:\n\n```bash\n$ docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://google/gemma-4-31B-it \\\n --dflash-model=hf://z-lab/gemma-4-31B-it-DFlash\n```\n\nOne nice benefit of ZML’s programming model is the compounding of features across platforms. For instance, DFlash transparently runs very well on Intel and Apple GPUs.\n\nRead about DFlash [on the Z-Lab blog](https://z-lab.ai/projects/dflash/) and the [currently released DFlash drafters](https://huggingface.co/collections/z-lab/dflash).\n\n## Optimized Docker image\n\nOn each platform, ZML/LLMD ships the platform runtime as part of the image in a sandboxed manner. Per file optimization reduces image sizes by 5-10x. On top of that careful ordering of layers and custom compression improves `docker pull` time by using download/extraction overlapping.\n\nThe result is a small image that can be pulled and run in seconds:\n\n```bash\n$ time docker pull zmlai/llmd:cuda\ncuda: Pulling from zmlai/llmd\n...\nStatus: Downloaded newer image for zmlai/llmd:cuda\ndocker.io/zmlai/llmd:cuda\n\nreal 0m13.766s\nuser 0m0.021s\nsys 0m0.022s\n```\n\n| Platform | Image | Size |\n| ---------- | --------------------------- | ---------- |\n| **CUDA** | `zmlai/llmd:cuda` | **1.7 GB** |\n| **ROCm** | `zmlai/llmd:rocm` | **3.9 GB** |\n| **TPU** | `zmlai/llmd:tpu` | **280 MB** |\n| **OneAPI** | `zmlai/llmd:oneapi` | **350 MB** |\n| **Metal** | `brew install zml/zml/llmd` | **140 MB** |\n\nEach image ships with everything it needs to run. For instance on Metal the Apple Metal compiler is shipped in the archive, or ROCm itself in the AMD image.\n\n### Built-in CUDA Compatibility\n\nOn the CUDA platform, ZML/LLMD ships with the [CUDA Compatibility layer](https://docs.nvidia.com/deploy/cuda-compatibility/latest/why-cuda-compatibility.html) transparently built-in that enables ZML/LLMD to run on a wide range of driver versions without intervention.\n\nThis feature doesn’t require any special configuration as the probe is done automatically at runtime. It is also fully sandboxed and doesn’t require any system libraries to be installed.\n\n## Automatic sharding support\n\nZML/LLMD supports tensor parallel inference on all platforms. It automatically shards the model across multiple devices and handles the communication between them transparently.\n\nCurrently only tensor or expert parallel sharding is supported. More control will be added in the near future, for instance pipeline parallelism.\n\n## Python-free Execution Path\n\nZML/LLMD runs on ZML’s own ML framework, ZML, built with Zig, MLIR, OpenXLA. It compiles binaries and runtimes ahead of time into a single, hermetic artifact. Since the ZML programming model is explicit compilation passes, once compiled, there are no hidden compilation happening in the hot path. Latencies are flat and predictable.\n\n## Performance\n\n### google/gemma-4-26B-A4B-it (bs=16)\n\n| NVIDIA H100x2 | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ------------- | ------- | ----- | ------ | ------ | ------ | ----- | ----- | ------ | ----- |\n| TTFT (ms) | 127.64 | 70.39 | 162.08 | 141.91 | 142.04 | 142.1 | 143.1 | 158.28 | 161.7 |\n| ITL (ms) | 12.03 | 0.01 | 85.58 | 11.67 | 12.28 | 13.14 | 16.35 | 68.24 | 74.42 |\n| Total tok/s | 1317.93 | | | | | | | | |\n\n| Intel B70x2 | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ------ | ------ | ------ | ----- | ------ | ------ | ------ | ------- |\n| TTFT (ms) | 426.05 | 117.36 | 1027.4 | 375.38 | 375.4 | 612.75 | 850.48 | 992.01 | 1023.86 |\n| ITL (ms) | 81.83 | 0.03 | 499.75 | 81.62 | 82.72 | 83.7 | 84.17 | 84.6 | 279.99 |\n| Total tok/s | 179.31 | | | | | | | | |\n\n| AMD MI300X | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ---- | ------ | ----- | ------ | ------ | ------ | ------ | ------ |\n| TTFT (ms) | 101.86 | 36.3 | 145.35 | 109.7 | 109.76 | 110.39 | 116.76 | 139.63 | 144.77 |\n| ITL (ms) | 18.52 | 0.02 | 69.33 | 18.4 | 18.82 | 19.12 | 19.26 | 40.41 | 47.75 |\n| Total tok/s | 858.8 | | | | | | | | |\n\n| M3 Max | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------- | ------- | ------- | ------- | ------ | ------- | ------ | ------- | ------- |\n| TTFT (ms) | 2364.71 | 1840.79 | 2608.73 | 2411.42 | 2411.5 | 2470.67 | 2539.7 | 2594.92 | 2607.35 |\n| ITL (ms) | 67.01 | 25.81 | 196.99 | 67.02 | 70.63 | 74.5 | 77.41 | 85.19 | 104.45 |\n| Total tok/s | 109.59 | | | | | | | | |\n\n### google/gemma-4-31B-it + z-lab/gemma-4-31B-it-DFlash (bs=16)\n\n| TPU v6ex4 | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ----- | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n| TTFT (ms) | 137.64 | 25.36 | 187.67 | 137.79 | 137.82 | 158.75 | 187.17 | 187.57 | 187.66 |\n| ITL (ms) | 16.73 | 0.01 | 345.35 | 0.02 | 0.08 | 71.85 | 73.76 | 75.63 | 323.83 |\n| Total tok/s | 943.7 | | | | | | | | |\n\n| AMD MI300X | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ----- | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n| TTFT (ms) | 254.95 | 57.23 | 368.84 | 268.09 | 268.14 | 268.23 | 283.34 | 351.74 | 367.13 |\n| ITL (ms) | 21.51 | 0.01 | 140.15 | 0.02 | 0.03 | 91.55 | 93.82 | 109.29 | 139.67 |\n| Total tok/s | 731.8 | | | | | | | | |\n\n### Qwen/Qwen3.6-27B (bs=8)\n\n| M3 Max | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| TTFT (ms) | 6052.4 | 2684.92 | 7285.46 | 6408.14 | 6408.16 | 6671.36 | 6978.41 | 7224.05 | 7279.32 |\n| ITL (ms) | 183.96 | 3.33 | 876.88 | 183.83 | 185.05 | 186.34 | 187.62 | 196.88 | 208.71 |\n| Total tok/s | 33.5 | | | | | | | | |\n\n| AMD MI300X | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ---- | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n| TTFT (ms) | 259.71 | 68.5 | 357.55 | 241.55 | 270.58 | 357.51 | 357.54 | 357.55 | 357.55 |\n| ITL (ms) | 30.97 | 0.01 | 115.9 | 31.02 | 31.15 | 31.24 | 31.29 | 31.42 | 34.38 |\n| Total tok/s | 511.6 | | | | | | | | |\n\n### Fast cold start\n\nZML/LLMD has been designed from the ground up to have fast cold start times, as shown here with Qwen 3.6-27B BF16:\n\n## Give it a try\n\nRun ZML/LLMD and point your favorite harness to it.\n\n**NVIDIA**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://Qwen/Qwen3.6-27B\n```\n\n**AMD**\n\n```bash\ndocker run -p 8000:8000 --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n --model=hf://Qwen/Qwen3.6-27B\n```\n\n**Intel**\n\n```bash\ndocker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n --model=hf://Qwen/Qwen3.6-27B\n```\n\n**Google TPU**\n\n```bash\ndocker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n --model=hf://Qwen/Qwen3.6-27B\n```\n\n**Apple Metal**\n\n```bash\nbrew install zml/zml/llmd\nllmd --model=hf://Qwen/Qwen3.6-27B\n```\n", "authors": [], "captured_at": "2026-09-08T10:17:10.166315+00:00", "category": "ai", "content_html": "

Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs transparently on 5 architectures: NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI and Apple Metal.

\n

\"\"

\n

Modern serving features

\n

ZML/LLMD supports modern serving features: continuous batching, paged attention, tensor parallel sharding, prefix caching, tool calling and does so on all platforms.

\n

Metrics are also exposed in the Prometheus format via the /metrics endpoint.

\n

Supported models

\n

ZML/LLMD alpha ships with support for the following models:

\n\n

DeepSeek, Kimi, GLM, MiniMax and StepFun models are coming soon.

\n

Native HuggingFace, S3 and GCS support

\n

Built on ZML, ZML/LLMD has native support for ZML’s VFS subsystem, which allows for zero-copy loading of models from HuggingFace (hf://), S3 (s3://), and GCS (gs://). This means that you can load models directly from these sources without having to download them first, saving time and storage space. Authentication is handled with the standard environment or paths for each backend.

\n

For instance, loading from HuggingFace is as simple as using the hf:// prefix on any flag that expects a path:

\n
$ docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n    --model=hf://Qwen/Qwen3-8B\n
\n

Notice we’re also passing the HF_TOKEN environment variable to authenticate.

\n

DFlash speculative decoding

\n

ZML/LLMD alpha ships with native support for DFlash on Gemma 4 series and soon Qwen series models. DFlash is a new speculative decoding algorithm that can speed up tok/s/user by up to 10x on supported models.

\n

To use it, just pass the --dflash-model flag when launching LLMD:

\n
$ docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n    --model=hf://google/gemma-4-31B-it \\\n    --dflash-model=hf://z-lab/gemma-4-31B-it-DFlash\n
\n

One nice benefit of ZML’s programming model is the compounding of features across platforms. For instance, DFlash transparently runs very well on Intel and Apple GPUs.

\n

Read about DFlash on the Z-Lab blog and the currently released DFlash drafters.

\n

Optimized Docker image

\n

On each platform, ZML/LLMD ships the platform runtime as part of the image in a sandboxed manner. Per file optimization reduces image sizes by 5-10x. On top of that careful ordering of layers and custom compression improves docker pull time by using download/extraction overlapping.

\n

The result is a small image that can be pulled and run in seconds:

\n
$ time docker pull zmlai/llmd:cuda\ncuda: Pulling from zmlai/llmd\n...\nStatus: Downloaded newer image for zmlai/llmd:cuda\ndocker.io/zmlai/llmd:cuda\n\nreal    0m13.766s\nuser    0m0.021s\nsys     0m0.022s\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
PlatformImageSize
CUDAzmlai/llmd:cuda1.7 GB
ROCmzmlai/llmd:rocm3.9 GB
TPUzmlai/llmd:tpu280 MB
OneAPIzmlai/llmd:oneapi350 MB
Metalbrew install zml/zml/llmd140 MB
\n

Each image ships with everything it needs to run. For instance on Metal the Apple Metal compiler is shipped in the archive, or ROCm itself in the AMD image.

\n

Built-in CUDA Compatibility

\n

On the CUDA platform, ZML/LLMD ships with the CUDA Compatibility layer transparently built-in that enables ZML/LLMD to run on a wide range of driver versions without intervention.

\n

This feature doesn’t require any special configuration as the probe is done automatically at runtime. It is also fully sandboxed and doesn’t require any system libraries to be installed.

\n

Automatic sharding support

\n

ZML/LLMD supports tensor parallel inference on all platforms. It automatically shards the model across multiple devices and handles the communication between them transparently.

\n

Currently only tensor or expert parallel sharding is supported. More control will be added in the near future, for instance pipeline parallelism.

\n

Python-free Execution Path

\n

ZML/LLMD runs on ZML’s own ML framework, ZML, built with Zig, MLIR, OpenXLA. It compiles binaries and runtimes ahead of time into a single, hermetic artifact. Since the ZML programming model is explicit compilation passes, once compiled, there are no hidden compilation happening in the hot path. Latencies are flat and predictable.

\n

Performance

\n

google/gemma-4-26B-A4B-it (bs=16)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
NVIDIA H100x2AvgMinMaxP50P75P90P95P99P99.9
TTFT (ms)127.6470.39162.08141.91142.04142.1143.1158.28161.7
ITL (ms)12.030.0185.5811.6712.2813.1416.3568.2474.42
Total tok/s1317.93
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Intel B70x2AvgMinMaxP50P75P90P95P99P99.9
TTFT (ms)426.05117.361027.4375.38375.4612.75850.48992.011023.86
ITL (ms)81.830.03499.7581.6282.7283.784.1784.6279.99
Total tok/s179.31
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
AMD MI300XAvgMinMaxP50P75P90P95P99P99.9
TTFT (ms)101.8636.3145.35109.7109.76110.39116.76139.63144.77
ITL (ms)18.520.0269.3318.418.8219.1219.2640.4147.75
Total tok/s858.8
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
M3 MaxAvgMinMaxP50P75P90P95P99P99.9
TTFT (ms)2364.711840.792608.732411.422411.52470.672539.72594.922607.35
ITL (ms)67.0125.81196.9967.0270.6374.577.4185.19104.45
Total tok/s109.59
\n

google/gemma-4-31B-it + z-lab/gemma-4-31B-it-DFlash (bs=16)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
TPU v6ex4AvgMinMaxP50P75P90P95P99P99.9
TTFT (ms)137.6425.36187.67137.79137.82158.75187.17187.57187.66
ITL (ms)16.730.01345.350.020.0871.8573.7675.63323.83
Total tok/s943.7
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
AMD MI300XAvgMinMaxP50P75P90P95P99P99.9
TTFT (ms)254.9557.23368.84268.09268.14268.23283.34351.74367.13
ITL (ms)21.510.01140.150.020.0391.5593.82109.29139.67
Total tok/s731.8
\n

Qwen/Qwen3.6-27B (bs=8)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
M3 MaxAvgMinMaxP50P75P90P95P99P99.9
TTFT (ms)6052.42684.927285.466408.146408.166671.366978.417224.057279.32
ITL (ms)183.963.33876.88183.83185.05186.34187.62196.88208.71
Total tok/s33.5
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
AMD MI300XAvgMinMaxP50P75P90P95P99P99.9
TTFT (ms)259.7168.5357.55241.55270.58357.51357.54357.55357.55
ITL (ms)30.970.01115.931.0231.1531.2431.2931.4234.38
Total tok/s511.6
\n

Fast cold start

\n

ZML/LLMD has been designed from the ground up to have fast cold start times, as shown here with Qwen 3.6-27B BF16:

\n

Give it a try

\n

Run ZML/LLMD and point your favorite harness to it.

\n

NVIDIA

\n
docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n    --model=hf://Qwen/Qwen3.6-27B\n
\n

AMD

\n
docker run -p 8000:8000 --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n    --model=hf://Qwen/Qwen3.6-27B\n
\n

Intel

\n
docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n    --model=hf://Qwen/Qwen3.6-27B\n
\n

Google TPU

\n
docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n    --model=hf://Qwen/Qwen3.6-27B\n
\n

Apple Metal

\n
brew install zml/zml/llmd\nllmd --model=hf://Qwen/Qwen3.6-27B\n
\n", "content_markdown": "Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs transparently on **5 architectures**: **NVIDIA CUDA**, **AMD ROCm**, **Google TPU**, **Intel oneAPI** and **Apple Metal**.\n\n![](https://zml.ai/posts/llmd/llmd1.png)\n\n## Modern serving features\n\nZML/LLMD supports modern serving features: **continuous batching**, **paged attention**, **tensor parallel sharding**, **prefix caching**, **tool calling** and does so on **all platforms**.\n\nMetrics are also exposed in the Prometheus format via the `/metrics` endpoint.\n\n## Supported models\n\nZML/LLMD alpha ships with support for the following models:\n\n- Qwen 2, 3, 3.5 and 3.6 series (dense and MoE)\n- Gemma 3 and 4 series (dense and MoE)\n- LFM2.5 series\n- Mistral 3 and Ministral series\n- LLaMa 2 and 3 series\n\nDeepSeek, Kimi, GLM, MiniMax and StepFun models are coming soon.\n\n## Native HuggingFace, S3 and GCS support\n\nBuilt on ZML, ZML/LLMD has native support for ZML’s VFS subsystem, which allows for zero-copy loading of models from HuggingFace (`hf://`), S3 (`s3://`), and GCS (`gs://`). This means that you can load models directly from these sources without having to download them first, saving time and storage space. Authentication is handled with the standard environment or paths for each backend.\n\nFor instance, loading from HuggingFace is as simple as using the `hf://` prefix on any flag that expects a path:\n\n```bash\n$ docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://Qwen/Qwen3-8B\n```\n\nNotice we’re also passing the `HF_TOKEN` environment variable to authenticate.\n\n## DFlash speculative decoding\n\nZML/LLMD alpha ships with native support for DFlash on Gemma 4 series and soon Qwen series models. DFlash is a new speculative decoding algorithm that can speed up tok/s/user by **up to 10x** on supported models.\n\nTo use it, just pass the `--dflash-model` flag when launching LLMD:\n\n```bash\n$ docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://google/gemma-4-31B-it \\\n --dflash-model=hf://z-lab/gemma-4-31B-it-DFlash\n```\n\nOne nice benefit of ZML’s programming model is the compounding of features across platforms. For instance, DFlash transparently runs very well on Intel and Apple GPUs.\n\nRead about DFlash [on the Z-Lab blog](https://z-lab.ai/projects/dflash/) and the [currently released DFlash drafters](https://huggingface.co/collections/z-lab/dflash).\n\n## Optimized Docker image\n\nOn each platform, ZML/LLMD ships the platform runtime as part of the image in a sandboxed manner. Per file optimization reduces image sizes by 5-10x. On top of that careful ordering of layers and custom compression improves `docker pull` time by using download/extraction overlapping.\n\nThe result is a small image that can be pulled and run in seconds:\n\n```bash\n$ time docker pull zmlai/llmd:cuda\ncuda: Pulling from zmlai/llmd\n...\nStatus: Downloaded newer image for zmlai/llmd:cuda\ndocker.io/zmlai/llmd:cuda\n\nreal 0m13.766s\nuser 0m0.021s\nsys 0m0.022s\n```\n\n| Platform | Image | Size |\n| ---------- | --------------------------- | ---------- |\n| **CUDA** | `zmlai/llmd:cuda` | **1.7 GB** |\n| **ROCm** | `zmlai/llmd:rocm` | **3.9 GB** |\n| **TPU** | `zmlai/llmd:tpu` | **280 MB** |\n| **OneAPI** | `zmlai/llmd:oneapi` | **350 MB** |\n| **Metal** | `brew install zml/zml/llmd` | **140 MB** |\n\nEach image ships with everything it needs to run. For instance on Metal the Apple Metal compiler is shipped in the archive, or ROCm itself in the AMD image.\n\n### Built-in CUDA Compatibility\n\nOn the CUDA platform, ZML/LLMD ships with the [CUDA Compatibility layer](https://docs.nvidia.com/deploy/cuda-compatibility/latest/why-cuda-compatibility.html) transparently built-in that enables ZML/LLMD to run on a wide range of driver versions without intervention.\n\nThis feature doesn’t require any special configuration as the probe is done automatically at runtime. It is also fully sandboxed and doesn’t require any system libraries to be installed.\n\n## Automatic sharding support\n\nZML/LLMD supports tensor parallel inference on all platforms. It automatically shards the model across multiple devices and handles the communication between them transparently.\n\nCurrently only tensor or expert parallel sharding is supported. More control will be added in the near future, for instance pipeline parallelism.\n\n## Python-free Execution Path\n\nZML/LLMD runs on ZML’s own ML framework, ZML, built with Zig, MLIR, OpenXLA. It compiles binaries and runtimes ahead of time into a single, hermetic artifact. Since the ZML programming model is explicit compilation passes, once compiled, there are no hidden compilation happening in the hot path. Latencies are flat and predictable.\n\n## Performance\n\n### google/gemma-4-26B-A4B-it (bs=16)\n\n| NVIDIA H100x2 | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ------------- | ------- | ----- | ------ | ------ | ------ | ----- | ----- | ------ | ----- |\n| TTFT (ms) | 127.64 | 70.39 | 162.08 | 141.91 | 142.04 | 142.1 | 143.1 | 158.28 | 161.7 |\n| ITL (ms) | 12.03 | 0.01 | 85.58 | 11.67 | 12.28 | 13.14 | 16.35 | 68.24 | 74.42 |\n| Total tok/s | 1317.93 | | | | | | | | |\n\n| Intel B70x2 | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ------ | ------ | ------ | ----- | ------ | ------ | ------ | ------- |\n| TTFT (ms) | 426.05 | 117.36 | 1027.4 | 375.38 | 375.4 | 612.75 | 850.48 | 992.01 | 1023.86 |\n| ITL (ms) | 81.83 | 0.03 | 499.75 | 81.62 | 82.72 | 83.7 | 84.17 | 84.6 | 279.99 |\n| Total tok/s | 179.31 | | | | | | | | |\n\n| AMD MI300X | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ---- | ------ | ----- | ------ | ------ | ------ | ------ | ------ |\n| TTFT (ms) | 101.86 | 36.3 | 145.35 | 109.7 | 109.76 | 110.39 | 116.76 | 139.63 | 144.77 |\n| ITL (ms) | 18.52 | 0.02 | 69.33 | 18.4 | 18.82 | 19.12 | 19.26 | 40.41 | 47.75 |\n| Total tok/s | 858.8 | | | | | | | | |\n\n| M3 Max | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------- | ------- | ------- | ------- | ------ | ------- | ------ | ------- | ------- |\n| TTFT (ms) | 2364.71 | 1840.79 | 2608.73 | 2411.42 | 2411.5 | 2470.67 | 2539.7 | 2594.92 | 2607.35 |\n| ITL (ms) | 67.01 | 25.81 | 196.99 | 67.02 | 70.63 | 74.5 | 77.41 | 85.19 | 104.45 |\n| Total tok/s | 109.59 | | | | | | | | |\n\n### google/gemma-4-31B-it + z-lab/gemma-4-31B-it-DFlash (bs=16)\n\n| TPU v6ex4 | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ----- | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n| TTFT (ms) | 137.64 | 25.36 | 187.67 | 137.79 | 137.82 | 158.75 | 187.17 | 187.57 | 187.66 |\n| ITL (ms) | 16.73 | 0.01 | 345.35 | 0.02 | 0.08 | 71.85 | 73.76 | 75.63 | 323.83 |\n| Total tok/s | 943.7 | | | | | | | | |\n\n| AMD MI300X | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ----- | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n| TTFT (ms) | 254.95 | 57.23 | 368.84 | 268.09 | 268.14 | 268.23 | 283.34 | 351.74 | 367.13 |\n| ITL (ms) | 21.51 | 0.01 | 140.15 | 0.02 | 0.03 | 91.55 | 93.82 | 109.29 | 139.67 |\n| Total tok/s | 731.8 | | | | | | | | |\n\n### Qwen/Qwen3.6-27B (bs=8)\n\n| M3 Max | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ------- | ------- | ------- | ------- | ------- | ------- | ------- | ------- |\n| TTFT (ms) | 6052.4 | 2684.92 | 7285.46 | 6408.14 | 6408.16 | 6671.36 | 6978.41 | 7224.05 | 7279.32 |\n| ITL (ms) | 183.96 | 3.33 | 876.88 | 183.83 | 185.05 | 186.34 | 187.62 | 196.88 | 208.71 |\n| Total tok/s | 33.5 | | | | | | | | |\n\n| AMD MI300X | Avg | Min | Max | P50 | P75 | P90 | P95 | P99 | P99.9 |\n| ----------- | ------ | ---- | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n| TTFT (ms) | 259.71 | 68.5 | 357.55 | 241.55 | 270.58 | 357.51 | 357.54 | 357.55 | 357.55 |\n| ITL (ms) | 30.97 | 0.01 | 115.9 | 31.02 | 31.15 | 31.24 | 31.29 | 31.42 | 34.38 |\n| Total tok/s | 511.6 | | | | | | | | |\n\n### Fast cold start\n\nZML/LLMD has been designed from the ground up to have fast cold start times, as shown here with Qwen 3.6-27B BF16:\n\n## Give it a try\n\nRun ZML/LLMD and point your favorite harness to it.\n\n**NVIDIA**\n\n```bash\ndocker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \\\n --model=hf://Qwen/Qwen3.6-27B\n```\n\n**AMD**\n\n```bash\ndocker run -p 8000:8000 --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \\\n --model=hf://Qwen/Qwen3.6-27B\n```\n\n**Intel**\n\n```bash\ndocker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \\\n --model=hf://Qwen/Qwen3.6-27B\n```\n\n**Google TPU**\n\n```bash\ndocker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \\\n --model=hf://Qwen/Qwen3.6-27B\n```\n\n**Apple Metal**\n\n```bash\nbrew install zml/zml/llmd\nllmd --model=hf://Qwen/Qwen3.6-27B\n```\n", "dateCreated": "2026-09-08T10:17:10.166315+00:00", "date_created": "2026-07-08T07:00:00+00:00", "date_modified": "2026-07-08T07:00:00+00:00", "date_published": "2026-07-08T07:00:00+00:00", "description": "Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs transparently on 5 architectures: NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI and Apple Metal. Modern serving features…", "external_url": "https://zml.ai/posts/llmd/", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/zml-ai/2026/07/2026-07-08-zml-llmd-alpha.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/zml-ai/2026/07/2026-07-08-zml-llmd-alpha.md" }, "id": "urn:aggr:item:6cff417eadf22045c363e6f386f353f85c752102", "isBasedOn": { "@id": "https://zml.ai/posts/llmd/" }, "mainEntity": { "@id": "https://zml.ai/posts/llmd/", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-07-08-zml-llmd-alpha/#webpage" }, "dateModified": "2026-07-08T07:00:00+00:00", "datePublished": "2026-07-08T07:00:00+00:00", "headline": "ZML/LLMD alpha", "timeRequired": "PT6M", "url": "https://zml.ai/posts/llmd/", "wordCount": 1143 }, "name": "Archived snapshot: ZML/LLMD alpha", "preview": null, "reading_minutes": 6, "replicated_at": null, "source": "zml-ai", "source_name": "ZML - Model to Metal", "summary": "Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs transparently on 5 architectures: NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI and Apple Metal. Modern serving features ZML/LLMD supports modern serving features: continuous batching, paged attention, tensor parallel sharding, prefix caching, tool calling and does so on all platforms. Metrics are also exposed in the Prometheus format via the /metrics endpoint.", "tags": [], "temporalCoverage": "2026-09-08T10:17:10.166315+00:00", "title": "ZML/LLMD alpha", "url": "https://aggr.aymericbeaumet.com/items/zml-ai/2026-07-08-zml-llmd-alpha/", "word_count": 1143 }