ZML - Model to Metal

ZML/LLMD 20261009.0

Tue, 29 Sep 2026 07:00:00 +0000

Article

Performance is all around me, and so the feeling grows.

October 9, 2026 inference / serving / NVIDIA / AMD / TPU / Metal / Intel

Another week, another release. We’re happy to announce the release of zml/llmd 20261009.0, which brings considerable performance improvements across the board.

Faster Compilation

In this release, we have spent a considerable amount of effort improving compilation performance. Notably, in configurations with multiple GPUs, autotuning is now done in parallel across GPUs. This has a significant impact on cold-start performance, as shown below.

These changes impact NVIDIA, AMD, Intel and Apple Metal backends.

Cold Start, deepseek-ai/DeepSeek-V4.1-Flash, 4xGB300, less is better

And here is a visual representation of the compilation time improvements (BEFORE / AFTER):

Faster Qwen3.8-27B

In this release, we have decided to focus on some fundamental performance work and apply it to Qwen/Qwen3.8-27B as a testing ground.

We chose to benchmark it with the ridiculous bs=1024 pure decode path because we found it tends to highlight inefficiencies better.

Token Generation, Qwen/Qwen3.8-27B, w16a16, bs=1024, 4xGB300, more is better

Apple M3 Max, nvidia/Qwen3.8-27B-NVFP4, pp1024/tg128, bs=1, more is better (tg tok/s)

Command lines

zml/llmd

llmd --model=/var/models/Qwen/Qwen3.8-27B --batch-size=1024

SGLang

uv run sglang serve \
    --trust-remote-code \
    --model-path /var/models/Qwen/Qwen3.8-27B \
    --mem-fraction-static 0.85 \
    --reasoning-parser qwen3 \
    --tool-call-parser qwen3_coder \
    --mamba-full-memory-ratio 3.67 \
    --host 0.0.0.0 \
    --port 30000 \
    --mamba-radix-cache-strategy extra_buffer_lazy \
    --mamba-ssm-dtype float32 \
    --max-running-requests 1024 \
    --tp 4 \
    --cuda-graph-max-bs-decode 1024 \
    --enable-torch-compile

Faster DeepSeek-V4.1-Flash

As stated in the previous blog post, we chose to release DeepSeek-V4.1-Flash early, even though we were not yet at the performance level we can achieve. This release brings incremental performance improvements to DeepSeek-V4.1-Flash, as shown below.

Token Generation, deepseek-ai/DeepSeek-V4.1-Flash, bs=1, 4xGB300 (tok/s)

We had missed enabling speculative decoding on Metal in the previous release. This release brings it back, and it works well. DFlash, DFlash2 and DSpark are all supported. If the model natively supports it, you can also pass the --speculative-drafter-model=native flag to enable it.

ZIO is now the default I/O backend

This release now enables ZIO as the default Zig std.Io backend.

ZIO is an asynchronous runtime for Zig, in the same spirit as Go’s runtime or Tokio: it schedules lightweight coroutines onto a pool of OS threads, and gives you blocking-looking network, file, and process I/O that’s actually backed by non-blocking, event-driven OS APIs under the hood.

Thanks to it, zml/llmd runs on top of kqueue on macOS and epoll or io_uring on Linux (detected at runtime).

ZIO is exactly what we wanted to build ourselves (and had started to prior to std.Io in older zml/zml versions). So a huge shout-out to Lukáš Lalinský. Great job.

The old std.Io.Threaded backend is still available and can be enabled with the --io-impl=threaded flag.

Try it today

NVIDIA

docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \
    --model=hf://deepseek-ai/DeepSeek-V4.1-Flash

AMD

docker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \
    --model=hf://deepseek-ai/DeepSeek-V4.1-Flash

Intel

docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \
    --model=hf://Qwen/Qwen3.8-27B

Google TPU

docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \
    --model=hf://Qwen/Qwen3.8-27B

Apple Metal

brew install zml/zml/llmd
llmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1

ZML/LLMD 20260929.0

Tue, 29 Sep 2026 07:00:00 +0000

Article

DeepSeek-V4.1-Flash lands on NVIDIA and AMD.

September 29, 2026 inference / serving / NVIDIA / AMD / DeepSeek

We’re happy to announce the release of zml/llmd 20260929.0, which notably brings deepseek-ai/DeepSeek-V4.1-Flash support on NVIDIA and AMD GPUs.

Models

This release adds 2 model families:

deepseek-ai/DeepSeek-V4.1-Flash (with native DSpark support)
meta-models/Muse-Glimmer-30B

As stated, DeepSeek is only available on platforms with enough VRAM to host it natively. We do not plan to support custom quants in the near future.

DeepSeek-V4.1-Flash

Obviously the highlight, this is our first release supporting a frontier model. ZML is now mature enough to support complex frontier models, and we wanted to bring it to you. Bear in mind this is a 763B model, weighing 510GB, so for now we only support it on platforms LLMD supports with enough VRAM, namely NVIDIA and AMD GPUs.

That said, this is a cool release because it pushes the limits and shows how powerful ZML can be: MXFP4 MoEs, block-FP8, DSpark speculative decoding, DSA, HCA, and more. Nothing was spared. And it all works.

Sadly, we couldn’t bring it to TPUs because Google told us there were no TPUv7s available “for at least a few weeks”. LLMD doesn’t support Trainium yet due to an AWS Neuron bug that has since been fixed. We’ll revisit that.

On performance, we’re about 20% slower than vLLM and 30% slower than SGLang on 4xGB300. We could’ve waited to close the gap, but we wanted to bring it to you now. The roadmap is clear and we’ll get there, so why wait?

Token Generation, deepseek-ai/DeepSeek-V4.1-Flash, bs=1, 4xGB300 (tok/s)

Speculative decoding is also supported on DeepSeek-V4.1-Flash, and it works well. It can be enabled with the --speculative-drafter-model=native flag.

Cold start

We’re pretty happy about our cold-start performance. We already have improvements that are not in this release that we can’t wait to share with you soon.

Cold Start, deepseek-ai/DeepSeek-V4.1-Flash, 4xGB300

Bringing frontier IR support to ZML

It was always possible in zml/zml to emit custom kernels as MLIR bytecode using the built-in codegen builder. This allows to leverage ZML’s memory allocation, stream management, concurrency and sharding while going off the trail if needed. We used that feature to emit Triton and Mosaic (Pallas) kernels.

Since the last release, we have added support for CuTe IR, CUDA Tile IR, and Fly IR.

Why does this matter? Well, because Triton cannot get the full performance of the underlying hardware. That’s why there’s Gluon now (which we’ll add support for soon).

On NVIDIA, CUDA Tile is faster than Triton, and CuTe is as fast as raw C++ CUDA (while still being a JIT). On AMD, Fly IR is faster than AMD’s preferred Triton path.

Using these new backends, we replaced the few custom kernels we had (MoE and PagedAttention) using their fastest respective IR. For instance, on NVIDIA, CuTe IR is as fast as C++ CUDA. So here goes.

As always with us, this was achieved with zero Python. No compromise.

Metal gains MXFP8, MXFP4 and NVFP4 block scaled dot products and block-FP8 MoE paths.

That means you can run NVFP4 models on Apple Silicon:

$ llmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1

The performance is nice, too:

Apple M3 Max, nvidia/Qwen3.8-27B-NVFP4, 1024/128, bs=1 (tg tok/s)

Various improvements

AMD

This release ships with ROCm 10.

Intel

This release brings fixes for four GPU configurations.

Serving diagnostics

Prometheus metrics (/metrics) now cover scheduling, request outcomes, prompt-token counts, and speculative-decoding efficiency.

ignore_eos is also now supported for benchmarking.

Try it today

NVIDIA

docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \
    --model=hf://deepseek-ai/DeepSeek-V4.1-Flash

AMD

docker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \
    --model=hf://deepseek-ai/DeepSeek-V4.1-Flash

Intel

docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \
    --model=hf://Qwen/Qwen3.8-27B

Google TPU

docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \
    --model=hf://Qwen/Qwen3.8-27B

Apple Metal

brew install zml/zml/llmd
llmd --model=hf://nvidia/Qwen3.8-27B-NVFP4 --batch-size=1

Laguna S 2.1

Tue, 21 Jul 2026 18:00:00 +0000

ZML - Model to Metal

Back to posts

Article

Poolside's open-weight model for agentic coding and long-horizon work.

July 21, 2026 inference / performance / NVIDIA / AMD / TPU / Intel / Apple / Metal / poolside / laguna

In close collaboration with Poolside, we’re happy to announce the support of Laguna S 2.1, their new open-weight model for agentic coding and long-horizon work.

It is a 118B total parameters Mixture-of-Experts model with 8B activated parameters per token and supports a context window of up to 1M tokens, available in thinking and no-thinking modes.

And it’s available today in ZML/LLMD with DFlash support. Most importantly (and as always with us), it’s available on all the supported platforms: NVIDIA, AMD, Intel, Google TPU and Apple Metal.

Run it today

NVIDIA

docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \
    --model=hf://poolside/Laguna-S-2.1 \
    --dflash-model=hf://poolside/Laguna-S-2.1-DFlash

AMD

docker run -p 8000:8000 --shm-size=256GB --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \
    --model=hf://poolside/Laguna-S-2.1 \
    --dflash-model=hf://poolside/Laguna-S-2.1-DFlash

Intel

docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \
    --model=hf://poolside/Laguna-S-2.1 \
    --dflash-model=hf://poolside/Laguna-S-2.1-DFlash

Google TPU

docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \
    --model=hf://poolside/Laguna-S-2.1 \
    --dflash-model=hf://poolside/Laguna-S-2.1-DFlash

Apple Metal

brew install zml/zml/llmd
llmd --model=hf://poolside/Laguna-S-2.1 --dflash-model=hf://poolside/Laguna-S-2.1-DFlash

Benchmarks

It’s a good model, sir.

ZML/LLMD alpha

Wed, 8 Jul 2026 07:00:00 +0000

Today we’re releasing ZML/LLMD. It’s a self-contained inference server that runs LLaMa, Gemma, Qwen and Mistral LLMs transparently on 5 architectures: NVIDIA CUDA, AMD ROCm, Google TPU, Intel oneAPI and Apple Metal.

Modern serving features

ZML/LLMD supports modern serving features: continuous batching, paged attention, tensor parallel sharding, prefix caching, tool calling and does so on all platforms.

Metrics are also exposed in the Prometheus format via the /metrics endpoint.

Supported models

ZML/LLMD alpha ships with support for the following models:

Qwen 2, 3, 3.5 and 3.6 series (dense and MoE)
Gemma 3 and 4 series (dense and MoE)
LFM2.5 series
Mistral 3 and Ministral series
LLaMa 2 and 3 series

DeepSeek, Kimi, GLM, MiniMax and StepFun models are coming soon.

Native HuggingFace, S3 and GCS support

Built on ZML, ZML/LLMD has native support for ZML’s VFS subsystem, which allows for zero-copy loading of models from HuggingFace (hf://), S3 (s3://), and GCS (gs://). This means that you can load models directly from these sources without having to download them first, saving time and storage space. Authentication is handled with the standard environment or paths for each backend.

For instance, loading from HuggingFace is as simple as using the hf:// prefix on any flag that expects a path:

$ docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \
    --model=hf://Qwen/Qwen3-8B

Notice we’re also passing the HF_TOKEN environment variable to authenticate.

DFlash speculative decoding

ZML/LLMD alpha ships with native support for DFlash on Gemma 4 series and soon Qwen series models. DFlash is a new speculative decoding algorithm that can speed up tok/s/user by up to 10x on supported models.

To use it, just pass the --dflash-model flag when launching LLMD:

$ docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \
    --model=hf://google/gemma-4-31B-it \
    --dflash-model=hf://z-lab/gemma-4-31B-it-DFlash

One nice benefit of ZML’s programming model is the compounding of features across platforms. For instance, DFlash transparently runs very well on Intel and Apple GPUs.

Read about DFlash on the Z-Lab blog and the currently released DFlash drafters.

Optimized Docker image

On each platform, ZML/LLMD ships the platform runtime as part of the image in a sandboxed manner. Per file optimization reduces image sizes by 5-10x. On top of that careful ordering of layers and custom compression improves docker pull time by using download/extraction overlapping.

The result is a small image that can be pulled and run in seconds:

$ time docker pull zmlai/llmd:cuda
cuda: Pulling from zmlai/llmd
...
Status: Downloaded newer image for zmlai/llmd:cuda
docker.io/zmlai/llmd:cuda

real    0m13.766s
user    0m0.021s
sys     0m0.022s

Platform	Image	Size
CUDA	`zmlai/llmd:cuda`	1.7 GB
ROCm	`zmlai/llmd:rocm`	3.9 GB
TPU	`zmlai/llmd:tpu`	280 MB
OneAPI	`zmlai/llmd:oneapi`	350 MB
Metal	`brew install zml/zml/llmd`	140 MB

Each image ships with everything it needs to run. For instance on Metal the Apple Metal compiler is shipped in the archive, or ROCm itself in the AMD image.

Built-in CUDA Compatibility

On the CUDA platform, ZML/LLMD ships with the CUDA Compatibility layer transparently built-in that enables ZML/LLMD to run on a wide range of driver versions without intervention.

This feature doesn’t require any special configuration as the probe is done automatically at runtime. It is also fully sandboxed and doesn’t require any system libraries to be installed.

Automatic sharding support

ZML/LLMD supports tensor parallel inference on all platforms. It automatically shards the model across multiple devices and handles the communication between them transparently.

Currently only tensor or expert parallel sharding is supported. More control will be added in the near future, for instance pipeline parallelism.

Python-free Execution Path

ZML/LLMD runs on ZML’s own ML framework, ZML, built with Zig, MLIR, OpenXLA. It compiles binaries and runtimes ahead of time into a single, hermetic artifact. Since the ZML programming model is explicit compilation passes, once compiled, there are no hidden compilation happening in the hot path. Latencies are flat and predictable.

Performance

google/gemma-4-26B-A4B-it (bs=16)

NVIDIA H100x2	Avg	Min	Max	P50	P75	P90	P95	P99	P99.9
TTFT (ms)	127.64	70.39	162.08	141.91	142.04	142.1	143.1	158.28	161.7
ITL (ms)	12.03	0.01	85.58	11.67	12.28	13.14	16.35	68.24	74.42
Total tok/s	1317.93

Intel B70x2	Avg	Min	Max	P50	P75	P90	P95	P99	P99.9
TTFT (ms)	426.05	117.36	1027.4	375.38	375.4	612.75	850.48	992.01	1023.86
ITL (ms)	81.83	0.03	499.75	81.62	82.72	83.7	84.17	84.6	279.99
Total tok/s	179.31

AMD MI300X	Avg	Min	Max	P50	P75	P90	P95	P99	P99.9
TTFT (ms)	101.86	36.3	145.35	109.7	109.76	110.39	116.76	139.63	144.77
ITL (ms)	18.52	0.02	69.33	18.4	18.82	19.12	19.26	40.41	47.75
Total tok/s	858.8

M3 Max	Avg	Min	Max	P50	P75	P90	P95	P99	P99.9
TTFT (ms)	2364.71	1840.79	2608.73	2411.42	2411.5	2470.67	2539.7	2594.92	2607.35
ITL (ms)	67.01	25.81	196.99	67.02	70.63	74.5	77.41	85.19	104.45
Total tok/s	109.59

google/gemma-4-31B-it + z-lab/gemma-4-31B-it-DFlash (bs=16)

TPU v6ex4	Avg	Min	Max	P50	P75	P90	P95	P99	P99.9
TTFT (ms)	137.64	25.36	187.67	137.79	137.82	158.75	187.17	187.57	187.66
ITL (ms)	16.73	0.01	345.35	0.02	0.08	71.85	73.76	75.63	323.83
Total tok/s	943.7

AMD MI300X	Avg	Min	Max	P50	P75	P90	P95	P99	P99.9
TTFT (ms)	254.95	57.23	368.84	268.09	268.14	268.23	283.34	351.74	367.13
ITL (ms)	21.51	0.01	140.15	0.02	0.03	91.55	93.82	109.29	139.67
Total tok/s	731.8

Qwen/Qwen3.6-27B (bs=8)

M3 Max	Avg	Min	Max	P50	P75	P90	P95	P99	P99.9
TTFT (ms)	6052.4	2684.92	7285.46	6408.14	6408.16	6671.36	6978.41	7224.05	7279.32
ITL (ms)	183.96	3.33	876.88	183.83	185.05	186.34	187.62	196.88	208.71
Total tok/s	33.5

AMD MI300X	Avg	Min	Max	P50	P75	P90	P95	P99	P99.9
TTFT (ms)	259.71	68.5	357.55	241.55	270.58	357.51	357.54	357.55	357.55
ITL (ms)	30.97	0.01	115.9	31.02	31.15	31.24	31.29	31.42	34.38
Total tok/s	511.6

Fast cold start

ZML/LLMD has been designed from the ground up to have fast cold start times, as shown here with Qwen 3.6-27B BF16:

Give it a try

Run ZML/LLMD and point your favorite harness to it.

NVIDIA

docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd:cuda \
    --model=hf://Qwen/Qwen3.6-27B

AMD

docker run -p 8000:8000 --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \
    --model=hf://Qwen/Qwen3.6-27B

Intel

docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \
    --model=hf://Qwen/Qwen3.6-27B

Google TPU

docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \
    --model=hf://Qwen/Qwen3.6-27B

Apple Metal

brew install zml/zml/llmd
llmd --model=hf://Qwen/Qwen3.6-27B

10x faster tokenization

Tue, 7 Apr 2026 08:00:00 +0000

Article

Integrating with the IREE tokenizer for a 10x uplift in tokenization performance.

April 7, 2026 monitoring / iree / tokenizer / tiktoken / huggingface / tokenizers / performance

Today we merged a pretty nice performance improvement to ZML.

PR #416 switches zml.tokenizer.Tokenizer from using the Hugging Face tokenizers crate to using the IREE project tokenizer when using Hugging Face tokenizer.json files.

The (upcoming) problem

Tokenizers performance is often overlooked, but it can have a significant impact on the overall latency of LLM inference. In some cases, it can even become the bottleneck of the entire system. This is particularly important as context are becoming bigger and bigger. For code generation, a 1M context window is regular. But a 1M context means 4MB of data (u32).

A bad (de)tokenization strategy can severely impact the overall latency of the system, and as such, it’s crucial to have a fast and efficient tokenizer.

ZML Tokenizers

ZML handles 2 types of tokenizers: Hugging Face tokenizer.json files and SentencePiece models. This is exposed through the zml.tokenizer.Tokenizer union type, which makes this transparent to the user:

var tokenizer: zml.tokenizer.Tokenizer = try .fromFile(allocator, io, "tokenizer.json");
defer tokenizer.deinit();
const token_id = tokenizer.tokenId("<|im_start|>") orelse return error.TokenNotFound;

The zml.tokenizer.Tokenizer types also support the Encoder and Decoder types for streaming purposes. We are also iterating on a std.Io.Reader/Writer based APIs for great composability.

For SentencePiece, we use the official google/sentencepiece implementation.

For Hugging Face, we leverage the tokenizers crate via a C trampoline.

The IREE tokenizer

The IREE tokenizer, introduced by Stella Laurenzo in her LinkedIn post is a C reimplementation of the Hugging Face and TikToken tokenizers. The interesting part is that it was generated by agents, orchestrated and reviewed by Ben Vanik.

It also happens to be 10x faster than everything else.

We’re sold.

Easily integrated via Bazel

At its core, the IREE tokenizer is a pretty straightforward C API. Simple right? Well, as always, there are strings attached. However, because ZML and IREE are built with Bazel, and thanks to our native translate-c support in rules_zig, integration was super easy.

IREE being a pretty big project. We had to be careful to only pull in the tokenizer and its dependencies. This is done via a git sparse checkout and a little patch to shim or remove a few graph dependencies. No biggie.

After that, we just had to depend on a few targets, bind it to our abstractions and we were good to go.

Benchmarks

With all said and done, here are the benchmarks when integrated in ZML with a ~100k tokens text:

Model	Hugging Face (ms)	IREE oneshot / stream (ms)	Speedup
liquidai/LFM2.5-1.2B-Instruct	40.5	8.5 / 7.3	4.8x / 5.5x
Qwen/Qwen3.5-9B	59.4	6.0 / 8.2	9.9x / 7.2x
mistralai/Ministral-3-3B-Reasoning-2512	59.2	7.4 / 6.9	8x / 8.6x

Sunsetting the Rust implementation

We’re pretty happy with the IREE tokenizer. It has been a long discussion at ZML on wether we should rewrite our own implementation in Zig, mostly because we had some ideas to optimize the tokenization process. That said, the performance uplift is so significant that we don’t think it’s worth it for now.

As such, we’ll be removing the Rust implementation and it will be done in a separate PR.

Happy hacking!

Introducing zml-smi

Mon, 30 Mar 2026 15:00:00 +0000

zml-smi is a universal diagnostic and monitoring tool for GPUs, TPUs and NPUs. It provides real-time insights into the performance and health of your hardware.

It is a mix between nvidia-smi and nvtop.

It transparently supports all the platforms ZML supports. That is NVIDIA, AMD, Google TPU and AWS Trainium devices. It will be extended to support more platforms in the future as ZML continues to expand its hardware support.

Getting started

You can download zml-smi from the official mirror.

$ curl -LO 'https://mirror.zml.ai/zml-smi/zml-smi-v0.2.tar.zst'
$ tar -xf zml-smi-v0.2.tar.zst
$ ./zml-smi/zml-smi

Listing devices

$ zml-smi

Monitoring devices

The --top flag provides real-time monitoring of device performance, including utilization, temperature, and memory usage.

$ zml-smi --top

Completely sandboxed

zml-smi doesn’t require any software on the target machine besides the device driver and the GLIBC (mostly due to the fact that some shared objects from vendors are loaded).

Metrics

Host

zml-smi displays host-level metrics such as CPU model and utilization, memory usage, and temperature.

Available metrics

Hostname, Kernel, CPU Model, CPU Core Count, Memory Used / Total, Uptime, Load Average (1m / 5m / 15m), Device Count

Processes

zml-smi also provides insights into the processes utilizing the devices, including their resource usage and command lines. This is available for all platforms.

Available metrics

PID, Device Index, Device Utilization, Device Memory, Process Command Line

NVIDIA

Metrics are given through the NVML library, which ships with the driver. As such, it is expected to be on the system.

Available metrics

GPU Utilization, Temperature, Power Draw, Encoder Utilization, Decoder Utilization, VRAM Used, VRAM Total, Memory Bus Width, Temperature, Fan Speed, Power Draw, Power Limit, Graphics Clock, SM Clock, Memory Clock, Max Graphics Clock, Max Memory Clock, PCIe Link Generation, PCIe Link Width, PCIe TX Throughput, PCIe RX Throughput

AMD

Metrics are provided through the AMD SMI library. zml-smi ships with it in its sandbox.

In order to support the latest AMD GPUs, zml-smi at build time downloads the amdgpu.ids file from both Mesa and ROCm (7.2.1 at the time of this article) and merges them together. This allows zml-smi to recognize and report on the latest AMD GPU models, even if they are not yet included in the official ROCm release. This is the case for Ryzen AI Max+ 395 (Strix Halo) for instance.

Sandboxing that file turned somewhat tricky. Because libdrm-amdgpu expects to find it in /opt/amdgpu/share/libdrm/amdgpu.ids, we had to get a bit creative. We didn’t want to install anything outside the binary sandbox. Nor did we want to patch that string inside libdrm.

So we created a shared object named zmlxrocm.so that is added to the DT_NEEDED section of libdrm_amdgpu.so.1. Then, fopen64 is renamed to zmlxrocm_fopen64, which is then provided by zmlxrocm.so. Since we now sit between libdrm and fopen64, we can intercept the call to fopen64, compare the path against /opt/amdgpu/share/libdrm/amdgpu.ids and redirect it to the sandboxed copy of the file.

Available metrics

GPU Utilization, Memory Usage, Temperature, Power Draw, VRAM Used, VRAM Total, Temperature, Fan Speed, Power Draw, Power Limit, Graphics Clock, SoC Clock, Memory Clock, Max Graphics Clock, Max Memory Clock, PCIe Bandwidth, PCIe Link Generation, PCIe Link Width

TPU

Metrics are provided via the local gRPC endpoint exposed by the TPU runtime. Those are the same metrics exposed to the tpu-info tool from Google.

Available metrics

TensorCore Duty Cycle, HBM Used, HBM Total

AWS Trainium

Metrics are provided through a private API found in libnrt.so, which zml-smi embeds in its sandbox. Those are the same metrics provided by the neuron-top utility.

Available metrics

Core Utilization, HBM Used, HBM Total, Tensor Memory, Constant Memory, Model Code, Shared Scratchpad, Nonshared Scratchpad, Runtime Memory, Driver Memory, DMA Rings, Collectives, Notifications

Introducing ZML/v2

Tue, 24 Mar 2026 15:00:00 +0000

ZML is an inference stack built close to the hardware. It lowers models directly onto NVIDIA, AMD, TPU, and Trainium targets from a single codebase, without depending on and suffering from the Python-heavy runtime layers that most of the ecosystem is built around.

The guiding idea behind zml/v1 was simplicity: give ZML a model and its weights, and the system would take care of compilation, placement, and execution for you. That made the first version approachable and effective for standard deployments, but it also baked too much behavior into implicit global state. As the project pushed into partial compilation, custom passes, sharding, quantization, and more backend-specific execution paths, those implicit shortcuts became constraints. ZML/v2 is the rewrite that makes those concepts explicit: platform ownership, compilation, memory, IO, and placement are now first-class, so advanced use cases can be expressed directly instead of forced through workarounds.

Composability as a core principle

ZML/v2 is a complete rewrite of the framework with a focus on developer experience, performance and most importantly composability. This is a fundamental departure from the original design which was a lot more turnkey but could make more complex programs a bit harder than necessary.

So we’ve decided to strongly orient the design towards composability while also offering an uplift in performance.

Explicit over implicit

The new API centers around zml.Platforms, which is the main abstraction across accelerators, runtimes, IO etc. In ZML, a platform is instantiated and from then on used to transfer data, build, compile and execute programs.

This is how it all starts:

const platform: *zml.Platform = try .auto(allocator, io, .{});

ZML offers various platforms selection mechanisms: compile-time, runtime and manual. The automatic runtime selection is the default and will pick the best available platform on the system, but users can also choose to compile for a specific platform or manually create a platform with custom options.

The then built program can now run transparently on all comptime-time enabled platforms, which can be enabled via the --@zml//platforms:{cuda,rocm,tpu,neuron}=true build flags. Doing so will download the runtime, strip it to its bare minimum and embed it inside the binary sandbox.

For instance, here is the entire CUDA sandbox:

$ tree --du -h bazel-bin/examples/llm/llm.runfiles/+cuda_packages+libpjrt_cuda/sandbox/
[2.9G]  bazel-bin/examples/llm/llm.runfiles/+cuda_packages+libpjrt_cuda/sandbox/
├── [ 82M]  bin
│   ├── [3.0M]  compat_probe
│   ├── [ 40M]  nvlink
│   └── [ 39M]  ptxas
├── [2.8G]  lib
│   ├── [409M]  compat
│   │   ├── [ 11M]  libcudadebugger.so.1
│   │   ├── [ 94M]  libcuda.so.1
│   │   ├── [ 86M]  libnvidia-gpucomp.so.590.48.01
│   │   ├── [ 24M]  libnvidia-nvvm70.so.4
│   │   ├── [ 75M]  libnvidia-nvvm.so.4
│   │   ├── [ 14K]  libnvidia-pkcs11-openssl3.so.590.48.01
│   │   ├── [ 31M]  libnvidia-ptxjitcompiler.so.1
│   │   └── [ 90M]  libnvidia-tileiras.so.590.48.01
│   ├── [479M]  libcublasLt.so.13
│   ├── [ 52M]  libcublas.so.13
│   ├── [740K]  libcudart.so.13
│   ├── [104M]  libcudnn_adv.so.9
│   ├── [2.3M]  libcudnn_cnn.so.9
│   ├── [233M]  libcudnn_engines_precompiled.so.9
│   ├── [ 26M]  libcudnn_engines_runtime_compiled.so.9
│   ├── [3.5M]  libcudnn_graph.so.9
│   ├── [ 56M]  libcudnn_heuristic.so.9
│   ├── [ 37M]  libcudnn_ops.so.9
│   ├── [126K]  libcudnn.so.9
│   ├── [286M]  libcufft.so.12
│   ├── [4.4M]  libcupti.so.13
│   ├── [136M]  libcusolver.so.12
│   ├── [162M]  libcusparse.so.12
│   ├── [218M]  libnccl.so.2
│   ├── [ 95M]  libnvJitLink.so.13
│   ├── [4.2M]  libnvrtc-builtins.so.13.1
│   ├── [109M]  libnvrtc.so.13
│   ├── [ 40M]  libnvshmem_host.so.3
│   ├── [ 39K]  libnvtx3interop.so
│   ├── [383M]  libpjrt_cuda.so
│   ├── [2.9M]  libzmlxcuda.so.0
│   ├── [118K]  libz.so.1
│   ├── [ 72K]  nvshmem_bootstrap_uid.so.3
│   └── [1023K]  nvshmem_transport_ibrc.so.4
└── [ 77M]  nvvm
    ├── [ 77M]  bin
    │   └── [ 77M]  cicc
    └── [458K]  libdevice
        └── [454K]  libdevice.10.bc

Pinned Memory Allocators

ZML/v2 enables the explicit allocation and use of pinned memory. Pinned memory is memory that is not movable nor swappable and thus can be directly accessed by the DMA engine of the accelerators. This functionality is exposed via the zml.mem.DmaAllocator which, combined with Zig’s explicit allocator model, enables unique optimizations and removes extraneous copies found in other frameworks.

Zero and overlapped copy primitives

Data transfers between host and accelerator now compose with std.Io.Writer via zml.io.MemoryWriter, enabling zero-copy and overlapped data movement across the board.

For instance, loading a model from disk to GPU memory now happens at SSD speed when using O_DIRECT, even with staging through the system memory, since transfers are overlapped. This enables NVIDIA GPUDirect-like performance on raw POSIX.

$ bazel build --config=release //examples/llm --@zml//platforms:cuda=true
$ bazel-bin/examples/llm/llm --model=/var/models/meta-llama/Llama-3.1-8B-Instruct/
...
info(llm): Selected backend: .cuda_fa2
info(llm): Loaded tokenizer [181.256ms]
info(llama): Loaded weights [14.96GiB, 1.165s, 12.83GiB/s]

Userland Virtual File System

Thanks to injectable std.Io, ZML implements a full userland VFS built into the program. It supports local files, HTTP endpoints, S3 buckets and Hugging Face repositories. Concretely, this means that models can be loaded directly from each of those sources without downloading to disk first, regardless of the underlying FS implementation.

Let’s try to register an HF implementation and load the tokenizer directly from Hugging Face:

var vfs: zml.io.VFS = try .init(init.gpa, init.io);
defer vfs.deinit();

var hf_vfs: zml.io.VFS.HF = try .auto(init.gpa, init.io, &http_client, init.environ_map);
defer hf_vfs.deinit();
try vfs.register("hf", hf_vfs.io());

// Use the VFS IO from now on
const io = vfs.io();

const tokenizer: zml.Tokenizer = blk: {
    const file = try std.Io.Dir.openFile(.cwd(), io, "hf://Qwen/Qwen3-8B/tokenizer.json", .{});
    defer file.close(io);
    var reader = file.reader(init.gpa, &.{});
    const bytes = try reader.interface.readAlloc(init.gpa, try file.length(io));
    defer init.gpa.free(bytes);
    break :blk try .fromBytes(init.gpa, io, bytes);
};

Or load the model directly from Hugging Face:

$ bazel-bin/examples/llm/llm --model=hf://meta-llama/Llama-3.1-8B-Instruct

Powerful sharding primitives

Sharding is now a first-class citizen of the programming model instead of an implicit backend detail. We introduce a new public zml.sharding API with PhysicalMesh, LogicalMesh, Strategy, Sharding, and Placement. Compile / load / buffer APIs now take explicit sharding information and finally zml.ops.manualComputation expresses a shard-local primitive similar to JAX’s shard_map.

It also fully leverages Shardy or GSPMD based on the target platform transparently.

Pluggable Attention backend

ZML/v2 provides a pluggable attention backend system, giving a clean way to select and configure different attention implementations without rewriting model code around each kernel.

Instead of baking backend-specific logic into every call site, zml.attention exposes an explicit Backend plus backend-typed Parameters, Metadata, and paged-attention Context, so the same high-level attention path can target vanilla behavior or optimized CUDA FlashAttention variants like Tri-Dao’s FA2 and FA3.

The backend is selected automatically based on the platform and accelerator.

In this release, we took the opportunity to integrate FlashAttention 2 and 3 built on CUDA for sm80 up to sm121.

Complete build sandboxing

Our build now runs entirely on the 100% hermetic LLVM toolchain @llvm. This enables full reproducibility, better cross-compilation support, and a more consistent development experience across platforms. This means that ZML supports remote execution on services such as BuildBuddy or NativeLink.

On macOS, we no longer rely on Xcode nor Command Line Tools to build ZML. The macOS SDK is downloaded as part of the build.

Cross compilation is also fully supported, enabling fully working containers for any supported ZML platform to be built right from a Mac.

//examples/llm

Try it today via the llm CLI:

$ bazel build --config=release //examples/llm --@zml//platforms:cuda=true
$ bazel-bin/examples/llm/llm --model=hf://Qwen/Qwen3.5-9B