---
title: 10x faster tokenization
link: https://zml.ai/posts/iree-tokenizer/
source: zml-ai
published: 2026-04-07T08:00:00Z
updated: 2026-04-07T08:00:00Z
first_seen: 2026-09-08T10:17:10.166315Z
summary: 'Today we merged a pretty nice performance improvement to ZML. PR #416 switches zml.tokenizer.Tokenizer from using the Hugging Face tokenizers crate to using the IREE project tokenizer when using Hugging Face tokenizer.json files. The (upcoming) problem Tokenizers performance is often overlooked, but it can have a significant impact on the overall latency of LLM inference. In some cases, it can even become the bottleneck of the entire system. This is particularly important as context are becoming bigger and bigger. For code generation, a 1M context window is regular. But a 1M context means 4MB of data (u32).'
content: extracted
html: 2026-04-07-10x-faster-tokenization.html
preview:
  file: 2026-04-07-10x-faster-tokenization.preview-76d49b5db3f0.webp
  width: 256
  height: 64
  color: '#282c37'
images:
- source: https://zml.ai/posts/iree-tokenizer/1-benchmark.jpg
  original:
    file: 2026-04-07-10x-faster-tokenization.image-ed32a0c20b69.jpg
    width: 800
    height: 200
  variants:
  - file: 2026-04-07-10x-faster-tokenization.image-39c6feb98fd0.webp
    width: 48
    height: 12
  color: '#0d1017'
---

Article

Integrating with the IREE tokenizer for a 10x uplift in tokenization performance.

April 7, 2026 monitoring / iree / tokenizer / tiktoken / huggingface / tokenizers / performance

Today we merged a pretty nice performance improvement to ZML.

[PR #416](https://github.com/zml/zml/pull/416) switches `zml.tokenizer.Tokenizer` from using the Hugging Face [`tokenizers`](https://crates.io/crates/tokenizers) crate to using the [IREE project tokenizer](https://github.com/iree-org/iree/blob/main/runtime/src/iree/tokenizer/README.md) when using Hugging Face `tokenizer.json` files.

![](https://zml.ai/posts/iree-tokenizer/1-benchmark.jpg)

## The (upcoming) problem

Tokenizers performance is often overlooked, but it can have a significant impact on the overall latency of LLM inference. In some cases, it can even become the bottleneck of the entire system. This is particularly important as context are becoming bigger and bigger. For code generation, a 1M context window is regular. But a 1M context means 4MB of data (`u32`).

A bad (de)tokenization strategy can severely impact the overall latency of the system, and as such, it’s crucial to have a fast and efficient tokenizer.

## ZML Tokenizers

ZML handles 2 types of tokenizers: Hugging Face `tokenizer.json` files and SentencePiece models. This is exposed through the `zml.tokenizer.Tokenizer` union type, which makes this transparent to the user:

```zig
var tokenizer: zml.tokenizer.Tokenizer = try .fromFile(allocator, io, "tokenizer.json");
defer tokenizer.deinit();
const token_id = tokenizer.tokenId("<|im_start|>") orelse return error.TokenNotFound;
```

The `zml.tokenizer.Tokenizer` types also support the `Encoder` and `Decoder` types for streaming purposes. We are also iterating on a `std.Io.Reader/Writer` based APIs for great composability.

For SentencePiece, we use the official [google/sentencepiece](https://github.com/google/sentencepiece) implementation.

For Hugging Face, we leverage the [tokenizers crate](https://crates.io/crates/tokenizers) via a [C trampoline](https://github.com/zml/zml/blob/70c5d1c5883813e13481cf35fab841f8a18152aa/zml/tokenizer/hftokenizers/hftokenizers.rs).

## The IREE tokenizer

The IREE tokenizer, introduced by [Stella Laurenzo in her LinkedIn post](https://www.linkedin.com/posts/stellalaurenzo_fastest-llm-tokenizer-on-the-internet-authored-activity-7435479758118989825--wDF) is a C reimplementation of the Hugging Face and TikToken tokenizers. The interesting part is that it was generated by agents, orchestrated and reviewed by Ben Vanik.

**It also happens to be 10x faster than everything else.**

![](https://zml.ai/posts/iree-tokenizer/1-benchmark.jpg)

We’re sold.

## Easily integrated via Bazel

At its core, the IREE tokenizer is a pretty straightforward C API. Simple right? Well, as always, there are strings attached. However, because ZML and IREE are built with Bazel, and thanks to our [native `translate-c` support in rules\_zig](https://github.com/zml/rules_zig/blob/zig-0.16/zig/private/common/translate_c.bzl), integration was super easy.

IREE being a pretty big project. We had to be careful to only pull in the tokenizer and its dependencies. This is done via a [git sparse checkout](https://git-scm.com/docs/git-sparse-checkout) and a [little patch](https://github.com/zml/zml/blob/master/third_party/iree/tokenizer-only.patch) to shim or remove a few graph dependencies. No biggie.

After that, we just had to [depend on a few targets](https://github.com/zml/zml/blob/adee932efa875d181cfc3ec8cbc4eb822db76035/zml/tokenizer/iree/BUILD.bazel#L9-L11), bind it to our abstractions and we were good to go.

## Benchmarks

With all said and done, here are the benchmarks when integrated in ZML with a ~100k tokens text:

| Model                                   | Hugging Face (ms) | IREE oneshot / stream (ms) | Speedup         |
| --------------------------------------- | ----------------- | -------------------------- | --------------- |
| liquidai/LFM2.5-1.2B-Instruct           | 40.5              | 8.5 / 7.3                  | **4.8x / 5.5x** |
| Qwen/Qwen3.5-9B                         | 59.4              | 6.0 / 8.2                  | **9.9x / 7.2x** |
| mistralai/Ministral-3-3B-Reasoning-2512 | 59.2              | 7.4 / 6.9                  | **8x / 8.6x**   |

## Sunsetting the Rust implementation

We’re pretty happy with the IREE tokenizer. It has been a long discussion at ZML on wether we should rewrite our own implementation in Zig, mostly because we had some ideas to optimize the tokenization process. That said, the performance uplift is so significant that we don’t think it’s worth it for now.

As such, we’ll be removing the Rust implementation and it will be done in a separate PR.

Happy hacking!