![\"thumbnail\"](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/thumbnail.png\")
Transformers v5 redesigns how tokenizers work. The big tokenizers reformat separates tokenizer design from trained vocabulary (much like how PyTorch separates neural network architecture from learned weights). The result is tokenizers you can inspect, customize, and train from scratch with far less friction.
\n\n\nTL;DR: This blog explains how tokenization works in Transformers and why v5 is a major redesign, with clearer internals, a clean class hierarchy, and a single fast backend. It’s a practical guide for anyone who wants to understand, customize, or train model-specific tokenizers instead of treating them as black boxes.
\n
tokenizers through transformerstransformersAutoTokenizer Automatically Selects the Correct Tokenizer Class\n\nFor experts: If you're already familiar with the concepts and want to understand the changes in v5, go to v5 Separates Tokenizer Architecture from Trained Vocab
\n
Before diving into the changes, let's quickly cover what tokenization does and how the pieces fit together.
\nLanguage models don't read raw text. They consume sequences of integers usually called token IDs or input IDs. Tokenization is the process of converting raw text into these token IDs. (Try the tokenization playground here to visualize tokenization.)
\nTokenization is a broad concept used across natural language processing and text processing generally. This post focuses specifically on tokenization for Large Language Models (LLMs) using the transformers and tokenizers libraries.
\n = \n\n = \n = \n\n\n# [9906, 1917]\n\n\n# ['Hello', 'Ġworld']\n\n\n\n
Ġworld(above) is a single token that represents the character sequence " world" (with the space).
A token is the smallest string unit the model sees. It can be a character, word, or subword chunk like "play" or "##ing" ("##" is a pattern, don't worry if you don't completely understand it now 🤗). The vocabulary maps each unique token to the token ID.
\n\n\n = \n\n\n# {'ÎĹÎľ': 106502, 'ĠPeel': 89694, '.languages': 91078, ...}\nA good tokenizer compresses text into the smallest amount of tokens. Fewer tokens means more usable context without increasing model size. Training a tokenizer boils down to finding the best compression rules for your datasets. For example, if you work on Chinese corpus you can sometimes find very nice surprises 😉.
\nTokenization happens in stages. Each stage transforms text before passing it to the next:
\n| Stage | \nPurpose | \nExample | \n
|---|---|---|
| Normalizer | \nStandardizes text (lowercasing, unicode normalization, whitespace cleanup) | \n"HELLO World" → "hello world" | \n
| Pre-tokenizer | \nSplits text into preliminary chunks | \n"hello world" → ["hello", " world"] | \n
| Model | \nApplies the tokenization algorithm (BPE, Unigram, etc.) | \n["hello", " world"] → [9906, 1917] | \n
| Post-processor | \nAdds special tokens (BOS, EOS, padding) | \n[9906, 1917] → [1, 9906, 1917, 2] | \n
| Decoder | \nConverts token IDs back to text | \n[9906, 1917] → "hello world" | \n
Each component is independent. You can swap normalizers or change the algorithm without rewriting everything else.
\n\n\nYou can access the rust based tokenizer through
\n_tokenizer. We go in more depth about it in this section
\n\n = \n\n\n# Replace(...)\n\n\n# Split(...)\n\n\n# BPE(...)\n\n\n# TemplateProcessing(...)\n\n\n# Sequence(decoders=[Replace(...), ByteFallback(), Fuse()])\nThe following algorithms dominate modern language model tokenizers:
\n\n\n = \n\n\n# BPE(...)\n\n\n = \n\n\n# Unigram(...)\n\n\n = \n\n\n# WordPiece(...)\nThe tokenizers library is a Rust-based tokenization engine. It is fast, efficient, and completely language model agnostic. The library handles the mechanics of converting text into token IDs and back. The tokenizers library is a general-purpose tool that implements the tokenization algorithms, but does not implement the conventions that connect those algorithms to specific language models.
Consider what happens when you use tokenizers directly with the SmolLM3-3B model:
\n\n = \n = \n = \n\n\n# [9906, 1917]\n\n# ['Hello', 'Ġworld']\nThe output is raw tokenization. You get token IDs and the string pieces they correspond to. Nothing more.
\nNow consider what's missing. The SmolLM3-3B is a conversational model. When you interact with it, you typically structure your input as a conversation with roles like "user" and "assistant". The language model expects special formatting tokens to indicate these roles. The raw tokenizers library has no concept of any of this.
The transformers library bridges this gap. The library is primarily known as a model definition library, but it also provides a tokenizer abstraction layer that wraps the raw tokenizers backend and adds model-aware functionality.
Here's the same tokenization with the transformers wrapper:
\n\n = \n\n# Format a conversation using the model's chat template\n = \n = \n = \n\n\n\n# <|im_start|>system\n# ...\n# <|im_start|>user\n# Give me a brief explanation of gravity in simple terms.<|im_end|>\n# <|im_start|>assistant\n\n = \nNotice how the special tokens like <|im_start|> and <|im_end|> are applied to the prompt before tokenizing. This is useful for the model to learn where a new sequence starts and ends.
The transformers tokenizer adds everything the raw library lacks:
apply_chat_template method formats conversations according to the model's expected format, inserting the correct special tokens and delimiters.truncation=True and the tokenizer will respect the model's maximum sequence length.return_tensors="pt"), NumPy arrays and others.\n\n\n
transformersimplements the tokenization API that is most commonly used in the entire ML community (encode,decode,convert_tokens_to_ids, etc.)
The transformers library organizes tokenizers into a class hierarchy. At the top sits a base class that defines the common interface. Below it, backend classes handle the actual tokenization using different engines. At the bottom, model-specific classes configure the backends for particular models.
![\"class](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/hierarchy.png\") | \n
|---|
| The class hierarchy for tokenizers inside transformers | \n
PreTrainedTokenizerBase defines the common interface for all tokenizersPreTrainedTokenizerBase is the abstract base class for all tokenizers in transformers. It defines the interface that every tokenizer must implement.
The base class handles functionality that doesn't depend on the tokenization backend:
\nbos_token, eos_token, pad_token, and unk_token are defined here. These properties provide access to the special tokens that models use to mark sequence boundaries and handle unknown inputs.__call__ method, encode, and encode_plus methods are defined here. These methods accept text input and return token IDs along with attention masks and other metadata.decode and batch_decode methods convert token IDs back to text.save_pretrained and from_pretrained methods handle downloading the correct files, reading information, saving tokenizers to disk etc.apply_chat_template method lives here, formatting conversations according to Jinja templates stored in the tokenizer configuration.Every tokenizer in transformers ultimately inherits from PreTrainedTokenizerBase. The base class ensures consistent behavior across all tokenizers, regardless of which backend they use for the actual tokenization.
TokenizersBackend wraps the tokenizers libraryTokenizersBackend is the primary backend class for most modern tokenizers. It inherits from PreTrainedTokenizerBase and wraps the Rust-based tokenizers library.
The class stores the Rust tokenizer object internally:
\n\n\n = # The Rust tokenizer\n ...\nWhen you call encoding methods on a TokenizersBackend tokenizer, the class delegates the actual tokenization to the Rust backend:
\n = \n ...\nThe Rust backend performs computationally intensive work, while the Python wrapper adds the model-aware features on top.
\nMany model-specific tokenizers inherit from TokenizersBackend, examples include:
LlamaTokenizerGemmaTokenizerThese model-specific classes configure the backend with the correct vocabulary, merge rules, special tokens, and normalization settings for their respective models.
\nPythonBackend provides a pure-Python mixinPythonBackend inherits from PreTrainedTokenizerBase and implements tokenization in pure Python. The class is aliased as PreTrainedTokenizer.
The pure-Python backend exists for several reasons:
\ntokenizers pipeline.\n\nThe Python backend is slower than the Rust backend. For most use cases, the Rust-backed
\nTokenizersBackendis preferred.
Model-specific tokenizers that inherit from PythonBackend (or its alias PreTrainedTokenizer) include some older or specialized models, like:
CTRLTokenizerCanineTokenizerSentencePieceBackend handles SentencePiece modelsSentencePieceBackend inherits from PythonBackend and provides integration with Google's SentencePiece library. SentencePiece is a standalone tokenization library that many models use, particularly those trained by Google.
The backend wraps a SentencePiece processor:
\n\n\n = \n \n ...\nModels that use SentencePiece tokenization inherit from this backend. Examples include:
\nSiglipTokenizerBartphoTokenizerThe SentencePiece backend inherits from PythonBackend rather than directly from PreTrainedTokenizerBase because it shares much of the same interface and padding/truncation logic.
AutoTokenizer is the recommended entry point for loading tokenizers. It automatically determines which tokenizer class to use for a given model and returns an instance of that class.
\n\n = \nBehind the scenes, AutoTokenizer performs these steps:
from_pretrained method fetches tokenizer_config.json from the Hub (or from a local directory).AutoTokenizer maintains a mapping called TOKENIZER_MAPPING_NAMES that maps model types to tokenizer class names: = \nAutoTokenizer imports the appropriate tokenizer class and calls its from_pretrained method.\n\nThe benefit of
\nAutoTokenizeris that you don't need to know which tokenizer class a model uses. Whether a model usesLlamaTokenizer,GPT2Tokenizer, orBertTokenizer, the sameAutoTokenizer.from_pretrained("model-name")call works.
The tokenizer system in transformers forms a layered architecture:
| Layer | \nComponent | \nResponsibility | \n
|---|---|---|
| Entry Point | \nAutoTokenizer | \nAutomatically selects and instantiates the correct tokenizer class | \n
| Model-Specific | \nLlamaTokenizer, GPT2Tokenizer, etc. | \nConfigures the backend with model-specific architecture of normalizer, pre tokenizer, etc, special tokens, and settings | \n
| Backend | \nTokenizersBackend, PythonBackend, SentencePieceBackend | \nImplements the actual tokenization using a specific engine | \n
| Base | \nPreTrainedTokenizerBase | \nDefines the common interface and shared functionality | \n
| Engine | \ntokenizers (Rust), SentencePiece, Pure Python | \nPerforms raw tokenization | \n
The most significant change in Transformers v5 is a philosophical shift in how tokenizers are defined. Tokenizers now work like PyTorch's nn.Module: you define the architecture first, then fill it with learned parameters.
In v4, tokenizers were black boxes tied to pretrained checkpoint files. If you loaded LlamaTokenizerFast, you couldn't easily answer basic questions about it:
The __init__ method gave no clues. You had to dig through serialized files or external documentation to understand what the tokenizer actually did.
![\"v4](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v4-llama.png\") | \n
|---|
LlamaTokenizerFast as seen in v4 transformers | \n
v4 also maintained two parallel implementations for every model:
\nLlamaTokenizer inheriting from PreTrainedTokenizer) andLlamaTokenizerFast inheriting from PreTrainedTokenizerFast).This meant:
\ntokenization_llama.py and tokenization_llama_fast.py)Worst of all, you couldn't create an empty tokenizer architecture. If you wanted to train a LLaMA-style tokenizer on your own data, there was no clean way to instantiate a "blank" LLaMA tokenizer and fill it with your vocabulary and merges. Tokenizers existed only as loaded checkpoints, not as configurable templates.
\nv5 treats tokenizer architecture (normalizer, pre-tokenizer, model type, post-processor, decoder) as distinct from trained parameters (vocabulary, merges). This mirrors how PyTorch separates model architecture from learned weights.
\nWith nn.Module, you define layers first:
\n\n = \n# Architecture defined; weights initialized randomly or loaded later\nV5 tokenizers follow the same pattern:
\n\n\n# Instantiate the architecture\n = \n\n# Train on your own data to fill in vocab and merges\n\nThe tokenizer class now explicitly declares its structure. Looking at LlamaTokenizer in v5, you can immediately see:
unk, bos, eos) sit at specific vocabulary positions▁ with spaces![\"v5](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v5-llama.png\") | \n
|---|
LlamaTokenizer as seen in v5 transformers | \n
This transparency was impossible in v4, where the same information was buried in serialized files.
\nv5 consolidates the two-file system into a single file per model. LlamaTokenizer now inherits from TokenizersBackend, which wraps the Rust-based tokenizer that was previously exposed as the “fast” implementation and is now the default.
The former “slow” Python implementation lives explicitly behind PythonBackend, and SentencePieceBackend remains for models that require it, but Rust-backed tokenization is the preferred default.
This change eliminates:
\nTokenizer vs TokenizerFast naming conventionUsers now have one clear entry point. Advanced users who need to customize can still access lower-level components, but the library no longer forces everyone to navigate two parallel implementations.
\nSuppose you want a tokenizer that behaves exactly like LLaMA's – same normalization, same pre-tokenization, same BPE model type – but trained on a domain-specific corpus (medical text, legal documents, a new language). In v4, this required manually reconstructing the tokenizer pipeline from low-level tokenizers library primitives. In v5, you can instantiate the architecture directly and call train:
\n\n\n# Initialize blank tokenizer\n = \n\n = \n\n\n = 1000\n \n yield \n\n = \n\n\n\n = \nThe resulting tokenizer will have your custom vocabulary and merge rules, but will process text identically to how a standard LLaMA tokenizer would with the same whitespace handling, same special token conventions, same decoding behavior.
\n| Aspect | \nV4 | \nV5 | \n
|---|---|---|
| Files per model | \nTwo (tokenization_X.py, tokenization_X_fast.py) | \nOne (tokenization_X.py) | \n
| Default backend | \nSplit between Python and Rust | \nRust (TokenizersBackend) preferred | \n
| Architecture visibility | \nHidden in serialized files | \nExplicit in class definition | \n
| Training from scratch | \nRequired manual pipeline construction | \ntokenizer.train(files=[...]) | \n
| Component inspection | \nDifficult, undocumented | \nDirect properties (tokenizer.normalizer, etc.) | \n
| Parent classes | \nPreTrainedTokenizer, PreTrainedTokenizerFast | \nTokenizersBackend (or SentencePieceBackend, PythonBackend) | \n
The shift from "tokenizers as loaded checkpoints" to "tokenizers as configurable architectures" makes the library more modular, more transparent, and more aligned with how practitioners think about building ML systems.
\nTransformers v5 brings three improvements to tokenization:
\nThe wrapper layer between tokenizers and Transformers remains essential. It adds model awareness, context lengths, chat templates, special tokens, that raw tokenization doesn't provide. V5 just makes that layer clearer and more customizable.
If you are looking to learn more about tokenization here are some resources:
\n\n", "content_markdown": "[](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/thumbnail.png)\n\n[Transformers v5](https://huggingface.co/blog/transformers-v5) redesigns how tokenizers work. The [big tokenizers reformat](https://github.com/huggingface/transformers/pull/40936/files) separates tokenizer design from trained vocabulary (much like how PyTorch separates neural network architecture from learned weights). The result is tokenizers you can *inspect*, *customize*, and *train* from scratch with far less friction.\n\n> TL;DR: This blog explains how tokenization works in Transformers and why v5 is a major redesign, with clearer internals, a clean class hierarchy, and a single fast backend. It’s a practical guide for anyone who wants to understand, customize, or train model-specific tokenizers instead of treating them as black boxes.\n\n## Table of Contents\n\n- [What is Tokenization?](https://huggingface.co/blog/tokenizers#what-is-tokenization)\n- [The Tokenization Pipeline](https://huggingface.co/blog/tokenizers#the-tokenization-pipeline)\n- [Tokenization Algorithms](https://huggingface.co/blog/tokenizers#tokenization-algorithms)\n- [Accessing `tokenizers` through `transformers`](https://huggingface.co/blog/tokenizers#accessing-tokenizers-through-transformers)\n- [The Tokenizer Class Hierarchy in `transformers`](https://huggingface.co/blog/tokenizers#the-tokenizer-class-hierarchy-in-transformers)\n- [`AutoTokenizer` Automatically Selects the Correct Tokenizer Class](https://huggingface.co/blog/tokenizers#autotokenizer-automatically-selects-the-correct-tokenizer-class)\n- [v5 Separates Tokenizer Architecture from Trained Vocab](https://huggingface.co/blog/tokenizers#v5-separates-tokenizer-architecture-from-trained-vocab)\n- [Summary](https://huggingface.co/blog/tokenizers#summary)\n\n> For experts: If you're already familiar with the concepts and want to understand the changes in v5, go to [v5 Separates Tokenizer Architecture from Trained Vocab](https://huggingface.co/blog/tokenizers#v5-separates-tokenizer-architecture-from-trained-vocab)\n\nBefore diving into the changes, let's quickly cover what tokenization does and how the pieces fit together.\n\n## What is tokenization?\n\nLanguage models don't read raw text. They consume sequences of integers usually called **token IDs or input IDs**. Tokenization is the process of converting raw text into these token IDs. (Try the tokenization playground [here](https://huggingface.co/spaces/Xenova/the-tokenizer-playground) to visualize tokenization.)\n\nTokenization is a broad concept used across natural language processing and text processing generally. This post focuses specifically on tokenization for Large Language Models (LLMs) using the [`transformers`](https://github.com/huggingface/transformers) and [`tokenizers`](https://github.com/huggingface/tokenizers) libraries.\n\n```python\nfrom transformers import AutoTokenizer\ntokenizer = AutoTokenizer.from_pretrained(\"HuggingFaceTB/SmolLM3-3B\")\n\ntext = \"Hello world\"\ntokens = tokenizer(text)\n\nprint(tokens[\"input_ids\"])\n# [9906, 1917]\n\nprint(tokenizer.convert_ids_to_tokens(tokens[\"input_ids\"]))\n# ['Hello', 'Ġworld']\n```\n\n> `Ġworld` (above) is a single token that represents the character sequence \" world\" (with the space).\n\nA **token** is the smallest string unit the model sees. It can be a character, word, or subword chunk like \"play\" or \"##ing\" (\"##\" is a pattern, don't worry if you don't completely understand it now 🤗). The **vocabulary** maps each unique token to the token ID.\n\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"HuggingFaceTB/SmolLM3-3B\")\nprint(tokenizer.vocab)\n\n# {'ÎĹÎľ': 106502, 'ĠPeel': 89694, '.languages': 91078, ...}\n```\n\nA good tokenizer *compresses* text into the smallest amount of tokens. Fewer tokens means more usable context without increasing model size. Training a tokenizer boils down to finding the best compression rules for your datasets. For example, if you work on Chinese corpus you can sometimes find [very nice surprises 😉](https://x.com/suchenzang/status/1697862650053660721).\n\n## The tokenization pipeline\n\nTokenization happens in stages. Each stage transforms text before passing it to the next:\n\n| Stage | Purpose | Example |\n| ------------------ | -------------------------------------------------------------------------- | --------------------------------------- |\n| **Normalizer** | Standardizes text (lowercasing, unicode normalization, whitespace cleanup) | `\"HELLO World\"` → `\"hello world\"` |\n| **Pre-tokenizer** | Splits text into preliminary chunks | `\"hello world\"` → `[\"hello\", \" world\"]` |\n| **Model** | Applies the tokenization algorithm (BPE, Unigram, etc.) | `[\"hello\", \" world\"]` → `[9906, 1917]` |\n| **Post-processor** | Adds special tokens (BOS, EOS, padding) | `[9906, 1917]` → `[1, 9906, 1917, 2]` |\n| **Decoder** | Converts token IDs back to text | `[9906, 1917]` → `\"hello world\"` |\n\nEach component is *independent*. You can swap [normalizers](https://huggingface.co/docs/tokenizers/en/api/normalizers) or change the [algorithm](https://huggingface.co/docs/tokenizers/en/api/models) without rewriting everything else.\n\n> You can access the rust based tokenizer through `_tokenizer`. We go in more depth about it in [this section](https://huggingface.co/blog/tokenizers#tokenizersbackend-wraps-the-tokenizers-library)\n\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"google/gemma-3-270m-it\")\n\nprint(f\"{tokenizer._tokenizer.normalizer=}\")\n# Replace(...)\n\nprint(f\"{tokenizer._tokenizer.pre_tokenizer=}\")\n# Split(...)\n\nprint(f\"{tokenizer._tokenizer.model=}\")\n# BPE(...)\n\nprint(f\"{tokenizer._tokenizer.post_processor=}\")\n# TemplateProcessing(...)\n\nprint(f\"{tokenizer._tokenizer.decoder=}\")\n# Sequence(decoders=[Replace(...), ByteFallback(), Fuse()])\n```\n\n## Tokenization algorithms\n\nThe following algorithms dominate modern language model tokenizers:\n\n1. **Byte Pair Encoding (BPE)** iteratively merges the most frequent character pairs. This algorithm is deterministic and widely used. (Read more about [BPE](https://huggingface.co/learn/llm-course/en/chapter6/5))\n\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"openai/gpt-oss-20b\")\nprint(tokenizer._tokenizer.model)\n\n# BPE(...)\n```\n\n2. **Unigram** takes a probabilistic approach, selecting the most likely segmentation from a large initial vocabulary. This is more flexible than the strict BPE. (Read more about [Unigram](https://huggingface.co/learn/llm-course/en/chapter6/7))\n\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"google-t5/t5-base\")\nprint(tokenizer._tokenizer.model)\n\n# Unigram(...)\n```\n\n3. **WordPiece** resembles BPE but uses different merge criteria based on likelihood. (Read more about [WordPiece](https://huggingface.co/learn/llm-course/en/chapter6/6))\n\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\nprint(tokenizer._tokenizer.model)\n\n# WordPiece(...)\n```\n\n## Accessing tokenizers through transformers\n\nThe [`tokenizers`](https://github.com/huggingface/tokenizers) library is a Rust-based tokenization engine. It is fast, efficient, and completely language model agnostic. The library handles the mechanics of converting text into token IDs and back. The `tokenizers` library is a general-purpose tool that implements the tokenization algorithms, but does not implement the conventions that connect those algorithms to specific language models.\n\nConsider what happens when you use `tokenizers` directly with the [`SmolLM3-3B`](http://hf.co/HuggingFaceTB/SmolLM3-3B) model:\n\n```python\nfrom tokenizers import Tokenizer\n\ntokenizer = Tokenizer.from_pretrained(\"HuggingFaceTB/SmolLM3-3B\")\ntext = \"Hello world\"\nencodings = tokenizer.encode(text)\n\nprint(encodings.ids)\n# [9906, 1917]\nprint(encodings.tokens)\n# ['Hello', 'Ġworld']\n```\n\nThe output is raw tokenization. You get token IDs and the string pieces they correspond to. Nothing more.\n\nNow consider what's missing. The `SmolLM3-3B` is a *conversational model*. When you interact with it, you typically structure your input as a conversation with roles like \"user\" and \"assistant\". The language model expects special formatting tokens to indicate these roles. The raw `tokenizers` library has no concept of any of this.\n\n### How do you bridge the gap between raw tokenization and model requirements?\n\nThe `transformers` library bridges this gap. The library is primarily known as a model definition library, but it also provides a tokenizer abstraction layer that wraps the raw `tokenizers` backend and adds model-aware functionality.\n\nHere's the same tokenization with the `transformers` wrapper:\n\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"HuggingFaceTB/SmolLM3-3B\")\n\n# Format a conversation using the model's chat template\nprompt = \"Give me a brief explanation of gravity in simple terms.\"\nmessages = [{\"role\": \"user\", \"content\": prompt}]\ntext = tokenizer.apply_chat_template(\n messages,\n tokenize=False,\n add_generation_prompt=True,\n)\n\nprint(text)\n\n# <|im_start|>system\n# ...\n# <|im_start|>user\n# Give me a brief explanation of gravity in simple terms.<|im_end|>\n# <|im_start|>assistant\n\nmodel_inputs = tokenizer([text], add_special_tokens=False, return_tensors=\"pt\")\n```\n\nNotice how the special tokens like `<|im_start|>` and `<|im_end|>` are applied to the prompt before tokenizing. This is useful for the model to learn where a new sequence starts and ends.\n\nThe `transformers` tokenizer adds everything the raw library lacks:\n\n- **Chat template application.** The `apply_chat_template` method formats conversations according to the model's expected format, inserting the correct special tokens and delimiters.\n- **Automatic special token insertion.** Beginning-of-sequence and end-of-sequence tokens are added where the model expects them.\n- **Truncation to context length.** You can specify `truncation=True` and the tokenizer will respect the model's maximum sequence length.\n- **Batch encoding with padding.** Multiple inputs can be padded to the same length with the correct padding token and direction.\n- **Return format options.** You can request PyTorch tensors (`return_tensors=\"pt\"`), NumPy arrays and others.\n\n> `transformers` implements the tokenization API that is most commonly used in the entire ML community (`encode`, `decode`, `convert_tokens_to_ids`, etc.)\n\n## The tokenizer class hierarchy in transformers\n\nThe `transformers` library organizes tokenizers into a class hierarchy. At the top sits a base class that defines the common interface. Below it, backend classes handle the actual tokenization using different engines. At the bottom, model-specific classes configure the backends for particular models.\n\n| [](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/hierarchy.png) |\n| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| The class hierarchy for tokenizers inside transformers |\n\n### `PreTrainedTokenizerBase` defines the common interface for all tokenizers\n\n[`PreTrainedTokenizerBase`](https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_utils_base.py#L964C7) is the abstract base class for all tokenizers in `transformers`. It defines the interface that every tokenizer must implement.\n\nThe base class handles functionality that doesn't depend on the tokenization backend:\n\n- **Special token properties.** Properties like `bos_token`, `eos_token`, `pad_token`, and `unk_token` are defined here. These properties provide access to the special tokens that models use to mark sequence boundaries and handle unknown inputs.\n- **Encoding interface.** The `__call__` method, `encode`, and `encode_plus` methods are defined here. These methods accept text input and return token IDs along with attention masks and other metadata.\n- **Decoding interface.** The `decode` and `batch_decode` methods convert token IDs back to text.\n- **Serialization.** The `save_pretrained` and `from_pretrained` methods handle downloading the correct files, reading information, saving tokenizers to disk etc.\n- **Chat template support.** The `apply_chat_template` method lives here, formatting conversations according to Jinja templates stored in the tokenizer configuration.\n\nEvery tokenizer in `transformers` ultimately inherits from `PreTrainedTokenizerBase`. The base class ensures consistent behavior across all tokenizers, regardless of which backend they use for the actual tokenization.\n\n### `TokenizersBackend` wraps the `tokenizers` library\n\n[`TokenizersBackend`](https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_utils_tokenizers.py#L80C7) is the primary backend class for most modern tokenizers. It inherits from `PreTrainedTokenizerBase` and wraps the Rust-based `tokenizers` library.\n\nThe class stores the Rust tokenizer object internally:\n\n```python\nclass TokenizersBackend(PreTrainedTokenizerBase):\n def __init__(self, tokenizer_object, ...):\n self._tokenizer = tokenizer_object # The Rust tokenizer\n ...\n```\n\nWhen you call encoding methods on a `TokenizersBackend` tokenizer, the class delegates the actual tokenization to the Rust backend:\n\n```python\ndef _batch_encode_plus(self, batch_text_or_text_pairs, ...):\n encodings = self._tokenizer.encode_batch(batch_text_or_text_pairs, ...)\n ...\n```\n\nThe Rust backend performs computationally intensive work, while the Python wrapper adds the model-aware features on top.\n\nMany model-specific tokenizers inherit from `TokenizersBackend`, examples include:\n\n- `LlamaTokenizer`\n- `GemmaTokenizer`\n\nThese model-specific classes configure the backend with the correct vocabulary, merge rules, special tokens, and normalization settings for their respective models.\n\n### `PythonBackend` provides a pure-Python mixin\n\n[`PythonBackend`](https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_python.py#L400) inherits from `PreTrainedTokenizerBase` and implements tokenization in pure Python. The class is aliased as [`PreTrainedTokenizer`](https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_python.py#L1400C1).\n\nThe pure-Python backend exists for several reasons:\n\n- **Custom tokenization logic.** Some models require tokenization behavior that doesn't fit the standard `tokenizers` pipeline.\n- **Legacy compatibility.** Older model implementations may rely on Python-specific behavior.\n\n> The Python backend is slower than the Rust backend. For most use cases, the Rust-backed `TokenizersBackend` is preferred.\n\nModel-specific tokenizers that inherit from `PythonBackend` (or its alias `PreTrainedTokenizer`) include some older or specialized models, like:\n\n- `CTRLTokenizer`\n- `CanineTokenizer`\n\n### `SentencePieceBackend` handles SentencePiece models\n\n[`SentencePieceBackend`](https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/tokenization_utils_sentencepiece.py#L46) inherits from `PythonBackend` and provides integration with Google's [SentencePiece](https://github.com/google/sentencepiece) library. SentencePiece is a standalone tokenization library that many models use, particularly those trained by Google.\n\nThe backend wraps a SentencePiece processor:\n\n```python\nclass SentencePieceBackend(PythonBackend):\n def __init__(self, vocab_file, ...):\n self.sp_model = spm.SentencePieceProcessor()\n self.sp_model.Load(vocab_file)\n ...\n```\n\nModels that use SentencePiece tokenization inherit from this backend. Examples include:\n\n- `SiglipTokenizer`\n- `BartphoTokenizer`\n\nThe SentencePiece backend inherits from `PythonBackend` rather than directly from `PreTrainedTokenizerBase` because it shares much of the same interface and padding/truncation logic.\n\n## AutoTokenizer automatically selects the correct tokenizer class\n\n[`AutoTokenizer`](https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/models/auto/tokenization_auto.py#L531) is the recommended entry point for loading tokenizers. It automatically determines which tokenizer class to use for a given model and returns an instance of that class.\n\n```python\nfrom transformers import AutoTokenizer\n\ntokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n```\n\nBehind the scenes, `AutoTokenizer` performs these steps:\n\n1. **Download the tokenizer configuration.** The `from_pretrained` method fetches `tokenizer_config.json` from the Hub (or from a local directory).\n2. **Identify the model type.** The configuration contains metadata that [identifies the model type](https://huggingface.co/openai-community/gpt2/blob/main/config.json#L12) (e.g., \"gpt2\", \"llama\", \"bert\").\n3. **Look up the tokenizer class.** `AutoTokenizer` maintains a mapping called [`TOKENIZER_MAPPING_NAMES`](https://github.com/huggingface/transformers/blob/7f52a2a4ea8ab49b7f069df7fac58a5b280d4919/src/transformers/models/auto/tokenization_auto.py#L64) that maps model types to tokenizer class names:\n\n```python\nTOKENIZER_MAPPING_NAMES = {\n \"gpt2\": \"GPT2Tokenizer\",\n \"llama\": \"LlamaTokenizer\",\n \"bert\": \"BertTokenizer\",\n ...\n}\n```\n\n4. **Instantiate the correct class.** `AutoTokenizer` imports the appropriate tokenizer class and calls its `from_pretrained` method.\n5. **Return the configured tokenizer.** You receive a fully configured, model-specific tokenizer ready for use.\n\n> The benefit of `AutoTokenizer` is that you don't need to know which tokenizer class a model uses. Whether a model uses `LlamaTokenizer`, `GPT2Tokenizer`, or `BertTokenizer`, the same `AutoTokenizer.from_pretrained(\"model-name\")` call works.\n\nThe tokenizer system in `transformers` forms a layered architecture:\n\n| Layer | Component | Responsibility |\n| -------------- | ------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- |\n| Entry Point | `AutoTokenizer` | Automatically selects and instantiates the correct tokenizer class |\n| Model-Specific | `LlamaTokenizer`, `GPT2Tokenizer`, etc. | Configures the backend with model-specific architecture of normalizer, pre tokenizer, etc, special tokens, and settings |\n| Backend | `TokenizersBackend`, `PythonBackend`, `SentencePieceBackend` | Implements the actual tokenization using a specific engine |\n| Base | `PreTrainedTokenizerBase` | Defines the common interface and shared functionality |\n| Engine | `tokenizers` (Rust), SentencePiece, Pure Python | Performs raw tokenization |\n\n## v5 Separates Tokenizer Architecture from Trained Vocab\n\nThe most significant change in Transformers v5 is a philosophical shift in how tokenizers are defined. **Tokenizers now work like PyTorch's `nn.Module`**: you define the architecture first, then fill it with learned parameters.\n\n### The problem with v4: tokenizers were opaque and tightly coupled\n\nIn v4, tokenizers were black boxes tied to pretrained checkpoint files. If you loaded `LlamaTokenizerFast`, you couldn't easily answer basic questions about it:\n\n- Is it BPE or Unigram?\n- How does it normalize text?\n- What pre-tokenization strategy does it use?\n- What are the special tokens and their positions?\n\nThe `__init__` method gave no clues. You had to dig through serialized files or external documentation to understand what the tokenizer actually did.\n\n| [](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v4-llama.png) |\n| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `LlamaTokenizerFast` as seen in v4 `transformers` |\n\nv4 also maintained two parallel implementations for every model:\n\n1. a \"slow\" Python tokenizer (`LlamaTokenizer` inheriting from `PreTrainedTokenizer`) and\n2. a \"fast\" Rust-backed tokenizer (`LlamaTokenizerFast` inheriting from `PreTrainedTokenizerFast`).\n\nThis meant:\n\n- **Two files per model** (e.g., `tokenization_llama.py` and `tokenization_llama_fast.py`)\n- **Code duplication** across hundreds of models\n- **Behavioral discrepancies** between slow and fast versions, leading to subtle bugs\n- **A growing test suite** dedicated to verifying that slow and fast tokenizers produced identical outputs\n- **User confusion** about which tokenizer to use and when\n\nWorst of all, you couldn't create an empty tokenizer architecture. If you wanted to train a LLaMA-style tokenizer on your own data, there was no clean way to instantiate a \"blank\" LLaMA tokenizer and fill it with your vocabulary and merges. Tokenizers existed only as loaded checkpoints, not as configurable templates.\n\n### The v5 solution: architecture and parameters are now separate\n\nv5 treats tokenizer architecture (normalizer, pre-tokenizer, model type, post-processor, decoder) as distinct from trained parameters (vocabulary, merges). This mirrors how PyTorch separates model architecture from learned weights.\n\n**With `nn.Module`, you define layers first:**\n\n```python\nfrom torch import nn\n\nmodel = nn.Sequential(\n nn.Embedding(vocab_size, embed_dim),\n nn.Linear(embed_dim, hidden_dim),\n)\n# Architecture defined; weights initialized randomly or loaded later\n```\n\n**V5 tokenizers follow the same pattern:**\n\n```python\nfrom transformers import LlamaTokenizer\n\n# Instantiate the architecture\ntokenizer = LlamaTokenizer()\n\n# Train on your own data to fill in vocab and merges\ntokenizer.train(files=[\"my_corpus.txt\"])\n```\n\nThe tokenizer class now explicitly declares its structure. Looking at `LlamaTokenizer` in v5, you can immediately see:\n\n- [It uses **BPE**](https://github.com/huggingface/transformers/blob/0a8465420eecbac1c6d7dd9f45c08dd96b8c5027/src/transformers/models/llama/tokenization_llama.py#L92) as its tokenization model\n- It may add a **prefix space** before text\n- Its special tokens (`unk`, `bos`, `eos`) sit at specific vocabulary positions\n- [It does **not normalize**](https://github.com/huggingface/transformers/blob/0a8465420eecbac1c6d7dd9f45c08dd96b8c5027/src/transformers/models/llama/tokenization_llama.py#L121) input text\n- [Its decoder](https://github.com/huggingface/transformers/blob/0a8465420eecbac1c6d7dd9f45c08dd96b8c5027/src/transformers/models/llama/tokenization_llama.py#L122) replaces the metaspace character `▁` with spaces\n\n| [](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/tokenizers/v5-llama.png) |\n| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `LlamaTokenizer` as seen in v5 `transformers` |\n\nThis transparency was impossible in v4, where the same information was buried in serialized files.\n\n### One file, one backend, one recommended path\n\nv5 consolidates the two-file system *into a single file per model*. `LlamaTokenizer` now inherits from `TokenizersBackend`, which wraps the Rust-based tokenizer that was previously exposed as the “fast” implementation and is now the default.\n\nThe former “slow” Python implementation lives explicitly behind `PythonBackend`, and `SentencePieceBackend` remains for models that require it, but **Rust-backed tokenization is the preferred default**.\n\nThis change eliminates:\n\n- Duplicate code across slow/fast implementations\n- The confusing `Tokenizer` vs `TokenizerFast` naming convention\n- Test suites dedicated to checking slow-fast parity\n\nUsers now have one clear entry point. Advanced users who need to customize can still access lower-level components, but the library no longer forces everyone to navigate two parallel implementations.\n\n### You can now train model specific tokenizers from scratch\n\nSuppose you want a tokenizer that behaves exactly like LLaMA's – same normalization, same pre-tokenization, same BPE model type – but trained on a domain-specific corpus (medical text, legal documents, a new language). In v4, this required manually reconstructing the tokenizer pipeline from low-level `tokenizers` library primitives. In v5, you can instantiate the architecture directly and call `train`:\n\n```python\nfrom transformers import LlamaTokenizer\nfrom datasets import load_dataset\n\n# Initialize blank tokenizer\ntokenizer = LlamaTokenizer()\n\ndataset = load_dataset(\"wikitext\", \"wikitext-2-raw-v1\", split=\"train\")\n\ndef get_training_corpus():\n batch = 1000\n for i in range(0, len(dataset), batch):\n yield dataset[i : i + batch][\"text\"]\n\ntrained_tokenizer = tokenizer.train_new_from_iterator(\n text_iterator=get_training_corpus(),\n vocab_size=32000,\n length=len(dataset),\n show_progress=True,\n)\n\ntrained_tokenizer.push_to_hub(\"my_custom_tokenizer\")\n\ntokenizer = LlamaTokenizer.from_pretrained(\"my_custom_tokenizer\")\n```\n\nThe resulting tokenizer will have your custom vocabulary and merge rules, but will process text identically to how a standard LLaMA tokenizer would with the same whitespace handling, same special token conventions, same decoding behavior.\n\n| Aspect | V4 | V5 |\n| ----------------------- | --------------------------------------------------- | ---------------------------------------------------------------- |\n| Files per model | Two (`tokenization_X.py`, `tokenization_X_fast.py`) | One (`tokenization_X.py`) |\n| Default backend | Split between Python and Rust | Rust (`TokenizersBackend`) preferred |\n| Architecture visibility | Hidden in serialized files | Explicit in class definition |\n| Training from scratch | Required manual pipeline construction | `tokenizer.train(files=[...])` |\n| Component inspection | Difficult, undocumented | Direct properties (`tokenizer.normalizer`, etc.) |\n| Parent classes | `PreTrainedTokenizer`, `PreTrainedTokenizerFast` | `TokenizersBackend` (or `SentencePieceBackend`, `PythonBackend`) |\n\nThe shift from \"tokenizers as loaded checkpoints\" to \"tokenizers as configurable architectures\" makes the library more modular, more transparent, and more aligned with how practitioners think about building ML systems.\n\n## Summary\n\nTransformers v5 brings three improvements to tokenization:\n\n1. **One file per model** instead of separate slow/fast implementations\n2. **Visible architecture** so you can inspect normalizers, pre-tokenizers, and decoders\n3. **Trainable templates** that let you create custom tokenizers matching any model's design\n\nThe wrapper layer between `tokenizers` and Transformers remains essential. It adds model awareness, context lengths, chat templates, special tokens, that raw tokenization doesn't provide. V5 just makes that layer clearer and more customizable.\n\nIf you are looking to learn more about tokenization here are some resources:\n\n- [Let's build the GPT Tokenizer](https://youtu.be/zduSFxRajkE?si=ZAfCjZjpyPHsnyfF)\n- [Gotchas in Tokenizer Behavior Every Developer Should Know](https://huggingface.co/blog/qgallouedec/gotchas-in-tokenizer-behavior)\n- [Chat Templates](https://huggingface.co/blog/chat-templates)\n- [A list of resources we have gathered from the community!](https://x.com/ariG23498/status/1999058214906888237)\n", "dateCreated": "2026-10-04T12:50:30.804479567+00:00", "date_created": "2025-12-18T00:00:00+00:00", "date_modified": "2025-12-18T00:00:00+00:00", "date_published": "2025-12-18T00:00:00+00:00", "description": "Transformers v5 redesigns how tokenizers work. The big tokenizers reformat separates tokenizer design from trained vocabulary (much like how PyTorch separates neural network architecture from learned weights). The result is tokenizers you…", "external_url": "https://huggingface.co/blog/tokenizers", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2025/12/2025-12-18-tokenization-in-transformers-v5-simpler-clearer-and-more.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2025/12/2025-12-18-tokenization-in-transformers-v5-simpler-clearer-and-more.md" }, "id": "urn:aggr:item:afac5f423297ddfd2fa8acdfd6ec4bab1d70bf18", "isBasedOn": { "@id": "https://huggingface.co/blog/tokenizers" }, "mainEntity": { "@id": "https://huggingface.co/blog/tokenizers", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-12-18-tokenization-in-transformers-v5-simpler-clearer-and-more/#webpage" }, "dateModified": "2025-12-18T00:00:00+00:00", "datePublished": "2025-12-18T00:00:00+00:00", "headline": "Tokenization in Transformers v5: Simpler, Clearer, and More Modular", "timeRequired": "PT13M", "url": "https://huggingface.co/blog/tokenizers", "wordCount": 2800 }, "name": "Archived snapshot: Tokenization in Transformers v5: Simpler, Clearer, and More Modular", "preview": null, "reading_minutes": 13, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00", "title": "Tokenization in Transformers v5: Simpler, Clearer, and More Modular", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-12-18-tokenization-in-transformers-v5-simpler-clearer-and-more/", "word_count": 2800 }