{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/philschmid-de/2024-12-25-fine-tune-classifier-with-modernbert-in-2025/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "Large Language Models (LLMs) have become ubiquitous in 2024. However, smaller, specialized models - particularly for classification tasks - remain critical for building efficient and cost-effective AI systems. One key use case is routing user prompts to the most appropriate LLM or selecting optimal few-shot examples, where fast, accurate classification is essential.\n\nThis blog post demonstrates how to fine-tune ModernBERT, a new state-of-the-art encoder model, for classifying user prompts to implement an intelligent LLM router. ModernBERT is a refreshed version of BERT models, with 8192 token context length, significantly better downstream performance, and much faster processing speeds.\n\nYou will learn how to:\n\n1. [Setup environment and install libraries](https://www.philschmid.de/fine-tune-modern-bert-in-2025#1-setup-environment-and-install-libraries)\n2. [Load and prepare the classification dataset](https://www.philschmid.de/fine-tune-modern-bert-in-2025#2-load-and-prepare-the-classification-dataset)\n3. [Fine-tune & evaluate ModernBERT with the Hugging Face `Trainer`](https://www.philschmid.de/fine-tune-modern-bert-in-2025#3-fine-tune--evaluate-modernbert-with-the-hugging-face-trainer)\n4. [Run inference & test model](https://www.philschmid.de/fine-tune-modern-bert-in-2025#4-run-inference--test-model)\n\n## Quick intro: ModernBERT\n\nModernBERT is a modernization of BERT maintaining full backward compatibility while delivering dramatic improvements through architectural innovations like rotary positional embeddings (RoPE), alternating attention patterns, and hardware-optimized design. The model comes in two sizes:\n\n- ModernBERT Base (139M parameters)\n- ModernBERT Large (395M parameters)\n\nModernBERT achieves state-of-the-art performance across classification, retrieval and code understanding tasks while being 2-4x faster than previous encoder models. This makes it ideal for high-throughput production applications like LLM routing, where both accuracy and latency are critical.\n\nModernBERT was trained on 2 trillion tokens of diverse data including web documents, code, and scientific articles - making it much more robust than traditional BERT models trained primarily on Wikipedia. This broader knowledge helps it better understand the nuances of user prompts across different domains.\n\nIf you want to learn more about ModernBERT's architecture and training process, check out the official [blog](https://huggingface.co/blog/modernbert).\n\n* * *\n\nNow let's get started building our LLM router with ModernBERT! šŸš€\n\n*Note: This tutorial was created and tested on an NVIDIA L4 GPU with 24GB of VRAM.*\n\n## 1\\. Setup environment and install libraries\n\nOur first step is to install Hugging Face Libraries and Pyroch, including transformers and datasets.\n\nPython\n\n```python\n# Install Pytorch & other libraries\n%pip install \"torch==2.4.1\" tensorboard \n%pip install flash-attn \"setuptools<71.0.0\" scikit-learn \n \n# Install Hugging Face libraries\n%pip install --upgrade \\\n \"datasets==3.1.0\" \\\n \"accelerate==1.2.1\" \\\n \"hf-transfer==0.1.8\"\n #\"transformers==4.47.1\" \\\n \n# ModernBERT is not yet available in an official release, so we need to install it from github\n%pip install \"git+https://github.com/huggingface/transformers.git@6e0515e99c39444caae39472ee1b2fd76ece32f1\" --upgrade\n \n```\n\nWe will use the [Hugging Face Hub](https://huggingface.co/models) as a remote model versioning service. This means we will automatically push our model, logs and information to the Hub during training. You must register on the [Hugging Face](https://huggingface.co/join) for this. After you have an account, we will use the `login` util from the `huggingface_hub` package to log into our account and store our token (access key) on the disk.\n\nPython\n\n```python\nfrom huggingface_hub import login\n \nlogin(token=\"\", add_to_git_credential=True) # ADD YOUR TOKEN HERE\n```\n\n## 2\\. Load and prepare the classification dataset\n\nIn our example we want to fine-tune ModernBERT to act as a router for user prompts. Therefore we need a classification dataset consisting of user prompts and their \"difficulty\" score. We are going to use the `DevQuasar/llm_router_dataset-synth` dataset, which is a synthetic dataset of ~15,000 user prompts with a difficulty score of \"large\\_llm\" (`1`) or \"small\\_llm\" (`0`).\n\nWe will use the `load_dataset()` method from the [šŸ¤— Datasets](https://huggingface.co/docs/datasets/index) library to load the `DevQuasar/llm_router_dataset-synth` dataset.\n\nPython\n\n```python\nfrom datasets import load_dataset\n \n# Dataset id from huggingface.co/dataset\n# dataset_id = \"DevQuasar/llm_router_dataset-synth\"\ndataset_id = \"legacy-datasets/banking77\"\n \n# Load raw dataset\nraw_dataset = load_dataset(dataset_id)\n \nprint(f\"Train dataset size: {len(raw_dataset['train'])}\")\nprint(f\"Test dataset size: {len(raw_dataset['test'])}\")\n```\n\nTrain dataset size: 10003 Test dataset size: 3080\n\nLetā€™s check out an example of the dataset.\n\nPython\n\n```python\nfrom random import randrange\n \nrandom_id = randrange(len(raw_dataset['train']))\nraw_dataset['train'][random_id]\n# {'id': '6225a9cd-5cba-4840-8e21-1f9cf2ded7e6',\n# 'prompt': 'How many legs does a spider have?',\n# 'label': 0}\n```\n\nTo train our model, we need to convert our text prompts to token IDs. This is done by a Tokenizer, which tokenizes the inputs (including converting the tokens to their corresponding IDs in the pre-trained vocabulary) if you want to learn more about this, outĀ **[chapter 6](https://huggingface.co/course/chapter6/1?fw=pt)**Ā of the [Hugging Face Course](https://huggingface.co/course/chapter1/1).\n\nPython\n\n```python\nfrom transformers import AutoTokenizer\n \n# Model id to load the tokenizer\n# model_id = \"answerdotai/ModernBERT-base\"\nmodel_id = \"google-bert/bert-base-uncased\"\n# Load Tokenizer\ntokenizer = AutoTokenizer.from_pretrained(model_id)\ntokenizer.model_max_length = 512 # set model_max_length to 512 as prompts are not longer than 1024 tokens\n \n# Tokenize helper function\ndef tokenize(batch):\n return tokenizer(batch['text'], padding='max_length', truncation=True, return_tensors=\"pt\")\n \n# Tokenize dataset\nraw_dataset = raw_dataset.rename_column(\"label\", \"labels\") # to match Trainer\ntokenized_dataset = raw_dataset.map(tokenize, batched=True,remove_columns=[\"text\"])\n \nprint(tokenized_dataset[\"train\"].features.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask','lable'])\n```\n\n## 3\\. Fine-tune & evaluate ModernBERT with the Hugging Face Trainer\n\nAfter we have processed our dataset, we can start training our model. We will use the [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) model. The first step is to load our model with `AutoModelForSequenceClassification` class from the [Hugging Face Hub](https://huggingface.co/answerdotai/ModernBERT-base). This will initialize the pre-trained ModernBERT weights with a classification head on top. Here we pass the number of classes (2) from our dataset and the label names to have readable outputs for inference.\n\nPython\n\n```python\nfrom transformers import AutoModelForSequenceClassification\n \n# Model id to load the tokenizer\n# model_id = \"answerdotai/ModernBERT-base\"\nmodel_id = \"google-bert/bert-base-uncased\"\n \n# Prepare model labels - useful for inference\nlabels = tokenized_dataset[\"train\"].features[\"labels\"].names\nnum_labels = len(labels)\nlabel2id, id2label = dict(), dict()\nfor i, label in enumerate(labels):\n label2id[label] = str(i)\n id2label[str(i)] = label\n \n# Download the model from huggingface.co/models\nmodel = AutoModelForSequenceClassification.from_pretrained(\n model_id, num_labels=num_labels, label2id=label2id, id2label=id2label,\n)\n```\n\nWe evaluate our model during training. TheĀ `Trainer`Ā supports evaluation during training by providing aĀ `compute_metrics` method. We use the `evaluate` library to calculate the [f1 metric](https://huggingface.co/spaces/evaluate-metric/f1) during training on our test split.\n\nPython\n\n```python\nimport numpy as np\nfrom sklearn.metrics import f1_score\n \n# Metric helper method\ndef compute_metrics(eval_pred):\n predictions, labels = eval_pred\n predictions = np.argmax(predictions, axis=1)\n score = f1_score(\n labels, predictions, labels=labels, pos_label=1, average=\"weighted\"\n )\n return {\"f1\": float(score) if score == 1 else score}\n```\n\nThe last step is to define the hyperparameters (`TrainingArguments`) we use for our training. Here we are adding optimizations introduced features for fast training times using `torch_compile` option in the `TrainingArguments`.\n\nWe also leverage theĀ [Hugging Face Hub](https://huggingface.co/models)Ā integration of theĀ `Trainer`Ā to push our checkpoints, logs, and metrics during training into a repository.\n\nPython\n\n```python\nfrom huggingface_hub import HfFolder\nfrom transformers import Trainer, TrainingArguments\n \n# Define training args\ntraining_args = TrainingArguments(\n output_dir= \"modernbert-llm-router\",\n per_device_train_batch_size=32,\n per_device_eval_batch_size=16,\n learning_rate=5e-5,\n\t\tnum_train_epochs=5,\n bf16=True, # bfloat16 training \n optim=\"adamw_torch_fused\", # improved optimizer \n # logging & evaluation strategies\n logging_strategy=\"steps\",\n logging_steps=100,\n eval_strategy=\"epoch\",\n save_strategy=\"epoch\",\n save_total_limit=2,\n load_best_model_at_end=True,\n metric_for_best_model=\"f1\",\n # push to hub parameters\n report_to=\"tensorboard\",\n push_to_hub=True,\n hub_strategy=\"every_save\",\n hub_token=HfFolder.get_token(),\n \n)\n \n# Create a Trainer instance\ntrainer = Trainer(\n model=model,\n args=training_args,\n train_dataset=tokenized_dataset[\"train\"],\n eval_dataset=tokenized_dataset[\"test\"],\n compute_metrics=compute_metrics,\n)\n```\n\nWe can start our training by using theĀ **`train`**Ā method of the `Trainer`.\n\nPython\n\n```python\n# Start training\ntrainer.train()\n```\n\nFine-tuning `answerdotai/ModernBERT-base` on ~15,000 synthetic prompts for 5 epochs took `321` seconds and our best model achieved a `f1` score of `0.993`. šŸš€ I also ran the training with `bert-base-uncased` to compare the training time and performance. The original BERT achieved a `f1` score of `0.99` and took `1048` seconds to train.\n\n*Note: ModernBERT and BERT both almost achieve the same performance. This indicates that the dataset is not challenging and probably could be solved using a logistic regression classifier. I ran the same code on the [banking77](https://huggingface.co/datasets/legacy-datasets/banking77) dataset. A dataset of ~13,000 customer service queries with 77 classes. There the ModernBERT outperformed the original BERT by 3% (f1 score of 0.93 vs 0.90)*\n\nLets save our final best model and tokenizer to the Hugging Face Hub and create a model card.\n\nPython\n\n```python\n# Save processor and create model card\ntokenizer.save_pretrained(\"modernbert-llm-router\")\ntrainer.create_model_card()\ntrainer.push_to_hub()\n```\n\n## 4\\. Run Inference & test model\n\nTo wrap up this tutorial, we will run inference on a few examples and test our model. We will use the `pipeline` method from the `transformers` library to run inference on our model.\n\nPython\n\n```python\nfrom transformers import pipeline\n \n# load model from huggingface.co/models using our repository id\nclassifier = pipeline(\"sentiment-analysis\", model=\"modernbert-llm-router\", device=0)\n \nsample = \"How does the structure and function of plasmodesmata affect cell-to-cell communication and signaling in plant tissues, particularly in response to environmental stresses?\"\n \n \npred = classifier(sample)\nprint(pred)\n# [{'label': 'large_llm', 'score': 1.0}]\n```\n\n## Conclusion\n\nIn this tutorial, we learned how to fine-tune ModernBERT for an LLM routing classification task. We demonstrated how to leverage the Hugging Face ecosystem to efficiently train and deploy a specialized classifier that can intelligently route user prompts to the most appropriate LLM model.\n\nUsing modern training optimizations like flash attention, fused optimizers and mixed precision, we were able to train our model efficiently. Comparing ModernBERT with the original BERT we reduced training time by approximately 3x (1048s vs 321s) on our dataset and outperformed the original BERT by 3% on a more challenging dataset. But more importantly, ModernBERT was trained on 2 trillion tokens, which are more diverse and up to date than the Wikipedia-based training data of the original BERT.\n\nThis example showcases how smaller, specialized models remain valuable in the age of large language models - particularly for high-throughput, latency-sensitive tasks like LLM routing. By using ModernBERT's improved architecture and broader training data, we can build more robust and efficient classification systems.\n", "authors": [], "captured_at": "2026-09-27T19:29:15.293681927+00:00", "category": "programming", "content_html": "

Large Language Models (LLMs) have become ubiquitous in 2024. However, smaller, specialized models - particularly for classification tasks - remain critical for building efficient and cost-effective AI systems. One key use case is routing user prompts to the most appropriate LLM or selecting optimal few-shot examples, where fast, accurate classification is essential.

\n

This blog post demonstrates how to fine-tune ModernBERT, a new state-of-the-art encoder model, for classifying user prompts to implement an intelligent LLM router. ModernBERT is a refreshed version of BERT models, with 8192 token context length, significantly better downstream performance, and much faster processing speeds.

\n

You will learn how to:

\n
    \n
  1. Setup environment and install libraries
    2. \n
  2. Load and prepare the classification dataset
    3. \n
  3. Fine-tune & evaluate ModernBERT with the Hugging Face Trainer
    4. \n
  4. Run inference & test model
    5. \n
\n

Quick intro: ModernBERT

\n

ModernBERT is a modernization of BERT maintaining full backward compatibility while delivering dramatic improvements through architectural innovations like rotary positional embeddings (RoPE), alternating attention patterns, and hardware-optimized design. The model comes in two sizes:

\n\n

ModernBERT achieves state-of-the-art performance across classification, retrieval and code understanding tasks while being 2-4x faster than previous encoder models. This makes it ideal for high-throughput production applications like LLM routing, where both accuracy and latency are critical.

\n

ModernBERT was trained on 2 trillion tokens of diverse data including web documents, code, and scientific articles - making it much more robust than traditional BERT models trained primarily on Wikipedia. This broader knowledge helps it better understand the nuances of user prompts across different domains.

\n

If you want to learn more about ModernBERT's architecture and training process, check out the official blog.

\n
\n

Now let's get started building our LLM router with ModernBERT! šŸš€

\n

Note: This tutorial was created and tested on an NVIDIA L4 GPU with 24GB of VRAM.

\n

1. Setup environment and install libraries

\n

Our first step is to install Hugging Face Libraries and Pyroch, including transformers and datasets.

\n

Python

\n
# Install Pytorch & other libraries\n%pip install "torch==2.4.1" tensorboard \n%pip install flash-attn "setuptools<71.0.0" scikit-learn \n \n# Install Hugging Face libraries\n%pip install  --upgrade \\\n  "datasets==3.1.0" \\\n  "accelerate==1.2.1" \\\n  "hf-transfer==0.1.8"\n  #"transformers==4.47.1" \\\n \n# ModernBERT is not yet available in an official release, so we need to install it from github\n%pip install "git+https://github.com/huggingface/transformers.git@6e0515e99c39444caae39472ee1b2fd76ece32f1" --upgrade\n \n
\n

We will use the Hugging Face Hub as a remote model versioning service. This means we will automatically push our model, logs and information to the Hub during training. You must register on the Hugging Face for this. After you have an account, we will use the login util from the huggingface_hub package to log into our account and store our token (access key) on the disk.

\n

Python

\n
from huggingface_hub import login\n \nlogin(token="", add_to_git_credential=True) # ADD YOUR TOKEN HERE\n
\n

2. Load and prepare the classification dataset

\n

In our example we want to fine-tune ModernBERT to act as a router for user prompts. Therefore we need a classification dataset consisting of user prompts and their "difficulty" score. We are going to use the DevQuasar/llm_router_dataset-synth dataset, which is a synthetic dataset of ~15,000 user prompts with a difficulty score of "large_llm" (1) or "small_llm" (0).

\n

We will use the load_dataset() method from the šŸ¤— Datasets library to load the DevQuasar/llm_router_dataset-synth dataset.

\n

Python

\n
from datasets import load_dataset\n \n# Dataset id from huggingface.co/dataset\n# dataset_id = "DevQuasar/llm_router_dataset-synth"\ndataset_id = "legacy-datasets/banking77"\n \n# Load raw dataset\nraw_dataset = load_dataset(dataset_id)\n \nprint(f"Train dataset size: {len(raw_dataset['train'])}")\nprint(f"Test dataset size: {len(raw_dataset['test'])}")\n
\n

Train dataset size: 10003 Test dataset size: 3080

\n

Letā€™s check out an example of the dataset.

\n

Python

\n
from random import randrange\n \nrandom_id = randrange(len(raw_dataset['train']))\nraw_dataset['train'][random_id]\n# {'id': '6225a9cd-5cba-4840-8e21-1f9cf2ded7e6',\n# 'prompt': 'How many legs does a spider have?',\n# 'label': 0}\n
\n

To train our model, we need to convert our text prompts to token IDs. This is done by a Tokenizer, which tokenizes the inputs (including converting the tokens to their corresponding IDs in the pre-trained vocabulary) if you want to learn more about this, outĀ chapter 6Ā of the Hugging Face Course.

\n

Python

\n
from transformers import AutoTokenizer\n \n# Model id to load the tokenizer\n# model_id = "answerdotai/ModernBERT-base"\nmodel_id = "google-bert/bert-base-uncased"\n# Load Tokenizer\ntokenizer = AutoTokenizer.from_pretrained(model_id)\ntokenizer.model_max_length = 512 # set model_max_length to 512 as prompts are not longer than 1024 tokens\n \n# Tokenize helper function\ndef tokenize(batch):\n    return tokenizer(batch['text'], padding='max_length', truncation=True, return_tensors="pt")\n \n# Tokenize dataset\nraw_dataset =  raw_dataset.rename_column("label", "labels") # to match Trainer\ntokenized_dataset = raw_dataset.map(tokenize, batched=True,remove_columns=["text"])\n \nprint(tokenized_dataset["train"].features.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask','lable'])\n
\n

3. Fine-tune & evaluate ModernBERT with the Hugging Face Trainer

\n

After we have processed our dataset, we can start training our model. We will use the answerdotai/ModernBERT-base model. The first step is to load our model with AutoModelForSequenceClassification class from the Hugging Face Hub. This will initialize the pre-trained ModernBERT weights with a classification head on top. Here we pass the number of classes (2) from our dataset and the label names to have readable outputs for inference.

\n

Python

\n
from transformers import AutoModelForSequenceClassification\n \n# Model id to load the tokenizer\n# model_id = "answerdotai/ModernBERT-base"\nmodel_id = "google-bert/bert-base-uncased"\n \n# Prepare model labels - useful for inference\nlabels = tokenized_dataset["train"].features["labels"].names\nnum_labels = len(labels)\nlabel2id, id2label = dict(), dict()\nfor i, label in enumerate(labels):\n    label2id[label] = str(i)\n    id2label[str(i)] = label\n \n# Download the model from huggingface.co/models\nmodel = AutoModelForSequenceClassification.from_pretrained(\n    model_id, num_labels=num_labels, label2id=label2id, id2label=id2label,\n)\n
\n

We evaluate our model during training. TheĀ TrainerĀ supports evaluation during training by providing aĀ compute_metrics method. We use the evaluate library to calculate the f1 metric during training on our test split.

\n

Python

\n
import numpy as np\nfrom sklearn.metrics import f1_score\n \n# Metric helper method\ndef compute_metrics(eval_pred):\n    predictions, labels = eval_pred\n    predictions = np.argmax(predictions, axis=1)\n    score = f1_score(\n            labels, predictions, labels=labels, pos_label=1, average="weighted"\n        )\n    return {"f1": float(score) if score == 1 else score}\n
\n

The last step is to define the hyperparameters (TrainingArguments) we use for our training. Here we are adding optimizations introduced features for fast training times using torch_compile option in the TrainingArguments.

\n

We also leverage theĀ Hugging Face HubĀ integration of theĀ TrainerĀ to push our checkpoints, logs, and metrics during training into a repository.

\n

Python

\n
from huggingface_hub import HfFolder\nfrom transformers import Trainer, TrainingArguments\n \n# Define training args\ntraining_args = TrainingArguments(\n    output_dir= "modernbert-llm-router",\n    per_device_train_batch_size=32,\n    per_device_eval_batch_size=16,\n    learning_rate=5e-5,\n\t\tnum_train_epochs=5,\n    bf16=True, # bfloat16 training \n    optim="adamw_torch_fused", # improved optimizer \n    # logging & evaluation strategies\n    logging_strategy="steps",\n    logging_steps=100,\n    eval_strategy="epoch",\n    save_strategy="epoch",\n    save_total_limit=2,\n    load_best_model_at_end=True,\n    metric_for_best_model="f1",\n    # push to hub parameters\n    report_to="tensorboard",\n    push_to_hub=True,\n    hub_strategy="every_save",\n    hub_token=HfFolder.get_token(),\n \n)\n \n# Create a Trainer instance\ntrainer = Trainer(\n    model=model,\n    args=training_args,\n    train_dataset=tokenized_dataset["train"],\n    eval_dataset=tokenized_dataset["test"],\n    compute_metrics=compute_metrics,\n)\n
\n

We can start our training by using theĀ trainĀ method of the Trainer.

\n

Python

\n
# Start training\ntrainer.train()\n
\n

Fine-tuning answerdotai/ModernBERT-base on ~15,000 synthetic prompts for 5 epochs took 321 seconds and our best model achieved a f1 score of 0.993. šŸš€ I also ran the training with bert-base-uncased to compare the training time and performance. The original BERT achieved a f1 score of 0.99 and took 1048 seconds to train.

\n

Note: ModernBERT and BERT both almost achieve the same performance. This indicates that the dataset is not challenging and probably could be solved using a logistic regression classifier. I ran the same code on the banking77 dataset. A dataset of ~13,000 customer service queries with 77 classes. There the ModernBERT outperformed the original BERT by 3% (f1 score of 0.93 vs 0.90)

\n

Lets save our final best model and tokenizer to the Hugging Face Hub and create a model card.

\n

Python

\n
# Save processor and create model card\ntokenizer.save_pretrained("modernbert-llm-router")\ntrainer.create_model_card()\ntrainer.push_to_hub()\n
\n

4. Run Inference & test model

\n

To wrap up this tutorial, we will run inference on a few examples and test our model. We will use the pipeline method from the transformers library to run inference on our model.

\n

Python

\n
from transformers import pipeline\n \n# load model from huggingface.co/models using our repository id\nclassifier = pipeline("sentiment-analysis", model="modernbert-llm-router", device=0)\n \nsample = "How does the structure and function of plasmodesmata affect cell-to-cell communication and signaling in plant tissues, particularly in response to environmental stresses?"\n \n \npred = classifier(sample)\nprint(pred)\n# [{'label': 'large_llm', 'score': 1.0}]\n
\n

Conclusion

\n

In this tutorial, we learned how to fine-tune ModernBERT for an LLM routing classification task. We demonstrated how to leverage the Hugging Face ecosystem to efficiently train and deploy a specialized classifier that can intelligently route user prompts to the most appropriate LLM model.

\n

Using modern training optimizations like flash attention, fused optimizers and mixed precision, we were able to train our model efficiently. Comparing ModernBERT with the original BERT we reduced training time by approximately 3x (1048s vs 321s) on our dataset and outperformed the original BERT by 3% on a more challenging dataset. But more importantly, ModernBERT was trained on 2 trillion tokens, which are more diverse and up to date than the Wikipedia-based training data of the original BERT.

\n

This example showcases how smaller, specialized models remain valuable in the age of large language models - particularly for high-throughput, latency-sensitive tasks like LLM routing. By using ModernBERT's improved architecture and broader training data, we can build more robust and efficient classification systems.

\n", "content_markdown": "Large Language Models (LLMs) have become ubiquitous in 2024. However, smaller, specialized models - particularly for classification tasks - remain critical for building efficient and cost-effective AI systems. One key use case is routing user prompts to the most appropriate LLM or selecting optimal few-shot examples, where fast, accurate classification is essential.\n\nThis blog post demonstrates how to fine-tune ModernBERT, a new state-of-the-art encoder model, for classifying user prompts to implement an intelligent LLM router. ModernBERT is a refreshed version of BERT models, with 8192 token context length, significantly better downstream performance, and much faster processing speeds.\n\nYou will learn how to:\n\n1. [Setup environment and install libraries](https://www.philschmid.de/fine-tune-modern-bert-in-2025#1-setup-environment-and-install-libraries)\n2. [Load and prepare the classification dataset](https://www.philschmid.de/fine-tune-modern-bert-in-2025#2-load-and-prepare-the-classification-dataset)\n3. [Fine-tune & evaluate ModernBERT with the Hugging Face `Trainer`](https://www.philschmid.de/fine-tune-modern-bert-in-2025#3-fine-tune--evaluate-modernbert-with-the-hugging-face-trainer)\n4. [Run inference & test model](https://www.philschmid.de/fine-tune-modern-bert-in-2025#4-run-inference--test-model)\n\n## Quick intro: ModernBERT\n\nModernBERT is a modernization of BERT maintaining full backward compatibility while delivering dramatic improvements through architectural innovations like rotary positional embeddings (RoPE), alternating attention patterns, and hardware-optimized design. The model comes in two sizes:\n\n- ModernBERT Base (139M parameters)\n- ModernBERT Large (395M parameters)\n\nModernBERT achieves state-of-the-art performance across classification, retrieval and code understanding tasks while being 2-4x faster than previous encoder models. This makes it ideal for high-throughput production applications like LLM routing, where both accuracy and latency are critical.\n\nModernBERT was trained on 2 trillion tokens of diverse data including web documents, code, and scientific articles - making it much more robust than traditional BERT models trained primarily on Wikipedia. This broader knowledge helps it better understand the nuances of user prompts across different domains.\n\nIf you want to learn more about ModernBERT's architecture and training process, check out the official [blog](https://huggingface.co/blog/modernbert).\n\n* * *\n\nNow let's get started building our LLM router with ModernBERT! šŸš€\n\n*Note: This tutorial was created and tested on an NVIDIA L4 GPU with 24GB of VRAM.*\n\n## 1\\. Setup environment and install libraries\n\nOur first step is to install Hugging Face Libraries and Pyroch, including transformers and datasets.\n\nPython\n\n```python\n# Install Pytorch & other libraries\n%pip install \"torch==2.4.1\" tensorboard \n%pip install flash-attn \"setuptools<71.0.0\" scikit-learn \n \n# Install Hugging Face libraries\n%pip install --upgrade \\\n \"datasets==3.1.0\" \\\n \"accelerate==1.2.1\" \\\n \"hf-transfer==0.1.8\"\n #\"transformers==4.47.1\" \\\n \n# ModernBERT is not yet available in an official release, so we need to install it from github\n%pip install \"git+https://github.com/huggingface/transformers.git@6e0515e99c39444caae39472ee1b2fd76ece32f1\" --upgrade\n \n```\n\nWe will use the [Hugging Face Hub](https://huggingface.co/models) as a remote model versioning service. This means we will automatically push our model, logs and information to the Hub during training. You must register on the [Hugging Face](https://huggingface.co/join) for this. After you have an account, we will use the `login` util from the `huggingface_hub` package to log into our account and store our token (access key) on the disk.\n\nPython\n\n```python\nfrom huggingface_hub import login\n \nlogin(token=\"\", add_to_git_credential=True) # ADD YOUR TOKEN HERE\n```\n\n## 2\\. Load and prepare the classification dataset\n\nIn our example we want to fine-tune ModernBERT to act as a router for user prompts. Therefore we need a classification dataset consisting of user prompts and their \"difficulty\" score. We are going to use the `DevQuasar/llm_router_dataset-synth` dataset, which is a synthetic dataset of ~15,000 user prompts with a difficulty score of \"large\\_llm\" (`1`) or \"small\\_llm\" (`0`).\n\nWe will use the `load_dataset()` method from the [šŸ¤— Datasets](https://huggingface.co/docs/datasets/index) library to load the `DevQuasar/llm_router_dataset-synth` dataset.\n\nPython\n\n```python\nfrom datasets import load_dataset\n \n# Dataset id from huggingface.co/dataset\n# dataset_id = \"DevQuasar/llm_router_dataset-synth\"\ndataset_id = \"legacy-datasets/banking77\"\n \n# Load raw dataset\nraw_dataset = load_dataset(dataset_id)\n \nprint(f\"Train dataset size: {len(raw_dataset['train'])}\")\nprint(f\"Test dataset size: {len(raw_dataset['test'])}\")\n```\n\nTrain dataset size: 10003 Test dataset size: 3080\n\nLetā€™s check out an example of the dataset.\n\nPython\n\n```python\nfrom random import randrange\n \nrandom_id = randrange(len(raw_dataset['train']))\nraw_dataset['train'][random_id]\n# {'id': '6225a9cd-5cba-4840-8e21-1f9cf2ded7e6',\n# 'prompt': 'How many legs does a spider have?',\n# 'label': 0}\n```\n\nTo train our model, we need to convert our text prompts to token IDs. This is done by a Tokenizer, which tokenizes the inputs (including converting the tokens to their corresponding IDs in the pre-trained vocabulary) if you want to learn more about this, outĀ **[chapter 6](https://huggingface.co/course/chapter6/1?fw=pt)**Ā of the [Hugging Face Course](https://huggingface.co/course/chapter1/1).\n\nPython\n\n```python\nfrom transformers import AutoTokenizer\n \n# Model id to load the tokenizer\n# model_id = \"answerdotai/ModernBERT-base\"\nmodel_id = \"google-bert/bert-base-uncased\"\n# Load Tokenizer\ntokenizer = AutoTokenizer.from_pretrained(model_id)\ntokenizer.model_max_length = 512 # set model_max_length to 512 as prompts are not longer than 1024 tokens\n \n# Tokenize helper function\ndef tokenize(batch):\n return tokenizer(batch['text'], padding='max_length', truncation=True, return_tensors=\"pt\")\n \n# Tokenize dataset\nraw_dataset = raw_dataset.rename_column(\"label\", \"labels\") # to match Trainer\ntokenized_dataset = raw_dataset.map(tokenize, batched=True,remove_columns=[\"text\"])\n \nprint(tokenized_dataset[\"train\"].features.keys())\n# dict_keys(['input_ids', 'token_type_ids', 'attention_mask','lable'])\n```\n\n## 3\\. Fine-tune & evaluate ModernBERT with the Hugging Face Trainer\n\nAfter we have processed our dataset, we can start training our model. We will use the [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) model. The first step is to load our model with `AutoModelForSequenceClassification` class from the [Hugging Face Hub](https://huggingface.co/answerdotai/ModernBERT-base). This will initialize the pre-trained ModernBERT weights with a classification head on top. Here we pass the number of classes (2) from our dataset and the label names to have readable outputs for inference.\n\nPython\n\n```python\nfrom transformers import AutoModelForSequenceClassification\n \n# Model id to load the tokenizer\n# model_id = \"answerdotai/ModernBERT-base\"\nmodel_id = \"google-bert/bert-base-uncased\"\n \n# Prepare model labels - useful for inference\nlabels = tokenized_dataset[\"train\"].features[\"labels\"].names\nnum_labels = len(labels)\nlabel2id, id2label = dict(), dict()\nfor i, label in enumerate(labels):\n label2id[label] = str(i)\n id2label[str(i)] = label\n \n# Download the model from huggingface.co/models\nmodel = AutoModelForSequenceClassification.from_pretrained(\n model_id, num_labels=num_labels, label2id=label2id, id2label=id2label,\n)\n```\n\nWe evaluate our model during training. TheĀ `Trainer`Ā supports evaluation during training by providing aĀ `compute_metrics` method. We use the `evaluate` library to calculate the [f1 metric](https://huggingface.co/spaces/evaluate-metric/f1) during training on our test split.\n\nPython\n\n```python\nimport numpy as np\nfrom sklearn.metrics import f1_score\n \n# Metric helper method\ndef compute_metrics(eval_pred):\n predictions, labels = eval_pred\n predictions = np.argmax(predictions, axis=1)\n score = f1_score(\n labels, predictions, labels=labels, pos_label=1, average=\"weighted\"\n )\n return {\"f1\": float(score) if score == 1 else score}\n```\n\nThe last step is to define the hyperparameters (`TrainingArguments`) we use for our training. Here we are adding optimizations introduced features for fast training times using `torch_compile` option in the `TrainingArguments`.\n\nWe also leverage theĀ [Hugging Face Hub](https://huggingface.co/models)Ā integration of theĀ `Trainer`Ā to push our checkpoints, logs, and metrics during training into a repository.\n\nPython\n\n```python\nfrom huggingface_hub import HfFolder\nfrom transformers import Trainer, TrainingArguments\n \n# Define training args\ntraining_args = TrainingArguments(\n output_dir= \"modernbert-llm-router\",\n per_device_train_batch_size=32,\n per_device_eval_batch_size=16,\n learning_rate=5e-5,\n\t\tnum_train_epochs=5,\n bf16=True, # bfloat16 training \n optim=\"adamw_torch_fused\", # improved optimizer \n # logging & evaluation strategies\n logging_strategy=\"steps\",\n logging_steps=100,\n eval_strategy=\"epoch\",\n save_strategy=\"epoch\",\n save_total_limit=2,\n load_best_model_at_end=True,\n metric_for_best_model=\"f1\",\n # push to hub parameters\n report_to=\"tensorboard\",\n push_to_hub=True,\n hub_strategy=\"every_save\",\n hub_token=HfFolder.get_token(),\n \n)\n \n# Create a Trainer instance\ntrainer = Trainer(\n model=model,\n args=training_args,\n train_dataset=tokenized_dataset[\"train\"],\n eval_dataset=tokenized_dataset[\"test\"],\n compute_metrics=compute_metrics,\n)\n```\n\nWe can start our training by using theĀ **`train`**Ā method of the `Trainer`.\n\nPython\n\n```python\n# Start training\ntrainer.train()\n```\n\nFine-tuning `answerdotai/ModernBERT-base` on ~15,000 synthetic prompts for 5 epochs took `321` seconds and our best model achieved a `f1` score of `0.993`. šŸš€ I also ran the training with `bert-base-uncased` to compare the training time and performance. The original BERT achieved a `f1` score of `0.99` and took `1048` seconds to train.\n\n*Note: ModernBERT and BERT both almost achieve the same performance. This indicates that the dataset is not challenging and probably could be solved using a logistic regression classifier. I ran the same code on the [banking77](https://huggingface.co/datasets/legacy-datasets/banking77) dataset. A dataset of ~13,000 customer service queries with 77 classes. There the ModernBERT outperformed the original BERT by 3% (f1 score of 0.93 vs 0.90)*\n\nLets save our final best model and tokenizer to the Hugging Face Hub and create a model card.\n\nPython\n\n```python\n# Save processor and create model card\ntokenizer.save_pretrained(\"modernbert-llm-router\")\ntrainer.create_model_card()\ntrainer.push_to_hub()\n```\n\n## 4\\. Run Inference & test model\n\nTo wrap up this tutorial, we will run inference on a few examples and test our model. We will use the `pipeline` method from the `transformers` library to run inference on our model.\n\nPython\n\n```python\nfrom transformers import pipeline\n \n# load model from huggingface.co/models using our repository id\nclassifier = pipeline(\"sentiment-analysis\", model=\"modernbert-llm-router\", device=0)\n \nsample = \"How does the structure and function of plasmodesmata affect cell-to-cell communication and signaling in plant tissues, particularly in response to environmental stresses?\"\n \n \npred = classifier(sample)\nprint(pred)\n# [{'label': 'large_llm', 'score': 1.0}]\n```\n\n## Conclusion\n\nIn this tutorial, we learned how to fine-tune ModernBERT for an LLM routing classification task. We demonstrated how to leverage the Hugging Face ecosystem to efficiently train and deploy a specialized classifier that can intelligently route user prompts to the most appropriate LLM model.\n\nUsing modern training optimizations like flash attention, fused optimizers and mixed precision, we were able to train our model efficiently. Comparing ModernBERT with the original BERT we reduced training time by approximately 3x (1048s vs 321s) on our dataset and outperformed the original BERT by 3% on a more challenging dataset. But more importantly, ModernBERT was trained on 2 trillion tokens, which are more diverse and up to date than the Wikipedia-based training data of the original BERT.\n\nThis example showcases how smaller, specialized models remain valuable in the age of large language models - particularly for high-throughput, latency-sensitive tasks like LLM routing. By using ModernBERT's improved architecture and broader training data, we can build more robust and efficient classification systems.\n", "dateCreated": "2026-09-27T19:29:15.293681927+00:00", "date_created": "2024-12-25T00:00:00+00:00", "date_modified": "2024-12-25T00:00:00+00:00", "date_published": "2024-12-25T00:00:00+00:00", "description": "Modern updated guide on how to fine-tune BERT models for classification tasks in 2025.", "external_url": "https://www.philschmid.de/fine-tune-modern-bert-in-2025", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/philschmid-de/2024/12/2024-12-25-fine-tune-classifier-with-modernbert-in-2025.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/philschmid-de/2024/12/2024-12-25-fine-tune-classifier-with-modernbert-in-2025.md" }, "id": "urn:aggr:item:f63cd1c50500749eb84b5259c159fa0681502060", "isBasedOn": { "@id": "https://www.philschmid.de/fine-tune-modern-bert-in-2025" }, "mainEntity": { "@id": "https://www.philschmid.de/fine-tune-modern-bert-in-2025", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/philschmid-de/2024-12-25-fine-tune-classifier-with-modernbert-in-2025/#webpage" }, "dateModified": "2024-12-25T00:00:00+00:00", "datePublished": "2024-12-25T00:00:00+00:00", "headline": "Fine-tune classifier with ModernBERT in 2025", "timeRequired": "PT7M", "url": "https://www.philschmid.de/fine-tune-modern-bert-in-2025", "wordCount": 1563 }, "name": "Archived snapshot: Fine-tune classifier with ModernBERT in 2025", "preview": null, "reading_minutes": 7, "replicated_at": null, "source": "philschmid-de", "source_name": "philschmid.de - RSS feed", "summary": "Modern updated guide on how to fine-tune BERT models for classification tasks in 2025.", "tags": [], "temporalCoverage": "2026-09-27T19:29:15.293681927+00:00", "title": "Fine-tune classifier with ModernBERT in 2025", "url": "https://aggr.aymericbeaumet.com/items/philschmid-de/2024-12-25-fine-tune-classifier-with-modernbert-in-2025/", "word_count": 1563 }