{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-09-09-mmbert-modernbert-goes-multilingual/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "## TL;DR\n\nThis blog post introduces [mmBERT](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4), a state-of-the-art massively multilingual encoder model trained on 3T+ tokens of text in over 1800 languages. It shows significant performance and speed improvements over previous multilingual models, being the first to improve upon XLM-R, while also developing new strategies for effectively learning low-resource languages. mmBERT builds upon ModernBERT for a blazingly fast architecture, and adds novel components to enable efficient multilingual learning.\n\nIf you are interested in trying out the models yourself, some example boilerplate is available [at the end of this blogpost!](https://huggingface.co/blog/mmbert#usage-examples)\n\n## Training Data\n\n![Distribution of each phase of pre-training](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/data_dist.jpg?raw=true)\\\n Figure 1: the training data is progressively annealed to include more languages and more uniform sampling throughout training.\n\nmmBERT was trained on a carefully curated multilingual dataset totaling over 3T tokens across three distinct training phases. The foundation of our training data consists of three primary open-source and high-quality web crawls that enable both multilingual coverage and data quality:\n\n**DCLM and Filtered DCLM** provides the highest quality English content available, serving as the backbone for strong English performance (with the filtered data coming from [Dolmino](https://huggingface.co/datasets/allenai/dolmino-mix-1124)). This dataset represents state-of-the-art web filtering techniques and forms a crucial component. Due to the high quality of this data, we use a signficantly higher proportion of English than previous generation multilingual encoder models (up to 18%).\n\n**FineWeb2** delivers broad [multilingual web content](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) covering over 1,800 languages. This dataset enables our extensive multilingual coverage while maintaining reasonable quality standards across diverse language families and scripts.\n\n**FineWeb2-HQ** consists of a [filtered subset of FineWeb2](https://huggingface.co/datasets/epfml/FineWeb2-HQ) focusing on 20 high-resource languages. This filtered version provides higher-quality multilingual content that bridges the gap between English-only filtered data and broad multilingual coverage.\n\nThe training data also incorporates specialized corpora from [Dolma](https://arxiv.org/abs/2402.00159), [MegaWika v2](https://arxiv.org/abs/2508.03828), [ProLong](https://arxiv.org/abs/2410.02660) and more: code repositories (StarCoder, ProLong), academic content (ArXiv, PeS2o), reference materials (Wikipedia, textbooks), and community discussions (StackExchange), along with instruction and mathematical datasets.\n\nThe key innovation in our data approach is the **progressive language inclusion strategy** shown in [Figure 1](https://huggingface.co/blog/mmbert#figure1). At each phase we progressively sample from a *flatter* distribution (i.e. closer to uniform), while also adding new languages. This means that high resource languages like Russian start off with a high percentage of the data (i.e. 9%) and then in the last phase of training end around half of that. We start with 60 high-resource languages during pre-training, expand to 110 languages during mid-training, and finally include all 1,833 languages from FineWeb2 during the decay phase. This allows us to maximize the impact of limited low-resource language data without excess reptitions and while maintaining high overall data quality.\n\n## Training Recipe and Novel Components\n\nmmBERT builds upon the [ModernBERT](https://huggingface.co/blog/modernbert) architecture but introduces several key innovations for multilingual learning:\n\n### Architecture\n\nWe use the same core architecture as ModernBERT-base with 22 layers and 1152 intermediate dimensions, but switch to the Gemma 2 tokenizer to better handle multilingual text. The base model has 110M non-embedding parameters (307M total due to the larger vocabulary), while the small variant has 42M non-embedding parameters (140M total).\n\n### Three-Phase Training Approach\n\nOur training follows a carefully designed three-phase schedule:\n\n1. **Pre-training (2.3T tokens)**: Warmup and stable learning rate phase using 60 languages with 30% mask rate\n2. **Mid-training (600B tokens)**: Context extension to 8192 tokens, higher-quality data, expanded to 110 languages with 15% mask rate\n3. **Decay phase (100B tokens)**: Inverse square root learning rate decay, all 1,833 languages included with 5% mask rate\n\n### Novel Training Techniques\n\n**Inverse Mask Ratio Schedule**: Instead of using a fixed masking rate, we progressively reduce the mask ratio from 30% → 15% → 5% across training phases. This allows the model to learn basic representations with higher masking early on, then focus on more nuanced understanding with lower masking rates.\n\n**Annealed Language Learning**: We dynamically adjust the temperature for multilingual data sampling from τ=0.7 → 0.5 → 0.3. This creates a progression from high-resource language bias toward more uniform sampling, enabling the model to build a strong multilingual foundation before learning low-resource languages.\n\n**Progressive Language Addition**: Rather than training on all languages simultaneously, we strategically add languages at each phase (60 → 110 → 1,833). This maximizes learning efficiency by avoiding excessive epochs on limited low-resource data while still achieving strong performance.\n\n**Model Merging**: We train three different variants during the decay phase (English-focused, 110-language, and all-language) and use TIES merging to combine their strengths into the final model.\n\n## Results\n\n### Natural Language Understanding (NLU)\n\n![GLUE performance](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/glue.jpeg?raw=true)\\\nTable 1: Performance on GLUE (English)\n\n**English Performance**: On the English GLUE benchmark ([Table 1](https://huggingface.co/blog/mmbert#table1)), mmBERT base achieves strong performance, substantially outperforming other multilingual models like XLM-R (multilingual RoBERTa) base and mGTE base, while remaining competitive to English-only models despite less than 25% of the mmBERT training data being English.\n\n![XTREME performance](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/xtreme.jpeg?raw=true)\\\nTable 2: Performance on XTREME (Multilingual)\n\n**Multilingual Performance**: mmBERT shows significant improvements on XTREME benchmark compared to XLM-R as demonstrated in [Table 2](https://huggingface.co/blog/mmbert#table2). Notable gains include strong performance on XNLI classification, substantial improvements in question answering tasks like TyDiQA, and competitive results across PAWS-X and XCOPA for cross-lingual understanding.\n\nThe model performs well across most categories, with the exception of some structured prediction tasks like NER and POS tagging, likely due to tokenizer differences that affect word boundary detection. On these categories, it performs about the same as the previous generation, but can be applied to more languages.\n\n### Retrieval Performance\n\n![MTEB v2 Eng performance](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/mteb-english.jpeg?raw=true)\\\nTable 3: Performance on MTEB v2 English\n\n**English Retrieval**: Even though mmBERT is designed for massively multilingual settings, in the MTEB v2 English benchmarks ([Table 3](https://huggingface.co/blog/mmbert#table3)), mmBERT shows significant gains over previous multilingual models and even ties the capabilities of English-only models like ModernBERT!\n\n![MTEB v2 Multilingual performance](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/mteb-multilingual.jpeg?raw=true)\\\nTable 4: Performance on MTEB v2 Multilingual\n\n**Multilingual Retrieval**: mmBERT shows consistent improvements on MTEB v2 multilingual benchmarks compared to other models ([Table 4](https://huggingface.co/blog/mmbert#table4)).\n\n![CoIR performance](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/coir.jpeg?raw=true)\\\nTable 5: Performance on CoIR code benchmark\n\n**Code Retrieval**: Due to the modern tokenizer (based on Gemma 2) mmBERT also shows strong coding performance ([Table 5](https://huggingface.co/blog/mmbert#table5)), making mmBERT suitable for any type of textual data. The only model that outperforms it is EuroBERT, which was able to use the non-publicly accessible Stack v2 dataset.\n\n## Learning Languages in the Decay Phase\n\nOne of mmBERT's most significant novel features is demonstrating that low-resource languages can be effectively learned during the short decay phase of training. We validated this approach by testing on languages only introduced during the final 100B token decay phase.\n\n![Improvements from adding new languages in the decay phase](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/low_resource_merge.jpg?raw=true)\\\n Figure 2: adding more than 1700 languages in the decay phase allows for rapid learning, which we keep through model merging.\n\n**Dramatic Performance Gains**: Testing on TiQuaD (Tigrinya) and FoQA (Faroese), we observed substantial improvements when these languages were included in the decay phase, as shown in [Figure 2](https://huggingface.co/blog/mmbert#figure2). The results demonstrate the effectiveness of our progressive language learning approach.\n\n**Competitive with Large Models**: Despite only seeing these languages in the final training phase, mmBERT achieves performance levels that exceed much larger models. On Faroese question answering where LLMs have been benchmarked, mmBERT outperforms Google Gemini 2.5 Pro and OpenAI o3.\n\n**Rapid Learning Mechanism**: The success of decay-phase language learning stems from the model's ability to leverage its strong multilingual foundation built during earlier phases. When exposed to new languages, the model can quickly adapt existing cross-lingual representations rather than learning from scratch.\n\n**Model Merging Benefits**: The final mmBERT models successfully retain most of the decay-phase improvements while benefiting from the English-focused and high-resource variants through TIES merging.\n\n## Efficiency Improvements\n\nmmBERT delivers substantial efficiency gains over previous multilingual encoder models through architectural improvements inherited from ModernBERT:\n\n![mmBERT is much more efficient than previous multilingual models](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/inference_times.jpg?raw=true)\\\n Figure 3: mmBERT is significantly more efficient than previous multilingual models, up to 2-4x as much!\n\n**Throughput Performance**: mmBERT processes text significantly faster than existing multilingual models across various sequence lengths, as demonstrated in [Figure 3](https://huggingface.co/blog/mmbert#figure3). Both the small and base models show substantial speed improvements over previous multilingual encoders.\n\n**Modern Architecture Benefits**: The efficiency gains come from two main technical improvements:\n\n- **Flash Attention 2**: Optimized attention computation for better memory usage and speed\n- **Unpadding techniques**: Elimination of unnecessary padding tokens during processing\n\n**Sequence Length Scaling**: Unlike older models limited to 512 tokens, mmBERT handles up to 8,192 tokens efficiently while maintaining high throughput. This makes it suitable for longer document processing tasks that are increasingly common in multilingual applications.\n\n**Energy Efficiency**: The combination of better throughput and modern architecture results in lower computational costs for inference, making mmBERT more practical for production deployments where multilingual support is needed at scale.\n\nThese efficiency improvements make mmBERT not just more accurate than previous multilingual encoders, but also significantly more practical for real usage.\n\n## Usage Examples\n\nYou can use these models with just a few lines of code!\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForMaskedLM\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"jhu-clsp/mmBERT-base\")\nmodel = AutoModelForMaskedLM.from_pretrained(\"jhu-clsp/mmBERT-base\")\n\ndef predict_masked_token(text):\n inputs = tokenizer(text, return_tensors=\"pt\")\n with torch.no_grad():\n outputs = model(**inputs)\n mask_indices = torch.where(inputs[\"input_ids\"] == tokenizer.mask_token_id)\n predictions = outputs.logits[mask_indices]\n top_tokens, top_indices = torch.topk(predictions, 5, dim=-1)\n return [tokenizer.decode(token) for token in top_indices[0]]\n\n# Works across languages\ntexts = [\n \"The capital of France is .\",\n \"La capital de España es .\",\n \"Die Hauptstadt von Deutschland ist .\",\n]\n\nfor text in texts:\n predictions = predict_masked_token(text)\n print(f\"Text: {text}\")\n print(f\"Predictions: {predictions}\\n\")\n```\n\n## Fine-tuning Examples\n\n### Encoders\n\nClick to see how to finetune this into a dense embedding model using Sentence Transformers\n\n```python\nimport argparse\n\nfrom datasets import load_dataset\nfrom sentence_transformers import (\n SentenceTransformer,\n SentenceTransformerTrainer,\n SentenceTransformerTrainingArguments,\n)\nfrom sentence_transformers.evaluation import TripletEvaluator\nfrom sentence_transformers.losses import CachedMultipleNegativesRankingLoss\nfrom sentence_transformers.training_args import BatchSamplers\n\ndef main():\n # parse the lr & model name\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--lr\", type=float, default=8e-5)\n parser.add_argument(\"--model_name\", type=str, default=\"jhu-clsp/mmBERT-small\")\n args = parser.parse_args()\n lr = args.lr\n model_name = args.model_name\n model_shortname = model_name.split(\"/\")[-1]\n\n # 1. Load a model to finetune\n model = SentenceTransformer(model_name)\n\n # 2. Load a dataset to finetune on\n dataset = load_dataset(\n \"sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1\",\n \"triplet-hard\",\n split=\"train\",\n )\n dataset_dict = dataset.train_test_split(test_size=1_000, seed=12)\n train_dataset = dataset_dict[\"train\"].select(range(1_250_000))\n eval_dataset = dataset_dict[\"test\"]\n\n # 3. Define a loss function\n loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=16) # Increase mini_batch_size if you have enough VRAM\n\n run_name = f\"{model_shortname}-DPR-{lr}\"\n # 4. (Optional) Specify training arguments\n args = SentenceTransformerTrainingArguments(\n # Required parameter:\n output_dir=f\"output/{model_shortname}/{run_name}\",\n # Optional training parameters:\n num_train_epochs=1,\n per_device_train_batch_size=512,\n per_device_eval_batch_size=512,\n warmup_ratio=0.05,\n fp16=False, # Set to False if GPU can't handle FP16\n bf16=True, # Set to True if GPU supports BF16\n batch_sampler=BatchSamplers.NO_DUPLICATES, # (Cached)MultipleNegativesRankingLoss benefits from no duplicates\n learning_rate=lr,\n # Optional tracking/debugging parameters:\n save_strategy=\"steps\",\n save_steps=500,\n save_total_limit=2,\n logging_steps=500,\n run_name=run_name, # Used in `wandb`, `tensorboard`, `neptune`, etc. if installed\n )\n\n # 5. (Optional) Create an evaluator & evaluate the base model\n dev_evaluator = TripletEvaluator(\n anchors=eval_dataset[\"query\"],\n positives=eval_dataset[\"positive\"],\n negatives=eval_dataset[\"negative\"],\n name=\"msmarco-co-condenser-dev\",\n )\n dev_evaluator(model)\n\n # 6. Create a trainer & train\n trainer = SentenceTransformerTrainer(\n model=model,\n args=args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n loss=loss,\n evaluator=dev_evaluator,\n )\n trainer.train()\n\n # 7. (Optional) Evaluate the trained model on the evaluator after training\n dev_evaluator(model)\n\n # 8. Save the model\n model.save_pretrained(f\"output/{model_shortname}/{run_name}/final\")\n\n # 9. (Optional) Push it to the Hugging Face Hub\n model.push_to_hub(run_name, private=False)\n\nif __name__ == \"__main__\":\n main()\n```\n\nClick to see how to finetune this into a multi-vector embedding model with PyLate\n\n```python\nfrom datasets import load_dataset\nfrom pylate import losses, models, utils\nfrom sentence_transformers import (\n SentenceTransformerTrainer,\n SentenceTransformerTrainingArguments,\n)\n\ndef main():\n # Load the datasets required for knowledge distillation (train, queries, documents)\n train = load_dataset(\n path=\"lightonai/ms-marco-en-bge\",\n name=\"train\",\n )\n\n queries = load_dataset(\n path=\"lightonai/ms-marco-en-bge\",\n name=\"queries\",\n )\n\n documents = load_dataset(\n path=\"lightonai/ms-marco-en-bge\",\n name=\"documents\",\n )\n\n # Set the transformation to load the documents/queries texts using the corresponding ids on the fly\n train.set_transform(\n utils.KDProcessing(queries=queries, documents=documents).transform,\n )\n\n # Define the base model, training parameters, and output directory\n num_train_epochs = 1\n lr = 8e-5\n batch_size = 16\n accum_steps = 1\n model_name = \"jhu-clsp/mmBERT-small\"\n model_shortname = model_name.split(\"/\")[-1]\n\n # Set the run name for logging and output directory\n run_name = f\"{model_shortname}-colbert-KD-{lr}\"\n output_dir = f\"output/{model_shortname}/{run_name}\"\n\n # Initialize the ColBERT model from the base model\n model = models.ColBERT(model_name_or_path=model_name)\n\n # Configure the training arguments (e.g., epochs, batch size, learning rate)\n args = SentenceTransformerTrainingArguments(\n output_dir=output_dir,\n num_train_epochs=num_train_epochs,\n per_device_train_batch_size=batch_size,\n fp16=False, # Set to False if you get an error that your GPU can't run on FP16\n bf16=True, # Set to True if you have a GPU that supports BF16\n run_name=run_name,\n logging_steps=10,\n learning_rate=lr,\n gradient_accumulation_steps=accum_steps,\n warmup_ratio=0.05,\n )\n\n # Use the Distillation loss function for training\n train_loss = losses.Distillation(model=model)\n\n # Initialize the trainer\n trainer = SentenceTransformerTrainer(\n model=model,\n args=args,\n train_dataset=train,\n loss=train_loss,\n data_collator=utils.ColBERTCollator(tokenize_fn=model.tokenize),\n )\n\n # Start the training process\n trainer.train()\n\n model.save_pretrained(f\"{output_dir}/final\")\n\nif __name__ == \"__main__\":\n main()\n```\n\nClick to see how to finetune this into a sparse retrieval model using Sentence Transformers\n\n```python\nimport logging\n\nfrom datasets import load_dataset\n\nfrom sentence_transformers import (\n SparseEncoder,\n SparseEncoderModelCardData,\n SparseEncoderTrainer,\n SparseEncoderTrainingArguments,\n)\nfrom sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator\nfrom sentence_transformers.sparse_encoder.losses import SparseMultipleNegativesRankingLoss, SpladeLoss\nfrom sentence_transformers.training_args import BatchSamplers\n\nlogging.basicConfig(format=\"%(asctime)s - %(message)s\", datefmt=\"%Y-%m-%d %H:%M:%S\", level=logging.INFO)\n\n# 1. Load a model to finetune with 2. (Optional) model card data\nmodel = SparseEncoder(\n \"jhu-clsp/mmBERT-small\",\n model_card_data=SparseEncoderModelCardData(\n language=\"en\",\n license=\"apache-2.0\",\n )\n)\n\n# 3. Load a dataset to finetune on\nfull_dataset = load_dataset(\"sentence-transformers/natural-questions\", split=\"train\").select(range(100_000))\ndataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)\ntrain_dataset = dataset_dict[\"train\"]\neval_dataset = dataset_dict[\"test\"]\n\n# 4. Define a loss function\nloss = SpladeLoss(\n model=model,\n loss=SparseMultipleNegativesRankingLoss(model=model),\n query_regularizer_weight=5e-5,\n document_regularizer_weight=3e-5,\n)\n\n# 5. (Optional) Specify training arguments\nrun_name = \"splade-distilbert-base-uncased-nq\"\nargs = SparseEncoderTrainingArguments(\n # Required parameter:\n output_dir=f\"models/{run_name}\",\n # Optional training parameters:\n num_train_epochs=1,\n per_device_train_batch_size=16,\n per_device_eval_batch_size=16,\n learning_rate=2e-5,\n warmup_ratio=0.1,\n fp16=True, # Set to False if you get an error that your GPU can't run on FP16\n bf16=False, # Set to True if you have a GPU that supports BF16\n batch_sampler=BatchSamplers.NO_DUPLICATES, # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch\n # Optional tracking/debugging parameters:\n eval_strategy=\"steps\",\n eval_steps=1000,\n save_strategy=\"steps\",\n save_steps=1000,\n save_total_limit=2,\n logging_steps=200,\n run_name=run_name, # Will be used in W&B if `wandb` is installed\n)\n\n# 6. (Optional) Create an evaluator & evaluate the base model\ndev_evaluator = SparseNanoBEIREvaluator(dataset_names=[\"msmarco\", \"nfcorpus\", \"nq\"], batch_size=16)\n\n# 7. Create a trainer & train\ntrainer = SparseEncoderTrainer(\n model=model,\n args=args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n loss=loss,\n evaluator=dev_evaluator,\n)\ntrainer.train()\n\n# 8. Evaluate the model performance again after training\ndev_evaluator(model)\n\n# 9. Save the trained model\nmodel.save_pretrained(f\"models/{run_name}/final\")\n\n# 10. (Optional) Push it to the Hugging Face Hub\nmodel.push_to_hub(run_name)\n```\n\nClick to see how to finetune this into a reranker model using Sentence Transformers\n\n```python\nimport logging\nimport traceback\n\nimport torch\nfrom datasets import load_dataset\n\nfrom sentence_transformers import SentenceTransformer\nfrom sentence_transformers.cross_encoder import (\n CrossEncoder,\n CrossEncoderModelCardData,\n CrossEncoderTrainer,\n CrossEncoderTrainingArguments,\n)\nfrom sentence_transformers.cross_encoder.evaluation import (\n CrossEncoderNanoBEIREvaluator,\n CrossEncoderRerankingEvaluator,\n)\nfrom sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss\nfrom sentence_transformers.evaluation import SequentialEvaluator\nfrom sentence_transformers.util import mine_hard_negatives\n\n# Set the log level to INFO to get more information\nlogging.basicConfig(format=\"%(asctime)s - %(message)s\", datefmt=\"%Y-%m-%d %H:%M:%S\", level=logging.INFO)\n\n\ndef main():\n model_name = \"jhu-clsp/mmBERT-small\"\n\n train_batch_size = 64\n num_epochs = 1\n num_hard_negatives = 5 # How many hard negatives should be mined for each question-answer pair\n\n # 1a. Load a model to finetune with 1b. (Optional) model card data\n model = CrossEncoder(\n model_name,\n model_card_data=CrossEncoderModelCardData(\n language=\"en\",\n license=\"apache-2.0\",\n ),\n )\n print(\"Model max length:\", model.max_length)\n print(\"Model num labels:\", model.num_labels)\n\n # 2a. Load the GooAQ dataset: https://huggingface.co/datasets/sentence-transformers/gooaq\n logging.info(\"Read the gooaq training dataset\")\n full_dataset = load_dataset(\"sentence-transformers/gooaq\", split=\"train\").select(range(100_000))\n dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)\n train_dataset = dataset_dict[\"train\"]\n eval_dataset = dataset_dict[\"test\"]\n logging.info(train_dataset)\n logging.info(eval_dataset)\n\n # 2b. Modify our training dataset to include hard negatives using a very efficient embedding model\n embedding_model = SentenceTransformer(\"sentence-transformers/static-retrieval-mrl-en-v1\", device=\"cpu\")\n hard_train_dataset = mine_hard_negatives(\n train_dataset,\n embedding_model,\n num_negatives=num_hard_negatives, # How many negatives per question-answer pair\n margin=0, # Similarity between query and negative samples should be x lower than query-positive similarity\n range_min=0, # Skip the x most similar samples\n range_max=100, # Consider only the x most similar samples\n sampling_strategy=\"top\", # Sample the top negatives from the range\n batch_size=4096, # Use a batch size of 4096 for the embedding model\n output_format=\"labeled-pair\", # The output format is (query, passage, label), as required by BinaryCrossEntropyLoss\n use_faiss=True,\n )\n logging.info(hard_train_dataset)\n\n # 2c. (Optionally) Save the hard training dataset to disk\n # hard_train_dataset.save_to_disk(\"gooaq-hard-train\")\n # Load again with:\n # hard_train_dataset = load_from_disk(\"gooaq-hard-train\")\n\n # 3. Define our training loss.\n # pos_weight is recommended to be set as the ratio between positives to negatives, a.k.a. `num_hard_negatives`\n loss = BinaryCrossEntropyLoss(model=model, pos_weight=torch.tensor(num_hard_negatives))\n\n # 4a. Define evaluators. We use the CrossEncoderNanoBEIREvaluator, which is a light-weight evaluator for English reranking\n nano_beir_evaluator = CrossEncoderNanoBEIREvaluator(\n dataset_names=[\"msmarco\", \"nfcorpus\", \"nq\"],\n batch_size=train_batch_size,\n )\n\n # 4b. Define a reranking evaluator by mining hard negatives given query-answer pairs\n # We include the positive answer in the list of negatives, so the evaluator can use the performance of the\n # embedding model as a baseline.\n hard_eval_dataset = mine_hard_negatives(\n eval_dataset,\n embedding_model,\n corpus=full_dataset[\"answer\"], # Use the full dataset as the corpus\n num_negatives=30, # How many documents to rerank\n batch_size=4096,\n include_positives=True,\n output_format=\"n-tuple\",\n use_faiss=True,\n )\n logging.info(hard_eval_dataset)\n reranking_evaluator = CrossEncoderRerankingEvaluator(\n samples=[\n {\n \"query\": sample[\"question\"],\n \"positive\": [sample[\"answer\"]],\n \"documents\": [sample[column_name] for column_name in hard_eval_dataset.column_names[2:]],\n }\n for sample in hard_eval_dataset\n ],\n batch_size=train_batch_size,\n name=\"gooaq-dev\",\n # Realistic setting: only rerank the positives that the retriever found\n # Set to True to rerank *all* positives\n always_rerank_positives=False,\n )\n\n # 4c. Combine the evaluators & run the base model on them\n evaluator = SequentialEvaluator([reranking_evaluator, nano_beir_evaluator])\n evaluator(model)\n\n # 5. Define the training arguments\n short_model_name = model_name if \"/\" not in model_name else model_name.split(\"/\")[-1]\n run_name = f\"reranker-{short_model_name}-gooaq-bce\"\n args = CrossEncoderTrainingArguments(\n # Required parameter:\n output_dir=f\"models/{run_name}\",\n # Optional training parameters:\n num_train_epochs=num_epochs,\n per_device_train_batch_size=train_batch_size,\n per_device_eval_batch_size=train_batch_size,\n learning_rate=2e-5,\n warmup_ratio=0.1,\n fp16=False, # Set to False if you get an error that your GPU can't run on FP16\n bf16=True, # Set to True if you have a GPU that supports BF16\n dataloader_num_workers=4,\n load_best_model_at_end=True,\n metric_for_best_model=\"eval_gooaq-dev_ndcg@10\",\n # Optional tracking/debugging parameters:\n eval_strategy=\"steps\",\n eval_steps=1000,\n save_strategy=\"steps\",\n save_steps=1000,\n save_total_limit=2,\n logging_steps=200,\n logging_first_step=True,\n run_name=run_name, # Will be used in W&B if `wandb` is installed\n seed=12,\n )\n\n # 6. Create the trainer & start training\n trainer = CrossEncoderTrainer(\n model=model,\n args=args,\n train_dataset=hard_train_dataset,\n loss=loss,\n evaluator=evaluator,\n )\n trainer.train()\n\n # 7. Evaluate the final model, useful to include these in the model card\n evaluator(model)\n\n # 8. Save the final model\n final_output_dir = f\"models/{run_name}/final\"\n model.save_pretrained(final_output_dir)\n\n # 9. (Optional) save the model to the Hugging Face Hub!\n # It is recommended to run `huggingface-cli login` to log into your Hugging Face account first\n try:\n model.push_to_hub(run_name)\n except Exception:\n logging.error(\n f\"Error uploading model to the Hugging Face Hub:\\n{traceback.format_exc()}To upload it manually, you can run \"\n f\"`huggingface-cli login`, followed by loading the model using `model = CrossEncoder({final_output_dir!r})` \"\n f\"and saving it using `model.push_to_hub('{run_name}')`.\"\n )\n\n\nif __name__ == \"__main__\":\n main()\n```\n\n## Model Family and Links\n\n**Standard Models:**\n\n- [mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small) (140M params total, 42M non-embed)\n- [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) (307M params total, 110M non-embed)\n\n**Research Resources:**\n\n- [🤗 mmBERT Model Collection](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4)\n- [📝 Paper](https://arxiv.org/abs/2509.06888)\n- [🗂️ Training Data](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p1-fineweb2-langs) (3T+ tokens, fully open)\n- [💻 GitHub Repository](https://github.com/jhu-clsp/mmBERT)\n- [📊 Training Checkpoints](https://huggingface.co/jhu-clsp/mmBERT-checkpoints) for studying training or continued pre-training\n", "authors": [], "captured_at": "2026-10-04T12:50:30.804479567+00:00", "category": "ai", "content_html": "

TL;DR

\n

This blog post introduces mmBERT, a state-of-the-art massively multilingual encoder model trained on 3T+ tokens of text in over 1800 languages. It shows significant performance and speed improvements over previous multilingual models, being the first to improve upon XLM-R, while also developing new strategies for effectively learning low-resource languages. mmBERT builds upon ModernBERT for a blazingly fast architecture, and adds novel components to enable efficient multilingual learning.

\n

If you are interested in trying out the models yourself, some example boilerplate is available at the end of this blogpost!

\n

Training Data

\n
\"Distribution
Figure 1: the training data is progressively annealed to include more languages and more uniform sampling throughout training.
\n

mmBERT was trained on a carefully curated multilingual dataset totaling over 3T tokens across three distinct training phases. The foundation of our training data consists of three primary open-source and high-quality web crawls that enable both multilingual coverage and data quality:

\n

DCLM and Filtered DCLM provides the highest quality English content available, serving as the backbone for strong English performance (with the filtered data coming from Dolmino). This dataset represents state-of-the-art web filtering techniques and forms a crucial component. Due to the high quality of this data, we use a signficantly higher proportion of English than previous generation multilingual encoder models (up to 18%).

\n

FineWeb2 delivers broad multilingual web content covering over 1,800 languages. This dataset enables our extensive multilingual coverage while maintaining reasonable quality standards across diverse language families and scripts.

\n

FineWeb2-HQ consists of a filtered subset of FineWeb2 focusing on 20 high-resource languages. This filtered version provides higher-quality multilingual content that bridges the gap between English-only filtered data and broad multilingual coverage.

\n

The training data also incorporates specialized corpora from Dolma, MegaWika v2, ProLong and more: code repositories (StarCoder, ProLong), academic content (ArXiv, PeS2o), reference materials (Wikipedia, textbooks), and community discussions (StackExchange), along with instruction and mathematical datasets.

\n

The key innovation in our data approach is the progressive language inclusion strategy shown in Figure 1. At each phase we progressively sample from a flatter distribution (i.e. closer to uniform), while also adding new languages. This means that high resource languages like Russian start off with a high percentage of the data (i.e. 9%) and then in the last phase of training end around half of that. We start with 60 high-resource languages during pre-training, expand to 110 languages during mid-training, and finally include all 1,833 languages from FineWeb2 during the decay phase. This allows us to maximize the impact of limited low-resource language data without excess reptitions and while maintaining high overall data quality.

\n

Training Recipe and Novel Components

\n

mmBERT builds upon the ModernBERT architecture but introduces several key innovations for multilingual learning:

\n

Architecture

\n

We use the same core architecture as ModernBERT-base with 22 layers and 1152 intermediate dimensions, but switch to the Gemma 2 tokenizer to better handle multilingual text. The base model has 110M non-embedding parameters (307M total due to the larger vocabulary), while the small variant has 42M non-embedding parameters (140M total).

\n

Three-Phase Training Approach

\n

Our training follows a carefully designed three-phase schedule:

\n
    \n
  1. Pre-training (2.3T tokens): Warmup and stable learning rate phase using 60 languages with 30% mask rate
    2. \n
  2. Mid-training (600B tokens): Context extension to 8192 tokens, higher-quality data, expanded to 110 languages with 15% mask rate
    3. \n
  3. Decay phase (100B tokens): Inverse square root learning rate decay, all 1,833 languages included with 5% mask rate
    4. \n
\n

Novel Training Techniques

\n

Inverse Mask Ratio Schedule: Instead of using a fixed masking rate, we progressively reduce the mask ratio from 30% → 15% → 5% across training phases. This allows the model to learn basic representations with higher masking early on, then focus on more nuanced understanding with lower masking rates.

\n

Annealed Language Learning: We dynamically adjust the temperature for multilingual data sampling from τ=0.7 → 0.5 → 0.3. This creates a progression from high-resource language bias toward more uniform sampling, enabling the model to build a strong multilingual foundation before learning low-resource languages.

\n

Progressive Language Addition: Rather than training on all languages simultaneously, we strategically add languages at each phase (60 → 110 → 1,833). This maximizes learning efficiency by avoiding excessive epochs on limited low-resource data while still achieving strong performance.

\n

Model Merging: We train three different variants during the decay phase (English-focused, 110-language, and all-language) and use TIES merging to combine their strengths into the final model.

\n

Results

\n

Natural Language Understanding (NLU)

\n
\"GLUE
Table 1: Performance on GLUE (English)
\n

English Performance: On the English GLUE benchmark (Table 1), mmBERT base achieves strong performance, substantially outperforming other multilingual models like XLM-R (multilingual RoBERTa) base and mGTE base, while remaining competitive to English-only models despite less than 25% of the mmBERT training data being English.

\n
\"XTREME
Table 2: Performance on XTREME (Multilingual)
\n

Multilingual Performance: mmBERT shows significant improvements on XTREME benchmark compared to XLM-R as demonstrated in Table 2. Notable gains include strong performance on XNLI classification, substantial improvements in question answering tasks like TyDiQA, and competitive results across PAWS-X and XCOPA for cross-lingual understanding.

\n

The model performs well across most categories, with the exception of some structured prediction tasks like NER and POS tagging, likely due to tokenizer differences that affect word boundary detection. On these categories, it performs about the same as the previous generation, but can be applied to more languages.

\n

Retrieval Performance

\n
\"MTEB
Table 3: Performance on MTEB v2 English
\n

English Retrieval: Even though mmBERT is designed for massively multilingual settings, in the MTEB v2 English benchmarks (Table 3), mmBERT shows significant gains over previous multilingual models and even ties the capabilities of English-only models like ModernBERT!

\n
\"MTEB
Table 4: Performance on MTEB v2 Multilingual
\n

Multilingual Retrieval: mmBERT shows consistent improvements on MTEB v2 multilingual benchmarks compared to other models (Table 4).

\n
\"CoIR
Table 5: Performance on CoIR code benchmark
\n

Code Retrieval: Due to the modern tokenizer (based on Gemma 2) mmBERT also shows strong coding performance (Table 5), making mmBERT suitable for any type of textual data. The only model that outperforms it is EuroBERT, which was able to use the non-publicly accessible Stack v2 dataset.

\n

Learning Languages in the Decay Phase

\n

One of mmBERT's most significant novel features is demonstrating that low-resource languages can be effectively learned during the short decay phase of training. We validated this approach by testing on languages only introduced during the final 100B token decay phase.

\n
\"Improvements
Figure 2: adding more than 1700 languages in the decay phase allows for rapid learning, which we keep through model merging.
\n

Dramatic Performance Gains: Testing on TiQuaD (Tigrinya) and FoQA (Faroese), we observed substantial improvements when these languages were included in the decay phase, as shown in Figure 2. The results demonstrate the effectiveness of our progressive language learning approach.

\n

Competitive with Large Models: Despite only seeing these languages in the final training phase, mmBERT achieves performance levels that exceed much larger models. On Faroese question answering where LLMs have been benchmarked, mmBERT outperforms Google Gemini 2.5 Pro and OpenAI o3.

\n

Rapid Learning Mechanism: The success of decay-phase language learning stems from the model's ability to leverage its strong multilingual foundation built during earlier phases. When exposed to new languages, the model can quickly adapt existing cross-lingual representations rather than learning from scratch.

\n

Model Merging Benefits: The final mmBERT models successfully retain most of the decay-phase improvements while benefiting from the English-focused and high-resource variants through TIES merging.

\n

Efficiency Improvements

\n

mmBERT delivers substantial efficiency gains over previous multilingual encoder models through architectural improvements inherited from ModernBERT:

\n
\"mmBERT
Figure 3: mmBERT is significantly more efficient than previous multilingual models, up to 2-4x as much!
\n

Throughput Performance: mmBERT processes text significantly faster than existing multilingual models across various sequence lengths, as demonstrated in Figure 3. Both the small and base models show substantial speed improvements over previous multilingual encoders.

\n

Modern Architecture Benefits: The efficiency gains come from two main technical improvements:

\n
    \n
  • Flash Attention 2: Optimized attention computation for better memory usage and speed
    • \n
  • Unpadding techniques: Elimination of unnecessary padding tokens during processing
    • \n
\n

Sequence Length Scaling: Unlike older models limited to 512 tokens, mmBERT handles up to 8,192 tokens efficiently while maintaining high throughput. This makes it suitable for longer document processing tasks that are increasingly common in multilingual applications.

\n

Energy Efficiency: The combination of better throughput and modern architecture results in lower computational costs for inference, making mmBERT more practical for production deployments where multilingual support is needed at scale.

\n

These efficiency improvements make mmBERT not just more accurate than previous multilingual encoders, but also significantly more practical for real usage.

\n

Usage Examples

\n

You can use these models with just a few lines of code!

\n
from transformers import AutoTokenizer, AutoModelForMaskedLM\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-base")\nmodel = AutoModelForMaskedLM.from_pretrained("jhu-clsp/mmBERT-base")\n\ndef predict_masked_token(text):\n    inputs = tokenizer(text, return_tensors="pt")\n    with torch.no_grad():\n        outputs = model(**inputs)\n    mask_indices = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)\n    predictions = outputs.logits[mask_indices]\n    top_tokens, top_indices = torch.topk(predictions, 5, dim=-1)\n    return [tokenizer.decode(token) for token in top_indices[0]]\n\n# Works across languages\ntexts = [\n    "The capital of France is <mask>.",\n    "La capital de España es <mask>.",\n    "Die Hauptstadt von Deutschland ist <mask>.",\n]\n\nfor text in texts:\n    predictions = predict_masked_token(text)\n    print(f"Text: {text}")\n    print(f"Predictions: {predictions}\\n")\n
\n

Fine-tuning Examples

\n

Encoders

\n

Click to see how to finetune this into a dense embedding model using Sentence Transformers

\n
import argparse\n\nfrom datasets import load_dataset\nfrom sentence_transformers import (\n    SentenceTransformer,\n    SentenceTransformerTrainer,\n    SentenceTransformerTrainingArguments,\n)\nfrom sentence_transformers.evaluation import TripletEvaluator\nfrom sentence_transformers.losses import CachedMultipleNegativesRankingLoss\nfrom sentence_transformers.training_args import BatchSamplers\n\ndef main():\n    # parse the lr & model name\n    parser = argparse.ArgumentParser()\n    parser.add_argument("--lr", type=float, default=8e-5)\n    parser.add_argument("--model_name", type=str, default="jhu-clsp/mmBERT-small")\n    args = parser.parse_args()\n    lr = args.lr\n    model_name = args.model_name\n    model_shortname = model_name.split("/")[-1]\n\n    # 1. Load a model to finetune\n    model = SentenceTransformer(model_name)\n\n    # 2. Load a dataset to finetune on\n    dataset = load_dataset(\n        "sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1",\n        "triplet-hard",\n        split="train",\n    )\n    dataset_dict = dataset.train_test_split(test_size=1_000, seed=12)\n    train_dataset = dataset_dict["train"].select(range(1_250_000))\n    eval_dataset = dataset_dict["test"]\n\n    # 3. Define a loss function\n    loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=16)  # Increase mini_batch_size if you have enough VRAM\n\n    run_name = f"{model_shortname}-DPR-{lr}"\n    # 4. (Optional) Specify training arguments\n    args = SentenceTransformerTrainingArguments(\n        # Required parameter:\n        output_dir=f"output/{model_shortname}/{run_name}",\n        # Optional training parameters:\n        num_train_epochs=1,\n        per_device_train_batch_size=512,\n        per_device_eval_batch_size=512,\n        warmup_ratio=0.05,\n        fp16=False,  # Set to False if GPU can't handle FP16\n        bf16=True,  # Set to True if GPU supports BF16\n        batch_sampler=BatchSamplers.NO_DUPLICATES,  # (Cached)MultipleNegativesRankingLoss benefits from no duplicates\n        learning_rate=lr,\n        # Optional tracking/debugging parameters:\n        save_strategy="steps",\n        save_steps=500,\n        save_total_limit=2,\n        logging_steps=500,\n        run_name=run_name,  # Used in `wandb`, `tensorboard`, `neptune`, etc. if installed\n    )\n\n    # 5. (Optional) Create an evaluator & evaluate the base model\n    dev_evaluator = TripletEvaluator(\n        anchors=eval_dataset["query"],\n        positives=eval_dataset["positive"],\n        negatives=eval_dataset["negative"],\n        name="msmarco-co-condenser-dev",\n    )\n    dev_evaluator(model)\n\n    # 6. Create a trainer & train\n    trainer = SentenceTransformerTrainer(\n        model=model,\n        args=args,\n        train_dataset=train_dataset,\n        eval_dataset=eval_dataset,\n        loss=loss,\n        evaluator=dev_evaluator,\n    )\n    trainer.train()\n\n    # 7. (Optional) Evaluate the trained model on the evaluator after training\n    dev_evaluator(model)\n\n    # 8. Save the model\n    model.save_pretrained(f"output/{model_shortname}/{run_name}/final")\n\n    # 9. (Optional) Push it to the Hugging Face Hub\n    model.push_to_hub(run_name, private=False)\n\nif __name__ == "__main__":\n    main()\n
\n

Click to see how to finetune this into a multi-vector embedding model with PyLate

\n
from datasets import load_dataset\nfrom pylate import losses, models, utils\nfrom sentence_transformers import (\n    SentenceTransformerTrainer,\n    SentenceTransformerTrainingArguments,\n)\n\ndef main():\n    # Load the datasets required for knowledge distillation (train, queries, documents)\n    train = load_dataset(\n        path="lightonai/ms-marco-en-bge",\n        name="train",\n    )\n\n    queries = load_dataset(\n        path="lightonai/ms-marco-en-bge",\n        name="queries",\n    )\n\n    documents = load_dataset(\n        path="lightonai/ms-marco-en-bge",\n        name="documents",\n    )\n\n    # Set the transformation to load the documents/queries texts using the corresponding ids on the fly\n    train.set_transform(\n        utils.KDProcessing(queries=queries, documents=documents).transform,\n    )\n\n    # Define the base model, training parameters, and output directory\n    num_train_epochs = 1\n    lr = 8e-5\n    batch_size = 16\n    accum_steps = 1\n    model_name = "jhu-clsp/mmBERT-small"\n    model_shortname = model_name.split("/")[-1]\n\n    # Set the run name for logging and output directory\n    run_name = f"{model_shortname}-colbert-KD-{lr}"\n    output_dir = f"output/{model_shortname}/{run_name}"\n\n    # Initialize the ColBERT model from the base model\n    model = models.ColBERT(model_name_or_path=model_name)\n\n    # Configure the training arguments (e.g., epochs, batch size, learning rate)\n    args = SentenceTransformerTrainingArguments(\n        output_dir=output_dir,\n        num_train_epochs=num_train_epochs,\n        per_device_train_batch_size=batch_size,\n        fp16=False,  # Set to False if you get an error that your GPU can't run on FP16\n        bf16=True,  # Set to True if you have a GPU that supports BF16\n        run_name=run_name,\n        logging_steps=10,\n        learning_rate=lr,\n        gradient_accumulation_steps=accum_steps,\n        warmup_ratio=0.05,\n    )\n\n    # Use the Distillation loss function for training\n    train_loss = losses.Distillation(model=model)\n\n    # Initialize the trainer\n    trainer = SentenceTransformerTrainer(\n        model=model,\n        args=args,\n        train_dataset=train,\n        loss=train_loss,\n        data_collator=utils.ColBERTCollator(tokenize_fn=model.tokenize),\n    )\n\n    # Start the training process\n    trainer.train()\n\n    model.save_pretrained(f"{output_dir}/final")\n\nif __name__ == "__main__":\n    main()\n
\n

Click to see how to finetune this into a sparse retrieval model using Sentence Transformers

\n
import logging\n\nfrom datasets import load_dataset\n\nfrom sentence_transformers import (\n    SparseEncoder,\n    SparseEncoderModelCardData,\n    SparseEncoderTrainer,\n    SparseEncoderTrainingArguments,\n)\nfrom sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator\nfrom sentence_transformers.sparse_encoder.losses import SparseMultipleNegativesRankingLoss, SpladeLoss\nfrom sentence_transformers.training_args import BatchSamplers\n\nlogging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)\n\n# 1. Load a model to finetune with 2. (Optional) model card data\nmodel = SparseEncoder(\n    "jhu-clsp/mmBERT-small",\n    model_card_data=SparseEncoderModelCardData(\n        language="en",\n        license="apache-2.0",\n    )\n)\n\n# 3. Load a dataset to finetune on\nfull_dataset = load_dataset("sentence-transformers/natural-questions", split="train").select(range(100_000))\ndataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)\ntrain_dataset = dataset_dict["train"]\neval_dataset = dataset_dict["test"]\n\n# 4. Define a loss function\nloss = SpladeLoss(\n    model=model,\n    loss=SparseMultipleNegativesRankingLoss(model=model),\n    query_regularizer_weight=5e-5,\n    document_regularizer_weight=3e-5,\n)\n\n# 5. (Optional) Specify training arguments\nrun_name = "splade-distilbert-base-uncased-nq"\nargs = SparseEncoderTrainingArguments(\n    # Required parameter:\n    output_dir=f"models/{run_name}",\n    # Optional training parameters:\n    num_train_epochs=1,\n    per_device_train_batch_size=16,\n    per_device_eval_batch_size=16,\n    learning_rate=2e-5,\n    warmup_ratio=0.1,\n    fp16=True,  # Set to False if you get an error that your GPU can't run on FP16\n    bf16=False,  # Set to True if you have a GPU that supports BF16\n    batch_sampler=BatchSamplers.NO_DUPLICATES,  # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch\n    # Optional tracking/debugging parameters:\n    eval_strategy="steps",\n    eval_steps=1000,\n    save_strategy="steps",\n    save_steps=1000,\n    save_total_limit=2,\n    logging_steps=200,\n    run_name=run_name,  # Will be used in W&B if `wandb` is installed\n)\n\n# 6. (Optional) Create an evaluator & evaluate the base model\ndev_evaluator = SparseNanoBEIREvaluator(dataset_names=["msmarco", "nfcorpus", "nq"], batch_size=16)\n\n# 7. Create a trainer & train\ntrainer = SparseEncoderTrainer(\n    model=model,\n    args=args,\n    train_dataset=train_dataset,\n    eval_dataset=eval_dataset,\n    loss=loss,\n    evaluator=dev_evaluator,\n)\ntrainer.train()\n\n# 8. Evaluate the model performance again after training\ndev_evaluator(model)\n\n# 9. Save the trained model\nmodel.save_pretrained(f"models/{run_name}/final")\n\n# 10. (Optional) Push it to the Hugging Face Hub\nmodel.push_to_hub(run_name)\n
\n

Click to see how to finetune this into a reranker model using Sentence Transformers

\n
import logging\nimport traceback\n\nimport torch\nfrom datasets import load_dataset\n\nfrom sentence_transformers import SentenceTransformer\nfrom sentence_transformers.cross_encoder import (\n    CrossEncoder,\n    CrossEncoderModelCardData,\n    CrossEncoderTrainer,\n    CrossEncoderTrainingArguments,\n)\nfrom sentence_transformers.cross_encoder.evaluation import (\n    CrossEncoderNanoBEIREvaluator,\n    CrossEncoderRerankingEvaluator,\n)\nfrom sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss\nfrom sentence_transformers.evaluation import SequentialEvaluator\nfrom sentence_transformers.util import mine_hard_negatives\n\n# Set the log level to INFO to get more information\nlogging.basicConfig(format="%(asctime)s - %(message)s", datefmt="%Y-%m-%d %H:%M:%S", level=logging.INFO)\n\n\ndef main():\n    model_name = "jhu-clsp/mmBERT-small"\n\n    train_batch_size = 64\n    num_epochs = 1\n    num_hard_negatives = 5  # How many hard negatives should be mined for each question-answer pair\n\n    # 1a. Load a model to finetune with 1b. (Optional) model card data\n    model = CrossEncoder(\n        model_name,\n        model_card_data=CrossEncoderModelCardData(\n            language="en",\n            license="apache-2.0",\n        ),\n    )\n    print("Model max length:", model.max_length)\n    print("Model num labels:", model.num_labels)\n\n    # 2a. Load the GooAQ dataset: https://huggingface.co/datasets/sentence-transformers/gooaq\n    logging.info("Read the gooaq training dataset")\n    full_dataset = load_dataset("sentence-transformers/gooaq", split="train").select(range(100_000))\n    dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)\n    train_dataset = dataset_dict["train"]\n    eval_dataset = dataset_dict["test"]\n    logging.info(train_dataset)\n    logging.info(eval_dataset)\n\n    # 2b. Modify our training dataset to include hard negatives using a very efficient embedding model\n    embedding_model = SentenceTransformer("sentence-transformers/static-retrieval-mrl-en-v1", device="cpu")\n    hard_train_dataset = mine_hard_negatives(\n        train_dataset,\n        embedding_model,\n        num_negatives=num_hard_negatives,  # How many negatives per question-answer pair\n        margin=0,  # Similarity between query and negative samples should be x lower than query-positive similarity\n        range_min=0,  # Skip the x most similar samples\n        range_max=100,  # Consider only the x most similar samples\n        sampling_strategy="top",  # Sample the top negatives from the range\n        batch_size=4096,  # Use a batch size of 4096 for the embedding model\n        output_format="labeled-pair",  # The output format is (query, passage, label), as required by BinaryCrossEntropyLoss\n        use_faiss=True,\n    )\n    logging.info(hard_train_dataset)\n\n    # 2c. (Optionally) Save the hard training dataset to disk\n    # hard_train_dataset.save_to_disk("gooaq-hard-train")\n    # Load again with:\n    # hard_train_dataset = load_from_disk("gooaq-hard-train")\n\n    # 3. Define our training loss.\n    # pos_weight is recommended to be set as the ratio between positives to negatives, a.k.a. `num_hard_negatives`\n    loss = BinaryCrossEntropyLoss(model=model, pos_weight=torch.tensor(num_hard_negatives))\n\n    # 4a. Define evaluators. We use the CrossEncoderNanoBEIREvaluator, which is a light-weight evaluator for English reranking\n    nano_beir_evaluator = CrossEncoderNanoBEIREvaluator(\n        dataset_names=["msmarco", "nfcorpus", "nq"],\n        batch_size=train_batch_size,\n    )\n\n    # 4b. Define a reranking evaluator by mining hard negatives given query-answer pairs\n    # We include the positive answer in the list of negatives, so the evaluator can use the performance of the\n    # embedding model as a baseline.\n    hard_eval_dataset = mine_hard_negatives(\n        eval_dataset,\n        embedding_model,\n        corpus=full_dataset["answer"],  # Use the full dataset as the corpus\n        num_negatives=30,  # How many documents to rerank\n        batch_size=4096,\n        include_positives=True,\n        output_format="n-tuple",\n        use_faiss=True,\n    )\n    logging.info(hard_eval_dataset)\n    reranking_evaluator = CrossEncoderRerankingEvaluator(\n        samples=[\n            {\n                "query": sample["question"],\n                "positive": [sample["answer"]],\n                "documents": [sample[column_name] for column_name in hard_eval_dataset.column_names[2:]],\n            }\n            for sample in hard_eval_dataset\n        ],\n        batch_size=train_batch_size,\n        name="gooaq-dev",\n        # Realistic setting: only rerank the positives that the retriever found\n        # Set to True to rerank *all* positives\n        always_rerank_positives=False,\n    )\n\n    # 4c. Combine the evaluators & run the base model on them\n    evaluator = SequentialEvaluator([reranking_evaluator, nano_beir_evaluator])\n    evaluator(model)\n\n    # 5. Define the training arguments\n    short_model_name = model_name if "/" not in model_name else model_name.split("/")[-1]\n    run_name = f"reranker-{short_model_name}-gooaq-bce"\n    args = CrossEncoderTrainingArguments(\n        # Required parameter:\n        output_dir=f"models/{run_name}",\n        # Optional training parameters:\n        num_train_epochs=num_epochs,\n        per_device_train_batch_size=train_batch_size,\n        per_device_eval_batch_size=train_batch_size,\n        learning_rate=2e-5,\n        warmup_ratio=0.1,\n        fp16=False,  # Set to False if you get an error that your GPU can't run on FP16\n        bf16=True,  # Set to True if you have a GPU that supports BF16\n        dataloader_num_workers=4,\n        load_best_model_at_end=True,\n        metric_for_best_model="eval_gooaq-dev_ndcg@10",\n        # Optional tracking/debugging parameters:\n        eval_strategy="steps",\n        eval_steps=1000,\n        save_strategy="steps",\n        save_steps=1000,\n        save_total_limit=2,\n        logging_steps=200,\n        logging_first_step=True,\n        run_name=run_name,  # Will be used in W&B if `wandb` is installed\n        seed=12,\n    )\n\n    # 6. Create the trainer & start training\n    trainer = CrossEncoderTrainer(\n        model=model,\n        args=args,\n        train_dataset=hard_train_dataset,\n        loss=loss,\n        evaluator=evaluator,\n    )\n    trainer.train()\n\n    # 7. Evaluate the final model, useful to include these in the model card\n    evaluator(model)\n\n    # 8. Save the final model\n    final_output_dir = f"models/{run_name}/final"\n    model.save_pretrained(final_output_dir)\n\n    # 9. (Optional) save the model to the Hugging Face Hub!\n    # It is recommended to run `huggingface-cli login` to log into your Hugging Face account first\n    try:\n        model.push_to_hub(run_name)\n    except Exception:\n        logging.error(\n            f"Error uploading model to the Hugging Face Hub:\\n{traceback.format_exc()}To upload it manually, you can run "\n            f"`huggingface-cli login`, followed by loading the model using `model = CrossEncoder({final_output_dir!r})` "\n            f"and saving it using `model.push_to_hub('{run_name}')`."\n        )\n\n\nif __name__ == "__main__":\n    main()\n
\n

Model Family and Links

\n

Standard Models:

\n\n

Research Resources:

\n\n", "content_markdown": "## TL;DR\n\nThis blog post introduces [mmBERT](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4), a state-of-the-art massively multilingual encoder model trained on 3T+ tokens of text in over 1800 languages. It shows significant performance and speed improvements over previous multilingual models, being the first to improve upon XLM-R, while also developing new strategies for effectively learning low-resource languages. mmBERT builds upon ModernBERT for a blazingly fast architecture, and adds novel components to enable efficient multilingual learning.\n\nIf you are interested in trying out the models yourself, some example boilerplate is available [at the end of this blogpost!](https://huggingface.co/blog/mmbert#usage-examples)\n\n## Training Data\n\n![Distribution of each phase of pre-training](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/data_dist.jpg?raw=true)\\\n Figure 1: the training data is progressively annealed to include more languages and more uniform sampling throughout training.\n\nmmBERT was trained on a carefully curated multilingual dataset totaling over 3T tokens across three distinct training phases. The foundation of our training data consists of three primary open-source and high-quality web crawls that enable both multilingual coverage and data quality:\n\n**DCLM and Filtered DCLM** provides the highest quality English content available, serving as the backbone for strong English performance (with the filtered data coming from [Dolmino](https://huggingface.co/datasets/allenai/dolmino-mix-1124)). This dataset represents state-of-the-art web filtering techniques and forms a crucial component. Due to the high quality of this data, we use a signficantly higher proportion of English than previous generation multilingual encoder models (up to 18%).\n\n**FineWeb2** delivers broad [multilingual web content](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) covering over 1,800 languages. This dataset enables our extensive multilingual coverage while maintaining reasonable quality standards across diverse language families and scripts.\n\n**FineWeb2-HQ** consists of a [filtered subset of FineWeb2](https://huggingface.co/datasets/epfml/FineWeb2-HQ) focusing on 20 high-resource languages. This filtered version provides higher-quality multilingual content that bridges the gap between English-only filtered data and broad multilingual coverage.\n\nThe training data also incorporates specialized corpora from [Dolma](https://arxiv.org/abs/2402.00159), [MegaWika v2](https://arxiv.org/abs/2508.03828), [ProLong](https://arxiv.org/abs/2410.02660) and more: code repositories (StarCoder, ProLong), academic content (ArXiv, PeS2o), reference materials (Wikipedia, textbooks), and community discussions (StackExchange), along with instruction and mathematical datasets.\n\nThe key innovation in our data approach is the **progressive language inclusion strategy** shown in [Figure 1](https://huggingface.co/blog/mmbert#figure1). At each phase we progressively sample from a *flatter* distribution (i.e. closer to uniform), while also adding new languages. This means that high resource languages like Russian start off with a high percentage of the data (i.e. 9%) and then in the last phase of training end around half of that. We start with 60 high-resource languages during pre-training, expand to 110 languages during mid-training, and finally include all 1,833 languages from FineWeb2 during the decay phase. This allows us to maximize the impact of limited low-resource language data without excess reptitions and while maintaining high overall data quality.\n\n## Training Recipe and Novel Components\n\nmmBERT builds upon the [ModernBERT](https://huggingface.co/blog/modernbert) architecture but introduces several key innovations for multilingual learning:\n\n### Architecture\n\nWe use the same core architecture as ModernBERT-base with 22 layers and 1152 intermediate dimensions, but switch to the Gemma 2 tokenizer to better handle multilingual text. The base model has 110M non-embedding parameters (307M total due to the larger vocabulary), while the small variant has 42M non-embedding parameters (140M total).\n\n### Three-Phase Training Approach\n\nOur training follows a carefully designed three-phase schedule:\n\n1. **Pre-training (2.3T tokens)**: Warmup and stable learning rate phase using 60 languages with 30% mask rate\n2. **Mid-training (600B tokens)**: Context extension to 8192 tokens, higher-quality data, expanded to 110 languages with 15% mask rate\n3. **Decay phase (100B tokens)**: Inverse square root learning rate decay, all 1,833 languages included with 5% mask rate\n\n### Novel Training Techniques\n\n**Inverse Mask Ratio Schedule**: Instead of using a fixed masking rate, we progressively reduce the mask ratio from 30% → 15% → 5% across training phases. This allows the model to learn basic representations with higher masking early on, then focus on more nuanced understanding with lower masking rates.\n\n**Annealed Language Learning**: We dynamically adjust the temperature for multilingual data sampling from τ=0.7 → 0.5 → 0.3. This creates a progression from high-resource language bias toward more uniform sampling, enabling the model to build a strong multilingual foundation before learning low-resource languages.\n\n**Progressive Language Addition**: Rather than training on all languages simultaneously, we strategically add languages at each phase (60 → 110 → 1,833). This maximizes learning efficiency by avoiding excessive epochs on limited low-resource data while still achieving strong performance.\n\n**Model Merging**: We train three different variants during the decay phase (English-focused, 110-language, and all-language) and use TIES merging to combine their strengths into the final model.\n\n## Results\n\n### Natural Language Understanding (NLU)\n\n![GLUE performance](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/glue.jpeg?raw=true)\\\nTable 1: Performance on GLUE (English)\n\n**English Performance**: On the English GLUE benchmark ([Table 1](https://huggingface.co/blog/mmbert#table1)), mmBERT base achieves strong performance, substantially outperforming other multilingual models like XLM-R (multilingual RoBERTa) base and mGTE base, while remaining competitive to English-only models despite less than 25% of the mmBERT training data being English.\n\n![XTREME performance](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/xtreme.jpeg?raw=true)\\\nTable 2: Performance on XTREME (Multilingual)\n\n**Multilingual Performance**: mmBERT shows significant improvements on XTREME benchmark compared to XLM-R as demonstrated in [Table 2](https://huggingface.co/blog/mmbert#table2). Notable gains include strong performance on XNLI classification, substantial improvements in question answering tasks like TyDiQA, and competitive results across PAWS-X and XCOPA for cross-lingual understanding.\n\nThe model performs well across most categories, with the exception of some structured prediction tasks like NER and POS tagging, likely due to tokenizer differences that affect word boundary detection. On these categories, it performs about the same as the previous generation, but can be applied to more languages.\n\n### Retrieval Performance\n\n![MTEB v2 Eng performance](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/mteb-english.jpeg?raw=true)\\\nTable 3: Performance on MTEB v2 English\n\n**English Retrieval**: Even though mmBERT is designed for massively multilingual settings, in the MTEB v2 English benchmarks ([Table 3](https://huggingface.co/blog/mmbert#table3)), mmBERT shows significant gains over previous multilingual models and even ties the capabilities of English-only models like ModernBERT!\n\n![MTEB v2 Multilingual performance](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/mteb-multilingual.jpeg?raw=true)\\\nTable 4: Performance on MTEB v2 Multilingual\n\n**Multilingual Retrieval**: mmBERT shows consistent improvements on MTEB v2 multilingual benchmarks compared to other models ([Table 4](https://huggingface.co/blog/mmbert#table4)).\n\n![CoIR performance](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/coir.jpeg?raw=true)\\\nTable 5: Performance on CoIR code benchmark\n\n**Code Retrieval**: Due to the modern tokenizer (based on Gemma 2) mmBERT also shows strong coding performance ([Table 5](https://huggingface.co/blog/mmbert#table5)), making mmBERT suitable for any type of textual data. The only model that outperforms it is EuroBERT, which was able to use the non-publicly accessible Stack v2 dataset.\n\n## Learning Languages in the Decay Phase\n\nOne of mmBERT's most significant novel features is demonstrating that low-resource languages can be effectively learned during the short decay phase of training. We validated this approach by testing on languages only introduced during the final 100B token decay phase.\n\n![Improvements from adding new languages in the decay phase](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/low_resource_merge.jpg?raw=true)\\\n Figure 2: adding more than 1700 languages in the decay phase allows for rapid learning, which we keep through model merging.\n\n**Dramatic Performance Gains**: Testing on TiQuaD (Tigrinya) and FoQA (Faroese), we observed substantial improvements when these languages were included in the decay phase, as shown in [Figure 2](https://huggingface.co/blog/mmbert#figure2). The results demonstrate the effectiveness of our progressive language learning approach.\n\n**Competitive with Large Models**: Despite only seeing these languages in the final training phase, mmBERT achieves performance levels that exceed much larger models. On Faroese question answering where LLMs have been benchmarked, mmBERT outperforms Google Gemini 2.5 Pro and OpenAI o3.\n\n**Rapid Learning Mechanism**: The success of decay-phase language learning stems from the model's ability to leverage its strong multilingual foundation built during earlier phases. When exposed to new languages, the model can quickly adapt existing cross-lingual representations rather than learning from scratch.\n\n**Model Merging Benefits**: The final mmBERT models successfully retain most of the decay-phase improvements while benefiting from the English-focused and high-resource variants through TIES merging.\n\n## Efficiency Improvements\n\nmmBERT delivers substantial efficiency gains over previous multilingual encoder models through architectural improvements inherited from ModernBERT:\n\n![mmBERT is much more efficient than previous multilingual models](https://github.com/JHU-CLSP/mmBERT/blob/main/assets/inference_times.jpg?raw=true)\\\n Figure 3: mmBERT is significantly more efficient than previous multilingual models, up to 2-4x as much!\n\n**Throughput Performance**: mmBERT processes text significantly faster than existing multilingual models across various sequence lengths, as demonstrated in [Figure 3](https://huggingface.co/blog/mmbert#figure3). Both the small and base models show substantial speed improvements over previous multilingual encoders.\n\n**Modern Architecture Benefits**: The efficiency gains come from two main technical improvements:\n\n- **Flash Attention 2**: Optimized attention computation for better memory usage and speed\n- **Unpadding techniques**: Elimination of unnecessary padding tokens during processing\n\n**Sequence Length Scaling**: Unlike older models limited to 512 tokens, mmBERT handles up to 8,192 tokens efficiently while maintaining high throughput. This makes it suitable for longer document processing tasks that are increasingly common in multilingual applications.\n\n**Energy Efficiency**: The combination of better throughput and modern architecture results in lower computational costs for inference, making mmBERT more practical for production deployments where multilingual support is needed at scale.\n\nThese efficiency improvements make mmBERT not just more accurate than previous multilingual encoders, but also significantly more practical for real usage.\n\n## Usage Examples\n\nYou can use these models with just a few lines of code!\n\n```python\nfrom transformers import AutoTokenizer, AutoModelForMaskedLM\nimport torch\n\ntokenizer = AutoTokenizer.from_pretrained(\"jhu-clsp/mmBERT-base\")\nmodel = AutoModelForMaskedLM.from_pretrained(\"jhu-clsp/mmBERT-base\")\n\ndef predict_masked_token(text):\n inputs = tokenizer(text, return_tensors=\"pt\")\n with torch.no_grad():\n outputs = model(**inputs)\n mask_indices = torch.where(inputs[\"input_ids\"] == tokenizer.mask_token_id)\n predictions = outputs.logits[mask_indices]\n top_tokens, top_indices = torch.topk(predictions, 5, dim=-1)\n return [tokenizer.decode(token) for token in top_indices[0]]\n\n# Works across languages\ntexts = [\n \"The capital of France is .\",\n \"La capital de España es .\",\n \"Die Hauptstadt von Deutschland ist .\",\n]\n\nfor text in texts:\n predictions = predict_masked_token(text)\n print(f\"Text: {text}\")\n print(f\"Predictions: {predictions}\\n\")\n```\n\n## Fine-tuning Examples\n\n### Encoders\n\nClick to see how to finetune this into a dense embedding model using Sentence Transformers\n\n```python\nimport argparse\n\nfrom datasets import load_dataset\nfrom sentence_transformers import (\n SentenceTransformer,\n SentenceTransformerTrainer,\n SentenceTransformerTrainingArguments,\n)\nfrom sentence_transformers.evaluation import TripletEvaluator\nfrom sentence_transformers.losses import CachedMultipleNegativesRankingLoss\nfrom sentence_transformers.training_args import BatchSamplers\n\ndef main():\n # parse the lr & model name\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--lr\", type=float, default=8e-5)\n parser.add_argument(\"--model_name\", type=str, default=\"jhu-clsp/mmBERT-small\")\n args = parser.parse_args()\n lr = args.lr\n model_name = args.model_name\n model_shortname = model_name.split(\"/\")[-1]\n\n # 1. Load a model to finetune\n model = SentenceTransformer(model_name)\n\n # 2. Load a dataset to finetune on\n dataset = load_dataset(\n \"sentence-transformers/msmarco-co-condenser-margin-mse-sym-mnrl-mean-v1\",\n \"triplet-hard\",\n split=\"train\",\n )\n dataset_dict = dataset.train_test_split(test_size=1_000, seed=12)\n train_dataset = dataset_dict[\"train\"].select(range(1_250_000))\n eval_dataset = dataset_dict[\"test\"]\n\n # 3. Define a loss function\n loss = CachedMultipleNegativesRankingLoss(model, mini_batch_size=16) # Increase mini_batch_size if you have enough VRAM\n\n run_name = f\"{model_shortname}-DPR-{lr}\"\n # 4. (Optional) Specify training arguments\n args = SentenceTransformerTrainingArguments(\n # Required parameter:\n output_dir=f\"output/{model_shortname}/{run_name}\",\n # Optional training parameters:\n num_train_epochs=1,\n per_device_train_batch_size=512,\n per_device_eval_batch_size=512,\n warmup_ratio=0.05,\n fp16=False, # Set to False if GPU can't handle FP16\n bf16=True, # Set to True if GPU supports BF16\n batch_sampler=BatchSamplers.NO_DUPLICATES, # (Cached)MultipleNegativesRankingLoss benefits from no duplicates\n learning_rate=lr,\n # Optional tracking/debugging parameters:\n save_strategy=\"steps\",\n save_steps=500,\n save_total_limit=2,\n logging_steps=500,\n run_name=run_name, # Used in `wandb`, `tensorboard`, `neptune`, etc. if installed\n )\n\n # 5. (Optional) Create an evaluator & evaluate the base model\n dev_evaluator = TripletEvaluator(\n anchors=eval_dataset[\"query\"],\n positives=eval_dataset[\"positive\"],\n negatives=eval_dataset[\"negative\"],\n name=\"msmarco-co-condenser-dev\",\n )\n dev_evaluator(model)\n\n # 6. Create a trainer & train\n trainer = SentenceTransformerTrainer(\n model=model,\n args=args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n loss=loss,\n evaluator=dev_evaluator,\n )\n trainer.train()\n\n # 7. (Optional) Evaluate the trained model on the evaluator after training\n dev_evaluator(model)\n\n # 8. Save the model\n model.save_pretrained(f\"output/{model_shortname}/{run_name}/final\")\n\n # 9. (Optional) Push it to the Hugging Face Hub\n model.push_to_hub(run_name, private=False)\n\nif __name__ == \"__main__\":\n main()\n```\n\nClick to see how to finetune this into a multi-vector embedding model with PyLate\n\n```python\nfrom datasets import load_dataset\nfrom pylate import losses, models, utils\nfrom sentence_transformers import (\n SentenceTransformerTrainer,\n SentenceTransformerTrainingArguments,\n)\n\ndef main():\n # Load the datasets required for knowledge distillation (train, queries, documents)\n train = load_dataset(\n path=\"lightonai/ms-marco-en-bge\",\n name=\"train\",\n )\n\n queries = load_dataset(\n path=\"lightonai/ms-marco-en-bge\",\n name=\"queries\",\n )\n\n documents = load_dataset(\n path=\"lightonai/ms-marco-en-bge\",\n name=\"documents\",\n )\n\n # Set the transformation to load the documents/queries texts using the corresponding ids on the fly\n train.set_transform(\n utils.KDProcessing(queries=queries, documents=documents).transform,\n )\n\n # Define the base model, training parameters, and output directory\n num_train_epochs = 1\n lr = 8e-5\n batch_size = 16\n accum_steps = 1\n model_name = \"jhu-clsp/mmBERT-small\"\n model_shortname = model_name.split(\"/\")[-1]\n\n # Set the run name for logging and output directory\n run_name = f\"{model_shortname}-colbert-KD-{lr}\"\n output_dir = f\"output/{model_shortname}/{run_name}\"\n\n # Initialize the ColBERT model from the base model\n model = models.ColBERT(model_name_or_path=model_name)\n\n # Configure the training arguments (e.g., epochs, batch size, learning rate)\n args = SentenceTransformerTrainingArguments(\n output_dir=output_dir,\n num_train_epochs=num_train_epochs,\n per_device_train_batch_size=batch_size,\n fp16=False, # Set to False if you get an error that your GPU can't run on FP16\n bf16=True, # Set to True if you have a GPU that supports BF16\n run_name=run_name,\n logging_steps=10,\n learning_rate=lr,\n gradient_accumulation_steps=accum_steps,\n warmup_ratio=0.05,\n )\n\n # Use the Distillation loss function for training\n train_loss = losses.Distillation(model=model)\n\n # Initialize the trainer\n trainer = SentenceTransformerTrainer(\n model=model,\n args=args,\n train_dataset=train,\n loss=train_loss,\n data_collator=utils.ColBERTCollator(tokenize_fn=model.tokenize),\n )\n\n # Start the training process\n trainer.train()\n\n model.save_pretrained(f\"{output_dir}/final\")\n\nif __name__ == \"__main__\":\n main()\n```\n\nClick to see how to finetune this into a sparse retrieval model using Sentence Transformers\n\n```python\nimport logging\n\nfrom datasets import load_dataset\n\nfrom sentence_transformers import (\n SparseEncoder,\n SparseEncoderModelCardData,\n SparseEncoderTrainer,\n SparseEncoderTrainingArguments,\n)\nfrom sentence_transformers.sparse_encoder.evaluation import SparseNanoBEIREvaluator\nfrom sentence_transformers.sparse_encoder.losses import SparseMultipleNegativesRankingLoss, SpladeLoss\nfrom sentence_transformers.training_args import BatchSamplers\n\nlogging.basicConfig(format=\"%(asctime)s - %(message)s\", datefmt=\"%Y-%m-%d %H:%M:%S\", level=logging.INFO)\n\n# 1. Load a model to finetune with 2. (Optional) model card data\nmodel = SparseEncoder(\n \"jhu-clsp/mmBERT-small\",\n model_card_data=SparseEncoderModelCardData(\n language=\"en\",\n license=\"apache-2.0\",\n )\n)\n\n# 3. Load a dataset to finetune on\nfull_dataset = load_dataset(\"sentence-transformers/natural-questions\", split=\"train\").select(range(100_000))\ndataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)\ntrain_dataset = dataset_dict[\"train\"]\neval_dataset = dataset_dict[\"test\"]\n\n# 4. Define a loss function\nloss = SpladeLoss(\n model=model,\n loss=SparseMultipleNegativesRankingLoss(model=model),\n query_regularizer_weight=5e-5,\n document_regularizer_weight=3e-5,\n)\n\n# 5. (Optional) Specify training arguments\nrun_name = \"splade-distilbert-base-uncased-nq\"\nargs = SparseEncoderTrainingArguments(\n # Required parameter:\n output_dir=f\"models/{run_name}\",\n # Optional training parameters:\n num_train_epochs=1,\n per_device_train_batch_size=16,\n per_device_eval_batch_size=16,\n learning_rate=2e-5,\n warmup_ratio=0.1,\n fp16=True, # Set to False if you get an error that your GPU can't run on FP16\n bf16=False, # Set to True if you have a GPU that supports BF16\n batch_sampler=BatchSamplers.NO_DUPLICATES, # MultipleNegativesRankingLoss benefits from no duplicate samples in a batch\n # Optional tracking/debugging parameters:\n eval_strategy=\"steps\",\n eval_steps=1000,\n save_strategy=\"steps\",\n save_steps=1000,\n save_total_limit=2,\n logging_steps=200,\n run_name=run_name, # Will be used in W&B if `wandb` is installed\n)\n\n# 6. (Optional) Create an evaluator & evaluate the base model\ndev_evaluator = SparseNanoBEIREvaluator(dataset_names=[\"msmarco\", \"nfcorpus\", \"nq\"], batch_size=16)\n\n# 7. Create a trainer & train\ntrainer = SparseEncoderTrainer(\n model=model,\n args=args,\n train_dataset=train_dataset,\n eval_dataset=eval_dataset,\n loss=loss,\n evaluator=dev_evaluator,\n)\ntrainer.train()\n\n# 8. Evaluate the model performance again after training\ndev_evaluator(model)\n\n# 9. Save the trained model\nmodel.save_pretrained(f\"models/{run_name}/final\")\n\n# 10. (Optional) Push it to the Hugging Face Hub\nmodel.push_to_hub(run_name)\n```\n\nClick to see how to finetune this into a reranker model using Sentence Transformers\n\n```python\nimport logging\nimport traceback\n\nimport torch\nfrom datasets import load_dataset\n\nfrom sentence_transformers import SentenceTransformer\nfrom sentence_transformers.cross_encoder import (\n CrossEncoder,\n CrossEncoderModelCardData,\n CrossEncoderTrainer,\n CrossEncoderTrainingArguments,\n)\nfrom sentence_transformers.cross_encoder.evaluation import (\n CrossEncoderNanoBEIREvaluator,\n CrossEncoderRerankingEvaluator,\n)\nfrom sentence_transformers.cross_encoder.losses import BinaryCrossEntropyLoss\nfrom sentence_transformers.evaluation import SequentialEvaluator\nfrom sentence_transformers.util import mine_hard_negatives\n\n# Set the log level to INFO to get more information\nlogging.basicConfig(format=\"%(asctime)s - %(message)s\", datefmt=\"%Y-%m-%d %H:%M:%S\", level=logging.INFO)\n\n\ndef main():\n model_name = \"jhu-clsp/mmBERT-small\"\n\n train_batch_size = 64\n num_epochs = 1\n num_hard_negatives = 5 # How many hard negatives should be mined for each question-answer pair\n\n # 1a. Load a model to finetune with 1b. (Optional) model card data\n model = CrossEncoder(\n model_name,\n model_card_data=CrossEncoderModelCardData(\n language=\"en\",\n license=\"apache-2.0\",\n ),\n )\n print(\"Model max length:\", model.max_length)\n print(\"Model num labels:\", model.num_labels)\n\n # 2a. Load the GooAQ dataset: https://huggingface.co/datasets/sentence-transformers/gooaq\n logging.info(\"Read the gooaq training dataset\")\n full_dataset = load_dataset(\"sentence-transformers/gooaq\", split=\"train\").select(range(100_000))\n dataset_dict = full_dataset.train_test_split(test_size=1_000, seed=12)\n train_dataset = dataset_dict[\"train\"]\n eval_dataset = dataset_dict[\"test\"]\n logging.info(train_dataset)\n logging.info(eval_dataset)\n\n # 2b. Modify our training dataset to include hard negatives using a very efficient embedding model\n embedding_model = SentenceTransformer(\"sentence-transformers/static-retrieval-mrl-en-v1\", device=\"cpu\")\n hard_train_dataset = mine_hard_negatives(\n train_dataset,\n embedding_model,\n num_negatives=num_hard_negatives, # How many negatives per question-answer pair\n margin=0, # Similarity between query and negative samples should be x lower than query-positive similarity\n range_min=0, # Skip the x most similar samples\n range_max=100, # Consider only the x most similar samples\n sampling_strategy=\"top\", # Sample the top negatives from the range\n batch_size=4096, # Use a batch size of 4096 for the embedding model\n output_format=\"labeled-pair\", # The output format is (query, passage, label), as required by BinaryCrossEntropyLoss\n use_faiss=True,\n )\n logging.info(hard_train_dataset)\n\n # 2c. (Optionally) Save the hard training dataset to disk\n # hard_train_dataset.save_to_disk(\"gooaq-hard-train\")\n # Load again with:\n # hard_train_dataset = load_from_disk(\"gooaq-hard-train\")\n\n # 3. Define our training loss.\n # pos_weight is recommended to be set as the ratio between positives to negatives, a.k.a. `num_hard_negatives`\n loss = BinaryCrossEntropyLoss(model=model, pos_weight=torch.tensor(num_hard_negatives))\n\n # 4a. Define evaluators. We use the CrossEncoderNanoBEIREvaluator, which is a light-weight evaluator for English reranking\n nano_beir_evaluator = CrossEncoderNanoBEIREvaluator(\n dataset_names=[\"msmarco\", \"nfcorpus\", \"nq\"],\n batch_size=train_batch_size,\n )\n\n # 4b. Define a reranking evaluator by mining hard negatives given query-answer pairs\n # We include the positive answer in the list of negatives, so the evaluator can use the performance of the\n # embedding model as a baseline.\n hard_eval_dataset = mine_hard_negatives(\n eval_dataset,\n embedding_model,\n corpus=full_dataset[\"answer\"], # Use the full dataset as the corpus\n num_negatives=30, # How many documents to rerank\n batch_size=4096,\n include_positives=True,\n output_format=\"n-tuple\",\n use_faiss=True,\n )\n logging.info(hard_eval_dataset)\n reranking_evaluator = CrossEncoderRerankingEvaluator(\n samples=[\n {\n \"query\": sample[\"question\"],\n \"positive\": [sample[\"answer\"]],\n \"documents\": [sample[column_name] for column_name in hard_eval_dataset.column_names[2:]],\n }\n for sample in hard_eval_dataset\n ],\n batch_size=train_batch_size,\n name=\"gooaq-dev\",\n # Realistic setting: only rerank the positives that the retriever found\n # Set to True to rerank *all* positives\n always_rerank_positives=False,\n )\n\n # 4c. Combine the evaluators & run the base model on them\n evaluator = SequentialEvaluator([reranking_evaluator, nano_beir_evaluator])\n evaluator(model)\n\n # 5. Define the training arguments\n short_model_name = model_name if \"/\" not in model_name else model_name.split(\"/\")[-1]\n run_name = f\"reranker-{short_model_name}-gooaq-bce\"\n args = CrossEncoderTrainingArguments(\n # Required parameter:\n output_dir=f\"models/{run_name}\",\n # Optional training parameters:\n num_train_epochs=num_epochs,\n per_device_train_batch_size=train_batch_size,\n per_device_eval_batch_size=train_batch_size,\n learning_rate=2e-5,\n warmup_ratio=0.1,\n fp16=False, # Set to False if you get an error that your GPU can't run on FP16\n bf16=True, # Set to True if you have a GPU that supports BF16\n dataloader_num_workers=4,\n load_best_model_at_end=True,\n metric_for_best_model=\"eval_gooaq-dev_ndcg@10\",\n # Optional tracking/debugging parameters:\n eval_strategy=\"steps\",\n eval_steps=1000,\n save_strategy=\"steps\",\n save_steps=1000,\n save_total_limit=2,\n logging_steps=200,\n logging_first_step=True,\n run_name=run_name, # Will be used in W&B if `wandb` is installed\n seed=12,\n )\n\n # 6. Create the trainer & start training\n trainer = CrossEncoderTrainer(\n model=model,\n args=args,\n train_dataset=hard_train_dataset,\n loss=loss,\n evaluator=evaluator,\n )\n trainer.train()\n\n # 7. Evaluate the final model, useful to include these in the model card\n evaluator(model)\n\n # 8. Save the final model\n final_output_dir = f\"models/{run_name}/final\"\n model.save_pretrained(final_output_dir)\n\n # 9. (Optional) save the model to the Hugging Face Hub!\n # It is recommended to run `huggingface-cli login` to log into your Hugging Face account first\n try:\n model.push_to_hub(run_name)\n except Exception:\n logging.error(\n f\"Error uploading model to the Hugging Face Hub:\\n{traceback.format_exc()}To upload it manually, you can run \"\n f\"`huggingface-cli login`, followed by loading the model using `model = CrossEncoder({final_output_dir!r})` \"\n f\"and saving it using `model.push_to_hub('{run_name}')`.\"\n )\n\n\nif __name__ == \"__main__\":\n main()\n```\n\n## Model Family and Links\n\n**Standard Models:**\n\n- [mmBERT-small](https://huggingface.co/jhu-clsp/mmBERT-small) (140M params total, 42M non-embed)\n- [mmBERT-base](https://huggingface.co/jhu-clsp/mmBERT-base) (307M params total, 110M non-embed)\n\n**Research Resources:**\n\n- [🤗 mmBERT Model Collection](https://huggingface.co/collections/jhu-clsp/mmbert-a-modern-multilingual-encoder-68b725831d7c6e3acc435ed4)\n- [📝 Paper](https://arxiv.org/abs/2509.06888)\n- [🗂️ Training Data](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p1-fineweb2-langs) (3T+ tokens, fully open)\n- [💻 GitHub Repository](https://github.com/jhu-clsp/mmBERT)\n- [📊 Training Checkpoints](https://huggingface.co/jhu-clsp/mmBERT-checkpoints) for studying training or continued pre-training\n", "dateCreated": "2026-10-04T12:50:30.804479567+00:00", "date_created": "2025-09-09T00:00:00+00:00", "date_modified": "2025-09-09T00:00:00+00:00", "date_published": "2025-09-09T00:00:00+00:00", "description": "TL;DR This blog post introduces mmBERT, a state-of-the-art massively multilingual encoder model trained on 3T+ tokens of text in over 1800 languages. It shows significant performance and speed improvements over previous multilingual…", "external_url": "https://huggingface.co/blog/mmbert", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2025/09/2025-09-09-mmbert-modernbert-goes-multilingual.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2025/09/2025-09-09-mmbert-modernbert-goes-multilingual.md" }, "id": "urn:aggr:item:71fd0879335d9d0ec4a34da0d675777bc824e1b2", "isBasedOn": { "@id": "https://huggingface.co/blog/mmbert" }, "mainEntity": { "@id": "https://huggingface.co/blog/mmbert", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-09-09-mmbert-modernbert-goes-multilingual/#webpage" }, "dateModified": "2025-09-09T00:00:00+00:00", "datePublished": "2025-09-09T00:00:00+00:00", "headline": "mmBERT: ModernBERT goes Multilingual", "timeRequired": "PT15M", "url": "https://huggingface.co/blog/mmbert", "wordCount": 3180 }, "name": "Archived snapshot: mmBERT: ModernBERT goes Multilingual", "preview": null, "reading_minutes": 15, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00", "title": "mmBERT: ModernBERT goes Multilingual", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2025-09-09-mmbert-modernbert-goes-multilingual/", "word_count": 3180 }