![\"NDCG@10](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/train-multi-vector-encoder/mve_medical_model_size_ndcg.png\")
Sentence Transformers is a Python library for using and training embedding and reranker models for a wide range of applications, such as retrieval augmented generation, semantic search, semantic textual similarity, and more. Its v6.0 update introduces a fourth model type: MultiVectorEncoder, for ColBERT-style late interaction retrieval, alongside a complete training approach for it. In this blogpost, I'll show you how to use it to finetune a multi-vector model that outperforms general-purpose retrievers on your data. This method can also train strong new multi-vector models from scratch. Everything below runs on pip install -U "sentence-transformers[train]".
Finetuning multi-vector models involves several components: the model itself, datasets, loss functions, training arguments, evaluators, and the trainer class. I'll have a look at each of these components, accompanied by practical examples of how they can be used for finetuning strong multi-vector models.
\nLastly, in the Evaluation section, I'll show you that my finetuned multi-vector-encoder/mLateOn-medical model, trained in 14.5 hours on a single RTX 3090 alongside this blogpost, easily outperforms every general-purpose retrieval model I could find on my medical retrieval evaluation: dense, sparse, lexical, and multi-vector alike.
\n
If you're interested in finetuning dense embedding models, sparse embedding models, or rerankers instead, then consider reading through my prior Training and Finetuning Embedding Models, Training and Finetuning Sparse Embedding Models, and Training and Finetuning Reranker Models blogposts.
\n\n\nThis blogpost is about training multi-vector models. If you want to learn how to use them, from loading and encoding to indexing in vector databases, see the companion Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers blogpost.
\n
A dense embedding model compresses a whole text into a single vector, and similarity is one dot product between two such summaries. A multi-vector model (also called a late-interaction or ColBERT-style model) skips that compression. It keeps one small vector per token and scores a query against a document with the MaxSim operator, where every query token finds its best-matching document token and the scores are summed. Token-level matching preserves exactly the fine-grained signals that a single vector has to average away, which usually means stronger retrieval, at the cost of a bigger index.
\nThe companion Multi-Vector Embedding Models blogpost covers the architecture, encoding, scoring, and indexing in detail, so I'll keep this section short and get to the training.
\n![\"Dense](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/multi-vector-encoder/maxsim_explainer.gif\")
Finetuning multi-vector models significantly improves their retrieval performance on your specific domain: the vocabulary, the query style, and the notion of relevance all differ between web search, legal discovery, code search, and scientific literature review. Because queries and documents are matched token by token, multi-vector models pick up fine-grained domain signals that single-vector models tend to average away, and they respond very well to even modest amounts of in-domain finetuning data.
\nBeyond that, most released retrieval models were configured for short passages. The classic ColBERT checkpoints truncate documents at 180 or 300 tokens, and many popular dense models at 256 or 512, because their MS MARCO-style training data rarely goes beyond that. If your documents are long, these models silently discard most of every document before scoring it. On my medical evaluation with passages averaging 941 tokens, I measured that this truncation costs up to 0.24 NDCG@10, considerably more than any difference between model architectures. When you train your own model, you configure the document length that your data needs.
\nLightOn ran into this same dynamic with code retrieval, where general LateOn wasn't enough and they trained LateOn-Code. Your domain, whether that's medical, legal, financial, or your company's internal documents, is not getting an official model. This blogpost shows you how to build it yourself, in a matter of hours, on a single consumer GPU.
\nTraining MultiVectorEncoder models involves the following components:
\nLet's take a closer look at each component.
\nMulti-vector training gives you a real choice of starting point, and it matters more than you might expect.
\nIf you want to further finetune an existing multi-vector model, you don't have to worry about the architecture at all:
\n\n\n# Loading in fp32 is preferred for training if your memory can handle it\n = \nThe checkpoint brings its own recipe along: its query and document marker tokens, its projection head, its scoring skiplist. For finetuning, you generally want to keep all of that and change only what your data demands. The first thing to check is the length configuration, since many released checkpoints cap documents at 180 to 512 tokens (see Why Finetune?), and my medical passages run to 1,400 tokens. The mLateOn family already serves the backbone's full 8192 token context, but if your starting checkpoint carries caps, lift them:
\n# Let the model read full documents instead of the caps it was trained with,\n# e.g. GTE-ModernColBERT-v1 ships with query_length=48 and document_length=300\n. = None\n. = None\nWith the per-task caps unset, truncation falls back to the tokenizer's model_max_length, which is why I configure that limit at load time above.
I made one more change, adding a punctuation skiplist that excludes punctuation tokens from document-side scoring and storage. In a 4-way ablation (none, punctuation, stopwords, both) it modestly won on quality, and it shrinks the document index by 9.6% on this data for free:
\n\n\n# model[2] is the MultiVectorMask module\n. = \n # token ids are cached, so re-resolve after changing\nYou can also point MultiVectorEncoder at any base transformer, and a fresh, randomly initialized token-level projection is appended for you:
\n\n = \n# MultiVectorEncoder(\n# (0): Transformer({..., 'architecture': 'ModernBertModel'})\n# (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, ...})\n# (2): MultiVectorMask({'skiplist_words': [], 'skiplist_tasks': ['document'], ...})\n# (3): Normalize({...})\n# )\nThat's the classic ColBERT pipeline: a Transformer producing contextualized token embeddings, a token-level Dense projecting each of them down to 128 dimensions, a MultiVectorMask deciding which tokens count during scoring, and a token-level Normalize. The projection starts random, so training is required before this model is useful. Interestingly, this works with strong dense embedding backbones too. A fresh projection on Alibaba-NLP/gte-modernbert-base reached within 0.03 of the existing-checkpoint starting points in my experiments, from nothing but the projection and 25k training pairs.
The classic ColBERT tokenization tricks ([MASK] query expansion, [Q] / [D] prefix tokens, a document length cap, a punctuation skiplist) are all off by default and configurable. See Creating Custom Models for the full set. For what it's worth, I tested [MASK] query expansion in four configurations for my domain finetune and none of them made a measurable difference, so don't feel obliged to reach for the classic recipe.
I measured this directly while preparing this blogpost, taking six starting points and training each with the identical recipe on 25k medical question-passage pairs from MIRIAD, then evaluating on 1,000 held-out questions against a 50,000 passage corpus:
\n| Starting point | \nZero-shot NDCG@10 | \nAfter 25k pairs | \nDelta | \n
|---|---|---|---|
| lightonai/mLateOn-unsupervised | \n0.9087 | \n0.9398 | \n+0.0311 | \n
| lightonai/mLateOn | \n0.9277 | \n0.9319 | \n+0.0042 | \n
| lightonai/LateOn-unsupervised | \n0.9026 | \n0.9206 | \n+0.0180 | \n
| lightonai/LateOn | \n0.9185 | \n0.9105 | \n-0.0080 | \n
| lightonai/GTE-ModernColBERT-v1 | \n0.9198 | \n0.9007 | \n-0.0191 | \n
| Fresh head on gte-modernbert-base | \n- | \n0.9177 | \n- | \n
The result surprised me, and it replicated across two model families. *The -unsupervised checkpoints adapt to a new domain far better than their finished siblings, overtaking them despite starting lower. These checkpoints sit after large-scale contrastive pretraining but before supervised finetuning on general retrieval, so they carry all the late-interaction structure with none of the general-purpose tuning that domain training then has to undo. The finished checkpoints, by contrast, barely moved or even regressed, at every learning rate I tried.
So, if the model family you like publishes a pre-supervised checkpoint, start there. If not, a fresh projection on a strong retrieval-pretrained backbone is a close runner-up. Continuing from a fully finished checkpoint is the weakest option for domain adaptation, despite being the most natural-feeling one.
\nThe MultiVectorEncoderTrainer uses datasets.Dataset or datasets.DatasetDict instances for training and evaluation. You can load data from the Hugging Face Datasets Hub or use local data in whatever format you prefer (e.g. CSV, JSON, Parquet, Arrow, or SQL).
Note: Lots of public datasets that work out of the box with Sentence Transformers have been tagged with sentence-transformers on the Hugging Face Hub, so you can easily find them on https://huggingface.co/datasets?other=sentence-transformers. Consider browsing through these to find ready-to-go datasets that might be useful for your tasks, domains, or languages.
You can use the load_dataset function to load data from datasets on the Hub:
\n\n = \n\n\n"""\nDataset({\n features: ['question', 'passage_text'],\n num_rows: 4467542\n})\n"""\nThis is the dataset I'll train on in this blogpost: 4.4 million medical questions from MIRIAD, each paired with the source passage that contains its answer (averaging 941 tokens). Simple (query, relevant passage) pairs like these are the easiest retrieval training data to collect for your own domain, and as you'll see, they're all you need.
\nYou can also use load_dataset for loading local data in common file formats:
\n\n = \n# or\n = \nAnd if your local data requires pre-processing, you can use datasets.Dataset.from_dict to initialize your dataset with a dictionary of lists:
\n\n = \n = \n# Open a file, perform preprocessing, filtering, cleaning, etc.\n# and append to the lists\n\n = \nIt is important that your dataset format matches your loss function (or that you choose a loss function that matches your dataset format). Verifying whether a dataset format works with a loss function involves two steps:
\nThere are two multi-vector specific conventions on top of this:
\nrouter_mapping training argument.(query, document_1, ..., document_N, scores) where scores is a list of N teacher scores per row. For KD datasets that store query and document IDs alongside separate text datasets (e.g. lightonai/ms-marco-en-bge), you can use resolve_ids to resolve the IDs to texts on the fly.Loss functions quantify how well a model performs for a given batch of data, allowing an optimizer to update the model weights to produce more favourable (i.e., lower) loss values. The right loss function for your task depends on the data you have and what you're trying to achieve. You can find a full list of options in the Loss Overview.
\nFor the common case of question-answer or question-passage pairs, the workhorse is in-batch negatives training with MultiVectorMultipleNegativesRankingLoss, where every other document in the batch acts as a negative for each query. Bigger batches mean more negatives and stronger training, so in practice you'll want its GradCache variant, CachedMultiVectorMultipleNegativesRankingLoss, which decouples the effective batch size from what fits on your GPU:
\n\n\n = \n\n = \nThe mini_batch_size parameter bounds the memory by encoding documents in chunks of this size, while the effective contrastive batch size (128 in my run below, and in my ablations bigger batches bought nothing further) stays a free choice. GradCache guarantees identical results regardless of the chunk size, so lower it for smaller GPUs at only a wall-clock cost. When your document lengths vary a lot, consider its sibling mini_batch_num_tokens, which packs each chunk to a total token budget instead of a document count, so a chunk of unusually long documents can never spike your memory (my mini_batch_size=16 at roughly 940 tokens per document corresponds to mini_batch_num_tokens=15_000).
One multi-vector specific trap is that the contrastive losses default to scale=1.0, unlike the dense embedding equivalent which defaults to scale=20.0. That 20.0 exists because a cosine similarity is a single value in [-1, 1], too narrow a range for a sharp softmax. A MaxSim score instead sums one best-match similarity per query token, so it already spans roughly [0, query_length]: a 32-token query can score up to 32. So don't copy scale=20.0 over from a dense training script, since it would saturate the softmax and kill your gradients.
For distillation from a stronger teacher, which is how the strongest general-purpose late-interaction models are trained, see MultiVectorDistillKLDivLoss and the Knowledge Distillation tab in the Training Overview documentation.
You can customize the training process using the MultiVectorEncoderTrainingArguments class. This class lets you adjust parameters that can impact training speed and help you understand what's happening during training.
For more information on the most useful training arguments, check out the Multi-Vector Encoder > Training Overview > Training Arguments. It's worth reading to get the most out of your training.
\nHere's an example, using the values from my actual training run:
\n\n\n\n = \nA few of these deserve a comment:
\nprompts: training does not automatically apply the prompts stored in the model, so map them onto your training columns explicitly. Here that is the checkpoint's [Q] marker for the question column and [D] for the passage column, keeping training consistent with inference.max_length (deliberately not set): this argument caps tokenization during training only, for when you want cheaper training than the model's full serving length. I measured what that shortcut costs on this data. Training at 512 tokens lost about 0.015 NDCG@10 for about 2x the speed, and the deficit did not shrink with more data, because the model simply never sees what got cut off. Leave it unset so training matches inference, unless you need the speedup more than the quality.learning_rate=1e-4: after a sweep from 5e-6 to 2e-4, I had the best luck with this higher-than-usual learning rate.To track your model's performance during training, you can pass an eval_dataset to the trainer for evaluation loss, but concrete retrieval metrics are much more informative. Sentence Transformers includes the following built-in evaluators for multi-vector models:
| Evaluator | \nRequired Data | \n
|---|---|
MultiVectorInformationRetrievalEvaluator | \nQueries, corpus, and relevant document mappings | \n
MultiVectorNanoBEIREvaluator | \nNo data required | \n
MultiVectorTripletEvaluator | \n(anchor, positive, negative) triplets | \n
MultiVectorRerankingEvaluator | \nList of {'query': '...', 'positive': [...], 'negative': [...]} dictionaries | \n
MultiVectorDistillationEvaluator | \nQueries with candidate documents and teacher scores | \n
For domain finetuning, the MultiVectorInformationRetrievalEvaluator built from your own held-out data is the one that matters. One tip on constructing it is that the corpus should be hard enough that models can be told apart. In my case the MIRIAD questions are generated from their own source passages, which makes retrieval unusually easy. Against just the 10k gold passages, nearly every model scored above 0.97 NDCG@10. If your evaluation saturates like that, add distractor passages (I use deduplicated passages from the training split) until the scores spread out:
\n\n\n = \n\n# Gold: 1,000 evaluation questions, each mapping to its own passage, with the\n# eval split's full ~10k unique passages as the initial corpus\n = \n = \n = \n = \n\n \n = f\n = \n \n = \n = \n\n# Distractors: unique train passages that make the haystack realistic\n = \n\n \n break\n \n \n = \n\n = \n# results = evaluator(model)\nThe MultiVectorEncoderTrainer is where all previous components come together. Here is the complete script that trained multi-vector-encoder/mLateOn-medical, the model from the introduction:
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n # 1. Load the starting checkpoint: contrastively pretrained, not yet supervised\n # Loading in fp32 is preferred for training if your memory can handle it\n = \n\n # 2. Lift the per-task length caps so training and inference see full medical passages\n . = None\n . = None\n\n # 3. Skip punctuation tokens during scoring: a small quality win and a 9.6% smaller index\n . = \n \n\n # 4. Load 1 million medical question-passage pairs\n = \n\n # 5. In-batch negatives with GradCache: large effective batch, memory-bounded chunks\n = \n\n # 6. A light dev evaluator to watch progress during training: 500 held-out questions\n # against the eval split's ~10k unique passages. The full 200k protocol runs afterwards.\n = \n , , , = , , , \n \n \n = f\n = \n \n = \n = \n = \n\n # 7. Training arguments, as discussed above\n = \n = \n\n # 8. Create a trainer & train\n = \n \n\n # 9. Save the trained model\n \n\n # 10. (Optional) Push it to the Hugging Face Hub\n \n \n \n \n\n\n\n \nThat's the whole recipe: a pre-supervised checkpoint, a million domain pairs, in-batch negatives, full document length, and a higher-than-usual learning rate. The run took 14.5 hours on my single RTX 3090 at a peak of 17.5 GB VRAM, and every one of those choices was the winner of a measured comparison rather than a guess.
\nFor readers on smaller budgets, my scaling experiments put 100k pairs (75 minutes of training) within 0.012 NDCG@10 of the full million-pair run. Most of the gain comes in the first hour.
\nThe MultiVectorEncoder trainer supports various transformers.TrainerCallback subclasses, including:
WandbCallback for logging training metrics to W&B if wandb is installedTensorBoardCallback for logging training metrics to TensorBoard if tensorboard is accessibleCodeCarbonCallback for tracking carbon emissions during training if codecarbon is installedEnable these via the report_to training argument, e.g. report_to=["wandb", "codecarbon"], with the required dependencies installed. It defaults to "none", and report_to="all" activates every integration whose dependency is installed.
Refer to the Transformers Callbacks documentation for more information on these callbacks and how to create your own.
\nTypically, top-performing general-purpose models are trained on multiple datasets simultaneously. However, this approach can be challenging due to the varying formats of each dataset. Fortunately, the MultiVectorEncoderTrainer allows you to train on multiple datasets without requiring a uniform format. Additionally, it provides the flexibility to apply different loss functions to each dataset. Here are the steps to train with multiple datasets at once:
datasets.Dataset instances (or a datasets.DatasetDict) as the train_dataset (and optionally also eval_dataset).Each training/evaluation batch will only contain samples from one of the datasets. The order in which batches are sampled from the multiple datasets is defined by the MultiDatasetBatchSamplers enum, which can be passed to the MultiVectorEncoderTrainingArguments via multi_dataset_batch_sampler. Valid options are:
MultiDatasetBatchSamplers.ROUND_ROBIN: Round-robin sampling from each dataset until one is exhausted. With this strategy, it's likely that not all samples from each dataset are used, but each dataset is sampled from equally.MultiDatasetBatchSamplers.PROPORTIONAL (default): Sample from each dataset in proportion to its size. With this strategy, all samples from each dataset are used and larger datasets are sampled from more frequently.To find out where the finetuned model stands, I evaluated it against over 50 retrieval model configurations across four architecture families on the MIRIAD evaluation set, built exactly as in the Evaluator section above, with 1,000 held-out medical questions searching 200,000 unique passages (the 10k gold passages hidden among 190k deduplicated distractors from the training split). This corpus is four times the size of the 50,000-passage one from Which starting point should you pick?, so scores are not comparable between the two tables.
\n
The headline results, with the full table in the collapsible below:
\n| Model | \nFamily | \nNDCG@10 | \n
|---|---|---|
| multi-vector-encoder/mLateOn-medical (mine) | \nMulti-vector, finetuned | \n0.9139 | \n
| lightonai/mLateOn | \nMulti-vector, zero-shot | \n0.8520 | \n
| lightonai/GTE-ModernColBERT-v1 (cap lifted) | \nMulti-vector, zero-shot | \n0.8502 | \n
| Qwen/Qwen3-Embedding-4B | \nDense, zero-shot | \n0.7817 | \n
| voyageai/voyage-4-nano | \nDense, zero-shot | \n0.7563 | \n
| BM25 | \nLexical | \n0.7501 | \n
| naver/splade-v3 | \nSparse, zero-shot | \n0.6853 | \n
The finetuned model tops the table, beating the strongest zero-shot model of any architecture by +0.062 NDCG@10. In other words, the strongest zero-shot model returns the right passage as the very first hit for 75.8% of the queries, while the finetuned model does so for 84.9%, cutting the rank-1 error by more than a third.
\nThe architecture pattern is just as clear, with the top of the table exclusively late interaction. On long documents, one vector per token beats one vector per document, even at matched training and matched backbones. DenseOn and LateOn share training data and architecture except for the head, and the late-interaction sibling wins by +0.12, with the multilingual pair (mDenseOn and mLateOn) replicating this at +0.13. Scale doesn't rescue single vectors either. Qwen3-Embedding-4B, the strongest dense model with roughly 33x the active (non-embedding) parameters of mine, still stops 0.13 short, and the 8B version scores lower than the 4B.
\nBM25 also performs surprisingly well, beating every sparse model, every truncation-capped multi-vector model, and all but three dense models: the multi-billion Qwen3-Embedding-4B and 8B, and voyage-4-nano, which reads its full 32k token context to edge past by just 0.006. Don't expect that to transfer to your own data though. MIRIAD's questions are generated from the passages, so the lexical overlap between a query and its gold passage is far larger than in typical retrieval, and BM25's unlimited context length lets it use every one of those overlapping words while most neural checkpoints truncate. A BM25 baseline is cheap and always worth running, just don't count on this margin.
\nThe full field at a glance, sorted by score and colored by architecture family.
\n![\"Sorted](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/train-multi-vector-encoder/mve_medical_ndcg_by_model.png\")
Click to see the full evaluation table
\n| Model | \nFamily | \nNDCG@10 | \nacc@1 | \n
|---|---|---|---|
| multi-vector-encoder/mLateOn-medical (mine) | \nMulti-vector, finetuned | \n0.9139 | \n0.849 | \n
| lightonai/mLateOn | \nMulti-vector | \n0.8520 | \n0.758 | \n
| lightonai/GTE-ModernColBERT-v1 @1024 | \nMulti-vector | \n0.8502 | \n0.763 | \n
| lightonai/LateOn @1024 | \nMulti-vector | \n0.8485 | \n0.760 | \n
| lightonai/mLateOn-unsupervised | \nMulti-vector | \n0.8304 | \n0.733 | \n
| mixedbread-ai/mxbai-edge-colbert-v0-32m @1024 | \nMulti-vector | \n0.8186 | \n0.727 | \n
| Qwen/Qwen3-Embedding-4B | \nDense | \n0.7817 | \n0.669 | \n
| Qwen/Qwen3-Embedding-8B | \nDense | \n0.7747 | \n0.654 | \n
| perplexity-ai/pplx-embed-v1-late-0.6b @1024 | \nMulti-vector | \n0.7702 | \n0.632 | \n
| lightonai/ColBERT-Zero | \nMulti-vector | \n0.7613 | \n0.675 | \n
| LiquidAI/LFM2.5-ColBERT-350M | \nMulti-vector | \n0.7582 | \n0.664 | \n
| voyageai/voyage-4-nano | \nDense | \n0.7563 | \n0.638 | \n
| BM25 | \nLexical | \n0.7501 | \n0.641 | \n
| jinaai/jina-embeddings-v5-text-small-retrieval | \nDense | \n0.7470 | \n0.620 | \n
| Qwen/Qwen3-Embedding-0.6B | \nDense | \n0.7408 | \n0.620 | \n
| perplexity-ai/pplx-embed-v1-0.6b | \nDense | \n0.7384 | \n0.615 | \n
| mixedbread-ai/mxbai-edge-colbert-v0-32m | \nMulti-vector | \n0.7350 | \n0.639 | \n
| mixedbread-ai/mxbai-edge-colbert-v0-17m | \nMulti-vector | \n0.7271 | \n0.631 | \n
| answerdotai/answerai-colbert-small-v1 @512 | \nMulti-vector | \n0.7264 | \n0.615 | \n
| lightonai/DenseOn @1024 | \nDense | \n0.7239 | \n0.597 | \n
| lightonai/mDenseOn @1024 | \nDense | \n0.7227 | \n0.585 | \n
| jinaai/jina-embeddings-v5-text-nano-retrieval | \nDense | \n0.7206 | \n0.587 | \n
| microsoft/harrier-oss-v1-0.6b | \nDense | \n0.7126 | \n0.572 | \n
| Alibaba-NLP/gte-modernbert-base | \nDense | \n0.7102 | \n0.582 | \n
| Snowflake/snowflake-arctic-embed-l-v2.0 | \nDense | \n0.7068 | \n0.568 | \n
| perplexity-ai/pplx-embed-v1-late-0.6b | \nMulti-vector | \n0.7008 | \n0.570 | \n
| google/embeddinggemma-300m | \nDense | \n0.7000 | \n0.563 | \n
| lightonai/DenseOn | \nDense | \n0.6943 | \n0.570 | \n
| naver/splade-v3 | \nSparse | \n0.6853 | \n0.574 | \n
| ibm-granite/granite-embedding-small-english-r2 | \nDense | \n0.6813 | \n0.546 | \n
| naver/splade-v3-distilbert | \nSparse | \n0.6806 | \n0.567 | \n
| codefuse-ai/F2LLM-v2-0.6B | \nDense | \n0.6799 | \n0.536 | \n
| colbert-ir/colbertv2.0 @512 | \nMulti-vector | \n0.6785 | \n0.571 | \n
| prithivida/Splade_PP_en_v1 | \nSparse | \n0.6755 | \n0.577 | \n
| lightonai/LateOn | \nMulti-vector | \n0.6713 | \n0.561 | \n
| tomaarsen/embeddinggemma-300m-miriad-unsloth | \nDense, finetuned | \n0.6705 | \n0.530 | \n
| lightonai/LateOn-regularized | \nMulti-vector | \n0.6673 | \n0.554 | \n
| lightonai/LateOn-unsupervised | \nMulti-vector | \n0.6672 | \n0.553 | \n
| lightonai/GTE-ModernColBERT-v1 | \nMulti-vector | \n0.6612 | \n0.555 | \n
| opensearch-project/opensearch-neural-sparse-encoding-v2-distill | \nSparse | \n0.6518 | \n0.531 | \n
| nomic-ai/nomic-embed-text-v1.5 (prompted) | \nDense | \n0.6387 | \n0.498 | \n
| mixedbread-ai/mxbai-embed-large-v1 | \nDense | \n0.6355 | \n0.502 | \n
| BAAI/bge-large-en-v1.5 | \nDense | \n0.6308 | \n0.498 | \n
| jinaai/jina-colbert-v2 @1024 | \nMulti-vector | \n0.6218 | \n0.504 | \n
| nomic-ai/nomic-embed-text-v1.5 | \nDense | \n0.6203 | \n0.487 | \n
| answerdotai/answerai-colbert-small-v1 | \nMulti-vector | \n0.6184 | \n0.514 | \n
| tomaarsen/splade-modernbert-base-miriad | \nSparse, finetuned | \n0.6142 | \n0.473 | \n
| NeuML/biomedbert-base-colbert | \nMulti-vector | \n0.5963 | \n0.463 | \n
| BAAI/bge-base-en-v1.5 | \nDense | \n0.5930 | \n0.454 | \n
| BAAI/bge-small-en-v1.5 | \nDense | \n0.5881 | \n0.457 | \n
| sentence-transformers/all-mpnet-base-v2 | \nDense | \n0.5159 | \n0.396 | \n
| jinaai/jina-colbert-v2 | \nMulti-vector | \n0.4992 | \n0.401 | \n
| mixedbread-ai/mxbai-colbert-large-v1 | \nMulti-vector | \n0.4690 | \n0.358 | \n
| sentence-transformers/static-retrieval-mrl-en-v1 | \nDense | \n0.4614 | \n0.323 | \n
| sentence-transformers/all-MiniLM-L6-v2 | \nDense | \n0.4458 | \n0.321 | \n
| colbert-ir/colbertv2.0 | \nMulti-vector | \n0.4347 | \n0.346 | \n
Models marked @N are evaluated with their document length cap lifted to N tokens, since their native caps (180 to 512 tokens) would otherwise truncate the 941-token average passages. For every multi-vector model this lift was worth +0.08 to +0.24 NDCG@10 over the as-served row, and even the dense DenseOn gained +0.03 from the same treatment.
Note that this does not mean that multi-vector-encoder/mLateOn-medical is the strongest model on all domains. It's simply the strongest in my domain. This is totally fine, as I just need this model to work well on my data.
\nDon't underestimate the power of finetuning multi-vector models on your domain. Fourteen and a half hours on a single consumer GPU produced a model that no general-purpose retriever comes close to on this data, and the recipe is a single script with no teacher model and no mined negatives!
\nThe fair objection to multi-vector retrieval is index size, and this domain is close to the worst case for it. Storing one vector per token, my model needs about 878 vectors per passage, so the 200,000-passage corpus takes roughly 45 GB at fp16, where a dense model needs well under 1 GB. Document length is what makes that gap so wide. The Natural Questions passages in the companion post average about 125 token vectors each, seven times fewer, so a corpus of short passages starts from a far smaller index than this one does. The HierarchicalTokenPooling module compresses exactly this by clustering each document's token embeddings and storing the cluster means, keeping roughly 1 / pool_factor of the vectors:
\n\n = \n = \nI measured it post-hoc on the finished model, with no pooling-aware training, and on long documents it is remarkably cheap.
\n![\"Embedding](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/train-multi-vector-encoder/mve_medical_index_size_ndcg.png\")
The solid points are uncompressed embeddings, so that every family is counted the same way and scored with exact search. You would not deploy any of them like that, though. Dense indexes routinely use int8 or binary quantization with rescoring, sparse indexes compress their postings, and multi-vector indexes use PLAID-style residual compression. Don't read those points as the disk you need to buy, but as relative storage cost.
\nToken pooling is the solid line. Halving the vector count costs 0.0033 NDCG@10 and leaves rank-1 accuracy untouched, and keeping only a quarter of them, at 11.2 GB, still scores 0.8991. The curve keeps going (I measured out to a tenth of the vectors, still at 0.8765) but there is little reason to push pooling that far once quantization is on the table, which is what the dashed line below is about.
\nThe dashed line is what a real deployment might look like. I gave Omar Khattab early access to the model and the benchmark, and he measured these configurations with fast-plaid at 1-bit residual quantization, using compact 17-bit centroid ids and 18-bit document ids instead of its ordinary unpacked 64-bit integers, plus document-side pruning:
\n| configuration | \nvectors kept | \nindex | \nNDCG@10 | \n
|---|---|---|---|
| 1-bit PLAID, all vectors | \n100% | \n3.37 GB | \n0.8984 | \n
| 1-bit PLAID + pruning | \n65% | \n2.23 GB | \n0.8830 | \n
| 1-bit PLAID + pruning | \n42% | \n1.45 GB | \n0.8642 | \n
That first row is 13x smaller than the raw embeddings, for 0.0155 NDCG@10. That is a far better trade than anywhere on the pooling curve. Quantization shrinks each vector while pooling and pruning cut how many you keep, so they compose, and quantization is the one to reach for first. Push further and the last row lands at 1.45 GB, smaller than the fp16 embeddings of Qwen3-Embedding-8B (1.64 GB), while scoring 0.0895 higher. The objection that multi-vector indexes are too big does not survive a properly configured index.
\nThe pruning here is naive, meant only to establish that token reduction works on top of quantization, so read the bottom two rows as a floor rather than the frontier. If you would rather not hand-tune quantization at all, the Indexing section of the companion post covers fast-plaid, Qdrant, Weaviate, and Vespa.
\nMulti-vector retrieval is only as expensive as its index. The raw embeddings for this corpus are 45 GB, and a properly configured index is at least 7x smaller at nearly the same accuracy. The index deserves as much of your attention as the checkpoint.
\nThanks to Omar Khattab for measuring the quantized and pruned index configurations in Optimizing the index, and for the discussions around late-interaction index costs.
\nThese pages have training examples with explanations as well as links to training scripts. You can use them to get familiar with the multi-vector training loop:
\nFor further learning, you may also want to explore the following resources on Sentence Transformers:
\nAnd here is an advanced page that might interest you:
\n\nAnd the companion blogpost, covering everything about using these models:
\n\n", "content_markdown": "[Sentence Transformers](https://sbert.net/) is a Python library for using and training embedding and reranker models for a wide range of applications, such as retrieval augmented generation, semantic search, semantic textual similarity, and more. Its v6.0 update introduces a fourth model type: `MultiVectorEncoder`, for ColBERT-style late interaction retrieval, alongside a complete training approach for it. In this blogpost, I'll show you how to use it to finetune a multi-vector model that outperforms general-purpose retrievers on your data. This method can also train strong new multi-vector models from scratch. Everything below runs on `pip install -U \"sentence-transformers[train]\"`.\n\nFinetuning multi-vector models involves several components: the model itself, datasets, loss functions, training arguments, evaluators, and the trainer class. I'll have a look at each of these components, accompanied by practical examples of how they can be used for finetuning strong multi-vector models.\n\nLastly, in the [Evaluation](https://huggingface.co/blog/train-multi-vector-encoder#evaluation) section, I'll show you that my finetuned [multi-vector-encoder/mLateOn-medical](https://huggingface.co/multi-vector-encoder/mLateOn-medical) model, trained in 14.5 hours on a single RTX 3090 alongside this blogpost, easily outperforms every general-purpose retrieval model I could find on my medical retrieval evaluation: dense, sparse, lexical, and multi-vector alike.\n\n[](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/train-multi-vector-encoder/mve_medical_model_size_ndcg.png)\n\nIf you're interested in finetuning dense embedding models, sparse embedding models, or rerankers instead, then consider reading through my prior [Training and Finetuning Embedding Models](https://huggingface.co/blog/train-sentence-transformers), [Training and Finetuning Sparse Embedding Models](https://huggingface.co/blog/train-sparse-encoder), and [Training and Finetuning Reranker Models](https://huggingface.co/blog/train-reranker) blogposts.\n\n> This blogpost is about *training* multi-vector models. If you want to learn how to *use* them, from loading and encoding to indexing in vector databases, see the companion [Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers](https://huggingface.co/blog/multi-vector-encoder) blogpost.\n\n## Table of Contents\n\n- [What are Multi-Vector models?](https://huggingface.co/blog/train-multi-vector-encoder#what-are-multi-vector-models)\n- [Why Finetune?](https://huggingface.co/blog/train-multi-vector-encoder#why-finetune)\n- [Training Components](https://huggingface.co/blog/train-multi-vector-encoder#training-components)\n- [Model](https://huggingface.co/blog/train-multi-vector-encoder#model)\n - [Finetuning an existing multi-vector model](https://huggingface.co/blog/train-multi-vector-encoder#finetuning-an-existing-multi-vector-model)\n - [Building one from a base transformer](https://huggingface.co/blog/train-multi-vector-encoder#building-one-from-a-base-transformer)\n - [Which starting point should you pick?](https://huggingface.co/blog/train-multi-vector-encoder#which-starting-point-should-you-pick)\n- [Dataset](https://huggingface.co/blog/train-multi-vector-encoder#dataset)\n - [Data on the Hugging Face Hub](https://huggingface.co/blog/train-multi-vector-encoder#data-on-the-hugging-face-hub)\n - [Local Data](https://huggingface.co/blog/train-multi-vector-encoder#local-data)\n - [Dataset Format](https://huggingface.co/blog/train-multi-vector-encoder#dataset-format)\n- [Loss Function](https://huggingface.co/blog/train-multi-vector-encoder#loss-function)\n- [Training Arguments](https://huggingface.co/blog/train-multi-vector-encoder#training-arguments)\n- [Evaluator](https://huggingface.co/blog/train-multi-vector-encoder#evaluator)\n- [Trainer](https://huggingface.co/blog/train-multi-vector-encoder#trainer)\n - [Callbacks](https://huggingface.co/blog/train-multi-vector-encoder#callbacks)\n - [Multi-Dataset Training](https://huggingface.co/blog/train-multi-vector-encoder#multi-dataset-training)\n- [Evaluation](https://huggingface.co/blog/train-multi-vector-encoder#evaluation)\n - [Optimizing the index](https://huggingface.co/blog/train-multi-vector-encoder#optimizing-the-index)\n- [Acknowledgements](https://huggingface.co/blog/train-multi-vector-encoder#acknowledgements)\n- [Additional Resources](https://huggingface.co/blog/train-multi-vector-encoder#additional-resources)\n - [Training Examples](https://huggingface.co/blog/train-multi-vector-encoder#training-examples)\n - [Documentation](https://huggingface.co/blog/train-multi-vector-encoder#documentation)\n\n## What are Multi-Vector models?\n\nA dense embedding model compresses a whole text into a single vector, and similarity is one dot product between two such summaries. A multi-vector model (also called a late-interaction or ColBERT-style model) skips that compression. It keeps **one small vector per token** and scores a query against a document with the MaxSim operator, where every query token finds its best-matching document token and the scores are summed. Token-level matching preserves exactly the fine-grained signals that a single vector has to average away, which usually means stronger retrieval, at the cost of a bigger index.\n\nThe companion [Multi-Vector Embedding Models](https://huggingface.co/blog/multi-vector-encoder) blogpost covers the architecture, encoding, scoring, and indexing in detail, so I'll keep this section short and get to the training.\n\n[](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/multi-vector-encoder/maxsim_explainer.gif)\n\n## Why Finetune?\n\nFinetuning multi-vector models significantly improves their retrieval performance on your specific domain: the vocabulary, the query style, and the notion of relevance all differ between web search, legal discovery, code search, and scientific literature review. Because queries and documents are matched token by token, multi-vector models pick up fine-grained domain signals that single-vector models tend to average away, and they respond very well to even modest amounts of in-domain finetuning data.\n\nBeyond that, most released retrieval models were configured for short passages. The classic ColBERT checkpoints truncate documents at 180 or 300 tokens, and many popular dense models at 256 or 512, because their MS MARCO-style training data rarely goes beyond that. If your documents are long, these models silently discard most of every document before scoring it. On my medical evaluation with passages averaging 941 tokens, I measured that this truncation costs up to 0.24 NDCG@10, considerably more than any difference between model architectures. When you train your own model, you configure the document length that *your* data needs.\n\nLightOn ran into this same dynamic with code retrieval, where general [LateOn](https://huggingface.co/lightonai/LateOn) wasn't enough and they trained [LateOn-Code](https://huggingface.co/lightonai/LateOn-Code). Your domain, whether that's medical, legal, financial, or your company's internal documents, is not getting an official model. This blogpost shows you how to build it yourself, in a matter of hours, on a single consumer GPU.\n\n## Training Components\n\nTraining MultiVectorEncoder models involves the following components:\n\n1. [**Model**](https://huggingface.co/blog/train-multi-vector-encoder#model): The model to finetune or the architecture to build fresh.\n2. [**Dataset**](https://huggingface.co/blog/train-multi-vector-encoder#dataset): The data used for training and evaluation.\n3. [**Loss Function**](https://huggingface.co/blog/train-multi-vector-encoder#loss-function): A function that measures the model's performance and guides the optimization process.\n4. [**Training Arguments**](https://huggingface.co/blog/train-multi-vector-encoder#training-arguments) (optional): Parameters that impact training performance, tracking, and debugging.\n5. [**Evaluator**](https://huggingface.co/blog/train-multi-vector-encoder#evaluator) (optional): A class for evaluating the model before, during, or after training.\n6. [**Trainer**](https://huggingface.co/blog/train-multi-vector-encoder#trainer): Brings together all training components.\n\nLet's take a closer look at each component.\n\n## Model\n\nMulti-vector training gives you a real choice of starting point, and it matters more than you might expect.\n\n### Finetuning an existing multi-vector model\n\nIf you want to further finetune an existing multi-vector model, you don't have to worry about the architecture at all:\n\n```python\nfrom sentence_transformers import MultiVectorEncoder\n\n# Loading in fp32 is preferred for training if your memory can handle it\nmodel = MultiVectorEncoder(\n \"lightonai/mLateOn-unsupervised\",\n model_kwargs={\"torch_dtype\": \"float32\"},\n processor_kwargs={\"model_max_length\": 8192}, # the tokenizer-level token limit\n)\n```\n\nThe checkpoint brings its own recipe along: its query and document marker tokens, its projection head, its scoring skiplist. For finetuning, you generally want to keep all of that and change only what your data demands. The first thing to check is the length configuration, since many released checkpoints cap documents at 180 to 512 tokens (see [Why Finetune?](https://huggingface.co/blog/train-multi-vector-encoder#why-finetune)), and my medical passages run to 1,400 tokens. The mLateOn family already serves the backbone's full 8192 token context, but if your starting checkpoint carries caps, lift them:\n\n```python\n# Let the model read full documents instead of the caps it was trained with,\n# e.g. GTE-ModernColBERT-v1 ships with query_length=48 and document_length=300\nmodel[0].query_length = None\nmodel[0].document_length = None\n```\n\nWith the per-task caps unset, truncation falls back to the tokenizer's `model_max_length`, which is why I configure that limit at load time above.\n\nI made one more change, adding a punctuation skiplist that excludes punctuation tokens from document-side scoring and storage. In a 4-way ablation (none, punctuation, stopwords, both) it modestly won on quality, and it shrinks the document index by 9.6% on this data for free:\n\n```python\nimport string\n\n# model[2] is the MultiVectorMask module\nmodel[2].skiplist_words = list(string.punctuation)\nmodel[2].resolve_with_tokenizer(model.tokenizer) # token ids are cached, so re-resolve after changing\n```\n\n### Building one from a base transformer\n\nYou can also point `MultiVectorEncoder` at any base transformer, and a fresh, randomly initialized token-level projection is appended for you:\n\n```python\nfrom sentence_transformers import MultiVectorEncoder\n\nmodel = MultiVectorEncoder(\"answerdotai/ModernBERT-base\", model_kwargs={\"torch_dtype\": \"float32\"})\n# MultiVectorEncoder(\n# (0): Transformer({..., 'architecture': 'ModernBertModel'})\n# (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, ...})\n# (2): MultiVectorMask({'skiplist_words': [], 'skiplist_tasks': ['document'], ...})\n# (3): Normalize({...})\n# )\n```\n\nThat's the classic ColBERT pipeline: a `Transformer` producing contextualized token embeddings, a token-level `Dense` projecting each of them down to 128 dimensions, a `MultiVectorMask` deciding which tokens count during scoring, and a token-level `Normalize`. The projection starts random, so training is required before this model is useful. Interestingly, this works with strong dense embedding backbones too. A fresh projection on [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) reached within 0.03 of the existing-checkpoint starting points in my experiments, from nothing but the projection and 25k training pairs.\n\nThe classic ColBERT tokenization tricks (`[MASK]` query expansion, `[Q]` / `[D]` prefix tokens, a document length cap, a punctuation skiplist) are all off by default and configurable. See [Creating Custom Models](https://sbert.net/docs/multi_vector_encoder/usage/custom_models.html) for the full set. For what it's worth, I tested `[MASK]` query expansion in four configurations for my domain finetune and none of them made a measurable difference, so don't feel obliged to reach for the classic recipe.\n\n### Which starting point should you pick?\n\nI measured this directly while preparing this blogpost, taking six starting points and training each with the identical recipe on 25k medical question-passage pairs from [MIRIAD](https://huggingface.co/datasets/tomaarsen/miriad-4.4M-split), then evaluating on 1,000 held-out questions against a 50,000 passage corpus:\n\n| Starting point | Zero-shot NDCG@10 | After 25k pairs | Delta |\n| -------------------------------------------------------------------------------------------- | ----------------- | --------------- | ----------- |\n| [lightonai/mLateOn-unsupervised](https://huggingface.co/lightonai/mLateOn-unsupervised) | 0.9087 | **0.9398** | **+0.0311** |\n| [lightonai/mLateOn](https://huggingface.co/lightonai/mLateOn) | 0.9277 | 0.9319 | +0.0042 |\n| [lightonai/LateOn-unsupervised](https://huggingface.co/lightonai/LateOn-unsupervised) | 0.9026 | **0.9206** | **+0.0180** |\n| [lightonai/LateOn](https://huggingface.co/lightonai/LateOn) | 0.9185 | 0.9105 | -0.0080 |\n| [lightonai/GTE-ModernColBERT-v1](https://huggingface.co/lightonai/GTE-ModernColBERT-v1) | 0.9198 | 0.9007 | -0.0191 |\n| Fresh head on [gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) | - | 0.9177 | - |\n\nThe result surprised me, and it replicated across two model families. \\*The `-unsupervised` checkpoints adapt to a new domain far better than their finished siblings, overtaking them despite starting lower. These checkpoints sit after large-scale contrastive pretraining but before supervised finetuning on general retrieval, so they carry all the late-interaction structure with none of the general-purpose tuning that domain training then has to undo. The finished checkpoints, by contrast, barely moved or even regressed, at every learning rate I tried.\n\nSo, if the model family you like publishes a pre-supervised checkpoint, start there. If not, a fresh projection on a strong retrieval-pretrained backbone is a close runner-up. Continuing from a fully finished checkpoint is the weakest option for domain adaptation, despite being the most natural-feeling one.\n\n## Dataset\n\nThe [`MultiVectorEncoderTrainer`](https://sbert.net/docs/package_reference/multi_vector_encoder/trainer.html) uses [`datasets.Dataset`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset) or [`datasets.DatasetDict`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict) instances for training and evaluation. You can load data from the [Hugging Face Datasets Hub](https://huggingface.co/datasets) or use local data in whatever format you prefer (e.g. CSV, JSON, Parquet, Arrow, or SQL).\n\n**Note:** Lots of public datasets that work out of the box with Sentence Transformers have been tagged with `sentence-transformers` on the Hugging Face Hub, so you can easily find them on [https://huggingface.co/datasets?other=sentence-transformers](https://huggingface.co/datasets?other=sentence-transformers). Consider browsing through these to find ready-to-go datasets that might be useful for your tasks, domains, or languages.\n\n### Data on the Hugging Face Hub\n\nYou can use the [`load_dataset`](https://huggingface.co/docs/datasets/main/en/package_reference/loading_methods#datasets.load_dataset) function to load data from datasets on the Hub:\n\n```python\nfrom datasets import load_dataset\n\ntrain_dataset = load_dataset(\"tomaarsen/miriad-4.4M-split\", split=\"train\")\n\nprint(train_dataset)\n\"\"\"\nDataset({\n features: ['question', 'passage_text'],\n num_rows: 4467542\n})\n\"\"\"\n```\n\nThis is the dataset I'll train on in this blogpost: 4.4 million medical questions from [MIRIAD](https://huggingface.co/datasets/miriad/miriad-4.4M), each paired with the source passage that contains its answer (averaging 941 tokens). Simple (query, relevant passage) pairs like these are the easiest retrieval training data to collect for your own domain, and as you'll see, they're all you need.\n\n### Local Data\n\nYou can also use [`load_dataset`](https://huggingface.co/docs/datasets/main/en/package_reference/loading_methods#datasets.load_dataset) for loading local data in common file formats:\n\n```python\nfrom datasets import load_dataset\n\ndataset = load_dataset(\"csv\", data_files=\"my_file.csv\")\n# or\ndataset = load_dataset(\"json\", data_files=\"my_file.json\")\n```\n\nAnd if your local data requires pre-processing, you can use [`datasets.Dataset.from_dict`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.from_dict) to initialize your dataset with a dictionary of lists:\n\n```python\nfrom datasets import Dataset\n\nqueries = []\ndocuments = []\n# Open a file, perform preprocessing, filtering, cleaning, etc.\n# and append to the lists\n\ndataset = Dataset.from_dict({\n \"query\": queries,\n \"document\": documents,\n})\n```\n\n### Dataset Format\n\nIt is important that your dataset format matches your loss function (or that you choose a loss function that matches your dataset format). Verifying whether a dataset format works with a loss function involves two steps:\n\n1. If your loss function requires a *Label* according to the [Loss Overview](https://sbert.net/docs/multi_vector_encoder/loss_overview.html) table, then your dataset must have a **column named \"label\" or \"score\"**. This column is automatically taken as the label.\n2. All columns not named \"label\" or \"score\" are considered *Inputs* according to the [Loss Overview](https://sbert.net/docs/multi_vector_encoder/loss_overview.html) table. The number of remaining columns must match the number of valid inputs for your chosen loss. The names of these columns are **irrelevant**, only the **order matters**.\n\nThere are two multi-vector specific conventions on top of this:\n\n- Positional query and document assignment: the first column is embedded as the *query* and all following columns as *documents*, regardless of the column names. This default can be overridden per column via the standard `router_mapping` training argument.\n- Knowledge distillation format: one column per candidate document, i.e. `(query, document_1, ..., document_N, scores)` where `scores` is a list of N teacher scores per row. For KD datasets that store query and document *IDs* alongside separate text datasets (e.g. [lightonai/ms-marco-en-bge](https://huggingface.co/datasets/lightonai/ms-marco-en-bge)), you can use [`resolve_ids`](https://sbert.net/docs/package_reference/util.html#sentence_transformers.util.dataset.resolve_ids) to resolve the IDs to texts on the fly.\n\n## Loss Function\n\nLoss functions quantify how well a model performs for a given batch of data, allowing an optimizer to update the model weights to produce more favourable (i.e., lower) loss values. The right loss function for your task depends on the data you have and what you're trying to achieve. You can find a full list of options in the [Loss Overview](https://sbert.net/docs/multi_vector_encoder/loss_overview.html).\n\nFor the common case of question-answer or question-passage pairs, the workhorse is in-batch negatives training with [`MultiVectorMultipleNegativesRankingLoss`](https://sbert.net/docs/package_reference/multi_vector_encoder/losses.html#multivectormultiplenegativesrankingloss), where every other document in the batch acts as a negative for each query. Bigger batches mean more negatives and stronger training, so in practice you'll want its GradCache variant, [`CachedMultiVectorMultipleNegativesRankingLoss`](https://sbert.net/docs/package_reference/multi_vector_encoder/losses.html#cachedmultivectormultiplenegativesrankingloss), which decouples the effective batch size from what fits on your GPU:\n\n```python\nfrom sentence_transformers import MultiVectorEncoder\nfrom sentence_transformers.multi_vector_encoder.losses import CachedMultiVectorMultipleNegativesRankingLoss\n\nmodel = MultiVectorEncoder(\"lightonai/mLateOn-unsupervised\", model_kwargs={\"torch_dtype\": \"float32\"})\n\nloss = CachedMultiVectorMultipleNegativesRankingLoss(\n model=model,\n mini_batch_size=16, # how many documents to encode per chunk: bounds memory, not quality\n)\n```\n\nThe `mini_batch_size` parameter bounds the memory by encoding documents in chunks of this size, while the effective contrastive batch size (128 in my run below, and in my ablations bigger batches bought nothing further) stays a free choice. GradCache guarantees identical results regardless of the chunk size, so lower it for smaller GPUs at only a wall-clock cost. When your document lengths vary a lot, consider its sibling `mini_batch_num_tokens`, which packs each chunk to a total token budget instead of a document count, so a chunk of unusually long documents can never spike your memory (my `mini_batch_size=16` at roughly 940 tokens per document corresponds to `mini_batch_num_tokens=15_000`).\n\nOne multi-vector specific trap is that the contrastive losses default to `scale=1.0`, unlike the dense embedding equivalent which defaults to `scale=20.0`. That 20.0 exists because a cosine similarity is a single value in \\[-1, 1\\], too narrow a range for a sharp softmax. A MaxSim score instead sums one best-match similarity per query token, so it already spans roughly \\[0, query\\_length\\]: a 32-token query can score up to 32. So don't copy `scale=20.0` over from a dense training script, since it would saturate the softmax and kill your gradients.\n\nFor distillation from a stronger teacher, which is how the strongest general-purpose late-interaction models are trained, see [`MultiVectorDistillKLDivLoss`](https://sbert.net/docs/package_reference/multi_vector_encoder/losses.html#multivectordistillkldivloss) and the Knowledge Distillation tab in the [Training Overview](https://sbert.net/docs/multi_vector_encoder/training_overview.html#trainer) documentation.\n\n## Training Arguments\n\nYou can customize the training process using the [`MultiVectorEncoderTrainingArguments`](https://sbert.net/docs/package_reference/multi_vector_encoder/training_args.html) class. This class lets you adjust parameters that can impact training speed and help you understand what's happening during training.\n\nFor more information on the most useful training arguments, check out the [Multi-Vector Encoder > Training Overview > Training Arguments](https://sbert.net/docs/multi_vector_encoder/training_overview.html#training-arguments). It's worth reading to get the most out of your training.\n\nHere's an example, using the values from my actual training run:\n\n```python\nfrom sentence_transformers import MultiVectorEncoderTrainingArguments\nfrom sentence_transformers.base.sampler import BatchSamplers\n\nargs = MultiVectorEncoderTrainingArguments(\n # Required parameter:\n output_dir=\"models/mLateOn-medical\",\n # Optional training parameters:\n num_train_epochs=1,\n per_device_train_batch_size=128, # the effective contrastive batch, thanks to GradCache\n per_device_eval_batch_size=16,\n learning_rate=1e-4,\n warmup_steps=0.05,\n prompts={\"question\": \"[Q] \", \"passage_text\": \"[D] \"}, # the checkpoint's markers, keyed by training column\n fp16=False, # Set to True if you have a GPU that supports FP16\n bf16=True, # Set to True if you have a GPU that supports BF16\n batch_sampler=BatchSamplers.NO_DUPLICATES, # in-batch negatives benefit from no duplicates\n # Optional tracking/debugging parameters:\n eval_strategy=\"steps\",\n eval_steps=0.1,\n save_strategy=\"steps\",\n save_steps=0.05,\n logging_steps=0.01,\n run_name=\"mLateOn-medical\", # Will be used in e.g. Trackio, W&B, etc.\n)\n```\n\nA few of these deserve a comment:\n\n- `prompts`: training does not automatically apply the prompts stored in the model, so map them onto your training columns explicitly. Here that is the checkpoint's `[Q]` marker for the question column and `[D]` for the passage column, keeping training consistent with inference.\n- `max_length` (deliberately not set): this argument caps tokenization during *training only*, for when you want cheaper training than the model's full serving length. I measured what that shortcut costs on this data. Training at 512 tokens lost about 0.015 NDCG@10 for about 2x the speed, and the deficit did not shrink with more data, because the model simply never sees what got cut off. Leave it unset so training matches inference, unless you need the speedup more than the quality.\n- `learning_rate=1e-4`: after a sweep from 5e-6 to 2e-4, I had the best luck with this higher-than-usual learning rate.\n\n## Evaluator\n\nTo track your model's performance during training, you can pass an `eval_dataset` to the trainer for evaluation loss, but concrete retrieval metrics are much more informative. Sentence Transformers includes the following built-in evaluators for multi-vector models:\n\n| Evaluator | Required Data |\n| --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |\n| [`MultiVectorInformationRetrievalEvaluator`](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html#multivectorinformationretrievalevaluator) | Queries, corpus, and relevant document mappings |\n| [`MultiVectorNanoBEIREvaluator`](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html#multivectornanobeirevaluator) | No data required |\n| [`MultiVectorTripletEvaluator`](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html#multivectortripletevaluator) | (anchor, positive, negative) triplets |\n| [`MultiVectorRerankingEvaluator`](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html#multivectorrerankingevaluator) | List of `{'query': '...', 'positive': [...], 'negative': [...]}` dictionaries |\n| [`MultiVectorDistillationEvaluator`](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html#multivectordistillationevaluator) | Queries with candidate documents and teacher scores |\n\nFor domain finetuning, the [`MultiVectorInformationRetrievalEvaluator`](https://sbert.net/docs/package_reference/multi_vector_encoder/evaluation.html#multivectorinformationretrievalevaluator) built from your own held-out data is the one that matters. One tip on constructing it is that the corpus should be hard enough that models can be told apart. In my case the MIRIAD questions are generated from their own source passages, which makes retrieval unusually easy. Against just the 10k gold passages, nearly every model scored above 0.97 NDCG@10. If your evaluation saturates like that, add *distractor* passages (I use deduplicated passages from the training split) until the scores spread out:\n\n```python\nfrom datasets import load_dataset\nfrom sentence_transformers.multi_vector_encoder.evaluation import MultiVectorInformationRetrievalEvaluator\n\ndataset = load_dataset(\"tomaarsen/miriad-4.4M-split\")\n\n# Gold: 1,000 evaluation questions, each mapping to its own passage, with the\n# eval split's full ~10k unique passages as the initial corpus\ncorpus = {}\nqueries = {}\nrelevant_docs = {}\npassage_to_id = {}\nfor idx, row in enumerate(dataset[\"eval\"]):\n if row[\"passage_text\"] not in passage_to_id:\n passage_to_id[row[\"passage_text\"]] = f\"p{len(passage_to_id)}\"\n corpus[passage_to_id[row[\"passage_text\"]]] = row[\"passage_text\"]\n if idx < 1_000:\n queries[f\"q{idx}\"] = row[\"question\"]\n relevant_docs[f\"q{idx}\"] = {passage_to_id[row[\"passage_text\"]]}\n\n# Distractors: unique train passages that make the haystack realistic\nseen = set(passage_to_id)\nfor row in dataset[\"train\"]:\n if len(corpus) >= 200_000:\n break\n if row[\"passage_text\"] not in seen:\n seen.add(row[\"passage_text\"])\n corpus[f\"d{len(corpus)}\"] = row[\"passage_text\"]\n\nevaluator = MultiVectorInformationRetrievalEvaluator(\n queries=queries,\n corpus=corpus,\n relevant_docs=relevant_docs,\n name=\"miriad-dev\",\n batch_size=16,\n)\n# results = evaluator(model)\n```\n\n## Trainer\n\nThe [`MultiVectorEncoderTrainer`](https://sbert.net/docs/package_reference/multi_vector_encoder/trainer.html) is where all previous components come together. Here is the complete script that trained [multi-vector-encoder/mLateOn-medical](https://huggingface.co/multi-vector-encoder/mLateOn-medical), the model from the introduction:\n\n```python\nimport logging\nimport string\nimport traceback\n\nfrom datasets import load_dataset\n\nfrom sentence_transformers import (\n MultiVectorEncoder,\n MultiVectorEncoderModelCardData,\n MultiVectorEncoderTrainer,\n MultiVectorEncoderTrainingArguments,\n)\nfrom sentence_transformers.base.sampler import BatchSamplers\nfrom sentence_transformers.multi_vector_encoder.evaluation import MultiVectorInformationRetrievalEvaluator\nfrom sentence_transformers.multi_vector_encoder.losses import CachedMultiVectorMultipleNegativesRankingLoss\n\nlogging.basicConfig(format=\"%(asctime)s - %(message)s\", datefmt=\"%Y-%m-%d %H:%M:%S\", level=logging.INFO)\n\n\ndef main():\n # 1. Load the starting checkpoint: contrastively pretrained, not yet supervised\n # Loading in fp32 is preferred for training if your memory can handle it\n model = MultiVectorEncoder(\n \"lightonai/mLateOn-unsupervised\",\n model_kwargs={\"torch_dtype\": \"float32\"},\n processor_kwargs={\"model_max_length\": 8192},\n model_card_data=MultiVectorEncoderModelCardData(\n language=\"en\",\n license=\"apache-2.0\",\n model_name=\"mLateOn finetuned on MIRIAD medical retrieval\",\n ),\n )\n\n # 2. Lift the per-task length caps so training and inference see full medical passages\n model[0].query_length = None\n model[0].document_length = None\n\n # 3. Skip punctuation tokens during scoring: a small quality win and a 9.6% smaller index\n model[2].skiplist_words = list(string.punctuation)\n model[2].resolve_with_tokenizer(model.tokenizer)\n\n # 4. Load 1 million medical question-passage pairs\n train_dataset = load_dataset(\"tomaarsen/miriad-4.4M-split\", split=\"train\").select(range(1_000_000))\n\n # 5. In-batch negatives with GradCache: large effective batch, memory-bounded chunks\n loss = CachedMultiVectorMultipleNegativesRankingLoss(model=model, mini_batch_size=16)\n\n # 6. A light dev evaluator to watch progress during training: 500 held-out questions\n # against the eval split's ~10k unique passages. The full 200k protocol runs afterwards.\n eval_split = load_dataset(\"tomaarsen/miriad-4.4M-split\", split=\"eval\")\n corpus, queries, relevant_docs, passage_to_id = {}, {}, {}, {}\n for idx, row in enumerate(eval_split):\n if row[\"passage_text\"] not in passage_to_id:\n passage_to_id[row[\"passage_text\"]] = f\"p{len(passage_to_id)}\"\n corpus[passage_to_id[row[\"passage_text\"]]] = row[\"passage_text\"]\n if idx < 500:\n queries[f\"q{idx}\"] = row[\"question\"]\n relevant_docs[f\"q{idx}\"] = {passage_to_id[row[\"passage_text\"]]}\n dev_evaluator = MultiVectorInformationRetrievalEvaluator(\n queries=queries, corpus=corpus, relevant_docs=relevant_docs, name=\"miriad-dev\", batch_size=16\n )\n\n # 7. Training arguments, as discussed above\n run_name = \"mLateOn-medical\"\n args = MultiVectorEncoderTrainingArguments(\n output_dir=f\"models/{run_name}\",\n num_train_epochs=1,\n per_device_train_batch_size=128,\n per_device_eval_batch_size=16,\n learning_rate=1e-4,\n warmup_steps=0.05,\n prompts={\"question\": \"[Q] \", \"passage_text\": \"[D] \"},\n fp16=False, # Set to True if you have a GPU that supports FP16\n bf16=True, # Set to True if you have a GPU that supports BF16\n batch_sampler=BatchSamplers.NO_DUPLICATES,\n eval_strategy=\"steps\",\n eval_steps=0.1,\n save_strategy=\"steps\",\n save_steps=0.05,\n logging_steps=0.01,\n run_name=run_name,\n )\n\n # 8. Create a trainer & train\n trainer = MultiVectorEncoderTrainer(\n model=model,\n args=args,\n train_dataset=train_dataset,\n loss=loss,\n evaluator=dev_evaluator,\n )\n trainer.train()\n\n # 9. Save the trained model\n model.save_pretrained(f\"models/{run_name}/final\")\n\n # 10. (Optional) Push it to the Hugging Face Hub\n try:\n model.push_to_hub(run_name)\n except Exception:\n logging.error(f\"Error uploading model to the Hugging Face Hub:\\n{traceback.format_exc()}\")\n\n\nif __name__ == \"__main__\":\n main()\n```\n\nThat's the whole recipe: a pre-supervised checkpoint, a million domain pairs, in-batch negatives, full document length, and a higher-than-usual learning rate. The run took 14.5 hours on my single RTX 3090 at a peak of 17.5 GB VRAM, and every one of those choices was the winner of a measured comparison rather than a guess.\n\nFor readers on smaller budgets, my scaling experiments put 100k pairs (75 minutes of training) within 0.012 NDCG@10 of the full million-pair run. Most of the gain comes in the first hour.\n\n### Callbacks\n\nThe MultiVectorEncoder trainer supports various [`transformers.TrainerCallback`](https://huggingface.co/docs/transformers/main_classes/callback#transformers.TrainerCallback) subclasses, including:\n\n- [`WandbCallback`](https://huggingface.co/docs/transformers/en/main_classes/callback#transformers.integrations.WandbCallback) for logging training metrics to W&B if `wandb` is installed\n- [`TensorBoardCallback`](https://huggingface.co/docs/transformers/en/main_classes/callback#transformers.integrations.TensorBoardCallback) for logging training metrics to TensorBoard if `tensorboard` is accessible\n- [`CodeCarbonCallback`](https://huggingface.co/docs/transformers/en/main_classes/callback#transformers.integrations.CodeCarbonCallback) for tracking carbon emissions during training if `codecarbon` is installed\n\nEnable these via the `report_to` training argument, e.g. `report_to=[\"wandb\", \"codecarbon\"]`, with the required dependencies installed. It defaults to `\"none\"`, and `report_to=\"all\"` activates every integration whose dependency is installed.\n\nRefer to the [Transformers Callbacks documentation](https://huggingface.co/docs/transformers/en/main_classes/callback) for more information on these callbacks and how to create your own.\n\n### Multi-Dataset Training\n\nTypically, top-performing general-purpose models are trained on multiple datasets simultaneously. However, this approach can be challenging due to the varying formats of each dataset. Fortunately, the [`MultiVectorEncoderTrainer`](https://sbert.net/docs/package_reference/multi_vector_encoder/trainer.html) allows you to train on multiple datasets without requiring a uniform format. Additionally, it provides the flexibility to apply different loss functions to each dataset. Here are the steps to train with multiple datasets at once:\n\n- Use a dictionary of [`datasets.Dataset`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset) instances (or a [`datasets.DatasetDict`](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.DatasetDict)) as the `train_dataset` (and optionally also `eval_dataset`).\n- (Optional) Use a dictionary of loss functions mapping dataset names to losses. Only required if you wish to use different loss functions for different datasets.\n\nEach training/evaluation batch will only contain samples from one of the datasets. The order in which batches are sampled from the multiple datasets is defined by the [`MultiDatasetBatchSamplers`](https://sbert.net/docs/package_reference/sentence_transformer/sampler.html#sentence_transformers.training_args.MultiDatasetBatchSamplers) enum, which can be passed to the [`MultiVectorEncoderTrainingArguments`](https://sbert.net/docs/package_reference/multi_vector_encoder/training_args.html) via `multi_dataset_batch_sampler`. Valid options are:\n\n- `MultiDatasetBatchSamplers.ROUND_ROBIN`: Round-robin sampling from each dataset until one is exhausted. With this strategy, it's likely that not all samples from each dataset are used, but each dataset is sampled from equally.\n- `MultiDatasetBatchSamplers.PROPORTIONAL` (default): Sample from each dataset in proportion to its size. With this strategy, all samples from each dataset are used and larger datasets are sampled from more frequently.\n\n## Evaluation\n\nTo find out where the finetuned model stands, I evaluated it against over 50 retrieval model configurations across four architecture families on the MIRIAD evaluation set, built exactly as in the [Evaluator](https://huggingface.co/blog/train-multi-vector-encoder#evaluator) section above, with 1,000 held-out medical questions searching 200,000 unique passages (the 10k gold passages hidden among 190k deduplicated distractors from the training split). This corpus is four times the size of the 50,000-passage one from [Which starting point should you pick?](https://huggingface.co/blog/train-multi-vector-encoder#which-starting-point-should-you-pick), so scores are not comparable between the two tables.\n\n[](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/train-multi-vector-encoder/mve_medical_model_size_ndcg.png)\n\nThe headline results, with the full table in the collapsible below:\n\n| Model | Family | NDCG@10 |\n| --------------------------------------------------------------------------------------------------------------- | --------------------------- | ---------- |\n| [**multi-vector-encoder/mLateOn-medical (mine)**](https://huggingface.co/multi-vector-encoder/mLateOn-medical) | **Multi-vector, finetuned** | **0.9139** |\n| [lightonai/mLateOn](https://huggingface.co/lightonai/mLateOn) | Multi-vector, zero-shot | 0.8520 |\n| [lightonai/GTE-ModernColBERT-v1](https://huggingface.co/lightonai/GTE-ModernColBERT-v1) (cap lifted) | Multi-vector, zero-shot | 0.8502 |\n| [Qwen/Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) | Dense, zero-shot | 0.7817 |\n| [voyageai/voyage-4-nano](https://huggingface.co/voyageai/voyage-4-nano) | Dense, zero-shot | 0.7563 |\n| BM25 | Lexical | 0.7501 |\n| [naver/splade-v3](https://huggingface.co/naver/splade-v3) | Sparse, zero-shot | 0.6853 |\n\nThe finetuned model tops the table, beating the strongest zero-shot model of any architecture by +0.062 NDCG@10. In other words, the strongest zero-shot model returns the right passage as the very first hit for 75.8% of the queries, while the finetuned model does so for 84.9%, cutting the rank-1 error by more than a third.\n\nThe architecture pattern is just as clear, with the top of the table exclusively late interaction. On long documents, one vector per token beats one vector per document, even at matched training and matched backbones. DenseOn and LateOn share training data and architecture except for the head, and the late-interaction sibling wins by +0.12, with the multilingual pair (mDenseOn and mLateOn) replicating this at +0.13. Scale doesn't rescue single vectors either. [Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B), the strongest dense model with roughly 33x the active (non-embedding) parameters of mine, still stops 0.13 short, and the 8B version scores lower than the 4B.\n\nBM25 also performs surprisingly well, beating every sparse model, every truncation-capped multi-vector model, and all but three dense models: the multi-billion [Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) and [8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B), and [voyage-4-nano](https://huggingface.co/voyageai/voyage-4-nano), which reads its full 32k token context to edge past by just 0.006. Don't expect that to transfer to your own data though. MIRIAD's questions are generated from the passages, so the lexical overlap between a query and its gold passage is far larger than in typical retrieval, and BM25's unlimited context length lets it use every one of those overlapping words while most neural checkpoints truncate. A BM25 baseline is cheap and always worth running, just don't count on this margin.\n\nThe full field at a glance, sorted by score and colored by architecture family.\n\n[](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/train-multi-vector-encoder/mve_medical_ndcg_by_model.png)\n\nClick to see the full evaluation table\n\n| Model | Family | NDCG@10 | acc@1 |\n| ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ---------- | ----- |\n| [**multi-vector-encoder/mLateOn-medical (mine)**](https://huggingface.co/multi-vector-encoder/mLateOn-medical) | Multi-vector, finetuned | **0.9139** | 0.849 |\n| [lightonai/mLateOn](https://huggingface.co/lightonai/mLateOn) | Multi-vector | 0.8520 | 0.758 |\n| [lightonai/GTE-ModernColBERT-v1](https://huggingface.co/lightonai/GTE-ModernColBERT-v1) @1024 | Multi-vector | 0.8502 | 0.763 |\n| [lightonai/LateOn](https://huggingface.co/lightonai/LateOn) @1024 | Multi-vector | 0.8485 | 0.760 |\n| [lightonai/mLateOn-unsupervised](https://huggingface.co/lightonai/mLateOn-unsupervised) | Multi-vector | 0.8304 | 0.733 |\n| [mixedbread-ai/mxbai-edge-colbert-v0-32m](https://huggingface.co/mixedbread-ai/mxbai-edge-colbert-v0-32m) @1024 | Multi-vector | 0.8186 | 0.727 |\n| [Qwen/Qwen3-Embedding-4B](https://huggingface.co/Qwen/Qwen3-Embedding-4B) | Dense | 0.7817 | 0.669 |\n| [Qwen/Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) | Dense | 0.7747 | 0.654 |\n| [perplexity-ai/pplx-embed-v1-late-0.6b](https://huggingface.co/perplexity-ai/pplx-embed-v1-late-0.6b) @1024 | Multi-vector | 0.7702 | 0.632 |\n| [lightonai/ColBERT-Zero](https://huggingface.co/lightonai/ColBERT-Zero) | Multi-vector | 0.7613 | 0.675 |\n| [LiquidAI/LFM2.5-ColBERT-350M](https://huggingface.co/LiquidAI/LFM2.5-ColBERT-350M) | Multi-vector | 0.7582 | 0.664 |\n| [voyageai/voyage-4-nano](https://huggingface.co/voyageai/voyage-4-nano) | Dense | 0.7563 | 0.638 |\n| BM25 | Lexical | 0.7501 | 0.641 |\n| [jinaai/jina-embeddings-v5-text-small-retrieval](https://huggingface.co/jinaai/jina-embeddings-v5-text-small-retrieval) | Dense | 0.7470 | 0.620 |\n| [Qwen/Qwen3-Embedding-0.6B](https://huggingface.co/Qwen/Qwen3-Embedding-0.6B) | Dense | 0.7408 | 0.620 |\n| [perplexity-ai/pplx-embed-v1-0.6b](https://huggingface.co/perplexity-ai/pplx-embed-v1-0.6b) | Dense | 0.7384 | 0.615 |\n| [mixedbread-ai/mxbai-edge-colbert-v0-32m](https://huggingface.co/mixedbread-ai/mxbai-edge-colbert-v0-32m) | Multi-vector | 0.7350 | 0.639 |\n| [mixedbread-ai/mxbai-edge-colbert-v0-17m](https://huggingface.co/mixedbread-ai/mxbai-edge-colbert-v0-17m) | Multi-vector | 0.7271 | 0.631 |\n| [answerdotai/answerai-colbert-small-v1](https://huggingface.co/answerdotai/answerai-colbert-small-v1) @512 | Multi-vector | 0.7264 | 0.615 |\n| [lightonai/DenseOn](https://huggingface.co/lightonai/DenseOn) @1024 | Dense | 0.7239 | 0.597 |\n| [lightonai/mDenseOn](https://huggingface.co/lightonai/mDenseOn) @1024 | Dense | 0.7227 | 0.585 |\n| [jinaai/jina-embeddings-v5-text-nano-retrieval](https://huggingface.co/jinaai/jina-embeddings-v5-text-nano-retrieval) | Dense | 0.7206 | 0.587 |\n| [microsoft/harrier-oss-v1-0.6b](https://huggingface.co/microsoft/harrier-oss-v1-0.6b) | Dense | 0.7126 | 0.572 |\n| [Alibaba-NLP/gte-modernbert-base](https://huggingface.co/Alibaba-NLP/gte-modernbert-base) | Dense | 0.7102 | 0.582 |\n| [Snowflake/snowflake-arctic-embed-l-v2.0](https://huggingface.co/Snowflake/snowflake-arctic-embed-l-v2.0) | Dense | 0.7068 | 0.568 |\n| [perplexity-ai/pplx-embed-v1-late-0.6b](https://huggingface.co/perplexity-ai/pplx-embed-v1-late-0.6b) | Multi-vector | 0.7008 | 0.570 |\n| [google/embeddinggemma-300m](https://huggingface.co/google/embeddinggemma-300m) | Dense | 0.7000 | 0.563 |\n| [lightonai/DenseOn](https://huggingface.co/lightonai/DenseOn) | Dense | 0.6943 | 0.570 |\n| [naver/splade-v3](https://huggingface.co/naver/splade-v3) | Sparse | 0.6853 | 0.574 |\n| [ibm-granite/granite-embedding-small-english-r2](https://huggingface.co/ibm-granite/granite-embedding-small-english-r2) | Dense | 0.6813 | 0.546 |\n| [naver/splade-v3-distilbert](https://huggingface.co/naver/splade-v3-distilbert) | Sparse | 0.6806 | 0.567 |\n| [codefuse-ai/F2LLM-v2-0.6B](https://huggingface.co/codefuse-ai/F2LLM-v2-0.6B) | Dense | 0.6799 | 0.536 |\n| [colbert-ir/colbertv2.0](https://huggingface.co/colbert-ir/colbertv2.0) @512 | Multi-vector | 0.6785 | 0.571 |\n| [prithivida/Splade\\_PP\\_en\\_v1](https://huggingface.co/prithivida/Splade_PP_en_v1) | Sparse | 0.6755 | 0.577 |\n| [lightonai/LateOn](https://huggingface.co/lightonai/LateOn) | Multi-vector | 0.6713 | 0.561 |\n| [tomaarsen/embeddinggemma-300m-miriad-unsloth](https://huggingface.co/tomaarsen/embeddinggemma-300m-miriad-unsloth) | Dense, finetuned | 0.6705 | 0.530 |\n| [lightonai/LateOn-regularized](https://huggingface.co/lightonai/LateOn-regularized) | Multi-vector | 0.6673 | 0.554 |\n| [lightonai/LateOn-unsupervised](https://huggingface.co/lightonai/LateOn-unsupervised) | Multi-vector | 0.6672 | 0.553 |\n| [lightonai/GTE-ModernColBERT-v1](https://huggingface.co/lightonai/GTE-ModernColBERT-v1) | Multi-vector | 0.6612 | 0.555 |\n| [opensearch-project/opensearch-neural-sparse-encoding-v2-distill](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-v2-distill) | Sparse | 0.6518 | 0.531 |\n| [nomic-ai/nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) (prompted) | Dense | 0.6387 | 0.498 |\n| [mixedbread-ai/mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1) | Dense | 0.6355 | 0.502 |\n| [BAAI/bge-large-en-v1.5](https://huggingface.co/BAAI/bge-large-en-v1.5) | Dense | 0.6308 | 0.498 |\n| [jinaai/jina-colbert-v2](https://huggingface.co/jinaai/jina-colbert-v2) @1024 | Multi-vector | 0.6218 | 0.504 |\n| [nomic-ai/nomic-embed-text-v1.5](https://huggingface.co/nomic-ai/nomic-embed-text-v1.5) | Dense | 0.6203 | 0.487 |\n| [answerdotai/answerai-colbert-small-v1](https://huggingface.co/answerdotai/answerai-colbert-small-v1) | Multi-vector | 0.6184 | 0.514 |\n| [tomaarsen/splade-modernbert-base-miriad](https://huggingface.co/tomaarsen/splade-modernbert-base-miriad) | Sparse, finetuned | 0.6142 | 0.473 |\n| [NeuML/biomedbert-base-colbert](https://huggingface.co/NeuML/biomedbert-base-colbert) | Multi-vector | 0.5963 | 0.463 |\n| [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) | Dense | 0.5930 | 0.454 |\n| [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) | Dense | 0.5881 | 0.457 |\n| [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) | Dense | 0.5159 | 0.396 |\n| [jinaai/jina-colbert-v2](https://huggingface.co/jinaai/jina-colbert-v2) | Multi-vector | 0.4992 | 0.401 |\n| [mixedbread-ai/mxbai-colbert-large-v1](https://huggingface.co/mixedbread-ai/mxbai-colbert-large-v1) | Multi-vector | 0.4690 | 0.358 |\n| [sentence-transformers/static-retrieval-mrl-en-v1](https://huggingface.co/sentence-transformers/static-retrieval-mrl-en-v1) | Dense | 0.4614 | 0.323 |\n| [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) | Dense | 0.4458 | 0.321 |\n| [colbert-ir/colbertv2.0](https://huggingface.co/colbert-ir/colbertv2.0) | Multi-vector | 0.4347 | 0.346 |\n\nModels marked `@N` are evaluated with their document length cap lifted to N tokens, since their native caps (180 to 512 tokens) would otherwise truncate the 941-token average passages. For every multi-vector model this lift was worth +0.08 to +0.24 NDCG@10 over the as-served row, and even the dense DenseOn gained +0.03 from the same treatment.\n\nNote that this does not mean that [multi-vector-encoder/mLateOn-medical](https://huggingface.co/multi-vector-encoder/mLateOn-medical) is the strongest model on *all* domains. It's simply the strongest in *my* domain. This is totally fine, as I just need this model to work well on my data.\n\nDon't underestimate the power of finetuning multi-vector models on your domain. Fourteen and a half hours on a single consumer GPU produced a model that no general-purpose retriever comes close to on this data, and the recipe is a single script with no teacher model and no mined negatives!\n\n### Optimizing the index\n\nThe fair objection to multi-vector retrieval is index size, and this domain is close to the worst case for it. Storing one vector per token, my model needs about 878 vectors per passage, so the 200,000-passage corpus takes roughly 45 GB at fp16, where a dense model needs well under 1 GB. Document length is what makes that gap so wide. The Natural Questions passages in the [companion post](https://huggingface.co/blog/multi-vector-encoder) average about 125 token vectors each, seven times fewer, so a corpus of short passages starts from a far smaller index than this one does. The [`HierarchicalTokenPooling`](https://sbert.net/docs/package_reference/multi_vector_encoder/modules.html#hierarchicaltokenpooling) module compresses exactly this by clustering each document's token embeddings and storing the cluster means, keeping roughly `1 / pool_factor` of the vectors:\n\n```python\nfrom sentence_transformers.multi_vector_encoder.modules import HierarchicalTokenPooling\n\npooling = HierarchicalTokenPooling(pool_factor=4)\ndocument_embeddings = model.encode_document(passages, token_pooling=pooling)\n```\n\nI measured it post-hoc on the finished model, with no pooling-aware training, and on long documents it is remarkably cheap.\n\n[](https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/train-multi-vector-encoder/mve_medical_index_size_ndcg.png)\n\nThe solid points are uncompressed embeddings, so that every family is counted the same way and scored with exact search. You would not deploy any of them like that, though. Dense indexes routinely use int8 or binary quantization with rescoring, sparse indexes compress their postings, and multi-vector indexes use PLAID-style residual compression. Don't read those points as the disk you need to buy, but as relative storage cost.\n\nToken pooling is the solid line. Halving the vector count costs 0.0033 NDCG@10 and leaves rank-1 accuracy untouched, and keeping only a quarter of them, at 11.2 GB, still scores 0.8991. The curve keeps going (I measured out to a tenth of the vectors, still at 0.8765) but there is little reason to push pooling that far once quantization is on the table, which is what the dashed line below is about.\n\nThe dashed line is what a real deployment might look like. I gave Omar Khattab early access to the model and the benchmark, and he measured these configurations with [fast-plaid](https://github.com/lightonai/fast-plaid) at 1-bit residual quantization, using compact 17-bit centroid ids and 18-bit document ids instead of its ordinary unpacked 64-bit integers, plus document-side pruning:\n\n| configuration | vectors kept | index | NDCG@10 |\n| ------------------------ | ------------ | ------- | ------- |\n| 1-bit PLAID, all vectors | 100% | 3.37 GB | 0.8984 |\n| 1-bit PLAID + pruning | 65% | 2.23 GB | 0.8830 |\n| 1-bit PLAID + pruning | 42% | 1.45 GB | 0.8642 |\n\nThat first row is 13x smaller than the raw embeddings, for 0.0155 NDCG@10. That is a far better trade than anywhere on the pooling curve. Quantization shrinks each vector while pooling and pruning cut how many you keep, so they compose, and quantization is the one to reach for first. Push further and the last row lands at 1.45 GB, *smaller* than the fp16 embeddings of [Qwen3-Embedding-8B](https://huggingface.co/Qwen/Qwen3-Embedding-8B) (1.64 GB), while scoring 0.0895 higher. The objection that multi-vector indexes are too big does not survive a properly configured index.\n\nThe pruning here is naive, meant only to establish that token reduction works on top of quantization, so read the bottom two rows as a floor rather than the frontier. If you would rather not hand-tune quantization at all, the [Indexing](https://huggingface.co/blog/multi-vector-encoder#indexing) section of the companion post covers fast-plaid, Qdrant, Weaviate, and Vespa.\n\nMulti-vector retrieval is only as expensive as its index. The raw embeddings for this corpus are 45 GB, and a properly configured index is at least 7x smaller at nearly the same accuracy. The index deserves as much of your attention as the checkpoint.\n\n## Acknowledgements\n\nThanks to [Omar Khattab](https://github.com/okhat) for measuring the quantized and pruned index configurations in [Optimizing the index](https://huggingface.co/blog/train-multi-vector-encoder#optimizing-the-index), and for the discussions around late-interaction index costs.\n\n## Additional Resources\n\n### Training Examples\n\nThese pages have training examples with explanations as well as links to training scripts. You can use them to get familiar with the multi-vector training loop:\n\n- [MIRIAD](https://sbert.net/examples/multi_vector_encoder/training/miriad/README.html): domain-specific training on medical retrieval, an earlier and simpler cousin of this blogpost's recipe\n- [MS MARCO](https://sbert.net/examples/multi_vector_encoder/training/msmarco/README.html): contrastive and knowledge distillation recipes\n- [Multimodal](https://sbert.net/examples/multi_vector_encoder/training/multimodal/README.html): ColPali-style visual document retrieval training\n- [PEFT Adapters](https://sbert.net/examples/multi_vector_encoder/training/peft/README.html): parameter-efficient finetuning with LoRA\n\n### Documentation\n\nFor further learning, you may also want to explore the following resources on Sentence Transformers:\n\n- [Installation](https://sbert.net/docs/installation.html)\n- [Quickstart](https://sbert.net/docs/quickstart.html)\n- [Usage](https://sbert.net/docs/multi_vector_encoder/usage/usage.html)\n- [Creating Custom Models](https://sbert.net/docs/multi_vector_encoder/usage/custom_models.html)\n- [Pretrained Models](https://sbert.net/docs/multi_vector_encoder/pretrained_models.html)\n- [Training Overview](https://sbert.net/docs/multi_vector_encoder/training_overview.html) (This blogpost is a distillation of the Training Overview documentation)\n- [Loss Overview](https://sbert.net/docs/multi_vector_encoder/loss_overview.html)\n- [API Reference](https://sbert.net/docs/package_reference/multi_vector_encoder/index.html)\n\nAnd here is an advanced page that might interest you:\n\n- [Distributed Training](https://sbert.net/docs/sentence_transformer/training/distributed.html)\n\nAnd the companion blogpost, covering everything about *using* these models:\n\n- [Multi-Vector (Late Interaction) Embedding Models with Sentence Transformers](https://huggingface.co/blog/multi-vector-encoder)\n", "dateCreated": "2026-10-19T01:26:53.998049939+00:00", "date_created": "2026-08-26T00:00:00+00:00", "date_modified": "2026-08-26T00:00:00+00:00", "date_published": "2026-08-26T00:00:00+00:00", "description": "Sentence Transformers is a Python library for using and training embedding and reranker models for a wide range of applications, such as retrieval augmented generation, semantic search, semantic textual similarity, and more. Its v6.0…", "external_url": "https://huggingface.co/blog/train-multi-vector-encoder", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/08/2026-08-26-training-and-finetuning-multi-vector-embedding-models-with.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/08/2026-08-26-training-and-finetuning-multi-vector-embedding-models-with.md" }, "id": "urn:aggr:item:0d00a6a331ca6b9ff36092b327c397da0f6a1da5", "isBasedOn": { "@id": "https://huggingface.co/blog/train-multi-vector-encoder" }, "mainEntity": { "@id": "https://huggingface.co/blog/train-multi-vector-encoder", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-08-26-training-and-finetuning-multi-vector-embedding-models-with/#webpage" }, "dateModified": "2026-08-26T00:00:00+00:00", "datePublished": "2026-08-26T00:00:00+00:00", "headline": "Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers", "timeRequired": "PT26M", "url": "https://huggingface.co/blog/train-multi-vector-encoder", "wordCount": 5676 }, "name": "Archived snapshot: Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers", "preview": null, "reading_minutes": 26, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-19T01:26:53.998049939+00:00", "title": "Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-08-26-training-and-finetuning-multi-vector-embedding-models-with/", "word_count": 5676 }