![\"Model](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/multimodal-sentence-transformers/vdr_plot.png\")
Sentence Transformers is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more. In my previous blogpost, I introduced the new multimodal capabilities, showing how to use embedding and reranker models that handle text, images, audio, and video. In this blogpost, I'll show you how to train or finetune these multimodal models on your own data.
\nAs a practical example, I'll walk through finetuning Qwen/Qwen3-VL-Embedding-2B for Visual Document Retrieval (VDR), the task of retrieving relevant document pages (as images, with charts, tables, and layout intact) for a given text query. The resulting tomaarsen/Qwen3-VL-Embedding-2B-vdr demonstrates how much performance you can gain by finetuning on your own domain. On my evaluation data, the finetuned model achieves an NDCG@10 of 0.947 compared to the base model's 0.888, and outperforms all existing VDR models I tested against, including models up to 4x its size.
\n\nIf you're new to multimodal models in Sentence Transformers, I recommend reading Multimodal Embedding & Reranker Models with Sentence Transformers first. For training text-only embedding, reranker, or sparse embedding models, see the Prior Blogposts section at the end.
\n
General-purpose multimodal embedding models like Qwen/Qwen3-VL-Embedding-2B are trained on diverse data to perform well across a wide range of languages and tasks: image-text matching, visual question answering, document understanding, and more. But this generality means the model is rarely the best choice for any specific task.
Consider Visual Document Retrieval: given a text query like "What was the company's Q3 revenue?", the model must find the most relevant document screenshot from a corpus of thousands. This requires understanding document layouts, charts, tables, and text, which is a very different skill from e.g. matching pictures of shoes with product descriptions.
\nBy finetuning on domain-specific data, the model can learn these specialized patterns. In my experiment, finetuning improved NDCG@10 from 0.888 to 0.947, ahead of every recent multimodal model I tested, including ones up to 4x larger.
\nTraining multimodal Sentence Transformer models involves the same components as training text-only models:
\nThe multimodal training pipeline uses the same SentenceTransformerTrainer as text-only training. The key difference is that your datasets contain images (or other modalities) alongside text, and the model's processor handles the image preprocessing automatically.
Let's walk through each component, using Visual Document Retrieval (matching text queries to document screenshots) as a running example.
\nThe most common approach is to finetune an existing multimodal embedding model, or to start from a Vision-Language Model (VLM) checkpoint. The Transformer module automatically detects supported modalities from the model's processor.
To finetune an existing multimodal embedding model (e.g. one that already has a modules.json file), you can pass processor_kwargs and model_kwargs to control preprocessing and model loading respectively. processor_kwargs are passed directly to AutoProcessor.from_pretrained(...) (e.g., image resolution bounds: higher max_pixels means higher quality but more memory), while model_kwargs are passed to the appropriate AutoModel.from_pretrained(...) call (e.g., precision, attention implementation):
\n\n = \nYou can also start from a fresh VLM checkpoint that hasn't been trained for embeddings yet. Sentence Transformers will attempt to recognize the architecture, infer the supported modalities from the processor, and set up the appropriate forward method and pooling. If the automatic detection doesn't work perfectly for a particular model, the configuration in the saved sentence_bert_config.json can be edited to adjust modality settings, forward methods, and output handling:
\n\n = \nIn both cases, the Transformer module inspects the processor to determine which modalities are available, and Pooling is added automatically if needed. You can verify the supported modalities:
\n# ['text', 'image', 'video', 'message']\n\n\n# True\nAlternative: Building multimodal models with Router
\nInstead of using a single VLM backbone, you can compose separate encoders for different modalities using the Router module. This lets you combine any existing encoders and route inputs to the appropriate one based on detected modality:
\n\n\n# Create separate encoders for different modalities\n = \n = \n = \n\n# SigLIP outputs pooled embeddings directly, so no separate Pooling module is needed\n = \n\n# Route inputs based on modality\n = \n\n = \n\n\nSince Router-based multimodal models use separate encoders per modality, their embedding spaces are initially unaligned. Training is required to align the spaces for meaningful cross-modal similarity. The
\nDenseprojection layer shown above helps map embeddings from different encoders into a shared space.
This approach is useful when you want to use lightweight, specialized encoders rather than a large VLM. You can also combine Router-based multimodality with task-based routing (e.g. different encoders for queries vs. documents) using route_mappings. See the Router documentation for advanced routing scenarios.
For this example, I use the tomaarsen/llamaindex-vdr-en-train-preprocessed dataset, a preprocessed English subset of llamaindex/vdr-multilingual-train. The source dataset was released alongside the Visual Document Retrieval Goes Multilingual blogpost by LlamaIndex, and consists of ~500k multilingual query-image samples collected from public internet PDFs, with queries synthetically generated using VLMs (gemini-1.5-pro and Qwen2-VL-72B). My preprocessed version filters to the 53,512 English samples and resolves 4 of the 16 ID-based hard negatives per sample into actual document screenshot images, so it can be used directly for training without further preprocessing:
\n\n = \n = \n = \nThe train config contains the first 10,000 samples, and the eval config contains the next 300 samples (a full config with all 53,512 samples is also available). For training, I select query, image, and negative_0 to form (anchor, positive, hard negative) triplets. Including additional hard negatives would likely improve the training signal, but each extra negative also increases memory usage and training time, so I stick with one. For evaluation, I keep all four hard negatives per query to build a more challenging retrieval corpus (more on that in the Evaluator section).
Just like text-only training, the dataset format must match your chosen loss function. The rules are the same:
\nFor multimodal datasets, the inputs can contain:
\n"array" and "sampling_rate" keys, or (if torchcodec is installed) torchcodec.AudioDecoder instances."array" and "video_metadata" keys, or (if torchcodec is installed) torchcodec.VideoDecoder instances.{"text": ..., "image": ...}. The keys must be "text", "image", "audio", or "video".The data collator automatically calls model.preprocess(), which detects the modality of each input and applies the appropriate preprocessing. No manual tokenization or image processing is needed.
\n\nMany Hugging Face datasets that work out of the box with Sentence Transformers have been tagged with
\nsentence-transformers, allowing you to easily find them at https://huggingface.co/datasets?other=sentence-transformers.
For this training, I use CachedMultipleNegativesRankingLoss, a common choice for retrieval tasks. It accepts (query, positive) pairs with any number of additional hard negative columns, from 0 up to n, as long as each sample has the same number of negatives. During training, the loss pushes each query's similarity to its positive up and its similarity to every negative down. The negatives come from two sources:
negative_0 in our triplet setup).More negatives per query means a stronger training signal, so a larger batch size directly improves training quality. Beyond that, the "cached" variant of the loss uses gradient caching to make large effective batch sizes feasible even when GPU memory is limited.
\nThe mini_batch_size parameter controls how many samples are processed at once during the cached forward passes. For large multimodal models, setting this to a small value (e.g., 1) is important to avoid out-of-memory errors without sacrificing the benefits of large effective batch sizes:
\n\n = \nTo produce embeddings that work well at multiple dimensionalities, I wrap the base loss with MatryoshkaLoss. This trains the model so that truncating the embedding to a smaller number of dimensions still yields good performance:
\n\n = \n = \nThis is especially useful for multimodal models, where embeddings can be large (2048 dimensions for Qwen3-VL). With Matryoshka training, you can use truncated embeddings (e.g., 256 or 128 dimensions) at deployment time for faster search with minimal quality loss. As I'll show in the Results section, the finetuned model achieves near-peak performance even at 512 dimensions.
\nThe SentenceTransformerTrainingArguments class lets you control training hyperparameters. Here's the configuration used for the VDR finetuning:
\n\n = \n = \nA few things to note for (multimodal) training:
\nbf16=True: bfloat16 is generally preferred over float16 due to better numerical stability.batch_sampler=BatchSamplers.NO_DUPLICATES: When using MultipleNegativesRankingLoss or its cached variant, having no duplicate samples in a batch ensures that every in-batch negative is a truly different sample.per_device_train_batch_size=64: This may seem large for a 2B parameter VLM, but CachedMultipleNegativesRankingLoss with mini_batch_size=1 handles the memory constraints through gradient caching.eval_steps, save_steps, and logging_steps: Setting these to a fraction (e.g., 0.1) means evaluation, saving, and logging will happen every 10% of an epoch, which is useful for monitoring training progress.To track retrieval performance before, during, and after training, I use the InformationRetrievalEvaluator. It computes standard retrieval metrics like NDCG@10, MAP, and Recall@k:
\n\n# Build the evaluation data from the eval dataset.\n# Queries and corpus use integer IDs: query 0's relevant document is corpus 0.\n = \n = \n = \n\n# Add hard negatives to the corpus with offset IDs (num_eval, 2*num_eval, ...)\n# so they don't collide with the positive document IDs (0..num_eval-1).\n = \n\n \n = \n\n# Each query's relevant document is the positive at the same index\n = \n\n = \nThe evaluator takes text queries, a corpus of images (including hard negatives), and a mapping of which documents are relevant to which queries. Note that the corpus contains a mix of positive and hard negative document screenshots, making this a challenging evaluation. Using batch_size=1 prevents out-of-memory issues during evaluation of the large VLM.
The SentenceTransformerTrainer brings everything together. Here's the complete training script:
\n\n\n\n\n\n\n\n\n# 1. Load a model to finetune with (optional) model card data\n = \n\n# 2. Load a dataset to finetune on: (query, positive, negative_0) triplets for training,\n# all 4 hard negatives retained for evaluation\n = \n = \n = \n\n# 3. Define a loss function\n = \n = \n\n# 4. (Optional) Specify training arguments\n = \n = \n\n# 5. (Optional) Create an evaluator & evaluate the base model\n = \n = \n = \n = \n\n \n = \n = \n\n = \n\n\n# 6. Create a trainer & train\n = \n\n\n# 7. (Optional) Evaluate at each Matryoshka dimension\n\n\n = \n \n\n# 8. Save the trained model\n\n\n# 9. (Optional) Push it to the Hugging Face Hub\n# This pushes to your personal namespace, e.g. {your_username}/Qwen3-VL-Embedding-2B-vdr\n\nThe training script is nearly identical to a text-only training script. The only differences are:
\nmodel_kwargs for precision and attention implementation, and processor_kwargs for image resolution bounds.CachedMultipleNegativesRankingLoss with mini_batch_size=1 to handle the large VLM without running out of memory.Everything else (the trainer, training arguments, dataset loading) works exactly the same as text-only training.
\nAfter training for just 1 epoch, the finetuned tomaarsen/Qwen3-VL-Embedding-2B-vdr model achieves an NDCG@10 of 0.947 on the evaluation set (300 queries, 1500 corpus documents, cosine similarity). This is a significant improvement over the base Qwen/Qwen3-VL-Embedding-2B model's 0.888, and outperforms all existing VDR models:
\n
Full NDCG@10 numbers by model (20 models)
\n| Model | \nParameters | \nNDCG@10 | \n
|---|---|---|
| tomaarsen/Qwen3-VL-Embedding-2B-vdr | \n2.1B | \n0.947 | \n
| Qwen/Qwen3-VL-Embedding-8B | \n8.1B | \n0.923 | \n
| nvidia/omni-embed-nemotron-3b | \n4.7B | \n0.915 | \n
| nvidia/llama-nemotron-embed-vl-1b-v2 | \n1.7B | \n0.912 | \n
| nomic-ai/nomic-embed-multimodal-7b | \n8.3B | \n0.912 | \n
| llamaindex/vdr-2b-multi-v1 | \n2.2B | \n0.912 | \n
| llamaindex/vdr-2b-v1 | \n2.2B | \n0.911 | \n
| nomic-ai/nomic-embed-multimodal-3b | \n3.8B | \n0.899 | \n
| Qwen/Qwen3-VL-Embedding-2B | \n2.1B | \n0.888 | \n
| LCO-Embedding/LCO-Embedding-Omni-7B | \n8.9B | \n0.888 | \n
| LCO-Embedding/LCO-Embedding-Omni-3B | \n4.7B | \n0.860 | \n
| BAAI/BGE-VL-v1.5-zs | \n7.6B | \n0.800 | \n
| BAAI/BGE-VL-v1.5-mmeb | \n7.6B | \n0.797 | \n
| BAAI/BGE-VL-MLLM-S2 | \n7.6B | \n0.792 | \n
| BidirLM/BidirLM-Omni-2.5B-Embedding | \n2.5B | \n0.775 | \n
| royokong/e5-v | \n8.4B | \n0.767 | \n
| BAAI/BGE-VL-MLLM-S1 | \n7.6B | \n0.710 | \n
| sentence-transformers/clip-ViT-L-14 | \n428M | \n0.611 | \n
| BAAI/BGE-VL-large | \n428M | \n0.467 | \n
| BAAI/BGE-VL-base | \n150M | \n0.335 | \n
The finetuned 2B model outperforms even the 8B Qwen3-VL-Embedding model, demonstrating the power of task-specific finetuning. Finetuning on your own domain is often worth considering, even when a larger general-purpose model is available!
\nThe comparison above uses full-size 2048-dim embeddings. Thanks to the Matryoshka training, the finetuned model also holds up well when truncated to fewer dimensions, letting you trade off embedding size and retrieval quality at deployment time:
\n![\"MRL](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/multimodal-sentence-transformers/vdr_plot_mrl.png\")
\n\nThe finetuned model's peak is at the full 2048 dimensions (0.948), but it stays within 0.3% of peak all the way down to 512 (4x smaller), and retains over 92% of peak even at 64 (32x smaller). Matryoshka training concentrates the most important information in the earlier dimensions, so moderate truncation costs very little performance.
\n
Full NDCG@10 numbers by dimension
\n| Dimensions | \nBase NDCG@10 | \nFinetuned NDCG@10 | \n
|---|---|---|
| 2048 (full) | \n0.8961 (100%) | \n0.9480 (100%) | \n
| 1536 | \n0.8940 (99.8%) | \n0.9439 (99.6%) | \n
| 1024 | \n0.8941 (99.8%) | \n0.9464 (99.8%) | \n
| 512 | \n0.8760 (97.8%) | \n0.9451 (99.7%) | \n
| 256 | \n0.8347 (93.2%) | \n0.9372 (98.9%) | \n
| 128 | \n0.7888 (88.0%) | \n0.9058 (95.5%) | \n
| 64 | \n0.6852 (76.5%) | \n0.8758 (92.4%) | \n
The gap between 1024 and 2048 dimensions is small (0.946 vs. 0.948), so I've saved the model with truncate_dim=1024 set in its configuration. This means that SentenceTransformer("tomaarsen/Qwen3-VL-Embedding-2B-vdr") produces 1024-dimensional embeddings by default, halving the storage footprint compared to the full 2048. If you want a different dimensionality, pass truncate_dim=N when loading to override it.
You can also finetune multimodal Cross Encoder (reranker) models using the same training infrastructure. The key difference is using CrossEncoderTrainer and Cross Encoder-specific loss functions. This section provides a brief overview; see the full training examples for complete, runnable scripts with dataset preparation and evaluation.
Here's a simplified example based on the doodles training script, which trains a reranker to match images with text captions:
\n\n\n\n\n\n\n# 1. Build the model from modules\n = \n\n# Extend chat template to support "query" and "document" roles\n = \n\n# LogitScore: score = log(P("1")) - log(P("0"))\n = \n\n = \n\n# 2. Define the loss\n = \n\n# 3. Multi-dataset training with separate directions\n = \n\nThere are multiple valid architectural choices for multimodal rerankers, including:
\nBoth approaches are demonstrated in the multimodal cross encoder training examples.
\nThe two scripts linked above split the training data into two datasets, one per direction (image-to-text and text-to-image), with a task-specific prompt for each that tells the model how to score in that direction. Each positive pair is then expanded with randomly sampled negatives so the loss sees a balanced mix of matches and non-matches.
\nThe Sentence Transformers repository includes several multimodal training examples:
\nAdditionally, the following pages may be useful to learn more about training with Sentence Transformers:
\nThe direct prerequisite and the prior training guides are listed first, followed by related technique posts that stack with multimodal training:
\nMatryoshkaLoss used in this post to produce embeddings that truncate gracefully.