{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-04-09-multimodal-embedding-reranker-models-with-sentence/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "[Sentence Transformers](https://sbert.net/) is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more. With the v5.4 update, you can now **encode and compare texts, images, audio, and videos** using the same familiar API. In this blogpost, I'll show you how to use these new multimodal capabilities for both embedding and reranking.\n\nMultimodal embedding models map inputs from different modalities into a shared embedding space, while multimodal reranker models score the relevance of mixed-modality pairs. This opens up use cases like visual document retrieval, cross-modal search, and multimodal RAG pipelines.\n\n> If you want to train your own multimodal models, check out the companion blogpost: [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers).\n\n## Table of Contents\n\n- [What are Multimodal Models?](https://huggingface.co/blog/multimodal-sentence-transformers#what-are-multimodal-models)\n- [Installation](https://huggingface.co/blog/multimodal-sentence-transformers#installation)\n- [Multimodal Embedding Models](https://huggingface.co/blog/multimodal-sentence-transformers#multimodal-embedding-models)\n - [Loading a Model](https://huggingface.co/blog/multimodal-sentence-transformers#loading-a-model)\n - [Encoding Images](https://huggingface.co/blog/multimodal-sentence-transformers#encoding-images)\n - [Cross-Modal Similarity](https://huggingface.co/blog/multimodal-sentence-transformers#cross-modal-similarity)\n - [Encoding Queries and Documents](https://huggingface.co/blog/multimodal-sentence-transformers#encoding-queries-and-documents)\n- [Multimodal Reranker Models](https://huggingface.co/blog/multimodal-sentence-transformers#multimodal-reranker-models)\n - [Ranking Mixed-Modality Documents](https://huggingface.co/blog/multimodal-sentence-transformers#ranking-mixed-modality-documents)\n - [Predicting Pair Scores](https://huggingface.co/blog/multimodal-sentence-transformers#predicting-pair-scores)\n- [Retrieve and Rerank](https://huggingface.co/blog/multimodal-sentence-transformers#retrieve-and-rerank)\n- [Input Formats and Configuration](https://huggingface.co/blog/multimodal-sentence-transformers#input-formats-and-configuration)\n - [Supported Input Types](https://huggingface.co/blog/multimodal-sentence-transformers#supported-input-types)\n - [Checking Modality Support](https://huggingface.co/blog/multimodal-sentence-transformers#checking-modality-support)\n - [Processor and Model kwargs](https://huggingface.co/blog/multimodal-sentence-transformers#processor-and-model-kwargs)\n- [Supported Models](https://huggingface.co/blog/multimodal-sentence-transformers#supported-models)\n- [Additional Resources](https://huggingface.co/blog/multimodal-sentence-transformers#additional-resources)\n\n## What are Multimodal Models?\n\nTraditional embedding models convert text into [fixed-size vectors](https://huggingface.co/blog/matryoshka#understanding-embeddings). Multimodal embedding models extend this by mapping inputs from different modalities (text, images, audio, or video) into a shared embedding space. This means you can compare a text query against image documents (or vice versa) using the same similarity functions you're already familiar with.\n\nSimilarly, traditional reranker (Cross Encoder) models compute relevance scores between pairs of texts. Multimodal rerankers can score pairs where one or both elements are images, combined text-image documents, or other modalities.\n\nFor example, you can compare a text query against image documents, find video clips matching a description, or build RAG pipelines that work across modalities.\n\n## Installation\n\nMultimodal models require some extra dependencies. Install the extras for the modalities you need (see [Installation](https://sbert.net/docs/installation.html) for more details):\n\n```bash\n# For image support\npip install -U \"sentence-transformers[image]\"\n\n# For audio support\npip install -U \"sentence-transformers[audio]\"\n\n# For video support\npip install -U \"sentence-transformers[video]\"\n\n# Mix and match as needed\npip install -U \"sentence-transformers[image,video,train]\"\n```\n\n> VLM-based models like Qwen3-VL-2B require a GPU with at least ~8 GB of VRAM. For the 8B variants, expect ~20 GB. If you don't have a local GPU, consider using a cloud GPU service or Google Colab. On CPU, these models will be extremely slow; text-only or CLIP models are better suited for CPU inference.\n\n## Multimodal Embedding Models\n\n### Loading a Model\n\nLoading a multimodal embedding model works exactly like loading a text-only model:\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n```\n\n> Some models might require a `revision` argument for now if the integration pull requests for the model is still pending. Once they're merged, you'll be able to load them without specifying a revision, like above.\n\nThe model automatically detects which modalities it supports, so there's nothing extra to configure. See [Processor and Model kwargs](https://huggingface.co/blog/multimodal-sentence-transformers#processor-and-model-kwargs) if you want to control things like image resolution or model precision.\n\n### Encoding Images\n\nWith a multimodal model loaded, [`model.encode()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode) accepts images alongside text. Images can be provided as URLs, local file paths, or PIL Image objects (see [Supported Input Types](https://huggingface.co/blog/multimodal-sentence-transformers#supported-input-types) for all accepted formats):\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n\n# Encode images from URLs\nimg_embeddings = model.encode([\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\",\n])\nprint(img_embeddings.shape)\n# (2, 2048)\n```\n\n### Cross-Modal Similarity\n\nYou can compute similarities between text embeddings and image embeddings, since the model maps both into the same space:\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n\n# Encode images\nimg_embeddings = model.encode([\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\",\n])\n\n# Encode text queries (one matching + one hard negative per image)\ntext_embeddings = model.encode([\n \"A green car parked in front of a yellow building\",\n \"A red car driving on a highway\",\n \"A bee on a pink flower\",\n \"A wasp on a wooden table\",\n])\n\n# Compute cross-modal similarities\nsimilarities = model.similarity(text_embeddings, img_embeddings)\nprint(similarities)\n# tensor([[0.5115, 0.1078],\n# [0.1999, 0.1108],\n# [0.1255, 0.6749],\n# [0.1283, 0.2704]])\n```\n\nAs expected, \"A green car parked in front of a yellow building\" is most similar to the car image (0.51), and \"A bee on a pink flower\" is most similar to the bee image (0.67). The hard negatives (\"A red car driving on a highway\", \"A wasp on a wooden table\") correctly receive lower scores.\n\nYou might notice that even the best matching scores (0.51, 0.67) aren't very close to 1.0. This is due to the [modality gap](https://arxiv.org/abs/2203.02053): embeddings from different modalities tend to cluster in separate regions of the space. Cross-modal similarities are typically lower than within-modal ones (e.g., text-to-text), but the relative ordering is preserved, so retrieval still works well.\n\n### Encoding Queries and Documents\n\nFor retrieval tasks, [`encode_query()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode_query) and [`encode_document()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode_document) are the recommended methods. Many retrieval models prepend different instruction prompts depending on whether the input is a query or a document, similar to how chat models might apply different system prompts depending on the goal. Model authors can specify their prompts in the model config, and `encode_query()` / `encode_document()` automatically load and apply the correct one:\n\n- `encode_query()` uses the model's `\"query\"` prompt (if available) and sets `task=\"query\"`.\n- `encode_document()` uses the first available prompt from `\"document\"`, `\"passage\"`, or `\"corpus\"`, and sets `task=\"document\"`.\n\nUnder the hood, both are thin wrappers around `encode()`, they just handle prompt selection for you. Here's what cross-modal retrieval looks like:\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n\n# Encode text queries with the query prompt\nquery_embeddings = model.encode_query([\n \"Find me a photo of a vehicle parked near a building\",\n \"Show me an image of a pollinating insect\",\n])\n\n# Encode document screenshots with the document prompt\ndoc_embeddings = model.encode_document([\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\",\n])\n\n# Compute similarities\nsimilarities = model.similarity(query_embeddings, doc_embeddings)\nprint(similarities)\n# tensor([[0.3907, 0.1490],\n# [0.1235, 0.4872]])\n```\n\nThese methods accept the same input types as `encode()` (images, URLs, multimodal dicts, etc.) and pass through the same parameters. For models without specialized query/document prompts, they behave identically to `encode()`.\n\n## Multimodal Reranker Models\n\nMultimodal reranker (CrossEncoder) models score the relevance between pairs of inputs, where each element can be text, an image, audio, video, or a combination. They tend to outperform embedding models in terms of quality, but are slower since they process each pair individually. The currently available pretrained multimodal rerankers focus on text and image inputs, but the architecture supports any modality that the underlying model can handle.\n\n### Ranking Mixed-Modality Documents\n\nThe [`rank()`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html#sentence_transformers.cross_encoder.CrossEncoder.rank) method scores and ranks a list of documents against a query, supporting mixed modalities:\n\n```python\nfrom sentence_transformers import CrossEncoder\n\nmodel = CrossEncoder(\"Qwen/Qwen3-VL-Reranker-2B\")\n\nquery = \"A green car parked in front of a yellow building\"\ndocuments = [\n # Image documents (URL or local file path)\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\",\n # Text document\n \"A vintage Volkswagen Beetle painted in bright green sits in a driveway.\",\n # Combined text + image document\n {\n \"text\": \"A car in a European city\",\n \"image\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n },\n]\n\nrankings = model.rank(query, documents)\nfor rank in rankings:\n print(f\"{rank['score']:.4f}\\t(document {rank['corpus_id']})\")\n\"\"\"\n0.9375 (document 0)\n0.5000 (document 3)\n-1.2500 (document 2)\n-2.4375 (document 1)\n\"\"\"\n```\n\nThe reranker correctly identifies the car image (document 0) as the most relevant result, followed by the combined text+image document about a car in a European city (document 3). The bee image (document 1) scores lowest. Keep in mind that the [modality gap](https://arxiv.org/abs/2203.02053) can influence absolute scores: text-image pair scores may occupy a different range than text-text or image-image pair scores.\n\nYou can also check which modalities a reranker supports using [`modalities`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html#sentence_transformers.cross_encoder.CrossEncoder.modalities) and [`supports()`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html#sentence_transformers.cross_encoder.CrossEncoder.supports), just like with embedding models:\n\n```python\nprint(model.modalities)\n# ['text', 'image', 'video', 'message']\n\nprint(model.supports(\"image\"))\n# True\n\n# Check if the model supports a specific pair of modalities\nprint(model.supports((\"image\", \"text\")))\n# True\n```\n\n### Predicting Pair Scores\n\nYou can also use [`predict()`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html#sentence_transformers.cross_encoder.CrossEncoder.predict) to get raw relevance scores for specific pairs of inputs:\n\n```python\nfrom sentence_transformers import CrossEncoder\n\nmodel = CrossEncoder(\"jinaai/jina-reranker-m0\", trust_remote_code=True)\n\nscores = model.predict([\n (\"A green car\", \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\"),\n (\"A bee on a flower\", \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\"),\n (\"A green car\", \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\"),\n])\nprint(scores)\n# [0.9389156 0.96922314 0.46063158]\n```\n\n## Retrieve and Rerank\n\nA common pattern is to use an embedding model for fast initial retrieval, then refine the top results with a reranker:\n\n```python\nfrom sentence_transformers import SentenceTransformer, CrossEncoder\n\n# Step 1: Retrieve with an embedding model\nembedder = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n\nquery = \"revenue growth chart\"\nquery_embedding = embedder.encode_query(query)\n\n# Pre-compute corpus embeddings (do this once, then store them)\ndocument_screenshots = [\n \"path/to/doc1.png\",\n \"path/to/doc2.png\",\n # ... potentially millions of document screenshots\n]\ncorpus_embeddings = embedder.encode_document(document_screenshots, show_progress_bar=True)\n\n# Simple cosine similarity retrieval, viable as long as embeddings fit in memory\nsimilarities = embedder.similarity(query_embedding, corpus_embeddings)\ntop_k_indices = similarities.argsort(descending=True)[0][:10]\n\n# Step 2: Rerank the top-k results with a reranker model\nreranker = CrossEncoder(\"nvidia/llama-nemotron-rerank-vl-1b-v2\", trust_remote_code=True)\n\ntop_k_documents = [document_screenshots[i] for i in top_k_indices]\nrankings = reranker.rank(query, top_k_documents)\nfor rank in rankings:\n print(f\"{rank['score']:.4f}\\t{top_k_documents[rank['corpus_id']]}\")\n```\n\nSince the corpus embeddings are pre-computed, the initial retrieval is fast even over millions of documents. The reranker then provides more accurate scoring over the smaller candidate set.\n\n## Input Formats and Configuration\n\n### Supported Input Types\n\nMultimodal models accept a variety of input formats. Here's a summary of what you can pass to [`model.encode()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode):\n\n| Modality | Accepted Formats |\n| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| **Text** | \\- Strings |\n| **Image** | \\- `PIL.Image.Image` objects - File paths (e.g. `\"./photo.jpg\"`) - URLs (e.g. `\"https://.../image.jpg\"`) - Numpy arrays, torch tensors |\n| **Audio** | \\- File paths (e.g. `\"./audio.wav\"`) - URLs (e.g. `\"https://.../audio.wav\"`) - Numpy/torch arrays - Dicts with `\"array\"` and `\"sampling_rate\"` keys - `torchcodec.AudioDecoder` instances |\n| **Video** | \\- File paths (e.g. `\"./video.mp4\"`) - URLs (e.g. `\"https://.../video.mp4\"`) - Numpy/torch arrays - Dicts with `\"array\"` and `\"video_metadata\"` keys - `torchcodec.VideoDecoder` instances |\n| **Multimodal** | \\- Dicts mapping modality names to values, e.g. `{\"text\": \"a caption\", \"image\": \"https://.../image.jpg\"}` Valid keys: `\"text\"`, `\"image\"`, `\"audio\"`, `\"video\"` |\n| **Message** | \\- List of message dicts with `\"role\"` and `\"content\"` keys, e.g. `[{\"role\": \"user\", \"content\": [...]}]` |\n\n### Checking Modality Support\n\nYou can check which modalities a model supports using the [`modalities`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.modalities) property and [`supports()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.supports) method:\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n\n# List all supported modalities\nprint(model.modalities)\n# ['text', 'image', 'video', 'message']\n\n# Check for a specific modality\nprint(model.supports(\"image\"))\n# True\nprint(model.supports(\"audio\"))\n# False\n```\n\nThe `\"message\"` modality indicates that the model accepts chat-style message inputs with interleaved content. In practice, you rarely need to use this directly. When you pass strings, URLs, or multimodal dicts, the model converts them to the appropriate message format internally. Sentence Transformers supports two message formats:\n\n- **Structured** (most VLMs, e.g. Qwen3-VL): Content is a list of typed dicts, e.g. `[{\"type\": \"text\", \"text\": \"...\"}, {\"type\": \"image\", \"image\": ...}]`\n- **Flat** (e.g. Deepseek-V3): Content is a direct value, e.g. `\"some text\"`\n\nThe format is auto-detected from the model's chat template.\n\nSince all inputs get converted into the same message format internally, you can mix input types in a single [`encode()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode) call:\n\n```python\nembeddings = model.encode([\n # A text input\n \"A green car parked in front of a yellow building\",\n # An image input (URL)\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n # A combined text + image input\n {\n \"text\": \"A car in a European city\",\n \"image\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n },\n])\n```\n\nClick here if you need to pass raw message inputs\n\nIf a model doesn't follow either format and you need full control, you can pass raw message dicts with `role` and `content` keys directly:\n\n```python\nembeddings = model.encode([\n [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\"},\n {\"type\": \"text\", \"text\": \"Describe this vehicle.\"},\n ],\n }\n ],\n])\n```\n\nThis bypasses the automatic format conversion and passes the messages directly to the processor's `apply_chat_template()`.\n\n### Processor and Model kwargs\n\nYou may want to control image resolution bounds or model precision. Use `processor_kwargs` and `model_kwargs` when loading the model:\n\n```python\nmodel = SentenceTransformer(\n \"Qwen/Qwen3-VL-Embedding-2B\",\n model_kwargs={\"attn_implementation\": \"flash_attention_2\", \"torch_dtype\": \"bfloat16\"},\n processor_kwargs={\"min_pixels\": 28 * 28, \"max_pixels\": 600 * 600},\n)\n```\n\n- `processor_kwargs` controls how inputs are preprocessed (e.g., image resolution bounds). Higher `max_pixels` means higher quality but more memory and compute. These are passed directly to `AutoProcessor.from_pretrained(...)`.\n- `model_kwargs` controls how the underlying model is loaded (e.g., precision, attention implementation). These are passed directly to the appropriate `AutoModel.from_pretrained(...)` call (e.g., `AutoModel`, `AutoModelForCausalLM`, `AutoModelForSequenceClassification`, etc., depending on the configuration of the model modules).\n\nSee the [SentenceTransformer](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer) API Reference documentation for more details on these kwargs.\n\n> In Sentence Transformers v5.4, `tokenizer_kwargs` has been renamed to `processor_kwargs` to reflect that multimodal models use processors rather than just tokenizers. The old name is still accepted but deprecated.\n\n## Supported Models\n\nHere are the multimodal models supported in v5.4, also available in the [v5.4 integrations collection](https://huggingface.co/collections/sentence-transformers/sentence-transformers-v54-integrations):\n\n### Supported Multimodal Embedding Models\n\n| Model | Parameters | Modalities | Revision |\n| ---------------------------------------------------------------------------------------------------- | ---------- | ------------------------- | ---------------------- |\n| [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) | 2B | Text, Image, Video | No `revision` needed |\n| [Qwen/Qwen3-VL-Embedding-8B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B) | 8B | Text, Image, Video | No `revision` needed |\n| [nvidia/llama-nemotron-embed-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-embed-vl-1b-v2) | 1.7B | Text, Image | No `revision` needed |\n| [nvidia/omni-embed-nemotron-3b](https://huggingface.co/nvidia/omni-embed-nemotron-3b) | 4.7B | Text, Image | No `revision` needed |\n| [LCO-Embedding/LCO-Embedding-Omni-3B](https://huggingface.co/LCO-Embedding/LCO-Embedding-Omni-3B) | 5B | Text, Image, Audio, Video | No `revision` needed |\n| [LCO-Embedding/LCO-Embedding-Omni-7B](https://huggingface.co/LCO-Embedding/LCO-Embedding-Omni-7B) | 9B | Text, Image, Audio, Video | No `revision` needed |\n| [BidirLM/BidirLM-Omni-2.5B-Embedding](https://huggingface.co/BidirLM/BidirLM-Omni-2.5B-Embedding) | 2.5B | Text, Image, Audio | No `revision` needed |\n| [BAAI/BGE-VL-base](https://huggingface.co/BAAI/BGE-VL-base) | 0.1B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-large](https://huggingface.co/BAAI/BGE-VL-large) | 0.4B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-MLLM-S1](https://huggingface.co/BAAI/BGE-VL-MLLM-S1) | 8B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-MLLM-S2](https://huggingface.co/BAAI/BGE-VL-MLLM-S2) | 8B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-v1.5-zs](https://huggingface.co/BAAI/BGE-VL-v1.5-zs) | 8B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-v1.5-mmeb](https://huggingface.co/BAAI/BGE-VL-v1.5-mmeb) | 8B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-Screenshot](https://huggingface.co/BAAI/BGE-VL-Screenshot) | 4B | Text, Image | No `revision` needed |\n| [royokong/e5-v](https://huggingface.co/royokong/e5-v) | 8B | Text, Image | No `revision` needed |\n| [eagerworks/eager-embed-v1](https://huggingface.co/eagerworks/eager-embed-v1) | 4B | Text, Image | `revision=\"refs/pr/2\"` |\n| [nomic-ai/nomic-embed-multimodal-3b](https://huggingface.co/nomic-ai/nomic-embed-multimodal-3b) | 5B | Text, Image | `revision=\"refs/pr/4\"` |\n| [nomic-ai/nomic-embed-multimodal-7b](https://huggingface.co/nomic-ai/nomic-embed-multimodal-7b) | 9B | Text, Image | `revision=\"refs/pr/3\"` |\n| [Haon-Chen/e5-omni-3B](https://huggingface.co/Haon-Chen/e5-omni-3B) | 5B | Text, Image, Audio, Video | `revision=\"refs/pr/2\"` |\n| [Haon-Chen/e5-omni-7B](https://huggingface.co/Haon-Chen/e5-omni-7B) | 9B | Text, Image, Audio, Video | `revision=\"refs/pr/1\"` |\n\n### Supported Multimodal Reranker Models\n\n| Model | Parameters | Modalities | Revision |\n| ------------------------------------------------------------------------------------------------------ | ---------- | ------------------ | -------------------- |\n| [Qwen/Qwen3-VL-Reranker-2B](https://huggingface.co/Qwen/Qwen3-VL-Reranker-2B) | 2B | Text, Image, Video | No `revision` needed |\n| [Qwen/Qwen3-VL-Reranker-8B](https://huggingface.co/Qwen/Qwen3-VL-Reranker-8B) | 8B | Text, Image, Video | No `revision` needed |\n| [nvidia/llama-nemotron-rerank-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2) | 2B | Text, Image | No `revision` needed |\n| [jinaai/jina-reranker-m0](https://huggingface.co/jinaai/jina-reranker-m0) | 2B | Text, Image | No `revision` needed |\n\n### Text-Only Reranker Models (also new in v5.4)\n\n| Model | Parameters | Revision |\n| ------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---------------------- |\n| [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) | 0.6B | No `revision` needed |\n| [Qwen/Qwen3-Reranker-4B](https://huggingface.co/Qwen/Qwen3-Reranker-4B) | 4B | No `revision` needed |\n| [Qwen/Qwen3-Reranker-8B](https://huggingface.co/Qwen/Qwen3-Reranker-8B) | 8B | No `revision` needed |\n| [mixedbread-ai/mxbai-rerank-base-v2](https://huggingface.co/mixedbread-ai/mxbai-rerank-base-v2) | 0.5B | No `revision` needed |\n| [mixedbread-ai/mxbai-rerank-large-v2](https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v2) | 2B | No `revision` needed |\n| [ContextualAI/ctxl-rerank-v2-instruct-multilingual-1b](https://huggingface.co/ContextualAI/ctxl-rerank-v2-instruct-multilingual-1b) | 1B | `revision=\"refs/pr/2\"` |\n| [ContextualAI/ctxl-rerank-v2-instruct-multilingual-2b](https://huggingface.co/ContextualAI/ctxl-rerank-v2-instruct-multilingual-2b) | 3B | `revision=\"refs/pr/1\"` |\n| [ContextualAI/ctxl-rerank-v2-instruct-multilingual-6b](https://huggingface.co/ContextualAI/ctxl-rerank-v2-instruct-multilingual-6b) | 7B | `revision=\"refs/pr/1\"` |\n\nClick here for a text-only reranker usage example\n\n```python\nfrom sentence_transformers import CrossEncoder\n\nmodel = CrossEncoder(\"mixedbread-ai/mxbai-rerank-base-v2\")\n\nquery = \"How do I bake sourdough bread?\"\ndocuments = [\n \"Sourdough bread requires a starter made from flour and water, fermented over several days.\",\n \"The history of bread dates back to ancient Egypt around 8000 BCE.\",\n \"To bake sourdough, mix your starter with flour, water, and salt, then let it rise overnight.\",\n \"Rye bread is a popular alternative to wheat-based breads in Northern Europe.\",\n]\n\npairs = [(query, doc) for doc in documents]\nscores = model.predict(pairs)\nprint(scores)\n# [ 7.3077507 -2.6217823 8.724761 -2.2488995]\n\nrankings = model.rank(query, documents)\nfor rank in rankings:\n print(f\"{rank['score']:.4f}\\t{documents[rank['corpus_id']]}\")\n# 8.7248 To bake sourdough, mix your starter with flour, water, and salt, then let it rise overnight.\n# 7.3078 Sourdough bread requires a starter made from flour and water, fermented over several days.\n# -2.2489 Rye bread is a popular alternative to wheat-based breads in Northern Europe.\n# -2.6218 The history of bread dates back to ancient Egypt around 8000 BCE.\n```\n\n### CLIP Models\n\nThe older CLIP models continue to be supported:\n\n| Model | ImageNet Zero-Shot Top-1 Accuracy | Notes |\n| ---------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ---------------------------------------- |\n| [sentence-transformers/clip-ViT-L-14](https://huggingface.co/sentence-transformers/clip-ViT-L-14) | 75.4 | |\n| [sentence-transformers/clip-ViT-B-16](https://huggingface.co/sentence-transformers/clip-ViT-B-16) | 68.1 | |\n| [sentence-transformers/clip-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) | 63.3 | |\n| [sentence-transformers/clip-ViT-B-32-multilingual-v1](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1) | N/A | Multilingual text encoder, 50+ languages |\n\nThese simple CLIP models still work well on lower-resource hardware.\n\nClick here for a CLIP usage example\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"sentence-transformers/clip-ViT-L-14\")\n\nimages = [\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\",\n \"https://huggingface.co/datasets/huggingface/cats-image/resolve/main/cats_image.jpeg\"\n]\ntexts = [\"A green car\", \"A bee on a flower\", \"Some cats on a couch\", \"One cat sitting in the window\"]\n\nimage_embeddings = model.encode(images)\ntext_embeddings = model.encode(texts)\nprint(image_embeddings.shape, text_embeddings.shape)\n# (3, 768) (4, 768)\n\nsimilarities = model.similarity(image_embeddings, text_embeddings)\nprint(similarities)\n# tensor([[0.2208, 0.1042, 0.0617, 0.0907], First image (car) is most similar to \"A green car\"\n# [0.1205, 0.2303, 0.0632, 0.0917], Second image (bee) is most similar to \"A bee on a flower\"\n# [0.1107, 0.0196, 0.2425, 0.1162]]) Third image (multiple cats) is most similar to \"Some cats on a couch\"\n```\n\n## Additional Resources\n\n### Documentation\n\n- [Sentence Transformer > Usage](https://sbert.net/docs/sentence_transformer/usage/usage.html)\n- [Sentence Transformer > Pretrained Models](https://sbert.net/docs/sentence_transformer/pretrained_models.html)\n- [Cross Encoder > Usage](https://sbert.net/docs/cross_encoder/usage/usage.html)\n- [Cross Encoder > Pretrained Models](https://sbert.net/docs/cross_encoder/pretrained_models.html)\n- [Installation](https://sbert.net/docs/installation.html)\n\n### Training\n\nTo learn how to finetune these multimodal models on your own data, see the companion blogpost: [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers).\n\n- [Sentence Transformer > Training Overview](https://sbert.net/docs/sentence_transformer/training_overview.html)\n- [Sentence Transformer > Training Examples](https://sbert.net/docs/sentence_transformer/training/examples.html)\n- [Cross Encoder > Training Overview](https://sbert.net/docs/cross_encoder/training_overview.html)\n- [Cross Encoder > Training Examples](https://sbert.net/docs/cross_encoder/training/examples.html)\n- [Sparse Encoder > Training Overview](https://sbert.net/docs/sparse_encoder/training_overview.html)\n- [Sparse Encoder > Training Examples](https://sbert.net/docs/sparse_encoder/training/examples.html)\n\n### Hugging Face Hub\n\n- [Sentence Transformers models on the Hub](https://huggingface.co/models?library=sentence-transformers)\n- [Sentence Transformers datasets on the Hub](https://huggingface.co/datasets?other=sentence-transformers)\n- [v5.4 Integrations Collection](https://huggingface.co/collections/sentence-transformers/sentence-transformers-v54-integrations)\n\n### Companion Blogposts\n\nThe training companion to this post and adjacent Sentence Transformers guides:\n\n- [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers): the direct training companion to this post, with a Visual Document Retrieval walkthrough.\n- [Training and Finetuning Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sentence-transformers): the general training guide for text-only bi-encoder embedding models.\n- [Training and Finetuning Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-reranker): Cross Encoder (reranker) training, applicable to text-only and multimodal rerankers.\n- [Training and Finetuning Sparse Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sparse-encoder): SPLADE training for sparse retrieval.\n- [🪆 Introduction to Matryoshka Embedding Models](https://huggingface.co/blog/matryoshka): variable-size embeddings; also applied to multimodal models in the training companion post.\n- [Train 400x faster Static Embedding Models with Sentence Transformers](https://huggingface.co/blog/static-embeddings): CPU-friendly text embedding models.\n- [Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval](https://huggingface.co/blog/embedding-quantization): post-training compression of embedding vectors.\n", "authors": [], "captured_at": "2026-10-04T12:50:30.804479567+00:00", "category": "ai", "content_html": "

Sentence Transformers is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more. With the v5.4 update, you can now encode and compare texts, images, audio, and videos using the same familiar API. In this blogpost, I'll show you how to use these new multimodal capabilities for both embedding and reranking.

\n

Multimodal embedding models map inputs from different modalities into a shared embedding space, while multimodal reranker models score the relevance of mixed-modality pairs. This opens up use cases like visual document retrieval, cross-modal search, and multimodal RAG pipelines.

\n
\n

If you want to train your own multimodal models, check out the companion blogpost: Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers.

\n
\n

Table of Contents

\n\n

What are Multimodal Models?

\n

Traditional embedding models convert text into fixed-size vectors. Multimodal embedding models extend this by mapping inputs from different modalities (text, images, audio, or video) into a shared embedding space. This means you can compare a text query against image documents (or vice versa) using the same similarity functions you're already familiar with.

\n

Similarly, traditional reranker (Cross Encoder) models compute relevance scores between pairs of texts. Multimodal rerankers can score pairs where one or both elements are images, combined text-image documents, or other modalities.

\n

For example, you can compare a text query against image documents, find video clips matching a description, or build RAG pipelines that work across modalities.

\n

Installation

\n

Multimodal models require some extra dependencies. Install the extras for the modalities you need (see Installation for more details):

\n
# For image support\npip install -U "sentence-transformers[image]"\n\n# For audio support\npip install -U "sentence-transformers[audio]"\n\n# For video support\npip install -U "sentence-transformers[video]"\n\n# Mix and match as needed\npip install -U "sentence-transformers[image,video,train]"\n
\n
\n

VLM-based models like Qwen3-VL-2B require a GPU with at least ~8 GB of VRAM. For the 8B variants, expect ~20 GB. If you don't have a local GPU, consider using a cloud GPU service or Google Colab. On CPU, these models will be extremely slow; text-only or CLIP models are better suited for CPU inference.

\n
\n

Multimodal Embedding Models

\n

Loading a Model

\n

Loading a multimodal embedding model works exactly like loading a text-only model:

\n
from sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")\n
\n
\n

Some models might require a revision argument for now if the integration pull requests for the model is still pending. Once they're merged, you'll be able to load them without specifying a revision, like above.

\n
\n

The model automatically detects which modalities it supports, so there's nothing extra to configure. See Processor and Model kwargs if you want to control things like image resolution or model precision.

\n

Encoding Images

\n

With a multimodal model loaded, model.encode() accepts images alongside text. Images can be provided as URLs, local file paths, or PIL Image objects (see Supported Input Types for all accepted formats):

\n
from sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")\n\n# Encode images from URLs\nimg_embeddings = model.encode([\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",\n])\nprint(img_embeddings.shape)\n# (2, 2048)\n
\n

Cross-Modal Similarity

\n

You can compute similarities between text embeddings and image embeddings, since the model maps both into the same space:

\n
from sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")\n\n# Encode images\nimg_embeddings = model.encode([\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",\n])\n\n# Encode text queries (one matching + one hard negative per image)\ntext_embeddings = model.encode([\n    "A green car parked in front of a yellow building",\n    "A red car driving on a highway",\n    "A bee on a pink flower",\n    "A wasp on a wooden table",\n])\n\n# Compute cross-modal similarities\nsimilarities = model.similarity(text_embeddings, img_embeddings)\nprint(similarities)\n# tensor([[0.5115, 0.1078],\n#         [0.1999, 0.1108],\n#         [0.1255, 0.6749],\n#         [0.1283, 0.2704]])\n
\n

As expected, "A green car parked in front of a yellow building" is most similar to the car image (0.51), and "A bee on a pink flower" is most similar to the bee image (0.67). The hard negatives ("A red car driving on a highway", "A wasp on a wooden table") correctly receive lower scores.

\n

You might notice that even the best matching scores (0.51, 0.67) aren't very close to 1.0. This is due to the modality gap: embeddings from different modalities tend to cluster in separate regions of the space. Cross-modal similarities are typically lower than within-modal ones (e.g., text-to-text), but the relative ordering is preserved, so retrieval still works well.

\n

Encoding Queries and Documents

\n

For retrieval tasks, encode_query() and encode_document() are the recommended methods. Many retrieval models prepend different instruction prompts depending on whether the input is a query or a document, similar to how chat models might apply different system prompts depending on the goal. Model authors can specify their prompts in the model config, and encode_query() / encode_document() automatically load and apply the correct one:

\n\n

Under the hood, both are thin wrappers around encode(), they just handle prompt selection for you. Here's what cross-modal retrieval looks like:

\n
from sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")\n\n# Encode text queries with the query prompt\nquery_embeddings = model.encode_query([\n    "Find me a photo of a vehicle parked near a building",\n    "Show me an image of a pollinating insect",\n])\n\n# Encode document screenshots with the document prompt\ndoc_embeddings = model.encode_document([\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",\n])\n\n# Compute similarities\nsimilarities = model.similarity(query_embeddings, doc_embeddings)\nprint(similarities)\n# tensor([[0.3907, 0.1490],\n#         [0.1235, 0.4872]])\n
\n

These methods accept the same input types as encode() (images, URLs, multimodal dicts, etc.) and pass through the same parameters. For models without specialized query/document prompts, they behave identically to encode().

\n

Multimodal Reranker Models

\n

Multimodal reranker (CrossEncoder) models score the relevance between pairs of inputs, where each element can be text, an image, audio, video, or a combination. They tend to outperform embedding models in terms of quality, but are slower since they process each pair individually. The currently available pretrained multimodal rerankers focus on text and image inputs, but the architecture supports any modality that the underlying model can handle.

\n

Ranking Mixed-Modality Documents

\n

The rank() method scores and ranks a list of documents against a query, supporting mixed modalities:

\n
from sentence_transformers import CrossEncoder\n\nmodel = CrossEncoder("Qwen/Qwen3-VL-Reranker-2B")\n\nquery = "A green car parked in front of a yellow building"\ndocuments = [\n    # Image documents (URL or local file path)\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",\n    # Text document\n    "A vintage Volkswagen Beetle painted in bright green sits in a driveway.",\n    # Combined text + image document\n    {\n        "text": "A car in a European city",\n        "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",\n    },\n]\n\nrankings = model.rank(query, documents)\nfor rank in rankings:\n    print(f"{rank['score']:.4f}\\t(document {rank['corpus_id']})")\n"""\n0.9375  (document 0)\n0.5000  (document 3)\n-1.2500 (document 2)\n-2.4375 (document 1)\n"""\n
\n

The reranker correctly identifies the car image (document 0) as the most relevant result, followed by the combined text+image document about a car in a European city (document 3). The bee image (document 1) scores lowest. Keep in mind that the modality gap can influence absolute scores: text-image pair scores may occupy a different range than text-text or image-image pair scores.

\n

You can also check which modalities a reranker supports using modalities and supports(), just like with embedding models:

\n
print(model.modalities)\n# ['text', 'image', 'video', 'message']\n\nprint(model.supports("image"))\n# True\n\n# Check if the model supports a specific pair of modalities\nprint(model.supports(("image", "text")))\n# True\n
\n

Predicting Pair Scores

\n

You can also use predict() to get raw relevance scores for specific pairs of inputs:

\n
from sentence_transformers import CrossEncoder\n\nmodel = CrossEncoder("jinaai/jina-reranker-m0", trust_remote_code=True)\n\nscores = model.predict([\n    ("A green car", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"),\n    ("A bee on a flower", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"),\n    ("A green car", "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"),\n])\nprint(scores)\n# [0.9389156  0.96922314 0.46063158]\n
\n

Retrieve and Rerank

\n

A common pattern is to use an embedding model for fast initial retrieval, then refine the top results with a reranker:

\n
from sentence_transformers import SentenceTransformer, CrossEncoder\n\n# Step 1: Retrieve with an embedding model\nembedder = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")\n\nquery = "revenue growth chart"\nquery_embedding = embedder.encode_query(query)\n\n# Pre-compute corpus embeddings (do this once, then store them)\ndocument_screenshots = [\n    "path/to/doc1.png",\n    "path/to/doc2.png",\n    # ... potentially millions of document screenshots\n]\ncorpus_embeddings = embedder.encode_document(document_screenshots, show_progress_bar=True)\n\n# Simple cosine similarity retrieval, viable as long as embeddings fit in memory\nsimilarities = embedder.similarity(query_embedding, corpus_embeddings)\ntop_k_indices = similarities.argsort(descending=True)[0][:10]\n\n# Step 2: Rerank the top-k results with a reranker model\nreranker = CrossEncoder("nvidia/llama-nemotron-rerank-vl-1b-v2", trust_remote_code=True)\n\ntop_k_documents = [document_screenshots[i] for i in top_k_indices]\nrankings = reranker.rank(query, top_k_documents)\nfor rank in rankings:\n    print(f"{rank['score']:.4f}\\t{top_k_documents[rank['corpus_id']]}")\n
\n

Since the corpus embeddings are pre-computed, the initial retrieval is fast even over millions of documents. The reranker then provides more accurate scoring over the smaller candidate set.

\n

Input Formats and Configuration

\n

Supported Input Types

\n

Multimodal models accept a variety of input formats. Here's a summary of what you can pass to model.encode():

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModalityAccepted Formats
Text- Strings
Image- PIL.Image.Image objects - File paths (e.g. "./photo.jpg") - URLs (e.g. "https://.../image.jpg") - Numpy arrays, torch tensors
Audio- File paths (e.g. "./audio.wav") - URLs (e.g. "https://.../audio.wav") - Numpy/torch arrays - Dicts with "array" and "sampling_rate" keys - torchcodec.AudioDecoder instances
Video- File paths (e.g. "./video.mp4") - URLs (e.g. "https://.../video.mp4") - Numpy/torch arrays - Dicts with "array" and "video_metadata" keys - torchcodec.VideoDecoder instances
Multimodal- Dicts mapping modality names to values, e.g. {"text": "a caption", "image": "https://.../image.jpg"} Valid keys: "text", "image", "audio", "video"
Message- List of message dicts with "role" and "content" keys, e.g. [{"role": "user", "content": [...]}]
\n

Checking Modality Support

\n

You can check which modalities a model supports using the modalities property and supports() method:

\n
from sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer("Qwen/Qwen3-VL-Embedding-2B")\n\n# List all supported modalities\nprint(model.modalities)\n# ['text', 'image', 'video', 'message']\n\n# Check for a specific modality\nprint(model.supports("image"))\n# True\nprint(model.supports("audio"))\n# False\n
\n

The "message" modality indicates that the model accepts chat-style message inputs with interleaved content. In practice, you rarely need to use this directly. When you pass strings, URLs, or multimodal dicts, the model converts them to the appropriate message format internally. Sentence Transformers supports two message formats:

\n\n

The format is auto-detected from the model's chat template.

\n

Since all inputs get converted into the same message format internally, you can mix input types in a single encode() call:

\n
embeddings = model.encode([\n    # A text input\n    "A green car parked in front of a yellow building",\n    # An image input (URL)\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",\n    # A combined text + image input\n    {\n        "text": "A car in a European city",\n        "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",\n    },\n])\n
\n

Click here if you need to pass raw message inputs

\n

If a model doesn't follow either format and you need full control, you can pass raw message dicts with role and content keys directly:

\n
embeddings = model.encode([\n    [\n        {\n            "role": "user",\n            "content": [\n                {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg"},\n                {"type": "text", "text": "Describe this vehicle."},\n            ],\n        }\n    ],\n])\n
\n

This bypasses the automatic format conversion and passes the messages directly to the processor's apply_chat_template().

\n

Processor and Model kwargs

\n

You may want to control image resolution bounds or model precision. Use processor_kwargs and model_kwargs when loading the model:

\n
model = SentenceTransformer(\n    "Qwen/Qwen3-VL-Embedding-2B",\n    model_kwargs={"attn_implementation": "flash_attention_2", "torch_dtype": "bfloat16"},\n    processor_kwargs={"min_pixels": 28 * 28, "max_pixels": 600 * 600},\n)\n
\n\n

See the SentenceTransformer API Reference documentation for more details on these kwargs.

\n
\n

In Sentence Transformers v5.4, tokenizer_kwargs has been renamed to processor_kwargs to reflect that multimodal models use processors rather than just tokenizers. The old name is still accepted but deprecated.

\n
\n

Supported Models

\n

Here are the multimodal models supported in v5.4, also available in the v5.4 integrations collection:

\n

Supported Multimodal Embedding Models

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelParametersModalitiesRevision
Qwen/Qwen3-VL-Embedding-2B2BText, Image, VideoNo revision needed
Qwen/Qwen3-VL-Embedding-8B8BText, Image, VideoNo revision needed
nvidia/llama-nemotron-embed-vl-1b-v21.7BText, ImageNo revision needed
nvidia/omni-embed-nemotron-3b4.7BText, ImageNo revision needed
LCO-Embedding/LCO-Embedding-Omni-3B5BText, Image, Audio, VideoNo revision needed
LCO-Embedding/LCO-Embedding-Omni-7B9BText, Image, Audio, VideoNo revision needed
BidirLM/BidirLM-Omni-2.5B-Embedding2.5BText, Image, AudioNo revision needed
BAAI/BGE-VL-base0.1BText, ImageNo revision needed
BAAI/BGE-VL-large0.4BText, ImageNo revision needed
BAAI/BGE-VL-MLLM-S18BText, ImageNo revision needed
BAAI/BGE-VL-MLLM-S28BText, ImageNo revision needed
BAAI/BGE-VL-v1.5-zs8BText, ImageNo revision needed
BAAI/BGE-VL-v1.5-mmeb8BText, ImageNo revision needed
BAAI/BGE-VL-Screenshot4BText, ImageNo revision needed
royokong/e5-v8BText, ImageNo revision needed
eagerworks/eager-embed-v14BText, Imagerevision="refs/pr/2"
nomic-ai/nomic-embed-multimodal-3b5BText, Imagerevision="refs/pr/4"
nomic-ai/nomic-embed-multimodal-7b9BText, Imagerevision="refs/pr/3"
Haon-Chen/e5-omni-3B5BText, Image, Audio, Videorevision="refs/pr/2"
Haon-Chen/e5-omni-7B9BText, Image, Audio, Videorevision="refs/pr/1"
\n

Supported Multimodal Reranker Models

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelParametersModalitiesRevision
Qwen/Qwen3-VL-Reranker-2B2BText, Image, VideoNo revision needed
Qwen/Qwen3-VL-Reranker-8B8BText, Image, VideoNo revision needed
nvidia/llama-nemotron-rerank-vl-1b-v22BText, ImageNo revision needed
jinaai/jina-reranker-m02BText, ImageNo revision needed
\n

Text-Only Reranker Models (also new in v5.4)

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelParametersRevision
Qwen/Qwen3-Reranker-0.6B0.6BNo revision needed
Qwen/Qwen3-Reranker-4B4BNo revision needed
Qwen/Qwen3-Reranker-8B8BNo revision needed
mixedbread-ai/mxbai-rerank-base-v20.5BNo revision needed
mixedbread-ai/mxbai-rerank-large-v22BNo revision needed
ContextualAI/ctxl-rerank-v2-instruct-multilingual-1b1Brevision="refs/pr/2"
ContextualAI/ctxl-rerank-v2-instruct-multilingual-2b3Brevision="refs/pr/1"
ContextualAI/ctxl-rerank-v2-instruct-multilingual-6b7Brevision="refs/pr/1"
\n

Click here for a text-only reranker usage example

\n
from sentence_transformers import CrossEncoder\n\nmodel = CrossEncoder("mixedbread-ai/mxbai-rerank-base-v2")\n\nquery = "How do I bake sourdough bread?"\ndocuments = [\n    "Sourdough bread requires a starter made from flour and water, fermented over several days.",\n    "The history of bread dates back to ancient Egypt around 8000 BCE.",\n    "To bake sourdough, mix your starter with flour, water, and salt, then let it rise overnight.",\n    "Rye bread is a popular alternative to wheat-based breads in Northern Europe.",\n]\n\npairs = [(query, doc) for doc in documents]\nscores = model.predict(pairs)\nprint(scores)\n# [ 7.3077507 -2.6217823  8.724761  -2.2488995]\n\nrankings = model.rank(query, documents)\nfor rank in rankings:\n    print(f"{rank['score']:.4f}\\t{documents[rank['corpus_id']]}")\n# 8.7248  To bake sourdough, mix your starter with flour, water, and salt, then let it rise overnight.\n# 7.3078  Sourdough bread requires a starter made from flour and water, fermented over several days.\n# -2.2489 Rye bread is a popular alternative to wheat-based breads in Northern Europe.\n# -2.6218 The history of bread dates back to ancient Egypt around 8000 BCE.\n
\n

CLIP Models

\n

The older CLIP models continue to be supported:

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
ModelImageNet Zero-Shot Top-1 AccuracyNotes
sentence-transformers/clip-ViT-L-1475.4
sentence-transformers/clip-ViT-B-1668.1
sentence-transformers/clip-ViT-B-3263.3
sentence-transformers/clip-ViT-B-32-multilingual-v1N/AMultilingual text encoder, 50+ languages
\n

These simple CLIP models still work well on lower-resource hardware.

\n

Click here for a CLIP usage example

\n
from sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer("sentence-transformers/clip-ViT-L-14")\n\nimages = [\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg",\n    "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",\n    "https://huggingface.co/datasets/huggingface/cats-image/resolve/main/cats_image.jpeg"\n]\ntexts = ["A green car", "A bee on a flower", "Some cats on a couch", "One cat sitting in the window"]\n\nimage_embeddings = model.encode(images)\ntext_embeddings = model.encode(texts)\nprint(image_embeddings.shape, text_embeddings.shape)\n# (3, 768) (4, 768)\n\nsimilarities = model.similarity(image_embeddings, text_embeddings)\nprint(similarities)\n# tensor([[0.2208, 0.1042, 0.0617, 0.0907],  First image (car) is most similar to "A green car"\n#         [0.1205, 0.2303, 0.0632, 0.0917],  Second image (bee) is most similar to "A bee on a flower"\n#         [0.1107, 0.0196, 0.2425, 0.1162]]) Third image (multiple cats) is most similar to "Some cats on a couch"\n
\n

Additional Resources

\n

Documentation

\n\n

Training

\n

To learn how to finetune these multimodal models on your own data, see the companion blogpost: Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers.

\n\n

Hugging Face Hub

\n\n

Companion Blogposts

\n

The training companion to this post and adjacent Sentence Transformers guides:

\n\n", "content_markdown": "[Sentence Transformers](https://sbert.net/) is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more. With the v5.4 update, you can now **encode and compare texts, images, audio, and videos** using the same familiar API. In this blogpost, I'll show you how to use these new multimodal capabilities for both embedding and reranking.\n\nMultimodal embedding models map inputs from different modalities into a shared embedding space, while multimodal reranker models score the relevance of mixed-modality pairs. This opens up use cases like visual document retrieval, cross-modal search, and multimodal RAG pipelines.\n\n> If you want to train your own multimodal models, check out the companion blogpost: [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers).\n\n## Table of Contents\n\n- [What are Multimodal Models?](https://huggingface.co/blog/multimodal-sentence-transformers#what-are-multimodal-models)\n- [Installation](https://huggingface.co/blog/multimodal-sentence-transformers#installation)\n- [Multimodal Embedding Models](https://huggingface.co/blog/multimodal-sentence-transformers#multimodal-embedding-models)\n - [Loading a Model](https://huggingface.co/blog/multimodal-sentence-transformers#loading-a-model)\n - [Encoding Images](https://huggingface.co/blog/multimodal-sentence-transformers#encoding-images)\n - [Cross-Modal Similarity](https://huggingface.co/blog/multimodal-sentence-transformers#cross-modal-similarity)\n - [Encoding Queries and Documents](https://huggingface.co/blog/multimodal-sentence-transformers#encoding-queries-and-documents)\n- [Multimodal Reranker Models](https://huggingface.co/blog/multimodal-sentence-transformers#multimodal-reranker-models)\n - [Ranking Mixed-Modality Documents](https://huggingface.co/blog/multimodal-sentence-transformers#ranking-mixed-modality-documents)\n - [Predicting Pair Scores](https://huggingface.co/blog/multimodal-sentence-transformers#predicting-pair-scores)\n- [Retrieve and Rerank](https://huggingface.co/blog/multimodal-sentence-transformers#retrieve-and-rerank)\n- [Input Formats and Configuration](https://huggingface.co/blog/multimodal-sentence-transformers#input-formats-and-configuration)\n - [Supported Input Types](https://huggingface.co/blog/multimodal-sentence-transformers#supported-input-types)\n - [Checking Modality Support](https://huggingface.co/blog/multimodal-sentence-transformers#checking-modality-support)\n - [Processor and Model kwargs](https://huggingface.co/blog/multimodal-sentence-transformers#processor-and-model-kwargs)\n- [Supported Models](https://huggingface.co/blog/multimodal-sentence-transformers#supported-models)\n- [Additional Resources](https://huggingface.co/blog/multimodal-sentence-transformers#additional-resources)\n\n## What are Multimodal Models?\n\nTraditional embedding models convert text into [fixed-size vectors](https://huggingface.co/blog/matryoshka#understanding-embeddings). Multimodal embedding models extend this by mapping inputs from different modalities (text, images, audio, or video) into a shared embedding space. This means you can compare a text query against image documents (or vice versa) using the same similarity functions you're already familiar with.\n\nSimilarly, traditional reranker (Cross Encoder) models compute relevance scores between pairs of texts. Multimodal rerankers can score pairs where one or both elements are images, combined text-image documents, or other modalities.\n\nFor example, you can compare a text query against image documents, find video clips matching a description, or build RAG pipelines that work across modalities.\n\n## Installation\n\nMultimodal models require some extra dependencies. Install the extras for the modalities you need (see [Installation](https://sbert.net/docs/installation.html) for more details):\n\n```bash\n# For image support\npip install -U \"sentence-transformers[image]\"\n\n# For audio support\npip install -U \"sentence-transformers[audio]\"\n\n# For video support\npip install -U \"sentence-transformers[video]\"\n\n# Mix and match as needed\npip install -U \"sentence-transformers[image,video,train]\"\n```\n\n> VLM-based models like Qwen3-VL-2B require a GPU with at least ~8 GB of VRAM. For the 8B variants, expect ~20 GB. If you don't have a local GPU, consider using a cloud GPU service or Google Colab. On CPU, these models will be extremely slow; text-only or CLIP models are better suited for CPU inference.\n\n## Multimodal Embedding Models\n\n### Loading a Model\n\nLoading a multimodal embedding model works exactly like loading a text-only model:\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n```\n\n> Some models might require a `revision` argument for now if the integration pull requests for the model is still pending. Once they're merged, you'll be able to load them without specifying a revision, like above.\n\nThe model automatically detects which modalities it supports, so there's nothing extra to configure. See [Processor and Model kwargs](https://huggingface.co/blog/multimodal-sentence-transformers#processor-and-model-kwargs) if you want to control things like image resolution or model precision.\n\n### Encoding Images\n\nWith a multimodal model loaded, [`model.encode()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode) accepts images alongside text. Images can be provided as URLs, local file paths, or PIL Image objects (see [Supported Input Types](https://huggingface.co/blog/multimodal-sentence-transformers#supported-input-types) for all accepted formats):\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n\n# Encode images from URLs\nimg_embeddings = model.encode([\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\",\n])\nprint(img_embeddings.shape)\n# (2, 2048)\n```\n\n### Cross-Modal Similarity\n\nYou can compute similarities between text embeddings and image embeddings, since the model maps both into the same space:\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n\n# Encode images\nimg_embeddings = model.encode([\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\",\n])\n\n# Encode text queries (one matching + one hard negative per image)\ntext_embeddings = model.encode([\n \"A green car parked in front of a yellow building\",\n \"A red car driving on a highway\",\n \"A bee on a pink flower\",\n \"A wasp on a wooden table\",\n])\n\n# Compute cross-modal similarities\nsimilarities = model.similarity(text_embeddings, img_embeddings)\nprint(similarities)\n# tensor([[0.5115, 0.1078],\n# [0.1999, 0.1108],\n# [0.1255, 0.6749],\n# [0.1283, 0.2704]])\n```\n\nAs expected, \"A green car parked in front of a yellow building\" is most similar to the car image (0.51), and \"A bee on a pink flower\" is most similar to the bee image (0.67). The hard negatives (\"A red car driving on a highway\", \"A wasp on a wooden table\") correctly receive lower scores.\n\nYou might notice that even the best matching scores (0.51, 0.67) aren't very close to 1.0. This is due to the [modality gap](https://arxiv.org/abs/2203.02053): embeddings from different modalities tend to cluster in separate regions of the space. Cross-modal similarities are typically lower than within-modal ones (e.g., text-to-text), but the relative ordering is preserved, so retrieval still works well.\n\n### Encoding Queries and Documents\n\nFor retrieval tasks, [`encode_query()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode_query) and [`encode_document()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode_document) are the recommended methods. Many retrieval models prepend different instruction prompts depending on whether the input is a query or a document, similar to how chat models might apply different system prompts depending on the goal. Model authors can specify their prompts in the model config, and `encode_query()` / `encode_document()` automatically load and apply the correct one:\n\n- `encode_query()` uses the model's `\"query\"` prompt (if available) and sets `task=\"query\"`.\n- `encode_document()` uses the first available prompt from `\"document\"`, `\"passage\"`, or `\"corpus\"`, and sets `task=\"document\"`.\n\nUnder the hood, both are thin wrappers around `encode()`, they just handle prompt selection for you. Here's what cross-modal retrieval looks like:\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n\n# Encode text queries with the query prompt\nquery_embeddings = model.encode_query([\n \"Find me a photo of a vehicle parked near a building\",\n \"Show me an image of a pollinating insect\",\n])\n\n# Encode document screenshots with the document prompt\ndoc_embeddings = model.encode_document([\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\",\n])\n\n# Compute similarities\nsimilarities = model.similarity(query_embeddings, doc_embeddings)\nprint(similarities)\n# tensor([[0.3907, 0.1490],\n# [0.1235, 0.4872]])\n```\n\nThese methods accept the same input types as `encode()` (images, URLs, multimodal dicts, etc.) and pass through the same parameters. For models without specialized query/document prompts, they behave identically to `encode()`.\n\n## Multimodal Reranker Models\n\nMultimodal reranker (CrossEncoder) models score the relevance between pairs of inputs, where each element can be text, an image, audio, video, or a combination. They tend to outperform embedding models in terms of quality, but are slower since they process each pair individually. The currently available pretrained multimodal rerankers focus on text and image inputs, but the architecture supports any modality that the underlying model can handle.\n\n### Ranking Mixed-Modality Documents\n\nThe [`rank()`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html#sentence_transformers.cross_encoder.CrossEncoder.rank) method scores and ranks a list of documents against a query, supporting mixed modalities:\n\n```python\nfrom sentence_transformers import CrossEncoder\n\nmodel = CrossEncoder(\"Qwen/Qwen3-VL-Reranker-2B\")\n\nquery = \"A green car parked in front of a yellow building\"\ndocuments = [\n # Image documents (URL or local file path)\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\",\n # Text document\n \"A vintage Volkswagen Beetle painted in bright green sits in a driveway.\",\n # Combined text + image document\n {\n \"text\": \"A car in a European city\",\n \"image\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n },\n]\n\nrankings = model.rank(query, documents)\nfor rank in rankings:\n print(f\"{rank['score']:.4f}\\t(document {rank['corpus_id']})\")\n\"\"\"\n0.9375 (document 0)\n0.5000 (document 3)\n-1.2500 (document 2)\n-2.4375 (document 1)\n\"\"\"\n```\n\nThe reranker correctly identifies the car image (document 0) as the most relevant result, followed by the combined text+image document about a car in a European city (document 3). The bee image (document 1) scores lowest. Keep in mind that the [modality gap](https://arxiv.org/abs/2203.02053) can influence absolute scores: text-image pair scores may occupy a different range than text-text or image-image pair scores.\n\nYou can also check which modalities a reranker supports using [`modalities`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html#sentence_transformers.cross_encoder.CrossEncoder.modalities) and [`supports()`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html#sentence_transformers.cross_encoder.CrossEncoder.supports), just like with embedding models:\n\n```python\nprint(model.modalities)\n# ['text', 'image', 'video', 'message']\n\nprint(model.supports(\"image\"))\n# True\n\n# Check if the model supports a specific pair of modalities\nprint(model.supports((\"image\", \"text\")))\n# True\n```\n\n### Predicting Pair Scores\n\nYou can also use [`predict()`](https://sbert.net/docs/package_reference/cross_encoder/cross_encoder.html#sentence_transformers.cross_encoder.CrossEncoder.predict) to get raw relevance scores for specific pairs of inputs:\n\n```python\nfrom sentence_transformers import CrossEncoder\n\nmodel = CrossEncoder(\"jinaai/jina-reranker-m0\", trust_remote_code=True)\n\nscores = model.predict([\n (\"A green car\", \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\"),\n (\"A bee on a flower\", \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\"),\n (\"A green car\", \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\"),\n])\nprint(scores)\n# [0.9389156 0.96922314 0.46063158]\n```\n\n## Retrieve and Rerank\n\nA common pattern is to use an embedding model for fast initial retrieval, then refine the top results with a reranker:\n\n```python\nfrom sentence_transformers import SentenceTransformer, CrossEncoder\n\n# Step 1: Retrieve with an embedding model\nembedder = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n\nquery = \"revenue growth chart\"\nquery_embedding = embedder.encode_query(query)\n\n# Pre-compute corpus embeddings (do this once, then store them)\ndocument_screenshots = [\n \"path/to/doc1.png\",\n \"path/to/doc2.png\",\n # ... potentially millions of document screenshots\n]\ncorpus_embeddings = embedder.encode_document(document_screenshots, show_progress_bar=True)\n\n# Simple cosine similarity retrieval, viable as long as embeddings fit in memory\nsimilarities = embedder.similarity(query_embedding, corpus_embeddings)\ntop_k_indices = similarities.argsort(descending=True)[0][:10]\n\n# Step 2: Rerank the top-k results with a reranker model\nreranker = CrossEncoder(\"nvidia/llama-nemotron-rerank-vl-1b-v2\", trust_remote_code=True)\n\ntop_k_documents = [document_screenshots[i] for i in top_k_indices]\nrankings = reranker.rank(query, top_k_documents)\nfor rank in rankings:\n print(f\"{rank['score']:.4f}\\t{top_k_documents[rank['corpus_id']]}\")\n```\n\nSince the corpus embeddings are pre-computed, the initial retrieval is fast even over millions of documents. The reranker then provides more accurate scoring over the smaller candidate set.\n\n## Input Formats and Configuration\n\n### Supported Input Types\n\nMultimodal models accept a variety of input formats. Here's a summary of what you can pass to [`model.encode()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode):\n\n| Modality | Accepted Formats |\n| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| **Text** | \\- Strings |\n| **Image** | \\- `PIL.Image.Image` objects - File paths (e.g. `\"./photo.jpg\"`) - URLs (e.g. `\"https://.../image.jpg\"`) - Numpy arrays, torch tensors |\n| **Audio** | \\- File paths (e.g. `\"./audio.wav\"`) - URLs (e.g. `\"https://.../audio.wav\"`) - Numpy/torch arrays - Dicts with `\"array\"` and `\"sampling_rate\"` keys - `torchcodec.AudioDecoder` instances |\n| **Video** | \\- File paths (e.g. `\"./video.mp4\"`) - URLs (e.g. `\"https://.../video.mp4\"`) - Numpy/torch arrays - Dicts with `\"array\"` and `\"video_metadata\"` keys - `torchcodec.VideoDecoder` instances |\n| **Multimodal** | \\- Dicts mapping modality names to values, e.g. `{\"text\": \"a caption\", \"image\": \"https://.../image.jpg\"}` Valid keys: `\"text\"`, `\"image\"`, `\"audio\"`, `\"video\"` |\n| **Message** | \\- List of message dicts with `\"role\"` and `\"content\"` keys, e.g. `[{\"role\": \"user\", \"content\": [...]}]` |\n\n### Checking Modality Support\n\nYou can check which modalities a model supports using the [`modalities`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.modalities) property and [`supports()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.supports) method:\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"Qwen/Qwen3-VL-Embedding-2B\")\n\n# List all supported modalities\nprint(model.modalities)\n# ['text', 'image', 'video', 'message']\n\n# Check for a specific modality\nprint(model.supports(\"image\"))\n# True\nprint(model.supports(\"audio\"))\n# False\n```\n\nThe `\"message\"` modality indicates that the model accepts chat-style message inputs with interleaved content. In practice, you rarely need to use this directly. When you pass strings, URLs, or multimodal dicts, the model converts them to the appropriate message format internally. Sentence Transformers supports two message formats:\n\n- **Structured** (most VLMs, e.g. Qwen3-VL): Content is a list of typed dicts, e.g. `[{\"type\": \"text\", \"text\": \"...\"}, {\"type\": \"image\", \"image\": ...}]`\n- **Flat** (e.g. Deepseek-V3): Content is a direct value, e.g. `\"some text\"`\n\nThe format is auto-detected from the model's chat template.\n\nSince all inputs get converted into the same message format internally, you can mix input types in a single [`encode()`](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer.encode) call:\n\n```python\nembeddings = model.encode([\n # A text input\n \"A green car parked in front of a yellow building\",\n # An image input (URL)\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n # A combined text + image input\n {\n \"text\": \"A car in a European city\",\n \"image\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n },\n])\n```\n\nClick here if you need to pass raw message inputs\n\nIf a model doesn't follow either format and you need full control, you can pass raw message dicts with `role` and `content` keys directly:\n\n```python\nembeddings = model.encode([\n [\n {\n \"role\": \"user\",\n \"content\": [\n {\"type\": \"image\", \"image\": \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\"},\n {\"type\": \"text\", \"text\": \"Describe this vehicle.\"},\n ],\n }\n ],\n])\n```\n\nThis bypasses the automatic format conversion and passes the messages directly to the processor's `apply_chat_template()`.\n\n### Processor and Model kwargs\n\nYou may want to control image resolution bounds or model precision. Use `processor_kwargs` and `model_kwargs` when loading the model:\n\n```python\nmodel = SentenceTransformer(\n \"Qwen/Qwen3-VL-Embedding-2B\",\n model_kwargs={\"attn_implementation\": \"flash_attention_2\", \"torch_dtype\": \"bfloat16\"},\n processor_kwargs={\"min_pixels\": 28 * 28, \"max_pixels\": 600 * 600},\n)\n```\n\n- `processor_kwargs` controls how inputs are preprocessed (e.g., image resolution bounds). Higher `max_pixels` means higher quality but more memory and compute. These are passed directly to `AutoProcessor.from_pretrained(...)`.\n- `model_kwargs` controls how the underlying model is loaded (e.g., precision, attention implementation). These are passed directly to the appropriate `AutoModel.from_pretrained(...)` call (e.g., `AutoModel`, `AutoModelForCausalLM`, `AutoModelForSequenceClassification`, etc., depending on the configuration of the model modules).\n\nSee the [SentenceTransformer](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html#sentence_transformers.SentenceTransformer) API Reference documentation for more details on these kwargs.\n\n> In Sentence Transformers v5.4, `tokenizer_kwargs` has been renamed to `processor_kwargs` to reflect that multimodal models use processors rather than just tokenizers. The old name is still accepted but deprecated.\n\n## Supported Models\n\nHere are the multimodal models supported in v5.4, also available in the [v5.4 integrations collection](https://huggingface.co/collections/sentence-transformers/sentence-transformers-v54-integrations):\n\n### Supported Multimodal Embedding Models\n\n| Model | Parameters | Modalities | Revision |\n| ---------------------------------------------------------------------------------------------------- | ---------- | ------------------------- | ---------------------- |\n| [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B) | 2B | Text, Image, Video | No `revision` needed |\n| [Qwen/Qwen3-VL-Embedding-8B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-8B) | 8B | Text, Image, Video | No `revision` needed |\n| [nvidia/llama-nemotron-embed-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-embed-vl-1b-v2) | 1.7B | Text, Image | No `revision` needed |\n| [nvidia/omni-embed-nemotron-3b](https://huggingface.co/nvidia/omni-embed-nemotron-3b) | 4.7B | Text, Image | No `revision` needed |\n| [LCO-Embedding/LCO-Embedding-Omni-3B](https://huggingface.co/LCO-Embedding/LCO-Embedding-Omni-3B) | 5B | Text, Image, Audio, Video | No `revision` needed |\n| [LCO-Embedding/LCO-Embedding-Omni-7B](https://huggingface.co/LCO-Embedding/LCO-Embedding-Omni-7B) | 9B | Text, Image, Audio, Video | No `revision` needed |\n| [BidirLM/BidirLM-Omni-2.5B-Embedding](https://huggingface.co/BidirLM/BidirLM-Omni-2.5B-Embedding) | 2.5B | Text, Image, Audio | No `revision` needed |\n| [BAAI/BGE-VL-base](https://huggingface.co/BAAI/BGE-VL-base) | 0.1B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-large](https://huggingface.co/BAAI/BGE-VL-large) | 0.4B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-MLLM-S1](https://huggingface.co/BAAI/BGE-VL-MLLM-S1) | 8B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-MLLM-S2](https://huggingface.co/BAAI/BGE-VL-MLLM-S2) | 8B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-v1.5-zs](https://huggingface.co/BAAI/BGE-VL-v1.5-zs) | 8B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-v1.5-mmeb](https://huggingface.co/BAAI/BGE-VL-v1.5-mmeb) | 8B | Text, Image | No `revision` needed |\n| [BAAI/BGE-VL-Screenshot](https://huggingface.co/BAAI/BGE-VL-Screenshot) | 4B | Text, Image | No `revision` needed |\n| [royokong/e5-v](https://huggingface.co/royokong/e5-v) | 8B | Text, Image | No `revision` needed |\n| [eagerworks/eager-embed-v1](https://huggingface.co/eagerworks/eager-embed-v1) | 4B | Text, Image | `revision=\"refs/pr/2\"` |\n| [nomic-ai/nomic-embed-multimodal-3b](https://huggingface.co/nomic-ai/nomic-embed-multimodal-3b) | 5B | Text, Image | `revision=\"refs/pr/4\"` |\n| [nomic-ai/nomic-embed-multimodal-7b](https://huggingface.co/nomic-ai/nomic-embed-multimodal-7b) | 9B | Text, Image | `revision=\"refs/pr/3\"` |\n| [Haon-Chen/e5-omni-3B](https://huggingface.co/Haon-Chen/e5-omni-3B) | 5B | Text, Image, Audio, Video | `revision=\"refs/pr/2\"` |\n| [Haon-Chen/e5-omni-7B](https://huggingface.co/Haon-Chen/e5-omni-7B) | 9B | Text, Image, Audio, Video | `revision=\"refs/pr/1\"` |\n\n### Supported Multimodal Reranker Models\n\n| Model | Parameters | Modalities | Revision |\n| ------------------------------------------------------------------------------------------------------ | ---------- | ------------------ | -------------------- |\n| [Qwen/Qwen3-VL-Reranker-2B](https://huggingface.co/Qwen/Qwen3-VL-Reranker-2B) | 2B | Text, Image, Video | No `revision` needed |\n| [Qwen/Qwen3-VL-Reranker-8B](https://huggingface.co/Qwen/Qwen3-VL-Reranker-8B) | 8B | Text, Image, Video | No `revision` needed |\n| [nvidia/llama-nemotron-rerank-vl-1b-v2](https://huggingface.co/nvidia/llama-nemotron-rerank-vl-1b-v2) | 2B | Text, Image | No `revision` needed |\n| [jinaai/jina-reranker-m0](https://huggingface.co/jinaai/jina-reranker-m0) | 2B | Text, Image | No `revision` needed |\n\n### Text-Only Reranker Models (also new in v5.4)\n\n| Model | Parameters | Revision |\n| ------------------------------------------------------------------------------------------------------------------------------------ | ---------- | ---------------------- |\n| [Qwen/Qwen3-Reranker-0.6B](https://huggingface.co/Qwen/Qwen3-Reranker-0.6B) | 0.6B | No `revision` needed |\n| [Qwen/Qwen3-Reranker-4B](https://huggingface.co/Qwen/Qwen3-Reranker-4B) | 4B | No `revision` needed |\n| [Qwen/Qwen3-Reranker-8B](https://huggingface.co/Qwen/Qwen3-Reranker-8B) | 8B | No `revision` needed |\n| [mixedbread-ai/mxbai-rerank-base-v2](https://huggingface.co/mixedbread-ai/mxbai-rerank-base-v2) | 0.5B | No `revision` needed |\n| [mixedbread-ai/mxbai-rerank-large-v2](https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v2) | 2B | No `revision` needed |\n| [ContextualAI/ctxl-rerank-v2-instruct-multilingual-1b](https://huggingface.co/ContextualAI/ctxl-rerank-v2-instruct-multilingual-1b) | 1B | `revision=\"refs/pr/2\"` |\n| [ContextualAI/ctxl-rerank-v2-instruct-multilingual-2b](https://huggingface.co/ContextualAI/ctxl-rerank-v2-instruct-multilingual-2b) | 3B | `revision=\"refs/pr/1\"` |\n| [ContextualAI/ctxl-rerank-v2-instruct-multilingual-6b](https://huggingface.co/ContextualAI/ctxl-rerank-v2-instruct-multilingual-6b) | 7B | `revision=\"refs/pr/1\"` |\n\nClick here for a text-only reranker usage example\n\n```python\nfrom sentence_transformers import CrossEncoder\n\nmodel = CrossEncoder(\"mixedbread-ai/mxbai-rerank-base-v2\")\n\nquery = \"How do I bake sourdough bread?\"\ndocuments = [\n \"Sourdough bread requires a starter made from flour and water, fermented over several days.\",\n \"The history of bread dates back to ancient Egypt around 8000 BCE.\",\n \"To bake sourdough, mix your starter with flour, water, and salt, then let it rise overnight.\",\n \"Rye bread is a popular alternative to wheat-based breads in Northern Europe.\",\n]\n\npairs = [(query, doc) for doc in documents]\nscores = model.predict(pairs)\nprint(scores)\n# [ 7.3077507 -2.6217823 8.724761 -2.2488995]\n\nrankings = model.rank(query, documents)\nfor rank in rankings:\n print(f\"{rank['score']:.4f}\\t{documents[rank['corpus_id']]}\")\n# 8.7248 To bake sourdough, mix your starter with flour, water, and salt, then let it rise overnight.\n# 7.3078 Sourdough bread requires a starter made from flour and water, fermented over several days.\n# -2.2489 Rye bread is a popular alternative to wheat-based breads in Northern Europe.\n# -2.6218 The history of bread dates back to ancient Egypt around 8000 BCE.\n```\n\n### CLIP Models\n\nThe older CLIP models continue to be supported:\n\n| Model | ImageNet Zero-Shot Top-1 Accuracy | Notes |\n| ---------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ---------------------------------------- |\n| [sentence-transformers/clip-ViT-L-14](https://huggingface.co/sentence-transformers/clip-ViT-L-14) | 75.4 | |\n| [sentence-transformers/clip-ViT-B-16](https://huggingface.co/sentence-transformers/clip-ViT-B-16) | 68.1 | |\n| [sentence-transformers/clip-ViT-B-32](https://huggingface.co/sentence-transformers/clip-ViT-B-32) | 63.3 | |\n| [sentence-transformers/clip-ViT-B-32-multilingual-v1](https://huggingface.co/sentence-transformers/clip-ViT-B-32-multilingual-v1) | N/A | Multilingual text encoder, 50+ languages |\n\nThese simple CLIP models still work well on lower-resource hardware.\n\nClick here for a CLIP usage example\n\n```python\nfrom sentence_transformers import SentenceTransformer\n\nmodel = SentenceTransformer(\"sentence-transformers/clip-ViT-L-14\")\n\nimages = [\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/car.jpg\",\n \"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg\",\n \"https://huggingface.co/datasets/huggingface/cats-image/resolve/main/cats_image.jpeg\"\n]\ntexts = [\"A green car\", \"A bee on a flower\", \"Some cats on a couch\", \"One cat sitting in the window\"]\n\nimage_embeddings = model.encode(images)\ntext_embeddings = model.encode(texts)\nprint(image_embeddings.shape, text_embeddings.shape)\n# (3, 768) (4, 768)\n\nsimilarities = model.similarity(image_embeddings, text_embeddings)\nprint(similarities)\n# tensor([[0.2208, 0.1042, 0.0617, 0.0907], First image (car) is most similar to \"A green car\"\n# [0.1205, 0.2303, 0.0632, 0.0917], Second image (bee) is most similar to \"A bee on a flower\"\n# [0.1107, 0.0196, 0.2425, 0.1162]]) Third image (multiple cats) is most similar to \"Some cats on a couch\"\n```\n\n## Additional Resources\n\n### Documentation\n\n- [Sentence Transformer > Usage](https://sbert.net/docs/sentence_transformer/usage/usage.html)\n- [Sentence Transformer > Pretrained Models](https://sbert.net/docs/sentence_transformer/pretrained_models.html)\n- [Cross Encoder > Usage](https://sbert.net/docs/cross_encoder/usage/usage.html)\n- [Cross Encoder > Pretrained Models](https://sbert.net/docs/cross_encoder/pretrained_models.html)\n- [Installation](https://sbert.net/docs/installation.html)\n\n### Training\n\nTo learn how to finetune these multimodal models on your own data, see the companion blogpost: [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers).\n\n- [Sentence Transformer > Training Overview](https://sbert.net/docs/sentence_transformer/training_overview.html)\n- [Sentence Transformer > Training Examples](https://sbert.net/docs/sentence_transformer/training/examples.html)\n- [Cross Encoder > Training Overview](https://sbert.net/docs/cross_encoder/training_overview.html)\n- [Cross Encoder > Training Examples](https://sbert.net/docs/cross_encoder/training/examples.html)\n- [Sparse Encoder > Training Overview](https://sbert.net/docs/sparse_encoder/training_overview.html)\n- [Sparse Encoder > Training Examples](https://sbert.net/docs/sparse_encoder/training/examples.html)\n\n### Hugging Face Hub\n\n- [Sentence Transformers models on the Hub](https://huggingface.co/models?library=sentence-transformers)\n- [Sentence Transformers datasets on the Hub](https://huggingface.co/datasets?other=sentence-transformers)\n- [v5.4 Integrations Collection](https://huggingface.co/collections/sentence-transformers/sentence-transformers-v54-integrations)\n\n### Companion Blogposts\n\nThe training companion to this post and adjacent Sentence Transformers guides:\n\n- [Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-multimodal-sentence-transformers): the direct training companion to this post, with a Visual Document Retrieval walkthrough.\n- [Training and Finetuning Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sentence-transformers): the general training guide for text-only bi-encoder embedding models.\n- [Training and Finetuning Reranker Models with Sentence Transformers](https://huggingface.co/blog/train-reranker): Cross Encoder (reranker) training, applicable to text-only and multimodal rerankers.\n- [Training and Finetuning Sparse Embedding Models with Sentence Transformers](https://huggingface.co/blog/train-sparse-encoder): SPLADE training for sparse retrieval.\n- [🪆 Introduction to Matryoshka Embedding Models](https://huggingface.co/blog/matryoshka): variable-size embeddings; also applied to multimodal models in the training companion post.\n- [Train 400x faster Static Embedding Models with Sentence Transformers](https://huggingface.co/blog/static-embeddings): CPU-friendly text embedding models.\n- [Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval](https://huggingface.co/blog/embedding-quantization): post-training compression of embedding vectors.\n", "dateCreated": "2026-10-04T12:50:30.804479567+00:00", "date_created": "2026-04-09T00:00:00+00:00", "date_modified": "2026-04-09T00:00:00+00:00", "date_published": "2026-04-09T00:00:00+00:00", "description": "Sentence Transformers is a Python library for using and training embedding and reranker models for applications like retrieval augmented generation, semantic search, and more. With the v5.4 update, you can now encode and compare texts,…", "external_url": "https://huggingface.co/blog/multimodal-sentence-transformers", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/04/2026-04-09-multimodal-embedding-reranker-models-with-sentence.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/04/2026-04-09-multimodal-embedding-reranker-models-with-sentence.md" }, "id": "urn:aggr:item:8bab4810d254e3c8c3e7dab94b2c99472cd70e45", "isBasedOn": { "@id": "https://huggingface.co/blog/multimodal-sentence-transformers" }, "mainEntity": { "@id": "https://huggingface.co/blog/multimodal-sentence-transformers", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-04-09-multimodal-embedding-reranker-models-with-sentence/#webpage" }, "dateModified": "2026-04-09T00:00:00+00:00", "datePublished": "2026-04-09T00:00:00+00:00", "headline": "Multimodal Embedding & Reranker Models with Sentence Transformers", "timeRequired": "PT15M", "url": "https://huggingface.co/blog/multimodal-sentence-transformers", "wordCount": 3172 }, "name": "Archived snapshot: Multimodal Embedding & Reranker Models with Sentence Transformers", "preview": null, "reading_minutes": 15, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-04T12:50:30.804479567+00:00", "title": "Multimodal Embedding & Reranker Models with Sentence Transformers", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-04-09-multimodal-embedding-reranker-models-with-sentence/", "word_count": 3172 }