![\"\"](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/embeddinggemma/embeddinggemma-300m-mteb-multilingual.png\")
![\"\"](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/embeddinggemma/embeddinggemma-300m-mteb-eng.png\")
Today, Google releases EmbeddingGemma, a state-of-the-art multilingual embedding model perfect for on-device use cases. Designed for speed and efficiency, the model features a compact size of 308M parameters and a 2K context window, unlocking new possibilities for mobile RAG pipelines, agents, and more. EmbeddingGemma is trained to support over 100 languages and is the highest-ranking text-only multilingual embedding model under 500M on the Massive Text Embedding Benchmark (MTEB) at the time of writing.
\nText embeddings have become the backbone of modern natural‑language applications, turning words, sentences, and documents into dense vectors that capture meaning, sentiment, and intent. These vectors enable fast similarity search, clustering, classification, and retrieval across massive corpora, powering everything from recommendation engines and semantic search to retrieval-augmented generation and code‑search tools. Embedding models that calculate these embeddings are widely used, with well over 200 million monthly downloads on Hugging Face.
\nBuilding on this foundation, Google DeepMind’s EmbeddingGemma arrives as the newest, most capable small multilingual embedding model yet. With just 308M parameters, a 2k‑token context window, and support for over 100 languages, EmbeddingGemma delivers state‑of‑the‑art performance on the Massive Multilingual Text Embedding Benchmark (MMTEB) while staying under 200 MB of RAM when quantized.
\nThe various design choices result in a very practical, open-source tool for computing high-quality multilingual embeddings on everyday devices.
\nIn this blogpost, we describe the EmbeddingGemma architecture and training, and show you how to use the model with various frameworks like Sentence Transformers, LangChain, LlamaIndex, Haystack, txtai, Transformers.js, Text Embedding Inference, and ONNX.
\nAfterwards, we demonstrate how to finetune EmbeddingGemma on your domain for even stronger performance. In our example, we finetune EmbeddingGemma on the Medical Instruction and Retrieval Dataset (MIRIAD). The resulting model, sentence-transformers/embeddinggemma-300m-medical, achieves state-of-the-art performance on our task: retrieving passages of scientific medical papers in response to detailed medical questions. It even outperforms models twice as big on this task.
\nEmbeddingGemma builds on the Gemma3 transformers backbone, but modified to use bi-directional attention instead of causal (one-way) attention. This means that earlier tokens in the sequence can attend to later tokens, effectively turning the architecture from a decoder into an encoder. Encoder models can outperform LLMs, which are decoders, on embedding tasks like retrieval (Weller et al., 2025). With this backbone, the model can process a sizable 2048 tokens at once, sufficient for typical retrieval inputs, especially given that larger inputs often result in information loss in the text embeddings.
\nBeyond the new Gemma3-based encoder backbone, which produces token embeddings, a mean pooling layer converts these token embeddings into text embeddings. Lastly, two dense layers transform the text embeddings into their final form, a 768-dimensional vector.
\nThe EmbeddingGemma model has been trained with Matryoshka Representation Learning (MRL), allowing you to truncate the 768‑dimensional output to 512, 256, or 128 dimensions on demand. This results in faster downstream processing and lower memory and disk space utilization. See the Sentence Transformers usage for a snippet showing how to perform this truncation.
\nThe model has been trained using a carefully curated, multilingual corpus totalling approximately 320 billion tokens. The proprietary dataset is a blend of publicly available web text, code and technical documentation, and synthetic task‑specific examples. It has been filtered to avoid Child Sexual Abuse Material (CSAM), sensitive data, and low-quality or unsafe content.
\nEmbeddingGemma was benchmarked on the MMTEB (Multilingual, v2) and MTEB (English, v2) suites, which span a wide range of tasks, domains, and languages. Despite its modest 308M‑parameter size, the model consistently beats comparable baselines while keeping a very small memory footprint.
\n| MTEB (Multilingual, v2) Performance | \nMTEB (English, v2) Performance | \n
|---|---|
| \n | \n
The results will be listed on the official MTEB Leaderboard. We exclude any model that has been trained on more than 20% of the MTEB data, to mitigate potential over‑fitting.
\nThe demo can also be experienced in full screen.
\nYour browser does not support the video tag.
\nExperience the demo yourself on a Desktop device.
\nEmbeddingGemma is integrated with many popular tools, making it easy to incorporate into your existing workflows and applications. The model has been integrated in Sentence Transformers, and thus also in projects that use Sentence Transformers behind the scenes, such as LangChain, LlamaIndex, Haystack, and txtai. See the examples below to get started with your preferred framework.
\nFor production deployments, you can use Text Embeddings Inference (TEI) to serve the model efficiently on various hardware configurations, and you can use Transformers.js for use in web applications.
\nRegardless of your framework choice, you should be mindful of the prompts. For embedding models, prompts are prepended to the input text to allow the model to distinguish between different tasks. EmbeddingGemma was trained with these prompt names and prompts, so they should also be included when using the model:
\nquery: "task: search result | query: ",document: "title: none | text: ",BitextMining: "task: search result | query: ",Clustering: "task: clustering | query: ",Classification: "task: classification | query: ",InstructionRetrieval: "task: code retrieval | query: ",MultilabelClassification: "task: classification | query: ",PairClassification: "task: sentence similarity | query: ",Reranking: "task: search result | query: ",Retrieval-query: "task: search result | query: ",Retrieval-document: "title: none | text: ",STS: "task: sentence similarity | query: ",Summarization: "task: summarization | query: "In Sentence Transformers, the query and document prompts are used automatically when calling model.encode_query and model.encode_document, but for other frameworks you might have to: $
The following example scripts will demonstrate this with various frameworks.
\nYou will need to install the following packages:
\n\n\nInference using Sentence Transformers is rather simple, see this example for semantic search:
\n\n\n# Download from the 🤗 Hub\n = \n\n# Run inference with queries and documents\n = \n = \n = \n = \n\n# (768,) (4, 768)\n\n# Compute similarities to determine a ranking\n = \n\n# tensor([[0.3011, 0.6359, 0.4930, 0.4889]])\n\n# Convert similarities to a ranking\n = \n\n# tensor([1, 2, 3, 0])\nencode_query method documentationencode_document method documentationsimilarity method documentationClick to see non-retrieval code
\nIf you’re not looking to use this model for Information Retrieval, then you’re likely best off using the most general encode method together with the model prompt that best describes your downstream task out of these options:
BitextMining: Find translated sentence pairs in two languages.Clustering: Find similar texts to group them together.Classification: Assign predefined labels to texts.InstructionRetrieval: Retrieve relevant code snippets based on natural language instructions.MultilabelClassification: Assign multiple labels to texts.PairClassification: Assign predefined labels to texts.Reranking: Reorder search results based on relevance.Retrieval-query: Retrieve documents based on a query.Retrieval-document: Retrieve documents based on their content.STS: Compute semantic textual similarity between texts.Summarization: Generate concise summaries of texts.\n\n# Download from the 🤗 Hub\n = \n\n# Let's inspect the configured prompts\n\n# {\n# "query": "task: search result | query: ",\n# "document": "title: none | text: ",\n# "BitextMining": "task: search result | query: ",\n# "Clustering": "task: clustering | query: ",\n# "Classification": "task: classification | query: ",\n# "InstructionRetrieval": "task: code retrieval | query: ",\n# "MultilabelClassification": "task: classification | query: ",\n# "PairClassification": "task: sentence similarity | query: ",\n# "Reranking": "task: search result | query: ",\n# "Retrieval-query": "task: search result | query: ",\n# "Retrieval-document": "title: none | text: ",\n# "STS": "task: sentence similarity | query: ",\n# "Summarization": "task: summarization | query: ",\n# }\n\n# Compute semantic textual similarity using texts, so let's use the STS prompt\n = \n = \n\n# (4, 768)\n\n# Compute similarities\n = \n\n"""\ntensor([[1.0000, 0.9305, 0.4660, 0.4326],\n [0.9305, 1.0000, 0.4227, 0.4434],\n [0.4660, 0.4227, 1.0000, 0.2638],\n [0.4326, 0.4434, 0.2638, 1.0000]])\n"""\nencode method documentationsimilarity method documentationClick to see how to truncate embedding dimensionality for faster and cheaper search
\nBecause google/embeddinggemma-300m was trained with MRL, the embeddings generated by this model can be truncated to lower dimensionalities without considerably hurting the evaluation performance. Embeddings with lower dimensionalities are both cheaper to store on disk and in memory, as well as faster for downstream tasks like retrieval, clustering, or classification.
In Sentence Transformers, you can set a lower dimensionality using the truncate_dim parameter on either the SentenceTransformer initialization or when calling model.encode/model.encode_query/model.encode_document:
\n\n# Download from the 🤗 Hub\n = \n\n# Run inference with queries and documents\n = \n = \n = \n = \n\n# (256,) (4, 256)\n\n# Compute similarities to determine a ranking\n = \n\n# tensor([[0.4016, 0.6715, 0.5283, 0.5261]])\n\n# Convert similarities to a ranking\n = \n\n# tensor([1, 2, 3, 0])\nNote that the ranking is preserved despite using 3x smaller embeddings compared to the full-sized embeddings.
\n\nIf you prefer, you can also use the LangChain HuggingFaceEmbeddings, which uses Sentence Transformers behind the scenes. Note that you'll have to tell LangChain to use the prompts called "query" and "document" for queries and documents, respectively. This example involves a simple information retrieval setup, but the same embedding model can be used in more complex scenarios.
You will need to install the following packages:
\n\n\n\n\n\n\n\n\n\n\n# Download the model from the 🤗 Hub. Also specify to use the "query" and "document" prompts\n# as defined in the model configuration, as LangChain doesn't automatically use them.\n# See https://huggingface.co/google/embeddinggemma-300m/blob/main/config_sentence_transformers.json\n = \n\n = \n\n# Create documents for the vector store\n = \n\n# Create vector store using FAISS. Setting distance_strategy to "MAX_INNER_PRODUCT" uses\n# FAISS' FlatIndexIP behind the scenes, which is optimized for inner product search. This\n# is what the model was trained for\n = \n\n# Search for top 3 similar documents\n = \n = \n\n# Print results\n\n \n"""\nText: Mars, known for its reddish appearance, is often referred to as the Red Planet. (score: 0.6359)\nText: Jupiter, the largest planet in our solar system, has a prominent red spot. (score: 0.4930)\nText: Saturn, famous for its rings, is sometimes mistaken for the Red Planet. (score: 0.4889)\n"""\nEmbeddingGemma is also supported in LlamaIndex as it uses Sentence Transformers under the hood. For the correct behaviour, you need to specify the query and document prompts as defined in the model configuration. Otherwise, your performance will be suboptimal. This script shows a rudimentary example of using EmbeddingGemma with LlamaIndex, but you can use the HuggingFaceEmbedding class in more difficult settings also.
You will need to install the following packages:
\n\n\n\n\n\n\n\n\n\n\n\n# Download from the 🤗 Hub. Also specify the query and document prompts as\n# defined in the model configuration, as LlamaIndex doesn't automatically load them.\n# See https://huggingface.co/google/embeddinggemma-300m/blob/main/config_sentence_transformers.json\n = \n\n = \n\n# Create a sample vector store\n = \n\n\n# Search for top k similar documents\n = \n = \n = \n\n# Print results\n\n \n"""\nText: Mars, known for its reddish appearance, is often referred to as the Red Planet. (score: 0.6359)\nText: Jupiter, the largest planet in our solar system, has a prominent red spot. (score: 0.4930)\nText: Saturn, famous for its rings, is sometimes mistaken for the Red Planet. (score: 0.4889)\n"""\nEmbeddingGemma can also be used with Haystack, a framework for building production-ready search and language applications. Like LangChain and LlamaIndex, Haystack uses Sentence Transformers behind the scenes and requires you to specify the appropriate prompts. The following example shows how to set up a basic retrieval pipeline using EmbeddingGemma with Haystack.
\nYou will need to install the following packages:
\n\n\n\n\n\n\n\n\n# Initialize the document store\n = \n\n# Initialize the document and query embedders\n = \n = \n\n\n\n = \n\n# Convert to Haystack documents and write to document store\n = \n = \n\n\n# Create a query pipeline using a query embedder and compatible retriever\n = \n\n\n\n\n# Search for top 3 similar documents\n = \n = \n\n# Print results\n\n \n"""\nText: Mars, known for its reddish appearance, is often referred to as the Red Planet. (score: 0.6359)\nText: Jupiter, the largest planet in our solar system, has a prominent red spot. (score: 0.4930)\nText: Saturn, famous for its rings, is sometimes mistaken for the Red Planet. (score: 0.4889)\n"""\ntxtai is also compatible with EmbeddingGemma. Like other frameworks, txtai utilizes Sentence Transformers under the hood and needs the appropriate prompts for optimal performance with EmbeddingGemma. The following example demonstrates how to set up a basic retrieval system with txtai.
\nYou will need to install the following packages:
\n\n\n\n\n\n# Download from the 🤗 Hub. Also specify the query and document prompts as\n# defined in the model configuration, as txtai doesn't automatically load them.\n# See https://huggingface.co/google/embeddinggemma-300m/blob/main/config_sentence_transformers.json\n = \n\n = \n\n# Create a sample vector store\n\n\n# Search for top k similar documents\n = \n = \n\n# Print results\n\n \n"""\nText: Mars, known for its reddish appearance, is often referred to as the Red Planet. (score: 0.6359)\nText: Jupiter, the largest planet in our solar system, has a prominent red spot. (score: 0.4930)\nText: Saturn, famous for its rings, is sometimes mistaken for the Red Planet. (score: 0.4889)\n"""\nYou can even run EmbeddingGemma 100% locally in your browser with Transformers.js! If you haven't already, you can install the library from NPM using:
\n\nYou can then compute embeddings as follows:
\n;\n\n// Download from the 🤗 Hub\nconst model_id = "onnx-community/embeddinggemma-300m-ONNX";\nconst tokenizer = await AutoTokenizer.;\nconst model = await AutoModel.;\n\n// Run inference with queries and documents\nconst prefixes = ;\nconst query = prefixes. + "Which planet is known as the Red Planet?";\nconst documents = .;\n\nconst inputs = await ;\nconst = await ;\n\n// Compute similarities to determine a ranking\nconst scores = await ;\nconst similarities = scores..;\n;\n// [ 0.30109718441963196, 0.6358831524848938, 0.4930494725704193, 0.48887503147125244 ]\n\n// Convert similarities to a ranking\nconst ranking = similarities..;\n;\n// [\n// { index: 1, score: 0.6358831524848938 },\n// { index: 2, score: 0.4930494725704193 },\n// { index: 3, score: 0.48887503147125244 },\n// { index: 0, score: 0.30109718441963196 }\n// ]\nYou can easily deploy EmbeddingGemma for both development and production using Text Embeddings Inference (TEI) version 1.8.1 or later.
\n\n\n\n\n\nIf you run the Docker container with the
\ncuda-1.8.1tag, it includes support for multiple GPU architectures: Turing, Ampere, Ada Lovelace, and Hopper. For a lighter image tailored to just your GPU, you can instead use a specific tag such asturing-1.8.1,1.8.1and86-1.8.1(Ampere),89-1.8.1(Ada Lovelace), orhopper-1.8.1.
Once deployed, regardless of the device or runtime, you can leverage the /v1/embeddings endpoint based on the OpenAI Embeddings API Specification to generate embeddings.
\nAlternatively, you can also leverage the /embed endpoint from the Text Embeddings Inference Embeddings API, which supports the prompt_name parameter, meaning there’s no need to manually prepend the prompt to the inputs but select it via prompt_name instead.
\n\n\nAdditionally, note that since
\ngoogle/embeddinggemma-300mwas trained with Matryoshka Representation Learning (MRL), you can also leverage thedimensionsparameter, on both/v1/embeddingsand/embed, to truncate the embeddings to lower dimensionalities (512, 256, and 128) without hurting the evaluation performance.
You can also run the model directly with ONNX Runtime, making it highly portable and cross-platform compatible. The example below shows usage in Python, but the same approach can be applied in other languages (Java, C#, C++, etc.) as well.
\n\n\n\n\n# Download from the 🤗 Hub\n = \n = # Download graph\n # Download weights\n = \n = \n\n# Run inference with queries and documents\n = \n = + \n = \n = \n\n = \n\n, = \n # (5, 768)\n\n# Compute similarities to determine a ranking\n = \n = \n = @ \n # [0.30109745 0.635883 0.49304956 0.48887485]\n\n# Convert similarities to a ranking\n = \n # [1 2 3 0]\nAs with all models compatible with the Sentence Transformers library, EmbeddingGemma can be easily fine-tuned on your specific dataset. To showcase this, we'll be finetuning google/embeddinggemma-300m on the Medical Instruction and RetrIeval Dataset (MIRIAD) dataset, such that our finetuned model becomes particularly adept at finding passages up to 1000 tokens from scientific medical papers given detailed medical questions. These passages can be used as crucial context for a generative model to answer questions more effectively.
Below, you can explore each key component of the finetuning process using expandable tabs. Each tab contains the relevant code and a detailed explanation.
\nModel
\n\n\n = \n# SentenceTransformer(\n# (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})\n# (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})\n# (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})\n# (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})\n# (4): Normalize()\n# )\nThis code loads the EmbeddingGemma model from Hugging Face, with optional model card metadata for documentation and sharing. The SentenceTransformer class loads the model weights and configuration, while the model_card_data argument attaches metadata useful for inclusion in the automatically generated model card.
Dataset
\n\n\n = \n = \n = \n# Dataset({\n# features: ['question', 'passage_text'],\n# num_rows: 100000\n# })\n# Dataset({\n# features: ['question', 'passage_text'],\n# num_rows: 1000\n# })\n# Dataset({\n# features: ['question', 'passage_text'],\n# num_rows: 1000\n# })\nThis code loads the MIRIAD dataset, or rather, a copy that has been divided into train, eval, and test splits. Using a large, high-quality dataset ensures the model learns meaningful representations, while subsetting allows for faster experimentation. The load_dataset function fetches the dataset from Hugging Face Datasets, and the .select() method limits the number of samples for each split.
Loss Function
\n\n\n = \nThis code defines the loss function for training, using Cached Multiple Negatives Ranking Loss (CMNRL). CMNRL is effective for retrieval tasks, as it uses in-batch negatives to efficiently train the model to distinguish between correct and incorrect pairs. The loss takes question-answer pairs and treats other answers in the batch as negatives, maximizing the distance between unrelated pairs in the embedding space. The mini_batch_size parameter controls the memory usage, but does not affect the training dynamics.
It's recommended to use this loss with a large per_device_train_batch_size in SentenceTransformerTrainingArguments and a low mini_batch_size in CachedMultipleNegativesRankingLoss for a strong training signal with low memory usage. Additionally, the NO_DUPLICATES batch sampler is recommended to avoid accidental false negatives.
Training Arguments
\n\n\n\n = \n = \nThis code sets up all hyperparameters and configuration for training, evaluation, and logging. Proper training arguments are crucial for efficient, stable, and reproducible training. The arguments control batch sizes, learning rate, mixed precision, evaluation and saving frequency, and more. Notably, the prompts dictionary maps dataset columns to prompts used by the model to distinguish queries from documents.
Evaluator
\n\n\n = \n = \n = \n = \n\nThis code sets up an evaluator for information retrieval, using queries and a corpus to measure model performance. Evaluation during training helps monitor progress and avoid overfitting. The evaluator computes retrieval metrics (NDCG, MRR, Recall, Precision, MAP, etc.) by checking if the model retrieves the correct passages for each query. It can be run before, during, and after training, and the results will be logged and incorporated in the automatically generated model card.
\nNote that this snippet in particular uses all (1k) evaluation questions against a corpus of all (1k) evaluation passages and 30k training passages, for a total of 31k documents. Evaluating only against evaluation passages is too simple for the model.
\nTrainer
\n\n\n = \n\nThis code initializes and runs the training loop, coordinating all components.
\nBelow is the complete script, combining all components above:
\n\n\n\n\n\n\n\n\n\n# Set the log level to INFO to get more information\n\n\n# 1. Load a model to finetune with 2. (Optional) model card data\n = \n\n# 3. Load a dataset to finetune on\n = \n = \n = \n\n# 4. Define a loss function. CachedMultipleNegativesRankingLoss (CMNRL) is a special variant of MNRL (a.k.a. InfoNCE),\n# which take question-answer pairs (or triplets, etc.) as input. It will take answers from other questions in the batch\n# as wrong answers, reducing the distance between the question and the true answer while increasing the distance to the\n# wrong answers, in the embedding space.\n# The (C)MNRL losses benefit from larger `per_device_train_batch_size` in the Training Arguments, as they can leverage\n# more in-batch negative samples. At the same time, the `mini_batch_size` does not affect training performance, but it\n# does limit the memory usage. A good trick is setting a high `per_device_train_batch_size` while keeping\n# `mini_batch_size` small.\n = \n\n# 5. (Optional) Specify training arguments\n = \n = \n\n# 6. (Optional) Create an evaluator using the evaluation queries and 31k answers & evaluate the base model\n = \n = \n = \n = \n\n\n# 7. Create a trainer & train\n = \n\n\n# (Optional) Evaluate the trained model on the evaluation set once more, this will also log the results\n# and include them in the model card\n\n\n = \n = \n = \n = \n\n\n# 8. Save the trained model\n = f\n\n\n# 9. (Optional) Push it to the Hugging Face Hub\n# It is recommended to run `huggingface-cli login` to log into your Hugging Face account first\n\n \n\n \nWe ran the full training script on an RTX 3090 with 24GB of VRAM, and the completed training and evaluating scripts took 5.5 hours. If desired, you can further reduce the memory footprint by reducing mini_batch_size on the CachedMultipleNegativesRankingLoss and batch_size on the InformationRetrievalEvaluator instances. See here the logs from our training run:
| Epoch | \nStep | \nTraining Loss | \nValidation Loss | \nmiriad-eval-1kq-31kd_cosine_ndcg@10 | \nmiriad-test-1kq-31kd_cosine_ndcg@10 | \n
|---|---|---|---|---|---|
| -1 | \n-1 | \n- | \n- | \n0.8474 | \n0.8340 | \n
| 0.0256 | \n20 | \n0.1019 | \n- | \n- | \n- | \n
| 0.0512 | \n40 | \n0.0444 | \n- | \n- | \n- | \n
| 0.0767 | \n60 | \n0.0408 | \n- | \n- | \n- | \n
| 0.1023 | \n80 | \n0.0462 | \n- | \n- | \n- | \n
| 0.1279 | \n100 | \n0.0542 | \n0.0525 | \n0.8616 | \n- | \n
| 0.1535 | \n120 | \n0.0454 | \n- | \n- | \n- | \n
| 0.1790 | \n140 | \n0.0403 | \n- | \n- | \n- | \n
| 0.2046 | \n160 | \n0.0463 | \n- | \n- | \n- | \n
| 0.2302 | \n180 | \n0.0508 | \n- | \n- | \n- | \n
| 0.2558 | \n200 | \n0.0497 | \n0.0449 | \n0.8643 | \n- | \n
| 0.2813 | \n220 | \n0.0451 | \n- | \n- | \n- | \n
| 0.3069 | \n240 | \n0.0445 | \n- | \n- | \n- | \n
| 0.3325 | \n260 | \n0.0489 | \n- | \n- | \n- | \n
| 0.3581 | \n280 | \n0.0452 | \n- | \n- | \n- | \n
| 0.3836 | \n300 | \n0.0461 | \n0.0406 | \n0.8832 | \n- | \n
| 0.4092 | \n320 | \n0.0415 | \n- | \n- | \n- | \n
| 0.4348 | \n340 | \n0.04 | \n- | \n- | \n- | \n
| 0.4604 | \n360 | \n0.0399 | \n- | \n- | \n- | \n
| 0.4859 | \n380 | \n0.0423 | \n- | \n- | \n- | \n
| 0.5115 | \n400 | \n0.0352 | \n0.0316 | \n0.8823 | \n- | \n
| 0.5371 | \n420 | \n0.0408 | \n- | \n- | \n- | \n
| 0.5627 | \n440 | \n0.0356 | \n- | \n- | \n- | \n
| 0.5882 | \n460 | \n0.0371 | \n- | \n- | \n- | \n
| 0.6138 | \n480 | \n0.0276 | \n- | \n- | \n- | \n
| 0.6394 | \n500 | \n0.028 | \n0.0280 | \n0.8807 | \n- | \n
| 0.6650 | \n520 | \n0.0302 | \n- | \n- | \n- | \n
| 0.6905 | \n540 | \n0.0345 | \n- | \n- | \n- | \n
| 0.7161 | \n560 | \n0.0325 | \n- | \n- | \n- | \n
| 0.7417 | \n580 | \n0.033 | \n- | \n- | \n- | \n
| 0.7673 | \n600 | \n0.0314 | \n0.0264 | \n0.8910 | \n- | \n
| 0.7928 | \n620 | \n0.033 | \n- | \n- | \n- | \n
| 0.8184 | \n640 | \n0.029 | \n- | \n- | \n- | \n
| 0.8440 | \n660 | \n0.0396 | \n- | \n- | \n- | \n
| 0.8696 | \n680 | \n0.0266 | \n- | \n- | \n- | \n
| 0.8951 | \n700 | \n0.0262 | \n0.0240 | \n0.8968 | \n- | \n
| 0.9207 | \n720 | \n0.0262 | \n- | \n- | \n- | \n
| 0.9463 | \n740 | \n0.0327 | \n- | \n- | \n- | \n
| 0.9719 | \n760 | \n0.0293 | \n- | \n- | \n- | \n
| 0.9974 | \n780 | \n0.0304 | \n- | \n- | \n- | \n
| -1 | \n-1 | \n- | \n- | \n0.9026 | \n0.8862 | \n
The performance of the base model was already excellent, with a strong 0.8340 NDCG@10 on our MIRIAD test set. Despite that, we were able to increase it considerably on this domain-specific dataset.
\n| Model | \nNumber of Parameters | \nNDCG@10 on miriad-test-1kq-31kd | \n
|---|---|---|
BAAI/bge-base-en-v1.5 | \n109M | \n0.7541 | \n
intfloat/multilingual-e5-small | \n118M | \n0.6852 | \n
ibm-granite/granite-embedding-125m-english | \n125M | \n0.7745 | \n
Snowflake/snowflake-arctic-embed-m-long | \n137M | \n0.7514 | \n
intfloat/multilingual-e5-base | \n278M | \n0.7052 | \n
Snowflake/snowflake-arctic-embed-m-v2.0 | \n305M | \n0.8467 | \n
BAAI/bge-large-en-v1.5 | \n335M | \n0.7727 | \n
mixedbread-ai/mxbai-embed-large-v1 | \n335M | \n0.7851 | \n
intfloat/multilingual-e5-large | \n560M | \n0.7318 | \n
Snowflake/snowflake-arctic-embed-l-v2.0 | \n568M | \n0.8433 | \n
Qwen/Qwen3-Embedding-0.6B | \n596M | \n0.8493 | \n
| \n | \n | \n |
google/embeddinggemma-300m (base) | \n268M | \n0.8340 | \n
sentence-transformers/embeddinggemma-300m-medical (fine-tuned) | \n268M | \n0.8862 | \n
Our fine-tuning process achieved a significant improvement of +0.0522 NDCG@10 on the test set, resulting in a model that comfortably outperforms any existing general-purpose embedding model on our specific task, at this model size. Additional time and compute investment would allow for even stronger results, such as hard negatives mining or training with more than 100k data pairs.
\nTo extend the finetuning approach from this post, or apply related techniques to EmbeddingGemma and other embedding models:
\n