{ "@context": "https://schema.org", "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-09-03-neomme-an-efficient-multimodal-native-and-multilingual/#webpage", "@type": [ "WebPage", "ArchiveComponent" ], "articleBody": "[![Hugging Face](https://img.shields.io/badge/Collection-FFD21E?style=for-the-badge&logo=huggingface&logoColor=000)](https://hf.co/collections/Hcompany/neomme) [![Hugging Face Paper](https://img.shields.io/badge/HF_Paper-FFD21E?style=for-the-badge&logo=huggingface&logoColor=000)](https://huggingface.co/papers/2609.01657) [![arXiv](https://img.shields.io/badge/arXiv-2609.01657-b31b1b.svg?style=for-the-badge)](https://arxiv.org/abs/2609.01657)\n\n![NeoMME logo](https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/assets/neomme/neomme_logo.webp?raw=true)\n\n## TL;DR\n\nWe introduce ***NeoMME***, a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models, *NeoMME* does not use a separate pretrained vision tower or a causal language model. A single bidirectional Transformer processes both text tokens and raw image patches, and we train the entire model from scratch with a masked discrete-diffusion objective.\n\nWe fine-tuned *NeoMME* for visual document retrieval using ColPali's page-image approach. *NeoMME*-Retriever returns dense and late-interaction embeddings in one forward pass. Both model sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and model size. At a matched 2048×2048 image input size on an NVIDIA L40S GPU, the 260M model encodes about 51 pages per second, or about twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from roughly 1.5 MB to 6 kB per page (255× smaller) while retaining more than 95% of baseline nDCG@10.\n\n*NeoMME* is available in Hugging Face Transformers. We release all model checkpoints under the Apache 2.0 license.\n\n- 🤗 [*NeoMME* collection](https://hf.co/collections/Hcompany/neomme)\n- 📄 [Technical report](https://huggingface.co/papers/2609.01657)\n- 🔎 [Visual RAG demo](https://huggingface.co/spaces/tonywu71/neomme-retriever-demo)\n\n## Why another multimodal encoder?\n\nMany recent visual document retrievers are adapted from pretrained generative visual language models. A separately pretrained vision encoder produces visual features, which a projector maps into the language model's input space. A causal decoder then processes the combined image and text representations. Retrieval, classification, and token labeling do not generate text autoregressively, so they do not require a causal decoder or the parameter and compute overhead of this architecture.\n\n[ModernBERT](https://huggingface.co/blog/modernbert) brought efficient architecture and training improvements to bidirectional encoders. For visual document retrieval, [ModernVBERT](https://huggingface.co/blog/paultltc/modernvbert) applied a bidirectional ModernBERT-style text encoder while retaining a separate pretrained SigLIP2 vision tower. We wanted to push this even further by designing and training a multimodal encoder without having to carry over the parameter and compute overhead of a VLM.\n\n***NeoMME*** (pronounced \"nee-oh-me\", IPA /ˈniː.oʊ.mi/) is a multilingual, multimodal foundation encoder that generates vector representations for input text and/or images using a single Transformer encoder. It is not based on an existing pretrained vision tower, text encoder, or text decoder.\n\n![Comparison of dual-tower, VLM, ModernVBERT, and NeoMME input paths](https://github.com/tonywu71/colpali-cookbooks/blob/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa/assets/neomme/architecture.webp?raw=true)\\\nUnlike dual-tower and VLM encoders, *NeoMME* processes image patches and text tokens in one bidirectional Transformer, without a pretrained vision tower or a pretrained text encoder or decoder.\n\nImages and text use the same computational path, so *NeoMME* can more easily support pretraining, fine-tuning, parallelization, and serving across both modalities.\n\n## *NeoMME* encoder backbone\n\n### One Transformer for images and text\n\n*NeoMME* comes in two sizes, [260M](https://huggingface.co/Hcompany/NeoMME-260M) and [800M](https://huggingface.co/Hcompany/NeoMME-800M). Both variants share the same architecture:\n\n- **Native multimodal inputs:** text inputs use factorized token embeddings, while images are divided into a grid of non-overlapping 32×32 patches and projected with a small MLP. Both enter the same Transformer encoder.\n- **Dynamic image resolution:** images keep their aspect ratio and size. This allows the model to use more tokens on a high-resolution, information-dense document page than on a smaller image with less content.\n- **Long bidirectional context:** both models have a context length of 16,384 tokens (enough for up to two standard 3840×2160 4K UHD images). Most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention.\n- **A modern encoder stack:** *NeoMME* uses recent encoder improvements such as grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs, among others.\n- **Multilingual text:** we trained a BPE tokenizer with a 131k-token vocabulary from scratch on multilingual text, code, mathematics, and machine-produced image transcripts.\n\n![Alternating sliding-window and global-attention layers in NeoMME](https://github.com/tonywu71/colpali-cookbooks/blob/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa/assets/neomme/encoder-stack.webp?raw=true)\\\nAlternating sliding-window and global-attention layers in the *NeoMME* encoder stack.\n\n### Learning from images through masked text\n\nWe pretrain *NeoMME* from scratch as a discrete masked-diffusion text denoiser. For each text-only example, we sample a corruption rate uniformly between 0 and 1. Each eligible text token is then independently masked at that rate.\n\nMultimodal examples use corruption rates between 0.3 and 1. The image patches remain visible while *NeoMME* reconstructs masked text. With light masking, the model can often recover a missing word from the surrounding text alone. For example, \"cat\" is a plausible completion of \"The \\[MASK\\] sat on the mat,\" even without an image. But high masking forces the model to learn image-grounded descriptions with little to no signal from the non-masked input text tokens.\n\n![Effect of text corruption on the textual and visual evidence available to NeoMME](https://github.com/tonywu71/colpali-cookbooks/blob/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa/assets/neomme/masked-diffusion.webp?raw=true)\\\nHigher text corruption removes language-only shortcuts and encourages *NeoMME* to use visible image evidence.\n\nPretraining mixes multilingual text, code, mathematics, natural images, and document images. Each model processes about 524 billion packed input tokens, including 290 billion tokens from text-only examples. This text budget is relatively small compared with ModernBERT's 2 trillion training token budget. Hence, we chose the NorMuon optimizer to improve data efficiency during training.\n\n## *NeoMME*-Retriever\n\nTo get a meaningful downstream evaluation of the backbone, we fine-tune *NeoMME* for visual document retrieval using the page-image methodology introduced by [ColPali](https://huggingface.co/blog/manu/colpali). While traditional text-based retrieval consists of retrieving text chunks, *NeoMME*-Retriever ranks document page screenshots and bypasses all the preprocessing OCR steps necessary to extract text from PDFs. Treating the pages as images preserves layout, charts, tables, font type and size, and other visual clues that cannot be captured even by a perfect OCR model.\n\n### A dual-head design for dense and late-interaction retrieval\n\n*NeoMME*-Retriever reuses the *NeoMME* backbone but adds two jointly trained heads on top of it for retrieval:\n\n- The dense head averages the backbone's hidden state vectors into a normalized vector (mean pooling). Dense embeddings are most common today: they are compact and work naturally with approximate nearest-neighbor (ANN) techniques for fast retrieval.\n- The late-interaction head projects each text token or image patch from the backbone's output hidden states to a 128-dimensional normalized vector. Compared to dense embeddings, the finer granularity preserves local matches between individual query tokens and image regions.\n\n![Late-interaction and dense retrieval heads for NeoMME](https://github.com/tonywu71/colpali-cookbooks/blob/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa/assets/neomme/retrieval-heads.webp?raw=true)\\\nLate-interaction and dense retrieval heads for both *NeoMME* model sizes.\n\n> Omar Khattab, who introduced late-interaction in ColBERT, [explains why](https://x.com/lateinteraction/status/1894696983077785980) the term is more precise than \"multi-vector.\" It describes the granularity and learnability of the scoring function, not simply the number of stored vectors.\n>\n> To learn more about late-interaction, we recommend reading [this crash course](https://meet.ameliechatelain.com/lectures/multi-vector-search/) by Amélie Chatelain.\n\nOne *NeoMME*-Retriever forward pass returns both representations, which gives you flexibility no matter your use case and infrastructure. We recommend using late-interaction embeddings in general since they are more powerful and can be used easily with open-source libraries like [NextPlaid](https://github.com/lightonai/next-plaid). However, if you have a very large corpora, you can run a single forward pass with *NeoMME*-Retriever to get the dense embedding, retrieve a small number of documents through an ANN index, and then use late-interaction to rerank the retrieved candidates.\n\n### Competitive retrieval at compact model sizes\n\nWe report nDCG@10 on ViDoRe v3. *NeoMME*-Retriever-260M reaches 0.523, the highest score among evaluated models strictly below 800M parameters. It is within 0.002 nDCG@10 of ColQwen2.5 while using about 14× fewer parameters. *NeoMME*-Retriever-800M reaches 0.556, within 0.009 nDCG@10 of the similarly sized [Vultron Retriever Flash (0.8B)](https://huggingface.co/vultr/VultronRetrieverFlash-Qwen3.5-0.8B). Both *NeoMME*-Retriever models lie on the model-size Pareto frontier.\n\n![ViDoRe v3 nDCG@10 versus model size](https://github.com/tonywu71/colpali-cookbooks/blob/c5e737e7e474363822b6f2d95290c0120e0fa993/assets/neomme/vidore-v3-model-size.webp?raw=true)\\\nViDoRe v3 nDCG@10 versus model size.\n\nViDoRe v1 and v2 use nDCG@5. On both benchmarks, *NeoMME*-Retriever-260M outperforms ColModernVBERT and the twice-larger ColSmol-500M. *NeoMME*-Retriever-800M outperforms ColPali v1.3 while using 3.6 times fewer parameters.\n\nVisual document retrieval performance on the ViDoRe benchmarks.\n| Model details | ViDoRe (nDCG@*k*) | | | |\n| ---------------------------------------------------------------------------------- | ----------------- | --------- | --------- | --------- |\n| [ColModernVBERT](https://huggingface.co/ModernVBERT/colmodernvbert) | 250M | 0.261† | 0.407‡ | 0.806‡ |\n| [ColSmol-256M](https://huggingface.co/vidore/colSmol-256M)† | 256M | 0.207 | 0.348 | 0.797 |\n| [*NeoMME*-260M](https://huggingface.co/Hcompany/NeoMME-260M-Retriever)‡ | 260M | **0.523** | **0.522** | **0.860** |\n| [ColSmol-500M](https://huggingface.co/vidore/colSmol-500M) | 500M | 0.340‡ | 0.455† | 0.825† |\n| [Vultron Flash](https://huggingface.co/vultr/VultronRetrieverFlash-Qwen3.5-0.8B)† | 850M | **0.565** | **0.604** | **0.882** |\n| [*NeoMME*-800M](https://huggingface.co/Hcompany/NeoMME-800M-Retriever)‡ | 800M | 0.556 | 0.559 | 0.874 |\n| [ColQwen2.5-v0.2](https://huggingface.co/vidore/colqwen2.5-v0.2)† | 3.75B | **0.524** | **0.601** | **0.895** |\n| [ColPali v1.3](https://huggingface.co/vidore/colpali-v1.3)† | 2.92B | 0.430 | 0.547 | 0.848 |\n\n† Scores from MTEB. ‡ Results from our own evaluations.\n\n### Making high-resolution retrieval practical for late-interaction\n\nLate-interaction storage scales linearly with the number of vectors in the output embedding. Higher-resolution images contain more patches, so they produce larger embeddings. For example, a 2048×2048 square page produces embeddings containing 4,200 vectors with *NeoMME*-Retriever, or about 2.1 MB in float32. Across the ViDoRe v3 benchmark, the measured average is about 1.5 MB per document.\n\nTo reduce the storage footprint of the late-interaction index, we combine two complementary compression methods:\n\n1. [Hierarchical token pooling](https://www.answer.ai/posts/colbert-pooling.html) clusters similar document vectors in a given multi-vector embedding and replaces each cluster with its mean, hence reducing the number of vectors stored for each page.\n2. [Asymmetric quantization](https://www.mixedbread.com/blog/asymmetric-quant) quantizes document embeddings to int8 or binary. Because query embeddings are not stored and only generated on-the-fly, they can be kept at a higher precision.\n\nWe tested this setup on ViDoRe v3. With a pooling factor 10 and int8 queries and documents, storage decreased from about 1.5 MB to 39 kB per page, a 39× reduction, while keeping more than 99% of the baseline nDCG@10. A more aggressive configuration uses pooling factor 8, int8 queries, and binary documents. That version uses 6 kB per page (255× smaller) and keeps more than 95% of the original retrieval quality.\n\n![Quality and storage frontier for the NeoMME-260M late-interaction index](https://github.com/tonywu71/colpali-cookbooks/blob/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa/assets/neomme/compression-frontier-260m.webp?raw=true)\\\nQuality and storage frontier for the *NeoMME*-260M late-interaction index on ViDoRe v3. Labels show pool factor, retained quality, compression, and storage.\n\nUsers can pick a compression setting from that frontier based on storage budget and required retrieval quality.\n\n#### Fast inference for cheaper multimodal corpus indexing\n\nBefore you can search a corpus, a retriever model must turn your documents into embeddings, which will be stored in a vector store like Qdrant, Weaviate, or Milvus. Faster encoding makes building and adding new documents to the index faster, thus reducing the GPU uptime and compute cost required.\n\nSo we measured image encoding speeds for *NeoMME*-Retriever against other multimodal document retrievers. We used preprocessed image tensors and calibrated the batch size separately for each model and image size. At a matched 2048×2048 input size on one NVIDIA L40S, *NeoMME*-Retriever-260M encodes about 51 pages per second, nearly twice ColModernVBERT's 26 pages per second. Both 260M and 800M *NeoMME*-Retriever models are also faster than the other models we compared on smaller input images.\n\n![Document-encoding throughput across image resolutions on an NVIDIA L40S](https://github.com/tonywu71/colpali-cookbooks/blob/c5e737e7e474363822b6f2d95290c0120e0fa993/assets/neomme/indexing-throughput-l40s.webp?raw=true)\\\nDocument-encoding throughput by retriever and input resolution on one NVIDIA L40S.\n\n### Try *NeoMME*-Retriever yourself!\n\n*NeoMME*-Retriever ([260M](https://huggingface.co/Hcompany/NeoMME-260M-Retriever) and [800M](https://huggingface.co/Hcompany/NeoMME-800M-Retriever)) returns dense and multi-vector embeddings together. The example below scores two text queries against two document-page images with MeanMaxSim late interaction and dense cosine similarity.\n\nClick to see the complete 🤗 `transformers` example snippet\n\n```bash\n# accelerate is an optional dependency needed only when using device_map=\"auto\".\npip install -U accelerate \"transformers @ git+https://github.com/huggingface/transformers.git@main\" \"sentence-transformers>=6.0.0\"\n```\n\n```python\nfrom typing import Any, Literal\n\nimport requests\nimport torch\nfrom PIL import Image\nfrom sentence_transformers.util import cos_sim, mean_maxsim\n\nfrom transformers import BatchFeature, NeoMMEForRetrieval, NeoMMEProcessor\n\n\ndef encode(\n messages: list[list[dict[str, Any]]],\n task: Literal[\"query\", \"document\"],\n) -> BatchFeature:\n return processor.apply_chat_template(\n messages,\n task=task,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n processor_kwargs={\"padding\": \"longest\"},\n )\n\n\nmodel_name = \"Hcompany/NeoMME-260M-Retriever\"\nprocessor = NeoMMEProcessor.from_pretrained(model_name)\nmodel = NeoMMEForRetrieval.from_pretrained(model_name, device_map=\"auto\")\n\n# Document images (our corpus)\nimage_urls = [\n \"https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/shift_kazakhstan.jpg?raw=true\",\n \"https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/energy_electricity_generation.jpg?raw=true\",\n]\ndocuments = [Image.open(requests.get(url, stream=True).raw) for url in image_urls]\n\n# Queries\nqueries = [\n \"Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?\",\n \"Which hour of the day had the highest overall electricity generation in 2019?\",\n]\n\ndocument_messages = [\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": document}]}] for document in documents\n]\nquery_messages = [[{\"role\": \"user\", \"content\": query}] for query in queries]\n\ninputs_documents = encode(document_messages, \"document\").to(model.device)\ninputs_text = encode(query_messages, \"query\").to(model.device)\n\nwith torch.inference_mode():\n document_outputs = model(**inputs_documents)\n query_outputs = model(**inputs_text)\n\nlate_scores = mean_maxsim(\n query_outputs.embeddings,\n document_outputs.embeddings,\n a_mask=inputs_text[\"attention_mask\"],\n b_mask=inputs_documents[\"attention_mask\"],\n)\ndense_scores = cos_sim(query_outputs.dense_embeddings, document_outputs.dense_embeddings)\n\n# Expected: late_scores[0, 0] > late_scores[0, 1] and late_scores[1, 1] > late_scores[1, 0].\nprint(late_scores, dense_scores)\n```\n\n### Fine-tuning with Sentence Transformers\n\nWe provide separate [dense](https://huggingface.co/Hcompany/NeoMME-260M-Retriever-ST-dense) and [late-interaction](https://huggingface.co/Hcompany/NeoMME-260M-Retriever-ST-late) checkpoints for fine-tuning with [Sentence Transformers v6](https://huggingface.co/blog/multi-vector-encoder). Following the same pattern as text encoders such as ModernBERT, Sentence Transformers loads the backbone through [`NeoMMEModel`](https://huggingface.co/docs/transformers/main/en/model_doc/neomme#transformers.NeoMMEModel) rather than the dual-head [`NeoMMEForRetrieval`](https://huggingface.co/docs/transformers/main/en/model_doc/neomme#transformers.NeoMMEForRetrieval) class. Sentence Transformers currently supports one retrieval head per model, so each checkpoint lets you fine-tune the dense or late-interaction head independently. To train both heads together, use `NeoMMEForRetrieval` with a custom `Trainer`.\n\n### From retrieval to visual RAG\n\nVisual document retrieval can be used as the first stage of a visual retrieval-augmented generation (RAG) system. Unlike text RAG, which retrieves extracted text chunks, visual RAG retrieves the original page images and sends them to a visual language model. The model can then use tables, plots, diagrams, and page layout that text extraction may flatten or omit. Here is how visual RAG works:\n\n1. Indexing: Convert each PDF page to an image, generate an embedding with a retrieval model, and store the embeddings in a vector store.\n2. Retrieval: Generate an embedding for the user's query with the same model and retrieve the top-k most relevant pages.\n3. Generation: Append the images after the query in the chat message (*e.g.*, `{query}{img_1}{img_2}...{img_k}`) and send it to a VLM to generate the answer.\n\nYou can test visual RAG directly with *NeoMME*-Retriever in our HF Space: 🤗 [tonywu71/neomme-retriever-demo](https://huggingface.co/spaces/tonywu71/neomme-retriever-demo).\n\n## Conclusion\n\n*NeoMME* replaces separate pretrained image and text encoders with one long-context bidirectional Transformer. We train it from scratch to process both multilingual text tokens and raw 32×32 image patches.\n\n*NeoMME*-Retriever is a fine-tuned version of *NeoMME* for visual document retrieval. One forward pass produces both dense and late-interaction representations. The 260M model outperforms all evaluated models strictly below 800M parameters and, at a matched 2048×2048 input size, encodes pages at about 2× ColModernVBERT's throughput. To reduce the large storage footprint of late-interaction embeddings for high-resolution documents, we experimented with hierarchical token pooling and asymmetric quantization and managed to reduce the late-interaction embeddings from roughly 1.5 MB to 6 kB per page, a 255× compression, while retaining more than 95% of the baseline nDCG@10.\n\nWe release all *NeoMME* model checkpoints and a day-zero Hugging Face Transformers implementation to allow practitioners to build efficient multimodal and multilingual representation models on top of our work.\n\n## Acknowledgements\n\n*NeoMME* began as a side quest between two good friends. We worked with limited time and compute, and we decided to share the results so the community can build on them. We thank H Company for supporting the work and providing the compute used to train *NeoMME*.\n\n## Citation\n\n```bibtex\n@misc{lac2026neommesingletowermultimodalnativemultilingual,\n title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},\n author={Aurélien Lac and Tony Wu},\n year={2026},\n eprint={2609.01657},\n archivePrefix={arXiv},\n primaryClass={cs.IR},\n url={https://arxiv.org/abs/2609.01657},\n}\n```\n", "authors": [], "captured_at": "2026-10-19T01:26:53.998049939+00:00", "category": "ai", "content_html": "
\"Hugging \"Hugging \"arXiv\"

\n

\"NeoMME

\n

TL;DR

\n

We introduce NeoMME, a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models, NeoMME does not use a separate pretrained vision tower or a causal language model. A single bidirectional Transformer processes both text tokens and raw image patches, and we train the entire model from scratch with a masked discrete-diffusion objective.

\n

We fine-tuned NeoMME for visual document retrieval using ColPali's page-image approach. NeoMME-Retriever returns dense and late-interaction embeddings in one forward pass. Both model sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and model size. At a matched 2048×2048 image input size on an NVIDIA L40S GPU, the 260M model encodes about 51 pages per second, or about twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from roughly 1.5 MB to 6 kB per page (255× smaller) while retaining more than 95% of baseline nDCG@10.

\n

NeoMME is available in Hugging Face Transformers. We release all model checkpoints under the Apache 2.0 license.

\n\n

Why another multimodal encoder?

\n

Many recent visual document retrievers are adapted from pretrained generative visual language models. A separately pretrained vision encoder produces visual features, which a projector maps into the language model's input space. A causal decoder then processes the combined image and text representations. Retrieval, classification, and token labeling do not generate text autoregressively, so they do not require a causal decoder or the parameter and compute overhead of this architecture.

\n

ModernBERT brought efficient architecture and training improvements to bidirectional encoders. For visual document retrieval, ModernVBERT applied a bidirectional ModernBERT-style text encoder while retaining a separate pretrained SigLIP2 vision tower. We wanted to push this even further by designing and training a multimodal encoder without having to carry over the parameter and compute overhead of a VLM.

\n

NeoMME (pronounced "nee-oh-me", IPA /ˈniː.oʊ.mi/) is a multilingual, multimodal foundation encoder that generates vector representations for input text and/or images using a single Transformer encoder. It is not based on an existing pretrained vision tower, text encoder, or text decoder.

\n

\"Comparison

Unlike dual-tower and VLM encoders, NeoMME processes image patches and text tokens in one bidirectional Transformer, without a pretrained vision tower or a pretrained text encoder or decoder.
\n

Images and text use the same computational path, so NeoMME can more easily support pretraining, fine-tuning, parallelization, and serving across both modalities.

\n

NeoMME encoder backbone

\n

One Transformer for images and text

\n

NeoMME comes in two sizes, 260M and 800M. Both variants share the same architecture:

\n\n
\"Alternating
Alternating sliding-window and global-attention layers in the NeoMME encoder stack.
\n

Learning from images through masked text

\n

We pretrain NeoMME from scratch as a discrete masked-diffusion text denoiser. For each text-only example, we sample a corruption rate uniformly between 0 and 1. Each eligible text token is then independently masked at that rate.

\n

Multimodal examples use corruption rates between 0.3 and 1. The image patches remain visible while NeoMME reconstructs masked text. With light masking, the model can often recover a missing word from the surrounding text alone. For example, "cat" is a plausible completion of "The [MASK] sat on the mat," even without an image. But high masking forces the model to learn image-grounded descriptions with little to no signal from the non-masked input text tokens.

\n
\"Effect
Higher text corruption removes language-only shortcuts and encourages NeoMME to use visible image evidence.
\n

Pretraining mixes multilingual text, code, mathematics, natural images, and document images. Each model processes about 524 billion packed input tokens, including 290 billion tokens from text-only examples. This text budget is relatively small compared with ModernBERT's 2 trillion training token budget. Hence, we chose the NorMuon optimizer to improve data efficiency during training.

\n

NeoMME-Retriever

\n

To get a meaningful downstream evaluation of the backbone, we fine-tune NeoMME for visual document retrieval using the page-image methodology introduced by ColPali. While traditional text-based retrieval consists of retrieving text chunks, NeoMME-Retriever ranks document page screenshots and bypasses all the preprocessing OCR steps necessary to extract text from PDFs. Treating the pages as images preserves layout, charts, tables, font type and size, and other visual clues that cannot be captured even by a perfect OCR model.

\n

A dual-head design for dense and late-interaction retrieval

\n

NeoMME-Retriever reuses the NeoMME backbone but adds two jointly trained heads on top of it for retrieval:

\n\n
\"Late-interaction
Late-interaction and dense retrieval heads for both NeoMME model sizes.
\n
\n

Omar Khattab, who introduced late-interaction in ColBERT, explains why the term is more precise than "multi-vector." It describes the granularity and learnability of the scoring function, not simply the number of stored vectors.

\n

To learn more about late-interaction, we recommend reading this crash course by Amélie Chatelain.

\n
\n

One NeoMME-Retriever forward pass returns both representations, which gives you flexibility no matter your use case and infrastructure. We recommend using late-interaction embeddings in general since they are more powerful and can be used easily with open-source libraries like NextPlaid. However, if you have a very large corpora, you can run a single forward pass with NeoMME-Retriever to get the dense embedding, retrieve a small number of documents through an ANN index, and then use late-interaction to rerank the retrieved candidates.

\n

Competitive retrieval at compact model sizes

\n

We report nDCG@10 on ViDoRe v3. NeoMME-Retriever-260M reaches 0.523, the highest score among evaluated models strictly below 800M parameters. It is within 0.002 nDCG@10 of ColQwen2.5 while using about 14× fewer parameters. NeoMME-Retriever-800M reaches 0.556, within 0.009 nDCG@10 of the similarly sized Vultron Retriever Flash (0.8B). Both NeoMME-Retriever models lie on the model-size Pareto frontier.

\n
\"ViDoRe
ViDoRe v3 nDCG@10 versus model size.
\n

ViDoRe v1 and v2 use nDCG@5. On both benchmarks, NeoMME-Retriever-260M outperforms ColModernVBERT and the twice-larger ColSmol-500M. NeoMME-Retriever-800M outperforms ColPali v1.3 while using 3.6 times fewer parameters.

\n

Visual document retrieval performance on the ViDoRe benchmarks.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Model detailsViDoRe (nDCG@k)
ColModernVBERT250M0.261†0.407‡0.806‡
ColSmol-256M†256M0.2070.3480.797
NeoMME-260M‡260M0.5230.5220.860
ColSmol-500M500M0.340‡0.455†0.825†
Vultron Flash†850M0.5650.6040.882
NeoMME-800M‡800M0.5560.5590.874
ColQwen2.5-v0.2†3.75B0.5240.6010.895
ColPali v1.3†2.92B0.4300.5470.848
\n

† Scores from MTEB. ‡ Results from our own evaluations.

\n

Making high-resolution retrieval practical for late-interaction

\n

Late-interaction storage scales linearly with the number of vectors in the output embedding. Higher-resolution images contain more patches, so they produce larger embeddings. For example, a 2048×2048 square page produces embeddings containing 4,200 vectors with NeoMME-Retriever, or about 2.1 MB in float32. Across the ViDoRe v3 benchmark, the measured average is about 1.5 MB per document.

\n

To reduce the storage footprint of the late-interaction index, we combine two complementary compression methods:

\n
    \n
  1. Hierarchical token pooling clusters similar document vectors in a given multi-vector embedding and replaces each cluster with its mean, hence reducing the number of vectors stored for each page.
    2. \n
  2. Asymmetric quantization quantizes document embeddings to int8 or binary. Because query embeddings are not stored and only generated on-the-fly, they can be kept at a higher precision.
    3. \n
\n

We tested this setup on ViDoRe v3. With a pooling factor 10 and int8 queries and documents, storage decreased from about 1.5 MB to 39 kB per page, a 39× reduction, while keeping more than 99% of the baseline nDCG@10. A more aggressive configuration uses pooling factor 8, int8 queries, and binary documents. That version uses 6 kB per page (255× smaller) and keeps more than 95% of the original retrieval quality.

\n
\"Quality
Quality and storage frontier for the NeoMME-260M late-interaction index on ViDoRe v3. Labels show pool factor, retained quality, compression, and storage.
\n

Users can pick a compression setting from that frontier based on storage budget and required retrieval quality.

\n

Fast inference for cheaper multimodal corpus indexing

\n

Before you can search a corpus, a retriever model must turn your documents into embeddings, which will be stored in a vector store like Qdrant, Weaviate, or Milvus. Faster encoding makes building and adding new documents to the index faster, thus reducing the GPU uptime and compute cost required.

\n

So we measured image encoding speeds for NeoMME-Retriever against other multimodal document retrievers. We used preprocessed image tensors and calibrated the batch size separately for each model and image size. At a matched 2048×2048 input size on one NVIDIA L40S, NeoMME-Retriever-260M encodes about 51 pages per second, nearly twice ColModernVBERT's 26 pages per second. Both 260M and 800M NeoMME-Retriever models are also faster than the other models we compared on smaller input images.

\n
\"Document-encoding
Document-encoding throughput by retriever and input resolution on one NVIDIA L40S.
\n

Try NeoMME-Retriever yourself!

\n

NeoMME-Retriever (260M and 800M) returns dense and multi-vector embeddings together. The example below scores two text queries against two document-page images with MeanMaxSim late interaction and dense cosine similarity.

\n

Click to see the complete 🤗 transformers example snippet

\n
# accelerate is an optional dependency needed only when using device_map="auto".\npip install -U accelerate "transformers @ git+https://github.com/huggingface/transformers.git@main" "sentence-transformers>=6.0.0"\n
\n
from typing import Any, Literal\n\nimport requests\nimport torch\nfrom PIL import Image\nfrom sentence_transformers.util import cos_sim, mean_maxsim\n\nfrom transformers import BatchFeature, NeoMMEForRetrieval, NeoMMEProcessor\n\n\ndef encode(\n    messages: list[list[dict[str, Any]]],\n    task: Literal["query", "document"],\n) -> BatchFeature:\n    return processor.apply_chat_template(\n        messages,\n        task=task,\n        tokenize=True,\n        return_dict=True,\n        return_tensors="pt",\n        processor_kwargs={"padding": "longest"},\n    )\n\n\nmodel_name = "Hcompany/NeoMME-260M-Retriever"\nprocessor = NeoMMEProcessor.from_pretrained(model_name)\nmodel = NeoMMEForRetrieval.from_pretrained(model_name, device_map="auto")\n\n# Document images (our corpus)\nimage_urls = [\n    "https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/shift_kazakhstan.jpg?raw=true",\n    "https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/energy_electricity_generation.jpg?raw=true",\n]\ndocuments = [Image.open(requests.get(url, stream=True).raw) for url in image_urls]\n\n# Queries\nqueries = [\n    "Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?",\n    "Which hour of the day had the highest overall electricity generation in 2019?",\n]\n\ndocument_messages = [\n    [{"role": "user", "content": [{"type": "image", "image": document}]}] for document in documents\n]\nquery_messages = [[{"role": "user", "content": query}] for query in queries]\n\ninputs_documents = encode(document_messages, "document").to(model.device)\ninputs_text = encode(query_messages, "query").to(model.device)\n\nwith torch.inference_mode():\n    document_outputs = model(**inputs_documents)\n    query_outputs = model(**inputs_text)\n\nlate_scores = mean_maxsim(\n    query_outputs.embeddings,\n    document_outputs.embeddings,\n    a_mask=inputs_text["attention_mask"],\n    b_mask=inputs_documents["attention_mask"],\n)\ndense_scores = cos_sim(query_outputs.dense_embeddings, document_outputs.dense_embeddings)\n\n# Expected: late_scores[0, 0] > late_scores[0, 1] and late_scores[1, 1] > late_scores[1, 0].\nprint(late_scores, dense_scores)\n
\n

Fine-tuning with Sentence Transformers

\n

We provide separate dense and late-interaction checkpoints for fine-tuning with Sentence Transformers v6. Following the same pattern as text encoders such as ModernBERT, Sentence Transformers loads the backbone through NeoMMEModel rather than the dual-head NeoMMEForRetrieval class. Sentence Transformers currently supports one retrieval head per model, so each checkpoint lets you fine-tune the dense or late-interaction head independently. To train both heads together, use NeoMMEForRetrieval with a custom Trainer.

\n

From retrieval to visual RAG

\n

Visual document retrieval can be used as the first stage of a visual retrieval-augmented generation (RAG) system. Unlike text RAG, which retrieves extracted text chunks, visual RAG retrieves the original page images and sends them to a visual language model. The model can then use tables, plots, diagrams, and page layout that text extraction may flatten or omit. Here is how visual RAG works:

\n
    \n
  1. Indexing: Convert each PDF page to an image, generate an embedding with a retrieval model, and store the embeddings in a vector store.
    2. \n
  2. Retrieval: Generate an embedding for the user's query with the same model and retrieve the top-k most relevant pages.
    3. \n
  3. Generation: Append the images after the query in the chat message (e.g., {query}{img_1}{img_2}...{img_k}) and send it to a VLM to generate the answer.
    4. \n
\n

You can test visual RAG directly with NeoMME-Retriever in our HF Space: 🤗 tonywu71/neomme-retriever-demo.

\n

Conclusion

\n

NeoMME replaces separate pretrained image and text encoders with one long-context bidirectional Transformer. We train it from scratch to process both multilingual text tokens and raw 32×32 image patches.

\n

NeoMME-Retriever is a fine-tuned version of NeoMME for visual document retrieval. One forward pass produces both dense and late-interaction representations. The 260M model outperforms all evaluated models strictly below 800M parameters and, at a matched 2048×2048 input size, encodes pages at about 2× ColModernVBERT's throughput. To reduce the large storage footprint of late-interaction embeddings for high-resolution documents, we experimented with hierarchical token pooling and asymmetric quantization and managed to reduce the late-interaction embeddings from roughly 1.5 MB to 6 kB per page, a 255× compression, while retaining more than 95% of the baseline nDCG@10.

\n

We release all NeoMME model checkpoints and a day-zero Hugging Face Transformers implementation to allow practitioners to build efficient multimodal and multilingual representation models on top of our work.

\n

Acknowledgements

\n

NeoMME began as a side quest between two good friends. We worked with limited time and compute, and we decided to share the results so the community can build on them. We thank H Company for supporting the work and providing the compute used to train NeoMME.

\n

Citation

\n
@misc{lac2026neommesingletowermultimodalnativemultilingual,\n      title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},\n      author={Aurélien Lac and Tony Wu},\n      year={2026},\n      eprint={2609.01657},\n      archivePrefix={arXiv},\n      primaryClass={cs.IR},\n      url={https://arxiv.org/abs/2609.01657},\n}\n
\n", "content_markdown": "[![Hugging Face](https://img.shields.io/badge/Collection-FFD21E?style=for-the-badge&logo=huggingface&logoColor=000)](https://hf.co/collections/Hcompany/neomme) [![Hugging Face Paper](https://img.shields.io/badge/HF_Paper-FFD21E?style=for-the-badge&logo=huggingface&logoColor=000)](https://huggingface.co/papers/2609.01657) [![arXiv](https://img.shields.io/badge/arXiv-2609.01657-b31b1b.svg?style=for-the-badge)](https://arxiv.org/abs/2609.01657)\n\n![NeoMME logo](https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/assets/neomme/neomme_logo.webp?raw=true)\n\n## TL;DR\n\nWe introduce ***NeoMME***, a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models, *NeoMME* does not use a separate pretrained vision tower or a causal language model. A single bidirectional Transformer processes both text tokens and raw image patches, and we train the entire model from scratch with a masked discrete-diffusion objective.\n\nWe fine-tuned *NeoMME* for visual document retrieval using ColPali's page-image approach. *NeoMME*-Retriever returns dense and late-interaction embeddings in one forward pass. Both model sizes lie on the ViDoRe v3 Pareto frontier for nDCG@10 and model size. At a matched 2048×2048 image input size on an NVIDIA L40S GPU, the 260M model encodes about 51 pages per second, or about twice ColModernVBERT's throughput. Hierarchical token pooling and asymmetric quantization reduce late-interaction index storage from roughly 1.5 MB to 6 kB per page (255× smaller) while retaining more than 95% of baseline nDCG@10.\n\n*NeoMME* is available in Hugging Face Transformers. We release all model checkpoints under the Apache 2.0 license.\n\n- 🤗 [*NeoMME* collection](https://hf.co/collections/Hcompany/neomme)\n- 📄 [Technical report](https://huggingface.co/papers/2609.01657)\n- 🔎 [Visual RAG demo](https://huggingface.co/spaces/tonywu71/neomme-retriever-demo)\n\n## Why another multimodal encoder?\n\nMany recent visual document retrievers are adapted from pretrained generative visual language models. A separately pretrained vision encoder produces visual features, which a projector maps into the language model's input space. A causal decoder then processes the combined image and text representations. Retrieval, classification, and token labeling do not generate text autoregressively, so they do not require a causal decoder or the parameter and compute overhead of this architecture.\n\n[ModernBERT](https://huggingface.co/blog/modernbert) brought efficient architecture and training improvements to bidirectional encoders. For visual document retrieval, [ModernVBERT](https://huggingface.co/blog/paultltc/modernvbert) applied a bidirectional ModernBERT-style text encoder while retaining a separate pretrained SigLIP2 vision tower. We wanted to push this even further by designing and training a multimodal encoder without having to carry over the parameter and compute overhead of a VLM.\n\n***NeoMME*** (pronounced \"nee-oh-me\", IPA /ˈniː.oʊ.mi/) is a multilingual, multimodal foundation encoder that generates vector representations for input text and/or images using a single Transformer encoder. It is not based on an existing pretrained vision tower, text encoder, or text decoder.\n\n![Comparison of dual-tower, VLM, ModernVBERT, and NeoMME input paths](https://github.com/tonywu71/colpali-cookbooks/blob/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa/assets/neomme/architecture.webp?raw=true)\\\nUnlike dual-tower and VLM encoders, *NeoMME* processes image patches and text tokens in one bidirectional Transformer, without a pretrained vision tower or a pretrained text encoder or decoder.\n\nImages and text use the same computational path, so *NeoMME* can more easily support pretraining, fine-tuning, parallelization, and serving across both modalities.\n\n## *NeoMME* encoder backbone\n\n### One Transformer for images and text\n\n*NeoMME* comes in two sizes, [260M](https://huggingface.co/Hcompany/NeoMME-260M) and [800M](https://huggingface.co/Hcompany/NeoMME-800M). Both variants share the same architecture:\n\n- **Native multimodal inputs:** text inputs use factorized token embeddings, while images are divided into a grid of non-overlapping 32×32 patches and projected with a small MLP. Both enter the same Transformer encoder.\n- **Dynamic image resolution:** images keep their aspect ratio and size. This allows the model to use more tokens on a high-resolution, information-dense document page than on a smaller image with less content.\n- **Long bidirectional context:** both models have a context length of 16,384 tokens (enough for up to two standard 3840×2160 4K UHD images). Most layers use symmetric sliding-window attention, while every sixth layer and the final layer use global attention.\n- **A modern encoder stack:** *NeoMME* uses recent encoder improvements such as grouped-query attention, query-key normalization, gated attention, 2D rotary position embeddings, and squared-ReLU MLPs, among others.\n- **Multilingual text:** we trained a BPE tokenizer with a 131k-token vocabulary from scratch on multilingual text, code, mathematics, and machine-produced image transcripts.\n\n![Alternating sliding-window and global-attention layers in NeoMME](https://github.com/tonywu71/colpali-cookbooks/blob/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa/assets/neomme/encoder-stack.webp?raw=true)\\\nAlternating sliding-window and global-attention layers in the *NeoMME* encoder stack.\n\n### Learning from images through masked text\n\nWe pretrain *NeoMME* from scratch as a discrete masked-diffusion text denoiser. For each text-only example, we sample a corruption rate uniformly between 0 and 1. Each eligible text token is then independently masked at that rate.\n\nMultimodal examples use corruption rates between 0.3 and 1. The image patches remain visible while *NeoMME* reconstructs masked text. With light masking, the model can often recover a missing word from the surrounding text alone. For example, \"cat\" is a plausible completion of \"The \\[MASK\\] sat on the mat,\" even without an image. But high masking forces the model to learn image-grounded descriptions with little to no signal from the non-masked input text tokens.\n\n![Effect of text corruption on the textual and visual evidence available to NeoMME](https://github.com/tonywu71/colpali-cookbooks/blob/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa/assets/neomme/masked-diffusion.webp?raw=true)\\\nHigher text corruption removes language-only shortcuts and encourages *NeoMME* to use visible image evidence.\n\nPretraining mixes multilingual text, code, mathematics, natural images, and document images. Each model processes about 524 billion packed input tokens, including 290 billion tokens from text-only examples. This text budget is relatively small compared with ModernBERT's 2 trillion training token budget. Hence, we chose the NorMuon optimizer to improve data efficiency during training.\n\n## *NeoMME*-Retriever\n\nTo get a meaningful downstream evaluation of the backbone, we fine-tune *NeoMME* for visual document retrieval using the page-image methodology introduced by [ColPali](https://huggingface.co/blog/manu/colpali). While traditional text-based retrieval consists of retrieving text chunks, *NeoMME*-Retriever ranks document page screenshots and bypasses all the preprocessing OCR steps necessary to extract text from PDFs. Treating the pages as images preserves layout, charts, tables, font type and size, and other visual clues that cannot be captured even by a perfect OCR model.\n\n### A dual-head design for dense and late-interaction retrieval\n\n*NeoMME*-Retriever reuses the *NeoMME* backbone but adds two jointly trained heads on top of it for retrieval:\n\n- The dense head averages the backbone's hidden state vectors into a normalized vector (mean pooling). Dense embeddings are most common today: they are compact and work naturally with approximate nearest-neighbor (ANN) techniques for fast retrieval.\n- The late-interaction head projects each text token or image patch from the backbone's output hidden states to a 128-dimensional normalized vector. Compared to dense embeddings, the finer granularity preserves local matches between individual query tokens and image regions.\n\n![Late-interaction and dense retrieval heads for NeoMME](https://github.com/tonywu71/colpali-cookbooks/blob/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa/assets/neomme/retrieval-heads.webp?raw=true)\\\nLate-interaction and dense retrieval heads for both *NeoMME* model sizes.\n\n> Omar Khattab, who introduced late-interaction in ColBERT, [explains why](https://x.com/lateinteraction/status/1894696983077785980) the term is more precise than \"multi-vector.\" It describes the granularity and learnability of the scoring function, not simply the number of stored vectors.\n>\n> To learn more about late-interaction, we recommend reading [this crash course](https://meet.ameliechatelain.com/lectures/multi-vector-search/) by Amélie Chatelain.\n\nOne *NeoMME*-Retriever forward pass returns both representations, which gives you flexibility no matter your use case and infrastructure. We recommend using late-interaction embeddings in general since they are more powerful and can be used easily with open-source libraries like [NextPlaid](https://github.com/lightonai/next-plaid). However, if you have a very large corpora, you can run a single forward pass with *NeoMME*-Retriever to get the dense embedding, retrieve a small number of documents through an ANN index, and then use late-interaction to rerank the retrieved candidates.\n\n### Competitive retrieval at compact model sizes\n\nWe report nDCG@10 on ViDoRe v3. *NeoMME*-Retriever-260M reaches 0.523, the highest score among evaluated models strictly below 800M parameters. It is within 0.002 nDCG@10 of ColQwen2.5 while using about 14× fewer parameters. *NeoMME*-Retriever-800M reaches 0.556, within 0.009 nDCG@10 of the similarly sized [Vultron Retriever Flash (0.8B)](https://huggingface.co/vultr/VultronRetrieverFlash-Qwen3.5-0.8B). Both *NeoMME*-Retriever models lie on the model-size Pareto frontier.\n\n![ViDoRe v3 nDCG@10 versus model size](https://github.com/tonywu71/colpali-cookbooks/blob/c5e737e7e474363822b6f2d95290c0120e0fa993/assets/neomme/vidore-v3-model-size.webp?raw=true)\\\nViDoRe v3 nDCG@10 versus model size.\n\nViDoRe v1 and v2 use nDCG@5. On both benchmarks, *NeoMME*-Retriever-260M outperforms ColModernVBERT and the twice-larger ColSmol-500M. *NeoMME*-Retriever-800M outperforms ColPali v1.3 while using 3.6 times fewer parameters.\n\nVisual document retrieval performance on the ViDoRe benchmarks.\n| Model details | ViDoRe (nDCG@*k*) | | | |\n| ---------------------------------------------------------------------------------- | ----------------- | --------- | --------- | --------- |\n| [ColModernVBERT](https://huggingface.co/ModernVBERT/colmodernvbert) | 250M | 0.261† | 0.407‡ | 0.806‡ |\n| [ColSmol-256M](https://huggingface.co/vidore/colSmol-256M)† | 256M | 0.207 | 0.348 | 0.797 |\n| [*NeoMME*-260M](https://huggingface.co/Hcompany/NeoMME-260M-Retriever)‡ | 260M | **0.523** | **0.522** | **0.860** |\n| [ColSmol-500M](https://huggingface.co/vidore/colSmol-500M) | 500M | 0.340‡ | 0.455† | 0.825† |\n| [Vultron Flash](https://huggingface.co/vultr/VultronRetrieverFlash-Qwen3.5-0.8B)† | 850M | **0.565** | **0.604** | **0.882** |\n| [*NeoMME*-800M](https://huggingface.co/Hcompany/NeoMME-800M-Retriever)‡ | 800M | 0.556 | 0.559 | 0.874 |\n| [ColQwen2.5-v0.2](https://huggingface.co/vidore/colqwen2.5-v0.2)† | 3.75B | **0.524** | **0.601** | **0.895** |\n| [ColPali v1.3](https://huggingface.co/vidore/colpali-v1.3)† | 2.92B | 0.430 | 0.547 | 0.848 |\n\n† Scores from MTEB. ‡ Results from our own evaluations.\n\n### Making high-resolution retrieval practical for late-interaction\n\nLate-interaction storage scales linearly with the number of vectors in the output embedding. Higher-resolution images contain more patches, so they produce larger embeddings. For example, a 2048×2048 square page produces embeddings containing 4,200 vectors with *NeoMME*-Retriever, or about 2.1 MB in float32. Across the ViDoRe v3 benchmark, the measured average is about 1.5 MB per document.\n\nTo reduce the storage footprint of the late-interaction index, we combine two complementary compression methods:\n\n1. [Hierarchical token pooling](https://www.answer.ai/posts/colbert-pooling.html) clusters similar document vectors in a given multi-vector embedding and replaces each cluster with its mean, hence reducing the number of vectors stored for each page.\n2. [Asymmetric quantization](https://www.mixedbread.com/blog/asymmetric-quant) quantizes document embeddings to int8 or binary. Because query embeddings are not stored and only generated on-the-fly, they can be kept at a higher precision.\n\nWe tested this setup on ViDoRe v3. With a pooling factor 10 and int8 queries and documents, storage decreased from about 1.5 MB to 39 kB per page, a 39× reduction, while keeping more than 99% of the baseline nDCG@10. A more aggressive configuration uses pooling factor 8, int8 queries, and binary documents. That version uses 6 kB per page (255× smaller) and keeps more than 95% of the original retrieval quality.\n\n![Quality and storage frontier for the NeoMME-260M late-interaction index](https://github.com/tonywu71/colpali-cookbooks/blob/481c6ec8866f6f5d39d2d9dd06b5547c2d9a67fa/assets/neomme/compression-frontier-260m.webp?raw=true)\\\nQuality and storage frontier for the *NeoMME*-260M late-interaction index on ViDoRe v3. Labels show pool factor, retained quality, compression, and storage.\n\nUsers can pick a compression setting from that frontier based on storage budget and required retrieval quality.\n\n#### Fast inference for cheaper multimodal corpus indexing\n\nBefore you can search a corpus, a retriever model must turn your documents into embeddings, which will be stored in a vector store like Qdrant, Weaviate, or Milvus. Faster encoding makes building and adding new documents to the index faster, thus reducing the GPU uptime and compute cost required.\n\nSo we measured image encoding speeds for *NeoMME*-Retriever against other multimodal document retrievers. We used preprocessed image tensors and calibrated the batch size separately for each model and image size. At a matched 2048×2048 input size on one NVIDIA L40S, *NeoMME*-Retriever-260M encodes about 51 pages per second, nearly twice ColModernVBERT's 26 pages per second. Both 260M and 800M *NeoMME*-Retriever models are also faster than the other models we compared on smaller input images.\n\n![Document-encoding throughput across image resolutions on an NVIDIA L40S](https://github.com/tonywu71/colpali-cookbooks/blob/c5e737e7e474363822b6f2d95290c0120e0fa993/assets/neomme/indexing-throughput-l40s.webp?raw=true)\\\nDocument-encoding throughput by retriever and input resolution on one NVIDIA L40S.\n\n### Try *NeoMME*-Retriever yourself!\n\n*NeoMME*-Retriever ([260M](https://huggingface.co/Hcompany/NeoMME-260M-Retriever) and [800M](https://huggingface.co/Hcompany/NeoMME-800M-Retriever)) returns dense and multi-vector embeddings together. The example below scores two text queries against two document-page images with MeanMaxSim late interaction and dense cosine similarity.\n\nClick to see the complete 🤗 `transformers` example snippet\n\n```bash\n# accelerate is an optional dependency needed only when using device_map=\"auto\".\npip install -U accelerate \"transformers @ git+https://github.com/huggingface/transformers.git@main\" \"sentence-transformers>=6.0.0\"\n```\n\n```python\nfrom typing import Any, Literal\n\nimport requests\nimport torch\nfrom PIL import Image\nfrom sentence_transformers.util import cos_sim, mean_maxsim\n\nfrom transformers import BatchFeature, NeoMMEForRetrieval, NeoMMEProcessor\n\n\ndef encode(\n messages: list[list[dict[str, Any]]],\n task: Literal[\"query\", \"document\"],\n) -> BatchFeature:\n return processor.apply_chat_template(\n messages,\n task=task,\n tokenize=True,\n return_dict=True,\n return_tensors=\"pt\",\n processor_kwargs={\"padding\": \"longest\"},\n )\n\n\nmodel_name = \"Hcompany/NeoMME-260M-Retriever\"\nprocessor = NeoMMEProcessor.from_pretrained(model_name)\nmodel = NeoMMEForRetrieval.from_pretrained(model_name, device_map=\"auto\")\n\n# Document images (our corpus)\nimage_urls = [\n \"https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/shift_kazakhstan.jpg?raw=true\",\n \"https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/examples/data/energy_electricity_generation.jpg?raw=true\",\n]\ndocuments = [Image.open(requests.get(url, stream=True).raw) for url in image_urls]\n\n# Queries\nqueries = [\n \"Quelle partie de la production pétrolière du Kazakhstan provient de champs en mer ?\",\n \"Which hour of the day had the highest overall electricity generation in 2019?\",\n]\n\ndocument_messages = [\n [{\"role\": \"user\", \"content\": [{\"type\": \"image\", \"image\": document}]}] for document in documents\n]\nquery_messages = [[{\"role\": \"user\", \"content\": query}] for query in queries]\n\ninputs_documents = encode(document_messages, \"document\").to(model.device)\ninputs_text = encode(query_messages, \"query\").to(model.device)\n\nwith torch.inference_mode():\n document_outputs = model(**inputs_documents)\n query_outputs = model(**inputs_text)\n\nlate_scores = mean_maxsim(\n query_outputs.embeddings,\n document_outputs.embeddings,\n a_mask=inputs_text[\"attention_mask\"],\n b_mask=inputs_documents[\"attention_mask\"],\n)\ndense_scores = cos_sim(query_outputs.dense_embeddings, document_outputs.dense_embeddings)\n\n# Expected: late_scores[0, 0] > late_scores[0, 1] and late_scores[1, 1] > late_scores[1, 0].\nprint(late_scores, dense_scores)\n```\n\n### Fine-tuning with Sentence Transformers\n\nWe provide separate [dense](https://huggingface.co/Hcompany/NeoMME-260M-Retriever-ST-dense) and [late-interaction](https://huggingface.co/Hcompany/NeoMME-260M-Retriever-ST-late) checkpoints for fine-tuning with [Sentence Transformers v6](https://huggingface.co/blog/multi-vector-encoder). Following the same pattern as text encoders such as ModernBERT, Sentence Transformers loads the backbone through [`NeoMMEModel`](https://huggingface.co/docs/transformers/main/en/model_doc/neomme#transformers.NeoMMEModel) rather than the dual-head [`NeoMMEForRetrieval`](https://huggingface.co/docs/transformers/main/en/model_doc/neomme#transformers.NeoMMEForRetrieval) class. Sentence Transformers currently supports one retrieval head per model, so each checkpoint lets you fine-tune the dense or late-interaction head independently. To train both heads together, use `NeoMMEForRetrieval` with a custom `Trainer`.\n\n### From retrieval to visual RAG\n\nVisual document retrieval can be used as the first stage of a visual retrieval-augmented generation (RAG) system. Unlike text RAG, which retrieves extracted text chunks, visual RAG retrieves the original page images and sends them to a visual language model. The model can then use tables, plots, diagrams, and page layout that text extraction may flatten or omit. Here is how visual RAG works:\n\n1. Indexing: Convert each PDF page to an image, generate an embedding with a retrieval model, and store the embeddings in a vector store.\n2. Retrieval: Generate an embedding for the user's query with the same model and retrieve the top-k most relevant pages.\n3. Generation: Append the images after the query in the chat message (*e.g.*, `{query}{img_1}{img_2}...{img_k}`) and send it to a VLM to generate the answer.\n\nYou can test visual RAG directly with *NeoMME*-Retriever in our HF Space: 🤗 [tonywu71/neomme-retriever-demo](https://huggingface.co/spaces/tonywu71/neomme-retriever-demo).\n\n## Conclusion\n\n*NeoMME* replaces separate pretrained image and text encoders with one long-context bidirectional Transformer. We train it from scratch to process both multilingual text tokens and raw 32×32 image patches.\n\n*NeoMME*-Retriever is a fine-tuned version of *NeoMME* for visual document retrieval. One forward pass produces both dense and late-interaction representations. The 260M model outperforms all evaluated models strictly below 800M parameters and, at a matched 2048×2048 input size, encodes pages at about 2× ColModernVBERT's throughput. To reduce the large storage footprint of late-interaction embeddings for high-resolution documents, we experimented with hierarchical token pooling and asymmetric quantization and managed to reduce the late-interaction embeddings from roughly 1.5 MB to 6 kB per page, a 255× compression, while retaining more than 95% of the baseline nDCG@10.\n\nWe release all *NeoMME* model checkpoints and a day-zero Hugging Face Transformers implementation to allow practitioners to build efficient multimodal and multilingual representation models on top of our work.\n\n## Acknowledgements\n\n*NeoMME* began as a side quest between two good friends. We worked with limited time and compute, and we decided to share the results so the community can build on them. We thank H Company for supporting the work and providing the compute used to train *NeoMME*.\n\n## Citation\n\n```bibtex\n@misc{lac2026neommesingletowermultimodalnativemultilingual,\n title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},\n author={Aurélien Lac and Tony Wu},\n year={2026},\n eprint={2609.01657},\n archivePrefix={arXiv},\n primaryClass={cs.IR},\n url={https://arxiv.org/abs/2609.01657},\n}\n```\n", "dateCreated": "2026-10-19T01:26:53.998049939+00:00", "date_created": "2026-09-03T13:13:48+00:00", "date_modified": "2026-09-03T13:13:48+00:00", "date_published": "2026-09-03T13:13:48+00:00", "description": "TL;DR We introduce NeoMME, a family of 260M and 800M multilingual multimodal encoders. Unlike many generative visual language models, NeoMME does not use a separate pretrained vision tower or a causal language model. A single bidirectional…", "external_url": "https://huggingface.co/blog/Hcompany/neomme", "git": { "commit": "ebb0941c5c06a2ab2377342d607535262a3e969e", "path": "items/huggingface-co/2026/09/2026-09-03-neomme-an-efficient-multimodal-native-and-multilingual.md", "permalink": "https://github.com/aymericbeaumet/aggr-instance/blob/ebb0941c5c06a2ab2377342d607535262a3e969e/items/huggingface-co/2026/09/2026-09-03-neomme-an-efficient-multimodal-native-and-multilingual.md" }, "id": "urn:aggr:item:477afbff5da0f06921c6002df289dd72497fba74", "isBasedOn": { "@id": "https://huggingface.co/blog/Hcompany/neomme" }, "mainEntity": { "@id": "https://huggingface.co/blog/Hcompany/neomme", "@type": "CreativeWork", "archivedAt": { "@id": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-09-03-neomme-an-efficient-multimodal-native-and-multilingual/#webpage" }, "dateModified": "2026-09-03T13:13:48+00:00", "datePublished": "2026-09-03T13:13:48+00:00", "headline": "NeoMME: an efficient Multimodal-native and Multilingual Encoder", "image": { "@type": "ImageObject", "caption": null, "height": 90, "url": "https://cdn-uploads.huggingface.co/production/uploads/6264f9655f6f2e14d6ac981c/GJ6FUbgFpq1x8RNOqzmz-.webp", "width": 160 }, "timeRequired": "PT11M", "url": "https://huggingface.co/blog/Hcompany/neomme", "wordCount": 2465 }, "name": "Archived snapshot: NeoMME: an efficient Multimodal-native and Multilingual Encoder", "preview": { "alt": null, "color": null, "height": 90, "placeholder": { "data_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=", "hash": "" }, "url": "https://cdn-uploads.huggingface.co/production/uploads/6264f9655f6f2e14d6ac981c/GJ6FUbgFpq1x8RNOqzmz-.webp", "width": 160 }, "reading_minutes": 11, "replicated_at": null, "source": "huggingface-co", "source_name": "Hugging Face - Blog", "summary": null, "tags": [], "temporalCoverage": "2026-10-19T01:26:53.998049939+00:00", "title": "NeoMME: an efficient Multimodal-native and Multilingual Encoder", "url": "https://aggr.aymericbeaumet.com/items/huggingface-co/2026-09-03-neomme-an-efficient-multimodal-native-and-multilingual/", "word_count": 2465 }