![\"Chart](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ocr/chart-rendering.png\")
\n\nWe have added Chandra and OlmOCR-2 to this blog, as well as OlmOCR Scores of the models đ«Ą
\n
TL;DR: The rise of powerful vision-language models has transformed document AI. Each model comes with unique strengths, making it tricky to choose the right one. Open-weight models offer better cost efficiency and privacy. To help you get started with them, weâve put together this guide.
\nIn this guide, youâll learn:
\nBy the end, youâll know how to choose the right OCR model, start building with it, and gain deeper insights into document AI. Letâs go!
\nOptical Character Recognition (OCR) is one of the earliest and longest running challenges in computer vision. Many of AIâs first practical applications focused on turning printed text into digital form.
\nWith the surge of vision-language models (VLMs), OCR has advanced significantly. Recently, many OCR models have been developed by fine-tuning existing VLMs. But todayâs capabilities extend far beyond OCR: you can retrieve documents by query or answer questions about them directly. Thanks to stronger vision features, these models can also handle low-quality scans, interpret complex elements like tables, charts, and images, and fuse text with visuals to answer open-ended questions across documents.
\nRecent models transcribe texts into a machine-readable format.
\nThe input can include:
OCR models convert them into machine-readable text that comes in many different formats like HTML, Markdown and more.
\nOn top of text, some models can also recognize:
\nSome models know where images are inside the document, extract their coordinates, and insert them appropriately between texts. Other models generate captions for images and insert them where they appear. This is especially useful if you are feeding the machine-readable output into an LLM. Example models are OlmOCR by AllenAI, or PaddleOCR-VL by PaddlePaddle.
\nModels use different machine-readable output formats, such as DocTags, HTML or Markdown (explained in the next section Output Formats). The way a model handles tables and charts often depends on the output format they are using. Some models treat charts like images: they are kept as is. Other models convert charts into markdown tables or JSON, e.g., a bar chart can be converted as follows.
\n
Similarly for tables, cells are converted into a machine-readable format while retaining context from headings and columns.
\n![\"Table](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ocr/table-rendering.png\")
Different OCR models have different output formats. Briefly, here are the common output formats used by modern models.
\nDocTag: DocTag is an XML-like format for documents that expresses location, text format, component-level information, and more. Below is an illustration of a paper parsed into DocTags. This format is employed by the open Docling models.
![\"DocTags\"](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ocr/doctags_v2.png\")
The right model depends on how you plan to use its outputs:
\nDocuments can have complex structures, like multi-column text blocks and floating figures. Older OCR models handled these documents by detecting words and then the layout of pages manually in post-processing to have the text rendered in reading order, which is brittle. Modern OCR models, on the other hand, incorporate layout metadata to help preserve reading order and accuracy. This metadata is called âanchorâ, it can come in bounding boxes. This process is also called as âgrounding/anchoringâ because it helps with reducing hallucination.
\nOCR models can either take in images and an optional text prompt, this depends on the model architecture and the pre-training setup.
\nSome OCR models support prompt-based task switching, e.g. granite-docling can parse an entire page with the prompt âConvert this page to Doclingâ while it can also take prompts like âConvert this formula to LaTeXâ along with a page full of formulas.
\nOther models, however, are trained only for parsing entire pages, and they are conditioned to do this through a system prompt.
\nFor instance, OlmOCR by AllenAI takes a long conditioning prompt. Like many others, OlmOCR is technically an OCR fine-tuned version of a VLM (Qwen2.5VL in this case), so you can prompt for other tasks, but its performance will not be on par with the OCR capabilities.
Weâve seen an incredible wave of new models this past year. Because so much work is happening in the open, these players build on and benefit from each otherâs work. A great example is AllenAIâs release of OlmOCR, which not only released a model but also the dataset used to train it. With these, others can build upon them in new directions. The field is incredibly active, but itâs not always obvious which model to use.
\nTo make things a bit easier, weâre putting together a non-exhaustive comparison of some of our current favorite models. All of the models below are layout-aware and can parse tables, charts, and math equations. The full list of languages each model supports are detailed in their model cards, so make sure to check them if youâre interested. All models below have open-source license except for Chandra having OpenRAIL license and Nanonets license being unclear. The average scores are taken from model cards of Chandra, OlmOCR, evaluated on OlmOCR Benchmark, which is English-only. Many of the models in this collection have been fine-tuned from Qwen2.5-VL or Qwen3-VL, so we also provide Qwen3-VL model below as well.
\n| Model Name | \nOutput formats | \nFeatures | \nModel Size | \nMultilingual? | \nAverage Score on OlmOCR Benchmark | \n
|---|---|---|---|---|---|
| Nanonets-OCR2-3B | \nstructured Markdown with semantic tagging (plus HTML tables, etc.) | \nCaptions images in the documents Signature & watermark extraction Handles checkboxes, flowcharts, and handwriting | \n4B | \nâ Supports English, Chinese, French, Arabic and more. | \nN/A | \n
| PaddleOCR-VL | \nMarkdown, JSON, HTML tables and charts | \nHandles handwriting, old documents Allows prompting Converts tables & charts to HTML Extracts and inserts images directly | \n0.9B | \nâ Supports 109 languages | \nN/A | \n
| dots.ocr | \nMarkdown, JSON | \nGrounding Extracts and inserts images Handles handwriting | \n3B | \nâ Multilingual with language info not available | \n79.1 ± 1.0 | \n
| OlmOCR-2 | \nMarkdown, HTML, LaTeX | \nGrounding Optimized for large-scale batch processing | \n8B | \nâEnglish-only | \n82.3 ± 1.1 | \n
| Granite-Docling-258M | \nDocTags | \nPrompt-based task switching Ability to prompt element locations with location tokens Rich output | \n258M | \nâ Supports English, Japanese, Arabic and Chinese. | \nN/A | \n
| DeepSeek-OCR | \nMarkdown, HTML | \nSupports general visual understanding Can parse and re-render all charts, tables, and more into HTML Handles handwriting Memory-efficient, solves text through image | \n3B | \nâ Supports nearly 100 languages | \n75.4 ± 1.0 | \n
| Chandra | \nMarkdown, HTML, JSON | \nGrounding Extracts and inserts images as is | \n9B | \nâ Supports 40+ languages | \n83.1 ± 0.9 | \n
| Qwen3-VL | \nVision Language Model can output in all formats | \nCan recognize ancient text Handles handwriting Extracts and inserts images as is | \n9B | \nâ Supports 32 languages | \nN/A | \n
While Qwen3-VL itself is a powerful and versatile vision-language model post-trained for document understanding and other tasks, it isnât optimized for a single, universal OCR prompt. In contrast, the other models were fine-tuned using one or a few fixed prompts specifically designed for OCR tasks. So to use Qwen3-VL, we recommend experimenting with prompts.
\nHereâs a small demo for you to try some of the latest models and compare their outputs.
\nThereâs no single best model, as every problem has different needs. Should tables be rendered in Markdown or HTML? Which elements should we extract? How should we quantify text accuracy and error rates? đ
\nWhile there are many evaluation datasets and tools, many donât answer these questions. So we suggest using the following benchmarks:
When testing different OCR models, we've found that the performance across different document types, languages, etc., varies a lot. Your domain may not be well represented in existing benchmarks! To make effective use of this new generation of VLM-based OCR models we suggest aiming to collect a dataset of representative examples of your task domain and testing a few different models to compare their performance.
\nMost OCR models are small, having between 3B and 7B parameters; you can even find models with fewer than 1B parameters, like PaddleOCR-VL. However, the cost also depends on the availability of optimized implementations for specialized inference frameworks. For example, OlmOCR-2 comes with vLLM and SGLang implementations, and the cost per million pages is 178 dollars (assuming on H100 for $2.69/hour). DeepSeek-OCR can process 200k+ pages per day on a single A100 with 40GB VRAM. With napkin math, we see that the cost per million pages is more or less similar to OlmOCR (although it depends on your A100 provider). If your use case remains unaffected, you can also opt for quantized versions of the models. The cost of running open-source models heavily depends on the hourly cost of the instance and the optimizations the model includes, but itâs guaranteed to be cheaper than many closed-source models out there on a larger scale.
\nWhile the past year has seen a surge in open OCR models, this hasn't been matched by as many open training and evaluation datasets. An exception is AllenAI's olmOCR-mix-0225, which has been used to train at least 72 models on the Hub â likely more, since not all models document their training data.
\nSharing more datasets could unlock even greater advances in open OCR models. There are several promising approaches for creating these datasets:
\nIt's worth noting that many such datasets exist but remain unused. Making them more readily available as 'training-ready' datasets carries a considerable potential for the open-source community.
\nWe have received many questions about getting started with OCR models, so here are a few ways you can use local inference tools and host remotely with Hugging Face.
\nMost cutting-edge models come with vLLM support and transformers implementation. You can get more info about how to serve each from the modelsâ own cards. For convenience, we show how to infer locally using vLLM here. The code below can differ from model to model, but for most models it looks like the following.
\n\nAnd then you can query as follows using e.g. OpenAI client.
\n\n\n\n = \n\n = \n\n\n \n return \n\n\n = \n return ..\n\n = \n\nTransformers
\nTransformers provides standard model definitions for easy inference and fine-tuning. Models available in transformers come with either official transformers implementation (model definitions within the library) or âremote codeâ implementations. Latter is defined by the model owners to enable easy loading of models into transformers interface, so you donât have to go through the model implementation. Below is an example loading Nanonets model using transformers implementation.
\n# make sure to install flash-attn and transformers\n\n\n = \n\n = \n\n\n = \n = \n = \n = \n = \n \n = \n = \n \n = \n return \n\n = \n\nMLX
\nMLX is an open-source machine learning framework for Apple Silicon. MLX-VLM is built on top of MLX to serve vision language models easily. You can explore all the OCR models available in MLX format here. They also come in quantized versions.
\nYou can install MLX-VLM as follows.
pip install -U mlx-vlm\nwget https://huggingface.co/datasets/merve/vlm_test_images/resolve/main/throughput_smolvlm.png\n\npython -m mlx_vlm.generate --model ibm-granite/granite-docling-258M-mlx --max-tokens 4096 --temperature 0.0 --prompt "Convert this chart to JSON." --image throughput_smolvlm.png \nInference Endpoints for Managed Deployment
\nYou can deploy OCR models compatible with vLLM or SGLang on Hugging Face Inference Endpoints, either from a model repository âDeployâ option or directly through Inference Endpoints interface. Inference Endpoints serve the cutting-edge models in a fully managed environment with GPU acceleration, auto-scaling, and monitoring without manually managing the infrastructure.
Here is a simple method of deploying nanonets using vLLM as the inference engine.
nanonets/Nanonets-OCR2-3B![\"Inference](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ocr/IE.png\")
![\"Inference](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/ocr/IE2.png\")
You can learn more about it here.
\nHugging Face Jobs for Batch Inference
\nFor many OCR applications, you want to do efficient batch inference, i.e., running a model across thousands of images as cheaply and efficiently as possible. A good approach is to use vLLM's offline inference mode. As discussed above, many recent VLM-based OCR models are supported by vLLM, which efficiently batches images and generates OCR outputs at scale.
\nTo make this even easier, we've created uv-scripts/ocr, a collection of ready-to-run OCR scripts that work with Hugging Face Jobs. These scripts let you run OCR on any dataset without needing your own GPU. Simply point the script at your input dataset, and it will:
\nFor example, to run OCR on 100 images:
\n\nThe scripts handle all the vLLM configuration and batching automatically, making batch OCR accessible without infrastructure setup.
\nIf you are interested in document AI, not just OCR, here are some of our recommendations.
\nVisual document retrieval is to retrieve the most relevant top-k documents when given a text query. If you have previously worked with retriever models, the difference is that you search directly on a stack of PDFs. Aside from using them standalone, you can also build multimodal RAG pipelines by combining them with a vision language model (find how to do so here). You can find all of them on Hugging Face Hub.
\nThere are two types of visual document retrievers, single-vector and multi-vector models. Single-vector models are more memory efficient and less performant; meanwhile, multi-vector models are more memory hungry and more performant. Most of these models often come with vLLM and transformers integrations, so you can index documents using them and then do a search easily using a vector DB.
\nIf you have a task at hand that only requires answering questions based on documents, you can use some of the vision language models that had document tasks in their training tasks. Weâve observed users trying to convert documents into text and passing the output to LLMs, but if your document has a complex layout, and your converted document outputs charts and so on in HTML, or images are captioned incorrectly, the LLM will miss out. Instead, feed your document and query to one of the advanced vision language models like Qwen3-VL not to miss out on any context.
\nIn this blog post, we wanted to give you an overview of how to pick your OCR model, existing cutting-edge models and capabilities, and the tools to get you started with OCR.
\nIf you want to learn more about OCR and vision language models, we encourage you to read the resources below.