![\"Compute](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/dots-ocr-ne/joules_cu.png\")
Every year our hardware is a little more powerful, our models a little smarter for each parameter. In 2025, it is more feasible than ever to run truly competitive models on-device. dots.ocr, a 3B parameter OCR model from RedNote, surpasses Gemini 2.5 Pro in OmniDocBench, making OCR a truly no compromises on-device use case. Running models on-device is certainly appealing to developers: no smuggling API keys, zero cost, and no network required. However, if we want these models to run on-device, we need to be mindful of the limited compute and power budgets.
\nEnter the Neural Engine, Apple's custom AI accelerator that has shipped with every Apple device since 2017. This accelerator is designed for high performance whilst sipping battery power. Some of our testing has found the Neural Engine to be 12x more power efficient than CPU, and 4x more power efficient than GPU.
\n
Whilst this all sounds very appealing, unfortunately the Neural Engine is only accessible through Core ML, Apple's closed source ML framework. Furthermore, even just converting a model from PyTorch to Core ML can present some challenges, and without a preconverted model or some knowledge of the sharp edges it can be arduous for developers. Luckily, Apple also offers MLX, a more modern and flexible ML framework that targets the GPU (not the Neural Engine), and can be used in conjunction with Core ML.
\n![\"NE](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/dots-ocr-ne/NE_Header.png\")
In this three part series, we will provide a reasoning trace of how we converted dots.ocr to run on-device, using a combination of CoreML and MLX. This process should be applicable to many other models, and we hope that this will help highlight the ideas and tools needed for developers looking to run their own models on-device.
To follow along, clone the repo. You'll need uv and hf installed to run the setup command:
\nIf you just want to skip ahead and use the converted model, you can download it here.
\nConverting from PyTorch to CoreML is a two step process:
\ntorch.jit.trace or, the more modern approach of torch.export)..mlpackage using coremltools.Whilst we do have a few knobs we can tweak for step 2, most of our control is in step 1, the graph we feed to coremltools.
Following the programmers litany of make it work, make it right, make it fast, we will first focus on getting the conversion working on GPU, in FLOAT32, and with static shapes. Once we have this working, we can dial down the precision and try and move to the Neural Engine.
Dots.OCR consists of two key components: A 1.2B parameter vision encoder trained from scratch, based on the NaViT architecture, and a Qwen2.5-1.5B backbone. We will be using CoreML to run the vision encoder, and MLX to run the LM backbone.
In order to convert a model, it's best to understand the structure and function before getting started. Looking at the original vision modelling file here, we can see that the vision encoder is similar to the QwenVL family. Like many vision encoders, the vision encoder for dots works on a patch basis, in this case 14x14 patches. The dots vision encoder is capable of processing videos and batches of images. This gives us an opportunity to simplify by only processing a single image at a time. This approach is frequent in on-device apps, where we convert a model that provides the essential functions and iterate if we want to process multiple images.
When kicking off the conversion process, it's best to start with a minimal viable model. This means removing any bells and whistles that are not strictly necessary for the model to function. In our case, dots has many different attention implementations available for both the vision encoder and the LM backbone. CoreML has lots of infrastructure oriented around the scaled_dot_product_attention operator, which they introduced in iOS 18. We can simplify the model by removing all of the other attention implementations and just focusing on simple sdpa (not the memory efficient variant) for now, commit here.
Once we've done this, we see a scary warning message when we load the model:
\n; \nThe model doesn't require Sliding Window Attention to function, so we can happily move on.
\nUsing torch.jit.trace is still the most mature method for converting models to CoreML. We usually encapsulate this in a simple harness that allows you to modify the compute units used and the precision selected.
You can check out the initial harness here. If we run the following on the original code implementation:
\n\nWe should bump into the first (of many) issues.
\nIt is rare that a model will convert first time. Often, you will need to progressively make changes further and further down the execution graph until you reach the final node.
\nOur first issue is the following error:
\n\n\nLuckily this error gives us quite a bit of information. We can look at the VisionRotaryEmbedding layer and see the following code:
\n = \n = \n return \nAlthough torch.arange has a dtype argument, coremltools ignores this for arange and always outputs int32. We can simply add a cast after the arange to fix this issue, commit here.
After fixing this, running the conversion again leads us to our next issue at repeat_interleave:
\n\nWhilst this error is less informative, we only have a single call to repeat_interleave in our vision encoder:
= \ncu_seqlens is used for masking variable length sequences in flash_attention_2. It's derived from the grid_thw tensor, which represents time, height and width. Since we are only processing a single image, we can simply remove this call, commit here.
Onto the next! This time, we get a more cryptic error:
\n\n\nThis is again due to the masking logic to handle variable length sequences. Since we are only processing a single image (not a video or batch of images), we don't really need attention masking at all! Therefore, we can just use a mask of all True. To prepare ourselves for the Neural Engine conversion, we also switch from using a boolean mask to a float mask of all zeros, as the Neural Engine does not support bool tensors commit here
With all of this done, the model should now successfully convert to CoreML! However, when we run the model, we get the following error:
\n\nThis reshape could be in multiple places! Luckily, we can use a previous warning message to help us track down the issue:
\nMost ML compilers do not like dynamic control flow. Luckily for us, as we are only processing a single image, we can simply remove the loop and process the single h, w pair, commit here.
And there we have it! If we run the conversion again, we should see that the model successfully converts and matches the original PyTorch precision:
\n\nNow that we've got the model working, let's evaluate the size and performance. The good news is the model is working, the bad news is that it's over 5GB! This is completely untenable for on device deployment! To benchmark the computation time, we can use the built in XCode tooling by calling:
\nopen DotsOCR_FLOAT32.mlpackage\nwhich will launch the XCode inspector for the model. After clicking + Performance Report and launching a report on all compute devices, you should see something like the following:
![\"GPU](\"https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/blog/dots-ocr-ne/perf_report_gpu.png\")
Over a second for a single forward pass of the vision encoder! We have lots of more work.
\nIn the second part of this series, we will work on the integration between CoreML and MLX, to run the full model on-device. In the third part, we will dive deep into the optimizations required to get this model running on the Neural Engine, including quantization and dynamic shapes.