A benchmark for evaluating atomic visual world knowledge in Multimodal LLMs.
\nAuthors Kimi Team
\nWe are releasing WorldVQA, a new benchmark designed to measure the factual correctness of Multimodal Large Language Models (MLLMs). While recent models have demonstrated impressive capabilities in visual reasoning and description, measuring their reliability regarding visual world knowledge remains a challenge.
\nWorldVQA focuses on a critical question: Does the model actually recognize the specific entity it sees, or is it merely hallucinating based on visual patterns?
\nOur results show that WorldVQA creates a significant challenge for frontier models. Even state-of-the-art models struggle to achieve high accuracy on long-tail visual knowledge, often falling below 50% accuracy. This benchmark aims to drive progress toward more factually reliable and knowledgeable multimodal AI.
\nThe dataset consists of 3,500 high-quality image-question pairs. The distribution aims to test a model's encyclopedic breadth across the world. The dataset distinguishes itself through three core design principles:
\n\n\nNote on Quality: To ensure the benchmark is a reliable gold standard, all images and question-answer pairs underwent rigorous multi-stage human verification to filter out noise and ambiguity.
\n
| Statistics | \nNumber | \n
|---|---|
| Data | \n\n |
| Total | \n3500 | \n
| Chinese (CN) | \n1260 (36%) | \n
| English (EN) | \n2240 (64%) | \n
| Category Categories | \n\n |
| Total categories | \n9 | \n
| Nature & Environment (Nature) | \n9.31% | \n
| Locations & Architecture (Geography) | \n14.63% | \n
| Culture, Arts & Crafts (Culture) | \n14.46% | \n
| Objects & Products (Objects) | \n12.49% | \n
| Vehicles, Craft & Transportation (Transportation) | \n8.74% | \n
| Entertainment, Media & Gaming (Entertainment) | \n14.60% | \n
| Brands, Logos & Graphic Design (Brands) | \n7.43% | \n
| Sports, Gear & Venues (Sports) | \n4.06% | \n
| Notable People & Public Figures (People) | \n14.29% | \n
| Difficulty | \n\n |
| Easy | \n31.16% | \n
| Medium | \n40.77% | \n
| Hard | \n28.07% | \n
In our experiments comparing model confidence with actual accuracy, we utilized two key metrics to measure the alignment between a model's subjective belief and its objective performance:
\nCalibration and Confidence Distribution Analysis. Left: Reliability diagrams plotting Actual Accuracy against Stated Confidence. To ensure statistical significance, only bins containing more than 20 samples are visualized. The size of each data point is proportional to the number of samples in that bin. The black dashed diagonal (y=x) represents perfect calibration, while colored dashed lines indicate the weighted average slope for each model. Right: The distribution of stated confidence scores across the full dataset (without sample thresholding). The plots reveal a severe overconfidence trend, with most models concentrating their predictions in the 90-100% confidence range.
\nOur experiments reveal that all evaluated models are currently far from the ideal state, exhibiting a universal tendency toward overconfidence.
\nWhile Kimi-K2.5 achieves best performance on both metrics—recording an ECE of 37.9% and a Slope of 0.550—there remains a significant gap to bridge in the pursuit of "honesty" and "alignment." Enhancing the self-awareness boundaries of multimodal models represents a critical direction for future exploration.
\nWorldVQA is a simple but challenging benchmark for evaluating the atomic visual knowledge of frontier models. Improving performance on WorldVQA is a necessary step for the next generation of AI agents. We are open-sourcing the WorldVQA dataset and evaluation scripts to help the community address the visual knowledge gap.
\n