PerceptionBench

Introducing PerceptionBench Evaluating Atomic Visual Perception in Multimodal Large Language Models Authors Kimi Team Overview We are releasing PerceptionBench, a benchmark that isolates visual perception and evaluates it as a set of atomic capabilities—discovered from how today's models fail, not defined in advance. From frontier-model failures across 42 benchmarks, we derive 10 atomic perceptual capabilities and construct 3,000 verified questions, each isolating a single capability and answerable by looking, with no reasoning or external knowledge required. Across sixteen frontier MLLMs, no model reaches 60% accuracy, and perception-related hallucination is the weakest capability on average. Models with nearly identical overall scores can diverge sharply in what they actually perceive. PerceptionBench is built to expose exactly where perception breaks, and to drive progress toward multimodal AI that sees faithfully and consistently. The Dataset Guided by the induced taxonomy, we select the most informative failures from the source benchmarks, decompose them into atomic sub-questions, and author additional questions on supplemented images. The retained and constructed samples together form an in-house pool of 17,000+ verified questions. The released benchmark subsamples 3,000 verified questions from the constructed portion—60% decomposed from attributed model failures and 40% newly authored—with category-level balancing and difficulty stratification to isolate atomic perceptual capabilities from confounding factors. The released benchmark distinguishes itself through three core design principles: Failure-Driven Taxonomy: Every category is discovered from real model failures, attributed to the earliest erroneous step across 42 existing benchmarks. Ten Atomic Perceptual Categories: Visual Relation, Counting, Attribute, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination. Perception, Not Reasoning or Knowledge: Samples are curated, decomposed, and difficulty-balanced so that difficulty stems from perception rather than reasoning or external knowledge. Each source benchmark captures a narrow slice of perception errors, and these slices overlap only weakly (mean pairwise weighted Jaccard 0.20). No single benchmark—or small group of them—covers perception as a whole, which motivates a capability-centric benchmark that aggregates and rebalances these fragmented views. Distribution of Tasks per Category Statistics Number Data Total 3,000 Atomic perceptual categories 10 Task Categories Depth 3D Perception Error 330 (11.00%) Visual Counting Error 330 (11.00%) Fine-Grained Recognition Error 290 (9.67%) Visual Relation Error 330 (11.00%) Visual Attribute Error 330 (11.00%) Visual Localization Error 330 (11.00%) Visual Comparison Error 279 (9.30%) Context Integration Error 255 (8.50%) Hallucination 271 (9.03%) OCR Error 255 (8.50%) Using PerceptionBench to Compare Models Conclusion PerceptionBench is a simple but challenging benchmark for evaluating the atomic visual perception of frontier models. It provides a capability-level standard for measuring and diagnosing the visual perception boundaries of multimodal models. We are open-sourcing the PerceptionBench dataset and evaluation code to help the community address the visual perception gap. Download the Data: https://github.com/MoonshotAI/PerceptionBench