Do You See Me

A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs

Aditya Kanade, Tanuja Ganu

Microsoft Research

Overview

Multimodal Large Language Models (MLLMs) have demonstrated promising reasoning capabilities, yet their visual perception—the ability to interpret and understand visual stimuli—remains a critical bottleneck. Strikingly, MLLMs can produce correct answers even while misinterpreting crucial visual elements, masking these underlying failures.

Our work centers on a fundamental question: Do current MLLMs exhibit human-level visual perception capabilities? To answer this, we introduce the Do You See Me benchmark, a scalable and systematic tool for evaluating the core visual skills of MLLMs.

Key Insights

We created a joint perception-reasoning dataset from 150 logic-based IQ questions to investigate whether correct reasoning implies correct perception. Our findings reveal a significant disconnect:

MLLMs often answer reasoning questions correctly while failing the corresponding perception questions that test their understanding of critical visual details.
Visual perception errors are common in the reasoning chains of MLLMs, even when the final answer is correct.

Our preliminary study on one leading MLLM revealed that for 29.0% of its correct answers to reasoning questions, the model still made visual perception errors. This highlights that final-answer accuracy can obscure critical perceptual shortcomings.

Do You See Me is grounded in established human psychology frameworks, which categorize visual perception into five core abilities: visual discrimination, figure-ground perception, spatial relations, closure, and memory. Our benchmark adapts these principles to systematically evaluate MLLMs across 1,758 images and 2,612 questions.

The benchmark evaluates MLLMs on seven subtasks designed to mirror these core human skills:

Shape Discrimination: Counting specific shapes in a cluttered image.
Joint Shape-Color Discrimination: Counting shapes of specific colors.
Letter Disambiguation: Recognizing textual characters.
Visual Form Constancy: Recognizing a pattern after geometric transformations.
Visual Spatial: Understanding object positions and spatial relationships.
Visual Figure-Ground: Distinguishing a target pattern from background noise.
Visual Closure: Matching incomplete shapes with their complete forms.

Our fully synthetic and programmatic benchmark allows for fine-grained control over task difficulty, enabling a rigorous analysis of MLLM capabilities.

Results: A Stark Performance Gap

Our comprehensive evaluation reveals a stark deficit between MLLM and human visual perception:

Humans achieve an average accuracy of 96.49% across the benchmark.
In stark contrast, the top-performing MLLMs average below 50% accuracy.
No single model consistently excels across all perceptual challenges, indicating scattered and unreliable capabilities.

Visual Closure

MLLMs show strong performance in completing partial shapes, with Claude Sonnet-3.5 achieving 91.48% accuracy.

Joint Shape-Color Discrimination

Gemini-1.5 Flash and Qwen2.5-VL-7B-Instruct are top performers in this 2D task, both achieving 81.86% accuracy.

Visual Spatial

Qwen2.5-VL-7B-Instruct leads this 2D category with an accuracy of 40.69%, though this is still far below human performance.

Letter Discrimination

MLLMs struggle significantly as task difficulty increases, with performance dropping to near-zero at medium difficulty.

As task complexity increases, MLLM performance drops drastically while human performance remains robust. Key observations:

For visual form constancy, the performance gap between humans and closed-source models widens from 12% on easy samples to 45% on hard samples.
In tasks with background noise (Visual Figure-Ground), MLLM accuracy is notably diminished compared to tasks with clean backgrounds.
Across all subtasks, MLLM performance consistently declines as the difficulty control parameters increase, often leading to near-zero accuracy in the most challenging settings.

Root Cause Analysis: Why Do MLLMs Fail?

Our analysis delves into the underlying reasons for these perceptual failures, identifying several fundamental limitations in current MLLM architectures.

We found that MLLMs frequently fail to focus on the parts of an image that are most relevant to a given question. Our analysis of patch-level attention maps revealed a critical issue:

Models often direct only sparse attention—around 10%—towards query-relevant object regions.

This failure to engage with crucial visual details severely limits their ability to ground language queries in visual evidence, representing a major perceptual bottleneck.

MLLMs struggle to interpret fine-grained visual details, especially for objects that are near or below the resolution of their visual encoder's patches (typically 14x14 pixels).

We observed a significant decline in task accuracy as object sizes approached or fell below this patch resolution.
For example, detecting a subtle 1° rotation was impossible for objects smaller than twice the patch size but trivial for larger objects.

This suggests a fundamental constraint in current encoders, limiting their ability to robustly interpret small objects or intricate details.

To test if more training data could solve these issues, we finetuned a model on over 67,000 new image-text pairs from our benchmark. The results were modest:

SFT improved the model's average accuracy by approximately 11% (from 40.91% to 51.75%).
Despite these gains, the finetuned model's performance remained significantly below the human accuracy of 96.49%.

This suggests that merely scaling up SFT with more benchmark-like data is insufficient to overcome these fundamental visual perception limitations.

We found that MLLM perceptual performance exhibited significant and often unpredictable variance when transitioning between 2D and 3D settings.

Strong proficiency in a 2D subtask did not reliably translate to similar efficacy in its 3D counterpart, and vice-versa.

This indicates that MLLM perception is not abstract but is heavily influenced by the nature of the training data (e.g., vast amounts of real-world, inherently 3D data).

Conclusion

Our study reveals a stark deficit in the visual perception abilities of contemporary MLLMs. These models often fail to perceive fine-grained visual details, misallocate attention, and cannot reliably overcome these issues even with targeted finetuning.

The results on the Do You See Me benchmark clearly indicate an urgent need to improve foundational visual skills in MLLMs, independently from high-level reasoning. Enhancing these perceptual capabilities is paramount for reducing hallucinations, improving reasoning, and building more reliable MLLMs for real-world applications.

The programmatic, scalable, and complexity-controlled approach of our benchmark offers a valuable resource for both evaluating current models and guiding the development of more perceptually grounded MLLMs.

Citation

If you find our work useful, please consider citing our paper:

@misc{kanade2025multidimensionalbenchmarkevaluating,
      title={Do You See Me : A Multidimensional Benchmark for Evaluating Visual Perception in Multimodal LLMs}, 
      author={Aditya Kanade and Tanuja Ganu},
      year={2025},
      eprint={2506.02022},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2506.02022}, 
}