What’s the point of having a smart machine if it can’t really ‘see’ the world properly? This is exactly the puzzle researchers are trying to solve with multimodal large language models, machines that are supposed to understand both text and images. A recent study uncovered that while these models can do some impressive things with language, they still struggle when it comes to correctly interpreting and reasoning about complex visual scenes, like city transit maps.
Meet ReasonMap, a special test created to assess how well these AI models can understand fine details in visual information. It features high-resolution maps from cities all over the world and asks the models to answer detailed questions about them. Surprisingly, the study found that open-source models often did better than the closed-source ones at these tasks, but both types struggled when the images were masked, proving the models rely heavily on visual inputs.
Imagine a world where AI could help us navigate new cities effortlessly, just by looking at a map. This research is a step towards that future, highlighting where AI needs to improve to make that vision possible. If machines can get better at visual reasoning, they could revolutionize how we interact with the world, making everything from urban planning to traveling simpler and smarter.
Did you know? Some AI models rely so much on images that they perform worse when parts of the image are hidden, highlighting their need for genuine visual information!
FAQs
What is the ReasonMap benchmark used for?
ReasonMap is a specially designed test to evaluate the fine-grained visual understanding and spatial reasoning abilities of multimodal large language models using detailed transit maps from multiple cities.
Why do open-source models outperform closed-source ones in visual reasoning?
The study suggests that open-source models may be better optimized for handling specific visual tasks due to their development and training processes, though the reason remains counterintuitive and might be due to how these models are fine-tuned.
How does masking visual inputs affect AI model performance?
When visual inputs are masked, the performance of AI models typically degrades, indicating their heavy reliance on visual data to answer questions accurately, showing the importance of actual visual perception in reasoning tasks.
How can improvements in AI visual reasoning impact everyday life?
Enhancing AI’s visual reasoning abilities could pave the way for smarter tech applications, like navigation aids that help users interact more intuitively with environments or planning tools that optimize city infrastructure effectively.
What surprising patterns did the study reveal about AI reasoning models?
The study found that base models often outperform reasoning models in open-source formats, which contrasts with findings in closed-source formats where reasoning variants excel, suggesting different optimization strategies or capabilities.
Background
In simple terms, multimodal large language models are advanced forms of artificial intelligence that can understand and process both text and images. These models have shown great promise in tasks requiring a mix of language and visual information. However, evaluating their ability to truly ‘see’ and interpret fine details in images—like those in transit maps—poses new challenges, which is why ReasonMap was developed.
History
The evolution of AI has seen significant breakthroughs in language processing and image recognition as separate domains. Recently, efforts have been made to merge these abilities in multimodal models. This study builds on previous research by testing the limits of these combined capabilities, specifically their visual reasoning skills, using varied and complex datasets such as high-resolution transit maps.
Based on “Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual Reasoning from Transit Maps” by Sicheng Feng, Song Wang, Shuyi Ouyang, Lingdong Kong, Zikai Song, Jianke Zhu, Huan Wang, Xinchao Wang, available on arXiv (arxiv.org/abs/2505.18675), used under CC BY 4.0 (creativecommons.org/licenses/by/4.0/).
