Imagine if your gaming skills were the new frontier for artificial intelligence. While today’s AI excels at complex tasks like coding and solving math problems, it falls short in areas we humans find almost second nature—like perceiving our environment, moving through spaces, and remembering what we’ve learned. Wouldn’t it be cool if an AI could master a video game just like you did? With our natural knack for these tasks, video games become the perfect playground for testing AI’s evolution in this area.
Researchers have introduced a benchmark called VideoGameBench, featuring 10 popular video games from the 1990s. These games are chosen for their intuitive design, which relies on the very skills AI struggles with. By letting AI models interact with these games in real-time, researchers can see how well they mimic human abilities. However, it’s not a walk in the park for these AI systems; even the most advanced models, like Gemini 2.5 Pro, complete only a tiny fraction of the challenges. Interestingly, one key hurdle is how fast these systems can think, or their ‘inference latency’—kind of like if your brain took too long to decide which button to press next.
The hope is that by refining AI through these benchmarks, we’ll see the development of smarter systems that can assist in everyday life. Imagine a future where your smart home system can effortlessly navigate your home to assist you or a personal AI that learns from its environment just like you do. It’s not just about playing games—this research could redefine how AI integrates into our lives, making technology work seamlessly with our natural instincts and habits.
Did you know the best AI can only complete up to 1.6% of a modified classic game challenge? These games are tougher for AI than they are for you!
FAQs
What is VideoGameBench and why is it important?
VideoGameBench is a benchmark composed of 10 popular video games from the 1990s, aimed at evaluating AI models on tasks that come naturally to humans, such as perception and spatial navigation. It’s important because it helps researchers understand how close AI is to mimicking these intuitive human skills.
How do current AI models perform on VideoGameBench?
Current AI models, such as Gemini 2.5 Pro, struggle significantly, managing to complete only a small fraction of the games. This showcases the challenges AI faces in tasks involving real-time decision-making and perception, which are intuitive for humans.
Why are video games a good test for AI’s human-like abilities?
Video games are designed to be easily learned and mastered by humans due to our natural inductive biases. They serve as an ideal testbed to evaluate AI’s ability to perform intuitive tasks like navigation and memory management.
What is inference latency and how does it affect AI performance?
Inference latency refers to the time it takes for an AI model to process information and make a decision. High inference latency can hinder AI performance in real-time environments, such as video games, where quick decision-making is essential.
Background
Vision-language models are a type of AI designed to understand and respond to visual inputs and language cues. They’re great at executing complex tasks but struggle with intuitive human skills like perception and navigation. Video games, especially classic ones, are created with these innate human abilities in mind, making them ideal for testing and improving AI systems in these areas.
History
AI has been tested on video games since the early days of machine learning, with programs like Deep Blue in chess and AlphaGo in the game Go. These tests demonstrated AI’s ability to handle complex strategy and calculations. However, games involving real-time interaction and human-like skills remained a challenge. This research builds on that history by focusing on those human-like skills that AI hasn’t yet mastered.
Based on “VideoGameBench: Can Vision-Language Models complete popular video games?” by Alex L. Zhang, Thomas L. Griffiths, Karthik R. Narasimhan, Ofir Press, available on arXiv (arxiv.org/abs/2505.18134), used under CC BY 4.0 (creativecommons.org/licenses/by/4.0/).
