Singularity not so near? New benchmark shows even top AI models score just 4% on AGI test

The race toward artificial general intelligence (AGI) has hit a sobering checkpoint as a new benchmark reveals the limitations of today’s most advanced AI systems. The ARC Prize Foundation’s ARC-AGI-2 test introduces efficiency metrics alongside performance standards, showing that even cutting-edge models score in the low single digits while costing significantly more than humans to complete basic reasoning tasks. This development signals a fundamental shift in how we evaluate AI progress, prioritizing not just raw capability but also computational efficiency.

The big picture: Current AI models, including OpenAI‘s sophisticated o3 systems, are failing a new benchmark designed to measure progress toward artificial general intelligence, scoring no higher than single digits out of 100.

How the benchmark works: ARC-AGI-2 tests AI models on seemingly simplistic tasks requiring symbolic interpretation and adaptability, while also factoring in the computational efficiency and cost of running the models.

• While OpenAI’s o3-low model scored 75.7% on the previous ARC-AGI-1 test, it achieved just 4% on the new benchmark.
• The test measures efficiency by comparing costs – human testers were paid $17 per task, while o3-low costs an estimated $200 to complete the same work.
• Every question in the benchmark has been solved by at least two humans in fewer than two attempts, providing a clear human performance baseline.

Between the lines: The new benchmark represents a philosophical shift in AI evaluation, moving beyond raw performance to consider the environmental and economic costs of increasingly powerful systems.

• The efficiency component addresses growing concerns about AI models becoming more energy-intensive and computationally expensive.
• This approach suggests that truly intelligent systems should be able to solve problems effectively without requiring excessive computational resources.

What experts are saying: Researchers are divided on the significance and framing of these benchmark tests in measuring progress toward AGI.

• Joseph Imperial from the University of Bath calls the new focus on balancing performance with efficiency “a big step towards a more realistic evaluation of AI models.”
• Catherine Flick of the University of Staffordshire argues that framing these tests as measuring intelligence is misleading, as they merely assess narrow task completion abilities.

The counterpoint: Critics suggest these benchmarks mislead the public about AI capabilities by equating task-specific performance with general intelligence.

• Flick warns against media interpretations that claim AI models are “passing human-level intelligence tests” when they’re simply responding accurately to specific prompts.

Looking ahead: As AI development continues, benchmark standards will likely keep evolving to match advancing capabilities.

• Future iterations might add new dimensions to evaluation, potentially including metrics like the minimum number of humans required to solve comparable tasks.
• The fundamental debate about what constitutes artificial general intelligence remains unresolved, with benchmarks serving as moving targets rather than definitive measures.