How a Strands agent took Claude Opus 5 from 30% to 99.95% on ARC-AGI-3

7.8 relevance

Breakthrough in AI agent reasoning on ARC-AGI benchmark with deep technical detail.

AI/ML dev.to

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For engineers building AI agents, this shows that open-source harnesses like Strands can dramatically amplify model reasoning by handling context management, state maintenance, and iterative learning, which is critical for moving beyond simple RAG or prompt engineering into autonomous, multi-step problem solving.

Key Takeaways

Evaluate your agent harness architecture as rigorously as your model choice, since the surrounding system for context and state management can multiply effective performance by 3x or more on novel tasks.

How a Strands agent took Claude Opus 5 from 30% to 99.95% on ARC-AGI-3

Summary

AWS engineers used the open-source Strands Agents SDK with Claude Opus 5 to achieve a 99.95% relative human action efficiency (RHAE) score on ARC-AGI-3's public benchmark, completing all 183 levels across 25 environments in a single 8-hour run costing $830 in tokens. The agent started with zero game knowledge in its system prompt, using generic action labels and raw numeric board representations, forcing it to experiment and learn through observation. This contrasts sharply with Claude Opus 5's standalone score of 30.16% on the same benchmark, demonstrating that agent harness design—not just model capability—determines performance on long-horizon novel problem-solving tasks.

Author

Morgan Willis