The Model Scored 30%. The Harness Scored 100%. Which One Did You Benchmark?

8 relevance

Benchmarking harness vs model performance is a deep technical insight into AI evaluation, highly relevant.

AI/ML dev.to

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For engineers building AI evaluation pipelines or agent systems, this exposes that benchmark scores are inseparable from harness design—memory, action budgets, and reasoning retention—meaning your model's reported performance is as much a measure of your infrastructure as your architecture.

Key Takeaways

Treat any benchmark score as a joint property of the model and the harness; always version and publish the harness configuration alongside the result.

The Model Scored 30%. The Harness Scored 100%. Which One Did You Benchmark?

Summary

On ARC-AGI-3, the same Claude Opus 5 model scored 30% in the official harness and 100% in NVIDIA's AVO harness, a 70-point gap driven entirely by changes to the code around the model—not the weights. Harnesses like Microsoft's Agent Lightning v1.0 now integrate reinforcement learning into the deployment loop, making the harness part of the trained artifact and blurring what a benchmark score actually measures. Every 100% result is on the public set only, with authors explicitly disclaiming controlled ablation or held-out generalization claims.

Author

Sergei Parfenov