Livenerf: Has Opus 5.5 been nerfed yet?

7.4 relevance

Open source tool for monitoring AI model behavior, directly actionable and highly relevant to the AI/ML workflow.

Open Source github.com

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For engineers building on frontier LLMs, this benchmark provides a rigorous, open-source methodology to audit model consistency, directly impacting the reliability of AI agents and production services where stealth degradation can silently erode performance.

Key Takeaways

Adopt pre-registered, deterministic eval protocols (like Livenerf's Inspect-based approach) to baseline model performance on launch day and continuously monitor for drift in accuracy and token economics.

Benchmark for tracking model capability after release. - ninjahawk/livenerf

Summary

Livenerf is an open-source, deterministic benchmark built on the UK AISI's Inspect framework to detect post-launch degradation in frontier models like Claude Opus 5.5. Running daily via headless Claude Code, it measures statistical drift across a pre-registered panel of 78 GPQA/MMLU-Pro questions, finding that lower model effort reduces output tokens by 62% but accuracy by only 8.3 points. The benchmark currently cannot distinguish Opus 5.5 from Opus 5 at 99% confidence, highlighting the subtlety of stealth model changes.

Author

ninjahawk