I Built Scenario Packs for Agent Regression Testing. The Integration, Not the Judge, Broke Me.
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Novel approach to agent regression testing with YAML scenarios, directly applicable to agent workflows.
Building scenario packs for agent regression testing revealed that the hardest part isn't scoring or rubric design—it's the integration layer between the evaluation harness and real, messy third-party agents that import ffmpeg at module scope, hardcode model versions, and write to /root. The EvalForge OSS harness enforces a correctness boundary by stripping expected outputs and metrics from the agent's invocation payload, ensuring the agent cannot game what it cannot see. The architecture uses a CLI runner, adapter pattern for agent integration, scorer, semantic judge, and diff engine against saved baselines.