I Told My LLM Critic to Be Adversarial. It Started Blocking Plans for Being 'Not Thorough Enough.'

6.7 relevance

Building an adversarial LLM critic engine, directly relevant to agent orchestration and AI workflows.

AI/ML dev.to

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For engineers building multi-agent systems with LLM reviewers, this demonstrates that relying on prompt engineering for severity classification is brittle; deterministic code guardrails are essential to prevent false positives from adversarial or over-thorough critics, especially in production CI/CD or plan-approval pipelines.

Key Takeaways

Enforce severity classification with a code-level allowlist of blocker families (e.g., frozenset) rather than trusting LLM prompts to correctly distinguish between safety defects and advisory concerns.

I Told My LLM Critic to Be Adversarial. It Started Blocking Plans for Being 'Not Thorough Enough.'

Summary

A developer building an open-source PlannerCritic engine discovered that prompting an LLM critic to be 'adversarial' caused it to block plans for completeness concerns rather than concrete safety defects, escalating 16 strict goals incorrectly. The fix combined a prompt change with explicit severity rules and a deterministic code guardrail using a frozenset of allowed blocker families (unsafe_sequencing, weak_rollback, unverified_dependencies, feasibility), which downgraded any blocker outside those families to a warning. After the fix, zero advisory findings appeared as blockers across 92 runs, with all 132 blockers tied to real structural defects, proving that prompt engineering alone was insufficient—LLMs lack a stable concept of severity, so code-level enforcement is necessary.

Author

Debashish Ghosal