Presentation: Building Reusable Evaluation Frameworks for Agentic AI Products

7.8 relevance

Highly actionable framework for AI agent evaluation with strong technical depth and personal relevance.

AI/ML infoq.com

Answer the board on AI readiness

Arrive with evidence, not opinion. See how your foundation benchmarks against your peers, get a clear read, and leave with a ranked place to start.

Why It Matters

For platform engineers building agentic systems, this offers a concrete blueprint for unifying experimental evaluation workflows with production observability, directly addressing the common friction between data science and engineering teams.

Key Takeaways

Adopt a hybrid evaluation strategy combining LLM-as-a-judge with deterministic rules, and instrument deep tracing across your agentic workflows to bridge experimental evals and production regressions.

Presentation: Building Reusable Evaluation Frameworks for Agentic AI Products

Summary

Susan Chang detailed Elastic's shift from siloed, ad-hoc agent evaluations to a unified production framework. The approach bridges Python data science evals with TypeScript production code, balancing LLM-as-a-judge with deterministic rules. Deep tracing forms the foundation for catching regressions across complex RAG and cybersecurity workloads like Elastic's Attack Discovery.

Author

Susan Chang