Managing AI Coding Costs at Scale

Databricks, Stripe, Coinbase, Uber, and Ramp have converged on cost management techniques for AI coding tools, achieving a 'dual mandate' of broad access and fixed per-user costs. The single largest lever is rapidly adopting more efficient models along the 'efficiency frontier' rather than frontier models, requiring custom benchmarks since public ones poorly predict real-world coding performance. Databricks open-sourced its Omnigent meta-harness and Unity AI Gateway to help enterprises implement these techniques, which include model routing, caching, and client-side modifications.

Patrick Wendell