I type-check AI-generated SDK code against the real package. Claude refused a third of my Stripe tasks.
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
Type-checking AI-generated SDK code against real packages is highly novel, technically deep, and directly relevant to AI coding agents and developer experience.
A developer building SDKProof, a tool that type-checks AI-generated SDK code against real packages, discovered Claude refused 41.3% of Stripe API tasks (62/150) versus 0% on Zod controls. The bug was that empty refusal responses compiled cleanly, masking failures until a harness check was added. Refusal rates varied dramatically by task: payment-intent creation was refused 10/10 times, while refunds were refused only 1/10, suggesting the model applies inconsistent safety heuristics to similar financial operations.