Architecting lean LLM caching: how to drop a 20M-row table without losing your AI memory
7.2 relevance
Score Breakdown
technical depth 8
novelty 6
actionability 9
community 4
strategic 5
personal 9
Scored daily by a customisable AI persona to surface the most relevant engineering leadership news.
LLM caching for agentic pipelines is technically deep, practical, and highly relevant.
Summary
For an agentic pipeline reloading 20M rows every two weeks, only 60k distinct inputs reach the LLM, allowing a tiny cache table (enrichment_cache) to survive the wipe of the main working table (data), dropping from 16GB to a few MB. The four-step process—wipe, load, propagate cache hits via SQL UPDATE, then query LLM only for misses—keeps the cache durable in Postgres without needing Redis. However, batch UPDATE across 20M rows triggers Postgres MVCC bloat, so the team adopted a CTAS-and-swap zero-bloat pattern (detailed in part 3).