Insights
Judgment, written down.
Notes on evaluation, retrieval, cost and production behavior, written for the people who run these systems.
Evaluation & observability2026-08-18
What a production LLM evaluation suite actually contains
Most teams evaluate prompts in a notebook. This is what belongs in the suite that gates releases instead: the dataset, the checks, the judge, the review and the gate.
Cost & performance2026-07-15
Measure before you optimize: the honest way to cut LLM spend
Cost work without per-request attribution is guesswork with a budget attached. What to instrument first, and which levers actually move the number.
Retrieval & data2026-06-20
Where RAG quality is actually lost
Retrieval answers go wrong for specific, findable reasons. A stage-by-stage breakdown of where quality leaks, and how to measure each stage on its own.