Caching and Batching LLM Calls
Caching avoids paying for the same LLM computation twice, and batching groups many requests together, and together they are usually the difference between an LLM pipeline being affordable and not.
A quant firm running LLM calls over thousands of news stories a day quickly discovers that the bill, and the latency, come almost entirely from two habits: sending the same long context over and over, and sending requests one at a time when they could be sent together. Caching and batching are the two standard fixes, and both are about eliminating wasted, repeated work rather than making any single call faster.
Prompt caching stores the model's internal processing of a fixed block of text — a system prompt, a set of instructions, a long reference document — so that later calls reusing that same block skip reprocessing it and pay a much lower cost for it. This helps most when many calls share a large, unchanging prefix (like instructions) but vary in a short, changing suffix (like a headline to classify).
Batching groups many independent requests into a single job submitted together rather than one-by-one, which the provider processes more cheaply and efficiently in bulk, typically in exchange for the results arriving asynchronously rather than instantly.
Cache the parts of a prompt that repeat across calls, and batch the calls that don't need an instant answer — most LLM-pipeline cost is duplicated context and per-call overhead, not the actual reasoning.
Worked example
A pipeline classifies 10,000 headlines a day, each call carrying a 2,000-token instruction block plus a 50-token headline. Without caching, that instruction block is paid for in full 10,000 times. With caching, it is paid for once at full price and then at a steep discount (often around 90% off) on every subsequent call, cutting the dominant cost by roughly an order of magnitude — even though nothing about the model's answers changes.
Batching and caching solve different problems and are usually combined: batch to cut per-call overhead across many jobs, cache to avoid re-paying for a repeated context within each job.
Related concepts
Practice in interviews
Further reading
- Anthropic, 'Prompt Caching' documentation