How to Cut Your LLM API Bill
Sergio JimenezOctober 21, 20266 min read
Most teams paying too much for LLM calls are not paying for intelligence. They are paying to resend the same text thousands of times a day.
These are the levers, ordered by how much they typically return for the effort. The first three usually account for the whole problem.
The short answer
| Lever | Typical saving | Effort |
|---|---|---|
| Prompt caching on a stable prefix | 50 to 90 percent on input | Hours |
| Route easy turns to a smaller model | 40 to 70 percent overall | Days |
| Stop resending history and documents | 30 to 60 percent | Days |
| Cap retries and loops | Prevents the spike, not the baseline | Hours |
| Shorten outputs | 10 to 30 percent | Hours |
| Batch what is not interactive | Around 50 percent on that traffic | Days |
Measure before you touch anything
You cannot optimise a bill you have not broken down. Log, per request: input tokens, output tokens, cached tokens, model, latency and which feature made the call.
Almost every team that does this finds the same thing, and it is never what they guessed: one feature nobody considered important is making 70 percent of the calls, usually something that runs on a timer or on every page load. Find that before writing any optimisation, because the biggest win is often deleting a call rather than making it cheaper.
Prompt caching, which is the cheapest large win
If your requests share a long stable prefix, a system prompt, a tool schema, a set of examples, caching lets the provider charge a fraction for that part on subsequent calls.
The requirement is that the prefix is byte identical and comes first. Which means: put everything stable at the top and everything variable at the bottom. A timestamp or a user name injected into the top of your system prompt invalidates the cache on every single call, and this one mistake is common enough to be worth checking first.
Route by difficulty, not by habit
Teams pick one capable model and send everything to it. But most traffic in a real product is not hard: classification, extraction, routing, short rewrites, simple answers from retrieved text. A small model does those as well and costs a fraction.
The pattern that works is a cheap classifier deciding the route, with an escalation path: if the small model's answer fails a confidence or validation check, retry on the large one. You pay the large model price only on the minority of turns that need it. In practice this lands between 40 and 70 percent of total spend, and it is the single largest structural saving available.
Stop resending everything
Three habits drive this, and all three are defaults nobody chose deliberately.
Full conversation history on every turn. Turn 30 resends turns 1 through 29. Keep the last few turns in full and a rolling summary of the rest. Quality rarely drops and the bill falls sharply.
Retrieving twenty passages when three would do. More context is not better context. Beyond a handful of relevant passages you are paying for tokens that also dilute the model's attention, so this is one of the rare changes that is cheaper and more accurate.
Whole documents instead of relevant sections. If you are pasting a 40 page PDF to answer a question about one clause, you are paying for 39 pages of nothing.
Cap the loops
This does not reduce your average bill; it prevents the invoice that ruins a month. An agent retrying a failing tool call will happily do it hundreds of times.
Set a maximum tool calls per turn, a maximum total tokens per conversation, and an alert on daily spend. All three are an afternoon of work. Teams add them after the first incident, and the incident always costs more than the afternoon.
Shorten the output
Output tokens cost several times more than input tokens on most models, so verbosity is expensive in a way that is easy to miss.
If you need structured data, ask for structured data and set a max tokens limit. A model producing a friendly paragraph explaining the JSON it is about to produce is burning the expensive kind of token for something you will discard.
Batch what nobody is waiting for
Most providers offer a batch mode at roughly half price for work that can tolerate a delay. Nightly summarisation, enrichment, classifying a backlog, generating embeddings: none of it is interactive, and all of it is usually running on the real time path because that is how it was first built.
What we would do in week one
- Add per request token logging tagged by feature.
- Look at what is actually making the calls. Delete anything that should not be.
- Move every stable instruction to the top of the prompt and turn on caching.
- Cap retries, loops and daily spend.
- Then, and only then, consider routing and model changes.
Steps one to four are days of work and routinely cut the bill by more than half. Step five is where the remaining structural saving is, and it is much easier to justify once you can measure the effect. Our notes on guardrails and evals cover the measurement side, which is what lets you change models without guessing at the quality cost.
Frequently asked questions
How do I reduce my LLM API costs?
In order of return: turn on prompt caching for a stable prefix, route easy turns to a smaller model with escalation when validation fails, stop resending full history and whole documents, cap retries and loops, shorten outputs, and move non interactive work to batch mode. The first three usually account for most of the bill.
Why is my OpenAI or Anthropic bill so high?
Usually volume of input tokens rather than difficulty of the work. The common causes are a long system prompt resent uncached on every call, full conversation history on every turn, retrieving far more passages than needed, and one background feature nobody considered making most of the calls. Log tokens per request by feature and the cause is normally obvious within a day.
Does prompt caching actually work?
Yes, and it is the cheapest large win available, typically 50 to 90 percent off the cached portion of input. The catch is that the prefix must be byte identical and come first. Injecting a timestamp or a user name near the top of the system prompt silently invalidates the cache on every call.
Should I use a smaller model to save money?
For most traffic, yes. Classification, extraction, routing, short rewrites and answering from retrieved text do not need a frontier model. Use a cheap model by default and escalate to the large one only when a validation or confidence check fails. That pattern typically cuts total spend by 40 to 70 percent.
Do output tokens cost more than input tokens?
Yes, several times more on most models. That makes verbosity disproportionately expensive. Request structured output, set a maximum token limit, and stop the model from narrating what it is about to do before doing it.
What is batch mode and when should I use it?
A provider mode that processes requests at roughly half price in exchange for a delay. Use it for anything nobody is waiting on: nightly summarisation, backlog classification, enrichment jobs and embedding generation. Much of this work runs on the interactive path only because that is how it was first written.
Where to start
Add token logging tagged by feature, then look at the top three callers after one day. The answer is almost always surprising, and it is almost always where the money is.
If the bill is already a problem and you want it looked at directly, send us the shape of your traffic and we will tell you which of these levers applies before you change anything.
