How to Cut an LLM API Bill 50-70% Without Anyone Noticing a Quality Drop

The bill is big because nothing is measuring it

An LLM bill that has quietly grown past what anyone expected almost always has the same shape. There is one model name hardcoded across the codebase, it is the best model the team could find during the prototype, every request goes to it at whatever context length the retrieval step happened to produce, and nobody can tell you which feature is responsible for which share of the spend. The invoice is a single number. There is nothing to act on in a single number.

So the first move is not a saving. It is getting the data that tells you which savings are available.

Lever 0: a gateway, because you cannot cut what you cannot see

Put every LLM call behind one proxy. LiteLLM is the usual choice because it speaks the OpenAI API shape and fronts most providers, so the application change is a base URL and a key:

client = OpenAI(
    base_url="https://llm-gateway.internal/v1",
    api_key=os.environ["GATEWAY_KEY"],
)

resp = client.chat.completions.create(
    model="support-answer",          # a route name, not a vendor model name
    messages=messages,
    extra_headers={"x-route-tag": "support/answer"},
)

The important detail is that the application now asks for support-answer, a route, rather than claude-sonnet-4-5 or gpt-5, a model. Which model serves that route becomes a gateway config value you can change without a deploy, and every request carries a tag you can group spend by.

model_list:
  - model_name: support-answer
    litellm_params:
      model: anthropic/claude-sonnet-4-5
  - model_name: support-classify
    litellm_params:
      model: anthropic/claude-haiku-4-5
litellm_settings:
  cache: true
  cache_params:
    type: redis
    ttl: 3600
  success_callback: ["prometheus"]

Give it a week. What comes back is a cost-per-route table, and it is almost never what the team predicted. The expensive thing is rarely the flagship feature. It is usually a background classifier nobody thinks about that runs on every inbound event, or a retrieval step that grew from five chunks to twenty during a quality push six months ago and never got tuned back down.

Lever 1: cache the parts of the prompt that never change

Most production prompts are a long stable prefix (system instructions, tool definitions, few-shot examples, sometimes a whole document) followed by a short variable suffix (this user’s actual question). Sending that prefix on every request and paying full input-token price for it is the single most mechanical waste in the average bill.

Both Anthropic and OpenAI support prompt caching on the prefix. With Anthropic it is explicit — you mark the cache breakpoint:

messages = [{
    "role": "system",
    "content": [{
        "type": "text",
        "text": SYSTEM_PROMPT + TOOL_DOCS,   # long, identical every call
        "cache_control": {"type": "ephemeral"},
    }],
}]

The rule that decides whether this works is prefix stability: the cached portion must be byte-identical across calls. The classic mistake is interpolating a timestamp, a request ID, or the user’s name into the system prompt, which invalidates the prefix on literally every request and produces a cache with a zero percent hit rate that everybody assumes is working. Sort your retrieved chunks deterministically for the same reason — if the retriever returns the same five documents in a different order, you have a different prefix.

Separately from provider-side prompt caching, cache whole responses at the gateway for requests that genuinely repeat. In a support or docs assistant, a meaningful share of questions are the same question. An exact-match cache on a normalized prompt hash is safe and cheap. A semantic cache, keyed on embedding similarity, catches more but will eventually serve a near-miss answer to a subtly different question, so gate it behind a high similarity threshold and keep it off anything where the answer depends on per-user state.

Lever 2: route by difficulty, not by habit

This is where the structural saving is. Look at the route table from the gateway and sort the traffic by what the request actually demands:

• Classification, extraction, routing, tagging, short summarization. A small fast model handles these at parity. This is usually a large share of call volume and it is nearly always sitting on a frontier model purely because that is what the first prototype used.
• Retrieval-grounded answering. The hard work is in retrieval. Once the right context is in the prompt, mid-tier models are frequently indistinguishable from frontier ones, because the task has been reduced to reading comprehension over supplied text.
• Multi-step reasoning, code generation, ambiguous judgment calls. Keep these on the frontier model. This is what you are paying for, and it is a smaller slice of traffic than it feels like.

The thing that makes this safe rather than reckless is that you decide with an eval set, not with a vibe. Pull 150 real requests from logs, spanning the classes above. Grade the current model’s answers once — human-graded, or LLM-judged against a rubric with a human spot-check of the disagreements. Then run the same set against the cheaper candidate and compare. You are looking for the request classes where the cheaper model scores within tolerance, and you move only those.

for case in eval_set:
    cheap = call("support-classify", case.prompt)
    good  = call("support-answer",  case.prompt)
    record(case.id, case.klass, grade(cheap, case.expected), grade(good, case.expected))
# then: per-class score delta and per-class cost delta, side by side

The output is a table of “this class of request, this quality delta, this cost delta,” and the routing decisions fall out of it. Keep the eval set in CI so a later prompt change or model upgrade cannot silently regress the classes you moved.

Lever 3: stop sending tokens the model does not use

Token trimming is unglamorous and it compounds with everything else, because it shrinks the input to every lever above.

The biggest single offender in RAG systems is top_k set high during an early quality push. Retrieving twenty chunks when five would do multiplies input cost on every request, and past a point it actively degrades answer quality — relevant context gets buried among marginally-related passages. Re-rank the retrieved set and pass the top few rather than passing everything the vector search returned. This is one of the recurring themes in what goes wrong with self-hosted RAG in production: the retrieval tuning that helps quality and the tuning that helps cost point in the same direction more often than teams expect.

Then check for these, in roughly this order of how often they show up:

• Unbounded conversation history. Full transcripts replayed on every turn. Summarize older turns past a window instead of resending them verbatim.
• Verbose tool definitions. Long JSON schemas with prose descriptions for every field, resent on every call, when a terse schema works identically.
• Retry storms. A tool call that fails schema validation and gets retried three times has cost you four full requests to produce one answer. Validate the arguments and repair them locally where you can, rather than paying the model to try again.
• Output length. max_tokens set generously and a prompt that never asks for brevity. Output tokens usually cost several times what input tokens cost.

Prove you did not break anything

Every one of these levers is reversible, and that is the point. The order I use is the order of blast radius: gateway first (visibility, no behaviour change), caching second (no behaviour change if the prefix is genuinely static), routing third (real behaviour change, gated by evals), trimming fourth (real behaviour change, gated by evals).

Ship each behind a percentage rollout, keep the eval suite running against production traffic samples, and watch three numbers together: cost per route, eval score per request class, and your actual product metric — resolution rate, deflection rate, whatever the feature exists to move. Cost falling while the eval score holds and the product metric holds is a real win. Cost falling while you only watch cost is how you find out two months later that answer quality slid and nobody connected the two.

On a gateway engagement that cut a client’s monthly LLM spend around 65%, the split was roughly what this article predicts: caching and prompt trimming were the fast mechanical wins available in the first week, and routing the classification and extraction traffic off the frontier model was the structural change that held the number down afterward. Nothing about the product got worse, because the eval set said so before each change shipped, not after.

If your bill is growing faster than your usage and nobody can point at which feature owns which share of it, that is a measurement problem before it is a cost problem. That is the work I do under LLM and AI cost optimization, and it is usually a short engagement that pays for itself inside a billing cycle.