Cut a company's LLM API bill by ~65% with a gateway
A product team was sending every request to a frontier model and watching the bill climb past usefulness. A gateway with caching, model routing, and prompt trimming cut spend ~65% with no drop in output quality.
The problem
A team had shipped an LLM feature fast by routing every call to the most capable model available. It worked, then it scaled, and the API bill went from a rounding error to a line item finance asked about every month. Nobody could see which features or queries drove the cost.
The approach
- ▸Put a gateway in front of every LLM call so spend became visible per feature, per route, and per model before changing anything.
- ▸Added a semantic cache for repeated and near-identical queries, which alone absorbed a large share of traffic that did not need a fresh generation.
- ▸Routed easy queries (classification, short rewrites, simple lookups) to a cheaper model and reserved the frontier model for genuinely hard generation.
- ▸Trimmed bloated prompts and oversized context windows, cutting input tokens per call without measurably hurting answer quality.
The result
- ✓~65% reduction in monthly LLM API spend
- ✓Caching served roughly 35% of requests with no model call at all
- ✓Per-feature cost dashboards, so spend stopped being a mystery line item
- ✓No measurable drop in output quality on tracked evals
stack: LLM Gateway · OpenAI · caching · model routing · prompt optimization
Most requests do not need your best model
The bill was high for a simple reason. Every call went to the strongest model, including the ones doing trivial work like tagging a message or rewriting a sentence. Once the gateway made spend visible per route, the fix picked itself. Cache the repeats, send the easy traffic to a cheaper model, and stop shipping 4,000-token prompts where 800 did the job. The frontier model still handles the hard generation, which is what it is worth paying for. The discipline that lasts is the per-feature cost view. When a new feature spikes the bill, the team sees exactly which one and why.
Done by Harshit Luthra, an independent infrastructure and AI engineering consultant. Bring me a similar problem →
Questions about this work
How do you cut an LLM API bill by ~65% without losing quality?+
Three levers. A semantic cache absorbed repeated and near-identical queries (about 35% of traffic with no model call), easy queries routed to a cheaper model while the frontier model kept the hard generation, and bloated prompts got trimmed. Tracked evals showed no measurable quality drop, because the expensive model still handled the work that needed it.
What does an LLM gateway give you besides cost savings?+
Visibility first. Per-feature, per-route, per-model spend data, so you know what to optimize before touching anything. Plus routing, caching, rate limits, and fallbacks across providers. When a new feature spikes the bill, you see exactly which one.
How much traffic can caching realistically absorb?+
Here a semantic cache served roughly 35% of requests with no model call at all. The exact share depends on how repetitive your traffic is, but most production LLM features have more near-duplicate queries than teams expect.