Shipped a self-hosted RAG assistant with an LLM gateway
A team with a promising RAG demo couldn't ship it. Accuracy was unmeasured and API costs were unpredictable. A measured pipeline behind an LLM gateway with hybrid serving made it production-ready and ~70% cheaper.
The problem
A product team had a RAG assistant that demoed well but couldn't go live. Retrieval accuracy was a guess, the model sometimes confidently hallucinated, and projected LLM API costs at scale were alarming and unpredictable.
The approach
- ▸Built an evaluation harness so retrieval and answer quality became real numbers, then fixed chunking and added re-ranking to lift retrieval accuracy.
- ▸Added anti-hallucination guardrails (grounding checks and a graceful "I don't know" path) so wrong answers stopped going out as confident ones.
- ▸Put an LLM gateway in front to route simple queries to a self-hosted open model on GPU (vLLM) and hard ones to a frontier API, with caching and per-request cost tracking.
- ▸Deployed model serving on Kubernetes with autoscaling so GPU capacity followed demand instead of running hot all day.
The result
- ✓RAG accuracy became a tracked metric the team could improve against
- ✓~70% lower inference cost versus the all-API baseline
- ✓Predictable spend with per-query cost visibility through the gateway
- ✓A serving stack the team can operate without outside help
stack: RAG · LLM Gateway · vLLM · Kubernetes · GPU serving
A good demo tells you nothing
The blocker was never the model. It was that nobody was measuring anything. A RAG demo will look great on the three questions you tried and fall apart on the fourth, and you won’t know until a user finds it. Once retrieval quality and hallucination rate were numbers on a dashboard, the team could improve them and decide what was good enough to ship. The cost win came from the gateway. Most queries don’t need a frontier model, so routing the easy ones to a self-hosted model and caching the repeats cut the bill hard without hurting quality.
Done by Harshit Luthra, an independent infrastructure and AI engineering consultant. Bring me a similar problem →
further reading
The method behind this
Self-Hosted RAG in Production: The Issues That Never Show Up in the Demo
Why RAG systems that demo beautifully fall apart in production, and the concrete fixes for retrieval quality, hallucination, cost, and staleness that a demo never forces you to solve.
13 min readHow to Cut an LLM API Bill 50-70% Without Anyone Noticing a Quality Drop
The four levers that actually move an OpenAI or Anthropic bill — gateway visibility, caching, model routing, and token trimming — and how to prove you didn't degrade quality doing it.
Questions about this work
How does an LLM gateway cut RAG inference cost by ~70%?+
Most queries don't need a frontier model. The gateway routed simple queries to a self-hosted open model on GPU (vLLM) and reserved the frontier API for hard ones, then cached repeats. That mix cut inference cost ~70% versus sending everything to the API, with no drop in answer quality because the easy traffic never needed the expensive model.
When is self-hosting an LLM worth it over an API?+
Above a certain steady volume, and when you want data control. APIs win on time to market and frontier quality with zero ops. Self-hosting wins on cost at scale but you take on GPU capacity and uptime. A hybrid behind a gateway often wins, which is what shipped here. I model real token volume to find the break-even.
How do you make a RAG demo production-ready?+
You turn accuracy into a number. An eval harness scored retrieval and answer quality, then chunking fixes and re-ranking lifted retrieval, and grounding plus an "I don't know" path stopped confident wrong answers. Once quality is measured, the team can decide what's good enough to ship.