Shipped a self-hosted RAG assistant with an LLM gateway
A team with a promising RAG demo couldn't ship it. Accuracy was unmeasured and API costs were unpredictable. A measured pipeline behind an LLM gateway with hybrid serving made it production-ready and ~70% cheaper.
The problem
A product team had a RAG assistant that demoed well but couldn't go live. Retrieval accuracy was a guess, the model sometimes confidently hallucinated, and projected LLM API costs at scale were alarming and unpredictable.
The approach
- ▸Built an evaluation harness so retrieval and answer quality became real numbers, then fixed chunking and added re-ranking to lift retrieval accuracy.
- ▸Added anti-hallucination guardrails (grounding checks and a graceful "I don't know" path) so wrong answers stopped going out as confident ones.
- ▸Put an LLM gateway in front to route simple queries to a self-hosted open model on GPU (vLLM) and hard ones to a frontier API, with caching and per-request cost tracking.
- ▸Deployed model serving on Kubernetes with autoscaling so GPU capacity followed demand instead of running hot all day.
The result
- ✓RAG accuracy became a tracked metric the team could improve against
- ✓~70% lower inference cost versus the all-API baseline
- ✓Predictable spend with per-query cost visibility through the gateway
- ✓A serving stack the team can operate without outside help
stack: RAG · LLM Gateway · vLLM · Kubernetes · GPU serving
A good demo tells you nothing
The blocker was never the model. It was that nobody was measuring anything. A RAG demo will look great on the three questions you tried and fall apart on the fourth, and you won’t know until a user finds it. Once retrieval quality and hallucination rate were numbers on a dashboard, the team could improve them and decide what was good enough to ship. The cost win came from the gateway. Most queries don’t need a frontier model, so routing the easy ones to a self-hosted model and caching the repeats cut the bill hard without hurting quality.
Done by Harshit Luthra, an independent infrastructure and AI engineering consultant. Bring me a similar problem →
Questions about this work
How does an LLM gateway cut RAG inference cost by ~70%?+
Most queries don't need a frontier model. The gateway routed simple queries to a self-hosted open model on GPU (vLLM) and reserved the frontier API for hard ones, then cached repeats. That mix cut inference cost ~70% versus sending everything to the API, with no drop in answer quality because the easy traffic never needed the expensive model.
When is self-hosting an LLM worth it over an API?+
Above a certain steady volume, and when you want data control. APIs win on time to market and frontier quality with zero ops. Self-hosting wins on cost at scale but you take on GPU capacity and uptime. A hybrid behind a gateway often wins, which is what shipped here. I model real token volume to find the break-even.
How do you make a RAG demo production-ready?+
You turn accuracy into a number. An eval harness scored retrieval and answer quality, then chunking fixes and re-ranking lifted retrieval, and grounding plus an "I don't know" path stopped confident wrong answers. Once quality is measured, the team can decide what's good enough to ship.