Took a support bot from 72% to 96% answer accuracy
A support bot was already live and quietly wrong about a quarter of the time, with no one measuring it. An eval harness, fixed retrieval and grounding, and guardrails took accuracy from 72% to 96%.
The problem
A company had a support chatbot in production that nobody trusted. It sounded confident and was wrong often enough that agents quietly told customers to ignore it. There were no metrics, so every complaint was a debate about whether the bot was actually broken or just had a bad day.
The approach
- ▸Built an eval harness with a labeled set of real questions and known-good answers, turning "the bot feels wrong" into a measured 72% accuracy baseline.
- ▸Traced failures and found most were retrieval, not the model, so I fixed chunking and added a re-ranking pass to put the right context in front of the model.
- ▸Added grounding checks and an "I don't know" path, so the bot stopped fabricating answers when retrieval came back empty or weak.
- ▸Wired the evals into CI, so every prompt or retrieval change was scored before shipping instead of being a guess.
The result
- ✓Answer accuracy rose from 72% to 96% on the held-out eval set
- ✓Hallucinated answers on empty retrieval dropped to near zero with the grounding path
- ✓Regression evals in CI caught quality drops before they reached production
- ✓Agent trust recovered enough that they stopped routing around the bot
stack: evals · RAG · OpenAI · re-ranking · guardrails
You cannot fix what you refuse to measure
The hardest part was not technical. It was that the bot was shipped without a way to tell good from bad, so every conversation about it was vibes. The eval harness ended that. Once 72% was on a screen, the failures were traceable, and most of them were retrieval, not the model hallucinating for fun. Better chunking and a re-ranker fed it the right context, and the grounding path handled the rest by letting the bot admit when it had nothing. Putting the evals in CI is what makes the 96% hold. Quality stops being a thing you hope for and becomes a thing the pipeline checks.
Done by Harshit Luthra, an independent infrastructure and AI engineering consultant. Bring me a similar problem →
Questions about this work
How do you measure whether a chatbot's answers are accurate?+
Build an eval set, 50-200 real questions with known-good answers, and score retrieval and generation separately on every change. That turned "the bot feels wrong" into a measured 72% baseline. Without it, every complaint is a debate about whether the bot is broken or just had a bad day.
What actually fixes a hallucinating support bot?+
Usually retrieval, not the model. Most failures here traced to the wrong context reaching the prompt, so better chunking and a re-ranking pass fixed the bulk of it. A grounding check plus an "I don't know" path handled the rest by letting the bot admit when retrieval came back empty. Accuracy went 72% to 96%.
How do you keep accuracy from regressing later?+
Wire the evals into CI so every prompt or retrieval change is scored before it ships. That's what makes the 96% hold, quality stops being something you hope for and becomes something the pipeline checks.