A queue full of the same questions.
Ticket volume was climbing faster than the team could hire, and most of it was repetitive: password resets, plan changes, “where's my invoice.” A previous off-the-shelf bot had made things worse — it guessed, got billing answers wrong, and trained customers to type “agent” immediately.
The brief was narrow and correct: resolve the boring tickets perfectly, never guess on anything touching money or accounts, and make every hand-off to a human start ten steps ahead.
Build the eval harness before the first prompt.
Trust in an agent comes from evaluation, not from the model. So we built the harness first: a golden set of 600 real conversations, each graded for correctness, tone, and whether the agent should have escalated. Nothing ships unless it clears the bar.
The second rule was a hard boundary. The agent can read anything, but any action that moves money or changes an account requires either a verified tool call with guardrails or a human. It is allowed to say “I don't know” and route — and that is treated as a success, not a failure.
Retrieval, tools, and a strict gate.
Grounded it in four years of tickets.
RAG over the full resolved-ticket history and the help centre, with citations back to the source so answers are checkable, not invented.
Gave it real tools, behind guardrails.
Read-only tool calls into the billing and account APIs for facts, and a small set of write actions (resend invoice, reset link) that are idempotent and logged.
Made escalation a first-class outcome.
When confidence is low or the topic is sensitive, it summarises the conversation, attaches the relevant account context, and routes to the right queue.
Built the eval harness and wired it to CI.
A 600-conversation golden set graded on every change, with a required pass rate before deploy and a weekly review of drift on live traffic.
Put observability on every answer.
Each response is traced with its retrieved sources, tool calls, and confidence, so the support team can audit any answer in seconds.
The boring third, handled. The hard cases, accelerated.
Live, the agent resolves 38% of incoming ticketsend-to-end with no human, and median first-response time fell from five hours to under one. Crucially, it has never taken an unauthorised account action — the guardrails, not the model, are what made that true.
The cases it can't close arrive on a human's desk summarised and pre-tagged, so the team spends its time on judgment instead of triage.