This year we shipped an assistant that reads incoming support tickets, suggests a category and priority, and drafts a first reply for a human to approve. The model was the easy part. Here is what took the real work.
Start with the humans’ labels
Before writing a prompt, we exported six months of tickets and how the support team had actually handled them. That history became our evaluation set. Without it, “the AI seems good” is an opinion; with it, it’s a number you can improve.
Confidence decides the workflow
The assistant doesn’t just give an answer — it says how sure it is. We route on that:
| Confidence | What happens | Share of tickets |
|---|---|---|
| High | Category and priority applied automatically; reply drafted | ~62% |
| Medium | Suggestion shown, agent confirms with one click | ~27% |
| Low | No suggestion — goes to the normal queue | ~11% |
That last row matters. An assistant that knows when to stay quiet earns far more trust than one that always has an opinion.
Humans stay in the loop — visibly
- Every AI-drafted reply is marked as a draft; nothing is sent without an agent.
- Agents can correct the category in one click, and every correction feeds the next evaluation run.
- Customers never talk to the model directly. It works for the team, not instead of it.
The goal was never to remove people from support. It was to remove the part of the job nobody wanted.
Measure what the team feels
Accuracy went up week by week, but the metric the team cared about was simpler: time to first human reply dropped by almost half. Triage stopped being a morning chore and became a glance.
If you’re considering AI in an internal workflow, start small, keep a human on every decision that reaches a customer, and let your own history tell you whether it’s working.