You automate support QA by having AI score every ticket against your own scorecard, then checking its scores against your human reviewers before you trust it. People keep calibration, disputed scores, high-risk tickets and coaching. The AI replaces the thin sample, not your judgment about what a good ticket looks like.
#Key takeaways
- AI scoring reads every ticket, so QA stops depending on a small sample.
- The scorecard stays yours: the head of support writes it and signs it off.
- Check the AI against your own reviewers on the same tickets before you rely on a single score.
- Complaints, high-risk cases and score disputes always go to a person, and coaching stays a human conversation.
#Why is a sampled QA review not enough?
A typical review reads a small slice of tickets. Illustrative example, the numbers are assumptions: a team closes 12,000 tickets and reviewers read 120. That is 1 percent. The other 99 percent are never scored, so a bad habit can run for a long time unseen.
AI scoring reads all of them. The gain is coverage, not a replacement for your reviewers. The aim is to point their attention at the tickets that need a person, instead of at whichever tickets the sample happened to pick.
#How do i set up AI scoring against my own scorecard?
Start with the scorecard you already use. Write each criterion as a clear question with a defined scale. Claude Developer Docs, Anthropic, accessed october 2026 says to structure evaluation so grading can be automated, including llm-graded scales such as a 1-5 tone rating on a support response.
For criteria, reuse what Anthropic's customer support agent guide reviews on a sample of conversations: whether the intent was understood, whether the reply was relevant, and whether escalations were right or wrong. Add your own, such as policy followed.
The head of support signs off the scorecard. The AI does not decide what good looks like. Stennir's AI consulting work starts here, with a scorecard in plain words that your reviewers already agree on.
#How do i know the AI scores can be trusted?
Before you trust a score, have two reviewers score the same 100 tickets blind, and let the AI score them too. Then compare. Illustrative example, the numbers are assumptions: the AI lands within one point of the reviewers on 90 of 100. Below your bar, reword the scorecard and run it again.
The same Anthropic page says it is best practice to grade with a different model from the one that produced the output. If a bot drafts replies, do not let that same model score them. Repeat the blind check for Arabic and English separately.
#What should stay with people?
- Calibration: reviewers re-score a fresh sample regularly and compare it with the AI.
- Disputes: when an agent challenges a score, a named reviewer decides.
- High-risk and complaint tickets: these go to a person and are not auto-scored.
- Coaching: the AI finds the pattern, a team lead has the conversation.
We are
Stennir scopes your scorecard, builds the scoring, and checks it against your own reviewers, so you decide what it may score.
We aren't
A QA dashboard that grades agents on its own rules, or ranks people with no human reviewing the result.
#What should a head of support do next?
Write down your current scorecard as plain questions, and pull 100 tickets your reviewers have already scored. That is enough to test the idea. See how AI for customer support fits your queue, then let Stennir's AI consulting scope it with you. book a 30-minute call, or take the free readiness assessment.
#Common questions
What does it need from us before it can score tickets?
It needs your scorecard written as clear questions, a set of tickets your reviewers have already scored, and read access to the ticket text. It also needs a rule for which tickets skip scoring and go straight to a person. The scorecard matters most, because the AI can only apply what you define.
Who decides what a good ticket looks like, and who signs off the scorecard?
The head of support decides, with the reviewers who know the work. The AI applies the scorecard and does not write it. Stennir helps turn your standards into clear questions with a scale, but your team signs off, and every later change goes through the same sign-off.
Does it score Arabic and English conversations the same way?
Stennir supports Arabic and English fully, others on request. You should still check each language on its own, because a score that matches your reviewers in English may not match them in Arabic. Run the same blind comparison on Arabic tickets and fix the wording before you rely on those scores.
What happens when an agent disagrees with a score?
The agent raises it, and a named reviewer re-scores the ticket by hand. That human score replaces the AI score. Keep the disputed tickets and use them to tighten the scorecard wording. If disputes cluster on one criterion, that criterion is probably unclear.
What could go wrong, and which tickets should always go to a human reviewer?
The AI can misread sarcasm, local phrasing or a missing policy detail, and score too kindly or too harshly. Send complaints, legal threats and safety cases to a person. Treat the AI score as a prompt for coaching, not a verdict on an agent.
