Case Study Summary
A mid-market SaaS support team wants to use AI-assisted QA to review more tickets without losing trust in the scoring process. The team already has a support QA scorecard, but managers are seeing three problems:
- Human reviewers disagree on partial resolutions.
- AI-assisted scores are higher than manual scores on polite but incomplete answers.
- AI-agent conversations are being scored like human-agent replies, which hides grounding and escalation failures.
The team runs a calibration cycle on 42 anonymized or synthetic cases before allowing AI scores into coaching dashboards.
Starting Setup
| Item | Starting state |
|---|---|
| Support channels | Email, chat, and AI-agent front door. |
| Ticket sample | 42 cases across billing, account access, product troubleshooting, refunds, and AI-agent containment. |
| Scorecard | Seven categories based on the AI customer support QA scorecard template. |
| Reviewers | Two QA reviewers plus one support operations lead. |
| AI scoring | AI-assisted scoring output with category score, rationale, and evidence excerpt. |
| Calibration goal | Decide whether AI scores can support coaching queues or should remain in testing. |
Problem 1: Score Drift by Queue
The first review showed that billing cases were scoring higher in AI-assisted QA than in manual review.
| Queue | Human average | AI-assisted average | Gap | Calibration note |
|---|---|---|---|---|
| Billing | 71 | 86 | +15 | AI missed refund-policy escalation requirements. |
| Account access | 82 | 84 | +2 | Scores mostly aligned. |
| Product troubleshooting | 76 | 79 | +3 | Partial-resolution rule was unclear. |
| AI-agent conversations | 68 | 88 | +20 | AI scorer overweighted containment and tone. |
The team did not treat the average score as proof that AI was better or worse. Instead, they reviewed category-level evidence.
Problem 2: Escalation Misses
Three synthetic/anonymized cases exposed a pattern.
| Case | What happened | Human finding | AI-assisted finding | Fix |
|---|---|---|---|---|
| BILL-014 | Customer asked for a refund exception after a failed renewal notice. | Escalation required. | Marked as resolved because answer was polite. | Add refund exception as an auto-fail if not escalated. |
| AI-006 | AI agent answered using an outdated billing article. | Source freshness failure. | Marked as accurate. | Require source date and current policy link for AI-agent cases. |
| TECH-021 | Agent gave a workaround but no permanent fix. | Partial resolution. | Full resolution. | Add examples for workaround, mitigation, and full resolution. |
The key lesson: escalation misses are not just lower scores. They are workflow risk. The team moved escalation handling into the critical-failure section of the rubric.
Problem 3: Reviewer Disagreement
Human reviewers disagreed most often when a customer received a workaround that reduced pain but did not solve the root issue.
| Pattern | Reviewer A interpretation | Reviewer B interpretation | Calibration decision |
|---|---|---|---|
| Workaround without root fix | Resolution should be low. | Resolution is acceptable if the customer can proceed. | Score as partial resolution unless a permanent fix or owner is documented. |
| Polite answer with wrong policy | Tone should preserve some score. | Policy error should dominate. | Critical policy errors cap total score. |
| AI-agent containment | Containment suggests success. | Containment can hide bad answers. | AI-agent cases require grounding and escalation checks. |
The team updated the scorecard examples before running the next scoring pass.
Calibration Fixes
| Fix | Owner | Why it mattered |
|---|---|---|
| Add partial-resolution examples | QA lead | Reduced reviewer disagreement on technical cases. |
| Add refund exception auto-fail | Support ops | Prevented policy-sensitive misses from averaging out. |
| Add AI-agent grounding category | AI support owner | Separated AI containment from answer quality. |
| Require evidence snippets for AI scores | QA analyst | Made scores auditable. |
| Route high-variance cases to human review | QA lead | Prevented low-confidence automation from reaching coaching dashboards. |
| Schedule monthly drift check | Support ops | Caught score shifts after policy or prompt changes. |
Before and After
| Metric | Before calibration | After calibration pass |
|---|---|---|
| Human-human category variance on partial-resolution cases | High | Lower after examples were added |
| AI-human gap on billing cases | +15 points | Reduced after escalation auto-fail rule |
| AI-agent grounding misses | Frequent in sample | Routed to human review unless evidence is present |
| Coaching-ready AI scores | Not approved | Approved only for low-risk triage and draft notes |
Do not present these as benchmark numbers. They are illustrative outcomes from a synthetic/anonymized sample workflow.
Final Decision
The team did not approve AI-assisted QA as a final scoring authority. It approved a narrower workflow:
| Use case | Decision |
|---|---|
| Trend reporting | Allowed for low-risk categories after human spot checks. |
| Coaching queue prioritization | Allowed as a draft signal with manager review. |
| Compliance-sensitive cases | Human review required. |
| Refund exceptions | Human review required. |
| AI-agent source-grounding failures | Human review required until evidence quality improves. |
| Vendor evaluation | Use the same scorecard in the proof of concept. |
This decision creates a practical bridge between manual QA and AI-assisted review without pretending automation is fully calibrated.
How to Reuse This Pattern
- Build or update the AI support QA scorecard.
- Run the AI customer support QA calibration test on a balanced sample.
- Use the calibration checklist to run the meeting.
- Compare tools with the AI customer support QA scorecard tools guide.
- Use the AI contact center QA software guide for broader platform selection.
For vendor and category context, link to the AI support hub, Zendesk AI, Intercom Fin, Intercom Fin vs Zendesk AI, AI conversation intelligence tools, and LLM evaluation tools.
FAQ
Is this a real customer case study?
No. It is a synthetic/anonymized sample designed to show the decisions a support team should document during AI QA calibration.
Why not approve AI QA for every ticket after calibration?
Calibration can support controlled use, but high-risk cases still need human review when privacy, refunds, safety, account access, regulated language, or AI-agent escalation is involved.
What is score drift?
Score drift happens when scores change because the scoring process changed, not because support quality changed. It can come from reviewer turnover, prompt edits, policy updates, knowledge-base changes, or channel mix.
What should a team fix first?
Fix the rubric before fixing the AI scorer. If human reviewers cannot apply the scorecard consistently, AI scoring comparisons will not be meaningful.
Which ClawNewbie pages should this case study link to?
Use the scorecard tutorial, calibration experiment, calibration checklist, scorecard tools review, contact-center QA review, AI support hub, Zendesk AI, Intercom Fin, and support-agent comparison pages.