Experiment Summary
This experiment compares three scoring methods on the same small support ticket set:
- Manual reviewer scores from two human QA reviewers.
- AI-assisted QA scores from an internal prompt, LLM evaluation workflow, or QA tool.
- Optional vendor auto-QA output from a platform being evaluated.
The goal is not to crown a winner. The goal is to decide whether AI-assisted scoring is reliable enough for a specific use case: reporting, triage, coaching queues, compliance review support, or continued testing only.
What You Need
| Input | Minimum setup |
|---|---|
| Ticket sample | 30-50 anonymized or synthetic support conversations. |
| Scorecard | A frozen version of the AI customer support QA scorecard. |
| Human reviewers | Two reviewers scoring independently before discussion. |
| AI scorer | One AI-assisted workflow with score, rationale, and evidence snippets. |
| Optional vendor output | Auto-QA output from the tool being tested, kept separate from human review. |
| Calibration log | A table for score variance, critical misses, disagreement causes, and final decisions. |
Test Design
Build a sample that is small enough to review deeply and broad enough to catch real failure modes.
| Case mix | Suggested count | Why include it |
|---|---|---|
| Straightforward resolved tickets | 8-10 | Establish baseline agreement. |
| Reopened or low-CSAT tickets | 5-8 | Test whether QA catches unresolved issues. |
| Refund, billing, or cancellation cases | 5-8 | Test policy interpretation. |
| Technical troubleshooting cases | 5-8 | Test diagnosis and source grounding. |
| AI-agent conversations | 5-10 | Test grounding, escalation, and containment risk. |
| Synthetic edge cases | 3-5 | Test privacy, safety, unsupported claims, or outdated policy. |
Use a case ID format that does not expose private customer data. Example: SYN-001, ANON-014, or AI-EDGE-003.
Scoring Categories
Use the same rubric across all scoring methods.
| Category | Max points | Critical failure? |
|---|---|---|
| Issue understanding | 15 | No |
| Resolution accuracy | 25 | Yes |
| Policy/source alignment | 20 | Yes |
| Escalation handling | 15 | Yes |
| Tone and clarity | 10 | No |
| Documentation/follow-up | 10 | No |
| AI-specific controls | 5 | Yes for AI-agent cases |
AI scores should include evidence. A score without an excerpt, source reference, or rationale should be marked as weak evidence even if the number looks reasonable.
Sample Results Format
The table below is a synthetic example that Publisher can use as an illustrative body module.
| Case | Scenario | Reviewer A | Reviewer B | AI-assisted score | Vendor auto-QA | Calibration finding |
|---|---|---|---|---|---|---|
| SYN-004 | Refund policy exception | 72 | 76 | 91 | 88 | AI over-scored because it missed the required escalation. |
| SYN-011 | Password reset with account-risk language | 94 | 90 | 89 | 92 | Scores aligned; AI cited the correct policy excerpt. |
| AI-EDGE-002 | AI agent answered from stale KB article | 68 | 70 | 86 | 83 | AI scorer missed source freshness and unsupported claim risk. |
| ANON-019 | Agent was polite but did not solve the issue | 61 | 58 | 78 | 80 | AI overweighted tone; rubric needs stronger resolution weighting. |
| SYN-027 | Complex technical workaround | 84 | 72 | 75 | 74 | Human disagreement came from unclear partial-resolution rule. |
The pattern matters more than the average score. If AI scores are close on easy cases but miss escalation, policy, or source freshness, the workflow is not ready for sensitive uses.
Metrics to Calculate
| Metric | What it tells you | Pass signal |
|---|---|---|
| Human-human variance | Whether the scorecard is clear enough for reviewers. | Category variance is explainable and reduced after calibration. |
| AI-human variance | Whether AI scoring tracks calibrated human review. | Variance is low on approved categories and high-risk misses are rare. |
| Critical-failure miss rate | Whether AI misses policy, safety, privacy, escalation, or source failures. | No unresolved critical misses before sensitive use. |
| Evidence quality rate | Whether AI scores can be audited. | Most AI scores include usable excerpts or source references. |
| False auto-fail rate | Whether AI over-penalizes acceptable conversations. | Disputed auto-fails are rare or routed to human review. |
| Low-confidence routing rate | Whether staffing can handle the review queue. | Review volume is realistic for the QA team. |
Decision Matrix
Use this matrix after the calibration meeting.
| Result | Decision |
|---|---|
| Humans disagree often | Fix the scorecard before testing vendors or AI prompts further. |
| Humans align, AI misses critical failures | Limit AI to draft notes or triage; require human audit for risk categories. |
| Humans align, AI evidence is weak | Require excerpts, policy links, or rationale before scores enter dashboards. |
| AI aligns on low-risk categories only | Use AI for trend reporting or reviewer prioritization, not final coaching decisions. |
| AI aligns by queue but fails on AI-agent cases | Split scorecards for human-agent and AI-agent conversations. |
| AI aligns with evidence across the sample | Consider a controlled pilot for coaching queues or reporting with dispute handling. |
How This Connects to Tool Selection
After this experiment, use the findings to evaluate tools. The AI customer support QA scorecard tools guide and AI contact center QA software guide should be read through the lens of your calibration results.
If your experiment shows source-grounding problems, prioritize evidence and knowledge-base controls. If it shows escalation misses, prioritize low-confidence routing and human audit workflows. If it shows reviewer disagreement, fix the rubric before buying more automation.
Related ClawNewbie pages:
- AI customer support QA calibration checklist - AI support tools hub - Zendesk AI - Intercom Fin - Intercom Fin vs Zendesk AI - AI conversation intelligence tools - LLM evaluation tools
Experiment Caveats
This experiment does not prove legal compliance, vendor superiority, or universal model accuracy. It only documents how one team can test scoring consistency on a defined ticket sample, scorecard version, AI scorer version, and calibration workflow.
Before using results externally, verify that the sample is anonymized, the vendor claims are sourced, and the approved use level is limited to what the evidence supports.
FAQ
Is this experiment a benchmark?
No. It is a practical calibration test for one team, one sample, and one scorecard. It should not be presented as a public benchmark across vendors.
What sample size should a small team use?
Start with 30-50 conversations for the first calibration test. Expand the sample when multiple languages, queues, channels, products, or regulated workflows are involved.
Should the AI scorer see the human scores?
No. Human reviewers and AI scoring should run independently first. Compare results only after blind scoring is complete.
What is a failed experiment?
The experiment fails if reviewers cannot apply the scorecard consistently, AI scores lack evidence, or AI scoring misses critical failures that would affect customers, compliance, or escalation.
What should happen after the test?
Update the rubric, prompts, evidence requirements, review thresholds, and tool evaluation criteria. Then run the sample case study pattern on a representative support queue.