What the Eval Rubric Generator actually produces

1. REQUIRED: Category Accuracy - Measures: Does the assigned category (billing/technical/refund/general) match the labeled ground truth? - Scoring: Binary (correct/incorrect) - Examples: - PASS: Email about invoice dispute → billing - FAIL: Email about product defect → general (should be technical) - Threshold for investigation: >5% error rate on any category 2. REQUIRED: Priority Alignment with SLA Rules - Measures: Does assigned priority (urgent/high/medium/low) follow your documented SLA rules (e.g., churn-risk language = urgent, refund requests = high)? - Scoring: Binary (correct/incorrect) - Examples: - PASS: "Considering switching to competitor" + refund request → urgent - FAIL: "Considering switching to competitor" → low priority - Threshold for investigation: Any failure on churn-risk detection, or >3% misalignment on high-priority cases 3. REQUIRED: No Policy-Violating Commitments - Measures: Does the draft response promise/authorize actions outside policy scope (refunds not pre-approved, technical fixes requiring escalation, discounts beyond limits)? - Scoring: Binary (violates policy/compliant) - Examples: - PASS: "I'll escalate your refund request to our billing team for review" (no commitment) - FAIL: "We'll issue a full refund immediately" (violates policy if email shows ineligible reason) - Threshold for investigation: Any single violation; stop the model at 1 violation per 50 emails 4. REQUIRED: Attachment Context Acknowledged - Measures: If the email explicitly mentions or includes an attachment (error log, screenshot, receipt), does the response reference or address it? - Scoring: Binary (acknowledged/ignored) - Examples: - PASS: Email: "See attached error.log" → Response: "I've reviewed the error log you attached and..." - FAIL: Email: "See attached error.log" → Response: generic troubleshooting with no mention of attachment - Threshold for investigation: >10% miss rate on emails with explicit attachments 5. OPTIONAL: Factual Hallucination Check - Measures: Does the draft invent account details, order numbers, or dates not present in the email? (LLM-as-judge) - Scoring: Scale 1-5: 1=multiple false details, 3=one possible misstatement, 5=no invented facts - Judge prompt: "Does this response reference specific account details, orders, or dates that were not in the original email? Note what is invented." - Judge reliability: Run 2 independent judge passes; if scores differ >1 point, escalate to human review - Examples: - PASS: Email vague on order details → Response: "Could you share your order number so I can look into this?" - FAIL: Email doesn't mention order ID → Response: "I see order #12345 was placed on March 15..." - Threshold for investigation: Any hallucinated fact >3; average score <4.0 across eval set 6. OPTIONAL: Response Conciseness & Actionability - Measures: Is the draft response under 200 words and does it clearly state the next step (what the agent or customer should do)? - Scoring: Binary (clear next step present / absent) - Examples: - PASS: "I'm checking with our tech team now. You'll hear from me by end of day with next steps." - FAIL: "Thank you for contacting us. We take all feedback seriously..." [ends with no action] - Threshold for investigation: >20% missing next-step statements 7. OPTIONAL: Tone Appropriateness to Category - Measures: Does the tone match the category and priority? (e.g., urgent issues warrant empathy; general inquiries can be lighter) — LLM-as-judge - Scoring: Scale 1-5: 1=dismissive/wrong register, 3=neutral but safe, 5=appropriately empathetic or professional - Judge prompt: "Is the tone in this response appropriate for a [category] issue at [priority] level? Would a customer feel heard?" - Judge reliability: Single judge pass; flag for human review only if score <3 - Examples: - PASS: Urgent churn risk: "I understand your frustration. I'm personally escalating this..." - FAIL: Urgent churn risk: "We appreciate your business. We'll review your case." (flat, formulaic) - Threshold for investigation: >15% scores <3; any score of 1 8. OPTIONAL: Constraint Adherence (Formatting & Length) - Measures: Does the draft fit in a single email window (~150–250 words) and avoid legal disclaimers not in your template? - Scoring: Binary (meets constraints / exceeds) - Examples: - PASS: 180-word response, no extra legal boilerplate - FAIL: 450-word response with custom liability language - Threshold for investigation: >10% constraint violations

1. Multilingual emails (e.g., subject in English, body in Spanish): Criterion 1 (Category Accuracy) and Criterion 2 (Priority Alignment) — verify model doesn't default to 'general' and miss priority signals in non-English text. 2. Churn-risk language with low urgency metadata (e.g., "considering switching" but low account tenure/spend): Criterion 2 (Priority Alignment) — ensure model weights explicit churn language even when customer lifetime value is low. 3. Refund request with ambiguous eligibility (return window vs. non-refundable service): Criterion 3 (Policy-Violating Commitments) — model must draft escalation, not promise. 4. Email with 2+ attachments, only 1 relevant (e.g., unrelated invoice + error screenshot): Criterion 4 (Attachment Context) — verify model addresses the *relevant* attachment, not all. 5. Technical issue that requires customer password/sensitive data in response: Criterion 3 (Policy-Violating Commitments) — catch responses that ask for passwords in plain-text draft. 6. Sarcasm or frustration expressed as technical complaint ("Your product is terrible, fix it"): Criterion 2 (Priority Alignment) and Criterion 7 (Tone Appropriateness) — ensure model detects urgency despite hostile tone and responds with empathy. 7. Email from known high-value customer with generic complaint: Criterion 2 (Priority Alignment) — model should *not* boost priority based on account metadata not in email; priority is determined by issue type + language, not CRM data. 8. Attachment filename suggests relevance but email text doesn't mention it (e.g., "error_critical.log" attached but email says "general question"): Criterion 4 (Attachment Context) — model should flag mismatch, not assume attachment intent.

CADENCE & TRIGGERS: - Runs nightly on full labeled test set (~300 tickets) + any new labeled tickets added that week - Manual eval run on-demand before any model release - Weekly spot-check: 10% random sample of draft responses scored by human agent (5 raters, majority vote on optional criteria) KILL CRITERIA (auto-halt deployment): - Required Criterion 1 (Category Accuracy): <93% (regression of >2% from last passing run) - Required Criterion 2 (Priority Alignment): Any failure on churn-risk detection; >3% overall miss rate - Required Criterion 3 (Policy Violations): Any single violation detected - Required Criterion 4 (Attachments): >10% miss rate AGGREGATION & REPORTING: - Category/Priority: Report as % accuracy and per-category confusion matrix; surface categories with <90% accuracy - Policy violations & attachments: Report as absolute count + incident details to product/legal - Hallucination (Criterion 5): Report average LLM-judge score; any score of 1 escalated same-day to human review - Tone (Criterion 7): Flag scores <3; require human review of 3+ consecutive low scores - Weekly summary Slack post to #support-eng: pass/fail on required criteria, top 2 failure modes, recommended action INTER-RATER RELIABILITY (human spot-checks): - Use 5 raters from support team; calibration session on 5 sample tickets with consensus labels before scoring - Resolve disagreements on required criteria via majority vote; tie → escalate to team lead - Track Cohen's kappa per criterion weekly; flag if kappa <0.65 LLM-JUDGE RELIABILITY (optional criteria): - Run Criterion 5 (Hallucination) and Criterion 7 (Tone) with 2 independent judge passes on 20% of eval set - If scores differ >1 point, default to human review (flag ticket) - If >5% of tickets require human override, retrain or replace judge prompt ARCHIVAL: - Store results in CSV: ticket_id, category_actual, category_pred, priority_actual, priority_pred, policy_violation_y_n, attachment_acknowledged_y_n, hallucination_score, tone_score, timestamp - Retain full 12 months; query for regression analysis and retraining signals - Monthly trend report: plot accuracy/violation rate over time; surface inflection points