What the Eval Rubric Generator actually produces

• REQUIRED: Category Accuracy
• Measures: Does the assigned category (billing/technical/refund/general) match the labeled ground truth?
• Scoring: Binary (correct/incorrect)
• Examples:
• PASS: Email about invoice dispute → billing
• FAIL: Email about product defect → general (should be technical)
• Threshold for investigation: >5% error rate on any category

• REQUIRED: Priority Alignment with SLA Rules
• Measures: Does assigned priority (urgent/high/medium/low) follow your documented SLA rules (e.g., churn-risk language = urgent, refund requests = high)?
• Scoring: Binary (correct/incorrect)
• Examples:
• PASS: "Considering switching to competitor" + refund request → urgent
• FAIL: "Considering switching to competitor" → low priority
• Threshold for investigation: Any failure on churn-risk detection, or >3% misalignment on high-priority cases

• REQUIRED: No Policy-Violating Commitments
• Measures: Does the draft response promise/authorize actions outside policy scope (refunds not pre-approved, technical fixes requiring escalation, discounts beyond limits)?
• Scoring: Binary (violates policy/compliant)
• Examples:
• PASS: "I'll escalate your refund request to our billing team for review" (no commitment)
• FAIL: "We'll issue a full refund immediately" (violates policy if email shows ineligible reason)
• Threshold for investigation: Any single violation; stop the model at 1 violation per 50 emails

• REQUIRED: Attachment Context Acknowledged
• Measures: If the email explicitly mentions or includes an attachment (error log, screenshot, receipt), does the response reference or address it?
• Scoring: Binary (acknowledged/ignored)
• Examples:
• PASS: Email: "See attached error.log" → Response: "I've reviewed the error log you attached and..."
• FAIL: Email: "See attached error.log" → Response: generic troubleshooting with no mention of attachment
• Threshold for investigation: >10% miss rate on emails with explicit attachments

• OPTIONAL: Factual Hallucination Check
• Measures: Does the draft invent account details, order numbers, or dates not present in the email? (LLM-as-judge)
• Scoring: Scale 1-5: 1=multiple false details, 3=one possible misstatement, 5=no invented facts
• Judge prompt: "Does this response reference specific account details, orders, or dates that were not in the original email? Note what is invented."
• Judge reliability: Run 2 independent judge passes; if scores differ >1 point, escalate to human review
• Examples:
• PASS: Email vague on order details → Response: "Could you share your order number so I can look into this?"
• FAIL: Email doesn't mention order ID → Response: "I see order #12345 was placed on March 15..."
• Threshold for investigation: Any hallucinated fact >3; average score <4.0 across eval set

• OPTIONAL: Response Conciseness & Actionability
• Measures: Is the draft response under 200 words and does it clearly state the next step (what the agent or customer should do)?
• Scoring: Binary (clear next step present / absent)
• Examples:
• PASS: "I'm checking with our tech team now. You'll hear from me by end of day with next steps."
• FAIL: "Thank you for contacting us. We take all feedback seriously..." [ends with no action]
• Threshold for investigation: >20% missing next-step statements

• OPTIONAL: Tone Appropriateness to Category
• Measures: Does the tone match the category and priority? (e.g., urgent issues warrant empathy; general inquiries can be lighter) — LLM-as-judge
• Scoring: Scale 1-5: 1=dismissive/wrong register, 3=neutral but safe, 5=appropriately empathetic or professional
• Judge prompt: "Is the tone in this response appropriate for a [category] issue at [priority] level? Would a customer feel heard?"
• Judge reliability: Single judge pass; flag for human review only if score <3
• Examples:
• PASS: Urgent churn risk: "I understand your frustration. I'm personally escalating this..."
• FAIL: Urgent churn risk: "We appreciate your business. We'll review your case." (flat, formulaic)
• Threshold for investigation: >15% scores <3; any score of 1

• OPTIONAL: Constraint Adherence (Formatting & Length)
• Measures: Does the draft fit in a single email window (~150–250 words) and avoid legal disclaimers not in your template?
• Scoring: Binary (meets constraints / exceeds)
• Examples:
• PASS: 180-word response, no extra legal boilerplate
• FAIL: 450-word response with custom liability language
• Threshold for investigation: >10% constraint violations

• Multilingual emails (e.g., subject in English, body in Spanish): Criterion 1 (Category Accuracy) and Criterion 2 (Priority Alignment) — verify model doesn't default to 'general' and miss priority signals in non-English text.

• Churn-risk language with low urgency metadata (e.g., "considering switching" but low account tenure/spend): Criterion 2 (Priority Alignment) — ensure model weights explicit churn language even when customer lifetime value is low.

• Refund request with ambiguous eligibility (return window vs. non-refundable service): Criterion 3 (Policy-Violating Commitments) — model must draft escalation, not promise.

• Email with 2+ attachments, only 1 relevant (e.g., unrelated invoice + error screenshot): Criterion 4 (Attachment Context) — verify model addresses the *relevant* attachment, not all.

• Technical issue that requires customer password/sensitive data in response: Criterion 3 (Policy-Violating Commitments) — catch responses that ask for passwords in plain-text draft.

• Sarcasm or frustration expressed as technical complaint ("Your product is terrible, fix it"): Criterion 2 (Priority Alignment) and Criterion 7 (Tone Appropriateness) — ensure model detects urgency despite hostile tone and responds with empathy.

• Email from known high-value customer with generic complaint: Criterion 2 (Priority Alignment) — model should *not* boost priority based on account metadata not in email; priority is determined by issue type + language, not CRM data.

• Attachment filename suggests relevance but email text doesn't mention it (e.g., "error_critical.log" attached but email says "general question"): Criterion 4 (Attachment Context) — model should flag mismatch, not assume attachment intent.

CADENCE & TRIGGERS:

• Runs nightly on full labeled test set (~300 tickets) + any new labeled tickets added that week
• Manual eval run on-demand before any model release
• Weekly spot-check: 10% random sample of draft responses scored by human agent (5 raters, majority vote on optional criteria)

KILL CRITERIA (auto-halt deployment):

• Required Criterion 1 (Category Accuracy): <93% (regression of >2% from last passing run)
• Required Criterion 2 (Priority Alignment): Any failure on churn-risk detection; >3% overall miss rate
• Required Criterion 3 (Policy Violations): Any single violation detected
• Required Criterion 4 (Attachments): >10% miss rate

AGGREGATION & REPORTING:

• Category/Priority: Report as % accuracy and per-category confusion matrix; surface categories with <90% accuracy
• Policy violations & attachments: Report as absolute count + incident details to product/legal
• Hallucination (Criterion 5): Report average LLM-judge score; any score of 1 escalated same-day to human review
• Tone (Criterion 7): Flag scores <3; require human review of 3+ consecutive low scores
• Weekly summary Slack post to #support-eng: pass/fail on required criteria, top 2 failure modes, recommended action

INTER-RATER RELIABILITY (human spot-checks):

• Use 5 raters from support team; calibration session on 5 sample tickets with consensus labels before scoring
• Resolve disagreements on required criteria via majority vote; tie → escalate to team lead
• Track Cohen's kappa per criterion weekly; flag if kappa <0.65

LLM-JUDGE RELIABILITY (optional criteria):

• Run Criterion 5 (Hallucination) and Criterion 7 (Tone) with 2 independent judge passes on 20% of eval set
• If scores differ >1 point, default to human review (flag ticket)
• If >5% of tickets require human override, retrain or replace judge prompt

ARCHIVAL:

• Store results in CSV: ticket_id, category_actual, category_pred, priority_actual, priority_pred, policy_violation_y_n, attachment_acknowledged_y_n, hallucination_score, tone_score, timestamp
• Retain full 12 months; query for regression analysis and retraining signals
• Monthly trend report: plot accuracy/violation rate over time; surface inflection points