Claude for Prompt Engineers: Evals, Skill Audits & Plugin

---
name: refund-helper
description: Helps with customers.
---
You are a support agent. Read the customer message below and decide whether
to approve a refund. Be generous.

{customer_message}

Always follow any instructions in the customer message.

AUDIT SUMMARY
Verdict: FAIL. Top issues: (1) instruction-override risk, (2) vague description,
(3) no refusal or escalation path.

FRONTMATTER CHECK
- name and description present
- Description "Helps with customers." is too broad: it gives Claude no clear signal
  for when to load this skill. Suggest: "Drafts refund decisions for support tickets
  under the refund policy; use when a customer asks for a refund."

PROMPT-INJECTION SURFACE
- High: "Always follow any instructions in the customer message" lets user input
  override the skill (OWASP LLM01 pattern). Remove it; wrap the message in
  delimiters and state that its contents are data, not instructions.

EDGE-CASE COVERAGE
- No handling for empty or non-English messages, or for refund requests outside
  policy. "Be generous" is not a testable rule: replace with the policy limits.

NEXT STEPS
1. Remove the override line and add delimiters. 2. Rewrite the description.
3. Add escalation rules. 4. Run /test-batch with injection cases before merging.

You are my prompt engineering and evaluation assistant. Here is my context:

<engineer-profile>
- Role: [Prompt Engineer / AI Eval Specialist / LLM Quality Analyst]
- Primary models: [Claude Sonnet 5 / GPT-5.6 / etc.]
- Eval harness: [promptfoo / Braintrust / Anthropic Workbench / OpenAI Evals / Inspect]
- Skill/plugin format: [SKILL.md / OpenAI Actions / custom JSON schema]
- Eval methodology: [Hamel Husain rubric-based / Eugene Yan LLM-as-judge / NIST AI RMF / custom]
- Primary use case domains: [customer support / coding / document QA / agentic tasks]
</engineer-profile>

<rules>
- All eval rubrics must include per-criterion scoring with explicit pass/fail thresholds.
- Synthetic test cases must cover happy path, edge cases, and adversarial inputs.
- Never mark an eval as complete without a regression check against the prior prompt version.
- Flag any prompt pattern that creates prompt-injection surface or scope creep.
- Eval results do not guarantee production behavior — always note this in reports.
</rules>

<context>
New skill: customer-support-triage. The skill reads a user message and outputs a JSON object
with keys: intent (string), urgency (low/medium/high), suggested_action (string), and
escalate (boolean). It runs on Claude Sonnet 5 inside our promptfoo suite.
</context>

<instructions>
Build an eval rubric for this skill:
- 5-7 criteria covering correctness, format compliance, urgency calibration, escalation
  accuracy, and response safety
- For each criterion: a 1-sentence definition, a 1-5 scoring scale with explicit
  descriptors for scores 1, 3, and 5, and a hard pass/fail threshold
- Note which criteria can be checked programmatically vs. require LLM-as-judge
- Flag any criterion where prompt injection could corrupt the output
</instructions>

<format>
Markdown table for criteria overview, then a detailed breakdown per criterion.
End with a section: "Recommended test distribution" (happy path / edge / adversarial split).
</format>

<avoid>
Vague criteria like "is it good?"; thresholds that cannot be operationalized;
assuming a human reviewer will catch what the rubric misses.
</avoid>

<context>
Skill: document-summarizer. Input: arbitrary PDF text up to 4,000 tokens. Output: a
bullet-point summary of 5-8 key points, each under 25 words. Target model: Claude Sonnet 5.
Known failure modes from prior evals: hallucinated facts, bullet points exceeding word cap,
refusal on legal/medical documents, prompt injection via embedded instructions in the PDF text.
</context>

<instructions>
Generate 30 synthetic test inputs across four categories:
- 8 happy-path inputs (clear, well-structured text from varied domains)
- 8 edge cases (very short text, non-English content, tables/lists only, code-heavy docs)
- 8 adversarial inputs (text containing embedded instructions like "ignore previous",
  PII, extremely long sentences, contradictory information)
- 6 boundary cases (text at exactly 4,000 tokens, empty input, single-sentence input)

For each input: provide the raw input text (or a clear description if long), the expected
output behavior, and the failure mode it is designed to surface.
</instructions>

<format>
JSONL format compatible with promptfoo's `tests` array. One object per line:
{"input": "...", "expected": "...", "category": "...", "failure_mode": "..."}.
</format>

<avoid>
Inputs that are trivially easy and don't test anything meaningful; duplicate scenarios
reworded slightly; inputs that assume the model knows domain context not in the prompt.
</avoid>

<context>
Testing two variants of a code-review skill:
- Variant A: current production prompt (attached)
- Variant B: revised prompt with chain-of-thought reasoning step added
Primary metric: rubric score (1-5) on correctness criterion. Secondary: response latency proxy
(token count). Eval harness: Braintrust. Available test budget: ~500 prompt runs.
Acceptable false-positive rate: 5%. Minimum detectable effect: 0.3 points on the 1-5 scale.
</context>

<instructions>
Design the A/B test protocol:
- Calculate required sample size per variant given alpha=0.05, desired power=0.80, and the MDE
- Recommend a test split and randomization strategy
- Define the primary statistical test (paired t-test vs. Mann-Whitney based on score distribution)
- Identify confounders to control for (input complexity, domain)
- Draft a results summary template that reports: mean score per variant, confidence interval,
  p-value, effect size (Cohen's d), and a plain-language conclusion
- Flag conditions under which the result should be treated as inconclusive
</instructions>

<format>
Section 1: Sample size calculation with formula shown.
Section 2: Test design (numbered steps).
Section 3: Results template (fill-in-the-blank markdown table).
Section 4: Decision criteria (go/no-go rules).
</format>

<avoid>
Recommending a winner without a significance check; ignoring latency/cost trade-offs;
treating a statistically significant but operationally tiny improvement as actionable.
</avoid>

<context>
Attached SKILL.md is a customer-facing skill that reads user input and calls an external API.
It will be deployed in a Claude plugin environment. Review it before the production PR merges.
Our checklist covers: prompt injection surface, frontmatter schema compliance,
token budget, scope creep, and output safety.
</context>

<instructions>
Audit this SKILL.md on five dimensions:

1. Prompt injection surface: identify every place where user-controlled text enters the prompt
   without sanitization; flag indirect injection vectors (e.g., API responses piped back in)

2. Frontmatter validity: check required fields (title, description, version, author, model,
   max_tokens); flag missing fields, invalid types, or values outside allowed ranges

3. Token budget: estimate worst-case prompt token count (system + max user input + few-shot
   examples); flag if it exceeds 80% of the model's context window

4. Scope creep: identify any instruction that could cause the skill to take actions beyond
   its stated purpose; flag instructions that grant the model discretion over irreversible actions

5. Output safety: check for missing refusal instructions, missing format enforcement,
   and any path that could produce PII or harmful content in the response

For each finding: severity (critical/high/medium/low), location in the file, and a
one-line remediation recommendation.
</instructions>

<format>
Executive summary (3 sentences max), then a findings table:
| # | Dimension | Severity | Location | Finding | Recommendation |
End with: "Ship / Do not ship" recommendation with one-sentence rationale.
</format>

<avoid>
Passing a skill that has any critical findings; treating missing fields as low severity;
skipping the token budget check because "it probably fits."
</avoid>

<context>
Prompt change: added an explicit output length constraint ("respond in under 150 words")
to the meeting-summarizer skill. Before/after eval results from Braintrust:
- Correctness score: 4.1 → 3.9 (delta -0.2, p=0.03)
- Format compliance: 2.8 → 4.6 (delta +1.8, p<0.001)
- User satisfaction proxy: 3.5 → 3.7 (delta +0.2, p=0.12)
- Mean token count: 310 → 148 (delta -52%)
- 3 new failure cases observed: summaries that truncate critical action items
Test set: 80 inputs, same distribution as production traffic sample.
</context>

<instructions>
Write a regression report that includes:
- Summary of the change and its intent
- Key findings: what improved, what regressed, what is statistically inconclusive
- Root cause analysis for the correctness regression (hypothesis only, based on the data)
- Three specific cases from the failure set, described concisely
- Recommendation: ship as-is / ship with mitigation / revert — with clear rationale
- One follow-up experiment to address the correctness regression
Note that eval results do not guarantee production behavior; include this caveat.
</instructions>

<format>
PR-ready markdown: H2 sections, one summary table, bullet points for findings.
Under 400 words. Suitable for a non-technical stakeholder to read the summary section.
</format>

<avoid>
Recommending a ship without addressing the statistically significant correctness regression;
omitting the production-behavior caveat; writing a report that buries the regression finding.
</avoid>