Profa
19 KiB
Skill Evaluation Research Report
Date: 2026-02-18 Status: Complete Sources: 10+ industry sources analyzed
Executive Summary
We analyzed how leading platforms evaluate AI agent skills and compared them against our AQE skill evaluation infrastructure. Our system is already more sophisticated than most in areas like trust tiers, multi-model testing, and learning integration. However, we have clear gaps in skill quality scoring (LLM-as-a-judge for how well skills are written), activation conflict detection, adaptive rubric-based grading, and eval-driven development workflow. This report synthesizes findings from Tessl, Anthropic's official guidance, OpenAI's eval methodology, Google Vertex AI's rubric systems, DeepEval, and Anthropic's engineering blog.
1. Tessl Framework Analysis
Source: docs.tessl.io/evaluate/evaluating-skills, docs.tessl.io/evaluate/evaluate-skill-quality-using-scenarios
Two-Layer Evaluation System
Layer 1: tessl skill review — Static Quality Assessment (LLM-as-a-Judge)
- Validation Checks: Deterministic grading on structure (frontmatter, line count, licensing, metadata) — ~12-15 structural criteria
- Implementation Score: LLM judges conciseness, actionability, workflow clarity, progressive disclosure — 1-4 point subscores per metric
- Activation Score: LLM judges description specificity, trigger term quality, distinctiveness/conflict risk
- Scoring benchmarks: 90%+ = best practice, 70-89% = good, <70% = needs revision
Layer 2: tessl eval run — Scenario-Based End-to-End Evaluation
- Each scenario has 3 files:
capability.text— which skill capability is targetedTASK.md— a real task for the agent to solvecriteria.json— grading rubric
- Scenarios can be auto-generated from skill instructions, then manually reviewed
- Agent actually executes the task, and output is graded against criteria
- CLI:
tessl eval run,tessl eval view,tessl eval list,tessl eval retry
Key Tessl Insight
Tessl separates skill quality (how well is the SKILL.md written?) from skill effectiveness (does the agent produce correct output?). We only do the latter.
2. Anthropic's Official Skill Authoring Guidance
Source: Skill authoring best practices
Evaluation-Driven Development (Most Important Takeaway)
Anthropic recommends building evals BEFORE writing extensive skill documentation:
- Identify gaps: Run Claude on representative tasks without a skill. Document failures.
- Create evaluations: Build 3+ scenarios that test those gaps
- Establish baseline: Measure Claude's performance without the skill
- Write minimal instructions: Just enough to pass evaluations
- Iterate: Execute evaluations, compare against baseline, refine
"This approach ensures you're solving actual problems rather than anticipating requirements that may never materialize."
Skill Quality Checklist (from Anthropic)
Core Quality:
- Description is specific, includes trigger terms, written in third person
- SKILL.md body under 500 lines
- Progressive disclosure with one-level-deep references
- No time-sensitive information
- Consistent terminology throughout
Key Principles:
- Conciseness is critical — context window is a shared resource. Challenge every paragraph: "Does Claude really need this?"
- Degrees of freedom — match specificity to task fragility (high freedom for flexible tasks, low freedom for fragile operations)
- Test with ALL models — what works for Opus may need more detail for Haiku
Naming Conventions:
- Use gerund form:
processing-pdfs,analyzing-spreadsheets(notpdf-helper,utils) - Lowercase, hyphens only, max 64 characters
Iterative Development with Claude A/B Pattern
- Claude A = skill author (helps design/refine the skill)
- Claude B = skill user (tests the skill on real tasks in a fresh context)
- Observe Claude B's behavior → bring findings back to Claude A → refine → repeat
Relevance to AQE
Many of our 97 skills likely violate these principles. An automated quality scorer could flag:
- Skills over 500 lines
- Inconsistent terminology
- Vague descriptions
- Time-sensitive content
- Deeply nested references
3. Anthropic's Agent Eval Engineering Guide
Source: Demystifying evals for AI agents
Three Grader Types
| Type | Speed | Cost | Best For |
|---|---|---|---|
| Code-based | Fast | Free | Exact match, regex, schema validation, outcome verification |
| Model-based | Medium | $$ | Subjective quality, reasoning assessment, rubric grading |
| Human | Slow | |
Gold-standard calibration, complex judgment |
Key Metrics
- pass@k: Probability of ≥1 success across k attempts (when one working solution suffices)
- pass^k: Probability ALL k trials succeed (critical for consistency-required agents)
Critical Best Practices
- Start with 20-50 tasks from REAL user failures — outperforms hundreds of synthetic ones
- Grade outputs, not paths — agents often discover valid approaches evaluators didn't anticipate
- Build in partial credit — recognize continua of success, not just binary pass/fail
- Read transcripts regularly — distinguish genuine failures from grading bugs
- Monitor saturation — when evals hit 100%, graduate them to regression suites and create harder ones
Relevance to AQE
Our must_contain keyword checks are code-based graders only. We're missing:
- Model-based grading for subjective quality dimensions
- Partial credit scoring (currently binary pass/fail per keyword)
- pass@k / pass^k metrics for measuring consistency across trials
- Saturation monitoring for knowing when to create harder evals
4. OpenAI's Eval Skills Methodology
Source: Testing Agent Skills Systematically with Evals
Four Success Dimensions
Every skill eval should measure across these dimensions:
- Outcome goals: Did the task complete correctly?
- Process goals: Did the agent follow intended steps?
- Style goals: Does output adhere to conventions?
- Efficiency goals: Were unnecessary operations minimized?
Evaluation Dataset Structure
Start with 10-20 test cases covering:
- Explicit invocation: Direct skill name reference
- Implicit invocation: Skill triggers via description match
- Contextual invocation: Real-world scenarios
- Negative controls: Cases where skill should NOT activate
Advanced Checks to Layer In
Beyond basic output validation:
- Command counting: Detect looping or inefficient workflows
- Token tracking: Monitor input/output usage for prompt bloat
- Build verification: Run actual builds as end-to-end validation
- Repository cleanliness: Ensure no unwanted files generated
- Permission validation: Verify least-privilege constraints
Key Insight
"The fundamental shift is from subjective assessment ('this feels better') to measurable validation."
Relevance to AQE
We're missing:
- Negative control test cases — verifying skills DON'T activate incorrectly
- Efficiency metrics — token usage and command counting per skill invocation
- Process goals — did the agent follow the right steps, not just get the right answer?
5. Google Vertex AI: Adaptive vs Static Rubrics
Source: Vertex AI Rubric-Based Metrics
Two Rubric Approaches
| Approach | Description | Best For |
|---|---|---|
| Adaptive Rubrics | Auto-generate unique pass/fail checks per prompt | Varied prompts, no reference answers |
| Static Rubrics | Fixed scoring guidelines across all prompts | Consistent benchmarks, standardized metrics |
Adaptive Rubrics (Recommended Default)
For each input, the system dynamically generates prompt-specific checks:
- "Is the response under 100 words?"
- "Does it mention the Sept 15 deadline?"
- "Does it return valid JSON with keys
risk_levelandnext_steps?"
These catch highly specific contracts that static rubrics miss.
Relevance to AQE
Our current eval system uses entirely static rubrics (must_contain patterns). Adopting adaptive rubrics would:
- Auto-generate context-specific validation per test case
- Catch skill-specific output requirements we currently miss
- Reduce manual effort writing
must_containlists
6. DeepEval Agent Metrics
Source: DeepEval AI Agent Evaluation, Tool Correctness Metric
Agent-Specific Metrics
| Metric | What It Measures |
|---|---|
| Tool Correctness | Did the agent select the right tools and call them with correct arguments? |
| Task Completion | Did the agent accomplish what the user asked? |
| Reasoning Quality | Was the agent's decision-making process sound? |
Key Insight
"AI agents fail in fundamentally different ways than simple LLM applications—an agent might select the right tool but pass wrong arguments, create a brilliant plan but fail to follow it, or complete the task but waste resources on redundant steps."
Relevance to AQE
For our MCP-integrated skills, we should evaluate:
- Did the skill invoke the correct MCP tools?
- Were tool arguments correct?
- Was the overall task completed?
- Were any redundant tool calls made?
7. Anthropic's Empirical Evaluation Guide
Source: Create strong empirical evaluations
Grading Method Hierarchy
Choose the fastest, most reliable, most scalable method:
- Code-based: Exact match, string match — fastest, most reliable
- LLM-based: Rubric-driven assessment — fast, flexible, requires calibration
- Human: Most flexible — slow, expensive, avoid if possible
Eval Methods by Quality Dimension
| Dimension | Method | Example |
|---|---|---|
| Task fidelity | Exact match | Sentiment == "positive" |
| Consistency | Cosine similarity | Paraphrased inputs → similar outputs |
| Relevance | ROUGE-L | Summary captures key information |
| Tone/Style | LLM Likert scale | Rate empathy 1-5 |
| Safety | LLM binary classification | Contains PHI? yes/no |
| Context use | LLM ordinal scale | References conversation history? 1-5 |
Tips for LLM-Based Grading
- Detailed, clear rubrics with specific criteria and examples
- Empirical output — "Output only 'correct' or 'incorrect'" or 1-5 scale
- Encourage reasoning — ask LLM to think first, then score (chain-of-thought improves grading accuracy)
- Use a different model to grade than the model that generated the output
Relevance to AQE
We should adopt:
- Cosine similarity for consistency testing across similar inputs
- LLM Likert scales for subjective quality dimensions
- Chain-of-thought grading — have the judge reason before scoring
- Cross-model grading — don't use the same model to both generate and evaluate
8. Comparison: AQE vs Industry
| Capability | AQE | Tessl | Anthropic | OpenAI | Vertex AI | DeepEval |
|---|---|---|---|---|---|---|
| Trust tiers | T0-T4 | None | None | None | None | None |
| Multi-model testing | Built-in | None | Recommended | None | None | None |
| Learning integration | MCP + ReasoningBank | None | None | None | None | None |
| Schema validation | 4-layer | Basic | None | None | None | None |
| Skill quality scoring | Missing | LLM judge | Checklist | None | None | None |
| Adaptive rubrics | Missing | None | None | None | Built-in | None |
| Negative controls | Missing | None | Recommended | Required | None | None |
| Tool correctness | Missing | None | None | None | None | Built-in |
| Partial credit | Missing | None | Recommended | None | None | None |
| pass@k / pass^k | Missing | None | Built-in | None | None | None |
| Eval-driven development | Missing | None | Core workflow | Core workflow | None | None |
| Activation conflict detection | Missing | Built-in | None | None | None | None |
| Auto-generated evals | Missing | Built-in | None | None | Adaptive | None |
| Efficiency metrics | Missing | None | None | Command/token counting | None | None |
| Cosine similarity grading | Missing | None | Documented | None | None | None |
Legend: Bold = strong implementation
9. Prioritized Recommendations
P0 — High Impact, Addresses Core Gaps
1. Eval-Driven Development Workflow
Why: Anthropic and OpenAI both make this their #1 recommendation. Write evals before skill content.
Implementation: Add aqe skill eval-init <skill-name> that creates baseline eval from task failures, then aqe skill eval-compare to measure improvement.
Effort: Medium
2. LLM-as-a-Judge Skill Quality Scorer
Why: Tessl has this, Anthropic recommends it. We have 39 Tier-0 skills with no quality signal. Implementation: Score each SKILL.md on: conciseness (under 500 lines), actionability, description specificity, trigger term quality, terminology consistency, progressive disclosure usage. Output a 0-100 score with specific improvement suggestions. Effort: Medium
3. Activation Conflict Detector
Why: With 97 skills, overlapping descriptions cause mis-routing. Tessl scores this as "distinctiveness." Implementation: Embed all skill descriptions → compute pairwise cosine similarity → flag pairs above 0.7 threshold → suggest description refinements. Effort: Low (we already have HNSW/vector infrastructure)
P1 — Medium Impact, Deepens Eval Quality
4. Adaptive Rubric Grading
Why: Our must_contain keyword matching misses context-specific requirements. Vertex AI's adaptive rubrics auto-generate checks.
Implementation: For each eval test case, have an LLM generate prompt-specific pass/fail checks, then grade the output against those checks.
Effort: Medium
5. Multi-Dimensional Grading (Beyond Keywords)
Why: Anthropic documents 6+ grading methods (exact match, cosine similarity, ROUGE-L, Likert, binary classification, ordinal). We only use keyword match.
Implementation: Add grading modes to eval YAML: grading_method: [keyword_match | cosine_similarity | llm_likert | llm_rubric | schema_check]
Effort: Medium-High
6. Negative Control Test Cases
Why: OpenAI requires testing when skills should NOT activate. We don't test this.
Implementation: Add negative_test_cases section to eval YAML — inputs where the skill should decline or redirect.
Effort: Low
P2 — Refinements and Efficiency
7. pass@k / pass^k Consistency Metrics
Why: Anthropic's engineering guide highlights these as critical for measuring agent reliability across multiple trials. Implementation: Run each eval test case k times, compute both metrics. Surface pass^k for skills used in CI/production. Effort: Low
8. Efficiency Metrics (Token/Command Tracking)
Why: OpenAI recommends tracking token usage and command counts per skill invocation to detect bloat.
Implementation: Add max_tokens, max_tool_calls fields to eval success criteria.
Effort: Low
9. Auto-Generate Eval YAML from SKILL.md
Why: 39 skills at Tier 0 need evals. Tessl auto-generates scenarios from skill instructions. Implementation: LLM reads SKILL.md → generates eval YAML with test cases, expected outputs, and success criteria → human reviews and refines. Effort: Medium
10. Tool Correctness Metrics (for MCP Skills)
Why: DeepEval shows agent-specific metrics are different. Our MCP-integrated skills should validate tool selection and arguments. Implementation: For MCP-calling skills, verify: correct tools selected, correct arguments passed, no redundant calls. Effort: Medium
10. What We Already Do Better Than Everyone
These are genuine competitive advantages we should maintain and publicize:
- Trust Tier System (T0-T4) — No other platform has a graduated trust system with clear upgrade paths
- Multi-Model Consistency Testing — Built-in variance tracking across Sonnet/Haiku is unique
- MCP + ReasoningBank Learning — Eval results feed into shared learning, no competitor does this
- Structured YAML Eval Format — More organized than Tessl's 3-file-per-scenario approach
- 4-Layer Validation Architecture — Schema → Validator → Eval Suite → CI is more rigorous than industry standard
- Fleet-Wide Pattern Sharing — Successful patterns propagate to other agents automatically
Sources
- Tessl: Evaluating Skills
- Tessl: Evaluate Skill Quality Using Scenarios
- Anthropic: Demystifying Evals for AI Agents
- Anthropic: Skill Authoring Best Practices
- Anthropic: Create Strong Empirical Evaluations
- OpenAI: Testing Agent Skills Systematically with Evals
- Google Vertex AI: Rubric-Based Metrics
- DeepEval: AI Agent Evaluation
- DeepEval: Tool Correctness Metric
- LLM Evaluation: 2026 Edition (Medium)
- IBM Research: Evaluating LLM-based Agents
- orq.ai: Agent Evaluation in 2025
- Rubric-Based Evaluation for Agentic Systems