dotnet__skills/eng/skill-validator/src/README.md

skill-validator

You've built a bunch of skills. But are they actually helping or just adding noise?

skill-validator finds out. It runs your agent with and without each skill, measures what changed, and tells you whether the skill is worth keeping.

Plugging into your CI, it ensures every new skill adds real value, and existing skills that stop helping when a new model comes out can be removed.

How it works

1. Discovers skills (directories with SKILL.md)
2. Reads evaluation scenarios from each skill's tests/eval.yaml
3. For each scenario, runs the agent without the skill (baseline) and with the skill
4. Collects metrics: token usage, tool calls, time, errors, task completion
5. Uses pairwise comparative judging — the LLM judge sees both outputs side-by-side and decides which is better (with position-swap bias mitigation)
6. Computes confidence intervals via bootstrapping across multiple runs
7. Compares results and produces a verdict with statistical significance
8. Saves detailed results (JSON + markdown) to .skill-validator-results/

Prerequisites

• Authenticated with GitHub via gh auth login (the GitHub Copilot SDK picks up your credentials automatically)

Download from pipeline artifacts

A nightly scheduled job publishes AOT-compiled .tar.gz archives and NuGet packages as assets on the skill-validator-nightly GitHub Release.

Run without .NET installed

To execute skill-validator without .NET installed, download the .tar.gz archive matching your platform, extract it, and run the skill-validator binary.

Run via dnx

With .NET 10+, you can run skill-validator without permanently installing it using dnx (dotnet execute). Download the RID-agnostic .nupkg from the skill-validator-nightly release, then point dnx at it:

# Run check directly from the downloaded nupkg
dnx Microsoft.DotNet.SkillValidator --source ./path/to/downloaded/ check --plugin ./path/to/plugin/

# Run evaluate directly from the downloaded nupkg
dnx Microsoft.DotNet.SkillValidator --source ./path/to/downloaded/ evaluate --tests-dir ./tests/my-plugin ./path/to/skills/

Usage

The tool has several subcommands:

• evaluate — LLM-based evaluation testing (requires a Copilot token)
• check — Static analysis of skills, plugins, and agents (no LLM, no token required)
• consolidate — Merge results from matrix jobs into a single summary
• rejudge — Re-run judging on previously saved sessions

All examples below use the skill-validator binary directly. If running from source, replace skill-validator with dotnet run --project eng/skill-validator/src --:

LLM evaluation (evaluate)

# Show evaluate help
skill-validator evaluate --help

# Evaluate a skill (--tests-dir is required)
skill-validator evaluate --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills/my-skill

# Verbose output with per-scenario breakdowns
skill-validator evaluate --verbose --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills

# Custom model and threshold
skill-validator evaluate --model claude-sonnet-4.5 --min-improvement 0.2 --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills

# Use a different model for judging vs agent runs
skill-validator evaluate --model gpt-5.3-codex --judge-model claude-opus-4.6-fast --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills

# Multiple runs for stability
skill-validator evaluate --runs 5 --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills

# Compute a shared baseline once, then reuse it across multiple skills/agents
skill-validator evaluate --baseline-out baseline.json --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills/skill-a
skill-validator evaluate --baseline-from baseline.json --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills/skill-b

# Override the default results directory (.skill-validator-results)
skill-validator evaluate --results-dir ./my-results --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills

# File reporters can also be specified explicitly.
skill-validator evaluate --reporter junit --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills

# Verdict-warn-only mode (verdict failures return exit 0, execution errors still fail)
skill-validator evaluate --verdict-warn-only --tests-dir ./tests/my-plugin ./plugins/my-plugin/skills

Static analysis (check)

# Show check help
skill-validator check --help

# Check an entire plugin (recommended — validates skills, agents, plugin.json)
skill-validator check --plugin ./plugins/my-plugin

# Check multiple plugins
skill-validator check --plugin ./plugins/my-plugin --plugin ./plugins/other-plugin

# Check only skills
skill-validator check --skills ./plugins/my-plugin/skills

# Check only agents
skill-validator check --agents ./plugins/my-plugin/agents

# Check with external dependency allow list
skill-validator check --plugin ./plugins/my-plugin --allowed-external-deps ./eng/allowed-external-deps.txt

# Verbose output
skill-validator check --verbose --plugin ./plugins/my-plugin

# Emit machine-readable JSON to stdout
skill-validator check --json --plugin ./plugins/my-plugin

check flags

Flag	Default	Description
--plugin <paths...>		Plugin directories to check — discovers skills, agents, plugin.json (recommended)
--skills <paths...>		Skill directories to check (skills only)
--agents <paths...>		Agent directories to check (agents only)
--allowed-external-deps <path>	(none)	Path to allowed-external-deps.txt; when omitted the external-deps check is skipped
--known-domains <path>	(none)	Path to known-domains.txt for reference scanning; when omitted the reference scan is skipped
--verbose	false	Show detailed output
--json	false	Write a machine-readable JSON report to stdout

--plugin must be used alone. --skills and --agents can be combined.

evaluate flags

Flag	Default	Description
<paths...>	(required)	Paths to skill directories or parent directories
--tests-dir <path>	(required)	Directory containing test subdirectories
--model <name>	claude-opus-4.6	Model for agent runs
--judge-model <name>	same as --model	Model for LLM judge (can be different)
--judge-mode <mode>	pairwise	Judge mode: pairwise, independent, or both
--min-improvement <n>	0.1	Minimum improvement score (0–1)
--runs <n>	5	Runs per scenario (averaged for stability)
--parallel-skills <n>	3	Max concurrent skills to evaluate
--parallel-scenarios <n>	3	Max concurrent scenarios per skill
--parallel-runs <n>	3	Max concurrent runs per scenario
--confidence-level <n>	0.95	Confidence level for statistical intervals (0–1)
--judge-timeout <n>	300	Judge LLM timeout in seconds
--require-completion	true	Fail if skill regresses task completion
--baseline-out <path>	(none)	After running, persist each scenario's averaged baseline (no-skill/no-agent reference) to this file for reuse. Mutually exclusive with --baseline-from.
--baseline-from <path>	(none)	Reuse a precomputed baseline from this file instead of re-running the baseline arm. Must match --model, --judge-model, and every scenario's prompt, setup inputs, and evaluation criteria. Mutually exclusive with --baseline-out.
--verdict-warn-only	false	Treat verdict failures as warnings (exit 0). Execution errors still fail.
--no-overfitting-check	false	Disable the LLM-based overfitting analysis (on by default)
--overfitting-fix	false	Generate eval.fixed.yaml with improved rubric items/assertions
--verbose	false	Show tool calls and agent events during runs
--reporter <spec>	console, json, markdown	Output format: console, json, junit, markdown.
--results-dir <path>	.skill-validator-results	Directory for file reporter output.
--keep-sessions	false	Preserve agent session data (sessions.db) in the results directory for later rejudge. Requires --results-dir.
--no-judge	false	Run the requested agent arms and persist sessions.db, but skip all judging. No baseline file is required. Use this to run baseline and treatment arms in parallel, then judge later with rejudge. Requires --results-dir; mutually exclusive with --baseline-out/--baseline-from.

Models are validated on startup — invalid model names fail fast with a list of available models.

Shared baseline reuse

Every evaluation runs each scenario through a baseline arm (the agent with no skill / no agent loaded) to establish a reference the skill-enhanced run is compared against. When you evaluate many skills or agents against the same test scenarios, that baseline arm is re-run every time — redundant work that also introduces run-to-run variance into the comparison.

--baseline-out and --baseline-from let you compute the baseline once and reuse it as a shared control group:

1. Produce a baseline file with --baseline-out baseline.json. After the run, each scenario's averaged baseline result (honoring --runs) is written to the file.
2. Reuse it with --baseline-from baseline.json on subsequent runs. The baseline arm is skipped entirely; the cached baseline is used for assertions, pairwise/independent judging, and metric deltas.

The baseline file records the --model and --judge-model, and per scenario a SHA-256 of the prompt plus a composite SHA-256 over (a) its setup inputs — the fixtures copied via copy_test_files, explicit setup files, and setup commands — and (b) the evaluation criteria that shape the stored result (rubric, assertions, expect/reject tools, and the turn/token/timeout limits). This is the analog of a target/input SHA. On reuse the validator fails fast if the agent model, the judge model, or any scenario's prompt-plus-setup-plus-criteria identity is missing from the file, so a stale or mismatched baseline can never be silently applied — and two scenarios that share a prompt but feed the agent different fixtures (e.g. a different build.binlog) or use different rubrics never reuse each other's baseline. Scenarios reused from the file are reported with the baseline-reused session phase and a reused baseline status.

Note: Setup commands are fingerprinted by their text (the recipe), not the artifacts they produce, so baseline reuse assumes setup commands are deterministic/hermetic — a command whose output changes between runs (e.g. fetching latest) will not invalidate a cached baseline.

The two options are mutually exclusive.

Decoupled runs and judging (parallel run pool → central judge)

The expensive part of an evaluation is the agent investigation, not the judging. The baseline and treatment runs have no dependency on each other — only the judge step pairs them. You can therefore split a run into two phases so baseline and treatment arms execute in one parallel pool, with judging deferred to a final barrier:

1. Run every arm with skill-validator evaluate --no-judge --results-dir <dir>. This runs the requested agent arms, persists sessions.db (including each run's baseline key — the prompt-plus-target SHA used for pairing), and performs no judge calls. It does not require a baseline file and exits 0 on successful runs. Run the baseline directory and each treatment directory concurrently.

2. Judge with rejudge, pointing it at a treatment results directory and the separate baseline results directory:

   skill-validator evaluate rejudge <treatment-results-dir> --baseline-dir <baseline-results-dir>
   

   rejudge pairs each treatment scenario with its baseline by the shared key (prompt SHA + target SHA), runs the same pairwise/independent judges evaluate runs inline, writes the reports, and applies the usual pass/fail gates (--min-improvement, --require-completion, …). Baseline and treatment must share the same --model; the judge model defaults to the value persisted in the treatment sessions.db (then the baseline's), and a mismatch between the two persisted judge models is rejected unless you pass --judge-model.

Without --baseline-dir, rejudge keeps its original single-directory behavior: it re-judges baseline+treatment runs that live in the same sessions.db.

Output

Results are displayed in the console with color-coded scores and metric deltas. By default, json and markdown reporters are enabled and write to .skill-validator-results/ (override with --results-dir). File reporters write to that directory:

• json — results.json with model, timestamp, and all verdicts
• junit — results.xml with JUnit XML test results
• markdown — summary.md with a results table, plus per-skill directories with per-scenario judge reports

See Investigating Results for how to diagnose poor scores, download artifacts, and interpret results.json.

Consolidating results across matrix jobs

When evaluating multiple plugins in parallel CI matrix jobs, use the consolidate subcommand to merge individual results.json files into a single markdown summary:

skill-validator consolidate --output summary.md results1.json results2.json

# Or use find/glob to discover files
skill-validator consolidate --output summary.md $(find ./all-results/ -name results.json)

Flag	Description
<files...>	Paths to results.json files to merge
--output <path>	Output file path for the consolidated markdown

Writing eval files

Each skill can include a tests/eval.yaml:

Per-eval configuration

An optional top-level config section lets you override parallelism settings for a specific eval file. This is useful for resource-intensive evaluations (e.g., skills that spawn heavy child processes like ML.NET model training) where the default concurrency would exceed runner memory limits.

config:
  max_parallel_scenarios: 1   # cap concurrent scenarios (default: use CLI value)
  max_parallel_runs: 2        # cap concurrent runs per scenario (default: use CLI value)

scenarios:
  - name: "Heavy scenario"
    prompt: "..."

Setting	Type	Description
max_parallel_scenarios	int (optional)	Maximum concurrent scenarios for this eval. The effective value is min(--parallel-scenarios, this value).
max_parallel_runs	int (optional)	Maximum concurrent runs per scenario for this eval. The effective value is min(--parallel-runs, this value).

Both settings are optional. When omitted, the CLI defaults (--parallel-scenarios, --parallel-runs) apply unchanged. The override can only reduce parallelism (via Math.Min), never increase it beyond the CLI value.

Scenarios

scenarios:
  - name: "Descriptive name of the scenario"
    prompt: "The prompt to send to the agent"
    setup:
      copy_test_files: true  # auto-copy all sibling files into agent work dir
      files:
        - path: "input.txt"
          content: "file content to create before the run"
        - path: "data.csv"
          source: "fixtures/sample-data.csv"  # relative to skill dir
      additional_required_skills:       # skills needed by the target in isolated runs
        - binlog-failure-analysis
      additional_required_agents:        # agents needed by the target in isolated runs
        - build-perf
    assertions:
      - type: "output_contains"
        value: "expected text"
      - type: "output_not_contains"
        value: "text that should not appear"
      - type: "output_matches"
        pattern: "regex pattern"
      - type: "output_not_matches"
        pattern: "regex that should not match"
      - type: "file_exists"
        path: "*.csv"
      - type: "file_not_exists"
        path: "*.csproj"
      - type: "file_contains"
        path: "*.cs"
        value: "stackalloc"
      - type: "exit_success"
    expect_tools: ["bash"]
    reject_tools: ["create_file"]
    max_turns: 15
    rubric:
      - "The output is well-formatted and clear"
      - "The agent correctly handled edge cases"
    timeout: 120

Assertion types

Type	Description
output_contains	Agent output contains value (case-insensitive)
output_not_contains	Agent output does NOT contain value
output_matches	Agent output matches pattern (regex)
output_not_matches	Agent output does NOT match pattern
file_exists	File matching path glob exists in work dir
file_not_exists	No file matching path glob exists in work dir
file_contains	File matching path glob contains value
exit_success	Agent produced non-empty output

Setup options

Option	Description
copy_test_files	When true, copies all files from the eval directory (except eval.yaml) into the agent working directory before each run. Useful when test fixtures live alongside the eval file.
files	Explicit list of files to create in the working directory. Each entry has a path and either inline content or a source path (relative to the skill directory). Applied after copy_test_files.
commands	Shell commands to run in the work directory before the agent starts (e.g. dotnet build -bl:build.binlog).
additional_required_skills	List of skill names (from the same plugin) to load in the isolated run alongside the target. Useful when an agent routes to specific skills or a skill depends on sibling skills. Does not affect baseline (nothing loaded) or plugin (everything loaded) runs.
additional_required_agents	List of agent names (from the same plugin) to register in the isolated run alongside the target. Same semantics as additional_required_skills but for agents.

Test fixture files

If a scenario requires files in the agent's working directory (e.g. .csproj, .sln, .cs files), place them alongside eval.yaml and opt into auto-copy:

tests/<plugin>/<skill-name>/
  eval.yaml
  MyProject.csproj
  Program.cs

scenarios:
  - name: "Diagnose build failure"
    prompt: "Why does this project fail to build?"
    setup:
      copy_test_files: true    # copies MyProject.csproj, Program.cs into work dir
    assertions:
      - type: "output_matches"
        pattern: "CS\\d{4}"

You can also create files inline or reference files from the skill directory:

setup:
  files:
    - path: "input.txt"
      content: "inline file content"
    - path: "data.csv"
      source: "fixtures/sample-data.csv"  # relative to skill directory

Scenario constraints

Constraints are declarative checks against run metrics — no regex or globs needed:

scenarios:
  - name: "Test C# scripting"
    prompt: "Test stackalloc with nint"
    expect_tools: ["bash"]           # agent must use these tools
    reject_tools: ["create_file"]    # agent must NOT use these tools
    max_turns: 10                    # agent must finish within N turns
    max_tokens: 5000                 # agent must use fewer than N tokens

Constraint	Description
expect_tools	List of tool names the agent must use
reject_tools	List of tool names the agent must NOT use
max_turns	Maximum number of agent turns allowed
max_tokens	Maximum token usage allowed

Constraints are evaluated alongside assertions — a failed constraint means a failed task.

Rubric

Rubric items are evaluated by an LLM judge that sees both the baseline and skill-enhanced outputs side-by-side (pairwise mode). The judge determines which output is better per criterion and by how much. Position bias is mitigated by running the comparison twice with swapped order and checking consistency.

In independent mode, rubric items are scored 1–5 per run. Quality metrics have the highest weight (0.70 combined) in the improvement score.

Skill profile analysis

Before running the A/B evaluation, skill-validator performs static analysis of each SKILL.md and reports a one-line profile:

📊 crank-benchmarking: 1,722 BPE tokens [chars/4: 2,156] (detailed ✓), 29 sections, 24 code blocks
   ⚠  No numbered workflow steps detected

This is grounded in SkillsBench findings (84 tasks, 7,308 trajectories):

• "Detailed" and "compact" skills work best (+18.8pp and +17.1pp improvement)
• "Comprehensive" skills hurt performance (–2.9pp) — long documents create cognitive overhead
• Sweet spot is 800–2,500 tokens (ecosystem median: 1,569 tokens)
• 2–3 focused skills outperform 4+ skills bundled together

When a skill fails validation, the profile warnings appear in the diagnosis to suggest what to fix.

Getting the token count for a skill

The check sub-command reports the BPE token count for each skill when run with --verbose:

skill-validator check --verbose --plugin ./plugins/my-plugin

The profile line shows two token measurements:

Value	Description
BPE tokens	Actual token count computed with the cl100k_base tokenizer (OpenAI). This is the primary size metric.
chars/4	Approximate token count (character count ÷ 4). Shown as a reference point.

The complexity tier is derived from the BPE token count:

Tier	Token range	Verdict
compact	< 400	✓ Good
detailed	400 – 2,500	✓ Recommended tier (sweet spot: 800–2,500)
standard	2,501 – 5,000	Approaching diminishing returns
comprehensive	> 5,000	✗ Performance degrades

Note: The check command writes to stdout. Use the default console output for human-readable summaries, or --json for a machine-readable payload you can pipe to a file or another tool. Warnings about skill size are always printed in console mode; the full profile line requires --verbose.

Metrics & scoring

The improvement score is a weighted sum. Quality is heavily prioritized — a skill that improves output quality will pass even if it uses more tokens:

Metric	Weight	What it measures
Quality (rubric)	0.40	Pairwise rubric comparison (or independent judge rubric scores)
Quality (overall)	0.30	Pairwise overall comparison (or independent judge holistic assessment)
Task completion	0.15	Did hard assertions pass?
Token reduction	0.05	Fewer tokens = more efficient
Error reduction	0.05	Fewer errors/retries
Tool call reduction	0.025	Fewer tool calls = more efficient
Time reduction	0.025	Faster completion

All efficiency metrics are clamped to [-1, 1] so extreme changes can't overwhelm quality gains.

A skill passes if its average improvement score across scenarios meets the threshold (default 10%).

Pairwise judging

By default (--judge-mode pairwise), the LLM judge sees both baseline and skill-enhanced outputs in a single prompt and makes a direct comparison. This is more reliable than independent scoring because:

• LLMs are better at relative comparison than absolute scoring
• Eliminates calibration drift between separate judge calls
• Directly answers "is the skill-enhanced version better?"

Position-swap bias mitigation: Each comparison is run twice — once with baseline first, once with skill first. If the judge picks the same winner in both orderings, the result is trusted. If it flips, the comparison defaults to a tie (flagged as inconsistent).

Overfitting detection

An eval can be "overfitted" — rewarding the agent for parroting the skill's specific phrasing rather than producing a genuinely better result. The overfitting judge runs in parallel with scenario execution (one LLM call per skill) and classifies each rubric item and assertion:

• outcome / broad — tests a correct result; not overfitted
• technique / narrow — tests a skill-specific method; moderately overfitted
• vocabulary — tests the skill's exact wording; highly overfitted

The per-element classifications are combined into a 0–1 score (✅ low ≤ 0.20, 🟡 moderate ≤ 0.50, 🔴 high). The score is informational only — it does not affect the pass/fail verdict. It appears in console output, the markdown results table, and the dashboard.

Use --overfitting-fix to generate an eval.fixed.yaml with suggested outcome-focused replacements for flagged items. Disable the check entirely with --no-overfitting-check.

See OverfittingDetection.md for the full design.

Statistical confidence

Results include bootstrap confidence intervals computed across individual runs. The output shows:

✓ my-skill  +18.5%  [+8.2%, +28.8%] significant  (g=+24.3%)
✗ other-skill  +6.3%  [-2.1%, +14.7%] not significant  (g=+8.1%)

• significant: the 95% CI doesn't cross zero — the improvement is real
• not significant: the CI crosses zero — could be noise
• g=: normalized gain, controlling for ceiling effects (a skill improving a strong baseline is harder than improving a weak one)

The default of 5 runs provides sufficient precision for significance testing (validated by SkillsBench).