mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Abhitej John
78054c1161
* Make Vally the sole LLM eval engine, retiring skill-validator evaluate Collapses the parallel skill-validator + Vally eval pipeline into a single Vally-only path while preserving every capability skill-validator provided: PR gate, PR comment, dotnet/skills-data historical push, and the dashboard. The skill-validator `check` linter is retained (skill-check.yml) pending microsoft/vally #463. - evaluation.yml: delete build-validator and the skill-validator `evaluate` job; single `evaluate:` job now uses vally-evaluation.yml. Forward the COPILOT_PAT_0..9 pool onto it (from #911) so the reusable workflow's token gate is satisfied. Rewire comment-on-pr / report-status / publish-* / deploy-dashboard onto `evaluate` and vally-results-* artifacts. - vally-evaluation.yml: pin @microsoft/vally-cli@0.9.0; keep the #911 prepare guard; write the generated per-plugin experiment to $GITHUB_WORKSPACE so relative eval globs resolve. - adapt.mjs: reconcile #887's runCompareWithRetry + conclusive/unmatched hardening with the consolidation helpers (nonActivation, compareByStim, roleToDashboard, etc.). Add consolidate.mjs and gen-experiment.mjs. - build-replay-sessions.ps1: read Vally executor-session-logs/events.jsonl instead of skill-validator session output. - Rename all tests/*/*/eval.vally.yaml to eval.yaml (Vally format is now the only eval format); author Vally evals for the four skills that lacked one. - CONTRIBUTING.md + InvestigatingResults: point contributors at the Vally harness; drop non-public links. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix setup-local-sdk eval for Vally schema The hand-authored setup-local-sdk eval used skill-validator grader/environment schema that Vally rejects at validation (no stimuli executed): - file-contains/file-not-contains graders require 'value:' not 'substring:' (output-contains correctly keeps 'substring:'). - environment.files entries require 'src:'/'dest:' (copy a fixture) rather than inline 'path:'/'content:'. Moved the global.json body into fixtures/global.json. Verified with 'vally lint --eval-spec' (0 errors) across all 97 eval.yaml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix markdownlint violations in migration docs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback on eval migration - Consolidate run-vally-evals.sh into a single neutral entrypoint eng/run-skill-evals.sh (deletes the thin wrapper; adapt.mjs stays in eng/vally-adapter/ since CI invokes it directly) - Rename reusable workflow vally-evaluation.yml -> evaluation-run.yml and update all references/self-trigger regexes - Rename local output dir vally-results -> eval-results - Bake prerequisite preflight (Node 20+, GITHUB_TOKEN/gh) into the runner - Simplify CONTRIBUTING "Running tests locally" prerequisites - Scrub harness name from consumer-facing surfaces Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reinstate overfitting detection under the Vally harness Overfitting detection previously ran inside skill-validator's `evaluate` command, which the Vally migration replaced, so it silently stopped running. Reinstate it as a standalone step wired into the Vally pipeline: - Add `skill-validator overfitting` command that discovers skills, parses each eval.yaml, runs the existing OverfittingJudge, and emits [{plugin, skill, overfittingResult}] JSON (per-skill failures non-fatal, bounded parallelism). - Add a Vally-format parser bridge (ParseEvalConfigFlexible) so the judge handles the current stimuli/graders eval.yaml format; the legacy scenarios-only parser rejected all 98 evals, so the judge never ran. - adapt.mjs: merge overfitting results onto each verdict via --overfitting <file> (keyed by plugin/skill); output is byte-identical when the flag is absent. - evaluation-run.yml: build skill-validator (shared cache with check), run the judge per leg reading GITHUB_TOKEN, and pass --overfitting to adapt.mjs so the dashboard/skills-data consume it unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reconcile #919 Vally eval configs into the sole eval.yaml convention PR #919 landed on main adding fresh Vally specs for three coverage-gap skills (setup-local-sdk, convert-blazor-server-to-webapp, dotnet-webapi) under the interim eval.vally.yaml filename. This branch collapses to a single Vally spec named eval.yaml, so the merge left each skill with both my migrated eval.yaml and #919's newer eval.vally.yaml (a git-clean but semantic duplicate the pipeline would not discover). Adopt #919's reviewed Vally content as the canonical eval.yaml for all three skills and drop the redundant eval.vally.yaml files, preserving the 'sole eval.yaml, zero eval.vally.yaml' invariant. Content is byte-identical to #919's blobs (b14121c6/47d43823/18a9362f), which already passed vally-evaluate on main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
104 lines
5.0 KiB
YAML
104 lines
5.0 KiB
YAML
name: coverage-analysis
|
|
description: Evaluates the dotnet-test/coverage-analysis skill
|
|
type: capability
|
|
config:
|
|
timeout: 6m
|
|
stimuli:
|
|
- name: Project-wide coverage analysis with existing Cobertura data
|
|
prompt: Run coverage analysis on this project. Coverage data is already in TestResults/coverage.cobertura.xml.
|
|
environment:
|
|
files:
|
|
- src: fixtures/shared/ContosoUniversity.sln
|
|
dest: ContosoUniversity.sln
|
|
- src: fixtures/shared/ContosoUniversity/ContosoUniversity.csproj
|
|
dest: ContosoUniversity/ContosoUniversity.csproj
|
|
- src: fixtures/shared/ContosoUniversity/StudentService.cs
|
|
dest: ContosoUniversity/StudentService.cs
|
|
- src: fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj
|
|
dest: ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj
|
|
- src: fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs
|
|
dest: ContosoUniversity.UnitTests/StudentServiceTests.cs
|
|
- src: fixtures/partial-coverage/coverage.cobertura.xml
|
|
dest: TestResults/coverage.cobertura.xml
|
|
graders:
|
|
- type: output-matches
|
|
config:
|
|
pattern: CRAP|crap
|
|
- type: output-matches
|
|
config:
|
|
pattern: CalculateGpa|Calculate_?Gpa
|
|
- type: output-matches
|
|
config:
|
|
pattern: hotspot|risk|coverage
|
|
- type: prompt
|
|
rubric:
|
|
- Discovers the solution and identifies the test project
|
|
- Parses the existing Cobertura XML to extract per-method coverage data
|
|
- Calculates CRAP scores and produces a sorted risk hotspot table
|
|
- Identifies CalculateGpa as the highest-risk method (high complexity, low coverage)
|
|
- Provides actionable recommendations for the riskiest methods
|
|
- Generates or displays a coverage analysis report
|
|
- name: Run coverage from scratch without existing data
|
|
prompt: Analyze code coverage and risk hotspots for this project. I haven't run any coverage yet.
|
|
environment:
|
|
files:
|
|
- src: fixtures/shared/ContosoUniversity.sln
|
|
dest: ContosoUniversity.sln
|
|
- src: fixtures/shared/ContosoUniversity/ContosoUniversity.csproj
|
|
dest: ContosoUniversity/ContosoUniversity.csproj
|
|
- src: fixtures/shared/ContosoUniversity/StudentService.cs
|
|
dest: ContosoUniversity/StudentService.cs
|
|
- src: fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj
|
|
dest: ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj
|
|
- src: fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs
|
|
dest: ContosoUniversity.UnitTests/StudentServiceTests.cs
|
|
graders:
|
|
- type: output-matches
|
|
config:
|
|
pattern: dotnet test
|
|
- type: output-matches
|
|
config:
|
|
pattern: coverage
|
|
- type: output-matches
|
|
config:
|
|
pattern: CRAP|crap|hotspot|risk
|
|
- type: prompt
|
|
rubric:
|
|
- Detects the coverage provider from the test project .csproj (coverlet.collector)
|
|
- Runs dotnet test with coverage collection to generate Cobertura XML
|
|
- Extracts per-method coverage from the generated data
|
|
- Computes CRAP scores and identifies risk hotspots
|
|
- Identifies CalculateGpa as untested or highest risk due to complexity with no coverage
|
|
- Provides prioritized recommendations for where to add tests
|
|
- name: Coverage plateau diagnosis
|
|
prompt: My coverage is stuck at 75% and I can't get it higher. What's blocking me? Coverage data is in
|
|
TestResults/coverage.cobertura.xml.
|
|
environment:
|
|
files:
|
|
- src: fixtures/shared/ContosoUniversity.sln
|
|
dest: ContosoUniversity.sln
|
|
- src: fixtures/shared/ContosoUniversity/ContosoUniversity.csproj
|
|
dest: ContosoUniversity/ContosoUniversity.csproj
|
|
- src: fixtures/shared/ContosoUniversity/StudentService.cs
|
|
dest: ContosoUniversity/StudentService.cs
|
|
- src: fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj
|
|
dest: ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj
|
|
- src: fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs
|
|
dest: ContosoUniversity.UnitTests/StudentServiceTests.cs
|
|
- src: fixtures/plateau/coverage.cobertura.xml
|
|
dest: TestResults/coverage.cobertura.xml
|
|
graders:
|
|
- type: output-matches
|
|
config:
|
|
pattern: CalculateGpa|Calculate_?Gpa
|
|
- type: output-matches
|
|
config:
|
|
pattern: 0%|uncovered|untested|no coverage
|
|
- type: prompt
|
|
rubric:
|
|
- Identifies CalculateGpa as the method blocking coverage improvement (0% coverage, highest complexity)
|
|
- Explains that 3 of 4 methods are well-covered but CalculateGpa drags down the average
|
|
- Quantifies the impact — covering CalculateGpa would raise overall coverage significantly
|
|
- Uses CRAP scores to distinguish the risky gap (CalculateGpa) from trivial ones
|
|
- Provides specific test recommendations for the uncovered method
|