* Make Vally the sole LLM eval engine, retiring skill-validator evaluate
Collapses the parallel skill-validator + Vally eval pipeline into a single
Vally-only path while preserving every capability skill-validator provided:
PR gate, PR comment, dotnet/skills-data historical push, and the dashboard.
The skill-validator `check` linter is retained (skill-check.yml) pending
microsoft/vally #463.
- evaluation.yml: delete build-validator and the skill-validator `evaluate`
job; single `evaluate:` job now uses vally-evaluation.yml. Forward the
COPILOT_PAT_0..9 pool onto it (from #911) so the reusable workflow's token
gate is satisfied. Rewire comment-on-pr / report-status / publish-* /
deploy-dashboard onto `evaluate` and vally-results-* artifacts.
- vally-evaluation.yml: pin @microsoft/vally-cli@0.9.0; keep the #911 prepare
guard; write the generated per-plugin experiment to $GITHUB_WORKSPACE so
relative eval globs resolve.
- adapt.mjs: reconcile #887's runCompareWithRetry + conclusive/unmatched
hardening with the consolidation helpers (nonActivation, compareByStim,
roleToDashboard, etc.). Add consolidate.mjs and gen-experiment.mjs.
- build-replay-sessions.ps1: read Vally executor-session-logs/events.jsonl
instead of skill-validator session output.
- Rename all tests/*/*/eval.vally.yaml to eval.yaml (Vally format is now the
only eval format); author Vally evals for the four skills that lacked one.
- CONTRIBUTING.md + InvestigatingResults: point contributors at the Vally
harness; drop non-public links.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix setup-local-sdk eval for Vally schema
The hand-authored setup-local-sdk eval used skill-validator grader/environment
schema that Vally rejects at validation (no stimuli executed):
- file-contains/file-not-contains graders require 'value:' not 'substring:'
(output-contains correctly keeps 'substring:').
- environment.files entries require 'src:'/'dest:' (copy a fixture) rather than
inline 'path:'/'content:'. Moved the global.json body into fixtures/global.json.
Verified with 'vally lint --eval-spec' (0 errors) across all 97 eval.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix markdownlint violations in migration docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback on eval migration
- Consolidate run-vally-evals.sh into a single neutral entrypoint
eng/run-skill-evals.sh (deletes the thin wrapper; adapt.mjs stays in
eng/vally-adapter/ since CI invokes it directly)
- Rename reusable workflow vally-evaluation.yml -> evaluation-run.yml and
update all references/self-trigger regexes
- Rename local output dir vally-results -> eval-results
- Bake prerequisite preflight (Node 20+, GITHUB_TOKEN/gh) into the runner
- Simplify CONTRIBUTING "Running tests locally" prerequisites
- Scrub harness name from consumer-facing surfaces
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reinstate overfitting detection under the Vally harness
Overfitting detection previously ran inside skill-validator's `evaluate`
command, which the Vally migration replaced, so it silently stopped running.
Reinstate it as a standalone step wired into the Vally pipeline:
- Add `skill-validator overfitting` command that discovers skills, parses
each eval.yaml, runs the existing OverfittingJudge, and emits
[{plugin, skill, overfittingResult}] JSON (per-skill failures non-fatal,
bounded parallelism).
- Add a Vally-format parser bridge (ParseEvalConfigFlexible) so the judge
handles the current stimuli/graders eval.yaml format; the legacy
scenarios-only parser rejected all 98 evals, so the judge never ran.
- adapt.mjs: merge overfitting results onto each verdict via
--overfitting <file> (keyed by plugin/skill); output is byte-identical
when the flag is absent.
- evaluation-run.yml: build skill-validator (shared cache with check),
run the judge per leg reading GITHUB_TOKEN, and pass --overfitting to
adapt.mjs so the dashboard/skills-data consume it unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reconcile #919 Vally eval configs into the sole eval.yaml convention
PR #919 landed on main adding fresh Vally specs for three coverage-gap
skills (setup-local-sdk, convert-blazor-server-to-webapp, dotnet-webapi)
under the interim eval.vally.yaml filename. This branch collapses to a
single Vally spec named eval.yaml, so the merge left each skill with both
my migrated eval.yaml and #919's newer eval.vally.yaml (a git-clean but
semantic duplicate the pipeline would not discover).
Adopt #919's reviewed Vally content as the canonical eval.yaml for all
three skills and drop the redundant eval.vally.yaml files, preserving the
'sole eval.yaml, zero eval.vally.yaml' invariant. Content is byte-identical
to #919's blobs (b14121c6 / 47d43823 / 18a9362f), which already passed
vally-evaluate on main.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reorganize plugins per agreed structure (#271)
Split the dotnet plugin into domain-specific plugins per discussion
in #271:
- dotnet: narrowed to common everyday C#/.NET coding tasks
- dotnet-diag: perf investigations, debugging, incident analysis
- dotnet-data: data access and Entity Framework
- dotnet-aspnet, dotnet-wpf, dotnet-winforms, dotnet-maui: scaffolded
empty to convey intent for future framework-specific skills
Moved skills, tests, and CODEOWNERS entries accordingly.
No file content was changed; all skill/test moves are pure renames.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve dotnet-diag and dotnet-winforms descriptions
- dotnet-diag: add '.NET' qualifier to clarify scope
- dotnet-winforms: use official 'Windows Forms (WinForms)' name
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add .NET qualifier to dotnet-data description
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove empty plugins; list them in CONTRIBUTING.md as future options
Remove the four empty placeholder plugins (dotnet-aspnet, dotnet-wpf,
dotnet-winforms, dotnet-maui) that had only plugin.json and no skills.
Add a 'Plugin organization' section to CONTRIBUTING.md that describes the
domain-specific plugin groupings introduced in this PR and lists the four
removed plugin names as reserved candidates for future contributions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Create dotnet-upgrade plugin with migration/upgrade skills
Move migrate-nullable-references, thread-abort-migration, and
dotnet-aot-compat from dotnet to the new dotnet-upgrade plugin.
Update CODEOWNERS paths (same owners), CONTRIBUTING.md plugin table,
and fix the thread-abort-migration SKILL.md link in CONTRIBUTING.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Abhitej John