* Make Vally the sole LLM eval engine, retiring skill-validator evaluate
Collapses the parallel skill-validator + Vally eval pipeline into a single
Vally-only path while preserving every capability skill-validator provided:
PR gate, PR comment, dotnet/skills-data historical push, and the dashboard.
The skill-validator `check` linter is retained (skill-check.yml) pending
microsoft/vally #463.
- evaluation.yml: delete build-validator and the skill-validator `evaluate`
job; single `evaluate:` job now uses vally-evaluation.yml. Forward the
COPILOT_PAT_0..9 pool onto it (from #911) so the reusable workflow's token
gate is satisfied. Rewire comment-on-pr / report-status / publish-* /
deploy-dashboard onto `evaluate` and vally-results-* artifacts.
- vally-evaluation.yml: pin @microsoft/vally-cli@0.9.0; keep the #911 prepare
guard; write the generated per-plugin experiment to $GITHUB_WORKSPACE so
relative eval globs resolve.
- adapt.mjs: reconcile #887's runCompareWithRetry + conclusive/unmatched
hardening with the consolidation helpers (nonActivation, compareByStim,
roleToDashboard, etc.). Add consolidate.mjs and gen-experiment.mjs.
- build-replay-sessions.ps1: read Vally executor-session-logs/events.jsonl
instead of skill-validator session output.
- Rename all tests/*/*/eval.vally.yaml to eval.yaml (Vally format is now the
only eval format); author Vally evals for the four skills that lacked one.
- CONTRIBUTING.md + InvestigatingResults: point contributors at the Vally
harness; drop non-public links.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix setup-local-sdk eval for Vally schema
The hand-authored setup-local-sdk eval used skill-validator grader/environment
schema that Vally rejects at validation (no stimuli executed):
- file-contains/file-not-contains graders require 'value:' not 'substring:'
(output-contains correctly keeps 'substring:').
- environment.files entries require 'src:'/'dest:' (copy a fixture) rather than
inline 'path:'/'content:'. Moved the global.json body into fixtures/global.json.
Verified with 'vally lint --eval-spec' (0 errors) across all 97 eval.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix markdownlint violations in migration docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback on eval migration
- Consolidate run-vally-evals.sh into a single neutral entrypoint
eng/run-skill-evals.sh (deletes the thin wrapper; adapt.mjs stays in
eng/vally-adapter/ since CI invokes it directly)
- Rename reusable workflow vally-evaluation.yml -> evaluation-run.yml and
update all references/self-trigger regexes
- Rename local output dir vally-results -> eval-results
- Bake prerequisite preflight (Node 20+, GITHUB_TOKEN/gh) into the runner
- Simplify CONTRIBUTING "Running tests locally" prerequisites
- Scrub harness name from consumer-facing surfaces
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reinstate overfitting detection under the Vally harness
Overfitting detection previously ran inside skill-validator's `evaluate`
command, which the Vally migration replaced, so it silently stopped running.
Reinstate it as a standalone step wired into the Vally pipeline:
- Add `skill-validator overfitting` command that discovers skills, parses
each eval.yaml, runs the existing OverfittingJudge, and emits
[{plugin, skill, overfittingResult}] JSON (per-skill failures non-fatal,
bounded parallelism).
- Add a Vally-format parser bridge (ParseEvalConfigFlexible) so the judge
handles the current stimuli/graders eval.yaml format; the legacy
scenarios-only parser rejected all 98 evals, so the judge never ran.
- adapt.mjs: merge overfitting results onto each verdict via
--overfitting <file> (keyed by plugin/skill); output is byte-identical
when the flag is absent.
- evaluation-run.yml: build skill-validator (shared cache with check),
run the judge per leg reading GITHUB_TOKEN, and pass --overfitting to
adapt.mjs so the dashboard/skills-data consume it unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reconcile #919 Vally eval configs into the sole eval.yaml convention
PR #919 landed on main adding fresh Vally specs for three coverage-gap
skills (setup-local-sdk, convert-blazor-server-to-webapp, dotnet-webapi)
under the interim eval.vally.yaml filename. This branch collapses to a
single Vally spec named eval.yaml, so the merge left each skill with both
my migrated eval.yaml and #919's newer eval.vally.yaml (a git-clean but
semantic duplicate the pipeline would not discover).
Adopt #919's reviewed Vally content as the canonical eval.yaml for all
three skills and drop the redundant eval.vally.yaml files, preserving the
'sole eval.yaml, zero eval.vally.yaml' invariant. Content is byte-identical
to #919's blobs (b14121c6 / 47d43823 / 18a9362f), which already passed
vally-evaluate on main.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* add a new dotnet11 plugin and a new system text json skill to it.
* Update plugins/dotnet11/skills/system-text-json-net11/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* added the option to run dotnet11 skills. by default they don't get included
* Update tests/dotnet11/system-text-json-net11/eval.yaml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add JSON output assertions to system-text-json-net11 eval.yaml
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8e302f3f-4328-44a0-845d-4eb0c930af63
Co-authored-by: ManishJayaswal <9527491+ManishJayaswal@users.noreply.github.com>
* addressed CR comments
* addressed more CR comments
* fixed agument to pass while enabling dotnet11 skills
* Remove dotnet11-specific filtering from skill-validator and evaluation workflow
Now that .NET 11 is the default SDK for the repo (global.json updated on main),
there is no need for special dotnet11 exclusion logic. The dotnet11 plugin skills
are treated like any other plugin and included in all evaluation runs.
Changes:
- Remove --include-dotnet11 CLI option and filtering logic from EvaluateCommand.cs
- Remove IncludeDotnet11 from ValidatorConfig model
- Remove --dotnet11 flag parsing from evaluation workflow gate job
- Remove dotnet11 exclusion logic from discover job
- Remove conditional .NET 11 preview SDK install step (global.json handles it)
- Remove dotnet11 from excludeFromSchedule list
- Remove all dotnet11-specific warning/status messages
- Remove section 9 from InvestigatingResults.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* address code review comments after rebase
* added the plugin to cursor and added MIT license to skill
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: [Manish Jayaswal] <[manishj@microsoft.com]>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ManishJayaswal <9527491+ManishJayaswal@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Amaury Levé