Files
Abhitej John 78054c1161 Migrate LLM evals to the Vally harness (#877)
* Make Vally the sole LLM eval engine, retiring skill-validator evaluate

Collapses the parallel skill-validator + Vally eval pipeline into a single
Vally-only path while preserving every capability skill-validator provided:
PR gate, PR comment, dotnet/skills-data historical push, and the dashboard.
The skill-validator `check` linter is retained (skill-check.yml) pending
microsoft/vally #463.

- evaluation.yml: delete build-validator and the skill-validator `evaluate`
  job; single `evaluate:` job now uses vally-evaluation.yml. Forward the
  COPILOT_PAT_0..9 pool onto it (from #911) so the reusable workflow's token
  gate is satisfied. Rewire comment-on-pr / report-status / publish-* /
  deploy-dashboard onto `evaluate` and vally-results-* artifacts.
- vally-evaluation.yml: pin @microsoft/vally-cli@0.9.0; keep the #911 prepare
  guard; write the generated per-plugin experiment to $GITHUB_WORKSPACE so
  relative eval globs resolve.
- adapt.mjs: reconcile #887's runCompareWithRetry + conclusive/unmatched
  hardening with the consolidation helpers (nonActivation, compareByStim,
  roleToDashboard, etc.). Add consolidate.mjs and gen-experiment.mjs.
- build-replay-sessions.ps1: read Vally executor-session-logs/events.jsonl
  instead of skill-validator session output.
- Rename all tests/*/*/eval.vally.yaml to eval.yaml (Vally format is now the
  only eval format); author Vally evals for the four skills that lacked one.
- CONTRIBUTING.md + InvestigatingResults: point contributors at the Vally
  harness; drop non-public links.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix setup-local-sdk eval for Vally schema

The hand-authored setup-local-sdk eval used skill-validator grader/environment
schema that Vally rejects at validation (no stimuli executed):
- file-contains/file-not-contains graders require 'value:' not 'substring:'
  (output-contains correctly keeps 'substring:').
- environment.files entries require 'src:'/'dest:' (copy a fixture) rather than
  inline 'path:'/'content:'. Moved the global.json body into fixtures/global.json.

Verified with 'vally lint --eval-spec' (0 errors) across all 97 eval.yaml.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix markdownlint violations in migration docs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback on eval migration

- Consolidate run-vally-evals.sh into a single neutral entrypoint
  eng/run-skill-evals.sh (deletes the thin wrapper; adapt.mjs stays in
  eng/vally-adapter/ since CI invokes it directly)
- Rename reusable workflow vally-evaluation.yml -> evaluation-run.yml and
  update all references/self-trigger regexes
- Rename local output dir vally-results -> eval-results
- Bake prerequisite preflight (Node 20+, GITHUB_TOKEN/gh) into the runner
- Simplify CONTRIBUTING "Running tests locally" prerequisites
- Scrub harness name from consumer-facing surfaces

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reinstate overfitting detection under the Vally harness

Overfitting detection previously ran inside skill-validator's `evaluate`
command, which the Vally migration replaced, so it silently stopped running.
Reinstate it as a standalone step wired into the Vally pipeline:

- Add `skill-validator overfitting` command that discovers skills, parses
  each eval.yaml, runs the existing OverfittingJudge, and emits
  [{plugin, skill, overfittingResult}] JSON (per-skill failures non-fatal,
  bounded parallelism).
- Add a Vally-format parser bridge (ParseEvalConfigFlexible) so the judge
  handles the current stimuli/graders eval.yaml format; the legacy
  scenarios-only parser rejected all 98 evals, so the judge never ran.
- adapt.mjs: merge overfitting results onto each verdict via
  --overfitting <file> (keyed by plugin/skill); output is byte-identical
  when the flag is absent.
- evaluation-run.yml: build skill-validator (shared cache with check),
  run the judge per leg reading GITHUB_TOKEN, and pass --overfitting to
  adapt.mjs so the dashboard/skills-data consume it unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reconcile #919 Vally eval configs into the sole eval.yaml convention

PR #919 landed on main adding fresh Vally specs for three coverage-gap
skills (setup-local-sdk, convert-blazor-server-to-webapp, dotnet-webapi)
under the interim eval.vally.yaml filename. This branch collapses to a
single Vally spec named eval.yaml, so the merge left each skill with both
my migrated eval.yaml and #919's newer eval.vally.yaml (a git-clean but
semantic duplicate the pipeline would not discover).

Adopt #919's reviewed Vally content as the canonical eval.yaml for all
three skills and drop the redundant eval.vally.yaml files, preserving the
'sole eval.yaml, zero eval.vally.yaml' invariant. Content is byte-identical
to #919's blobs (b14121c6 / 47d43823 / 18a9362f), which already passed
vally-evaluate on main.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-07-21 15:22:31 -07:00

116 lines
5.6 KiB
YAML
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
name: create-blazor-project
description: Evaluates the dotnet-blazor/create-blazor-project skill
type: capability
config:
timeout: 20m
stimuli:
- name: University course catalog with enrollment form
prompt: |
Create a Blazor Web App for a university course catalog.
Requirements:
- Browse departments and courses in a searchable, filterable list
- View course detail pages (description, schedule, credits, instructor bio)
- Students submit an enrollment request form (name, student ID, course selection) — standard HTML form, no wizards or real-time validation
- Homepage shows featured courses and department links
- No authentication — open to all students and visitors
- Hosted on the university intranet with fast, reliable connectivity
graders:
- type: output-contains
config:
substring: dotnet new blazor
- type: output-matches
config:
pattern: -int(eractivity)? None
- type: output-not-contains
config:
substring: blazorwasm
- type: output-not-contains
config:
substring: blazorserver
- type: output-matches
config:
pattern: (Static SSR|no.interactive|None.*interactivity)
- type: prompt
rubric:
- Correct template and configuration — single-project structure (no .Client project), Program.cs calls AddRazorComponents() without any AddInteractive* chains, App.razor does NOT set @rendermode on Routes or HeadOutlet
- Recognizes that Static SSR uses standard HTML POST forms and enhanced navigation — no interactivity needed for a course catalog with enrollment
- Documents the static rendering choice — explains why the app uses no interactive render mode
- Plans the domain features — identifies the pages/components needed for browsing courses and submitting enrollments
- name: Recipe community with interactive ratings on static pages
prompt: |
Create a new Blazor Web App project for a recipe sharing community.
Run `dotnet new blazor` with the appropriate options, create the project structure,
and implement the pages.
Requirements:
- Browse recipes by category — read-only content pages
- Each recipe page has a star-rating widget (click to rate 1–5 stars)
- A "Submit Recipe" page with a form (title, ingredients, steps)
- Homepage displays recipe categories (static content)
- Most pages are read-only; only the rating widget and submit form need interactivity
graders:
- type: output-contains
config:
substring: dotnet new blazor
- type: output-matches
config:
pattern: -int(eractivity)? Server
- type: output-not-matches
config:
pattern: dotnet new blazor[^\n]*-ai
- type: output-not-contains
config:
substring: blazorwasm
- type: output-not-contains
config:
substring: blazorserver
- type: output-matches
config:
pattern: (per.page|per-page|opt.in)
- type: prompt
rubric:
- Correct template and configuration — Program.cs registers AddInteractiveServerComponents() and AddInteractiveServerRenderMode(), App.razor does NOT set @rendermode on Routes so interactivity is per-page opt-in
- Static pages (browse, homepage, recipe content) remain SSR for performance while interactive components (rating widget, comments) opt in with @rendermode InteractiveServer
- Documents the per-page strategy — explains why most pages are static and only specific features opt in to interactivity
- Differentiates which features need interactivity (rating widget, form submission) vs which can be static (content browsing)
- name: Global logistics tracking for worldwide users
prompt: |
Create a Blazor Web App for a logistics company that tracks shipments.
Requirements:
- Dispatchers view a list of shipments with status and destination
- Customers check their shipment status and estimated delivery
- Users are worldwide — round-trip latency to a single server is noticeable
- The app must load fast on first visit, then feel snappy without server round-trips
- Every page involves interactive controls
- Staff and customers authenticate with individual accounts
graders:
- type: output-contains
config:
substring: dotnet new blazor
- type: output-matches
config:
pattern: -int(eractivity)? Auto
- type: output-matches
config:
pattern: (-ai|--all-interactive)
- type: output-not-contains
config:
substring: blazorwasm
- type: output-not-contains
config:
substring: blazorserver
- type: output-matches
config:
pattern: (-au|--auth) Individual
- type: output-matches
config:
pattern: (\.Client|Auto.*mode|dual.*execution)
- type: prompt
rubric:
- Correct template and configuration — two-project structure with .Client project, Server Program.cs registers both AddInteractiveServerComponents() and AddInteractiveWebAssemblyComponents(), interactive components live in .Client
- Components cannot directly access server resources — all data through HTTP APIs, both server and client Program.cs register matching services, component code must not assume execution environment
- Documents the Auto mode trade-offs — explains why this rendering strategy suits the requirements
- Addresses the latency requirement — explains why Auto mode solves worldwide latency (Server first, then switches to WebAssembly for client-side speed)