* Surface activation-only evaluation failures
Add post-activation continuation telemetry to Vally results, PR summaries, and dashboard evidence so models that load a skill and stop can be diagnosed separately from routing and content failures. Relates to #1160.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix activation continuation diagnostics
Use ordered trajectory events to distinguish pre-activation tools from true post-activation continuation, prioritize runtime diagnosis over preference-loss advice, and retain plugin-arm failures on target dormancy cases.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Describe the observed absence of counted pass data without assuming that a skill has no boolean graders.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5b4ee7ec-7244-4668-a151-5abee0ef2c4a
Keep one in-flight initialization request, retry failed index loads, and clarify why the derived index is excluded from plugin tabs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5b4ee7ec-7244-4668-a151-5abee0ef2c4a
When the baseline metric is 0, reduction() is null and the Tokens/Time delta
cell rendered a blank percent label. Show an explicit n/a instead, matching the
rollup cells, so the delta is never an ambiguous blank.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
Generate a retained skill-value.json index from per-plugin histories so the default dashboard tab loads only the compact value data instead of every full benchmark file. Keep derived data out of plugin tabs and harden purge enumeration.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5b4ee7ec-7244-4668-a151-5abee0ef2c4a
- Drill-down paired-observation count now uses min(baseline.n, treatment.n),
matching the table's paired-n, so inconsistent or legacy entries cannot
overstate the paired sample size.
- signedPct helper renders an exact-or-rounded zero reduction as "0%" with no
sign, so "no change" no longer reads as a "+0%" regression (deltaCell and the
rollup percent both use it).
- When skill-value.js is present but there are no evaluation plugins, fall back
to the Token Usage tab instead of leaving the page on the Skill Value panel's
permanent "Loading...".
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
- Load skill-value.js (and token-usage.js) before dashboard.js so
window.initSkillValue exists when the default Skill Value tab renders on
first load; previously the fallback activated the first plugin tab instead.
- Aggregate key now includes the plugin, so two plugins that share a skill
name no longer blend histories (the view is grouped Plugin -> Skill -> Model).
- valueSentence distinguishes "not-passed rate still gathering data" (pass
checks present but below the gate) from "No pass/fail data" (no boolean
graders), and describes the unit as counted trials rather than scenarios.
- Document that only the primary judge (vally-results-*) feeds benchmark data;
the dual-judge re-score (vally-crossjudge-*) feeds only judge-comparison.json,
so Skill Value and Quality/Efficiency both reflect the primary judge.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
Adds a Plugin -> Skill -> Model "Skill Value" tab that surfaces, per skill,
the token and time a skill saves and how often scenarios pass without it.
Reuses the existing publish-eval-data job by emitting an entries.SkillValue
block from generate-benchmark-data.ps1 (no new job, no adapter change).
Fixes in this change:
- Quality/Efficiency stay one entry per executor model per run. The scheduled
benchmark merge now groups by (model, judge) for Skill Value, but passes
-SkillValueOnly for the second+ judge of a model so dual-judge runs no longer
duplicate same-model Quality/Efficiency points.
- Pass telemetry is labelled honestly. adapt.mjs notes the per-arm pass booleans
may include LLM-grader results, so the field is hasPassData (not
hasDeterministic), the column reads "Not-passed", and the claim reads "do not
pass their checks" with a tooltip caveat rather than an objective failure gate.
- Single-model groups show that model's real deltas; multi-model rollups read
"N model/judge result(s) show measured value" instead of an ambiguous 0/m.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
* Improve dense dashboard charts
Hide oversized quality-chart legends when their rendered series exceed the compact threshold, and direct viewers to the existing model filter for the color key.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f626eeb3-fc9f-4e9f-9a88-4dbde7594557
* Add accessible model markers to dashboard charts
Pair model colors with stable shape and rotation markers in quality charts and the Models filter. Reserve issue-marker shapes and keep vanilla points visible when series overlap.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f626eeb3-fc9f-4e9f-9a88-4dbde7594557
---------
Copilot-Session: f626eeb3-fc9f-4e9f-9a88-4dbde7594557
The model->colour map lived in a single module-level ctiveModelColors
that every
enderPlugin overwrote. Because draw() now runs lazily on
filter toggles (after a plugin was first rendered), switching plugin tabs
and then toggling a filter on an earlier plugin recoloured its charts and
summary dots using the later plugin's map.
Capture the map in a plugin-scoped pluginModelColors const and restore
the module global from it at the top of every draw(), so each plugin
always renders with its own canonical colours.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
The per-model view can get busy once several executor models share one plugin's
history. Add a checkbox filter bar (all models enabled by default); toggling a
model re-renders the summary table and every quality/efficiency chart for just
the selected models. Model colours stay canonical (bound to full history) so
hiding a model never recolours the others, and at least one model stays enabled
so the view is never empty. Chart instances are now tracked and destroyed on
each re-render to avoid canvas-reuse leaks.
Verified with a Playwright render check: filter bar shows one checkbox per model
all enabled; unchecking a model drops it from the summary and every chart while
the rest keep their colours; re-checking restores it.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
The summary container is a CSS grid, so the table card's flex:1 1 100% was
ignored and the card was squeezed into one ~200px grid track, clipping the
Delta Isolated / Delta Plugin columns. Add grid-column:1/-1 so the table spans
the whole row. Caught by a Playwright render check of a 3-model fixture.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
Cross-family evaluation interleaves several executor models in one plugin's
benchmark history. The summary cards averaged the last 50 runs without grouping
by model and the 'Model' card showed only the last-appended model, while each
quality trend drew one line per variant across all models - blending families
into a single average and a single zig-zag line.
Group the recent window by model into a per-model quality table (Skilled /
Plugin / Vanilla + deltas, one row per model) and redraw each trend with one
line per model (colour = model, dash = variant, spanGaps:false) so a line never
bridges two families. Per-point issue markers and tooltips are preserved.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
* Make Vally the sole LLM eval engine, retiring skill-validator evaluate
Collapses the parallel skill-validator + Vally eval pipeline into a single
Vally-only path while preserving every capability skill-validator provided:
PR gate, PR comment, dotnet/skills-data historical push, and the dashboard.
The skill-validator `check` linter is retained (skill-check.yml) pending
microsoft/vally #463.
- evaluation.yml: delete build-validator and the skill-validator `evaluate`
job; single `evaluate:` job now uses vally-evaluation.yml. Forward the
COPILOT_PAT_0..9 pool onto it (from #911) so the reusable workflow's token
gate is satisfied. Rewire comment-on-pr / report-status / publish-* /
deploy-dashboard onto `evaluate` and vally-results-* artifacts.
- vally-evaluation.yml: pin @microsoft/vally-cli@0.9.0; keep the #911 prepare
guard; write the generated per-plugin experiment to $GITHUB_WORKSPACE so
relative eval globs resolve.
- adapt.mjs: reconcile #887's runCompareWithRetry + conclusive/unmatched
hardening with the consolidation helpers (nonActivation, compareByStim,
roleToDashboard, etc.). Add consolidate.mjs and gen-experiment.mjs.
- build-replay-sessions.ps1: read Vally executor-session-logs/events.jsonl
instead of skill-validator session output.
- Rename all tests/*/*/eval.vally.yaml to eval.yaml (Vally format is now the
only eval format); author Vally evals for the four skills that lacked one.
- CONTRIBUTING.md + InvestigatingResults: point contributors at the Vally
harness; drop non-public links.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix setup-local-sdk eval for Vally schema
The hand-authored setup-local-sdk eval used skill-validator grader/environment
schema that Vally rejects at validation (no stimuli executed):
- file-contains/file-not-contains graders require 'value:' not 'substring:'
(output-contains correctly keeps 'substring:').
- environment.files entries require 'src:'/'dest:' (copy a fixture) rather than
inline 'path:'/'content:'. Moved the global.json body into fixtures/global.json.
Verified with 'vally lint --eval-spec' (0 errors) across all 97 eval.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix markdownlint violations in migration docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback on eval migration
- Consolidate run-vally-evals.sh into a single neutral entrypoint
eng/run-skill-evals.sh (deletes the thin wrapper; adapt.mjs stays in
eng/vally-adapter/ since CI invokes it directly)
- Rename reusable workflow vally-evaluation.yml -> evaluation-run.yml and
update all references/self-trigger regexes
- Rename local output dir vally-results -> eval-results
- Bake prerequisite preflight (Node 20+, GITHUB_TOKEN/gh) into the runner
- Simplify CONTRIBUTING "Running tests locally" prerequisites
- Scrub harness name from consumer-facing surfaces
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reinstate overfitting detection under the Vally harness
Overfitting detection previously ran inside skill-validator's `evaluate`
command, which the Vally migration replaced, so it silently stopped running.
Reinstate it as a standalone step wired into the Vally pipeline:
- Add `skill-validator overfitting` command that discovers skills, parses
each eval.yaml, runs the existing OverfittingJudge, and emits
[{plugin, skill, overfittingResult}] JSON (per-skill failures non-fatal,
bounded parallelism).
- Add a Vally-format parser bridge (ParseEvalConfigFlexible) so the judge
handles the current stimuli/graders eval.yaml format; the legacy
scenarios-only parser rejected all 98 evals, so the judge never ran.
- adapt.mjs: merge overfitting results onto each verdict via
--overfitting <file> (keyed by plugin/skill); output is byte-identical
when the flag is absent.
- evaluation-run.yml: build skill-validator (shared cache with check),
run the judge per leg reading GITHUB_TOKEN, and pass --overfitting to
adapt.mjs so the dashboard/skills-data consume it unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reconcile #919 Vally eval configs into the sole eval.yaml convention
PR #919 landed on main adding fresh Vally specs for three coverage-gap
skills (setup-local-sdk, convert-blazor-server-to-webapp, dotnet-webapi)
under the interim eval.vally.yaml filename. This branch collapses to a
single Vally spec named eval.yaml, so the merge left each skill with both
my migrated eval.yaml and #919's newer eval.vally.yaml (a git-clean but
semantic duplicate the pipeline would not discover).
Adopt #919's reviewed Vally content as the canonical eval.yaml for all
three skills and drop the redundant eval.vally.yaml files, preserving the
'sole eval.yaml, zero eval.vally.yaml' invariant. Content is byte-identical
to #919's blobs (b14121c6 / 47d43823 / 18a9362f), which already passed
vally-evaluate on main.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make vanilla series distinguishable on overlapping dashboard charts
Vanilla lines were hard to tell apart from the isolated/plugin lines when their values coincided. Give every vanilla series hollow diamond markers (larger than the round markers of the other series, transparent fill so the underlying line shows through) and a more prominent dash, across the quality and efficiency charts.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
The dashboard-session-data branch on this repo had grown to ~480MB and was bloating clones. Session data has been migrated to the standalone dotnet/skills-data repository (branch of the same name). This PR redirects:
- dashboard.js -> reads manifest from dotnet/skills-data
- evaluation.yml PR-comment link -> points at dotnet/skills-data
- evaluation.yml publish-session-data job -> clones/pushes to dotnet/skills-data using a new SKILLS_DATA_TOKEN secret (fine-grained PAT with contents:write on the data repo)
Also adds --depth 1 to the deploy clone so future runs do not re-import history.
AGENTVIZ replay UI requires no changes; it accepts an arbitrary manifest URL via query string. raw.githubusercontent.com serves the new repo with Access-Control-Allow-Origin: * so the dotnet.github.io/skills/replay page can load it cross-origin.
The old branch on dotnet/skills is left in place for now and will be deleted manually after one successful run end-to-end.
* Move token usage data into separate branch
* Address PR review: add help doc, fix title injection, add token-usage fallback
* Add -SkipBenchmarkData switch, clean stale files on branch push
* Add plugin-aware skill validation with activation detection
* Address Copilot review comments (round 4)
- DESIGN doc: mark Targeted Skill Activation TODO as implemented
- ValidateCommand: fix misleading --plugin-dir comment
- Comparator: set PluginScore to null when no plugin runs exist
- ComparatorTests: copy missing fields in rehydrated ScenarioComparison
- AgentRunner: fix GetPluginClient docstring, route RunAgentCore through it
- AgentRunner: stage isolated skill to temp dir for true single-skill isolation
- dashboard.js: fix triple chart comment to match actual series names
- RunnerTests: update assertions for new isolated skill staging behavior
* Remove implementation doc
* Security Update
Identified and remediated 15 vulnerabilities across the skill-validator
runtime, dashboard, and CI/CD pipelines.
Changes:
- Runtime (AgentRunner.cs): path traversal prevention, deny-by-default
permissions, env scrubbing, MCP server hardening
- Regex (AssertionEvaluator.cs): ReDoS timeout
- Logging (SkillDiscovery.cs): exception logging
- Dashboard (dashboard.html, dashboard.js): SRI hash, XSS escaping
- CI/CD (5 workflow files): SHA pinning, persist-credentials: false
- Tests (RunnerTests.cs): 45+ new security test cases
MCP Server Defense-in-Depth (5 layers):
1. Command allowlist: only dotnet, node, npx, python, python3, uvx
2. Path rejection: no /, \, or .. in command names
3. Args validation: per-runtime dangerous flag blocklist
4. Env sanitization: strips PATH, LD_PRELOAD, NODE_OPTIONS,
DOTNET_STARTUP_HOOKS, etc.
5. Cwd dropped: custom working directories not passed to SDK
Subprocess Isolation:
- Sensitive env vars + prefix patterns scrubbed from child processes
- 120-second timeout with Kill(entireProcessTree: true)
- Deny-by-default permission model
Supply Chain:
- All GitHub Action references pinned to full commit SHAs
- All checkout steps set persist-credentials: false
- Chart.js CDN pinned to v4.4.7 with SRI integrity hash
* Update .github/workflows/skill-validator.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update eng/skill-validator/src/Services/AgentRunner.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update eng/skill-validator/src/Services/AgentRunner.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update tests to use fullCommandText field name
Update ExtractsCommandProperty and PrefersPathOverCommand tests to use
the fullCommandText property instead of command, matching the updated
permission request schema in AgentRunner.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add 'dnx' to allowed MCP commands
* Address PR review feedback: security hardening fixes
- CheckPermission: normalize paths with TrimEndingDirectorySeparator to
prevent double-separator false denials when dirs have trailing slash
- SanitizeMcpArgs: reject prefix forms (-econsole.log) and --flag=value
variants that bypass exact-match blocklist checks
- Setup file path traversal: normalize canonicalWorkDir and
canonicalSkillPath to avoid double-separator in StartsWith checks
- IsAllowedMcpCommand: use GetFileName instead of
GetFileNameWithoutExtension to block dotnet.bat/node.cmd; only strip
.exe on Windows
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* prune
* fix
* Fix concurrency setting
* Add binlog generation and upload to skill-validator workflow
Produce binary logs for build, test, and pack steps using /bl: flag.
Upload binlogs as artifacts (with if: always()) for build diagnostics.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve CheckPermission diagnostics and fix AOT build error
Log allowed directories alongside denied paths to help diagnose
permission mismatches. Replace JsonSerializer.Serialize with manual
string building to fix IL3050 AOT compilation error.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Extract fileName from permission requests
Some permission requests use 'fileName' instead of 'path'. Handle
this property to avoid false denials for file-level operations.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* nownar nu1903 temporarily
* Log config dir in verbose mode alongside work dir
Include skilled/baseline label for consistency with the work dir log.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add skilled/baseline label to permission denial logs
Helps identify which run type triggered the denial when debugging.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix empty runLabel brackets and add labels to judge permission checks
Conditionally format the label suffix to avoid empty () when no
runLabel is supplied. Pass 'judge' and 'pairwise-judge' labels from
the respective judge call sites.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Allow pairwise judge to read from skilled work directory
The pairwise judge compares baseline and skilled runs in one session,
but previously only allowed reads from the baseline work directory.
Add additionalAllowedDirs parameter to CheckPermission and pass the
skilled work directory through PairwiseJudgeOptions.SkilledWorkDir.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix tests to match allow-by-default for unrecognized permission requests
CheckPermission was changed to allow requests with no extractable
path/command (allow-by-default), but four tests still expected the
old deny-by-default behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove NoWarn
* Pin discover job checkout action to full commit SHA
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
* Purge dashboard entries older than 14 days
When updating plugin JSON files, filter out Quality and Efficiency
entries whose date is older than 14 days. If all entries are purged,
delete the JSON file so it is excluded from the components.json
manifest and no longer appears in the dashboard.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move stale file cleanup to workflow level
Address review feedback: the delete-if-empty branch in the script was
unreachable because fresh entries are always appended before purging.
Instead, add a workflow step that scans all existing data files after
per-plugin generation and removes any whose entries are entirely older
than 14 days.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move stale file purge logic into the script
Add a -PurgeStaleFiles/-DataDir parameter set so the workflow calls
the script instead of duplicating the retention logic inline. The
retention window is defined once in the script.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make retention days configurable via workflow env var
Add -RetentionDays parameter to the script (mandatory for
-PurgeStaleFiles, optional for generate mode) and define
DASHBOARD_RETENTION_DAYS env var at the workflow level so the value
is controlled in one place.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback: fix comment and add entries guard
- Fix workflow comment to accurately describe the purge behavior
- Guard against missing 'entries' key in purge mode to avoid silently
skipping malformed plugin JSON files
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eng/dashboard/generate-benchmark-data.ps1
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update .github/workflows/evaluation.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Refine parameter description in generate-benchmark-data.ps1 and add safeDirName function to sanitize skill names in reporter.ts
* refactor: make safeDirName a true sanitizer instead of a throwing validator
- Slugify skill names for directory use instead of hard-failing on
unexpected characters from SKILL.md frontmatter display names.
- Use path.basename() for robust single-segment check instead of
string.includes('..') which false-positives on names like 'foo..bar'.
- Throw on genuinely invalid names (empty, '.', '..', or all-special-chars).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix empty benches arrays in benchmark data generation
The json reporter lost per-skill verdict.json generation during a
refactor (b461016), causing generate-benchmark-data.ps1 to find no
verdict files and produce empty benches arrays.
Fix by:
- Restoring verdict.json generation in the json reporter
- Updating generate-benchmark-data.ps1 to read verdicts directly from
results.json instead of scanning for per-skill verdict.json files
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify generate-benchmark-data to accept ResultsFile path directly
Replace ResultsDir parameter with ResultsFile since the script only
needs the results.json path. Update evaluation.yml caller accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eng/dashboard/generate-benchmark-data.ps1
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Restructure repo to use src/<component> layout
Adopt the same repository structure as dotnet/skills:
- Move skills/ and tests/ under src/dotnet/
- Add plugin.json for the csharp-scripts component
- Add AGENTS.md with plugin versioning instructions
- Update marketplace.json files to point to new paths
- Update CI workflows for new directory structure
- Move contributor guidance from README into CONTRIBUTING.md
- Streamline README to match dotnet/skills format
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add evaluation workflow and dashboard infrastructure
Add GitHub Actions workflow for skill evaluation with parallel test
execution, response grading, and summary generation. Include:
- CI workflow (.github/workflows/evaluation.yml) with matrix-based
parallel test runs and configurable parameters
- Evaluation scripts (eng/evaluation/) for invoking Copilot, parsing
stats, evaluating responses, and generating summaries
- Dashboard (eng/dashboard/) with HTML/JS visualization and benchmark
data generation script
- Updated CONTRIBUTING.md with evaluation documentation reference
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Amaury Levé