* feat: update build-validator job and README for Node.js version requirement
- Update an existing comment if it exists with the total summary
- Build the skill-validator just once and upload
- Clean-up when passing CLI args to skill-validator
* Update .github/workflows/evaluation.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Prune dev dependencies before uploading validator artifact
Reduces artifact size by removing devDependencies (typescript, vitest, etc.)
from node_modules/ since the evaluate job only needs the built dist/.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Updates
* Consolidate step summary into comment-on-pr job
Move GITHUB_STEP_SUMMARY writing from per-matrix Generate Summary step
into the existing Consolidate summaries step in comment-on-pr job.
This produces a single unified summary instead of fragmented per-component
summaries. The job now runs for all event types, with the PR comment
step scoped to pull_request events only.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refactor reporters to use --results-dir for file output
- Remove --no-save-results option and saveRunResults function
- Reporters no longer accept inline paths (e.g., json:path)
- Add --results-dir option: when specified, file reporters (json, junit,
markdown) write to that directory; when omitted, they write to console
- Enhance markdown reporter with per-scenario judge reports including
rubric scores, reasoning, and agent output
- Update workflow, README, and CONTRIBUTING docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Default --results-dir to .skill-validator-results and enable json/markdown reporters by default
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix unresolved matrix expression in skipped evaluate job name
Replace if: has_entries with if: !cancelled() && !failure() so the job
proceeds with an empty matrix, creating zero instances instead of a
skipped job with unresolved expressions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update required nodejs version in CONTRIBUTING doc
* Address PR feedback: fix timeout message, improve reporter output handling
- Use effective timeout value in error message to avoid 'undefineds'
- Only create resultsDir when file-based reporters are selected
- Write results into timestamped subdirectory (yyyyMMdd-HHmmss) to avoid stale output
- Use scenario-level improvementScore for markdown verdict icons
- Update workflow to find summary.md in timestamped subdirs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update bin-obj clash skill
* Fix PR comment step: use gh api instead of gh pr comment
gh pr comment requires a git repo checkout, but comment-on-pr job
only downloads artifacts. Switch to gh api for creating comments.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Touch one more skill
* Address PR feedback: remove dead code, fix formatting, improve workflow
- Remove dead validation check in cli.ts (resultsDir always has default)
- Fix missing spaces in variable declarations in reporter.ts
- Quote command substitution in consolidation script
- Use explicit needs.build-validator.result == 'success' condition
- Strip duplicate headers when consolidating multi-component summaries
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace markdown concatenation with consolidate subcommand
Instead of fragile bash string manipulation to combine per-component
summary.md files (which caused duplicate headers, empty tables, and
repeated model lines), add a 'consolidate' subcommand that reads
results.json files and generates a single clean markdown summary.
The subcommand takes explicit file paths as arguments — the caller
controls exactly which files to merge. Runner-specific concerns
(comment markers, run URLs) stay in the workflow script.
- Extract generateMarkdownSummary() from reportMarkdown() in reporter.ts
- Add 'skill-validator consolidate' subcommand in cli.ts
- Update workflow to use the new subcommand
- Document consolidate in README.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add copy_test_files support and clean up test metadata
- Add copy_test_files option to eval.yaml setup schema for opt-in
auto-copy of test fixture files into agent working directory
- Update all msbuild eval.yaml files to enable copy_test_files
- Remove unused README.md, expected-output.md, and eval-test-prompt.txt
from test directories
- Document setup options in README.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix copy_test_files to place files at workDir root
Use readdir + per-entry cp instead of cp(dir, dir) which would
nest files under a subdirectory. Addresses PR review feedback.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reduce parallel skill count to 3 due to quota issues
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel runs support to skill-validator
- Add --parallel-runs flag (default: 8) to control max concurrent runs
- Refactor run loop to use p-limit for controlled parallelism
- All runs within a scenario now execute concurrently (up to limit)
- Significantly reduces validation time with multiple runs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel runs support to skill-validator
- Add --parallel-runs flag (default: 1) to control max concurrent runs
- Refactor run loop to use p-limit for controlled parallelism
- All runs within a scenario now execute concurrently (up to limit)
- Add workflow input and env variable (default: 3) for CI
- Add input validation for runs and parallelRuns (clamp to >= 1)
- Show spinner progress for both sequential and parallel modes
- Remove unused imports and variables
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve description
* Add DEFAULT_RUNS env variable to workflow
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel skills, coordinated spinner, and allSettled
- Add --parallel-skills flag to evaluate multiple skills concurrently
- Use single coordinated spinner shared across all parallel skills
- Prefix output with [skill] or [skill/scenario] for disambiguation
(scenario omitted when eval has only one scenario)
- Use Promise.allSettled so one skill failure doesn't discard others
- Per-scenario runLimit so parallel-runs is scoped per scenario
- Add workflow input and env variable for parallel-skills (default: 3)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Temporarily use Token 2
* Allow skill-validator step to fail without blocking the job
Results are still uploaded and published to the dashboard even when
skills fail validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove test changes
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Only evaluate changed components in PRs
Update evaluation workflow to detect which components have changed
in pull requests and only run evaluations for those, similar to
how validate-skills and validate-agents workflows work.
- Add change detection using git diff for PRs
- Add has_components output to skip evaluation when nothing changed
- Add no-changes job to report when no components were modified
- Push/schedule/workflow_dispatch still evaluate all components
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update validate-skills.yml
* Add validate-agents.yml to workflow paths
* Update workflow to trigger on changes to itself
* Update skill-validator.yml
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace hardcoded src/dotnet paths with a discover job that finds all
components under src/ with skills/ and tests/ directories. Each
component is evaluated in parallel via matrix strategy. Artifacts and
benchmark data are now generated per-component.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move skill-validator sources and tests from the external
adityamandaleeka/skill-validator repository into eng/skill-validator/.
Update evaluation.yml to build and run from the local copy instead of
checking out the external repo. Add skill-validator.yml CI pipeline
for building and testing the validator itself.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The third-party action peter-evans/create-or-update-comment@v4 is not
allowed by repository policy. Replace with inline gh pr comment command
which is pre-installed on GitHub-hosted runners.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restructure repo to use src/<component> layout
Adopt the same repository structure as dotnet/skills:
- Move skills/ and tests/ under src/dotnet/
- Add plugin.json for the csharp-scripts component
- Add AGENTS.md with plugin versioning instructions
- Update marketplace.json files to point to new paths
- Update CI workflows for new directory structure
- Move contributor guidance from README into CONTRIBUTING.md
- Streamline README to match dotnet/skills format
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add evaluation workflow and dashboard infrastructure
Add GitHub Actions workflow for skill evaluation with parallel test
execution, response grading, and summary generation. Include:
- CI workflow (.github/workflows/evaluation.yml) with matrix-based
parallel test runs and configurable parameters
- Evaluation scripts (eng/evaluation/) for invoking Copilot, parsing
stats, evaluating responses, and generating summaries
- Dashboard (eng/dashboard/) with HTML/JS visualization and benchmark
data generation script
- Updated CONTRIBUTING.md with evaluation documentation reference
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Viktor Hofer