* Add copy_test_files support and clean up test metadata
- Add copy_test_files option to eval.yaml setup schema for opt-in
auto-copy of test fixture files into agent working directory
- Update all msbuild eval.yaml files to enable copy_test_files
- Remove unused README.md, expected-output.md, and eval-test-prompt.txt
from test directories
- Document setup options in README.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix copy_test_files to place files at workDir root
Use readdir + per-entry cp instead of cp(dir, dir) which would
nest files under a subdirectory. Addresses PR review feedback.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reduce parallel skill count to 3 due to quota issues
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel runs support to skill-validator
- Add --parallel-runs flag (default: 8) to control max concurrent runs
- Refactor run loop to use p-limit for controlled parallelism
- All runs within a scenario now execute concurrently (up to limit)
- Significantly reduces validation time with multiple runs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel runs support to skill-validator
- Add --parallel-runs flag (default: 1) to control max concurrent runs
- Refactor run loop to use p-limit for controlled parallelism
- All runs within a scenario now execute concurrently (up to limit)
- Add workflow input and env variable (default: 3) for CI
- Add input validation for runs and parallelRuns (clamp to >= 1)
- Show spinner progress for both sequential and parallel modes
- Remove unused imports and variables
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve description
* Add DEFAULT_RUNS env variable to workflow
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel skills, coordinated spinner, and allSettled
- Add --parallel-skills flag to evaluate multiple skills concurrently
- Use single coordinated spinner shared across all parallel skills
- Prefix output with [skill] or [skill/scenario] for disambiguation
(scenario omitted when eval has only one scenario)
- Use Promise.allSettled so one skill failure doesn't discard others
- Per-scenario runLimit so parallel-runs is scoped per scenario
- Add workflow input and env variable for parallel-skills (default: 3)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Temporarily use Token 2
* Allow skill-validator step to fail without blocking the job
Results are still uploaded and published to the dashboard even when
skills fail validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove test changes
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Only evaluate changed components in PRs
Update evaluation workflow to detect which components have changed
in pull requests and only run evaluations for those, similar to
how validate-skills and validate-agents workflows work.
- Add change detection using git diff for PRs
- Add has_components output to skip evaluation when nothing changed
- Add no-changes job to report when no components were modified
- Push/schedule/workflow_dispatch still evaluate all components
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update validate-skills.yml
* Add validate-agents.yml to workflow paths
* Update workflow to trigger on changes to itself
* Update skill-validator.yml
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace hardcoded src/dotnet paths with a discover job that finds all
components under src/ with skills/ and tests/ directories. Each
component is evaluated in parallel via matrix strategy. Artifacts and
benchmark data are now generated per-component.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move skill-validator sources and tests from the external
adityamandaleeka/skill-validator repository into eng/skill-validator/.
Update evaluation.yml to build and run from the local copy instead of
checking out the external repo. Add skill-validator.yml CI pipeline
for building and testing the validator itself.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The third-party action peter-evans/create-or-update-comment@v4 is not
allowed by repository policy. Replace with inline gh pr comment command
which is pre-installed on GitHub-hosted runners.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restructure repo to use src/<component> layout
Adopt the same repository structure as dotnet/skills:
- Move skills/ and tests/ under src/dotnet/
- Add plugin.json for the csharp-scripts component
- Add AGENTS.md with plugin versioning instructions
- Update marketplace.json files to point to new paths
- Update CI workflows for new directory structure
- Move contributor guidance from README into CONTRIBUTING.md
- Streamline README to match dotnet/skills format
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add evaluation workflow and dashboard infrastructure
Add GitHub Actions workflow for skill evaluation with parallel test
execution, response grading, and summary generation. Include:
- CI workflow (.github/workflows/evaluation.yml) with matrix-based
parallel test runs and configurable parameters
- Evaluation scripts (eng/evaluation/) for invoking Copilot, parsing
stats, evaluating responses, and generating summaries
- Dashboard (eng/dashboard/) with HTML/JS visualization and benchmark
data generation script
- Updated CONTRIBUTING.md with evaluation documentation reference
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Viktor Hofer