* Add copy_test_files support and clean up test metadata
- Add copy_test_files option to eval.yaml setup schema for opt-in
auto-copy of test fixture files into agent working directory
- Update all msbuild eval.yaml files to enable copy_test_files
- Remove unused README.md, expected-output.md, and eval-test-prompt.txt
from test directories
- Document setup options in README.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix copy_test_files to place files at workDir root
Use readdir + per-entry cp instead of cp(dir, dir) which would
nest files under a subdirectory. Addresses PR review feedback.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reduce parallel skill count to 3 due to quota issues
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel runs support to skill-validator
- Add --parallel-runs flag (default: 8) to control max concurrent runs
- Refactor run loop to use p-limit for controlled parallelism
- All runs within a scenario now execute concurrently (up to limit)
- Significantly reduces validation time with multiple runs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel runs support to skill-validator
- Add --parallel-runs flag (default: 1) to control max concurrent runs
- Refactor run loop to use p-limit for controlled parallelism
- All runs within a scenario now execute concurrently (up to limit)
- Add workflow input and env variable (default: 3) for CI
- Add input validation for runs and parallelRuns (clamp to >= 1)
- Show spinner progress for both sequential and parallel modes
- Remove unused imports and variables
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve description
* Add DEFAULT_RUNS env variable to workflow
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel skills, coordinated spinner, and allSettled
- Add --parallel-skills flag to evaluate multiple skills concurrently
- Use single coordinated spinner shared across all parallel skills
- Prefix output with [skill] or [skill/scenario] for disambiguation
(scenario omitted when eval has only one scenario)
- Use Promise.allSettled so one skill failure doesn't discard others
- Per-scenario runLimit so parallel-runs is scoped per scenario
- Add workflow input and env variable for parallel-skills (default: 3)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Temporarily use Token 2
* Allow skill-validator step to fail without blocking the job
Results are still uploaded and published to the dashboard even when
skills fail validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove test changes
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Rename 12 test directories to match their corresponding skill names
(e.g. bin-obj-clash -> check-bin-obj-clash, build-errors-cs -> common-build-errors)
- Allow empty scenarios in eval.yaml schema for placeholder tests
- Add skip logic in CLI for skills with empty scenarios
- Create placeholder eval.yaml (scenarios: []) for 8 skills without tests
- Restore --strict --require-evals in evaluation workflow
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move skill-validator sources and tests from the external
adityamandaleeka/skill-validator repository into eng/skill-validator/.
Update evaluation.yml to build and run from the local copy instead of
checking out the external repo. Add skill-validator.yml CI pipeline
for building and testing the validator itself.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Viktor Hofer