* Add tests for binlog-failure-analysis skill
* Add support for using MCP within eval tests
* Update the skill and testcase - to better target the usage
* Change maxLevels default value from 4 to 2
* Add --warn-only option to skill-validator for non-fatal skill failures
This avoids the continue-on-error setting the yml which sometimes hides real issues.
* Change default runs from 1 to 2 in evaluation.yml
Updated the number of runs to 2 for better testing balance.
* Rename --warn-only to --threshold-warn-only and fail on execution errors
- Rename --warn-only to --threshold-warn-only to clarify it only applies
to threshold-based verdict failures, not execution errors.
- Track rejected evaluation promises and return exit code 1 for execution
errors even in --threshold-warn-only mode.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove redundant --strict option, clarify --threshold-warn-only
--strict only implied --require-evals, which already exists as its own
flag. Remove --strict to avoid confusion about its interaction with
--threshold-warn-only. Update --threshold-warn-only help text to clarify
it only suppresses threshold failures, not runtime/config errors.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Only post PR comment when evaluate matrix succeeds
When a new push triggers cancel-in-progress, the evaluate matrix jobs
get cancelled but comment-on-pr still ran due to always(). This caused
partial/stale results to be posted. Require evaluate to succeed before
posting the PR comment.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add --verdict-warn-only flag with failureKind tracking
--verdict-warn-only treats verdict failures (threshold, completion
regression, skill not activated) as warnings (exit 0), while execution
errors and --require-evals missing_eval failures remain fatal.
Add failureKind field to SkillVerdict to categorize failure reasons.
Remove --strict (redundant with --require-evals).
Update README and evaluation.yml with documentation.
Allow comment-on-pr to run on evaluate failure (skip only on cancel).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refine parameter description in generate-benchmark-data.ps1 and add safeDirName function to sanitize skill names in reporter.ts
* refactor: make safeDirName a true sanitizer instead of a throwing validator
- Slugify skill names for directory use instead of hard-failing on
unexpected characters from SKILL.md frontmatter display names.
- Use path.basename() for robust single-segment check instead of
string.includes('..') which false-positives on names like 'foo..bar'.
- Throw on genuinely invalid names (empty, '.', '..', or all-special-chars).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix empty benches arrays in benchmark data generation
The json reporter lost per-skill verdict.json generation during a
refactor (b461016), causing generate-benchmark-data.ps1 to find no
verdict files and produce empty benches arrays.
Fix by:
- Restoring verdict.json generation in the json reporter
- Updating generate-benchmark-data.ps1 to read verdicts directly from
results.json instead of scanning for per-skill verdict.json files
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify generate-benchmark-data to accept ResultsFile path directly
Replace ResultsDir parameter with ResultsFile since the script only
needs the results.json path. Update evaluation.yml caller accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eng/dashboard/generate-benchmark-data.ps1
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
When a PR modifies files under .github/workflows/ or
eng/skill-validator/, all component evaluations are triggered since
these infra changes can affect any evaluation result. To prevent
quota issues, infra-triggered evaluations use 1 run instead of the
default.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix extractJson picking up non-JSON brace groups (e.g., C# code)
extractJson now validates each brace-matched candidate with JSON.parse
before returning it. If the candidate isn't valid JSON (even after
escape sanitization), it skips to the next brace group. This prevents
C# code snippets or other non-JSON content from being mistakenly
extracted as the judge's JSON response.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add test for skipping brace groups with invalid escapes
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The skill-validator reporter creates timestamped subdirectories
(YYYYMMDD-HHMMSS format), not run-* directories. The benchmark
step was looking for run-* and never finding results. Also sort
descending to pick the latest run when multiple exist.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix JSON parse error from invalid LLM escape sequences in pairwise judge
Wrap JSON.parse in try/catch and retry after stripping invalid escape
sequences (e.g., \' or \a) that LLMs sometimes produce in reasoning text.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR feedback: improve error context and add parser tests
- Preserve original error and direction in all failure paths
- Only attempt escape sanitization when invalid escapes are detected
- Include JSON snippet in error messages for debugging
- Export parsePairwiseResponse for direct testing
- Add 7 tests covering valid JSON, invalid escapes, malformed JSON,
missing JSON, and direction reversal
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Extract shared LLM JSON parsing into llm-json.ts
- Create llm-json.ts with extractJson() and parseLlmJson() helpers
- Refactor judge.ts and pairwise-judge.ts to use shared helpers
- Remove duplicated extractOutermostJson and escape-sanitization logic
- Add 11 tests for the shared module (extractJson + parseLlmJson)
- Add 8 tests for judge.ts parseJudgeResponse
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: update build-validator job and README for Node.js version requirement
- Update an existing comment if it exists with the total summary
- Build the skill-validator just once and upload
- Clean-up when passing CLI args to skill-validator
* Update .github/workflows/evaluation.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Prune dev dependencies before uploading validator artifact
Reduces artifact size by removing devDependencies (typescript, vitest, etc.)
from node_modules/ since the evaluate job only needs the built dist/.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Updates
* Consolidate step summary into comment-on-pr job
Move GITHUB_STEP_SUMMARY writing from per-matrix Generate Summary step
into the existing Consolidate summaries step in comment-on-pr job.
This produces a single unified summary instead of fragmented per-component
summaries. The job now runs for all event types, with the PR comment
step scoped to pull_request events only.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refactor reporters to use --results-dir for file output
- Remove --no-save-results option and saveRunResults function
- Reporters no longer accept inline paths (e.g., json:path)
- Add --results-dir option: when specified, file reporters (json, junit,
markdown) write to that directory; when omitted, they write to console
- Enhance markdown reporter with per-scenario judge reports including
rubric scores, reasoning, and agent output
- Update workflow, README, and CONTRIBUTING docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Default --results-dir to .skill-validator-results and enable json/markdown reporters by default
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix unresolved matrix expression in skipped evaluate job name
Replace if: has_entries with if: !cancelled() && !failure() so the job
proceeds with an empty matrix, creating zero instances instead of a
skipped job with unresolved expressions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update required nodejs version in CONTRIBUTING doc
* Address PR feedback: fix timeout message, improve reporter output handling
- Use effective timeout value in error message to avoid 'undefineds'
- Only create resultsDir when file-based reporters are selected
- Write results into timestamped subdirectory (yyyyMMdd-HHmmss) to avoid stale output
- Use scenario-level improvementScore for markdown verdict icons
- Update workflow to find summary.md in timestamped subdirs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update bin-obj clash skill
* Fix PR comment step: use gh api instead of gh pr comment
gh pr comment requires a git repo checkout, but comment-on-pr job
only downloads artifacts. Switch to gh api for creating comments.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Touch one more skill
* Address PR feedback: remove dead code, fix formatting, improve workflow
- Remove dead validation check in cli.ts (resultsDir always has default)
- Fix missing spaces in variable declarations in reporter.ts
- Quote command substitution in consolidation script
- Use explicit needs.build-validator.result == 'success' condition
- Strip duplicate headers when consolidating multi-component summaries
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace markdown concatenation with consolidate subcommand
Instead of fragile bash string manipulation to combine per-component
summary.md files (which caused duplicate headers, empty tables, and
repeated model lines), add a 'consolidate' subcommand that reads
results.json files and generates a single clean markdown summary.
The subcommand takes explicit file paths as arguments — the caller
controls exactly which files to merge. Runner-specific concerns
(comment markers, run URLs) stay in the workflow script.
- Extract generateMarkdownSummary() from reportMarkdown() in reporter.ts
- Add 'skill-validator consolidate' subcommand in cli.ts
- Update workflow to use the new subcommand
- Document consolidate in README.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add copy_test_files support and clean up test metadata
- Add copy_test_files option to eval.yaml setup schema for opt-in
auto-copy of test fixture files into agent working directory
- Update all msbuild eval.yaml files to enable copy_test_files
- Remove unused README.md, expected-output.md, and eval-test-prompt.txt
from test directories
- Document setup options in README.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix copy_test_files to place files at workDir root
Use readdir + per-entry cp instead of cp(dir, dir) which would
nest files under a subdirectory. Addresses PR review feedback.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reduce parallel skill count to 3 due to quota issues
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel runs support to skill-validator
- Add --parallel-runs flag (default: 8) to control max concurrent runs
- Refactor run loop to use p-limit for controlled parallelism
- All runs within a scenario now execute concurrently (up to limit)
- Significantly reduces validation time with multiple runs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel runs support to skill-validator
- Add --parallel-runs flag (default: 1) to control max concurrent runs
- Refactor run loop to use p-limit for controlled parallelism
- All runs within a scenario now execute concurrently (up to limit)
- Add workflow input and env variable (default: 3) for CI
- Add input validation for runs and parallelRuns (clamp to >= 1)
- Show spinner progress for both sequential and parallel modes
- Remove unused imports and variables
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve description
* Add DEFAULT_RUNS env variable to workflow
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add parallel skills, coordinated spinner, and allSettled
- Add --parallel-skills flag to evaluate multiple skills concurrently
- Use single coordinated spinner shared across all parallel skills
- Prefix output with [skill] or [skill/scenario] for disambiguation
(scenario omitted when eval has only one scenario)
- Use Promise.allSettled so one skill failure doesn't discard others
- Per-scenario runLimit so parallel-runs is scoped per scenario
- Add workflow input and env variable for parallel-skills (default: 3)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Temporarily use Token 2
* Allow skill-validator step to fail without blocking the job
Results are still uploaded and published to the dashboard even when
skills fail validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove test changes
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Only evaluate changed components in PRs
Update evaluation workflow to detect which components have changed
in pull requests and only run evaluations for those, similar to
how validate-skills and validate-agents workflows work.
- Add change detection using git diff for PRs
- Add has_components output to skip evaluation when nothing changed
- Add no-changes job to report when no components were modified
- Push/schedule/workflow_dispatch still evaluate all components
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update validate-skills.yml
* Add validate-agents.yml to workflow paths
* Update workflow to trigger on changes to itself
* Update skill-validator.yml
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Enhance README with new plugin and workflow sections
Added new sections for distribution templates, agentic workflows, and Copilot extension in the README.
* Add MIT License to the project
* Add LICENSE file
* Update README.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
- Rename 12 test directories to match their corresponding skill names
(e.g. bin-obj-clash -> check-bin-obj-clash, build-errors-cs -> common-build-errors)
- Allow empty scenarios in eval.yaml schema for placeholder tests
- Add skip logic in CLI for skills with empty scenarios
- Create placeholder eval.yaml (scenarios: []) for 8 skills without tests
- Restore --strict --require-evals in evaluation workflow
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Convert all 13 dotnet-msbuild test scenarios from the old
expected-output.md format to skill-validator eval.yaml format.
Each test has assertions and rubric criteria derived from the
original evaluation checklists. Includes a negative test for
domain-check (MSBuild skills should not activate for npm).
Also add eng/skill-validator/dist/ to .gitignore.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace hardcoded src/dotnet paths with a discover job that finds all
components under src/ with skills/ and tests/ directories. Each
component is evaluated in parallel via matrix strategy. Artifacts and
benchmark data are now generated per-component.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move skill-validator sources and tests from the external
adityamandaleeka/skill-validator repository into eng/skill-validator/.
Update evaluation.yml to build and run from the local copy instead of
checking out the external repo. Add skill-validator.yml CI pipeline
for building and testing the validator itself.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Viktor Hofer