* Add tests for binlog-failure-analysis skill
* Add support for using MCP within eval tests
* Update the skill and testcase - to better target the usage
* Change maxLevels default value from 4 to 2
* Fix extractJson picking up non-JSON brace groups (e.g., C# code)
extractJson now validates each brace-matched candidate with JSON.parse
before returning it. If the candidate isn't valid JSON (even after
escape sanitization), it skips to the next brace group. This prevents
C# code snippets or other non-JSON content from being mistakenly
extracted as the judge's JSON response.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add test for skipping brace groups with invalid escapes
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix JSON parse error from invalid LLM escape sequences in pairwise judge
Wrap JSON.parse in try/catch and retry after stripping invalid escape
sequences (e.g., \' or \a) that LLMs sometimes produce in reasoning text.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR feedback: improve error context and add parser tests
- Preserve original error and direction in all failure paths
- Only attempt escape sanitization when invalid escapes are detected
- Include JSON snippet in error messages for debugging
- Export parsePairwiseResponse for direct testing
- Add 7 tests covering valid JSON, invalid escapes, malformed JSON,
missing JSON, and direction reversal
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Extract shared LLM JSON parsing into llm-json.ts
- Create llm-json.ts with extractJson() and parseLlmJson() helpers
- Refactor judge.ts and pairwise-judge.ts to use shared helpers
- Remove duplicated extractOutermostJson and escape-sanitization logic
- Add 11 tests for the shared module (extractJson + parseLlmJson)
- Add 8 tests for judge.ts parseJudgeResponse
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Rename 12 test directories to match their corresponding skill names
(e.g. bin-obj-clash -> check-bin-obj-clash, build-errors-cs -> common-build-errors)
- Allow empty scenarios in eval.yaml schema for placeholder tests
- Add skip logic in CLI for skills with empty scenarios
- Create placeholder eval.yaml (scenarios: []) for 8 skills without tests
- Restore --strict --require-evals in evaluation workflow
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move skill-validator sources and tests from the external
adityamandaleeka/skill-validator repository into eng/skill-validator/.
Update evaluation.yml to build and run from the local copy instead of
checking out the external repo. Add skill-validator.yml CI pipeline
for building and testing the validator itself.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jan Krivanek