* Port skill-validator from TypeScript to .NET C#
Replace the TypeScript skill-validator with a .NET 10 C# implementation.
Uses System.CommandLine for CLI parsing, GitHub.Copilot.SDK for Copilot
communication, YamlDotNet for YAML, and xUnit v3 with MTP v2 for testing.
All 135 unit tests ported and passing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update CI workflows for .NET skill-validator
Replace Node.js setup and npm commands with dotnet build/test/publish.
Use global.json for SDK version instead of hardcoded dotnet-version.
Publish self-contained binary for evaluate runners.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix permission denied for Copilot SDK native binary in CI
Restore execute permissions on the native copilot binary after
artifact download, which strips file permissions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* nit updates
* Apply upstream changes: --verdict-warn-only, safeDirName, fix benches
Port three origin/main commits to the C# skill-validator and workflow:
- Add --verdict-warn-only flag replacing --strict; verdict failures
(threshold, completion, activation) become warnings while execution
errors and --require-evals missing_eval failures remain fatal.
- Add FailureKind tracking to SkillVerdict and Comparator.
- Add SafeDirName sanitizer for skill directory names in Reporter.
- Add per-skill verdict.json generation in JSON reporter.
- Update evaluation.yml: remove continue-on-error, use --verdict-warn-only,
change default infra runs to 2, skip comment-on-pr on cancellation,
use ResultsFile for benchmark data generation.
- Update generate-benchmark-data.ps1 to read verdicts from results.json.
- Update README with new CLI flags and CI example.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update CONTRIBUTING.md for .NET skill-validator
Replace Node.js/npm prerequisites and commands with .NET 10 SDK
equivalents (dotnet run).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update test commands in CONTRIBUTING.md to use .csproj file
* Add RunArguments for default results-dir and verbose logging
- Set RunArguments in SkillValidator.csproj using MSBuild NormalizeDirectory
to pass --results-dir pointing to artifacts/TestResults/skill-validator/
- Log configured results dir at startup when --verbose is enabled
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix path
* Fix AgentRunner: capture tool completion events and implement permission checking
- Add ToolExecutionCompleteEvent handler to capture Success/Result data
from tool completions, fixing timeline showing all tools as FAIL
- Add ToolExecutionStartEvent arguments capture for timeline detail
- Implement CheckPermission with path extraction from PermissionRequest
ExtensionData, replacing the stub that always returned true
- Update Judge and PairwiseJudge to pass PermissionRequest directly
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CheckPermission tests for cross-platform CI (Linux)
Use Path.GetTempPath() and Path.Combine() instead of hardcoded Windows
paths so tests pass on both Windows and Linux CI runners.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Capture skills & tokens usages
* Fix thread safety, resource leaks, and null safety in skill-validator
- Use ConcurrentBag and SemaphoreSlim with double-check locking for
_workDirs and _sharedClient in AgentRunner (race conditions under
parallel runs/scenarios)
- Make Spinner IDisposable; protect _frame/_message/_active with lock
and Interlocked; use 'using var' at call site
- Use 'using var' for timeout Timers in Judge and PairwiseJudge to
prevent leaks on exception paths
- Replace perRunPairwise[0] fallback with FirstOrDefault() to avoid
IndexOutOfRangeException when list is empty
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eng/skill-validator/README.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address PR review feedback
- Add per-type required field validation in ParseAssertion (e.g.
file_contains requires both path and value, output_matches requires
pattern)
- Fix JsonDocument leaks in LlmJson: use 'using' and Clone() on
RootElement in both ParseLlmJson and TryParseJson
- Pin Microsoft.Extensions.FileSystemGlobbing to 10.0.0 instead of
floating 10.* for reproducible builds
- Fix README Development section to reference SkillValidator.slnx
instead of .sln
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Jan Krivanek