mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-opus48
20 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 dependabot[bot]
|
7079f6c3ae |
Bump the github-actions-dependencies group with 3 updates (#526)
Bumps the github-actions-dependencies group with 3 updates: [actions/github-script](https://github.com/actions/github-script), [actions/cache](https://github.com/actions/cache) and [actions/upload-artifact](https://github.com/actions/upload-artifact). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/ed597411d8f924073f98dfc5c65a23a2325f34cd...3a2844b7e9c422d3c10d287c895573f7108da1b3) Updates `actions/cache` from 5.0.4 to 5.0.5 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/668228422ae6a00e4ad889ee87cd7109ec5666a7...27d5ce7f107fe9357f9df03efb73ab90386fccae) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/bbbca2ddaa5d8feaa63e36b76fdaad77386f024f...043fb46d1a93c77aae656e7c1c64a875d1fc6a0a) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/cache dependency-version: 5.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions-dependencies - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Viktor Hofer
|
7747fcb427 |
Move skill-validator docs under src/ and include in packages (#476)
* Move skill-validator docs under src/ and include in packages Move README.md to src/ and InvestigatingResults.md + OverfittingDetection.md to src/docs/ so they ship in the NuGet package and .tar.gz archives. - Include docs/*.md in NuGet pack and publish output via CopyToPublishDirectory - Add LICENSE to publish output (tar.gz only, not NuGet — uses PackageLicenseExpression) - Replace enumerated docOnlyFiles lists with a regex pattern in CI workflows - Deduplicate eval.yaml docs between CONTRIBUTING.md and the shipping README - Add test fixture files walkthrough to README (moved from CONTRIBUTING.md) - Add AGENTS.md guidance: src/ is the packaging boundary Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update link format in failure diagnosis message * Update AGENTS.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
604c46c8c6 | Fix eval investigation docs: download command, schema accuracy, example script (#449) | ||
|
dependabot[bot]
|
8e27c14ea4 |
Bump actions/download-artifact in the github-actions-dependencies group (#396)
Bumps the github-actions-dependencies group with 1 update: [actions/download-artifact](https://github.com/actions/download-artifact). Updates `actions/download-artifact` from 8.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
Copilot
|
0bb3606942 |
Fix release job skipped by transitive dependency chain from discover (#395)
* Initial plan * Fix release job being skipped due to tainted dependency chain from discover The release job was always skipped for schedule and workflow_dispatch events because the discover job (a transitive dependency via build) is skipped for non-PR events, which taints the dependency chain. Without always() in its if condition, GitHub skips release without evaluating the expression. Add always() so the if expression is always evaluated, and add an explicit needs.build.result == 'success' guard so the release only runs when the build actually passed. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Abhitej John
|
bb974f7360 |
Security Update (#331)
* Security Update Identified and remediated 15 vulnerabilities across the skill-validator runtime, dashboard, and CI/CD pipelines. Changes: - Runtime (AgentRunner.cs): path traversal prevention, deny-by-default permissions, env scrubbing, MCP server hardening - Regex (AssertionEvaluator.cs): ReDoS timeout - Logging (SkillDiscovery.cs): exception logging - Dashboard (dashboard.html, dashboard.js): SRI hash, XSS escaping - CI/CD (5 workflow files): SHA pinning, persist-credentials: false - Tests (RunnerTests.cs): 45+ new security test cases MCP Server Defense-in-Depth (5 layers): 1. Command allowlist: only dotnet, node, npx, python, python3, uvx 2. Path rejection: no /, \, or .. in command names 3. Args validation: per-runtime dangerous flag blocklist 4. Env sanitization: strips PATH, LD_PRELOAD, NODE_OPTIONS, DOTNET_STARTUP_HOOKS, etc. 5. Cwd dropped: custom working directories not passed to SDK Subprocess Isolation: - Sensitive env vars + prefix patterns scrubbed from child processes - 120-second timeout with Kill(entireProcessTree: true) - Deny-by-default permission model Supply Chain: - All GitHub Action references pinned to full commit SHAs - All checkout steps set persist-credentials: false - Chart.js CDN pinned to v4.4.7 with SRI integrity hash * Update .github/workflows/skill-validator.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update eng/skill-validator/src/Services/AgentRunner.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update eng/skill-validator/src/Services/AgentRunner.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests to use fullCommandText field name Update ExtractsCommandProperty and PrefersPathOverCommand tests to use the fullCommandText property instead of command, matching the updated permission request schema in AgentRunner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add 'dnx' to allowed MCP commands * Address PR review feedback: security hardening fixes - CheckPermission: normalize paths with TrimEndingDirectorySeparator to prevent double-separator false denials when dirs have trailing slash - SanitizeMcpArgs: reject prefix forms (-econsole.log) and --flag=value variants that bypass exact-match blocklist checks - Setup file path traversal: normalize canonicalWorkDir and canonicalSkillPath to avoid double-separator in StartsWith checks - IsAllowedMcpCommand: use GetFileName instead of GetFileNameWithoutExtension to block dotnet.bat/node.cmd; only strip .exe on Windows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * prune * fix * Fix concurrency setting * Add binlog generation and upload to skill-validator workflow Produce binary logs for build, test, and pack steps using /bl: flag. Upload binlogs as artifacts (with if: always()) for build diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve CheckPermission diagnostics and fix AOT build error Log allowed directories alongside denied paths to help diagnose permission mismatches. Replace JsonSerializer.Serialize with manual string building to fix IL3050 AOT compilation error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extract fileName from permission requests Some permission requests use 'fileName' instead of 'path'. Handle this property to avoid false denials for file-level operations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * nownar nu1903 temporarily * Log config dir in verbose mode alongside work dir Include skilled/baseline label for consistency with the work dir log. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add skilled/baseline label to permission denial logs Helps identify which run type triggered the denial when debugging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix empty runLabel brackets and add labels to judge permission checks Conditionally format the label suffix to avoid empty () when no runLabel is supplied. Pass 'judge' and 'pairwise-judge' labels from the respective judge call sites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Allow pairwise judge to read from skilled work directory The pairwise judge compares baseline and skilled runs in one session, but previously only allowed reads from the baseline work directory. Add additionalAllowedDirs parameter to CheckPermission and pass the skilled work directory through PairwiseJudgeOptions.SkilledWorkDir. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix tests to match allow-by-default for unrecognized permission requests CheckPermission was changed to allow requests with no extractable path/command (allow-by-default), but four tests still expected the old deny-by-default behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove NoWarn * Pin discover job checkout action to full commit SHA Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |
||
|
Jan Krivanek
|
4a4306f563 |
Disable scheduled workflows on forks (#318)
* Disable schedules on forks * Reflect PR comments |
||
|
Viktor Hofer
|
7be01578bd | Rename status job to skill-validator-status | ||
|
Viktor Hofer
|
8fecbba151 |
Rename status job to skill-validator-status (#322)
* Rename status job to skill-validator-status * Update .github/workflows/skill-validator.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
7bd004ee9f |
Enable skill-validator as a required status check (#321)
* Enable skill-validator as a required status check - Remove path filters from pull_request trigger - Add discover job to skip the expensive 5-platform build matrix when no skill-validator files changed - Add report-status job that posts commit status 'skill-validator': - No relevant changes → success - Build passed → success - Build failed → failure - Schedule and workflow_dispatch triggers unchanged (discover is skipped, build runs unconditionally) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Handle fork PRs with pull_request_target for status posting Fork PRs have restricted GITHUB_TOKEN and can't post commit statuses via pull_request trigger. Add pull_request_target trigger with a fork-status job that posts success for fork PRs (they should never change skill-validator). - Add pull_request_target trigger - Gate build to skip for pull_request_target events - Gate report-status to non-fork PRs only - Add fork-status job for fork PRs (always posts success) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Enable skill-validator as a required check - Remove path filters from pull_request trigger so the workflow runs on every PR - Add discover job to skip the expensive 5-platform build matrix when no skill-validator files changed - Add status job that can be required in the GitHub ruleset: succeeds when no relevant changes or when the build passes, fails otherwise - No commit status API calls or pull_request_target needed — just a plain job whose check run is required Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
750f56313b |
Fix fork PR evaluation concurrency cancelling runs on unrelated comments (#272)
* Fix fork PR evaluation concurrency cancelling runs on unrelated comments The concurrency group used the PR number alone, so any comment on a PR (not just /evaluate) would cancel an in-progress evaluation. Now only /evaluate comments share a group; other comments get a unique group keyed by comment ID so they never cancel real evaluation runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * remove push trigger from skill-validator.yml --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
05a3633f2a |
Fix create_release input check in skill-validator.yml (#261)
* Add workflow_dispatch trigger to skill-validator.yml Add a workflow_dispatch trigger with an optional create_release boolean input so the workflow can be run manually from the Actions tab. When create_release is true, the release job runs and retention is set to 90 days, matching the scheduled nightly behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use string comparison for inputs and guard release to main Compare inputs.create_release against 'true' (string) instead of true (boolean) since workflow_dispatch inputs are surfaced as strings. Also restrict the release job to refs/heads/main to prevent accidental overwrites of the nightly tag from non-main branches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix create_release input check to use truthiness workflow_dispatch inputs with type: boolean are actual booleans in the inputs context, so comparing against the string 'true' never matches. Use truthy evaluation (inputs.create_release) instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
83fb975ac7 |
Add workflow_dispatch trigger to skill-validator.yml (#259)
* Add workflow_dispatch trigger to skill-validator.yml Add a workflow_dispatch trigger with an optional create_release boolean input so the workflow can be run manually from the Actions tab. When create_release is true, the release job runs and retention is set to 90 days, matching the scheduled nightly behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use string comparison for inputs and guard release to main Compare inputs.create_release against 'true' (string) instead of true (boolean) since workflow_dispatch inputs are surfaced as strings. Also restrict the release job to refs/heads/main to prevent accidental overwrites of the nightly tag from non-main branches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
78f1ef2d53 |
skill-validator: improve UX for published tool (#223)
* skill-validator: improve UX for published tool - Make paths argument required (OneOrMore arity) so invoking without args shows full help text instead of a cryptic error - Add Installation section to README covering dotnet tool install from nightly release, platform-specific AOT packages, and pipeline artifacts - Update all usage examples to use the skillvalidator command - Update CI integration example to install from nightly release - Include README.md as NuGet package readme Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR feedback: clarify release wording and chmod instructions - Replace ambiguous 'GitHub pre-release artifacts' with a direct link to the skill-validator-nightly GitHub Release - Use 'chmod +x' instead of 'give +x permissions' - Capitalize NuGet consistently Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align ToolCommandName with AssemblyName (skill-validator) Fixes naming mismatch where dotnet tool install produced 'skillvalidator' but the AOT binary was 'skill-validator'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update README.md * Publish tar.gz archives in skill-validator workflow - Add tar.gz creation from dotnet pack publish output for each RID - Upgrade upload-artifact to v7 with archive: false (no double-zip) - Upgrade download-artifact to v8 for non-zipped artifact support - Include tar.gz files in nightly GitHub release Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update README * Update README --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
54e6fe8485 |
Enable NativeAOT publishing for SkillValidator (#221)
* Enable NativeAOT publishing for SkillValidator Migrate all serialization to source-generated, AOT-compatible patterns: - Add PublishAot to csproj and Vecc.YamlDotNet.Analyzers.StaticGenerator package - Create SkillValidatorJsonContext with source-generated serializers for all types - Create SkillValidatorYamlContext with static YAML deserialization for eval config and frontmatter parsing - Change AgentEvent.Data from Dictionary<string, object?> to Dictionary<string, JsonNode?> for AOT-safe serialization - Replace anonymous type in Reporter.cs with named ResultsOutput record - Fix JsonStringEnumConverter to generic AOT-safe JsonStringEnumConverter<T> - Move ConsolidateData to Models.cs for JSON context accessibility - Switch EvalSchema and SkillDiscovery to StaticDeserializerBuilder - Add RawFrontmatter model for strongly-typed YAML frontmatter parsing - Update all JsonSerializer call sites to use context-based overloads The published binary is a single 14.6 MB native executable. All 198 existing tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add NativeAOT publish matrix and daily scheduled builds to CI - Single build job with matrix for linux-x64, linux-arm64, win-x64, win-arm64, osx-arm64 - Each matrix entry: build, test, NativeAOT publish, pack NuGet, upload tarball - collect-packages job: builds RID-agnostic package, merges with per-RID packages into a single skill-validator-nupkgs artifact - Add daily schedule trigger (08:00 UTC) with 90-day retention - Keep existing path-based triggers for push/PR Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add NativeAOT publish matrix and daily scheduled builds to CI - Single build job with matrix for linux-x64, linux-arm64, win-x64, win-arm64, osx-arm64 - Each matrix entry: build, test, NativeAOT publish, pack NuGet, upload binary - Upload publish directory directly (Actions wraps in zip automatically) - collect-packages job: builds RID-agnostic package, merges with per-RID packages into a single skill-validator-nupkgs artifact - Add daily schedule trigger (08:00 UTC) with 90-day retention - Keep existing path-based triggers for push/PR Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix flaky DelayClampedToRemainingBudget test Increase totalTimeoutMs from 200ms to 2000ms. The tight budget was causing the CTS to fire before the retry attempt on slower CI runners (macOS arm64). The test still validates that the 60s base delay gets clamped — the 5s wall-time assertion is the real check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Cancel in-progress workflow runs on new push Add concurrency group so that pushing a new commit to a PR automatically cancels any still-running workflow for that ref. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix per-RID NuGet pack with --use-current-runtime The pack step was producing the same agnostic package for every RID. Adding --use-current-runtime ensures each matrix entry produces a RID-specific nupkg that gets collected into the combined artifact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant publish step; pack already produces native binary dotnet pack --use-current-runtime already runs publish internally, producing the native AOT binary in artifacts/publish/. The separate dotnet publish step was redundant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename nupkgs artifact to nuget-bundled Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove redundant PublishTrimmed; implied by PublishAot Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix flaky ExponentialBackoff_DelaysIncrease test Increase baseDelayMs from 50ms to 200ms so that OS scheduling jitter on slow CI runners (macOS arm64) doesn't dominate the measured gaps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove per-RID publish directory artifacts; nupkgs suffice NuGet packages are zip files containing the native binaries, so the separate skill-validator-<rid> publish folder uploads are redundant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove collect-packages job; produce agnostic nupkg from linux-x64 Pack the RID-agnostic package from the linux-x64 matrix entry instead of a separate downstream job, eliminating the bundle step entirely. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename artifact names from nuget-* to skill-validator-* Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename agnostic artifact to skill-validator Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify NuGet artifact uploads; remove rm -rf and collect-packages Use precise glob patterns with the stable package name to distinguish RID-specific from agnostic nupkgs. The agnostic upload uses a negation pattern to exclude the RID-specific package on linux-x64. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extract retention-days and agnostic RID into workflow env vars Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add nightly release job using gh release On scheduled runs, downloads all nupkg artifacts and publishes them as a rolling 'skill-validator-nightly' pre-release on the repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore PropertyNameCaseInsensitive for JSON deserialization The original MCPServerDef deserialization used PropertyNameCaseInsensitive which was dropped when switching to source-generated context. Add it back to the JsonSourceGenerationOptions to preserve flexible property casing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
228faf6991 |
Bump the github-actions-dependencies group across 1 directory with 5 updates (#194)
Bumps the github-actions-dependencies group with 5 updates in the / directory: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `4` | `6` | | [actions/github-script](https://github.com/actions/github-script) | `7` | `8` | | [actions/setup-dotnet](https://github.com/actions/setup-dotnet) | `4` | `5` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4` | `7` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `4` | `8` | Updates `actions/checkout` from 4 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4...v6) Updates `actions/github-script` from 7 to 8 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/v7...v8) Updates `actions/setup-dotnet` from 4 to 5 - [Release notes](https://github.com/actions/setup-dotnet/releases) - [Commits](https://github.com/actions/setup-dotnet/compare/v4...v5) Updates `actions/upload-artifact` from 4 to 7 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4...v7) Updates `actions/download-artifact` from 4 to 8 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/v4...v8) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/github-script dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/setup-dotnet dependency-version: '5' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/upload-artifact dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/download-artifact dependency-version: '8' dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Viktor Hofer
|
cfc22971e1 |
Port skill-validator from TypeScript to .NET C# (#106)
* Port skill-validator from TypeScript to .NET C# Replace the TypeScript skill-validator with a .NET 10 C# implementation. Uses System.CommandLine for CLI parsing, GitHub.Copilot.SDK for Copilot communication, YamlDotNet for YAML, and xUnit v3 with MTP v2 for testing. All 135 unit tests ported and passing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update CI workflows for .NET skill-validator Replace Node.js setup and npm commands with dotnet build/test/publish. Use global.json for SDK version instead of hardcoded dotnet-version. Publish self-contained binary for evaluate runners. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix permission denied for Copilot SDK native binary in CI Restore execute permissions on the native copilot binary after artifact download, which strips file permissions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * nit updates * Apply upstream changes: --verdict-warn-only, safeDirName, fix benches Port three origin/main commits to the C# skill-validator and workflow: - Add --verdict-warn-only flag replacing --strict; verdict failures (threshold, completion, activation) become warnings while execution errors and --require-evals missing_eval failures remain fatal. - Add FailureKind tracking to SkillVerdict and Comparator. - Add SafeDirName sanitizer for skill directory names in Reporter. - Add per-skill verdict.json generation in JSON reporter. - Update evaluation.yml: remove continue-on-error, use --verdict-warn-only, change default infra runs to 2, skip comment-on-pr on cancellation, use ResultsFile for benchmark data generation. - Update generate-benchmark-data.ps1 to read verdicts from results.json. - Update README with new CLI flags and CI example. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update CONTRIBUTING.md for .NET skill-validator Replace Node.js/npm prerequisites and commands with .NET 10 SDK equivalents (dotnet run). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update test commands in CONTRIBUTING.md to use .csproj file * Add RunArguments for default results-dir and verbose logging - Set RunArguments in SkillValidator.csproj using MSBuild NormalizeDirectory to pass --results-dir pointing to artifacts/TestResults/skill-validator/ - Log configured results dir at startup when --verbose is enabled Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix path * Fix AgentRunner: capture tool completion events and implement permission checking - Add ToolExecutionCompleteEvent handler to capture Success/Result data from tool completions, fixing timeline showing all tools as FAIL - Add ToolExecutionStartEvent arguments capture for timeline detail - Implement CheckPermission with path extraction from PermissionRequest ExtensionData, replacing the stub that always returned true - Update Judge and PairwiseJudge to pass PermissionRequest directly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix CheckPermission tests for cross-platform CI (Linux) Use Path.GetTempPath() and Path.Combine() instead of hardcoded Windows paths so tests pass on both Windows and Linux CI runners. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Capture skills & tokens usages * Fix thread safety, resource leaks, and null safety in skill-validator - Use ConcurrentBag and SemaphoreSlim with double-check locking for _workDirs and _sharedClient in AgentRunner (race conditions under parallel runs/scenarios) - Make Spinner IDisposable; protect _frame/_message/_active with lock and Interlocked; use 'using var' at call site - Use 'using var' for timeout Timers in Judge and PairwiseJudge to prevent leaks on exception paths - Replace perRunPairwise[0] fallback with FirstOrDefault() to avoid IndexOutOfRangeException when list is empty Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eng/skill-validator/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR review feedback - Add per-type required field validation in ParseAssertion (e.g. file_contains requires both path and value, output_matches requires pattern) - Fix JsonDocument leaks in LlmJson: use 'using' and Clone() on RootElement in both ParseLlmJson and TryParseJson - Pin Microsoft.Extensions.FileSystemGlobbing to 10.0.0 instead of floating 10.* for reproducible builds - Fix README Development section to reference SkillValidator.slnx instead of .sln Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
0b85b5c33d |
Only evaluate changed components in PRs (#66)
* Only evaluate changed components in PRs Update evaluation workflow to detect which components have changed in pull requests and only run evaluations for those, similar to how validate-skills and validate-agents workflows work. - Add change detection using git diff for PRs - Add has_components output to skip evaluation when nothing changed - Add no-changes job to report when no components were modified - Push/schedule/workflow_dispatch still evaluate all components Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update validate-skills.yml * Add validate-agents.yml to workflow paths * Update workflow to trigger on changes to itself * Update skill-validator.yml --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
ad23394552 |
Add discover+matrix to evaluation workflow for multi-component support
Replace hardcoded src/dotnet paths with a discover job that finds all components under src/ with skills/ and tests/ directories. Each component is evaluated in parallel via matrix strategy. Artifacts and benchmark data are now generated per-component. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
11bf689e82 |
Inline skill-validator into eng/skill-validator
Move skill-validator sources and tests from the external adityamandaleeka/skill-validator repository into eng/skill-validator/. Update evaluation.yml to build and run from the local copy instead of checking out the external repo. Add skill-validator.yml CI pipeline for building and testing the validator itself. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |