* Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop
WHY
Tools that surface skills (Copilot CLI, Claude Code, Codex, Cursor) read a
plugin's version directly from its checked-in manifest. With no versioning
discipline, a plugin's behavior can change while its advertised version stays
flat, so clients never learn to re-pull, and there is no human-readable signal
of what changed. We want correct, current versions in the repo with minimal
manual work and without bloating the marketplace clone.
WHAT
- Per-plugin semantic versioning via Nerdbank.GitVersioning (NBGV). Each plugin
owns a version.json whose pathFilters exclude the generated manifests and the
version.json itself, so version height tracks real content changes only.
- The computed version is materialized into the checked-in manifests
(plugin.json and .codex-plugin/plugin.json) so every consumer reads a current
value with no build step on their side.
- eng/version/Sync-PluginVersions.ps1 is the single workhorse. It resolves the
set of changed plugins from a git diff, computes each version with nbgv
(predicting the squash-merge height for PRs), and either reports or stamps.
AUTOMATIONS (two, low-touch by design)
- /version-bump: an admin/maintainer comments the command on a PR and the
affected plugins are stamped on the PR branch. Gated on collaborator
permission (admin/write/maintain); forks are rejected before any privileged
step. No other PRs are auto-modified.
- weekly-version-sync: a Monday backstop (and workflow_dispatch) that stamps any
drift on main, opens/updates a single bot PR, and explains the per-plugin
reason. This self-heals anything that merged without a bump.
We deliberately did NOT auto-edit contributor PRs or add a noisy advisory
comment bot; maintainers stay in control and the signal stays clean.
SECURITY (multi-model adversarial review: GPT-5.5 + Gemini 3.1 Pro)
- Supply chain (High, both models): dotnet tool restore would have honored a
nuget.config authored in the PR tree, letting an attacker remap the nbgv
package source to a malicious feed and run code in the privileged
contents:write context. Mitigated with a trusted eng/version/nuget.config
(clear + nuget.org-only + packageSourceMapping), overlaid from main and used
via --configfile so PR-supplied configs are ignored. No nuget.config is
tracked in the repo today, so this path was genuinely exploitable.
- TOCTOU (Medium): /version-bump now checks out the authorized head SHA rather
than the mutable branch name; a racing push fails non-fast-forward, which is
the safe outcome.
- Injection: Set-ManifestVersion uses a MatchEvaluator (not a replacement
string) so a "$"-bearing version cannot re-expand, plus a strict
major.minor.patch guard that throws on a malformed base, leaving manifests
untouched.
- A base-only version.json bump (0.1 -> 0.2) is correctly detected and stamped.
VERIFIED
End-to-end against a real NBGV git harness: content-scoped predict, base-only
bump -> x.y.0, docs-only -> [], weekly drift stamping, malformed-base guard,
and --configfile restore (exit 0). actionlint passes on both workflows.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Copilot review feedback
- Add missing plugins/dotnet-test-migration/version.json so it participates
in versioning (it was the only plugin without one; manifests are at 0.1.0).
- CONTRIBUTING: the two manifests are not byte-identical; say the version is
duplicated across two manifest files instead.
- weekly-version-sync: include version.json in commit attribution so a
base-only bump is explained rather than showing 'no attributable commits'.
- Get-NbgvInfo: capture nbgv stderr and include it in the thrown error so CI
failures are diagnosable, while keeping stdout clean for JSON parsing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove agentic workflows
The agentic workflows were useful when CCA didn't support loading skills. That's now solved and supported. The agentic workflows come with a couple of issues (limited chars) which make it hard to use. Remove them in favor of the plugin system.
* Remove Agentic Workflows section from README
Removed section about Agentic Workflows from README.
* Remove stale /agentic-workflows references from CONTRIBUTING.md, CODEOWNERS, and docs
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8f129e2f-156e-4ee9-9897-896cd1981513
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Move skill-validator docs under src/ and include in packages
Move README.md to src/ and InvestigatingResults.md + OverfittingDetection.md
to src/docs/ so they ship in the NuGet package and .tar.gz archives.
- Include docs/*.md in NuGet pack and publish output via CopyToPublishDirectory
- Add LICENSE to publish output (tar.gz only, not NuGet — uses PackageLicenseExpression)
- Replace enumerated docOnlyFiles lists with a regex pattern in CI workflows
- Deduplicate eval.yaml docs between CONTRIBUTING.md and the shipping README
- Add test fixture files walkthrough to README (moved from CONTRIBUTING.md)
- Add AGENTS.md guidance: src/ is the packaging boundary
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update link format in failure diagnosis message
* Update AGENTS.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add evaluation troubleshooting guide and link from PR comments
Add InvestigatingResults.md with:
- How to find and download evaluation artifacts from workflow runs
- Full results.json schema documentation
- Common failure patterns (timeouts, empty output, variance, rubric
issues, skill not activated) with diagnosis steps and fixes
- Example analysis script for AI agents to parse results
Link the doc from:
- Reporter.cs: show a link in the PR comment when any skill fails
- skill-validator README.md: Output section
- CONTRIBUTING.md: CI evaluation section
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update InvestigatingResults.md
* Address review feedback: fix hard-coded URLs and optional plugin run docs
- Reporter.cs: Use repo-relative link instead of hard-coded GitHub URL
- evaluation.yml: Use github.sha instead of hard-coded 'main' branch
- InvestigatingResults.md: Clarify skilledPlugin run is optional
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* THROWAWAY: add trailing newline to trigger eval workflow
This change is intentionally trivial and will be reverted before merge.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #441 review feedback (round 2)
- Reporter.cs: Remove leading / from relative link
- evaluation.yml: Use jq on results.json instead of grepping for emoji
- evaluation.yml: Use needs.gate.outputs.head_sha for revision-pinned URL
- InvestigatingResults.md: Mark pluginImprovementScore/pluginBreakdown as optional
- InvestigatingResults.md: Fix skill-not-activated symptoms to use
skillActivation fields instead of turnCount
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use specific skillActivation field names and fix timestamp description
- Use skillActivationIsolated/skillActivationPlugin instead of generic skillActivation
- Clarify timestamp is when results were written, not run start
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix timestamp field description (was lost in prior commit)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert "THROWAWAY: add trailing newline to trigger eval workflow"
This reverts commit 83bd7f7601.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Integrate reference scanner into skill-validator CheckCommand
- Port reference scanner logic from scan.ps1 to C# ReferenceScanner service
- Add --known-domains option to CheckCommand for reference scanning
- Move known-domains.txt and allowed-external-deps.txt to eng/
- Remove standalone scan.ps1 and reference-validation.yml workflow
- Reference scanning now runs as part of skill-check.yml via skill-validator
- Add 34 unit tests for ReferenceScanner (all 463 tests pass)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR feedback: repoRoot derivation, early validation, .agents scanning, SRI policy
- Derive repoRoot from --known-domains file path instead of cwd
- Fail early with clear error when known-domains file is missing
- Add .agents/**/*.md scanning to DiscoverFiles (regression fix)
- Fix SRI policy loophole: SRI suppresses SCRIPT-NO-SRI only, domain
and HTTPS checks still apply
- Use actual known-domains file path in error messages instead of
hardcoded eng/known-domains.txt
- Update doc comments to reflect full set of scanned paths
- Add tests for .agents discovery and SRI domain enforcement
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove repo-specific paths from ReferenceScanner
DiscoverFiles now only scans directories it is given -- no hardcoded
knowledge of .agents, agentic-workflows, eng, or README.md. The tool
is repo-agnostic; callers control what gets scanned via --plugin dirs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Drop repoRoot from ReferenceScanner API
ScanFile/ScanFiles no longer take a repoRoot parameter. Finding paths
use the file path as provided by the caller, keeping the scanner
completely decoupled from any repo layout assumptions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Prune stale domains and add json.schemastore.org to known-domains.txt
Removed 10 entries no longer referenced by any plugin content:
agentskills.io, visualstudio.com, mcr.microsoft.com, dotnet.github.io,
github.com/dotnet/docs, github.com/dotnet/dotnet-api-docs,
github.com/dotnet/skills, github.com/github/gh-aw,
github.com/modelcontextprotocol/csharp-sdk, developers.openai.com.
Added json.schemastore.org (referenced by dotnet-template-engine).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix case-insensitive regex, update docs, add tests
- Make UrlRegex and PipeToShellRegex case-insensitive to match the
original PowerShell behavior (HTTP://, CURL, Wget all now detected)
- Update DiscoverFiles doc comment to remove stale 'individual files' claim
- Narrow CONTRIBUTING.md scope description to match actual plugin-only scanning
- Add tests for mixed/upper-case URL and pipe-to-shell detection
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove IPv6 loopback support, fix pipe-to-shell allowlist case sensitivity
- Remove Ipv6LoopbackRegex and IPv6 test data (not needed)
- Add case-insensitive + culture-invariant matching for pipe-to-shell allowlist
- Add mixed-case allowlist test
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make evaluate a subcommand with tests-dir as required argument
Change EvaluateCommand from RootCommand to a regular Command('evaluate')
so it is a subcommand like check, consolidate, and rejudge. Convert
--tests-dir from an optional Option to a required positional Argument
placed after the skill paths.
Update .csproj RunArguments, evaluation-run workflow, README, and
CONTRIBUTING.md to reflect the new CLI syntax.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Default --parallel-skills/scenarios/runs to 3
Update the default from 1 to 3 for all three parallelism options and
remove the now-redundant overrides from RunArguments in the .csproj.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace RunArguments with launchSettings.json, publish in skill-check
Add Properties/launchSettings.json with default evaluate args for local
development. Remove RunArguments from .csproj. Update skill-check.yml to
publish the validator and invoke the binary directly instead of using
dotnet run.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Keep --tests-dir as a required option, not positional
A variadic paths argument greedily consumes all positional values,
making a trailing positional tests-dir impossible to parse. Keep
--tests-dir as a named required option which matches the previous UX.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move testsDirOpt after resultsDirOpt, drop evaluate from CONTRIBUTING examples
Reorder testsDirOpt declaration and registration to follow resultsDirOpt.
Remove the evaluate subcommand from CONTRIBUTING.md dotnet run examples
since the launch profile already provides it.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add external reference scanner for skill/agent content
- PowerShell scanner (eng/reference-scanner/scan.ps1) checks for:
- http:// URLs (blocked except localhost and XML namespace URIs)
- External <script> tags without SRI integrity attribute
- Pipe-to-shell patterns (except dotnet install scripts)
- URLs referencing domains not in .github/known-domains.txt
- CI workflow runs on PRs and pushes touching skill/agent paths
- CONTRIBUTING.md updated with external references policy
- README.md: fixed http:// to https:// for agentskills.io
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move known-domains.txt to eng/reference-scanner/, fix comments, add ollama.com
- Moved from .github/ to eng/reference-scanner/ (co-located with scanner)
- Updated header comments: errors not informational notices, removed misleading org-scoping language
- Added ollama.com (pre-existing reference in dotnet-ai skill)
- Updated all references in scan.ps1, CI workflow, CONTRIBUTING.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback
- Add ? and # as valid URL terminators in path-scoped domain matching
- Strip # from host extraction in Test-KnownDomain and Test-HttpNotHttps
- Make script/SRI regex patterns case-insensitive
- Emit error finding on file read failure instead of silent skip
- Fix help comment (two levels up, not three)
- Unify CONTRIBUTING.md error language (all checks are CI-blocking)
- Add eng/reference-scanner/** to workflow path triggers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add cdn.jsdelivr.net to known-domains (used by dashboard after #331 adds SRI)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove cdn.jsdelivr.net from known-domains
jsdelivr serves arbitrary npm packages -- keeping it off the allowlist
means any use requires both SRI and reviewer approval (two gates).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Skip domain check for scripts with SRI, fix single-line file handling
Scripts with integrity attributes are secured by hash -- the domain
is irrelevant. This means cdn.jsdelivr.net does not need to be in
known-domains.txt as long as SRI is present.
Also wrap Get-Content in @() to handle single-line files correctly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address round-2 review feedback
- Multi-line <script> tag detection: join file content so SRI attributes
spanning lines are correctly detected
- Safe relative path: use GetRelativePath() with fallback instead of
bare Substring that throws for out-of-repo files
- Remove dead notice plumbing: all findings are errors now
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Scan only changed files on PRs, full scan on push to main
PRs scan only changed .md/.html files so pre-existing issues on main
don't block unrelated PRs. Push to main runs the full -All scan.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace marketplace.json symlink with file copy
On Windows without developer mode, git checks out symlinks as plain text
files containing the target path. This caused Claude Code to fail parsing
.claude-plugin/marketplace.json since it contained the string
'../.github/plugin/marketplace.json' instead of valid JSON.
Replace the symlink with an actual copy of the file and update
CONTRIBUTING.md to instruct contributors to update both locations.
Fixes#300
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Reorganize plugins per agreed structure (#271)
Split the dotnet plugin into domain-specific plugins per discussion
in #271:
- dotnet: narrowed to common everyday C#/.NET coding tasks
- dotnet-diag: perf investigations, debugging, incident analysis
- dotnet-data: data access and Entity Framework
- dotnet-aspnet, dotnet-wpf, dotnet-winforms, dotnet-maui: scaffolded
empty to convey intent for future framework-specific skills
Moved skills, tests, and CODEOWNERS entries accordingly.
No file content was changed; all skill/test moves are pure renames.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve dotnet-diag and dotnet-winforms descriptions
- dotnet-diag: add '.NET' qualifier to clarify scope
- dotnet-winforms: use official 'Windows Forms (WinForms)' name
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add .NET qualifier to dotnet-data description
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove empty plugins; list them in CONTRIBUTING.md as future options
Remove the four empty placeholder plugins (dotnet-aspnet, dotnet-wpf,
dotnet-winforms, dotnet-maui) that had only plugin.json and no skills.
Add a 'Plugin organization' section to CONTRIBUTING.md that describes the
domain-specific plugin groupings introduced in this PR and lists the four
removed plugin names as reserved candidates for future contributions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Create dotnet-upgrade plugin with migration/upgrade skills
Move migrate-nullable-references, thread-abort-migration, and
dotnet-aot-compat from dotnet to the new dotnet-upgrade plugin.
Update CODEOWNERS paths (same owners), CONTRIBUTING.md plugin table,
and fix the thread-abort-migration SKILL.md link in CONTRIBUTING.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clarify that when-to-use guidance belongs in description frontmatter
The agent runtime reads the description field to decide whether to load a
skill. Moving when-to-use/not-use guidance into the description avoids
loading the full SKILL.md unnecessarily, reducing token load.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CONTRIBUTING.md frontmatter example and tip consistency
- Update description placeholder to include 'when not to use' guidance
- Remove brittle line-number anchors from SKILL.md link
- Change 'token load' to 'token usage' for clarity
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update contributing guidelines
Clarify evaluation criteria for third-party tools and add ownership requirement for skills.
* Update CONTRIBUTING.md to remove skill owner requirement
Removed requirement for every skill to have an associated owner.
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Tim Heuer <4821+timheuer@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Port skill-validator from TypeScript to .NET C#
Replace the TypeScript skill-validator with a .NET 10 C# implementation.
Uses System.CommandLine for CLI parsing, GitHub.Copilot.SDK for Copilot
communication, YamlDotNet for YAML, and xUnit v3 with MTP v2 for testing.
All 135 unit tests ported and passing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update CI workflows for .NET skill-validator
Replace Node.js setup and npm commands with dotnet build/test/publish.
Use global.json for SDK version instead of hardcoded dotnet-version.
Publish self-contained binary for evaluate runners.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix permission denied for Copilot SDK native binary in CI
Restore execute permissions on the native copilot binary after
artifact download, which strips file permissions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* nit updates
* Apply upstream changes: --verdict-warn-only, safeDirName, fix benches
Port three origin/main commits to the C# skill-validator and workflow:
- Add --verdict-warn-only flag replacing --strict; verdict failures
(threshold, completion, activation) become warnings while execution
errors and --require-evals missing_eval failures remain fatal.
- Add FailureKind tracking to SkillVerdict and Comparator.
- Add SafeDirName sanitizer for skill directory names in Reporter.
- Add per-skill verdict.json generation in JSON reporter.
- Update evaluation.yml: remove continue-on-error, use --verdict-warn-only,
change default infra runs to 2, skip comment-on-pr on cancellation,
use ResultsFile for benchmark data generation.
- Update generate-benchmark-data.ps1 to read verdicts from results.json.
- Update README with new CLI flags and CI example.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update CONTRIBUTING.md for .NET skill-validator
Replace Node.js/npm prerequisites and commands with .NET 10 SDK
equivalents (dotnet run).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update test commands in CONTRIBUTING.md to use .csproj file
* Add RunArguments for default results-dir and verbose logging
- Set RunArguments in SkillValidator.csproj using MSBuild NormalizeDirectory
to pass --results-dir pointing to artifacts/TestResults/skill-validator/
- Log configured results dir at startup when --verbose is enabled
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix path
* Fix AgentRunner: capture tool completion events and implement permission checking
- Add ToolExecutionCompleteEvent handler to capture Success/Result data
from tool completions, fixing timeline showing all tools as FAIL
- Add ToolExecutionStartEvent arguments capture for timeline detail
- Implement CheckPermission with path extraction from PermissionRequest
ExtensionData, replacing the stub that always returned true
- Update Judge and PairwiseJudge to pass PermissionRequest directly
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CheckPermission tests for cross-platform CI (Linux)
Use Path.GetTempPath() and Path.Combine() instead of hardcoded Windows
paths so tests pass on both Windows and Linux CI runners.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Capture skills & tokens usages
* Fix thread safety, resource leaks, and null safety in skill-validator
- Use ConcurrentBag and SemaphoreSlim with double-check locking for
_workDirs and _sharedClient in AgentRunner (race conditions under
parallel runs/scenarios)
- Make Spinner IDisposable; protect _frame/_message/_active with lock
and Interlocked; use 'using var' at call site
- Use 'using var' for timeout Timers in Judge and PairwiseJudge to
prevent leaks on exception paths
- Replace perRunPairwise[0] fallback with FirstOrDefault() to avoid
IndexOutOfRangeException when list is empty
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eng/skill-validator/README.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address PR review feedback
- Add per-type required field validation in ParseAssertion (e.g.
file_contains requires both path and value, output_matches requires
pattern)
- Fix JsonDocument leaks in LlmJson: use 'using' and Clone() on
RootElement in both ParseLlmJson and TryParseJson
- Pin Microsoft.Extensions.FileSystemGlobbing to 10.0.0 instead of
floating 10.* for reproducible builds
- Fix README Development section to reference SkillValidator.slnx
instead of .sln
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: update build-validator job and README for Node.js version requirement
- Update an existing comment if it exists with the total summary
- Build the skill-validator just once and upload
- Clean-up when passing CLI args to skill-validator
* Update .github/workflows/evaluation.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Prune dev dependencies before uploading validator artifact
Reduces artifact size by removing devDependencies (typescript, vitest, etc.)
from node_modules/ since the evaluate job only needs the built dist/.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Updates
* Consolidate step summary into comment-on-pr job
Move GITHUB_STEP_SUMMARY writing from per-matrix Generate Summary step
into the existing Consolidate summaries step in comment-on-pr job.
This produces a single unified summary instead of fragmented per-component
summaries. The job now runs for all event types, with the PR comment
step scoped to pull_request events only.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refactor reporters to use --results-dir for file output
- Remove --no-save-results option and saveRunResults function
- Reporters no longer accept inline paths (e.g., json:path)
- Add --results-dir option: when specified, file reporters (json, junit,
markdown) write to that directory; when omitted, they write to console
- Enhance markdown reporter with per-scenario judge reports including
rubric scores, reasoning, and agent output
- Update workflow, README, and CONTRIBUTING docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Default --results-dir to .skill-validator-results and enable json/markdown reporters by default
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix unresolved matrix expression in skipped evaluate job name
Replace if: has_entries with if: !cancelled() && !failure() so the job
proceeds with an empty matrix, creating zero instances instead of a
skipped job with unresolved expressions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update required nodejs version in CONTRIBUTING doc
* Address PR feedback: fix timeout message, improve reporter output handling
- Use effective timeout value in error message to avoid 'undefineds'
- Only create resultsDir when file-based reporters are selected
- Write results into timestamped subdirectory (yyyyMMdd-HHmmss) to avoid stale output
- Use scenario-level improvementScore for markdown verdict icons
- Update workflow to find summary.md in timestamped subdirs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update bin-obj clash skill
* Fix PR comment step: use gh api instead of gh pr comment
gh pr comment requires a git repo checkout, but comment-on-pr job
only downloads artifacts. Switch to gh api for creating comments.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Touch one more skill
* Address PR feedback: remove dead code, fix formatting, improve workflow
- Remove dead validation check in cli.ts (resultsDir always has default)
- Fix missing spaces in variable declarations in reporter.ts
- Quote command substitution in consolidation script
- Use explicit needs.build-validator.result == 'success' condition
- Strip duplicate headers when consolidating multi-component summaries
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace markdown concatenation with consolidate subcommand
Instead of fragile bash string manipulation to combine per-component
summary.md files (which caused duplicate headers, empty tables, and
repeated model lines), add a 'consolidate' subcommand that reads
results.json files and generates a single clean markdown summary.
The subcommand takes explicit file paths as arguments — the caller
controls exactly which files to merge. Runner-specific concerns
(comment markers, run URLs) stay in the workflow script.
- Extract generateMarkdownSummary() from reportMarkdown() in reporter.ts
- Add 'skill-validator consolidate' subcommand in cli.ts
- Update workflow to use the new subcommand
- Document consolidate in README.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restructure repo to use src/<component> layout
Adopt the same repository structure as dotnet/skills:
- Move skills/ and tests/ under src/dotnet/
- Add plugin.json for the csharp-scripts component
- Add AGENTS.md with plugin versioning instructions
- Update marketplace.json files to point to new paths
- Update CI workflows for new directory structure
- Move contributor guidance from README into CONTRIBUTING.md
- Streamline README to match dotnet/skills format
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add evaluation workflow and dashboard infrastructure
Add GitHub Actions workflow for skill evaluation with parallel test
execution, response grading, and summary generation. Include:
- CI workflow (.github/workflows/evaluation.yml) with matrix-based
parallel test runs and configurable parameters
- Evaluation scripts (eng/evaluation/) for invoking Copilot, parsing
stats, evaluating responses, and generating summaries
- Dashboard (eng/dashboard/) with HTML/JS visualization and benchmark
data generation script
- Updated CONTRIBUTING.md with evaluation documentation reference
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Abhitej John