* Add external reference scanner for skill/agent content
- PowerShell scanner (eng/reference-scanner/scan.ps1) checks for:
- http:// URLs (blocked except localhost and XML namespace URIs)
- External <script> tags without SRI integrity attribute
- Pipe-to-shell patterns (except dotnet install scripts)
- URLs referencing domains not in .github/known-domains.txt
- CI workflow runs on PRs and pushes touching skill/agent paths
- CONTRIBUTING.md updated with external references policy
- README.md: fixed http:// to https:// for agentskills.io
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move known-domains.txt to eng/reference-scanner/, fix comments, add ollama.com
- Moved from .github/ to eng/reference-scanner/ (co-located with scanner)
- Updated header comments: errors not informational notices, removed misleading org-scoping language
- Added ollama.com (pre-existing reference in dotnet-ai skill)
- Updated all references in scan.ps1, CI workflow, CONTRIBUTING.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback
- Add ? and # as valid URL terminators in path-scoped domain matching
- Strip # from host extraction in Test-KnownDomain and Test-HttpNotHttps
- Make script/SRI regex patterns case-insensitive
- Emit error finding on file read failure instead of silent skip
- Fix help comment (two levels up, not three)
- Unify CONTRIBUTING.md error language (all checks are CI-blocking)
- Add eng/reference-scanner/** to workflow path triggers
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add cdn.jsdelivr.net to known-domains (used by dashboard after #331 adds SRI)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove cdn.jsdelivr.net from known-domains
jsdelivr serves arbitrary npm packages -- keeping it off the allowlist
means any use requires both SRI and reviewer approval (two gates).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Skip domain check for scripts with SRI, fix single-line file handling
Scripts with integrity attributes are secured by hash -- the domain
is irrelevant. This means cdn.jsdelivr.net does not need to be in
known-domains.txt as long as SRI is present.
Also wrap Get-Content in @() to handle single-line files correctly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address round-2 review feedback
- Multi-line <script> tag detection: join file content so SRI attributes
spanning lines are correctly detected
- Safe relative path: use GetRelativePath() with fallback instead of
bare Substring that throws for out-of-repo files
- Remove dead notice plumbing: all findings are errors now
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Scan only changed files on PRs, full scan on push to main
PRs scan only changed .md/.html files so pre-existing issues on main
don't block unrelated PRs. Push to main runs the full -All scan.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace marketplace.json symlink with file copy
On Windows without developer mode, git checks out symlinks as plain text
files containing the target path. This caused Claude Code to fail parsing
.claude-plugin/marketplace.json since it contained the string
'../.github/plugin/marketplace.json' instead of valid JSON.
Replace the symlink with an actual copy of the file and update
CONTRIBUTING.md to instruct contributors to update both locations.
Fixes#300
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Reorganize plugins per agreed structure (#271)
Split the dotnet plugin into domain-specific plugins per discussion
in #271:
- dotnet: narrowed to common everyday C#/.NET coding tasks
- dotnet-diag: perf investigations, debugging, incident analysis
- dotnet-data: data access and Entity Framework
- dotnet-aspnet, dotnet-wpf, dotnet-winforms, dotnet-maui: scaffolded
empty to convey intent for future framework-specific skills
Moved skills, tests, and CODEOWNERS entries accordingly.
No file content was changed; all skill/test moves are pure renames.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve dotnet-diag and dotnet-winforms descriptions
- dotnet-diag: add '.NET' qualifier to clarify scope
- dotnet-winforms: use official 'Windows Forms (WinForms)' name
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add .NET qualifier to dotnet-data description
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove empty plugins; list them in CONTRIBUTING.md as future options
Remove the four empty placeholder plugins (dotnet-aspnet, dotnet-wpf,
dotnet-winforms, dotnet-maui) that had only plugin.json and no skills.
Add a 'Plugin organization' section to CONTRIBUTING.md that describes the
domain-specific plugin groupings introduced in this PR and lists the four
removed plugin names as reserved candidates for future contributions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Create dotnet-upgrade plugin with migration/upgrade skills
Move migrate-nullable-references, thread-abort-migration, and
dotnet-aot-compat from dotnet to the new dotnet-upgrade plugin.
Update CODEOWNERS paths (same owners), CONTRIBUTING.md plugin table,
and fix the thread-abort-migration SKILL.md link in CONTRIBUTING.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clarify that when-to-use guidance belongs in description frontmatter
The agent runtime reads the description field to decide whether to load a
skill. Moving when-to-use/not-use guidance into the description avoids
loading the full SKILL.md unnecessarily, reducing token load.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CONTRIBUTING.md frontmatter example and tip consistency
- Update description placeholder to include 'when not to use' guidance
- Remove brittle line-number anchors from SKILL.md link
- Change 'token load' to 'token usage' for clarity
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update contributing guidelines
Clarify evaluation criteria for third-party tools and add ownership requirement for skills.
* Update CONTRIBUTING.md to remove skill owner requirement
Removed requirement for every skill to have an associated owner.
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Tim Heuer <4821+timheuer@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Port skill-validator from TypeScript to .NET C#
Replace the TypeScript skill-validator with a .NET 10 C# implementation.
Uses System.CommandLine for CLI parsing, GitHub.Copilot.SDK for Copilot
communication, YamlDotNet for YAML, and xUnit v3 with MTP v2 for testing.
All 135 unit tests ported and passing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update CI workflows for .NET skill-validator
Replace Node.js setup and npm commands with dotnet build/test/publish.
Use global.json for SDK version instead of hardcoded dotnet-version.
Publish self-contained binary for evaluate runners.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix permission denied for Copilot SDK native binary in CI
Restore execute permissions on the native copilot binary after
artifact download, which strips file permissions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* nit updates
* Apply upstream changes: --verdict-warn-only, safeDirName, fix benches
Port three origin/main commits to the C# skill-validator and workflow:
- Add --verdict-warn-only flag replacing --strict; verdict failures
(threshold, completion, activation) become warnings while execution
errors and --require-evals missing_eval failures remain fatal.
- Add FailureKind tracking to SkillVerdict and Comparator.
- Add SafeDirName sanitizer for skill directory names in Reporter.
- Add per-skill verdict.json generation in JSON reporter.
- Update evaluation.yml: remove continue-on-error, use --verdict-warn-only,
change default infra runs to 2, skip comment-on-pr on cancellation,
use ResultsFile for benchmark data generation.
- Update generate-benchmark-data.ps1 to read verdicts from results.json.
- Update README with new CLI flags and CI example.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update CONTRIBUTING.md for .NET skill-validator
Replace Node.js/npm prerequisites and commands with .NET 10 SDK
equivalents (dotnet run).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update test commands in CONTRIBUTING.md to use .csproj file
* Add RunArguments for default results-dir and verbose logging
- Set RunArguments in SkillValidator.csproj using MSBuild NormalizeDirectory
to pass --results-dir pointing to artifacts/TestResults/skill-validator/
- Log configured results dir at startup when --verbose is enabled
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix path
* Fix AgentRunner: capture tool completion events and implement permission checking
- Add ToolExecutionCompleteEvent handler to capture Success/Result data
from tool completions, fixing timeline showing all tools as FAIL
- Add ToolExecutionStartEvent arguments capture for timeline detail
- Implement CheckPermission with path extraction from PermissionRequest
ExtensionData, replacing the stub that always returned true
- Update Judge and PairwiseJudge to pass PermissionRequest directly
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix CheckPermission tests for cross-platform CI (Linux)
Use Path.GetTempPath() and Path.Combine() instead of hardcoded Windows
paths so tests pass on both Windows and Linux CI runners.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Capture skills & tokens usages
* Fix thread safety, resource leaks, and null safety in skill-validator
- Use ConcurrentBag and SemaphoreSlim with double-check locking for
_workDirs and _sharedClient in AgentRunner (race conditions under
parallel runs/scenarios)
- Make Spinner IDisposable; protect _frame/_message/_active with lock
and Interlocked; use 'using var' at call site
- Use 'using var' for timeout Timers in Judge and PairwiseJudge to
prevent leaks on exception paths
- Replace perRunPairwise[0] fallback with FirstOrDefault() to avoid
IndexOutOfRangeException when list is empty
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eng/skill-validator/README.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address PR review feedback
- Add per-type required field validation in ParseAssertion (e.g.
file_contains requires both path and value, output_matches requires
pattern)
- Fix JsonDocument leaks in LlmJson: use 'using' and Clone() on
RootElement in both ParseLlmJson and TryParseJson
- Pin Microsoft.Extensions.FileSystemGlobbing to 10.0.0 instead of
floating 10.* for reproducible builds
- Fix README Development section to reference SkillValidator.slnx
instead of .sln
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat: update build-validator job and README for Node.js version requirement
- Update an existing comment if it exists with the total summary
- Build the skill-validator just once and upload
- Clean-up when passing CLI args to skill-validator
* Update .github/workflows/evaluation.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Prune dev dependencies before uploading validator artifact
Reduces artifact size by removing devDependencies (typescript, vitest, etc.)
from node_modules/ since the evaluate job only needs the built dist/.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Updates
* Consolidate step summary into comment-on-pr job
Move GITHUB_STEP_SUMMARY writing from per-matrix Generate Summary step
into the existing Consolidate summaries step in comment-on-pr job.
This produces a single unified summary instead of fragmented per-component
summaries. The job now runs for all event types, with the PR comment
step scoped to pull_request events only.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refactor reporters to use --results-dir for file output
- Remove --no-save-results option and saveRunResults function
- Reporters no longer accept inline paths (e.g., json:path)
- Add --results-dir option: when specified, file reporters (json, junit,
markdown) write to that directory; when omitted, they write to console
- Enhance markdown reporter with per-scenario judge reports including
rubric scores, reasoning, and agent output
- Update workflow, README, and CONTRIBUTING docs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Default --results-dir to .skill-validator-results and enable json/markdown reporters by default
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix unresolved matrix expression in skipped evaluate job name
Replace if: has_entries with if: !cancelled() && !failure() so the job
proceeds with an empty matrix, creating zero instances instead of a
skipped job with unresolved expressions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update required nodejs version in CONTRIBUTING doc
* Address PR feedback: fix timeout message, improve reporter output handling
- Use effective timeout value in error message to avoid 'undefineds'
- Only create resultsDir when file-based reporters are selected
- Write results into timestamped subdirectory (yyyyMMdd-HHmmss) to avoid stale output
- Use scenario-level improvementScore for markdown verdict icons
- Update workflow to find summary.md in timestamped subdirs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update bin-obj clash skill
* Fix PR comment step: use gh api instead of gh pr comment
gh pr comment requires a git repo checkout, but comment-on-pr job
only downloads artifacts. Switch to gh api for creating comments.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Touch one more skill
* Address PR feedback: remove dead code, fix formatting, improve workflow
- Remove dead validation check in cli.ts (resultsDir always has default)
- Fix missing spaces in variable declarations in reporter.ts
- Quote command substitution in consolidation script
- Use explicit needs.build-validator.result == 'success' condition
- Strip duplicate headers when consolidating multi-component summaries
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace markdown concatenation with consolidate subcommand
Instead of fragile bash string manipulation to combine per-component
summary.md files (which caused duplicate headers, empty tables, and
repeated model lines), add a 'consolidate' subcommand that reads
results.json files and generates a single clean markdown summary.
The subcommand takes explicit file paths as arguments — the caller
controls exactly which files to merge. Runner-specific concerns
(comment markers, run URLs) stay in the workflow script.
- Extract generateMarkdownSummary() from reportMarkdown() in reporter.ts
- Add 'skill-validator consolidate' subcommand in cli.ts
- Update workflow to use the new subcommand
- Document consolidate in README.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restructure repo to use src/<component> layout
Adopt the same repository structure as dotnet/skills:
- Move skills/ and tests/ under src/dotnet/
- Add plugin.json for the csharp-scripts component
- Add AGENTS.md with plugin versioning instructions
- Update marketplace.json files to point to new paths
- Update CI workflows for new directory structure
- Move contributor guidance from README into CONTRIBUTING.md
- Streamline README to match dotnet/skills format
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add evaluation workflow and dashboard infrastructure
Add GitHub Actions workflow for skill evaluation with parallel test
execution, response grading, and summary generation. Include:
- CI workflow (.github/workflows/evaluation.yml) with matrix-based
parallel test runs and configurable parameters
- Evaluation scripts (eng/evaluation/) for invoking Copilot, parsing
stats, evaluating responses, and generating summaries
- Dashboard (eng/dashboard/) with HTML/JS visualization and benchmark
data generation script
- Updated CONTRIBUTING.md with evaluation documentation reference
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Dan Moseley