* Add agent and plugin spec conformance validation
Extend the validation pipeline to check .agent.md files and plugin.json
files alongside skills. Previously only SKILL.md files were validated.
Agent validation (AgentProfiler):
- Frontmatter required (error)
- Name and description required (error)
- Name format: same rules as skills (lowercase alphanum + hyphens, 64 max)
- Name must match filename minus .agent.md suffix
- Description max 1024 chars
- Body max 500 lines
Plugin validation (PluginValidator):
- Name required, must match directory name
- Version and description required
- Description max 1024 chars
- Skills path required and must exist
- Agents path warned if specified but missing
Discovery (SkillDiscovery):
- DiscoverAgents() walks up from skill paths to plugin roots,
scans agents/ directories for .agent.md files
- DiscoverPlugins() finds plugin.json from skill paths
Integration (ValidateCommand):
- Runs after skill discovery, before evaluation
- Errors block with exit code 1 (same as spec_conformance_failure)
All 4 existing agents and 2 plugins pass cleanly.
25 new tests (13 agent, 12 plugin).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Code review fixes: spec comments, ValidateNameFormat, kind-specific messages
- Refactor ValidateName into ValidateNameFormat (format-only) + ValidateName
(format + directory match) so agent/plugin validation doesn't abuse the
skill directory-match check by passing name twice.
- Warning messages now say 'Agent name' or 'Plugin name' instead of
'Skill name' when called from AgentProfiler/PluginValidator.
- Add spec URL comments to AgentProfiler and PluginValidator referencing
agentskills.io/specification, VS Code agent-plugins docs, and Claude Code
plugins-reference.
- Add missing agent name edge-case tests: leading/trailing hyphen,
consecutive hyphens, and message-prefix assertions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove unused agents path from plugins that have no agents
The dotnet, dotnet-data, and dotnet-upgrade plugins declared an agents
path but had no agents/ directory, producing validation warnings. Remove
the field; it can be added back when agents are actually created.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Promote agent/plugin name validation from warnings to errors
The agentskills.io spec uses 'Must' for all name constraints (length,
character set, no leading/trailing hyphens, no consecutive hyphens).
For a publishing validator these should block, not just warn. Agent
filename mismatch is also promoted since the IDE won't discover the
agent if the name doesn't match.
Skill name validation stays as warnings (lenient loading, not part
of this PR).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback
- Fix FrontmatterRegex: remove RegexOptions.Multiline so a horizontal
rule (---) in the body is not mistaken for frontmatter. Detection and
stripping now both anchor to start of string.
- Use filename as fallback identifier in AgentProfile when agent.Name
is empty, so error output shows [agent:foo.agent.md] not [agent:].
- Add TryGetSafeSubdirectory to PluginValidator: rejects absolute paths
and parent-directory traversal in skills/agents path fields. Used by
both PluginValidator and SkillDiscovery.DiscoverAgents.
- Use DirectoryName as fallback for plugin result name when Name is
empty, so error output shows [plugin:my-dir] not [plugin:].
- Add tests for path traversal rejection and name fallback behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Detect orphaned test directories with no matching plugin/skill
Add FindOrphanedTestDirectories to SkillDiscovery: checks that every
directory under tests/{plugin}/{skill}/ has a corresponding directory
under plugins/{plugin}/skills/{skill}/. Plugin-level orphans (tests/
directory with no matching plugins/ directory) are also flagged.
Integrated into ValidateCommand after plugin/agent validation so it
runs on every validation pass. Errors are blocking (exit 1).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #279 review feedback
- Use case-sensitive (Ordinal) comparison for agent filename matching
since agent names are required to be lowercase by spec.
- Replace OrdinalIgnoreCase path-under-root check in TryGetSafeSubdirectory
with Path.GetRelativePath + '..' prefix check for correct behavior on
case-sensitive filesystems.
- Let ParsePluginJson throw JsonException on malformed JSON instead of
silently returning null. ValidateCommand catches it as a blocking error.
- Add test for malformed plugin.json.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Increase eval timeouts for dump-collect and dotnet-trace-collect to 120s
The with-skill runs for these skills were hitting the 60s scenario timeout
in CI, causing the agent to be killed before producing output. Other skills
in the same plugin already use 120s. The 30s decline/negative-test scenario
is left unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix fixture paths in clr-activation-debugging and analyzing-dotnet-performance evals
The source paths referenced tests/dotnet/ but the fixtures live under
tests/dotnet-diag/. This caused file-not-found exceptions at runtime,
which set hasRejections and caused unconditional exit 1 (bypassing
--verdict-warn-only).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Surface execution errors in validation summary
Previously, skills that threw exceptions (e.g. missing fixture files)
were silently excluded from the verdict count. The summary showed
'2/4 skills passed' with no indication that 2 other skills crashed
entirely. Now the summary shows '2/6 skills passed (2 rejected due
to execution errors)' and lists the specific error messages near
the summary rather than buried in the middle of the log.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reduce parallelism for infra-change evaluation runs
When infrastructure changes trigger evaluation of all plugins
simultaneously, the high default parallelism (5x5x5) causes API
rate limits (429s) and scenario timeouts from contention. Reduce
to 2x3x3 for infra-change runs only; regular single-plugin PR
evaluations keep the default 5x5x5.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Promote skill spec conformance checks from warnings to errors
The agentskills.io spec (https://agentskills.io/specification) uses
'Must' for all of these constraints:
- name: 'Must be 1-64 characters'
- name: 'May only contain lowercase alphanumeric and hyphens'
- name: 'Must not start or end with -'
- name: 'Must not contain consecutive hyphens'
- name: 'Must match the parent directory name'
- description: 'Must be 1-1024 characters' (non-empty)
- compatibility: 'Must be 1-500 characters if provided'
These were previously warnings following the lenient-loading guidance
from the client implementation guide, but that guidance is for client
loaders, not publishing validators. For a CI tool that gates what gets
published, spec violations should be blocking errors.
All 25 existing skills already comply — no changes needed to skills.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback: empty compatibility, empty name, test
- Error when compatibility field is provided but empty/whitespace,
matching spec requirement of 1-500 characters when provided.
- Add explicit check for empty/whitespace name with targeted error
message instead of falling through to regex mismatch.
- Add test for empty compatibility string rejection.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Enforce description size limits (per-skill 1024, per-plugin 15K)
Add infrastructure to reject skills with oversized descriptions, per issue #222:
- SkillProfiler: check description length, flag DescriptionTooLong
- ValidateCommand: block oversized descriptions (survives --verdict-warn-only)
- New eng/validate-descriptions.ps1: validates per-skill and per-plugin aggregate limits
- New CI workflow: runs validation on PRs touching plugins/
- build.ps1: adds size checks to local build validation
- Unit tests for all new SkillProfiler behavior
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update --verdict-warn-only help text to mention description_too_long
Address review feedback: the option description now lists description_too_long
alongside execution errors and --require-evals as conditions that remain
fatal even when --verdict-warn-only is enabled.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add agentskills.io spec conformance checks to SkillProfiler
Validate name rules (max 64 chars, lowercase+hyphens, no edge/consecutive
hyphens, must match directory), compatibility field max 500 chars, body
max 500 lines, and file reference depth.
Scan of existing skills found 2 violations:
- dotnet-msbuild/msbuild-antipatterns: 582 lines (max 500)
- dotnet-msbuild/directory-build-organization: cross-skill ref 3 levels deep
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback on spec conformance checks
- Fix MakeSkill SkillMdPath to derive from resolved path (SkillProfileTests.cs)
- Fix RepoRoot in build.ps1 to be actual repo root (was agentic-workflows/)
- Fix body line count off-by-one: trim trailing newlines before splitting
- Fix file ref depth: normalize paths, handle .. traversals, strip #anchor,
compute depth from directory segments only
- Add tests for parent traversal, anchor fragments, ./ normalization,
and trailing newline body count
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Consolidate description validation into skill-validator C#
Move the per-plugin aggregate description limit (15K chars) from the
standalone PowerShell script into ValidateCommand, where it runs after
skill discovery. This eliminates the separate validate-descriptions.ps1,
its CI workflow, its PowerShell tests, and the delegation from build.ps1.
Also rename FailureKind from "description_too_long" to the generic
"spec_conformance_failure" so future spec checks reuse the same kind.
Deleted:
- eng/validate-descriptions.ps1
- eng/tests/validate-descriptions.tests.ps1
- .github/workflows/validate-descriptions.yml
Added:
- ValidateCommand.CheckAggregateDescriptionLimits()
- ValidateCommand.DerivePluginName()
- 6 C# tests for aggregate limit + DerivePluginName
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Upgrade body-lines and file-ref-depth from warnings to blocking errors
Now that #247 has merged and fixed the two existing violations
(msbuild-antipatterns 582 lines, directory-build-organization cross-ref),
these spec conformance checks can be blocking.
Also replaces the DescriptionTooLong bool with a general Errors list on
SkillProfile, so all spec conformance violations (description length,
body lines, file ref depth) flow through the same mechanism.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert unrelated lock file changes
The lock file diffs were side effects of running build.ps1 to verify
the validation changes — they picked up upstream content changes
unrelated to this PR.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reorganize plugins per agreed structure (#271)
Split the dotnet plugin into domain-specific plugins per discussion
in #271:
- dotnet: narrowed to common everyday C#/.NET coding tasks
- dotnet-diag: perf investigations, debugging, incident analysis
- dotnet-data: data access and Entity Framework
- dotnet-aspnet, dotnet-wpf, dotnet-winforms, dotnet-maui: scaffolded
empty to convey intent for future framework-specific skills
Moved skills, tests, and CODEOWNERS entries accordingly.
No file content was changed; all skill/test moves are pure renames.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve dotnet-diag and dotnet-winforms descriptions
- dotnet-diag: add '.NET' qualifier to clarify scope
- dotnet-winforms: use official 'Windows Forms (WinForms)' name
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add .NET qualifier to dotnet-data description
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove empty plugins; list them in CONTRIBUTING.md as future options
Remove the four empty placeholder plugins (dotnet-aspnet, dotnet-wpf,
dotnet-winforms, dotnet-maui) that had only plugin.json and no skills.
Add a 'Plugin organization' section to CONTRIBUTING.md that describes the
domain-specific plugin groupings introduced in this PR and lists the four
removed plugin names as reserved candidates for future contributions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Create dotnet-upgrade plugin with migration/upgrade skills
Move migrate-nullable-references, thread-abort-migration, and
dotnet-aot-compat from dotnet to the new dotnet-upgrade plugin.
Update CODEOWNERS paths (same owners), CONTRIBUTING.md plugin table,
and fix the thread-abort-migration SKILL.md link in CONTRIBUTING.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix fork PR evaluation concurrency cancelling runs on unrelated comments
The concurrency group used the PR number alone, so any comment on a PR
(not just /evaluate) would cancel an in-progress evaluation. Now only
/evaluate comments share a group; other comments get a unique group
keyed by comment ID so they never cancel real evaluation runs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* remove push trigger from skill-validator.yml
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove eyes reaction from trigger comment when workflow finishes
- Post a status comment on completion (success, failure, or skipped)
- On success, link to the evaluation summary table comment
- On failure, link to the workflow run
- On skip (no skills), explain why no evaluation ran
- Skip posting success comment if no table exists to avoid duplicates
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Initial plan
* Narrow evaluation workflow path triggers to only evaluation-related files
Replace broad '.github/workflows/**' trigger with specific evaluation
workflow files (evaluation.yml, evaluation-run.yml, evaluation-fork-pr.yml)
to prevent evaluation from running on PRs that only change unrelated workflows.
Also update the infra-change detection regex to match the same narrowed set.
Co-authored-by: JanKrivanek <3809076+JanKrivanek@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JanKrivanek <3809076+JanKrivanek@users.noreply.github.com>
- msbuild-antipatterns: Move AP-16 through AP-21 and checklist to
references/additional-antipatterns.md (body: 384 lines, was 583)
- directory-build-organization: Replace cross-skill reference to
../msbuild-antipatterns/SKILL.md with local reference file
references/targetframework-props-pitfall.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add workflow_dispatch trigger to skill-validator.yml
Add a workflow_dispatch trigger with an optional create_release boolean
input so the workflow can be run manually from the Actions tab. When
create_release is true, the release job runs and retention is set to
90 days, matching the scheduled nightly behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use string comparison for inputs and guard release to main
Compare inputs.create_release against 'true' (string) instead of true
(boolean) since workflow_dispatch inputs are surfaced as strings. Also
restrict the release job to refs/heads/main to prevent accidental
overwrites of the nightly tag from non-main branches.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix create_release input check to use truthiness
workflow_dispatch inputs with type: boolean are actual booleans in the
inputs context, so comparing against the string 'true' never matches.
Use truthy evaluation (inputs.create_release) instead.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add workflow_dispatch trigger to skill-validator.yml
Add a workflow_dispatch trigger with an optional create_release boolean
input so the workflow can be run manually from the Actions tab. When
create_release is true, the release job runs and retention is set to
90 days, matching the scheduled nightly behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use string comparison for inputs and guard release to main
Compare inputs.create_release against 'true' (string) instead of true
(boolean) since workflow_dispatch inputs are surfaced as strings. Also
restrict the release job to refs/heads/main to prevent accidental
overwrites of the nightly tag from non-main branches.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Upload skill-validator-dist as tar.gz with archive: false to preserve
execute permissions through artifact round-trip
- Remove chmod +x workarounds in evaluate and comment-on-pr jobs
- Include README.md in publish output via CopyToPublishDirectory
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: improve UX for published tool
- Make paths argument required (OneOrMore arity) so invoking without
args shows full help text instead of a cryptic error
- Add Installation section to README covering dotnet tool install from
nightly release, platform-specific AOT packages, and pipeline artifacts
- Update all usage examples to use the skillvalidator command
- Update CI integration example to install from nightly release
- Include README.md as NuGet package readme
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address PR feedback: clarify release wording and chmod instructions
- Replace ambiguous 'GitHub pre-release artifacts' with a direct link
to the skill-validator-nightly GitHub Release
- Use 'chmod +x' instead of 'give +x permissions'
- Capitalize NuGet consistently
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Align ToolCommandName with AssemblyName (skill-validator)
Fixes naming mismatch where dotnet tool install produced 'skillvalidator'
but the AOT binary was 'skill-validator'.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update README.md
* Publish tar.gz archives in skill-validator workflow
- Add tar.gz creation from dotnet pack publish output for each RID
- Upgrade upload-artifact to v7 with archive: false (no double-zip)
- Upgrade download-artifact to v8 for non-zipped artifact support
- Include tar.gz files in nightly GitHub release
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update README
* Update README
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Enable NativeAOT publishing for SkillValidator
Migrate all serialization to source-generated, AOT-compatible patterns:
- Add PublishAot to csproj and Vecc.YamlDotNet.Analyzers.StaticGenerator package
- Create SkillValidatorJsonContext with source-generated serializers for all types
- Create SkillValidatorYamlContext with static YAML deserialization for eval config
and frontmatter parsing
- Change AgentEvent.Data from Dictionary<string, object?> to
Dictionary<string, JsonNode?> for AOT-safe serialization
- Replace anonymous type in Reporter.cs with named ResultsOutput record
- Fix JsonStringEnumConverter to generic AOT-safe JsonStringEnumConverter<T>
- Move ConsolidateData to Models.cs for JSON context accessibility
- Switch EvalSchema and SkillDiscovery to StaticDeserializerBuilder
- Add RawFrontmatter model for strongly-typed YAML frontmatter parsing
- Update all JsonSerializer call sites to use context-based overloads
The published binary is a single 14.6 MB native executable.
All 198 existing tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add NativeAOT publish matrix and daily scheduled builds to CI
- Single build job with matrix for linux-x64, linux-arm64, win-x64, win-arm64, osx-arm64
- Each matrix entry: build, test, NativeAOT publish, pack NuGet, upload tarball
- collect-packages job: builds RID-agnostic package, merges with per-RID packages
into a single skill-validator-nupkgs artifact
- Add daily schedule trigger (08:00 UTC) with 90-day retention
- Keep existing path-based triggers for push/PR
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add NativeAOT publish matrix and daily scheduled builds to CI
- Single build job with matrix for linux-x64, linux-arm64, win-x64, win-arm64, osx-arm64
- Each matrix entry: build, test, NativeAOT publish, pack NuGet, upload binary
- Upload publish directory directly (Actions wraps in zip automatically)
- collect-packages job: builds RID-agnostic package, merges with per-RID packages
into a single skill-validator-nupkgs artifact
- Add daily schedule trigger (08:00 UTC) with 90-day retention
- Keep existing path-based triggers for push/PR
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix flaky DelayClampedToRemainingBudget test
Increase totalTimeoutMs from 200ms to 2000ms. The tight budget was
causing the CTS to fire before the retry attempt on slower CI runners
(macOS arm64). The test still validates that the 60s base delay gets
clamped — the 5s wall-time assertion is the real check.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Cancel in-progress workflow runs on new push
Add concurrency group so that pushing a new commit to a PR
automatically cancels any still-running workflow for that ref.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix per-RID NuGet pack with --use-current-runtime
The pack step was producing the same agnostic package for every RID.
Adding --use-current-runtime ensures each matrix entry produces a
RID-specific nupkg that gets collected into the combined artifact.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove redundant publish step; pack already produces native binary
dotnet pack --use-current-runtime already runs publish internally,
producing the native AOT binary in artifacts/publish/. The separate
dotnet publish step was redundant.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename nupkgs artifact to nuget-bundled
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove redundant PublishTrimmed; implied by PublishAot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix flaky ExponentialBackoff_DelaysIncrease test
Increase baseDelayMs from 50ms to 200ms so that OS scheduling jitter
on slow CI runners (macOS arm64) doesn't dominate the measured gaps.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove per-RID publish directory artifacts; nupkgs suffice
NuGet packages are zip files containing the native binaries, so the
separate skill-validator-<rid> publish folder uploads are redundant.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove collect-packages job; produce agnostic nupkg from linux-x64
Pack the RID-agnostic package from the linux-x64 matrix entry instead
of a separate downstream job, eliminating the bundle step entirely.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename artifact names from nuget-* to skill-validator-*
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename agnostic artifact to skill-validator
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify NuGet artifact uploads; remove rm -rf and collect-packages
Use precise glob patterns with the stable package name to distinguish
RID-specific from agnostic nupkgs. The agnostic upload uses a negation
pattern to exclude the RID-specific package on linux-x64.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Extract retention-days and agnostic RID into workflow env vars
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add nightly release job using gh release
On scheduled runs, downloads all nupkg artifacts and publishes them
as a rolling 'skill-validator-nightly' pre-release on the repo.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restore PropertyNameCaseInsensitive for JSON deserialization
The original MCPServerDef deserialization used PropertyNameCaseInsensitive
which was dropped when switching to source-generated context. Add it back
to the JsonSourceGenerationOptions to preserve flexible property casing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reject eval configs with an empty scenarios list at parse time,
rather than silently accepting them. This ensures authors don't
accidentally ship eval files that test nothing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add nuget-trusted-publishing skill
Knowledge-driven skill for setting up NuGet trusted publishing (OIDC)
on GitHub Actions repos, replacing long-lived API keys with short-lived tokens.
Replaces #34 which could not be reopened after history rewrite.
* Address review: move to plugins/tests layout, fix phase sequencing, add CODEOWNERS
- Move skill from src/dotnet/skills/ to plugins/dotnet/skills/nuget-trusted-publishing/
- Move eval from src/dotnet/tests/ to tests/dotnet/nuget-trusted-publishing/
- Fix Phase 3/4 sequencing conflict: clarify that drafting the workflow is OK
before policy confirmation, but publishing/removing secrets should wait
- Add CODEOWNERS entry for nuget-trusted-publishing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Warn when eval prompt explicitly references skill name
Add a profile warning when an eval scenario prompt contains the skill
name (e.g. 'Use the migrate-dotnet10-to-dotnet11 skill'). This pattern
biases baseline runs (agent wastes time searching for a non-existent
skill file) and forces activation rather than testing organic discovery.
The warning flows through the existing ProfileWarnings pipeline so it
surfaces in console output, JSON results, and the 'Possible causes'
section when a verdict fails.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: guard empty skill name, add unit tests
Guard against empty/whitespace skill names that would false-positive on
string.Contains. Add three unit tests covering: prompt mentions skill
name (warns), prompt does not mention skill name (no warn), and empty
skill name (no warn).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add networking investigation support to dotnet-trace-collect skill
- Add Networking Issues section with System.Net.* EventSource providers
for modern .NET (Http, NameResolution, Security, Sockets)
- Clarify .NET Framework needs only PerfView /ThreadTime for networking
- Add Kestrel providers to Slow Requests section for ASP.NET Core
- Use /Providers syntax for PerfView command-line examples
- Update reference files with networking provider examples
- Add two networking eval scenarios (HTTP 500s, timeouts)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix networking trace commands to explicitly specify --profile when using --providers
When --providers is specified, it overrides the default profiles. Updated the
dotnet-trace collect command to include --profile dotnet-common,dotnet-sampled-thread-time
and the collect-linux command to include --profile dotnet-common,cpu-sampling,thread-time.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix dotnet-trace reference to show correct default profiles
The default profiles are dotnet-common and dotnet-sampled-thread-time,
not CPU sampling.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reusable workflow inputs typed as 'number' reject string values
from job outputs. Change pr-number to type: string and update
the skip condition and default accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update evaluation-fork-pr.yml
* Address PR review: least-privilege permissions and resilient status reporting
- Remove statuses: write from workflow-level permissions (already on
gate and report-status jobs)
- Make pending status step non-blocking with continue-on-error
- Use 'error' state for cancelled/infrastructure issues vs 'failure'
for actual evaluation failures
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add better retry for judge
* Extract funtionality and guard against single scenario failing whole eval run
* Reflect PR comments
* Reflect PR comments
* Apply timeout and parallelism defaults to evaluation-run.yml
Upstream refactored the evaluation workflow to use a reusable workflow
(evaluation-run.yml). Move our timeout (60->90min) and parallelism
(3->5) changes to that file.
* Replace environment gate with /evaluate comment trigger
Replace pull_request_target + environment approval with an issue_comment
trigger. Maintainers with write+ access post '/evaluate' on a fork PR
to start the evaluation. This avoids the deployment entries that GitHub
environments create on the repo page.
Security model:
- Workflow YAML from default branch (enforced by issue_comment trigger)
- Validator built from default branch
- Fork content is untrusted data only
- Only write+ users can trigger via permission check
Also removes the environment input from the reusable workflow since it
is no longer needed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use PR base SHA for diff instead of hardcoded origin/main
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Enable evaluation for fork PRs via reusable workflow
Refactor the evaluation workflow into three files:
- evaluation-run.yml: reusable workflow with build-validator, evaluate,
and comment-on-pr jobs (shared by both callers)
- evaluation.yml: thin caller for pull_request and schedule triggers;
skips fork PRs which are handled by the new fork workflow
- evaluation-fork-pr.yml: thin caller using pull_request_target with
environment-gated secrets for fork PRs
Also clear GITHUB_TOKEN from the process environment after the
CopilotClient reads it, so eval scenarios and setup commands cannot
access the credential.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review: harden permissions and cleanup
- Add persist-credentials: false to untrusted content checkout
- Add explicit permissions (contents: read) to build-validator and
evaluate jobs to limit blast radius
- Guard comment-on-pr against skipped evaluate result
- Remove unused is_infra input from reusable workflow
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restrict discover job permissions and document workflow-level scope
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Skip token randomization on forked PRs
On forked PRs, the token randomization step is skipped and the
COPILOT_GITHUB_TOKEN secret is used directly as a fallback.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove fork guard from skill-validator step
The step now runs on forked PRs using the COPILOT_GITHUB_TOKEN
fallback when the randomization step is skipped.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update README with VS Code plugin instructions
Added instructions for enabling VS Code plugin support and using agent plugins.
* Update README.md
---------
Co-authored-by: Larry Ewing <lewing@microsoft.com>
Forks that activate Actions (by pushing a branch for a PR) inadvertently
start running the evaluation schedule every 2 hours. Each run builds and
uploads a ~95 MB artifact, consuming the fork owner's Actions minutes and
storage. The scheduled runs always fail on forks because the required
COPILOT_GITHUB_TOKEN secrets are not available.
Add a guard on the discover job to skip scheduled runs on forks. All
other jobs depend on discover, so this gates the entire workflow for
the schedule event. PR and workflow_dispatch triggers are unaffected.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add expect_activation: false to iOS crash log scenario (negative test)
- Remove 3 script-runner scenarios that never activated the skill and
created persistent CI noise (they tested PowerShell script correctness,
not skill activation)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add dotnet-trace-collect skill for .NET performance diagnostics
Add a new skill that guides developers through capturing diagnostic
artifacts for production .NET performance issues. Covers tool selection,
data collection commands, and analysis recommendations across Windows,
Linux, containers, and Kubernetes for both modern .NET and .NET Framework.
Includes eval.yaml with 12 test scenarios covering high CPU, memory leaks,
excessive GC, hangs, slow requests, native call stacks, Windows containers,
long-running repros, and environment-specific tool trade-offs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Remove overly broad eval assertion.
* Add workload context check to Step 1
Ask whether the agent is running in the context of the workload.
If so, run diagnostic commands directly; otherwise provide guidance.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Prefer console tools over dotnet-monitor in K8s when in workload context
When running in the context of the workload with console access,
prefer dotnet-trace, dotnet-trace collect-linux, and dotnet-dump
over dotnet-monitor to avoid authentication setup. Fall back to
dotnet-monitor when console access is not available or it is
already deployed.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update tool reference: prefer console tools over dotnet-monitor in workload context
Update dotnet-counters container comment and dotnet-monitor trade-offs
to note that console tools are preferred when running in the workload
context with console access.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Delegate dump collection to dump-collect skill
This skill focuses on trace collection. All dump collection guidance
and execution is now delegated to the dump-collect skill. References
to dotnet-dump, procdump, createdump, gcore, and Task Manager for
dump collection are replaced with delegation to dump-collect.
Dump analysis rows removed from Step 4 analysis table since dump
artifacts are handled by the other skill.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove dotnet-dump section and dump references from tool reference
The entire dotnet-dump section is removed since dump collection is
delegated to the dump-collect skill. Also removed dump/gcdump
endpoints from dotnet-monitor REST API examples (replaced gcdump
with GC trace endpoint), and cleaned up remaining procdump/dotnet-dump
references in PerfView and dotnet-monitor trade-offs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update hangs guidance: start with a trace before a dump
Reorder the hangs workflow to capture a trace first. A trace can
reveal livelocks, thread starvation, or partial forward progress.
Only escalate to a dump if the trace does not explain the hang,
which indicates a potential true deadlock.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix dotnet-trace collect profile: use dotnet-sampled-thread-time
cpu-sampling is only available for collect-linux. Replace with
dotnet-sampled-thread-time in the dotnet-trace collect example.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename dotnet-trace section header to dotnet-trace collect
Clarify the section heading to distinguish dotnet-trace collect from
dotnet-trace collect-linux, which has its own dedicated section.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Split tool-compatibility.md into per-tool reference files
Split the monolithic tool-compatibility.md (603 lines) into 6
individual reference files, one per tool:
- dotnet-counters.md (47 lines)
- dotnet-trace-collect.md (66 lines)
- dotnet-trace-collect-linux.md (76 lines)
- dotnet-monitor.md (93 lines)
- perfview.md (125 lines)
- perfcollect.md (177 lines)
This reduces context usage since the skill only needs to load 1-2
tool references per scenario (50-250 lines) instead of all 603 lines.
The tool selection logic remains entirely in SKILL.md; references are
only needed after the tool is chosen.
Updated SKILL.md to replace 3 generic tool-compatibility.md links
with an environment-to-reference-file lookup table. Follows the same
pattern used by the dump-collect skill.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add collect-linux limitations: symbol resolution and cross-namespace
Document two additional limitations for dotnet-trace collect-linux:
- Native symbols must be on disk during capture for resolution
- Cannot trace cross-namespace processes
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add aka.ms direct download links for dotnet-trace
Add curl/wget download option using aka.ms links for Linux x64,
Arm64, musl x64, and Windows x64. This avoids the need for a
.NET SDK, which is useful in container environments.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove dotnet-trace report from analysis tools table
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update evals to cover recent skill changes
Updated existing scenarios:
- Hang diagnosis: expect trace-first (livelock/starvation check)
before escalating to dump for true deadlocks
- Memory leak (Linux + .NET Framework): expect delegation to
dump-collect skill instead of direct dotnet-dump commands
Added new scenarios:
- K8s with console access: prefer console tools over dotnet-monitor
- Container without SDK: aka.ms direct download for dotnet-trace
- collect-linux cross-namespace limitation awareness
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove dotnet-counters from skill
dotnet-counters was causing confusion where the skill would recommend
dotnet-counters but download dotnet-trace via aka.ms links. Since traces
provide richer diagnostic data, remove all dotnet-counters references
and focus the skill on trace collection.
- Remove dotnet-counters.md reference file
- Remove dotnet-counters from lookup table, container tools list,
live monitoring section, and analysis table in SKILL.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update K8s eval to prefer dotnet-trace over dotnet-monitor
The skill recommends console-based tools (dotnet-trace) as the preferred
approach when the user has console access to a container. Update the
High CPU in Kubernetes scenario to match: expect dotnet-trace as
primary, dotnet-monitor as alternative for no-console-access cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add missing context to eval prompts to prevent user prompting
Several eval prompts were missing required inputs (deployment context,
admin privileges, symptom, .NET version) that the skill needs to select
the right tool. This caused the validator to hang waiting for clarifying
questions that never come in eval mode.
Changes:
- Add 'running directly on a VM/server' to non-container scenarios
- Add symptom to .NET 10 native stacks and container installation prompts
- Add .NET version and admin access to slow requests prompt
- Add admin access to PerfView triggers prompt
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Strengthen dump-collect delegation language to 'invoke' not 'delegate'
The skill was using passive 'delegate to dump-collect skill' phrasing
which LLMs may interpret as merely mentioning the skill rather than
actually invoking it. Changed all references to use 'invoke the
dump-collect skill' with explicit 'do not provide dump commands directly'
in key sections. Updated eval rubrics to match.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove mention of dotnet-monitor for Windows Container CPU eval
The result should be completely focused on PerfView usage and not
dotnet-monitor.
* Clarify StopOn triggers must fire on interesting event, not recovery
Strengthened the circular buffer + StopOn guidance to explain WHY
triggering on recovery is wrong: the circular buffer continuously
overwrites old data, so the interesting behavior may be lost by the
time collection stops. Added 'Critical' callout and 'Never trigger
on recovery' in the pitfalls table.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Explain why console tools are preferred over dotnet-monitor
When recommending console-based tools over dotnet-monitor in containers,
explicitly state the reason: dotnet-monitor requires authentication
configuration and sidecar deployment, making console tools easier to
set up when you have direct access.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add kernel requirement to cross-namespace eval rubric
The judge was flagging the kernel >= 6.4 with CONFIG_USER_EVENTS=y
detail as oddly specific, but it is accurate per the collect-linux
reference documentation. Add rubric item so the judge recognizes
this as correct.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Scope dump-collect to modern .NET only, provide direct commands for .NET Framework
The dump-collect skill only supports modern .NET (.NET Core 3.0+).
Updated all .NET Framework references to provide dump commands directly
(procdump -ma, Task Manager) instead of invoking dump-collect. Updated
the Memory dumps header, decision matrix, .NET Framework sections, and
the .NET Framework memory leak eval to match.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove .NET 10 collect-linux cross-namespace eval scenario
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove auth setup rubric from K8s console access eval
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move skill to standard directory structure and update CODEOWNERS
Moved skill from src/dotnet/skills/ to plugins/dotnet/skills/ and
tests from src/dotnet/tests/ to tests/dotnet/ to match the existing
directory structure. Updated CODEOWNERS to add the new skill with
@brianrob as owner.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Fix devops-aw triggering
* fix: add pull_request to pre_activation gate so PR trigger works
* fix: enable deep investigation dispatches
- Fix COPILOT_GITHUB_TOKEN in investigate workflow to use _3 (fine-grained PAT)
- Strengthen Step 5 dispatch instructions to be MANDATORY
- Add first-run note: all critical findings must be dispatched
- Add verification checklist before finishing
* fix: update frontmatter hash to match .md source
* fix: investigation uses add-comment instead of update-issue
update-issue fails on workflow_dispatch (no triggering issue context).
Switch to add-comment with explicit item_number targeting the health issue.
Remove update_issue from safe-outputs config, tools, and validation.
* fix: address PR review feedback
- Fix concurrency group: use stable ref-based key with cancel-in-progress
instead of run_id which defeated mutual exclusion (reviewer feedback)
- Remove temporary pull_request trigger and pre_activation gating
(was only needed for testing, reviewer correctly flagged mismatch)
- Remove slash command references from docs and shared knowledge
since slash_command trigger was removed (reviewer feedback)
Dan Moseley