* Enable evaluation as a required status check
- Remove path filters from evaluation.yml pull_request trigger so the
workflow runs on every PR
- Add statuses: write permission and a report-status job that posts a
commit status with context 'evaluation' for non-fork PRs
- Change evaluation-fork-pr.yml commit status context from
'evaluation (fork PRs)' to 'evaluation' so both workflows share the
same status context
- The 'evaluation' status can now be configured as a required check
in the GitHub ruleset
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback: handle discover failure, least-privilege perms
- Remove workflow-level statuses: write (job already declares it)
- Broaden report-status condition to handle discover failure/cancellation
instead of leaving PRs stuck with no status
- Exclude fork PRs (discover skipped) to avoid conflicting with
evaluation-fork-pr.yml
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Handle fork PRs in evaluation.yml for required status check
- Remove fork guard from discover so it runs for all PRs
- Add is_fork output to detect fork PRs
- Skip run-evaluation for fork PRs (no access to secrets)
- Post 'pending' status for fork PRs with skill changes, prompting
the author to use /evaluate
- Post 'success' for fork PRs with no skill changes
- This ensures every PR gets an 'evaluation' commit status, making
it viable as a required check without fork PRs getting stuck
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add statuses: write at workflow level (required for job-level grant)
Job-level permissions cannot exceed the workflow-level permissions.
The report-status job needs statuses: write but was getting read-only
because the workflow level didn't include it.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use pull_request_target for fork PR status posting
Fork PRs need an 'evaluation' commit status but pull_request triggers
have restricted GITHUB_TOKEN permissions for forks. Use
pull_request_target in evaluation-fork-pr.yml instead, which runs in
the base repo context with full permissions.
The new fork-status job:
- Triggers on pull_request_target for fork PRs only
- Checks if the PR changes any skills (git diff only, no code execution)
- Posts 'pending' status if skills changed (needs /evaluate)
- Posts 'success' if no skills to evaluate
evaluation.yml reverts to the original fork guard on discover (skips
fork PRs entirely). report-status excludes skipped discover results
so it only posts for non-fork PRs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: fix header comment, use shallow checkout
- Correct security model comment: pull_request_target runs from the
base branch, not necessarily the default branch
- Use default shallow checkout (depth 1) instead of fetch-depth: 0
- Diff against HEAD instead of base SHA since checkout is the base
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify evaluation.yml: use plain check-run job instead of commit status API
Replace report-status job (which called gh api to post commit statuses)
with a plain evaluation job whose check run can be required directly.
Removes the need for statuses: write permission in this workflow.
Fork PRs are handled by evaluation-fork-pr.yml which still uses commit
statuses (necessary because issue_comment check runs are not associated
with the PR head SHA).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix fork-status: pin checkout to base SHA, use merge-base for diff
- Pin actions/checkout to the PR base SHA to avoid checking out the
merge ref which includes untrusted fork content
- Use explicit base/head SHAs with merge-base for reliable diff,
consistent with the non-fork workflow's discovery logic
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename evaluation job and context to evaluation-status
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: least-privilege permissions, full clone for merge-base
- Add permissions: {} to evaluation-status job (evaluation.yml)
- Use fetch-depth: 0 for fork-status checkout so merge-base works
- Remove --depth=1 from PR head fetch
- Keep statuses: write at workflow level (needed by reusable workflow)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix header comment: evaluation → evaluation-status
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Enable skill-validator as a required status check
- Remove path filters from pull_request trigger
- Add discover job to skip the expensive 5-platform build matrix when
no skill-validator files changed
- Add report-status job that posts commit status 'skill-validator':
- No relevant changes → success
- Build passed → success
- Build failed → failure
- Schedule and workflow_dispatch triggers unchanged (discover is
skipped, build runs unconditionally)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Handle fork PRs with pull_request_target for status posting
Fork PRs have restricted GITHUB_TOKEN and can't post commit statuses
via pull_request trigger. Add pull_request_target trigger with a
fork-status job that posts success for fork PRs (they should never
change skill-validator).
- Add pull_request_target trigger
- Gate build to skip for pull_request_target events
- Gate report-status to non-fork PRs only
- Add fork-status job for fork PRs (always posts success)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Enable skill-validator as a required check
- Remove path filters from pull_request trigger so the workflow runs
on every PR
- Add discover job to skip the expensive 5-platform build matrix when
no skill-validator files changed
- Add status job that can be required in the GitHub ruleset: succeeds
when no relevant changes or when the build passes, fails otherwise
- No commit status API calls or pull_request_target needed — just a
plain job whose check run is required
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve CODEOWNERS validation workflow
- Remove path filters from pull_request trigger so it can be a required check
- Add owner sufficiency validation: each skill/test folder must have either
2+ individual owners or 1+ team in CODEOWNERS
- Update issue creation to report both missing entries and insufficient owners
- Update failure step to trigger on either validation failure
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update CODEOWNERS to add additional code owners
Added @agocke and @hoyosjs as code owners for migrate-nullable-references and dotnet-trace-collect skills.
* Update CODEOWNERS for dotnet-trace-collect
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace marketplace.json symlink with file copy
On Windows without developer mode, git checks out symlinks as plain text
files containing the target path. This caused Claude Code to fail parsing
.claude-plugin/marketplace.json since it contained the string
'../.github/plugin/marketplace.json' instead of valid JSON.
Replace the symlink with an actual copy of the file and update
CONTRIBUTING.md to instruct contributors to update both locations.
Fixes#300
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Include error details in evaluation failure PR comments
When evaluation fails due to validation errors (e.g., description too long),
the PR comment previously only said 'Evaluation failed. View workflow run',
requiring a click-through to discover the actual error.
Two fixes:
- Reporter.cs: GenerateMarkdownSummary() now renders an Errors section for
verdicts with FailureKind set, so the marker comment shows what went wrong.
- evaluation-fork-pr.yml: The report-status job now downloads result artifacts
and extracts error reasons from results.json, including them directly in the
failure comment posted to the PR.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* TEST: inflate build-perf-diagnostics description to trigger validation error
Temporarily makes the description 1221 chars (limit is 1024) to verify that
the new error reporting shows the validation failure in the PR comment.
This commit should be reverted after verifying the error display.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten error filter to pre-eval failures and add tests
Address review feedback:
- Restrict the Errors section in markdown summary to verdicts with no
scenarios (pre-eval failures like spec_conformance_failure), avoiding
duplication with threshold/regression failures already shown in the table.
- Add three tests: errors section appears for pre-eval failures, is omitted
when all verdicts pass, and is omitted for failures that have scenarios.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix error details in evaluation-run.yml post-completion-comment
The failure comment was posted by evaluation-run.yml's post-completion-comment
job (shared by both same-repo and fork PR workflows), not just by the fork PR
workflow. Apply the same artifact download + error extraction to this job so
error details appear for same-repo PRs too.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Promote eval prompt skill-name mention from warning to error
When an eval scenario prompt explicitly references the skill name,
it biases baseline runs and forces activation instead of testing
organic discovery. Move the check from warnings to errors so it
blocks evaluation via the existing profile.Errors fail-fast path.
No existing evals on main have this pattern.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Sanitize @ in error output to prevent mention injection
Neutralize @ characters with a zero-width space in all three locations
where error text from PR-controlled content is rendered into PR comments:
Reporter.cs, evaluation-run.yml, and evaluation-fork-pr.yml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Distinguish errors vs warnings in evaluation failure comments
Pre-eval structural errors (spec conformance, missing eval) get a
red X icon, while eval verdict failures (below threshold, completion
regression) get a warning icon. This makes it clear which failures
are blocking issues vs expected score variance.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Only show pre-eval errors in failure comment, link to results table
The failure comment no longer repeats verdict warnings (below
threshold, completion regression) that are already visible in the
results table. It only shows pre-eval structural errors (spec
conformance failures) and links to the results table for full details.
Three cases:
- Pre-eval errors: show them + link to results table
- No errors but failed (verdict): just link to results table
- No results table: fallback with workflow run link
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove duplicate failure comment from fork workflow
The reusable evaluation-run.yml already posts failure comments when
pr-number is provided. The fork workflow was duplicating this by
posting its own failure comment, resulting in two comments per failure.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert "TEST: inflate build-perf-diagnostics description to trigger validation error"
This reverts commit 5f9056695a.
* Address review feedback: boundary-aware match, markdown escaping, safe find loop
- SkillProfiler: use boundary-aware regex for skill-name-in-prompt check
to avoid false positives on short skill names (e.g. 'go')
- Reporter: wrap skill name in inline code and escape backticks/newlines
to prevent markdown injection from PR-controlled content
- evaluation-run.yml: use find -print0 with while read loop to handle
paths with spaces; match Reporter inline-code format in jq output
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clean up comments in evaluation-fork-pr.yml
Removed redundant comments regarding failure comments for fork PRs.
* Refine comments on eval prompts and skill discovery
Clarify comment regarding eval prompts and skill discovery errors.
* Update eng/skill-validator/src/Services/Reporter.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update .github/workflows/evaluation-run.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Reapply "TEST: inflate build-perf-diagnostics description to trigger validation error"
This reverts commit 331ed870c5.
* Fix test assertion to match updated markdown error format
The Reporter.GenerateMarkdownSummary method was updated in 95ede45 to wrap
both skill name and reason in a single backtick pair for better markdown
injection prevention, but the test assertion was not updated to match.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eng/skill-validator/src/Services/Reporter.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update .github/workflows/evaluation-run.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Link to newest results-table comment instead of oldest
The GitHub comments API returns oldest-first, so head -1 was linking
the failure comment to the oldest results-table marker comment rather
than the most recent run. Use tail -1 to select the newest match.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Consolidate PR comment jobs into a single comment-on-pr job
Merge comment-on-pr and post-completion-comment into one job that always
posts a new comment with the full results summary. When no result
artifacts exist (evaluate skipped or failed entirely), a short failure
message is posted instead.
This removes the find-and-update-existing-comment pattern, the separate
error extraction via jq, the PR_COMMENT_MARKER, and the comment_action
workflow output.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Guard consolidation step against missing validator binary
When result artifacts download succeeds but the validator binary
download/extract fails, the consolidate command would fail silently
(step is continue-on-error) and no PR comment would be posted. Check
that the validator executable exists before attempting consolidation,
falling through to the generic failure message otherwise.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Skip results table header when there are no scenario rows
Only emit the markdown table when at least one scenario row exists,
so error-only summaries show just the heading and errors section
without an empty table.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Display errors as bullet points in markdown summary
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Show validation errors with status emoji in heading
Use '### ❌ Skill validation errors:' heading for the errors section
to make the failure status immediately visible.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert "Reapply "TEST: inflate build-perf-diagnostics description to trigger validation error""
This reverts commit 59d5d8dfd4.
* Update eng/skill-validator/tests/OverfittingJudgeTests.cs
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update .github/workflows/evaluation-run.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Purge dashboard entries older than 14 days
When updating plugin JSON files, filter out Quality and Efficiency
entries whose date is older than 14 days. If all entries are purged,
delete the JSON file so it is excluded from the components.json
manifest and no longer appears in the dashboard.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move stale file cleanup to workflow level
Address review feedback: the delete-if-empty branch in the script was
unreachable because fresh entries are always appended before purging.
Instead, add a workflow step that scans all existing data files after
per-plugin generation and removes any whose entries are entirely older
than 14 days.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move stale file purge logic into the script
Add a -PurgeStaleFiles/-DataDir parameter set so the workflow calls
the script instead of duplicating the retention logic inline. The
retention window is defined once in the script.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make retention days configurable via workflow env var
Add -RetentionDays parameter to the script (mandatory for
-PurgeStaleFiles, optional for generate mode) and define
DASHBOARD_RETENTION_DAYS env var at the workflow level so the value
is controlled in one place.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback: fix comment and add entries guard
- Fix workflow comment to accurately describe the purge behavior
- Guard against missing 'entries' key in purge mode to avoid silently
skipping malformed plugin JSON files
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eng/dashboard/generate-benchmark-data.ps1
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update .github/workflows/evaluation.yml
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add verdict footnotes to evaluation results table
The results table showed a Verdict column (pass/fail) based on a weighted
composite score, but only displayed the raw quality delta. When the
composite disagreed with quality (e.g., quality improved but token/time
overhead made the composite negative), the verdict was unexplained.
Add a Notes column with footnote references [1], [2] etc. that link to
explanations below the table. Footnotes are only added when the verdict
disagrees with the quality delta direction, keeping the table clean for
straightforward cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* TEST: inflate csharp-scripts description to exercise verdict footnotes
Expands the skill description to trigger token overhead in evaluation,
which should produce a scenario where the verdict footnote feature
activates. **Revert before merge.**
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback on BuildVerdictFootnote
- Return null for neutral verdicts (ImprovementScore == 0) and unknown
quality delta (null), avoiding misleading footnotes for the yellow
neutral icon or when the table shows delta as dash.
- Align quality direction threshold with the F1 rounding used in the
delta column so footnotes only fire when the displayed value disagrees
with the verdict.
- Use DefaultWeights.Values instead of hard-coded weight literals to
avoid drift if weights change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix roundedDelta==0 footnote bug and improve test coverage
- Return null from BuildVerdictFootnote when rounded quality delta is zero,
preventing misleading 'offset quality gain' text when the table shows 0.0
- Make test ImprovementScore values consistent with breakdown * DefaultWeights
- Add test for 'quality dropped but composite positive' branch
- Add test for roundedDelta==0 edge case
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve results table: collapse quality columns and clarify footnotes
- Merge Baseline, With Skill, and delta columns into a single Quality
column showing 'base -> skill' with the better score bolded and an
arrow indicator, reducing table width for narrow viewports
- Reword footnotes from cryptic 'composite -X% -- factors offset
quality gain' to readable 'Quality improved but weighted score is
-X% due to: factors'
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Show raw metric values in verdict footnotes
Footnotes now include the actual baseline and skill values for each
contributing metric, e.g.:
tokens (1000 -> 8000), tool calls (5 -> 15), time (2.1s -> 8.5s)
This lets readers immediately see the magnitude of each factor without
digging into workflow logs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use emoji arrows, align footnote gating with Quality column
- Replace tiny unicode arrows with emoji (📈/📉) in Quality column
- Use raw quality delta for footnote direction instead of rounded delta,
matching FormatQualityCell which compares unrounded scores
- Rename test to MarkdownTable_NoFootnoteWhenQualityUnchanged with
truly equal scores instead of near-zero delta
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make footnote references bidirectional hyperlinks
Table [1] links jump to the footnote explanation at the bottom,
and footnote [1] links jump back to the table row.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restore dash assertion in ShowsDashWhenNoOverfitting test
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix flaky ReturnsEmptyForNonSkillDirectory test
Use a fresh empty temp directory instead of Path.GetTempPath() which
can contain stray SKILL.md files left by other tests or processes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use <strong> instead of ** for bold inside HTML anchor tags
GitHub markdown does not render ** inside raw HTML elements.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Merge Notes into Verdict column and add footnote for quality-unchanged cases
- Remove Notes column (7 -> 6 columns) to reduce table width; footnote
reference now appears next to the verdict icon
- Add footnote when quality is unchanged but verdict is negative, with
text: 'Quality unchanged but weighted score is -X% due to: ...'
- Update tests accordingly
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert "TEST: inflate csharp-scripts description to exercise verdict footnotes"
This reverts commit d86dee0eee.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
* Add dotnet-maui-doctor skill and refs
Add a new .agents/skills/dotnet-maui-doctor skill bundle. Includes SKILL.md describing an autonomous workflow to detect and remediate .NET MAUI environment issues and platform-specific reference docs for installation commands, platform requirements, troubleshooting, Microsoft OpenJDK guidance, and a WorkloadDependencies discovery guide that pulls authoritative versions from NuGet. Documents validation/remediation steps for .NET SDK, workloads, JDK, Android SDK, Xcode, and Windows SDK (macOS/Windows/Linux) and includes a temporary recommendation to prefer Microsoft OpenJDK (JDK 21) until manifests are updated.
* MAUI doesn't quite yet support dotnet run
* Add dotnet-maui plugin and dotnet-maui-doctor skill
Introduce the dotnet-maui plugin and move the dotnet-maui-doctor skill under plugins/ layout. Update marketplace, CODEOWNERS, and README accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add comprehensive eval scenarios for dotnet-maui-doctor
10 scenarios covering: macOS/Linux/Windows setup, JDK vendor enforcement,
workload update/repair guardrails, dynamic version discovery, JAVA_HOME
misconception, Android SDK diagnosis, out-of-scope decline, and stale
workload remediation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refine eval scenarios: remove weak scenarios, fix overfitting
Remove 'Resist hardcoding' (penalizes skill — baseline gives more useful
direct answer) and 'Decline runtime bug' (skill makes agent refuse to
help). Rework Android SDK scenario to test CI-oriented package discovery
where baseline lacks knowledge. Replace vocabulary-level NuGet/API
assertions with outcome-focused rubric items across all scenarios.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove hardcoded JDK version workarounds in favor of dynamic discovery
Replace temporary JDK 21 workaround notes with guidance to use the
version from WorkloadDependencies.json. Replace hardcoded JDK paths
in troubleshooting.md with {VERSION} placeholders.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve eval scenarios: fix JAVA_HOME timeout, macOS tie, reduce overfitting
- JAVA_HOME: rewrite prompt to request conceptual answer (prevents tool timeout)
- macOS Xcode: add installation source guidance, remove equal-penalty assertion
- Remove irrelevant workload update/repair assertions from 3 off-topic scenarios
- Keep workload assertions in 5 scenarios where workload management is the topic
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reframe JAVA_HOME scenario as prevent-harmful-action pattern
Rewrite prompt from conceptual yes/no question (which the agent answered
from wrong general knowledge without activating the skill) to a scenario
where the user is about to apply incorrect Stack Overflow advice. This
mirrors the successful workload guardrail pattern.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reduce overfitting: rephrase assertions and rubric to be outcome-focused
- Replace literal '--version' assertions with broader 'version|pin' pattern
- Rephrase Xcode rubric from skill-specific source to version control risk
- Rephrase version discovery rubric from 'NuGet API' to outcome-focused
- Rephrase workload install rubric from '--version flag' to 'pinned version'
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Inline critical JAVA_HOME auto-detection guidance into SKILL.md
The JAVA_HOME eval scenario scored poorly because the agent couldn't read
the reference file (references/microsoft-openjdk.md) and fell back on
general knowledge, which incorrectly says JAVA_HOME must be set.
Changes:
- Task 5: Add inline JAVA_HOME guidance with decision table showing that
JAVA_HOME is NOT required and MAUI auto-detects JDK installations
- Common Pitfalls: Add 'Unnecessary JAVA_HOME' entry with unset commands
This ensures the agent knows JAVA_HOME is unnecessary even when reference
files can't be loaded. File stays at 210 lines (limit: 500).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Inline JAVA_HOME auto-detection knowledge into SKILL.md
Add JAVA_HOME guidance directly to Task 5, Task 9, and Common Pitfalls
so the agent knows JAVA_HOME is not required even when reference files
can't be read. Based on Redth's original microsoft-openjdk.md reference.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address code review feedback from Copilot
- Fix cmd.exe syntax in installation-commands.md (use PowerShell $env:)
- Replace hardcoded simulator device/runtime with placeholders
- Add System.IO.Compression.FileSystem assembly load for PS 5.1 compat
- Clarify workload reinstall example is macOS-specific (omit ios on Linux)
- Replace hardcoded android-35 with $API_LEVEL placeholder
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address jonathanpeppers review feedback
- Add Apple Developer login/2FA note for Xcode downloads
- Fix Windows SDK detection: extract KitsRoot10 and version subkeys
instead of dumping full registry (was outputting noise)
- Replace manual NuGet API approach with dotnet workload search version
CLI command for discovering workload versions
- Reframe JAVA_HOME guidance: report as anomaly rather than prescribing
unset/set commands (per jonathanpeppers recommendation)
- Use $env:ProgramFiles instead of hardcoded C:\Program Files paths
in PowerShell commands
- Update SKILL.md to match: softer JAVA_HOME guidance, improved
Windows SDK detection command
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Redth review feedback
- Remove Windows SDK detection commands — installed automatically
via MAUI workload/VS Installer (per Redth recommendation)
- Use dotnet workload search version --format json --take 1 for
workload version discovery instead of NuGet search API (per both
Redth and jonathanpeppers)
- Update workload-dependencies-discovery.md Step 2 to match
- Expand Android SDK detection with known filesystem paths
instead of relying solely on environment variables
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Align eval rubric with reviewer feedback to reduce overfitting
Replace technique-specific rubric items with outcome-focused criteria:
- Replace PowerShell API query check with dotnet CLI discovery check
- Replace manifest extraction check with workload-version-based discovery
- Replace prescriptive version discovery with CLI/docs verification
- Convert workload update/repair regex assertions to rubric criteria
- Remove Invoke-RestMethod assertion (no longer using API queries)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Align JAVA_HOME guidance with reviewer consensus, add Xcode download expectations
- Consistently use ⚠️ anomaly approach for non-Microsoft JAVA_HOME across
microsoft-openjdk.md and troubleshooting.md (matching SKILL.md)
- Remove contradictory Note in microsoft-openjdk.md that conflicted with table
- Reframe unset commands as user's choice, not automatic action
- Add Xcode download size (~12GB) and time expectations per Redth's feedback
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update plugins/dotnet-maui/skills/dotnet-maui-doctor/references/workload-dependencies-discovery.md
Co-authored-by: Jonathan Peppers <jonathan.peppers@gmail.com>
* Update JDK example to version 21 per jonathanpeppers feedback
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: redth <jondick@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Jonathan Peppers <jonathan.peppers@gmail.com>
* Workaround gh-aw concurrency group issue
* Clarify concurrency bug comment per code review
Rephrase comment to make clear the bug causes mutual cancellation,
not a 2-run limit. The 2-run cap is the workaround, not the bug.
Co-authored-by: Copilot <copilot@github.com>
---------
Co-authored-by: Copilot <copilot@github.com>
Move the 'Post completion comment' step from evaluation-fork-pr.yml into
evaluation-run.yml as a dedicated post-completion-comment job, so both
evaluation.yml and evaluation-fork-pr.yml share the same logic.
The fork-pr workflow retains a slim step that only handles the skipped case
(no entries / discover failed) since evaluation-run.yml isn't invoked then.
Also extract the '<!-- skill-validator-results -->' marker into a workflow-
level PR_COMMENT_MARKER env var to eliminate duplication.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The evaluation workflow was using a two-point diff (base_sha..head_sha) to
detect changed files, which includes files changed on main that the PR branch
hasn't merged yet. This caused false 'Infrastructure changes detected' triggers
when evaluation workflow files were modified on main after a PR branch diverged.
Switch to computing the merge-base first, so only changes actually introduced
by the PR are considered.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix 429 rate limiting in evaluation pipeline
- P0: Add retry with exponential backoff to ListModelsAsync() in ValidateCommand
using the existing RetryHelper (3 retries, 2s base delay, 60s budget)
- P1: Reduce scheduled-run parallelism to match infra-change settings (2/3/3)
to avoid overloading the API when all 5 plugins run in parallel
- P2: Assign Copilot tokens deterministically by matrix job index instead of
randomly, eliminating ~96% token collision probability
- P3: Add retry with backoff to AgentRunner.RunAgent() for mid-eval 429s
(2 retries, 5s base delay, scenario timeout + 60s budget)
Fixesdotnet/skills#168
* Delete docs/fix-429-rate-limiting.md
* Add agent and plugin spec conformance validation
Extend the validation pipeline to check .agent.md files and plugin.json
files alongside skills. Previously only SKILL.md files were validated.
Agent validation (AgentProfiler):
- Frontmatter required (error)
- Name and description required (error)
- Name format: same rules as skills (lowercase alphanum + hyphens, 64 max)
- Name must match filename minus .agent.md suffix
- Description max 1024 chars
- Body max 500 lines
Plugin validation (PluginValidator):
- Name required, must match directory name
- Version and description required
- Description max 1024 chars
- Skills path required and must exist
- Agents path warned if specified but missing
Discovery (SkillDiscovery):
- DiscoverAgents() walks up from skill paths to plugin roots,
scans agents/ directories for .agent.md files
- DiscoverPlugins() finds plugin.json from skill paths
Integration (ValidateCommand):
- Runs after skill discovery, before evaluation
- Errors block with exit code 1 (same as spec_conformance_failure)
All 4 existing agents and 2 plugins pass cleanly.
25 new tests (13 agent, 12 plugin).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Code review fixes: spec comments, ValidateNameFormat, kind-specific messages
- Refactor ValidateName into ValidateNameFormat (format-only) + ValidateName
(format + directory match) so agent/plugin validation doesn't abuse the
skill directory-match check by passing name twice.
- Warning messages now say 'Agent name' or 'Plugin name' instead of
'Skill name' when called from AgentProfiler/PluginValidator.
- Add spec URL comments to AgentProfiler and PluginValidator referencing
agentskills.io/specification, VS Code agent-plugins docs, and Claude Code
plugins-reference.
- Add missing agent name edge-case tests: leading/trailing hyphen,
consecutive hyphens, and message-prefix assertions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove unused agents path from plugins that have no agents
The dotnet, dotnet-data, and dotnet-upgrade plugins declared an agents
path but had no agents/ directory, producing validation warnings. Remove
the field; it can be added back when agents are actually created.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Promote agent/plugin name validation from warnings to errors
The agentskills.io spec uses 'Must' for all name constraints (length,
character set, no leading/trailing hyphens, no consecutive hyphens).
For a publishing validator these should block, not just warn. Agent
filename mismatch is also promoted since the IDE won't discover the
agent if the name doesn't match.
Skill name validation stays as warnings (lenient loading, not part
of this PR).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback
- Fix FrontmatterRegex: remove RegexOptions.Multiline so a horizontal
rule (---) in the body is not mistaken for frontmatter. Detection and
stripping now both anchor to start of string.
- Use filename as fallback identifier in AgentProfile when agent.Name
is empty, so error output shows [agent:foo.agent.md] not [agent:].
- Add TryGetSafeSubdirectory to PluginValidator: rejects absolute paths
and parent-directory traversal in skills/agents path fields. Used by
both PluginValidator and SkillDiscovery.DiscoverAgents.
- Use DirectoryName as fallback for plugin result name when Name is
empty, so error output shows [plugin:my-dir] not [plugin:].
- Add tests for path traversal rejection and name fallback behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Detect orphaned test directories with no matching plugin/skill
Add FindOrphanedTestDirectories to SkillDiscovery: checks that every
directory under tests/{plugin}/{skill}/ has a corresponding directory
under plugins/{plugin}/skills/{skill}/. Plugin-level orphans (tests/
directory with no matching plugins/ directory) are also flagged.
Integrated into ValidateCommand after plugin/agent validation so it
runs on every validation pass. Errors are blocking (exit 1).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #279 review feedback
- Use case-sensitive (Ordinal) comparison for agent filename matching
since agent names are required to be lowercase by spec.
- Replace OrdinalIgnoreCase path-under-root check in TryGetSafeSubdirectory
with Path.GetRelativePath + '..' prefix check for correct behavior on
case-sensitive filesystems.
- Let ParsePluginJson throw JsonException on malformed JSON instead of
silently returning null. ValidateCommand catches it as a blocking error.
- Add test for malformed plugin.json.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Increase eval timeouts for dump-collect and dotnet-trace-collect to 120s
The with-skill runs for these skills were hitting the 60s scenario timeout
in CI, causing the agent to be killed before producing output. Other skills
in the same plugin already use 120s. The 30s decline/negative-test scenario
is left unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix fixture paths in clr-activation-debugging and analyzing-dotnet-performance evals
The source paths referenced tests/dotnet/ but the fixtures live under
tests/dotnet-diag/. This caused file-not-found exceptions at runtime,
which set hasRejections and caused unconditional exit 1 (bypassing
--verdict-warn-only).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Surface execution errors in validation summary
Previously, skills that threw exceptions (e.g. missing fixture files)
were silently excluded from the verdict count. The summary showed
'2/4 skills passed' with no indication that 2 other skills crashed
entirely. Now the summary shows '2/6 skills passed (2 rejected due
to execution errors)' and lists the specific error messages near
the summary rather than buried in the middle of the log.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reduce parallelism for infra-change evaluation runs
When infrastructure changes trigger evaluation of all plugins
simultaneously, the high default parallelism (5x5x5) causes API
rate limits (429s) and scenario timeouts from contention. Reduce
to 2x3x3 for infra-change runs only; regular single-plugin PR
evaluations keep the default 5x5x5.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Promote skill spec conformance checks from warnings to errors
The agentskills.io spec (https://agentskills.io/specification) uses
'Must' for all of these constraints:
- name: 'Must be 1-64 characters'
- name: 'May only contain lowercase alphanumeric and hyphens'
- name: 'Must not start or end with -'
- name: 'Must not contain consecutive hyphens'
- name: 'Must match the parent directory name'
- description: 'Must be 1-1024 characters' (non-empty)
- compatibility: 'Must be 1-500 characters if provided'
These were previously warnings following the lenient-loading guidance
from the client implementation guide, but that guidance is for client
loaders, not publishing validators. For a CI tool that gates what gets
published, spec violations should be blocking errors.
All 25 existing skills already comply — no changes needed to skills.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback: empty compatibility, empty name, test
- Error when compatibility field is provided but empty/whitespace,
matching spec requirement of 1-500 characters when provided.
- Add explicit check for empty/whitespace name with targeted error
message instead of falling through to regex mismatch.
- Add test for empty compatibility string rejection.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Enforce description size limits (per-skill 1024, per-plugin 15K)
Add infrastructure to reject skills with oversized descriptions, per issue #222:
- SkillProfiler: check description length, flag DescriptionTooLong
- ValidateCommand: block oversized descriptions (survives --verdict-warn-only)
- New eng/validate-descriptions.ps1: validates per-skill and per-plugin aggregate limits
- New CI workflow: runs validation on PRs touching plugins/
- build.ps1: adds size checks to local build validation
- Unit tests for all new SkillProfiler behavior
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update --verdict-warn-only help text to mention description_too_long
Address review feedback: the option description now lists description_too_long
alongside execution errors and --require-evals as conditions that remain
fatal even when --verdict-warn-only is enabled.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add agentskills.io spec conformance checks to SkillProfiler
Validate name rules (max 64 chars, lowercase+hyphens, no edge/consecutive
hyphens, must match directory), compatibility field max 500 chars, body
max 500 lines, and file reference depth.
Scan of existing skills found 2 violations:
- dotnet-msbuild/msbuild-antipatterns: 582 lines (max 500)
- dotnet-msbuild/directory-build-organization: cross-skill ref 3 levels deep
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback on spec conformance checks
- Fix MakeSkill SkillMdPath to derive from resolved path (SkillProfileTests.cs)
- Fix RepoRoot in build.ps1 to be actual repo root (was agentic-workflows/)
- Fix body line count off-by-one: trim trailing newlines before splitting
- Fix file ref depth: normalize paths, handle .. traversals, strip #anchor,
compute depth from directory segments only
- Add tests for parent traversal, anchor fragments, ./ normalization,
and trailing newline body count
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Consolidate description validation into skill-validator C#
Move the per-plugin aggregate description limit (15K chars) from the
standalone PowerShell script into ValidateCommand, where it runs after
skill discovery. This eliminates the separate validate-descriptions.ps1,
its CI workflow, its PowerShell tests, and the delegation from build.ps1.
Also rename FailureKind from "description_too_long" to the generic
"spec_conformance_failure" so future spec checks reuse the same kind.
Deleted:
- eng/validate-descriptions.ps1
- eng/tests/validate-descriptions.tests.ps1
- .github/workflows/validate-descriptions.yml
Added:
- ValidateCommand.CheckAggregateDescriptionLimits()
- ValidateCommand.DerivePluginName()
- 6 C# tests for aggregate limit + DerivePluginName
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Upgrade body-lines and file-ref-depth from warnings to blocking errors
Now that #247 has merged and fixed the two existing violations
(msbuild-antipatterns 582 lines, directory-build-organization cross-ref),
these spec conformance checks can be blocking.
Also replaces the DescriptionTooLong bool with a general Errors list on
SkillProfile, so all spec conformance violations (description length,
body lines, file ref depth) flow through the same mechanism.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert unrelated lock file changes
The lock file diffs were side effects of running build.ps1 to verify
the validation changes — they picked up upstream content changes
unrelated to this PR.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reorganize plugins per agreed structure (#271)
Split the dotnet plugin into domain-specific plugins per discussion
in #271:
- dotnet: narrowed to common everyday C#/.NET coding tasks
- dotnet-diag: perf investigations, debugging, incident analysis
- dotnet-data: data access and Entity Framework
- dotnet-aspnet, dotnet-wpf, dotnet-winforms, dotnet-maui: scaffolded
empty to convey intent for future framework-specific skills
Moved skills, tests, and CODEOWNERS entries accordingly.
No file content was changed; all skill/test moves are pure renames.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve dotnet-diag and dotnet-winforms descriptions
- dotnet-diag: add '.NET' qualifier to clarify scope
- dotnet-winforms: use official 'Windows Forms (WinForms)' name
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add .NET qualifier to dotnet-data description
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove empty plugins; list them in CONTRIBUTING.md as future options
Remove the four empty placeholder plugins (dotnet-aspnet, dotnet-wpf,
dotnet-winforms, dotnet-maui) that had only plugin.json and no skills.
Add a 'Plugin organization' section to CONTRIBUTING.md that describes the
domain-specific plugin groupings introduced in this PR and lists the four
removed plugin names as reserved candidates for future contributions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Create dotnet-upgrade plugin with migration/upgrade skills
Move migrate-nullable-references, thread-abort-migration, and
dotnet-aot-compat from dotnet to the new dotnet-upgrade plugin.
Update CODEOWNERS paths (same owners), CONTRIBUTING.md plugin table,
and fix the thread-abort-migration SKILL.md link in CONTRIBUTING.md.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix fork PR evaluation concurrency cancelling runs on unrelated comments
The concurrency group used the PR number alone, so any comment on a PR
(not just /evaluate) would cancel an in-progress evaluation. Now only
/evaluate comments share a group; other comments get a unique group
keyed by comment ID so they never cancel real evaluation runs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* remove push trigger from skill-validator.yml
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Remove eyes reaction from trigger comment when workflow finishes
- Post a status comment on completion (success, failure, or skipped)
- On success, link to the evaluation summary table comment
- On failure, link to the workflow run
- On skip (no skills), explain why no evaluation ran
- Skip posting success comment if no table exists to avoid duplicates
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Initial plan
* Narrow evaluation workflow path triggers to only evaluation-related files
Replace broad '.github/workflows/**' trigger with specific evaluation
workflow files (evaluation.yml, evaluation-run.yml, evaluation-fork-pr.yml)
to prevent evaluation from running on PRs that only change unrelated workflows.
Also update the infra-change detection regex to match the same narrowed set.
Co-authored-by: JanKrivanek <3809076+JanKrivanek@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: JanKrivanek <3809076+JanKrivanek@users.noreply.github.com>
- msbuild-antipatterns: Move AP-16 through AP-21 and checklist to
references/additional-antipatterns.md (body: 384 lines, was 583)
- directory-build-organization: Replace cross-skill reference to
../msbuild-antipatterns/SKILL.md with local reference file
references/targetframework-props-pitfall.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add workflow_dispatch trigger to skill-validator.yml
Add a workflow_dispatch trigger with an optional create_release boolean
input so the workflow can be run manually from the Actions tab. When
create_release is true, the release job runs and retention is set to
90 days, matching the scheduled nightly behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use string comparison for inputs and guard release to main
Compare inputs.create_release against 'true' (string) instead of true
(boolean) since workflow_dispatch inputs are surfaced as strings. Also
restrict the release job to refs/heads/main to prevent accidental
overwrites of the nightly tag from non-main branches.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix create_release input check to use truthiness
workflow_dispatch inputs with type: boolean are actual booleans in the
inputs context, so comparing against the string 'true' never matches.
Use truthy evaluation (inputs.create_release) instead.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add workflow_dispatch trigger to skill-validator.yml
Add a workflow_dispatch trigger with an optional create_release boolean
input so the workflow can be run manually from the Actions tab. When
create_release is true, the release job runs and retention is set to
90 days, matching the scheduled nightly behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use string comparison for inputs and guard release to main
Compare inputs.create_release against 'true' (string) instead of true
(boolean) since workflow_dispatch inputs are surfaced as strings. Also
restrict the release job to refs/heads/main to prevent accidental
overwrites of the nightly tag from non-main branches.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Upload skill-validator-dist as tar.gz with archive: false to preserve
execute permissions through artifact round-trip
- Remove chmod +x workarounds in evaluate and comment-on-pr jobs
- Include README.md in publish output via CopyToPublishDirectory
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: improve UX for published tool
- Make paths argument required (OneOrMore arity) so invoking without
args shows full help text instead of a cryptic error
- Add Installation section to README covering dotnet tool install from
nightly release, platform-specific AOT packages, and pipeline artifacts
- Update all usage examples to use the skillvalidator command
- Update CI integration example to install from nightly release
- Include README.md as NuGet package readme
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address PR feedback: clarify release wording and chmod instructions
- Replace ambiguous 'GitHub pre-release artifacts' with a direct link
to the skill-validator-nightly GitHub Release
- Use 'chmod +x' instead of 'give +x permissions'
- Capitalize NuGet consistently
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Align ToolCommandName with AssemblyName (skill-validator)
Fixes naming mismatch where dotnet tool install produced 'skillvalidator'
but the AOT binary was 'skill-validator'.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update README.md
* Publish tar.gz archives in skill-validator workflow
- Add tar.gz creation from dotnet pack publish output for each RID
- Upgrade upload-artifact to v7 with archive: false (no double-zip)
- Upgrade download-artifact to v8 for non-zipped artifact support
- Include tar.gz files in nightly GitHub release
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update README
* Update README
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Enable NativeAOT publishing for SkillValidator
Migrate all serialization to source-generated, AOT-compatible patterns:
- Add PublishAot to csproj and Vecc.YamlDotNet.Analyzers.StaticGenerator package
- Create SkillValidatorJsonContext with source-generated serializers for all types
- Create SkillValidatorYamlContext with static YAML deserialization for eval config
and frontmatter parsing
- Change AgentEvent.Data from Dictionary<string, object?> to
Dictionary<string, JsonNode?> for AOT-safe serialization
- Replace anonymous type in Reporter.cs with named ResultsOutput record
- Fix JsonStringEnumConverter to generic AOT-safe JsonStringEnumConverter<T>
- Move ConsolidateData to Models.cs for JSON context accessibility
- Switch EvalSchema and SkillDiscovery to StaticDeserializerBuilder
- Add RawFrontmatter model for strongly-typed YAML frontmatter parsing
- Update all JsonSerializer call sites to use context-based overloads
The published binary is a single 14.6 MB native executable.
All 198 existing tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add NativeAOT publish matrix and daily scheduled builds to CI
- Single build job with matrix for linux-x64, linux-arm64, win-x64, win-arm64, osx-arm64
- Each matrix entry: build, test, NativeAOT publish, pack NuGet, upload tarball
- collect-packages job: builds RID-agnostic package, merges with per-RID packages
into a single skill-validator-nupkgs artifact
- Add daily schedule trigger (08:00 UTC) with 90-day retention
- Keep existing path-based triggers for push/PR
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add NativeAOT publish matrix and daily scheduled builds to CI
- Single build job with matrix for linux-x64, linux-arm64, win-x64, win-arm64, osx-arm64
- Each matrix entry: build, test, NativeAOT publish, pack NuGet, upload binary
- Upload publish directory directly (Actions wraps in zip automatically)
- collect-packages job: builds RID-agnostic package, merges with per-RID packages
into a single skill-validator-nupkgs artifact
- Add daily schedule trigger (08:00 UTC) with 90-day retention
- Keep existing path-based triggers for push/PR
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix flaky DelayClampedToRemainingBudget test
Increase totalTimeoutMs from 200ms to 2000ms. The tight budget was
causing the CTS to fire before the retry attempt on slower CI runners
(macOS arm64). The test still validates that the 60s base delay gets
clamped — the 5s wall-time assertion is the real check.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Cancel in-progress workflow runs on new push
Add concurrency group so that pushing a new commit to a PR
automatically cancels any still-running workflow for that ref.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix per-RID NuGet pack with --use-current-runtime
The pack step was producing the same agnostic package for every RID.
Adding --use-current-runtime ensures each matrix entry produces a
RID-specific nupkg that gets collected into the combined artifact.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove redundant publish step; pack already produces native binary
dotnet pack --use-current-runtime already runs publish internally,
producing the native AOT binary in artifacts/publish/. The separate
dotnet publish step was redundant.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename nupkgs artifact to nuget-bundled
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove redundant PublishTrimmed; implied by PublishAot
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix flaky ExponentialBackoff_DelaysIncrease test
Increase baseDelayMs from 50ms to 200ms so that OS scheduling jitter
on slow CI runners (macOS arm64) doesn't dominate the measured gaps.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove per-RID publish directory artifacts; nupkgs suffice
NuGet packages are zip files containing the native binaries, so the
separate skill-validator-<rid> publish folder uploads are redundant.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove collect-packages job; produce agnostic nupkg from linux-x64
Pack the RID-agnostic package from the linux-x64 matrix entry instead
of a separate downstream job, eliminating the bundle step entirely.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename artifact names from nuget-* to skill-validator-*
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename agnostic artifact to skill-validator
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify NuGet artifact uploads; remove rm -rf and collect-packages
Use precise glob patterns with the stable package name to distinguish
RID-specific from agnostic nupkgs. The agnostic upload uses a negation
pattern to exclude the RID-specific package on linux-x64.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Extract retention-days and agnostic RID into workflow env vars
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add nightly release job using gh release
On scheduled runs, downloads all nupkg artifacts and publishes them
as a rolling 'skill-validator-nightly' pre-release on the repo.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restore PropertyNameCaseInsensitive for JSON deserialization
The original MCPServerDef deserialization used PropertyNameCaseInsensitive
which was dropped when switching to source-generated context. Add it back
to the JsonSourceGenerationOptions to preserve flexible property casing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reject eval configs with an empty scenarios list at parse time,
rather than silently accepting them. This ensures authors don't
accidentally ship eval files that test nothing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add nuget-trusted-publishing skill
Knowledge-driven skill for setting up NuGet trusted publishing (OIDC)
on GitHub Actions repos, replacing long-lived API keys with short-lived tokens.
Replaces #34 which could not be reopened after history rewrite.
* Address review: move to plugins/tests layout, fix phase sequencing, add CODEOWNERS
- Move skill from src/dotnet/skills/ to plugins/dotnet/skills/nuget-trusted-publishing/
- Move eval from src/dotnet/tests/ to tests/dotnet/nuget-trusted-publishing/
- Fix Phase 3/4 sequencing conflict: clarify that drafting the workflow is OK
before policy confirmation, but publishing/removing secrets should wait
- Add CODEOWNERS entry for nuget-trusted-publishing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Warn when eval prompt explicitly references skill name
Add a profile warning when an eval scenario prompt contains the skill
name (e.g. 'Use the migrate-dotnet10-to-dotnet11 skill'). This pattern
biases baseline runs (agent wastes time searching for a non-existent
skill file) and forces activation rather than testing organic discovery.
The warning flows through the existing ProfileWarnings pipeline so it
surfaces in console output, JSON results, and the 'Possible causes'
section when a verdict fails.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: guard empty skill name, add unit tests
Guard against empty/whitespace skill names that would false-positive on
string.Contains. Add three unit tests covering: prompt mentions skill
name (warns), prompt does not mention skill name (no warn), and empty
skill name (no warn).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add networking investigation support to dotnet-trace-collect skill
- Add Networking Issues section with System.Net.* EventSource providers
for modern .NET (Http, NameResolution, Security, Sockets)
- Clarify .NET Framework needs only PerfView /ThreadTime for networking
- Add Kestrel providers to Slow Requests section for ASP.NET Core
- Use /Providers syntax for PerfView command-line examples
- Update reference files with networking provider examples
- Add two networking eval scenarios (HTTP 500s, timeouts)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix networking trace commands to explicitly specify --profile when using --providers
When --providers is specified, it overrides the default profiles. Updated the
dotnet-trace collect command to include --profile dotnet-common,dotnet-sampled-thread-time
and the collect-linux command to include --profile dotnet-common,cpu-sampling,thread-time.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix dotnet-trace reference to show correct default profiles
The default profiles are dotnet-common and dotnet-sampled-thread-time,
not CPU sampling.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reusable workflow inputs typed as 'number' reject string values
from job outputs. Change pr-number to type: string and update
the skip condition and default accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Jan Krivanek