* Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop
WHY
Tools that surface skills (Copilot CLI, Claude Code, Codex, Cursor) read a
plugin's version directly from its checked-in manifest. With no versioning
discipline, a plugin's behavior can change while its advertised version stays
flat, so clients never learn to re-pull, and there is no human-readable signal
of what changed. We want correct, current versions in the repo with minimal
manual work and without bloating the marketplace clone.
WHAT
- Per-plugin semantic versioning via Nerdbank.GitVersioning (NBGV). Each plugin
owns a version.json whose pathFilters exclude the generated manifests and the
version.json itself, so version height tracks real content changes only.
- The computed version is materialized into the checked-in manifests
(plugin.json and .codex-plugin/plugin.json) so every consumer reads a current
value with no build step on their side.
- eng/version/Sync-PluginVersions.ps1 is the single workhorse. It resolves the
set of changed plugins from a git diff, computes each version with nbgv
(predicting the squash-merge height for PRs), and either reports or stamps.
AUTOMATIONS (two, low-touch by design)
- /version-bump: an admin/maintainer comments the command on a PR and the
affected plugins are stamped on the PR branch. Gated on collaborator
permission (admin/write/maintain); forks are rejected before any privileged
step. No other PRs are auto-modified.
- weekly-version-sync: a Monday backstop (and workflow_dispatch) that stamps any
drift on main, opens/updates a single bot PR, and explains the per-plugin
reason. This self-heals anything that merged without a bump.
We deliberately did NOT auto-edit contributor PRs or add a noisy advisory
comment bot; maintainers stay in control and the signal stays clean.
SECURITY (multi-model adversarial review: GPT-5.5 + Gemini 3.1 Pro)
- Supply chain (High, both models): dotnet tool restore would have honored a
nuget.config authored in the PR tree, letting an attacker remap the nbgv
package source to a malicious feed and run code in the privileged
contents:write context. Mitigated with a trusted eng/version/nuget.config
(clear + nuget.org-only + packageSourceMapping), overlaid from main and used
via --configfile so PR-supplied configs are ignored. No nuget.config is
tracked in the repo today, so this path was genuinely exploitable.
- TOCTOU (Medium): /version-bump now checks out the authorized head SHA rather
than the mutable branch name; a racing push fails non-fast-forward, which is
the safe outcome.
- Injection: Set-ManifestVersion uses a MatchEvaluator (not a replacement
string) so a "$"-bearing version cannot re-expand, plus a strict
major.minor.patch guard that throws on a malformed base, leaving manifests
untouched.
- A base-only version.json bump (0.1 -> 0.2) is correctly detected and stamped.
VERIFIED
End-to-end against a real NBGV git harness: content-scoped predict, base-only
bump -> x.y.0, docs-only -> [], weekly drift stamping, malformed-base guard,
and --configfile restore (exit 0). actionlint passes on both workflows.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Copilot review feedback
- Add missing plugins/dotnet-test-migration/version.json so it participates
in versioning (it was the only plugin without one; manifests are at 0.1.0).
- CONTRIBUTING: the two manifests are not byte-identical; say the version is
duplicated across two manifest files instead.
- weekly-version-sync: include version.json in commit attribution so a
base-only bump is explained rather than showing 'no attributable commits'.
- Get-NbgvInfo: capture nbgv stderr and include it in the thrown error so CI
failures are diagnosable, while keeping stdout clean for JSON parsing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Harden vally evaluation against command injection; source tokens from copilot-pat-pool
Close a command-injection vector where fork-PR-controlled path segments flowed
into a bash run: body via matrix.entry.plugin interpolation in the token-bearing
job.
- Allowlist-validate matrix entries (plugin/name/skills_path) before they enter
the matrix in evaluation.yml, and re-validate defense-in-depth as the first
step of the vally-evaluate job.
- Pass all PR-controlled values through env: vars and reference them quoted; no
raw expression interpolation remains in any run: body.
- Build workflow_dispatch entries with jq --arg instead of string interpolation.
- Replace secrets: inherit with explicit secret passing to reduce blast radius.
- Source Copilot tokens from the copilot-pat-pool environment (COPILOT_PAT_0..9)
via the job environment, with COPILOT_GITHUB_TOKEN as an empty-pool fallback.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address Copilot review: reject empty skills_path and quote dirname
- Fail the matrix-entry validation when skills_path is empty (never a
legitimate state for vally entries) instead of silently skipping the loop.
- Quote the inner $(dirname ...) in EVAL_NAME to avoid word-splitting.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address jeffhandley review: drop PAT fallback, explain regex anchors
- Remove the COPILOT_GITHUB_TOKEN fallback secret from vally-evaluation.yml
(workflow_call.secrets, the token-selection step, and evaluation.yml's
caller-side secrets passing). An empty copilot-pat-pool is now a hard
configuration failure instead of falling back to a separate secret.
- Add short comments explaining why bash's ^/$ anchors are safe here
without the \A/\z equivalents used in the PowerShell discover job.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Harden regex validation against locale-dependent bracket ranges
Add LC_ALL=C before the allowlist regex checks in both the prepare job and
the Validate matrix entry step. POSIX bracket ranges like [A-Za-z] can
behave unexpectedly under non-C locale collation; forcing the C locale
makes the anchor/character-class behavior deterministic regardless of the
runner's default locale. Found via cross-model (GPT-5.5) review of the
prior fixup commit.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address review: constrain vally by skills_path, reject empty skills_path
- vally: select eval specs per matrix leg's skills_path so sharded plugins
run only their own skills (no N-times re-run / PAT-pool token waste)
- evaluation.yml: hard-fail on empty skills_path in the matrix allowlist
- vally: quote inner dirname substitution in the Write summary step
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Reject whitespace-only skills_path in matrix validation
A value like a single space passed the bash `-z` empty check and then
word-split to zero loop iterations, letting an effectively-empty
skills_path slip through. Count word-split segments instead so empty and
whitespace-only both hard-fail. Mirror the intent on the PowerShell
discover side with a trim-based emptiness filter.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Bind skills_path segments to the matrix entry's plugin
The skills_path allowlist previously validated only the generic shape
(plugins/<any>/skills[/<skill>]); it did not require each segment to
belong to the entry's own plugin. A mismatched entry (plugin=foo +
skills_path=plugins/bar/skills) would pass, and downstream steps assume
the plugins/$PLUGIN/skills prefix.
Add a literal (non-regex) prefix binding in both validators:
- vally-evaluation.yml: [[ seg != prefix && seg != prefix/* ]] with
skills_prefix="plugins/$ENTRY_PLUGIN/skills".
- evaluation.yml: case-sensitive -cne plus Ordinal StartsWith against
"plugins/$($e.plugin)/skills".
Literal comparison avoids regex-metachar pitfalls from '.'/'-' in plugin
names; -cne/Ordinal keeps it case-sensitive to match Linux path
semantics. Legitimate entries are always same-plugin, so this rejects
no valid input.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Reduce evaluation fan-out to avoid Copilot rate limits
Group PR evaluation entries by plugin and preserve existing
executionShard bucketing so large PRs stop launching one skill-validator
job per changed skill. Cap PR matrix concurrency and keep PR grouped legs
at parallel-skills=1 so each assigned Copilot token stays below the
rate-limit threshold.
Keep vally on the proven reusable-workflow matrix pattern, cap its
concurrency, and only synthesize manual entries when inputs.entries is
empty so workflow_call runs use the caller-provided grouped entries.
Also skip agent.* and eval-less skills when constructing vally
--eval-filter arguments.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Make vally full runs work before the experiment file lands
PR 872 is intentionally based on main only, but main does not have the
new repo-root dotnet-skills.experiment.yaml yet. The vally workflow was
hard-coded to that file, so full branch-level runs from PR 872 failed
before starting any trials.
When the repo file is absent, synthesize an equivalent temporary
experiment file in the runner and use that for read_override() and vally
experiment run. That keeps PR 872 self-contained while preserving the
same per-skill skilled-vs-baseline experiment shape.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* fix: reinforce no-script-execution constraint in devops-health-check
The agent was hallucinating Python script creation (run_analysis2.py)
and attempting to execute it, which was correctly blocked by the bash
allowlist security policy. Strengthen the efficiency guideline to
explicitly state that python/python3/node are NOT in the bash
allowlist and will be blocked, directing the agent to use jq instead.
Fixes#841
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: narrow wording to 'general-purpose language runtimes' per review
Addresses review feedback: 'any script interpreter' was too broad since
jq/sed are technically interpreters but ARE in the allowlist.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add csharp-development skill and update CODEOWNERS and README.md
- Introduced a new skill for C# development guidance.
- Updated CODEOWNERS to include reviewers for the new skill.
- Modified README.md to list the new csharp-development skill.
- Added evaluation scenarios for the csharp-development skill.
* Refactor C# evaluation scenarios to enhance production code review and validation guidance
* Refactor production code review scenarios to improve clarity and focus on async error handling
* Update CODEOWNERS to reflect current ownership and streamline reviewer assignments
* Remove csharp-development skill documentation and evaluation tests
* Remove csharp-development skill from README
* Fix formatting in Constants.g.cs by adding missing semicolon
* Revert Constants.g.cs to main
* Remove csharp-development skill from CODEOWNERS
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update CODEOWNERS to include additional reviewers for dotnet-ai skills
* Move dotnet skills to dotnet-util to keep dotnet plugin focused and targeted.
* Add CODEOWNERS, SKILL.md, package-types.md, and publish-workflow.md for NuGet trusted publishing; implement eval.yaml for dotnet-pinvoke and dotnet-util skills
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update README files to enhance plugin descriptions and remove outdated LSP section
* Rename repository title from 'dotnet' to 'dotnet-util' in README.md for clarity.
* Refine README.md to clarify included .NET skills and remove redundant descriptions
* Add 'dotnet-util' plugin for utility skills in .NET development
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add 'dotnet-util' plugin for utility skills in .NET development
* Add dotnet-util to Codex marketplace manifest
* Align dotnet-util marketplace descriptions
* Refactor CODEOWNERS and update README for nuget-trusted-publishing; add evaluation scenarios and workflow template for trusted publishing
* Add P/Invoke diagnostics and type mapping references; implement NuGet trusted publishing setup
- Created diagnostics.md for common pitfalls and failure modes in P/Invoke.
- Added type-mapping.md for native-to-.NET type mappings.
- Introduced nuget-trusted-publishing skill for setting up OIDC-based NuGet publishing.
- Added package-types.md detailing structural requirements for various NuGet package types.
- Created publish-workflow.md as a template for GitHub Actions workflows for NuGet publishing.
* Add evaluation YAML files for csharp-scripts, dotnet-pinvoke, and nuget-trusted-publishing skills
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Complete nuget-trusted-publishing move: remove from dotnet, add to dotnet-specialized README
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Remove eval.vally.yaml and eval.yaml files for nuget-trusted-publishing skill
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add NuGet trusted publishing skill and related references
- Introduced `nuget-trusted-publishing` skill for setting up OIDC-based trusted publishing on GitHub Actions.
- Added detailed documentation in SKILL.md covering prerequisites, process, safety rules, and troubleshooting.
- Created `package-types.md` to outline structural requirements for various NuGet package types.
- Developed `publish-workflow.md` as a template for GitHub Actions workflows for NuGet publishing.
- Removed obsolete `dotnet-specialized` plugin and updated `dotnet` plugin description.
- Added evaluation tests for `nuget-trusted-publishing` skill to ensure proper guidance for users.
- Implemented evaluation tests for `dotnet-pinvoke` and `csharp-scripts` skills to validate functionality.
* Update dotnet plugin description to include high-level .NET development skills
* Add NuGet trusted publishing and P/Invoke documentation
- Introduced type mapping reference for native-to-.NET types in dotnet-pinvoke.
- Added a comprehensive guide for setting up NuGet trusted publishing using OIDC in GitHub Actions.
- Created a reference document detailing structural requirements for various NuGet package types.
- Provided a template for GitHub Actions workflow for publishing NuGet packages with trusted publishing.
- Implemented evaluation scenarios for testing the dotnet-pinvoke and nuget-trusted-publishing skills, including prompts and assertions for expected outputs.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add CODEOWNERS entries for setup-local-sdk
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update dotnet README to list setup-local-sdk skill
* Add setup-local-sdk to skills section in README
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Require environment input to pat_pool shared workflow.
- Updates the PAT pool shared workflow to latest version
- Updates the validate-pat-pool workflow to match other repos
- Updates all agentic workflows to use the updated PAT pool import
with the standard secret names used across other repos
- The new version of the import does not require the 'needs' workaround
but the issue-triage workflow's 'roles: all' configuration requires
a different workaround to ensure the pre_activation job exists for
the pat_pool job to be able to depend on it (necessary for ordering).
* Update markdown bullet style
* Apply Copilot PR feedback
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Bump versions to match gh-aw
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
PR #658 bumped .github/aw/actions-lock.json to v0.79.8 but its automation could not recompile the .lock.yml files, because the GITHUB_TOKEN it runs under lacks the 'workflow' scope. PR #815 then shipped prompt fixes but likewise did not re-pin the runtime. As a result main is in a split state: actions-lock.json declares v0.79.8 while all 8 compiled workflows still pin and execute the v0.77.5 runtime (setup@v0.77.5).
Run 'gh aw compile' (CLI pinned to v0.79.8) so the deployed workflows actually run the version main already declares, eliminating the declared-vs-deployed drift. This only re-pins the runtime and regenerates compiled output (the 8 .lock.yml files, the generated agentics-maintenance.yml, and the compile-managed dependabot entry); it makes no prompt/behaviour changes, so #815's groom and health-check fixes are preserved verbatim (groom body_hash unchanged at a705abb0).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rebase setup-local-sdk skill
Replay PR #508 onto current main and address the remaining setup-local-sdk review feedback. The install examples now use fail-fast temporary downloads, the team scripts preserve existing global.json content, and eval assertions cover the critical paths/version/workload behavior.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address setup-local-sdk review comments
Add the missing skill license metadata and fix the PowerShell .gitignore regex examples so they correctly detect an existing .dotnet/ entry.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Stabilize setup-local-sdk evals
Reduce overfit-prone assertions, make incompatible host handling explicit, configure base SDK resolution before slow workload installs, and tighten exact-version/team-script validation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address setup-local-sdk follow-up review
Include the shared dotnet skill reviewer team in CODEOWNERS, make .gitignore appends newline-safe, and tighten the incompatible-host eval to reward stopping before local install commands.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten exact local SDK eval
Measure project-level SDK resolution and exact roll-forward behavior using global.json artifact assertions so the scenario distinguishes PATH-based local installs from SDK paths setup.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address install script review feedback
Use the documented lowercase dotnet-install quality value and call out both bash and PowerShell official install script URLs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Increase team script eval timeout
Give the team install script scenario the same realistic long-running timeout as other scenarios that install preview SDK bits.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add Windows local SDK check
Document the PowerShell equivalent for detecting an existing local SDK and give exact SDK installs enough time to complete in eval.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Stabilize team script eval
Focus the team script scenario on script/config generation, create reproducible setup guidance before long downloads, and avoid brittle transcript assertions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make mktemp usage portable
Use BSD/macOS-compatible mktemp templates in setup-local-sdk bash snippets and generated team install script.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(devops-health): stop recurring groom node_id and health-check tool/token failures
Groom (fixes 780/792/797): MCP issue_read(get_comments) sometimes omits node_id, which hide-comment requires. The agent was reporting missing_data and failing the run. Make node_id best-effort: skip hiding comments without a node_id and continue; the weekly cleanup workflow removes them by age.
Health check (fixes 810/791/788): the agent attempted sed and git commands (incl. manual git plumbing on the cache-memory dir) that the bash allowlist denied, looping on the denials and exhausting the 25M effective-token budget. Add sed and git to the allowlist, and add guidelines to not git-manage cache-memory and not retry denied commands.
Recompiled lock files with pinned gh-aw v0.77.5.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix(groom): clarify skipped-comment count goes in noop summary only
Address PR review: Step 6 only emits a summary via noop when no other safe-output calls were made, so 'note in the Step 6 summary' was misleading. Track the skipped count internally and include it in the noop message only when that summary is emitted.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move the .NET test framework/platform migration skills (migrate-mstest-v1v2-to-v3, migrate-mstest-v3-to-v4, migrate-vstest-to-mtp, migrate-xunit-to-mstest, migrate-xunit-to-xunit-v3) and the test-migration orchestrator agent out of dotnet-test into a new dedicated dotnet-test-migration plugin, along with their evals. Update all marketplace manifests, READMEs, CODEOWNERS, and .vally.yaml accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add csharp-development skill and update CODEOWNERS and README.md
- Introduced a new skill for C# development guidance.
- Updated CODEOWNERS to include reviewers for the new skill.
- Modified README.md to list the new csharp-development skill.
- Added evaluation scenarios for the csharp-development skill.
* Refactor C# evaluation scenarios to enhance production code review and validation guidance
* Refactor production code review scenarios to improve clarity and focus on async error handling
* Update CODEOWNERS to reflect current ownership and streamline reviewer assignments
* Remove csharp-development skill documentation and evaluation tests
* Remove csharp-development skill from README
* Fix formatting in Constants.g.cs by adding missing semicolon
* Revert Constants.g.cs to main
* Remove csharp-development skill from CODEOWNERS
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update CODEOWNERS to include additional reviewers for dotnet-ai skills
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Fix evaluation.yml: quote run-name so '#' is not parsed as a YAML comment
The run-name added in #746 is an unquoted plain scalar containing
` #{0}`. In YAML, a space followed by '#' starts a comment, so everything
from '#{0} ...' onward was stripped, leaving an unterminated ${{ }}
expression. The file still parses as YAML (yaml.safe_load succeeds) but
GitHub Actions rejects it ("This run likely failed because of a workflow
file issue"), which broke every evaluation run on main after #746 merged.
Wrapping the value in double quotes keeps the full expression intact;
verified with actionlint.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add authoring-github-workflows skill + actionlint CI gate
Prevents the class of bug that broke evaluation on main (#746): an unquoted
`${{ }}` workflow expression containing `#`, which YAML treats as a comment
and silently truncates, producing a file that parses as YAML but that GitHub
Actions refuses to run ("This run likely failed because of a workflow file
issue", no jobs started).
Two layers:
- Knowledge: .agents/skills/authoring-github-workflows/SKILL.md teaches the
`#`-as-comment trap and other quoting rules, and to validate with actionlint
(plain YAML linters accept the truncated form).
- Enforcement: .github/workflows/actionlint.yml runs a pinned actionlint on
hand-authored workflows for any PR/push touching .github/workflows. Generated
gh-aw files (DO NOT EDIT headers) are skipped; shellcheck/pyflakes are off so
the gate focuses on workflow/expression errors. .github/actionlint.yaml
declares the repo's custom runner labels (ubuntu-slim, windows-11-arm).
Verified locally: actionlint exits 0 on all hand-authored workflows, and
exits 1 on the original unquoted run-name, confirming the gate catches it.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix markdownlint MD038: remove spaces inside code spans in SKILL.md
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review comments: actionlint config trigger, curl -f, syntactic-vs-semantic scope
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Harden actionlint workflow: pin checkout to SHA, disable persisted creds, verify actionlint checksum
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The run-name added in #746 is an unquoted plain scalar containing
` #{0}`. In YAML, a space followed by '#' starts a comment, so everything
from '#{0} ...' onward was stripped, leaving an unterminated ${{ }}
expression. The file still parses as YAML (yaml.safe_load succeeds) but
GitHub Actions rejects it ("This run likely failed because of a workflow
file issue"), which broke every evaluation run on main after #746 merged.
Wrapping the value in double quotes keeps the full expression intact;
verified with actionlint.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The triage worker added the 'evaluate-now' label via GITHUB_TOKEN, but label events emitted by GITHUB_TOKEN do not start workflows (GitHub's recursion guard), so evaluation.yml's pull_request_target:[labeled] entry point never fired for the bot (repro: PR #745). workflow_dispatch and repository_dispatch are the only token-initiated events exempt from that guard.
The worker now dispatches evaluation.yml directly via 'gh workflow run' with a pr_number input, routed through the existing gate job so the path is identical to /evaluate. A dispatched run's head_sha is the default branch (not the PR head), so idempotency now matches the deterministic run name 'Evaluate PR #<n> @ <sha7>'. The 'evaluate-now' label remains a valid human entry point. Worker granted actions:write for 'gh workflow run'.
* Pin github/gh-aw-actions to commit SHA and add pinning guard
The agentic workflows referenced github/gh-aw-actions/setup and
setup-cli by mutable tag (@v0.77.5) because .github/aw/actions-lock.json
still carried a stale pre-migration entry (github/gh-aw/actions/setup@v0.71.5)
with no entry for the action the compiled workflows actually use. With no
matching lock entry, gh aw compile fell back to emitting the bare tag.
Refresh the lock with SHA-pinned entries for github/gh-aw-actions/setup@v0.77.5
and setup-cli@v0.77.5 (commit 3ea13c02...), and regenerate the workflows via
gh aw compile so every uses: ref, decorative comment, and manifest sha is
pinned. This mirrors how dotnet/msbuild pins the same action and unblocks
enabling org-level "require actions pinned to a full-length commit SHA".
Also add enforce-action-pinning.yml, a self-contained PR/push check that
fails any workflow referencing an action by tag/branch. The agentic
workflows are schedule/issue-triggered and never run on PRs, so neither
/evaluate nor the runtime org policy gates them pre-merge; this check does.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Remove enforce-action-pinning guard workflow
Drop the CI guard in favor of relying on the org/repo 'Require actions
pinned to a full-length commit SHA' setting, per review feedback. The
lock-file fix and SHA pins remain the root-cause fix.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Lower the GitHub MCP integrity filter (min-integrity: none, allowed-repos: public) so the triage agent can read issue bodies/comments from external contributors and read-only org members instead of getting a [Filtered] placeholder. Harden the now-less-filtered path: disable issue-body edits on update-issue (body: false) and add an Untrusted content prompt section. Recompiled lock file.
Adopt the shared workflow import pattern from dotnet/runtime PR #127946,
replacing the per-workflow select-copilot-pat action + custom job (shipped in
#736) with a reusable shared/pat_pool.md import.
What changed:
- Add .github/workflows/shared/pat_pool.md: an import that defines a `pat_pool`
job (inline bash, no separate action) exposing a `pat_number` output, plus an
import-schema mapping COPILOT_PAT_0..7 to this repo's pool secrets
(COPILOT_GITHUB_TOKEN, COPILOT_GITHUB_TOKEN_2..8).
- Add .github/workflows/shared/pat_pool.README.md documenting the pattern.
- Convert all 8 agentic workflows to `imports: - shared/pat_pool.md` +
`engine.env` `case(needs.pat_pool.outputs.pat_number ...)`.
- Delete the now-unused .github/actions/select-copilot-pat action.
- Add .github/workflows/validate-pat-pool.yml: a daily standalone workflow that
validates each pool PAT with a Copilot CLI request and summarizes pool health.
Wiring note (adaptation from runtime): consuming workflows declare
`on.needs: [pat_pool]` instead of runtime's `needs: [pre_activation]` +
`on.permissions: {}`. This wires pat_pool ahead of the pre_activation and
activation jobs so the selected PAT is validated by the activation job and used
by the agent, and it works for `roles: all` workflows (issue-triage), which do
not produce a pre_activation job for the runtime workaround to attach to.
Compiled with gh-aw v0.77.5. Verified end-to-end with a temporary
test-pat-rotation workflow (since removed): a run selected token #2 of the
3-token pool and the agent job observed pat_number='2', confirming the rotated
PAT reaches the agent.
* Fix Copilot PAT rotation for gh-aw v0.77.5 runtime
The PAT-rotation stop-gap wired the rotated token into engine.env via needs.pre_activation.outputs.copilot_pat_number. Because the compiled agent job depends only on 'activation' (not the built-in 'pre_activation'), that needs reference evaluated to an empty string in the agent job, so the case() fell through to the default COPILOT_GITHUB_TOKEN and rotation never reached the agent. gh-aw v0.77.5 surfaces this as a compiler warning.
Replace the pre_activation step-injection with a 'select_copilot_pat' custom job wired via on.needs. As a user-defined job referenced in engine.env, the compiler makes it a direct dependency of the agent job, so needs.select_copilot_pat.outputs.copilot_pat_number resolves correctly in both the activation and agent jobs. Same action, same secret pool, same case() expression. Recompiled all workflows with gh-aw v0.77.5.
* Add temporary test-pat-rotation workflow to validate rotation
Non-destructive pull_request-triggered workflow that selects a pool token and asserts, in the agent job, that needs.select_copilot_pat.outputs.copilot_pat_number is non-empty (the exact value that was silently empty with the old pre_activation wiring). To be removed after validation.
* Work around gh-aw v0.77.5 invalid-YAML rendering of top-level if
gh-aw v0.77.5 emits the top-level frontmatter `if:` on the built-in
pre_activation job WITHOUT a ${{ }} wrapper. When the condition starts with
`!` (the fork guards), the emitted `if: !(...)` is invalid YAML (a leading `!`
starts a YAML tag), which GitHub rejects as a workflow-file startup failure.
v0.68.3 wrapped it (valid); v0.77.5 does not. Confirmed independent of the
PAT-rotation change via a minimal probe.
Wrap the fork-guard conditions in parentheses so the emitted scalar starts
with `(` instead of `!` (semantically identical). Affects close-stale-prs,
devops-health-check, devops-health-groom, markdown-linter, pr-malicious-scan.
* Update select-copilot-pat README for the custom-job + on.needs pattern
* Remove temporary test-pat-rotation workflow (rotation validated)
The health check located its dashboard issue solely via the devops-health label search. When GitHub silently dropped the pinned dashboard (#288) from its issue search/list index, the workflow could not find it, created a duplicate (#695), and abandoned the pinned issue -- leaving the pinned dashboard stale for days.
Step 4.1 now resolves the dashboard by a cached issue number first (updating it directly by number, which works even when the issue is missing from search), persists that number to cache-memory every run, falls back to label + pinned-issue lookup, and consolidates duplicates. Documents the new health-dashboard-issue cache key.
Lock file unchanged: the prompt body is runtime-imported from these .md files.
* Fix malicious-scan repeat-spam + integrity-filter blocks
Root causes (observed on PR #237):
1. Agent's emitted add_comment body did not include the HTML marker line
(<!-- pr-malicious-scan:fingerprint=... -->), so both the orchestrator's
pre-dispatch check and the agent's own Step 1 idempotency lookup failed
to find a prior scan for the same head SHA. Result: hourly re-dispatch.
2. The github MCP tools (pull_request_read, list_pull_requests,
search_pull_requests) are blocked by the gh-aw integrity filter on PRs
from non-approved authors -- exactly the population this scanner targets.
Result: 'Integrity filter blocked N items' notes in every comment.
Fixes:
- pr-malicious-scan.agent.md: drop the github MCP toolset, add 'gh' to the
bash allowlist, and instruct the agent to use 'gh api' for all PR data
reads (PAT-authenticated, not subject to the integrity filter).
- Strengthen Step 5: the HTML marker MUST be the first line of the comment
body. Add a defense-in-depth note that the orchestrator also accepts the
visible-body sentinel.
- pr-triage-batch.yml + pr-triage-act.sh: match prior scans by EITHER the
HTML marker OR the visible-body sentinel ('Automated diff scan' + backticked
sha7), so a missing marker on a previously-emitted comment no longer
triggers re-dispatch.
Workflow disabled remotely while this lands.
* Orchestrator-only dispatch + integrity-filter opt-out
Replace the per-push pull_request_target trigger and the gh-api workaround
with the documented gh-aw pattern:
- pr-malicious-scan.agent.md: drop pull_request_target; trigger only via
workflow_dispatch from the orchestrator. Restore the github MCP toolset
with min-integrity: none (the documented level for spam-detection /
analytics workflows; safe-outputs still gates every mutation). Drop the
'gh' bash hack and visible-body sentinel requirements.
- pr-triage-batch.yml: orchestrator now posts a deterministic
'<!-- pr-malicious-scan:dispatched=SHORT --> ' comment BEFORE calling
gh workflow run. That comment is the source of truth for 'a scan has
been initiated for this head SHA' and survives every agent-side failure
mode (PAT outage, integrity block, dropped HTML marker). Dedup matches
either that orchestrator marker OR the agent's own fingerprint marker.
- pr-triage-act.sh: drop the visible-body-sentinel fallback; match the
orchestrator dispatched marker plus the agent fingerprint marker.
Validated: gh aw compile clean; bash -n clean for both worker script and
orchestrator embedded script; markdownlint clean; dedup query and POST
api tested live against PR #713.
* PR triage workflows: orchestrator, worker, and evaluate-now label
Implements docs/design/pr-triage-workflows.md:
- pr-triage-batch.yml: hourly orchestrator that classifies open PRs
- pr-triage.yml + pr-triage-act.sh: per-PR worker (state recompute,
label reconciliation, eval-trigger, ping comments with cool-down)
- evaluation.yml: gate job now also handles pull_request_target [labeled]
with the evaluate-now label as a second entry point alongside /evaluate
* Add temporary push triggers for testing pr-triage workflows
* test: live-run pr-triage worker once
* test: re-run worker for cool-down check
* fix: age gate uses created_at and applies only before first ping
* Remove temporary test triggers and inline test marker
* Add pr-malicious-scan agent workflow; replace design doc with brief overview
- New: .github/workflows/pr-malicious-scan.agent.md + compiled .lock.yml.
Static diff scanner for external (non-trusted) PR contributors. Triggers
on pull_request_target [opened/synchronize/reopened] and workflow_dispatch.
Surfaces findings as code-scanning alerts plus a single maintainer-ping
comment per head SHA when high-severity / workflow-tamper / supply-chain
hits. Never executes PR head code.
- docs/design/pr-triage-workflows.md replaced with a brief overview + diagram.
The full implementation plan is kept locally as
docs/design/pr-triage-workflows-plan.md (gitignored).
- pr-triage-batch.yml's existing dispatch-scanner branch now resolves to
the new scanner; orchestrator unchanged.
* Fix markdownlint MD038 (pipe inside code span) in malicious-scan agent
The dashboard-session-data branch on this repo had grown to ~480MB and was bloating clones. Session data has been migrated to the standalone dotnet/skills-data repository (branch of the same name). This PR redirects:
- dashboard.js -> reads manifest from dotnet/skills-data
- evaluation.yml PR-comment link -> points at dotnet/skills-data
- evaluation.yml publish-session-data job -> clones/pushes to dotnet/skills-data using a new SKILLS_DATA_TOKEN secret (fine-grained PAT with contents:write on the data repo)
Also adds --depth 1 to the deploy clone so future runs do not re-import history.
AGENTVIZ replay UI requires no changes; it accepts an arbitrary manifest URL via query string. raw.githubusercontent.com serves the new repo with Access-Control-Allow-Origin: * so the dotnet.github.io/skills/replay page can load it cross-origin.
The old branch on dotnet/skills is left in place for now and will be deleted manually after one successful run end-to-end.
The publish-eval-data job iterates discover.outputs.plugins and looks for
"all-results/skill-validator-results-<plugin>", but when the discover job fans a
plugin into multiple shards the artifacts are named
"skill-validator-results-<plugin>--shard-<tag>". The exact-match lookup misses
all of them and the step logs "No run results found for dotnet-msbuild,
skipping" -- which is exactly what the 2026-05-27 scheduled run did (confirmed
in the publish-eval-data logs of run 26484154365).
Fix: locate every matching artifact dir (exact name OR --shard-* suffix),
collect the latest run dir's results.json from each, and -- when there are
multiple -- concat their verdicts arrays into a single synthetic results.json
before calling generate-benchmark-data.ps1. Calling the script per-shard would
have emitted N separate datapoints with the same commit/timestamp and skewed
the dashboard.
publish-token-data and publish-session-data were already shard-tolerant and
need no changes.
PR #473 moved the failure investigation prompt out of the PR comment and
only into the workflow run summary. This made it harder for users to
discover — they had to click through to the run details to find it.
Move it back to the PR comment so the copy-paste prompt is visible
inline on the PR, while keeping it in the workflow summary as well.
* Shard plugin evaluation via executionShard tag in eval.yaml
MCP-heavy plugins like dotnet-msbuild reliably exhaust the GitHub-hosted runner host after ~87 min of cumulative MCP traffic (Copilot CLI plus dotnet dnx Microsoft.AITools.BinlogMcp child processes), which has produced no dashboard data for dotnet-msbuild scheduled runs since 2026-05-20. Cutting per-runner concurrency alone was insufficient: at the default 18-way concurrency the runner OOMs in 17-30 min; at 3-way it survives longer but still hits a host-level termination at ~87 min.
Two mitigations, both required:
1. Sharding (this change). The discover step reads an optional top-level 'executionShard:' string from each tests/<plugin>/<skill>/eval.yaml and groups skills by tag, emitting one matrix entry per shard so each shard runs on a fresh runner. Untagged skills fall into a 'default' bucket. Plugins where all skills resolve to a single bucket emit one entry (unchanged behavior). The validator already calls IgnoreUnmatchedProperties so no Models.cs change is needed. Shard names follow '<plugin>--shard-<tag>' so the existing 'skill-validator-results-<plugin>--*' artifact glob keeps working.
2. Per-MCP-plugin concurrency caps (already in place). When plugin.json declares mcpServers, parallelism is forced to 1x3x1 with --judge-timeout 600 to avoid acute OOM within a shard.
Also adds a workflow_dispatch 'plugin' input for targeted re-runs of a single plugin, and tags 10 dotnet-msbuild eval.yamls (heavy/medium); the remaining 8 fall into 'default'. Empirically the three shards complete in parallel in ~30-40 min each, well inside the 180-min job timeout, and successfully upload dashboard data.
* Address PR review: lenient shard regex and validate dispatch plugin input
- executionShard regex now tolerates optional leading whitespace, so an accidentally indented key still groups correctly instead of silently falling back to the default bucket.
- workflow_dispatch 'plugin' input is now validated against ^[a-zA-Z0-9._-]+$ before any Join-Path/Test-Path use, preventing path-traversal style values (e.g. '../foo') from being injected into filesystem paths.
* Add dotnet-blazor plugin and convert-blazor-server-to-webapp skill
Add the dotnet-blazor plugin with 9 skills covering Blazor Web App development:
- plan-ui-change: Plan and scaffold UI features in Blazor Web Apps
- create-blazor-project: Create new Blazor projects with proper render mode setup
- author-component: Author Razor components with parameters, events, lifecycle
- coordinate-components: Share state across components using CascadingValueSource/scoped services
- use-js-interop: Call JavaScript from Blazor and vice versa
- fetch-and-send-data: HTTP data access with proper service patterns
- configure-auth: Set up ASP.NET Core Identity and authorization in Blazor
- collect-user-input: Build forms with EditForm, validation, and file uploads
- support-prerendering: Handle prerendering lifecycle and state persistence
Add convert-blazor-server-to-webapp skill to the dotnet-aspnet plugin for
migrating .NET 7 Blazor Server apps to .NET 8+ Blazor Web App architecture.
All 10 skills pass evaluation with quality improvements ranging from 12-50%
and overfitting scores within acceptable thresholds.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eng/allowed-external-deps.txt
* Update .gitignore
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix vally eval isolation: pin baseline to 0 skills, skilled to 1
vally evaluations were not isolating skills properly. In the failing run
https://github.com/dotnet/skills/actions/runs/26021981688, the dotnet-test
`code-testing-agent` eval reported "Found 77 skill(s)" for baseline and
"Found 22 skill(s)" for the skilled run instead of the expected 0 and 1.
Root cause: `vally eval` derives skills via `discoverSkills(skillRoot)`
where `skillRoot = opts.skillDir ?? cwd`:
- Baseline omitted `--skill-dir`, so vally walked up from the repo root
to `.vally.yaml` and loaded every skill under `plugins/` (77 total).
- The CI workflow's skilled run passed `--skill-dir` pointing at the
plugin-level `plugins/<plugin>/skills` (from
`matrix.entry.skills_path`). That directory has no `SKILL.md` at
its root, so vally's Step-4 child scan picked up every sibling skill in
the plugin (22 for dotnet-test).
Fix applied to both `eng/vally-adapter/run-vally-evals.sh` and
`.github/workflows/vally-evaluation.yml`:
- Baseline now passes `--skill-dir <empty mktemp dir>` so vally
discovers 0 skills.
- Skilled now uses `plugins/<plugin>/skills/<eval-name>` (the specific
skill directory, not the plugin-level one) so vally matches Step-1
(SKILL.md at root) and discovers exactly 1 skill.
- Evals with no matching `SKILL.md` (the `agent.*` orchestrator evals
that declare their own `environment.skills`) are explicitly skipped
with a warning -- baseline-vs-skilled isolation is not meaningful for
them, since they require multi-skill setup by design.
- `Find eval specs` now narrows to a single eval when
`workflow_dispatch` is invoked with a specific skill, so single-skill
dispatches do not accidentally run every eval in the plugin.
- Empty temp dirs are created per-eval and cleaned up after each run to
avoid any cross-run contention under parallel execution.
After the fix the CI log should report `Found 0 skill(s)` for baseline
and `Found 1 skill(s): <skill-name>` for skilled.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: trap-based cleanup for EMPTY_SKILL_DIR; fail-loud on missing dispatch skill
- run-vally-evals.sh: register a RETURN trap immediately after creating
EMPTY_SKILL_DIR so it is removed on every exit path, including aborts
triggered by set -euo pipefail (e.g. if
ode adapt.mjs exits
non-zero inside the log-capture block). Removes the unconditional
rm at the end of the function which would not run on errors.
- vally-evaluation.yml: when workflow_dispatch supplies a skill input
but no ests/<plugin>/<skill>/eval.vally.yaml exists, emit an
explicit ::error:: that includes the missing candidate path and fail
the job, instead of silently falling through to the generic
`No eval.vally.yaml files found for <plugin>` warning.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix evaluation timeouts and improve prompts for dotnet-test skills
- writing-mstest-tests: increase global vally timeout from 3m to 5m to match
the 300s eval.yaml timeout for the complex 'Modernize legacy test patterns' stimulus
- code-testing-agent: increase timeout from 40m/2400s to 60m/3600s
- mtp-hot-reload: increase timeout from 6m to 10m and remove overly strict
expect_tools constraint on launchSettings stimulus
- migrate-mstest-v1v2-to-v3: rephrase prompt for assertion overload errors to
mention CS1501 and cover AreNotEqual/AreSame alongside AreEqual
* Increase eval timeouts for mtp-hot-reload and migrate-mstest-v1v2-to-v3 scenarios
* Strengthen activation prompts and add missing vally scenarios
- migrate-mstest-v1v2-to-v3: Add explicit 'MSTest v2 to v3' migration
language in .testsettings and DataRow prompts (both eval.yaml and
eval.vally.yaml)
- writing-mstest-tests: Add 'MSTest' keyword and skill-specific
terminology to all prompts that lacked activation signal
- writing-mstest-tests: Add 5 missing vally scenarios (string
assertions, comparison assertions, collection/null/reference
assertions, conditional execution, parallelization)
- Add service-registry fixture files for the collection assertions
scenario
* Improve skill activation via SKILL.md descriptions and prompt alignment
Three-pronged fix for systematic NOT ACTIVATED failures:
1. SKILL.md descriptions: Restructure writing-mstest-tests, migrate-
mstest-v1v2-to-v3, and code-testing-agent descriptions to use the
proven 'Use when user says' pattern with quoted trigger phrases
(matching the working run-tests pattern)
2. Vally prompts: Each prompt now contains at least one exact quoted
trigger phrase from its SKILL.md description to ensure strong
description-to-prompt matching
3. Specific fixes:
- writing-mstest-tests: All 15 prompts now include MSTest-specific
trigger phrases (Assert.ThrowsExactly, sealed test class, DataRow,
DynamicData, Assert.HasCount, Assert.IsInstanceOfType, etc.)
- migrate-mstest-v1v2-to-v3: Add '.NET 5 dropped' and 'framework
compatibility issues' to trigger phrases; strengthen dropped-TFM
prompt
- code-testing-agent: Add 'generate comprehensive unit tests' and
'achieve high code coverage' to trigger phrases
* Revert SKILL.md description changes that caused activation regression
The description restructuring in 67056d8ec broke skill activation:
- migrate-mstest-v1v2-to-v3: went from 6/9 activating to 0/9
- writing-mstest-tests: lost isolated activation
- code-testing-agent: unnecessary change to working description
Restore all three SKILL.md files to their pre-regression state.
Keep the vally prompt improvements from 259727e.
* Fix writing-mstest-tests plugin-mode activation
- Add MSTest carve-out to code-testing-agent DO NOT USE FOR to prevent
it from intercepting MSTest-specific prompts
- Add test parallelization/MSTest.Sdk keywords to writing-mstest-tests
description (scenario 15 failed even in isolated mode)
- Strengthen prompts to be more MSTest-specific:
- Scenario 1: emphasize MSTest patterns over generic 'write tests'
- Scenario 4: remove vague 'something seems off', directly ask to
fix swapped Assert.AreEqual arguments
- Scenario 5: reframe as modernization request with specific APIs
- Scenario 13: name specific MSTest assertion APIs in the prompt
* Fix writing-mstest-tests description over 1024 char limit
Shorten the USE FOR section by removing redundant 'help me write
comprehensive tests' and condensing the assertion API list.
Description is now 944 chars (limit: 1024).
* Increase vally job timeout from 60m to 180m
The code-testing-agent eval has a 60m per-stimulus timeout, and vally
runs multiple configurations (baseline, isolated, plugin). The 60m
job-level timeout was causing the job to be cancelled mid-evaluation.
Match the 180m timeout from the evaluate job in evaluation.yml.
* Address PR review comments
- Add IsPackable=false to writing-mstest-tests fixture csproj for
consistency with other dotnet-test fixture projects
- Fix grammar in migrate-mstest-v1v2-to-v3 eval prompts: add comma
after 'upgrade' and add 'file' for clarity (both eval.yaml and
eval.vally.yaml)
* Trim dotnet-test aggregate description size under 15,000 limit
Consolidate duplicate filter keyword mentions in migrate-vstest-to-mtp
and shorten the cross-reference in migrate-xunit-to-xunit-v3 while
keeping the --filter-class/--filter-trait/--filter-query trigger
keywords for skill activation.
* Fix vally skill activation for 9 failing eval scenarios
Improve eval.vally.yaml prompts that were too specific, causing the model
to answer from general knowledge without activating the skill:
- migrate-mstest-v1v2-to-v3: Remove leading question that gives away the
answer ('Is .NET 5 dropped?'); use multi-target fixture matching eval.yaml
- writing-mstest-tests (7 scenarios): Add 'following MSTest best practices'
to trigger skill reading; make prompts more open-ended (don't name exact
APIs like OSCondition, Assert.AreSame); match eval.yaml prompt style
- code-testing-agent: Clarify app complexity to encourage skill pipeline
The pattern: vally prompts that give away exact API names or answers let
the model skip reading the skill. More open-ended prompts referencing
'best practices' trigger skill activation.
* Improve skills and eval prompts for regression scenarios
migrate-xunit-to-xunit-v3 SKILL.md:
- Restructure from 16 steps to 13, merging redundant steps
- Move test platform selection from step 12 to step 4
- Mark steps 6-12 as conditional with prioritization note
- Remove external URL references that caused web_fetch detours
writing-mstest-tests SKILL.md:
- Add Response Guidelines for proportional responses
- Remove redundant Validation checklist and Common Pitfalls table
(all items already covered in workflow steps)
- Reduces token count from 3,305 to 2,944
migrate-mstest-v1v2-to-v3 eval.vally.yaml:
- Relax DataRow rubric to account for MSTest 3.8.0 fixing the
16-parameter constructor limit
writing-mstest-tests eval.vally.yaml:
- Make 3 prompts more open-ended by removing API name hints
(Assert.HasCount, Assert.StartsWith, DynamicData to ValueTuples)
so baseline model cannot answer confidently without the skill
* fix(groom): prune resolved rows, add First Seen column, handle duplicate sections
- Rename 'Status' column to 'Investigation' for clarity
- Add 'First Seen' column showing when finding was first detected
- Remove resolved investigation rows instead of keeping them forever
- Handle duplicate Investigation Results sections by merging into one
- Update health-check workflow to use matching column schema
- Drop resolved rows from health-check table preservation logic
* fix(health-check): remove duplicate Investigation Results section
The health-check and groom workflows both created a '## Investigation
Results' section, causing duplicates in issue #288. The groom owns this
section via its replace-island block — remove it from the health-check
template and update the guideline accordingly.
* fix(health-check): restore Investigation Results as placeholder section
The health-check replaces the entire issue body, so removing the section
entirely caused it to disappear. Instead, keep it as a placeholder that:
- Preserves existing rows from the groom's island markers
- Appends new 'Dispatched' rows for findings dispatched in this run
- Does NOT add island markers itself (the groom adds those)
This ensures the section survives health-check body rewrites while the
groom still owns the detailed content via replace-island.
* fix: address PR review comments on groom/health-check workflows
- Fix duplicate section guidance: clarify replace-island only updates
first occurrence, doesn't auto-remove later duplicates
- Match example dispatched row text with health-check template
- Fix First Seen source: parse from summary line (first seen YYYY-MM-DD)
instead of non-existent **First seen:** field
- Preserve all non-resolved rows (not just Done rows) in health-check
- Fix Step 4.2 to align with prune behavior in Step 4.3
- Add schema migration for legacy 4-column tables in health-check
- Add link to dispatched investigation rows
* fix(groom): use MCP tools with min-integrity:none instead of unauthenticated gh CLI
The previous fix (PR #627) instructed the agent to use 'gh api' for reading
issue comments. However, the gh CLI is NOT authenticated inside the sandbox —
the framework deliberately strips credentials via the 'Clean git credentials'
step. The system prompt also explicitly states 'gh is not authenticated'.
Root cause analysis of run 25481332080 (pre-fix) shows the MCP issue_read
tool DID return comments — only 1 item out of many was filtered by integrity
policy. The agent overreacted to the [Filtered] warning and called
report_incomplete instead of working with the available data.
This fix:
- Sets min-integrity: none under tools.github so the integrity filter
does not remove any comments. This is safe because the workflow only
processes comments from github-actions[bot] (Step 2 filters by author).
- Reverts to MCP issue_read with get_comments (the proper tool for reading
comments in the sandbox).
- Removes gh from bash tools (not usable without auth).
- Adds explicit guidance: if filtered items appear, continue working with
available data — do NOT call report_incomplete or missing_tool.
- Documents that gh CLI is NOT authenticated in the sandbox.
Closes#625
* fix: address review comments - scope repos, bound comment fetch, fix jq guideline
Move assertion-quality, test-gap-analysis, test-smell-detection, and
test-tagging from dotnet-experimental to dotnet-test. These skills have
demonstrated sufficient eval coverage, unique value beyond baseline model
capabilities, and are already integrated into the test-quality-auditor
agent pipeline.
Changes:
- Move skills and tests, dropping the exp- prefix
- Update all cross-references in skill descriptions and routing tables
- Update test-quality-auditor agent and eval scenarios
- Update CODEOWNERS for the moved paths
- Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks)
- Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability
Skills held back for further iteration:
- exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios
- exp-test-maintainability: quality gains offset by token overhead
* Fix groom workflow: use gh api to bypass MCP integrity filter
The MCP gateway's integrity policy intermittently blocks access to issue
comments (report_incomplete: 'Resource issue:dotnet/skills#288 has lower
integrity than agent requires'). This caused issue #625.
Fix: Add 'gh' to allowed bash tools and instruct the agent to use
'gh api --paginate' for fetching comments. This bypasses the MCP gateway
entirely and also provides node_id fields required by hide-comment
(which the MCP issue_read tool did not return).
* Address PR review: fix --paginate JSON, add author filter, narrow guideline scope
* Fix triage workflow: separate label and assignee operations
- Use add_labels tool for labeling (Step 6) instead of update_issue
- Reserve update_issue exclusively for assignee assignment (Step 4)
- Increase update_issue max to 2 for assignment flexibility
- Add explicit workflow_dispatch guidance: agent must pass
issue_number/item_number on all safe-output calls when there
is no triggering issue context
- Fixes issues where owners were mentioned in comments but not
actually assigned to the issue
* Apply suggestions from code review
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Abhitej John