Closes the 'refactoring-only endpoint too narrow' risk flagged by the
cross-family rubber-duck. Adds two boundary stimuli never used in trim tuning:
9. Separate a smuggled behavior change from a requested rename (rename +
smuggled 10%->12% discount change) -> should separate/flag, not bundle.
10. Recognize a behavior-changing simplification is not a refactor (remove
free-shipping-over-\ rule under a 'cleanup' framing) -> should not
silently change observable behavior.
Executor reverted to claude-opus-4.6 / judge gpt-5.5 (re-judge) to match the
stringent Confirm A family where boundary behavior looked weakest.
Eval-branch only; not on PR #873.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3c9f9823-7f2f-4f7d-9d1b-f2b9e7a20c60
Robustness probe for the csharp-refactoring trim: all prior CI trajectories
were executed by claude-opus-4.6. Flip executor to gpt-5.5 (GPT family) and
CI judge to claude-opus-4.8 so the (executor, judge) pair stays cross-family.
Tests whether trimmed-vs-current non-inferiority holds under a different model
family. Eval-branch only; not on PR #873.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3c9f9823-7f2f-4f7d-9d1b-f2b9e7a20c60
Rewrite the 'not behavior-preserving' escalate bullet so an explicitly
requested bug fix or feature is completed via normal development (not
refused / 'nothing to change'), addressing the one scenario where v1
regressed. Bump experiment runs 3->5 to tighten the aggregate CI.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3c9f9823-7f2f-4f7d-9d1b-f2b9e7a20c60
Trim study for PR #873: csharp-refactoring-trim is a ~30% shorter rewrite of csharp-refactoring (drops When-to-use/Inputs/Outputs boilerplate + verbose LSP launch prose, compresses the dbc-overlapping hazards section) while preserving every rubric-rewarded behavior. Adds a 'trimmed' experiment arm and bumps runs to 3 so CI generates baseline/skilled/trimmed trajectories for local cross-family re-judging. Eval-only scratch branch; not part of the PR.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3c9f9823-7f2f-4f7d-9d1b-f2b9e7a20c60
CI-only eval branch (NOT part of PR #873). Carries the uncommitted
cost-trim + PR-comment fix on both SKILL.md, the eval.yaml rubric
re-alignment, and two new eval.vally.yaml specs so the vally CI can
evaluate whether the current skill content helps before merging.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3c9f9823-7f2f-4f7d-9d1b-f2b9e7a20c60
* Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop
WHY
Tools that surface skills (Copilot CLI, Claude Code, Codex, Cursor) read a
plugin's version directly from its checked-in manifest. With no versioning
discipline, a plugin's behavior can change while its advertised version stays
flat, so clients never learn to re-pull, and there is no human-readable signal
of what changed. We want correct, current versions in the repo with minimal
manual work and without bloating the marketplace clone.
WHAT
- Per-plugin semantic versioning via Nerdbank.GitVersioning (NBGV). Each plugin
owns a version.json whose pathFilters exclude the generated manifests and the
version.json itself, so version height tracks real content changes only.
- The computed version is materialized into the checked-in manifests
(plugin.json and .codex-plugin/plugin.json) so every consumer reads a current
value with no build step on their side.
- eng/version/Sync-PluginVersions.ps1 is the single workhorse. It resolves the
set of changed plugins from a git diff, computes each version with nbgv
(predicting the squash-merge height for PRs), and either reports or stamps.
AUTOMATIONS (two, low-touch by design)
- /version-bump: an admin/maintainer comments the command on a PR and the
affected plugins are stamped on the PR branch. Gated on collaborator
permission (admin/write/maintain); forks are rejected before any privileged
step. No other PRs are auto-modified.
- weekly-version-sync: a Monday backstop (and workflow_dispatch) that stamps any
drift on main, opens/updates a single bot PR, and explains the per-plugin
reason. This self-heals anything that merged without a bump.
We deliberately did NOT auto-edit contributor PRs or add a noisy advisory
comment bot; maintainers stay in control and the signal stays clean.
SECURITY (multi-model adversarial review: GPT-5.5 + Gemini 3.1 Pro)
- Supply chain (High, both models): dotnet tool restore would have honored a
nuget.config authored in the PR tree, letting an attacker remap the nbgv
package source to a malicious feed and run code in the privileged
contents:write context. Mitigated with a trusted eng/version/nuget.config
(clear + nuget.org-only + packageSourceMapping), overlaid from main and used
via --configfile so PR-supplied configs are ignored. No nuget.config is
tracked in the repo today, so this path was genuinely exploitable.
- TOCTOU (Medium): /version-bump now checks out the authorized head SHA rather
than the mutable branch name; a racing push fails non-fast-forward, which is
the safe outcome.
- Injection: Set-ManifestVersion uses a MatchEvaluator (not a replacement
string) so a "$"-bearing version cannot re-expand, plus a strict
major.minor.patch guard that throws on a malformed base, leaving manifests
untouched.
- A base-only version.json bump (0.1 -> 0.2) is correctly detected and stamped.
VERIFIED
End-to-end against a real NBGV git harness: content-scoped predict, base-only
bump -> x.y.0, docs-only -> [], weekly drift stamping, malformed-base guard,
and --configfile restore (exit 0). actionlint passes on both workflows.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Copilot review feedback
- Add missing plugins/dotnet-test-migration/version.json so it participates
in versioning (it was the only plugin without one; manifests are at 0.1.0).
- CONTRIBUTING: the two manifests are not byte-identical; say the version is
duplicated across two manifest files instead.
- weekly-version-sync: include version.json in commit attribution so a
base-only bump is explained rather than showing 'no attributable commits'.
- Get-NbgvInfo: capture nbgv stderr and include it in the thrown error so CI
failures are diagnosable, while keeping stdout clean for JSON parsing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Harden vally evaluation against command injection; source tokens from copilot-pat-pool
Close a command-injection vector where fork-PR-controlled path segments flowed
into a bash run: body via matrix.entry.plugin interpolation in the token-bearing
job.
- Allowlist-validate matrix entries (plugin/name/skills_path) before they enter
the matrix in evaluation.yml, and re-validate defense-in-depth as the first
step of the vally-evaluate job.
- Pass all PR-controlled values through env: vars and reference them quoted; no
raw expression interpolation remains in any run: body.
- Build workflow_dispatch entries with jq --arg instead of string interpolation.
- Replace secrets: inherit with explicit secret passing to reduce blast radius.
- Source Copilot tokens from the copilot-pat-pool environment (COPILOT_PAT_0..9)
via the job environment, with COPILOT_GITHUB_TOKEN as an empty-pool fallback.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address Copilot review: reject empty skills_path and quote dirname
- Fail the matrix-entry validation when skills_path is empty (never a
legitimate state for vally entries) instead of silently skipping the loop.
- Quote the inner $(dirname ...) in EVAL_NAME to avoid word-splitting.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address jeffhandley review: drop PAT fallback, explain regex anchors
- Remove the COPILOT_GITHUB_TOKEN fallback secret from vally-evaluation.yml
(workflow_call.secrets, the token-selection step, and evaluation.yml's
caller-side secrets passing). An empty copilot-pat-pool is now a hard
configuration failure instead of falling back to a separate secret.
- Add short comments explaining why bash's ^/$ anchors are safe here
without the \A/\z equivalents used in the PowerShell discover job.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Harden regex validation against locale-dependent bracket ranges
Add LC_ALL=C before the allowlist regex checks in both the prepare job and
the Validate matrix entry step. POSIX bracket ranges like [A-Za-z] can
behave unexpectedly under non-C locale collation; forcing the C locale
makes the anchor/character-class behavior deterministic regardless of the
runner's default locale. Found via cross-model (GPT-5.5) review of the
prior fixup commit.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address review: constrain vally by skills_path, reject empty skills_path
- vally: select eval specs per matrix leg's skills_path so sharded plugins
run only their own skills (no N-times re-run / PAT-pool token waste)
- evaluation.yml: hard-fail on empty skills_path in the matrix allowlist
- vally: quote inner dirname substitution in the Write summary step
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Reject whitespace-only skills_path in matrix validation
A value like a single space passed the bash `-z` empty check and then
word-split to zero loop iterations, letting an effectively-empty
skills_path slip through. Count word-split segments instead so empty and
whitespace-only both hard-fail. Mirror the intent on the PowerShell
discover side with a trim-based emptiness filter.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Bind skills_path segments to the matrix entry's plugin
The skills_path allowlist previously validated only the generic shape
(plugins/<any>/skills[/<skill>]); it did not require each segment to
belong to the entry's own plugin. A mismatched entry (plugin=foo +
skills_path=plugins/bar/skills) would pass, and downstream steps assume
the plugins/$PLUGIN/skills prefix.
Add a literal (non-regex) prefix binding in both validators:
- vally-evaluation.yml: [[ seg != prefix && seg != prefix/* ]] with
skills_prefix="plugins/$ENTRY_PLUGIN/skills".
- evaluation.yml: case-sensitive -cne plus Ordinal StartsWith against
"plugins/$($e.plugin)/skills".
Literal comparison avoids regex-metachar pitfalls from '.'/'-' in plugin
names; -cne/Ordinal keeps it case-sensitive to match Linux path
semantics. Legitimate entries are always same-plugin, so this rejects
no valid input.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Reduce evaluation fan-out to avoid Copilot rate limits
Group PR evaluation entries by plugin and preserve existing
executionShard bucketing so large PRs stop launching one skill-validator
job per changed skill. Cap PR matrix concurrency and keep PR grouped legs
at parallel-skills=1 so each assigned Copilot token stays below the
rate-limit threshold.
Keep vally on the proven reusable-workflow matrix pattern, cap its
concurrency, and only synthesize manual entries when inputs.entries is
empty so workflow_call runs use the caller-provided grouped entries.
Also skip agent.* and eval-less skills when constructing vally
--eval-filter arguments.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Make vally full runs work before the experiment file lands
PR 872 is intentionally based on main only, but main does not have the
new repo-root dotnet-skills.experiment.yaml yet. The vally workflow was
hard-coded to that file, so full branch-level runs from PR 872 failed
before starting any trials.
When the repo file is absent, synthesize an equivalent temporary
experiment file in the runner and use that for read_override() and vally
experiment run. That keeps PR 872 self-contained while preserving the
same per-skill skilled-vs-baseline experiment shape.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Add Code of Conduct document based on Contributor Covenant
* Rename CODE-OF-CONDUCT.md to CODE_OF_CONDUCT.md for GitHub auto-detection
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* fix: reinforce no-script-execution constraint in devops-health-check
The agent was hallucinating Python script creation (run_analysis2.py)
and attempting to execute it, which was correctly blocked by the bash
allowlist security policy. Strengthen the efficiency guideline to
explicitly state that python/python3/node are NOT in the bash
allowlist and will be blocked, directing the agent to use jq instead.
Fixes#841
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: narrow wording to 'general-purpose language runtimes' per review
Addresses review feedback: 'any script interpreter' was too broad since
jq/sed are technically interpreters but ARE in the allowlist.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sharpen the description boundary between migrate-static-to-wrapper and
generate-testability-wrappers so prompts that replace call sites when the
abstraction is already registered in DI activate the migration skill.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The 'Decline request to write new tests from scratch' non-activation
scenario writes a full MSTest suite from scratch (code generation) and
was hitting its 300s wall-clock timeout. Per InvestigatingResults.md,
code-generation scenarios often need 360s.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The 'Use comparison assertions for boundary testing' scenario chronically hit its 240s wall-clock timeout. Raise it to 360s to match the other code-generation scenarios in this eval file, per InvestigatingResults.md timeout guidance.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Scale report depth to suite size so a tiny test file gets a focused,
direct answer instead of a padded multi-section dashboard. Preserves
full-template behavior for substantial suites and keeps all
rubric-relevant substance (assertion-free/trivial identification,
quality verdict, concrete recommendations).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The 'Suggest hot reload for failing test in MTP project (SDK 9)' vally
stimulus forced bash usage on a purely advisory question, triggering a
slow SDK 9.0.200 NuGet restore that exceeded the timeout. Align with the
source-of-truth eval.yaml which does not constrain this scenario.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* dotnet-test: make code-testing agent tools declarations Claude Code-compatible
PR #847 added `tools: ["agent", "skill", "read", "search", "edit", "execute"]`
to the code-testing-* agents to enable VS Code / Copilot CLI subagent fan-out.
Those lowercase aliases map to real tools in VS Code and the Copilot CLI, but
Claude Code matches `tools:` against its own vocabulary (Task, Skill, Read,
Glob, Grep, Edit, Write, Bash). None of the aliases matched, so when these
agents are loaded into Claude Code via --plugin-dir and selected with
`claude --agent`, the agent was granted ZERO tools. A tool-less model asked
to generate tests emits a textual <tool_call> block and exits after one turn,
producing no file changes.
Append the Claude Code tool names to each agent's `tools:` list so the same
declaration works across all three runtimes (each honors the names it knows and
ignores the foreign ones):
- Orchestrators (generator, implementer): add Task, Skill, Read, Glob, Grep,
Edit, Write, Bash (Task is the Claude Code equivalent of the `agent`
fan-out tool).
- Workers (researcher, planner, builder, tester, fixer, linter): add Skill,
Read, Glob, Grep, Edit, Write, Bash.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* skill-validator: complete built-in tools + add cross-host tool portability check
Two related follow-ups to the agent tools fix:
1. Address the skill-check review feedback. The validator's BuiltInTools set was
missing three legitimate host tool spellings that are not case-insensitive
matches of existing entries, so they were flagged as non-built-in:
- "write" — Claude Code file-creation tool (Copilot CLI / VS Code: "create")
- "agent" — Copilot CLI / VS Code subagent fan-out tool (Claude Code: "task")
- "execute" — Copilot CLI / VS Code run-command tool (Claude Code: "bash")
"agent" and "execute" were already flagged before this branch (introduced by
the fan-out PR); adding them to BuiltInTools clears the pre-existing warnings.
2. Add a cross-host tool portability check (CheckAgentToolPortability) so an
agent that declares a capability for only one host is flagged. Tool names are
matched case-sensitively (hosts resolve tools by exact spelling), so an agent
that lists e.g. only "edit" (Copilot / VS Code) without "Edit"/"Write"
(Claude Code) is reported as working on one host and silently tool-less on the
other. Findings are advisory (do not fail CI) and allowlistable via
"agent-tool-portability:AGENT:capability". Wired into the agents loop in
CheckCommand and covered by unit tests.
Also make the one existing single-host agent (optimizing-dotnet-performance)
portable by adding its Claude Code tool spellings, so the new check reports a
clean tree.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Several dotnet-test eval scenarios repeatedly hit their wall-clock timeout
in the evaluation dashboard (skilled + plugin arms), producing empty output
and failing all assertions. Increase per-scenario timeouts for the ones that
timed out most often across recent runs:
- writing-mstest-tests: 'Write unit tests for a service class',
'Write data-driven tests for a calculator',
'Use string assertions for format validation' 180->360;
'Use comparison assertions for boundary testing' 120->240
- coverage-analysis: 'Coverage plateau diagnosis',
'Project-wide coverage analysis with existing Cobertura data' 300->480
- test-gap-analysis: 'Decline request to write new tests from scratch' 120->300;
'Acknowledge well-tested code with few surviving mutations' 300->420
- run-tests: 'Filter xUnit v3 tests by trait on MTP',
'Filter xUnit v3 tests by class pattern and trait using query filter language' 120->240
Per eng/skill-validator/src/docs/InvestigatingResults.md, timeouts are the
highest-priority failure mode; code-generation scenarios often need 360s.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Consolidate dotnet-test-frameworks into test-analysis-extensions
Remove the orphaned dotnet-test-frameworks reference skill, whose content
was a duplicate subset of test-analysis-extensions/extensions/dotnet.md.
Nothing actually loaded it. Redirect the remaining references and drop its
eval tests.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Make exp-test-maintainability Step 1 self-contained and fix activation
Address PR review: exp-test-maintainability (dotnet-experimental plugin)
must not point at test-analysis-extensions (dotnet-test plugin) — inline the
.NET framework markers instead. Also strengthen the description/When-to-Use
triggers so the 'each new case needs a whole new method / suggest a better
structure' scenario reliably activates the skill instead of routing to a
test-writing skill.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Revert exp-test-maintainability description broadening; keep self-contained markers
The description broadening regressed activation (3/4 -> 0/4 across a
variance-dominated single eval run, CV up to 924%). Description is the
activation lever, so restore the original wording and keep only the
review-comment fix (inlined .NET framework markers in Step 1).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix scenario 1 eval: use project fixture so the skill activates
Scenario 1 pasted a self-contained snippet inline and asked to 'suggest a
better structure', which the base model answers directly without loading the
skill -> NOT ACTIVATED with no headroom. Convert it to the project-fixture
pattern used by the activating scenarios (3 & 4): move the repetitive
InputValidatorTests into a Validation.Tests fixture and ask the model to
analyze the project. Mirror the change in eval.vally.yaml.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix scenario 2 eval: use project fixture so the skill activates
Scenario 2 also pasted a self-contained snippet inline (well-maintained
Auth.Tests) and asked to 'review these tests', which the base model handles
without loading the skill. Convert to the project-fixture pattern used by the
activating scenarios; add fixtures/well-maintained/Auth.Tests. Mirror in
eval.vally.yaml. All four scenarios now reference a project on disk.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Add reject_tools to exp-test-maintainability evals to create headroom
All four scenarios scored NOT ACTIVATED / no-headroom (#8 in
InvestigatingResults.md): baselines are already 4.0-5.0/5, so loading the
skill only adds token/tool overhead with no offsetting quality gain.
Match the convention of the sibling analysis skill test-anti-patterns:
reject bash/edit/create on every scenario. This skill is analysis-only
('do not modify any files'), so the constraint is a no-op for correct
behavior but levels the playing field between baseline and skilled runs —
scoring focuses on answer quality (weight 0.40+0.30) instead of tool-induced
overhead. Applied to both eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address review: make well-maintained fixture self-contained and deterministic
- Import System.Security.Authentication for AuthenticationException
- Replace wall-clock assertion (token.ExpiresAt > DateTime.UtcNow) with a
deterministic one anchored on a fixed clock (FixedNow), and drive the
injected FakeClock from that fixed time. This removes the flakiness and
makes the FakeClock injection meaningful — a better exemplar for the
'well-maintained' scenario.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Bump dotnet-test plugin to 0.2.0
Signals the VS Code subagent fan-out fix (#847) to existing installs.
Minor bump; no breaking changes.
* Sync .codex-plugin manifest to 0.2.0
Keeps plugins/dotnet-test/.codex-plugin/plugin.json in sync with the main
plugin.json, matching the convention other plugins follow (e.g. dotnet-ai).
Addresses PR review feedback.
Claude Code discovers MCP servers via .claude-plugin/plugin.json inside
each plugin directory. The existing .codex-plugin/ manifest only works
for Codex CLI. Add the Claude Code-specific manifest with inline
mcpServers config, and also declare mcpServers in the root marketplace
manifest entry for belt-and-suspenders coverage.
Fixes#840
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add VS Code subagent metadata to dotnet-test code-testing agents
Enable the code-testing-* Research-Plan-Implement pipeline to fan out as
subagents in VS Code while keeping the GitHub Copilot CLI working.
Frontmatter (VS Code "coordinator/worker" pattern; portable tool aliases that
map in both VS Code and the CLI):
- code-testing-generator: tools: [agent, read, search, edit, execute] + an
agents: list (researcher, planner, implementer, builder, tester, fixer,
linter); softened the three runSubagent({ agent, prompt }) blocks to
tool-agnostic delegation prose.
- code-testing-implementer: tools: [agent, read, search, edit, execute] +
agents: (builder, tester, fixer, linter).
- Leaf agents (researcher/planner/builder/tester/fixer/linter):
tools: [read, search, edit, execute].
Why explicit tools (not ["*"]): VS Code has no all-tools wildcard for ools:
and a subagent's ools: overrides its inherited set, so ["*"] matched nothing
and stripped subagents of file tools (they ran without read/edit/search). The
CLI treats ["*"] as all-tools, so this was VS-Code-specific. The portable
aliases agent/read/search/edit/execute map to real tools in both environments;
agents: is ignored by the CLI.
README: document the VS Code chat.subagents.allowInvocationsFromSubagents
setting (off by default) needed for the nested implementer->builder/tester/
fixer/linter layer to fan out on large scopes; the CLI has no such gate.
Validated end to end: VS Code shows researcher->planner->implementer fanning out
with real file I/O (no "subagents lack file tools" warning); CLI fan-out intact
with skills still loading and tests passing. An all-tools baseline used only
tools within this enumerated set, confirming no CLI capability is restricted.
* Include skill tool in code-testing agent tool allowlists
Address PR review: the explicit `tools:` allowlists omitted the `skill`
tool, but every code-testing-* agent's prompt instructs calling skills
(e.g. `code-testing-extensions` for per-language guidance, `test-gap-analysis`,
`assertion-quality`). Because `tools:` is an override, omitting `skill` can
prevent the agents from loading those skills in environments that gate skill
invocation by the allowlist.
Add `skill` to all eight agents:
- orchestrators (generator, implementer): [agent, skill, read, search, edit, execute]
- workers (researcher/planner/builder/tester/fixer/linter): [skill, read, search, edit, execute]
Re-verified in the Copilot CLI: full fan-out (researcher -> planner ->
implementer/tester), the `code-testing-extensions` skill is invoked, and the
generated tests pass.
* Add csharp-development skill and update CODEOWNERS and README.md
- Introduced a new skill for C# development guidance.
- Updated CODEOWNERS to include reviewers for the new skill.
- Modified README.md to list the new csharp-development skill.
- Added evaluation scenarios for the csharp-development skill.
* Refactor C# evaluation scenarios to enhance production code review and validation guidance
* Refactor production code review scenarios to improve clarity and focus on async error handling
* Update CODEOWNERS to reflect current ownership and streamline reviewer assignments
* Remove csharp-development skill documentation and evaluation tests
* Remove csharp-development skill from README
* Fix formatting in Constants.g.cs by adding missing semicolon
* Revert Constants.g.cs to main
* Remove csharp-development skill from CODEOWNERS
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update CODEOWNERS to include additional reviewers for dotnet-ai skills
* Move dotnet skills to dotnet-util to keep dotnet plugin focused and targeted.
* Add CODEOWNERS, SKILL.md, package-types.md, and publish-workflow.md for NuGet trusted publishing; implement eval.yaml for dotnet-pinvoke and dotnet-util skills
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update README files to enhance plugin descriptions and remove outdated LSP section
* Rename repository title from 'dotnet' to 'dotnet-util' in README.md for clarity.
* Refine README.md to clarify included .NET skills and remove redundant descriptions
* Add 'dotnet-util' plugin for utility skills in .NET development
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add 'dotnet-util' plugin for utility skills in .NET development
* Add dotnet-util to Codex marketplace manifest
* Align dotnet-util marketplace descriptions
* Refactor CODEOWNERS and update README for nuget-trusted-publishing; add evaluation scenarios and workflow template for trusted publishing
* Add P/Invoke diagnostics and type mapping references; implement NuGet trusted publishing setup
- Created diagnostics.md for common pitfalls and failure modes in P/Invoke.
- Added type-mapping.md for native-to-.NET type mappings.
- Introduced nuget-trusted-publishing skill for setting up OIDC-based NuGet publishing.
- Added package-types.md detailing structural requirements for various NuGet package types.
- Created publish-workflow.md as a template for GitHub Actions workflows for NuGet publishing.
* Add evaluation YAML files for csharp-scripts, dotnet-pinvoke, and nuget-trusted-publishing skills
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Complete nuget-trusted-publishing move: remove from dotnet, add to dotnet-specialized README
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Remove eval.vally.yaml and eval.yaml files for nuget-trusted-publishing skill
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add NuGet trusted publishing skill and related references
- Introduced `nuget-trusted-publishing` skill for setting up OIDC-based trusted publishing on GitHub Actions.
- Added detailed documentation in SKILL.md covering prerequisites, process, safety rules, and troubleshooting.
- Created `package-types.md` to outline structural requirements for various NuGet package types.
- Developed `publish-workflow.md` as a template for GitHub Actions workflows for NuGet publishing.
- Removed obsolete `dotnet-specialized` plugin and updated `dotnet` plugin description.
- Added evaluation tests for `nuget-trusted-publishing` skill to ensure proper guidance for users.
- Implemented evaluation tests for `dotnet-pinvoke` and `csharp-scripts` skills to validate functionality.
* Update dotnet plugin description to include high-level .NET development skills
* Add NuGet trusted publishing and P/Invoke documentation
- Introduced type mapping reference for native-to-.NET types in dotnet-pinvoke.
- Added a comprehensive guide for setting up NuGet trusted publishing using OIDC in GitHub Actions.
- Created a reference document detailing structural requirements for various NuGet package types.
- Provided a template for GitHub Actions workflow for publishing NuGet packages with trusted publishing.
- Implemented evaluation scenarios for testing the dotnet-pinvoke and nuget-trusted-publishing skills, including prompts and assertions for expected outputs.
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Add CODEOWNERS entries for setup-local-sdk
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update dotnet README to list setup-local-sdk skill
* Add setup-local-sdk to skills section in README
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Add TFM-aware buildTransitive->build forwarding guidance to msbuild skills
buildTransitive/*.props should forward through the corresponding build/*.props (ownership chain buildTransitive -> build -> shared) rather than importing buildMultiTargeting/ directly. When build/ is packed per-TFM (build/<tfm>/), the forwarder must include the TFM segment and derive it from the file own folder, not $(TargetFramework) (NuGet nearest-match can serve a different asset folder), otherwise transitive consumers hit MSB4019.
Updates extension-points (new Forwarding chain section), msbuild-antipatterns AP-13, and the msbuild-code-review agent. Lesson learned from microsoft/testfx#9431.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clarify ambiguous props/targets glob in forwarder review check
Address review feedback on dotnet/skills#836: spell out ".props/.targets forwarders" instead of the ambiguous `buildTransitive/*.props|targets`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten forwarding-chain guidance to reduce skill token overhead
Condense the new extension-points Forwarding chain section (+26 -> +11 lines) and the AP-13 note by dropping the redundant non-TFM example and self-evident derivation explanation, keeping the chain rule, MSB4019 cause, and the TFM derivation expression. Lower token footprint addresses the skill-validator weighted-score token penalty without losing substance.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review nits: AP range and forward-slash path
Update the additional anti-patterns range to AP-16 through AP-22 (the reference doc now includes AP-22), and use a forward-slash build/MyPackage.props in the forwarding-chain prose to match the build/<tfm>/ convention used in the section.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Condense AP-13 forwarding note to a cross-reference to reduce token overhead
The full TFM-forwarding guidance lives in extension-points; AP-13 only needs a concise pointer. Removes duplication and trims the msbuild-antipatterns skill footprint flagged by the skill-validator token penalty.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Address remaining review nits: targets forwarders, single-source wording, quoted MSBuildThisFileDirectory
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Adds eval.vally.yaml for all 9 dotnet-blazor skills, converted from the legacy skill-validator eval.yaml specs, and registers a dotnet-blazor suite in .vally.yaml so the shadow Vally pipeline discovers them.
Conversion preserves prompts, assertions, and rubric verbatim. Legacy setup blocks map to Vally environment config: 'dotnet new' scaffolding commands become environment.commands. Verified end-to-end by running the author-component eval (baseline + skilled + adapt), which passed the shadow threshold.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add MSTest deflection to assertion-quality and test-anti-patterns skills
The writing-mstest-tests skill failed to activate in the plugin arm for
six MSTest-specific scenarios (fix swapped Assert.AreEqual, modernize
legacy patterns, type/string assertions, DynamicData with ValueTuples)
because sibling skills captured the routing. assertion-quality and
test-anti-patterns matched these prompts but lacked explicit deflection
to writing-mstest-tests. Add DO NOT USE entries pointing MSTest test
writing/fixing/modernizing to writing-mstest-tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: restore general assertion-fix deflection, align test-anti-patterns
- assertion-quality: restore the general 'fixing or rewriting assertions'
deflection (dropped in the prior edit) alongside the MSTest deflection, and
fix punctuation to use comma-separated DO NOT USE items.
- test-anti-patterns: align the 'writing new tests' deflection to also point
MSTest test-writing to writing-mstest-tests, matching the body guidance.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Require environment input to pat_pool shared workflow.
- Updates the PAT pool shared workflow to latest version
- Updates the validate-pat-pool workflow to match other repos
- Updates all agentic workflows to use the updated PAT pool import
with the standard secret names used across other repos
- The new version of the import does not require the 'needs' workaround
but the issue-triage workflow's 'roles: all' configuration requires
a different workaround to ensure the pre_activation job exists for
the pat_pool job to be able to depend on it (necessary for ordering).
* Update markdown bullet style
* Apply Copilot PR feedback
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Bump versions to match gh-aw
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* skill-validator: restore 15K aggregate cap, document it as the real Copilot CLI skill-menu budget
The per-plugin aggregate description cap had been raised 15,000 -> 20,000
-> 22,000 under the belief that 15K was 'a local repo policy, NOT a
documented Copilot constraint'. That belief was wrong: the GitHub Copilot
CLI renders the model-facing <available_skills> menu under a hard 15,000-
char budget (the agent SDK's SKILL_CHAR_BUDGET, default 15e3, confirmed in
CLI 1.0.36 and 1.0.61). Skills are listed alphabetically and emitted with
their full <description> only until the budget is exhausted; every skill
past the cut-off collapses to a bare name with no description and can no
longer be reliably model-activated. Raising the validator cap merely
masked this silent menu truncation — e.g. dotnet-test's run-tests and
test-* skills stopped activating in plugin eval runs because they fell
into the name-only overflow.
Changes:
- SkillProfiler.MaxAggregateDescriptionLength: 22,000 -> 15,000, with the
comment rewritten to document the real Copilot CLI budget (and correct
the prior 'not a documented constraint' claim).
- CheckCommand aggregate now excludes skills marked
'disable-model-invocation: true' — the CLI drops those from the menu, so
they do not consume the budget. This makes the cap satisfiable by hiding
reference / agent-orchestrated primitives rather than only by trimming.
- InvestigatingResults.md: document plugin-arm-only non-activation caused
by skill-menu budget overflow, and how to fix it.
Note: dotnet-test currently exceeds 15K and must be slimmed below it
(via disable-model-invocation on reference/primitive skills plus
description trims) before this cap can go green repo-wide.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: use source-generated regex for disable-model-invocation check
Address review: replace Regex.IsMatch(pattern-string) with a
[GeneratedRegex] partial method (AOT-friendly, no per-call cache lookup),
matching FrontmatterParser's style. Runs once per skill during checks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: parse disable-model-invocation via YAML to avoid block-scalar false positives
The regex-based check matched any line in the frontmatter, so a block-scalar description that merely mentioned 'disable-model-invocation: true' on its own line was wrongly treated as disabling model invocation. Parse the frontmatter with the existing YAML deserializer (which correctly handles block scalars) by adding a DisableModelInvocation field to SkillFrontmatter, and drop the regex entirely.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add eval coverage for dotnet-test/writing-mstest-tests
Enrich the ServiceRegistry collection/null/reference scenario with deterministic file_contains assertions for the modern MSTest assertion APIs Assert.IsNull, Assert.AreSame, Assert.Contains, Assert.DoesNotContain, Assert.IsEmpty, and Assert.IsNotEmpty.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reject legacy CollectionAssert/StringAssert helpers in collection scenario
Address PR review: file_contains is a case-sensitive substring check, so the Assert.Contains and Assert.DoesNotContain value checks would also pass for CollectionAssert.* and StringAssert.* helpers. Add file_not_contains guards to keep the scenario a deterministic check for the modern Assert.* API.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add eval coverage for dotnet-test/dotnet-test-frameworks
Add a deterministic output_matches assertion for Assert.AreEqual in the MSTest try/catch refactor scenario, covering the previously-uncovered MSTest assertion CodePattern.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten Assert.AreEqual assertion to require message comparison
Address review feedback: the output_matches pattern now requires the exception message comparison (Assert.AreEqual(..., ex.Message)) rather than matching Assert.AreEqual anywhere. The CodePattern remains covered via the rubric evidence.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enrich the NUnit misleading-names scenario with rubric items and an output_matches assertion covering the [Category] code pattern, reading the test body before classifying, and classifying each test method. Add a Go standard-testing scenario (report-only) with a fixture to cover leaving source files unmodified for report-only frameworks.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Abhitej John