* Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop
WHY
Tools that surface skills (Copilot CLI, Claude Code, Codex, Cursor) read a
plugin's version directly from its checked-in manifest. With no versioning
discipline, a plugin's behavior can change while its advertised version stays
flat, so clients never learn to re-pull, and there is no human-readable signal
of what changed. We want correct, current versions in the repo with minimal
manual work and without bloating the marketplace clone.
WHAT
- Per-plugin semantic versioning via Nerdbank.GitVersioning (NBGV). Each plugin
owns a version.json whose pathFilters exclude the generated manifests and the
version.json itself, so version height tracks real content changes only.
- The computed version is materialized into the checked-in manifests
(plugin.json and .codex-plugin/plugin.json) so every consumer reads a current
value with no build step on their side.
- eng/version/Sync-PluginVersions.ps1 is the single workhorse. It resolves the
set of changed plugins from a git diff, computes each version with nbgv
(predicting the squash-merge height for PRs), and either reports or stamps.
AUTOMATIONS (two, low-touch by design)
- /version-bump: an admin/maintainer comments the command on a PR and the
affected plugins are stamped on the PR branch. Gated on collaborator
permission (admin/write/maintain); forks are rejected before any privileged
step. No other PRs are auto-modified.
- weekly-version-sync: a Monday backstop (and workflow_dispatch) that stamps any
drift on main, opens/updates a single bot PR, and explains the per-plugin
reason. This self-heals anything that merged without a bump.
We deliberately did NOT auto-edit contributor PRs or add a noisy advisory
comment bot; maintainers stay in control and the signal stays clean.
SECURITY (multi-model adversarial review: GPT-5.5 + Gemini 3.1 Pro)
- Supply chain (High, both models): dotnet tool restore would have honored a
nuget.config authored in the PR tree, letting an attacker remap the nbgv
package source to a malicious feed and run code in the privileged
contents:write context. Mitigated with a trusted eng/version/nuget.config
(clear + nuget.org-only + packageSourceMapping), overlaid from main and used
via --configfile so PR-supplied configs are ignored. No nuget.config is
tracked in the repo today, so this path was genuinely exploitable.
- TOCTOU (Medium): /version-bump now checks out the authorized head SHA rather
than the mutable branch name; a racing push fails non-fast-forward, which is
the safe outcome.
- Injection: Set-ManifestVersion uses a MatchEvaluator (not a replacement
string) so a "$"-bearing version cannot re-expand, plus a strict
major.minor.patch guard that throws on a malformed base, leaving manifests
untouched.
- A base-only version.json bump (0.1 -> 0.2) is correctly detected and stamped.
VERIFIED
End-to-end against a real NBGV git harness: content-scoped predict, base-only
bump -> x.y.0, docs-only -> [], weekly drift stamping, malformed-base guard,
and --configfile restore (exit 0). actionlint passes on both workflows.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Copilot review feedback
- Add missing plugins/dotnet-test-migration/version.json so it participates
in versioning (it was the only plugin without one; manifests are at 0.1.0).
- CONTRIBUTING: the two manifests are not byte-identical; say the version is
duplicated across two manifest files instead.
- weekly-version-sync: include version.json in commit attribution so a
base-only bump is explained rather than showing 'no attributable commits'.
- Get-NbgvInfo: capture nbgv stderr and include it in the thrown error so CI
failures are diagnosable, while keeping stdout clean for JSON parsing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Consolidate dotnet-test-frameworks into test-analysis-extensions
Remove the orphaned dotnet-test-frameworks reference skill, whose content
was a duplicate subset of test-analysis-extensions/extensions/dotnet.md.
Nothing actually loaded it. Redirect the remaining references and drop its
eval tests.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Make exp-test-maintainability Step 1 self-contained and fix activation
Address PR review: exp-test-maintainability (dotnet-experimental plugin)
must not point at test-analysis-extensions (dotnet-test plugin) — inline the
.NET framework markers instead. Also strengthen the description/When-to-Use
triggers so the 'each new case needs a whole new method / suggest a better
structure' scenario reliably activates the skill instead of routing to a
test-writing skill.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Revert exp-test-maintainability description broadening; keep self-contained markers
The description broadening regressed activation (3/4 -> 0/4 across a
variance-dominated single eval run, CV up to 924%). Description is the
activation lever, so restore the original wording and keep only the
review-comment fix (inlined .NET framework markers in Step 1).
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix scenario 1 eval: use project fixture so the skill activates
Scenario 1 pasted a self-contained snippet inline and asked to 'suggest a
better structure', which the base model answers directly without loading the
skill -> NOT ACTIVATED with no headroom. Convert it to the project-fixture
pattern used by the activating scenarios (3 & 4): move the repetitive
InputValidatorTests into a Validation.Tests fixture and ask the model to
analyze the project. Mirror the change in eval.vally.yaml.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix scenario 2 eval: use project fixture so the skill activates
Scenario 2 also pasted a self-contained snippet inline (well-maintained
Auth.Tests) and asked to 'review these tests', which the base model handles
without loading the skill. Convert to the project-fixture pattern used by the
activating scenarios; add fixtures/well-maintained/Auth.Tests. Mirror in
eval.vally.yaml. All four scenarios now reference a project on disk.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Add reject_tools to exp-test-maintainability evals to create headroom
All four scenarios scored NOT ACTIVATED / no-headroom (#8 in
InvestigatingResults.md): baselines are already 4.0-5.0/5, so loading the
skill only adds token/tool overhead with no offsetting quality gain.
Match the convention of the sibling analysis skill test-anti-patterns:
reject bash/edit/create on every scenario. This skill is analysis-only
('do not modify any files'), so the constraint is a no-op for correct
behavior but levels the playing field between baseline and skilled runs —
scoring focuses on answer quality (weight 0.40+0.30) instead of tool-induced
overhead. Applied to both eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address review: make well-maintained fixture self-contained and deterministic
- Import System.Security.Authentication for AuthenticationException
- Replace wall-clock assertion (token.ExpiresAt > DateTime.UtcNow) with a
deterministic one anchored on a fixed clock (FixedNow), and drive the
injected FakeClock from that fixed time. This removes the flakiness and
makes the FakeClock injection meaningful — a better exemplar for the
'well-maintained' scenario.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The Codex CLI requires .codex-plugin/plugin.json as the plugin manifest
entry point. Without it, 'codex plugin add' fails with 'missing plugin.json'
even though the marketplace listing works.
This adds .codex-plugin/plugin.json to all 14 plugin directories, with paths
relative to the plugin root per the Codex docs. Also updates the agents
marketplace to use dotnet-aspnetcore (per #711 rename) and adds missing
dotnet-blazor and dotnet11 entries.
Fixes#578Fixes#724
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move assertion-quality, test-gap-analysis, test-smell-detection, and
test-tagging from dotnet-experimental to dotnet-test. These skills have
demonstrated sufficient eval coverage, unique value beyond baseline model
capabilities, and are already integrated into the test-quality-auditor
agent pipeline.
Changes:
- Move skills and tests, dropping the exp- prefix
- Update all cross-references in skill descriptions and routing tables
- Update test-quality-auditor agent and eval scenarios
- Update CODEOWNERS for the moved paths
- Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks)
- Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability
Skills held back for further iteration:
- exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios
- exp-test-maintainability: quality gains offset by token overhead
* Fix dotnet-test-frameworks skill activation for evaluations
The skill had user-invocable: false and a description saying 'DO NOT USE
directly', which prevented the model from activating it during evaluation.
All 6 eval scenarios default to expect_activation: true, causing
SkillNotActivated failures on CI.
- Remove user-invocable: false from both dotnet-test and experimental copies
- Rewrite description with activation keywords matching eval prompts
(assertion mapping, framework identification, skip annotations, exception
assertions, lifecycle conversion, integration test markers)
- Preserve note that other analysis skills also load this as reference data
* Restore user-invocable: false, fix cross-refs, add expect_activation: false
Address review comment: remove exp-* skill references from dotnet-test-
frameworks description since those skills reference exp-dotnet-test-
frameworks instead.
Optimize eval results: this reference data skill's content is knowledge
the model already has (baseline scores 5.0/5 on most scenarios). Direct
activation adds token/time overhead without quality improvement:
- Restore user-invocable: false (correct for reference data skill)
- Shorten description to not trigger activation
- Add expect_activation: false to all 6 scenarios
The skill's value is realized when loaded by consumer skills (e.g.
test-anti-patterns), not via direct user queries.
* Deduplicate test skill references and clarify skill boundaries
- Move platform-detection.md and filter-syntax.md to plugins/dotnet-test/shared/,
removing 3 identical copies of each from run-tests, mtp-hot-reload, and
migrate-vstest-to-mtp reference directories.
- Move dotnet.md from exp-test-smell-detection/extensions/ to shared/ as
dotnet-test-frameworks.md in both dotnet-test and dotnet-experimental plugins.
Update exp-assertion-quality, exp-test-boilerplate-detection, exp-test-tagging,
and test-anti-patterns to reference the shared file instead of inlining
framework detection tables.
- Differentiate test-anti-patterns (quick pragmatic review) from
exp-test-smell-detection (deep formal audit with academic taxonomy) by updating
descriptions and cross-referencing each other in When Not to Use sections.
- Update skill-validator to allow ../../shared/ file references while still
blocking other parent-directory traversals. Add tests for the new rule.
* Switch from shared/ directories to hidden reference skills
Replace the plugin-level shared/ directories with non-invocable reference
skills (user-invocable: false) that other skills reference by name.
- Create platform-detection, filter-syntax, and dotnet-test-frameworks as
hidden skills under plugins/dotnet-test/skills/. These contain the
detection tables and syntax references previously duplicated across
run-tests, mtp-hot-reload, and migrate-vstest-to-mtp.
- Create exp-dotnet-test-frameworks as a hidden skill under
plugins/dotnet-experimental/skills/ for the experimental test analysis
skills (exp-test-smell-detection, exp-assertion-quality, etc.).
- Update all consuming skills to reference these by skill name in backtick
notation instead of file links.
- Revert the skill-validator ../../shared/ exception — no longer needed
since all references now use the standard skill name mechanism.
* Merge exp-test-boilerplate-detection into exp-test-maintainability
exp-test-maintainability was only 6 calibration rules with no workflow.
exp-test-boilerplate-detection had the full 5-category detection workflow,
examples, calibration, and validation. Merge the boilerplate content into
exp-test-maintainability (the broader, more user-facing name) and add the
two unique maintainability rules (DisplayName guidance, DataRow vs
DynamicData preference) to Category 3.
- Replace exp-test-maintainability SKILL.md with the merged content
- Move test fixtures from exp-test-boilerplate-detection to exp-test-maintainability
- Merge eval.yaml scenarios (4 total: 2 from each original skill)
- Delete exp-test-boilerplate-detection skill and tests
- Update all cross-references in exp-test-smell-detection, dotnet-test-frameworks,
exp-dotnet-test-frameworks, and CODEOWNERS
* Add cross-references to test-anti-patterns for deep mock and duplication analysis
Point users to exp-mock-usage-analysis from the Over-mocking entry and
to exp-test-maintainability from the Duplicate tests entry.
* Add exp-dotnet-test-frameworks to CODEOWNERS
* Improve run-tests SDK 10 MTP detection for blame-hang scenario
Inline the critical SDK 10 detection signal (global.json test.runner)
directly in Step 1 instead of deferring entirely to the platform-detection
skill. This makes the distinction between SDK 10 (no -- separator) and
SDK 8/9 (requires -- separator) more prominent.
Add a quick detection summary table, strengthen the Common Pitfalls entry
for SDK 10 with a blame-hang-timeout example, and keep the
platform-detection skill reference for the full detection logic.
* Improve skill activation keywords in descriptions
- exp-test-maintainability: Add 'suggest a better test structure',
'consolidate similar test methods', 'convert copy-paste tests to
data-driven parameterized tests' to match prompts like 'each new case
needs a whole new method, suggest a better structure'.
- test-anti-patterns: Add 'review tests', 'find test problems', 'check
test quality', 'audit tests for common mistakes' to match review-style
prompts that don't use the word 'anti-pattern'.
- run-tests: Add 'hang timeout', 'blame-hang', 'blame-crash', 'TUnit'
to match SDK 10 blame scenarios and TUnit filter scenarios that were
intermittently not activating.
* Add exp-mock-usage-analysis skill and evaluation
New experimental skill that detects unused, redundant, or unnecessary
mocks in .NET test suites. Covers Moq, NSubstitute, FakeItEasy, and
manual test doubles.
- SKILL.md with 6-step workflow: gather code, classify mocks, detect
anti-patterns, incorporate runtime data, generate recommendations,
report findings
- eval.yaml with 10 scenarios: trivial type mocking, unused setups,
redundant configs, stable utility mocks, well-placed mocks (no false
positives), NSubstitute support, FakeItEasy support, setup sprawl,
and 2 non-activation boundary tests
- Fixture files for each scenario with production + test code
- CODEOWNERS entry for @dotnet/dotnet-testing
* Remove no-headroom scenarios, increase NSubstitute timeout
Remove 4 scenarios where baseline already scores 5.0/5 (model handles
these well without skill guidance):
- Detect mocking of DTOs, records, and enums
- Recognize well-placed mocks without inventing false positives
- Decline when asked about non-mock test anti-patterns
- Decline mock framework migration request
Increase NSubstitute scenario timeout from 120s to 180s to avoid
baseline timeouts impacting scoring.
Remaining 6 scenarios all show positive skill impact (3→5, 3→4, etc).
* Document areas the model handles well without the skill
Add a subsection under 'When Not to Use' listing mock-related tasks
where the base model already scores 5.0/5 without skill guidance:
- Mocking trivial types (DTOs, records, enums)
- Recognizing well-placed mocks (no false positives)
* Fix md
* Improve
* Add exp-test-smell-detection skill and evaluation
- Language-agnostic core skill detecting 10 test smells (conditional logic,
mystery guest, sleepy test, assertion-free, eager test, magic numbers,
sensitive equality, exception handling, general fixture, ignored tests)
- .NET extension with framework-specific detection tables (MSTest/xUnit/NUnit/TUnit)
- Reference catalog with 19 smells from testsmells.org research
- Evaluation with 4 scenarios: smelly suite, clean suite, integration tests, non-activation
- CODEOWNERS entry for @dotnet/dotnet-testing
* Improve integration test fixture and non-activation timeout
- Add real smells to integration test fixture (Thread.Sleep, conditional
logic, assertion-free test) mixed with legitimate integration patterns
- Update rubric to test nuanced distinction between real smells and
acceptable integration test patterns
- Increase non-activation scenario timeout from 120s to 300s to prevent
timeout-related scoring penalties
* Fix test projects
* Add domains for testsmells
* Remove extra domains
New experimental skill that performs pseudo-mutation analysis on .NET
production code to find gaps in existing test suites. Covers boundary,
boolean/logic, return value, exception removal, arithmetic, and
null-check mutation categories.
Includes eval.yaml with 4 scenarios (boundary gaps, logic gaps,
well-tested code recognition, non-activation) and fixture projects.
Two new experimental skills for dotnet-experimental plugin:
- exp-test-boilerplate-detection: Detects duplicate boilerplate patterns
across .NET test suites and identifies refactoring opportunities.
Categories: repeated construction, assertion patterns, copy-paste
methods, duplicated setup/teardown, repeated infrastructure.
- exp-assertion-quality: Analyzes assertion variety and depth across
.NET test suites. Produces metrics dashboard including assertion
count, type spread, trivial %, assertion-free tests, and negative
assertion coverage.
Both skills include eval.yaml with 3-4 scenarios each (positive,
negative, and non-activation) plus test fixtures.
New experimental skill that assesses test maintainability: duplication,
test size, data-driven patterns, display names, builder/helper extraction,
and shared setup.
Abhitej John