* Make dotnet-test analysis skills and auditor agent polyglot
Five test analysis skills and the test-quality-auditor agent in
plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/
TUnit cues, even though the underlying techniques (assertion diversity,
anti-pattern detection, pseudo-mutation gap analysis, academic smell
catalog, trait tagging) apply equally to other test frameworks.
This change makes them polyglot across .NET, Python (pytest/unittest),
TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG),
Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin
(JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest).
Changes
-------
* New reference skill `test-analysis-extensions` with per-language
extension files (dotnet, python, typescript, java, go, ruby, rust,
swift, kotlin, powershell, cpp). Each documents test markers,
assertion APIs, sleep / time / random APIs, skip annotations,
setup/teardown, mystery-guest indicators, integration markers, and
tag-support capability for the corresponding framework.
* `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`,
`test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter
now lists all polyglot frameworks (keeping .NET trigger words for
backward compat); bodies reference `test-analysis-extensions` and
add per-language calibration (e.g., Go/Rust table-driven loops are
idiomatic and not Conditional Test Logic; pytest bare `assert` is
canonical; missing-await on async assertions is a new Critical smell).
* `test-quality-auditor` agent now runs language detection, gates
.NET-only pipeline steps (coverage-analysis, CRAP, detect-static-
dependencies, testability migration, experimental dotnet skills), and
recommends native tooling for non-.NET projects via an explicit
Capability Matrix.
* README updated to split sections by polyglot vs .NET-only.
* New polyglot eval scenarios added (Python/pytest, TS/Jest,
Java/JUnit) alongside existing C# scenarios in
`test-anti-patterns`, `assertion-quality`,
`test-smell-detection`.
The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis,
crap-score, detect-static-dependencies, run-tests, filter-syntax,
platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are
unchanged and continue to activate only for .NET work.
Validated with skill-validator `check --plugin ./plugins/dotnet-test`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #707 review feedback
* typescript.md: replace xpect.fail() recommendation (Jest has no
such API) with hrow new Error(...) / explicit failing assertion.
* test-tagging SKILL.md description: clarify that unittest has no
canonical tag syntax (pytest markers are the auto-edit mechanism;
unittest is report-only). Switch doctest reference to "decorators"
rather than implying a Catch2-style [tags] mechanism exists.
* test-tagging SKILL.md auto-edit list and table: use
`* doctest::test_suite("tag")` decorator chain syntax for doctest
in both the bullet list and the per-framework table, matching the
example already shown.
* cpp.md extension: spell out the Catch2 vs doctest tagging
distinction in the capability matrix (Catch2 [tag], doctest
* doctest::test_suite("tag") decorator).
Validated with `dotnet run --project eng/skill-validator/src -- check
--plugin ./plugins/dotnet-test` (23 skills + 11 agents).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Both skills suffer plugin-mode activation failures: in plugin runs the model
bypasses the skill tool entirely for ~5/6 of test-anti-patterns audit prompts
and ~2/3 of test-smell-detection prompts, answering directly from training.
Variant B repositions the two skills as umbrella + narrow specialist:
* test-anti-patterns becomes THE umbrella audit skill for all pragmatic
test-quality reviews (anti-patterns and smell-style prompts alike). Its
description explicitly absorbs smell-audit triggers.
* test-smell-detection is repositioned as a narrow niche that only fires
when the user explicitly asks for the testsmells.org / academic 19-smell
catalog with citable smell names from the research literature.
Eval prompts for test-smell-detection are rewritten to match the narrow
niche (explicit testsmells.org / 19-smell catalog requests) so we test
specialist activation rather than the umbrella scenarios.
Variant B of a 3-PR experiment to fix activation. Companion PRs:
* Variant A = activation-only rewrite, both skills keep overlap;
* Variant C = full merge into test-anti-patterns.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Multi-model skill-validator eval (claude-haiku-4.5, claude-opus-4.6,
gpt-5.3-codex, gpt-5.4-mini) showed two issues with this skill:
1. Quality regression on stronger models (+0.15 haiku, -0.14 opus,
-0.17 codex, -0.29 gpt-5.4-mini): the skill added tokens and tool
calls without lifting rubric quality on larger models.
2. Plugin-mode activation loss: the skill activated reliably in
isolated mode but was almost never picked in plugin mode where it
competed with siblings (code-testing-agent, etc.).
Changes:
- Trim BPE tokens 2,487 -> 1,472 (still detailed tier, well within
the 800-2,500 sweet spot per SkillsBench guidance). Collapsed long
smell-by-smell prose, `Why Test Smells Matter`, and pitfalls into a
compact taxonomy table plus a short decision procedure / output
checklist. Removed bulky examples. Kept taxonomy, calibration,
negative guidance, and references intact.
- Add prompt-shaped USE FOR triggers drawn from the eval prompts
(`review test smells`, `review these tests and tell me if there are`
`any problematic patterns`, `check my tests for test design`
`problems or anti-patterns`, `give us an objective assessment of`
`our integration tests`, plus explicit smell names).
- Strengthen DO NOT USE FOR clause: `write new tests / scaffold tests`
`from scratch / generate a complete MSTest test suite` now defers
cleanly to code-testing-generator (covers the negative eval scenario).
Validation: skill-validator check --plugin clean for this skill;
build + 545/545 tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move assertion-quality, test-gap-analysis, test-smell-detection, and
test-tagging from dotnet-experimental to dotnet-test. These skills have
demonstrated sufficient eval coverage, unique value beyond baseline model
capabilities, and are already integrated into the test-quality-auditor
agent pipeline.
Changes:
- Move skills and tests, dropping the exp- prefix
- Update all cross-references in skill descriptions and routing tables
- Update test-quality-auditor agent and eval scenarios
- Update CODEOWNERS for the moved paths
- Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks)
- Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability
Skills held back for further iteration:
- exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios
- exp-test-maintainability: quality gains offset by token overhead
Amaury Levé