* Make dotnet-test analysis skills and auditor agent polyglot
Five test analysis skills and the test-quality-auditor agent in
plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/
TUnit cues, even though the underlying techniques (assertion diversity,
anti-pattern detection, pseudo-mutation gap analysis, academic smell
catalog, trait tagging) apply equally to other test frameworks.
This change makes them polyglot across .NET, Python (pytest/unittest),
TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG),
Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin
(JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest).
Changes
-------
* New reference skill `test-analysis-extensions` with per-language
extension files (dotnet, python, typescript, java, go, ruby, rust,
swift, kotlin, powershell, cpp). Each documents test markers,
assertion APIs, sleep / time / random APIs, skip annotations,
setup/teardown, mystery-guest indicators, integration markers, and
tag-support capability for the corresponding framework.
* `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`,
`test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter
now lists all polyglot frameworks (keeping .NET trigger words for
backward compat); bodies reference `test-analysis-extensions` and
add per-language calibration (e.g., Go/Rust table-driven loops are
idiomatic and not Conditional Test Logic; pytest bare `assert` is
canonical; missing-await on async assertions is a new Critical smell).
* `test-quality-auditor` agent now runs language detection, gates
.NET-only pipeline steps (coverage-analysis, CRAP, detect-static-
dependencies, testability migration, experimental dotnet skills), and
recommends native tooling for non-.NET projects via an explicit
Capability Matrix.
* README updated to split sections by polyglot vs .NET-only.
* New polyglot eval scenarios added (Python/pytest, TS/Jest,
Java/JUnit) alongside existing C# scenarios in
`test-anti-patterns`, `assertion-quality`,
`test-smell-detection`.
The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis,
crap-score, detect-static-dependencies, run-tests, filter-syntax,
platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are
unchanged and continue to activate only for .NET work.
Validated with skill-validator `check --plugin ./plugins/dotnet-test`.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #707 review feedback
* typescript.md: replace xpect.fail() recommendation (Jest has no
such API) with hrow new Error(...) / explicit failing assertion.
* test-tagging SKILL.md description: clarify that unittest has no
canonical tag syntax (pytest markers are the auto-edit mechanism;
unittest is report-only). Switch doctest reference to "decorators"
rather than implying a Catch2-style [tags] mechanism exists.
* test-tagging SKILL.md auto-edit list and table: use
`* doctest::test_suite("tag")` decorator chain syntax for doctest
in both the bullet list and the per-framework table, matching the
example already shown.
* cpp.md extension: spell out the Catch2 vs doctest tagging
distinction in the capability matrix (Catch2 [tag], doctest
* doctest::test_suite("tag") decorator).
Validated with `dotnet run --project eng/skill-validator/src -- check
--plugin ./plugins/dotnet-test` (23 skills + 11 agents).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
The four agents (code-testing-generator, test-migration, testability-migration,
test-quality-auditor) were the only dotnet-test agents declaring a 'tools:'
list. Per Jan's suggestion on #660, drop the field so they inherit the runtime's
default tool surface (matching the convention used by every other agent in the
repo).
Follow-up to #660.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The skill-validator only recognizes 'bash' and 'powershell' as built-in shell tools, not 'terminal'. Update the four affected agents in the dotnet-test plugin to use both for cross-platform support.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move assertion-quality, test-gap-analysis, test-smell-detection, and
test-tagging from dotnet-experimental to dotnet-test. These skills have
demonstrated sufficient eval coverage, unique value beyond baseline model
capabilities, and are already integrated into the test-quality-auditor
agent pipeline.
Changes:
- Move skills and tests, dropping the exp- prefix
- Update all cross-references in skill descriptions and routing tables
- Update test-quality-auditor agent and eval scenarios
- Update CODEOWNERS for the moved paths
- Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks)
- Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability
Skills held back for further iteration:
- exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios
- exp-test-maintainability: quality gains offset by token overhead
The test-anti-patterns skill was not reliably activated in plugin mode
because sibling descriptions overlapped:
- test-quality-auditor agent: narrowed to multi-skill audit pipelines
only, explicitly excluding single-file reviews
- writing-mstest-tests skill: removed 'review' and 'anti-patterns' from
description, added explicit exclusion for test quality auditing
- test-anti-patterns skill: expanded trigger phrases to better match
review prompts (flaky tests, duplication, tests that pass without
verifying)
* Add test-quality-auditor hub agent for .NET test suite analysis
Introduces a new test-quality-auditor agent that triages test quality
concerns to specialized analysis skills: assertion quality, test smells,
anti-patterns, mock usage, test gaps, maintainability, coverage risk,
and test tagging. Includes a comprehensive audit pipeline that chains
multiple skills for broad health checks.
* Address Copilot review: clarify diagnostic role and coverage routing
- Reword intro to acknowledge file-modifying workflows (test tagging)
are only used with explicit user confirmation
- Clarify coverage-analysis vs crap-score: prefer coverage-analysis for
project-wide analysis, reserve crap-score for targeted method/class scope
Amaury Levé