mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00

Files

T

Amaury Levé 680292e04c Make dotnet-test analysis skills and auditor agent polyglot (#707 )

* Make dotnet-test analysis skills and auditor agent polyglot

Five test analysis skills and the test-quality-auditor agent in
plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/
TUnit cues, even though the underlying techniques (assertion diversity,
anti-pattern detection, pseudo-mutation gap analysis, academic smell
catalog, trait tagging) apply equally to other test frameworks.

This change makes them polyglot across .NET, Python (pytest/unittest),
TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG),
Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin
(JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest).

Changes
-------

* New reference skill `test-analysis-extensions` with per-language
  extension files (dotnet, python, typescript, java, go, ruby, rust,
  swift, kotlin, powershell, cpp). Each documents test markers,
  assertion APIs, sleep / time / random APIs, skip annotations,
  setup/teardown, mystery-guest indicators, integration markers, and
  tag-support capability for the corresponding framework.
* `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`,
  `test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter
  now lists all polyglot frameworks (keeping .NET trigger words for
  backward compat); bodies reference `test-analysis-extensions` and
  add per-language calibration (e.g., Go/Rust table-driven loops are
  idiomatic and not Conditional Test Logic; pytest bare `assert` is
  canonical; missing-await on async assertions is a new Critical smell).
* `test-quality-auditor` agent now runs language detection, gates
  .NET-only pipeline steps (coverage-analysis, CRAP, detect-static-
  dependencies, testability migration, experimental dotnet skills), and
  recommends native tooling for non-.NET projects via an explicit
  Capability Matrix.
* README updated to split sections by polyglot vs .NET-only.
* New polyglot eval scenarios added (Python/pytest, TS/Jest,
  Java/JUnit) alongside existing C# scenarios in
  `test-anti-patterns`, `assertion-quality`,
  `test-smell-detection`.

The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis,
crap-score, detect-static-dependencies, run-tests, filter-syntax,
platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are
unchanged and continue to activate only for .NET work.

Validated with skill-validator `check --plugin ./plugins/dotnet-test`.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR #707 review feedback

* typescript.md: replace xpect.fail() recommendation (Jest has no
  such API) with 	hrow new Error(...) / explicit failing assertion.
* test-tagging SKILL.md description: clarify that unittest has no
  canonical tag syntax (pytest markers are the auto-edit mechanism;
  unittest is report-only). Switch doctest reference to "decorators"
  rather than implying a Catch2-style [tags] mechanism exists.
* test-tagging SKILL.md auto-edit list and table: use
  `* doctest::test_suite("tag")` decorator chain syntax for doctest
  in both the bullet list and the per-framework table, matching the
  example already shown.
* cpp.md extension: spell out the Catch2 vs doctest tagging
  distinction in the capability matrix (Catch2 [tag], doctest
  * doctest::test_suite("tag") decorator).

Validated with `dotnet run --project eng/skill-validator/src -- check
--plugin ./plugins/dotnet-test` (23 skills + 11 agents).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

2026-06-01 16:46:26 +02:00

7.6 KiB

Raw Permalink Blame History

dotnet-test

Skills and agents for running, generating, analyzing, migrating, and improving tests. Originally built for .NET (MSTest, xUnit, NUnit, TUnit) and platforms (VSTest, Microsoft.Testing.Platform); the test-generation pipeline and the five test-analysis skills (anti-patterns, smells, assertion quality, gap analysis, tagging) plus the test-quality-auditor agent are polyglot and also work with Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/Jasmine/node:test), Java (JUnit 4/5/TestNG), Go (testing/testify), Ruby (RSpec/Minitest), Rust (built-in/proptest), Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest/Boost.Test).

When to use this plugin

Run tests (.NET only) — execute dotnet test with automatic platform/framework detection and filter syntax
Generate tests (polyglot) — scaffold comprehensive unit tests for any language via a multi-agent pipeline
Migrate tests (.NET only) — upgrade MSTest v1/v2 → v3 → v4, xUnit v2 → v3, or VSTest → Microsoft.Testing.Platform
Audit test quality (polyglot) — detect anti-patterns, test smells, assertion gaps, and (for .NET) coverage risks
Improve testability (.NET only) — find static dependencies, generate wrappers, and migrate call sites to injectable abstractions
Measure coverage (.NET only) — collect code coverage, compute CRAP scores, and surface risk hotspots

Skills

Test execution

Skill	Description
run-tests	Run .NET tests via `dotnet test` with platform/framework auto-detection and filter support
mtp-hot-reload	Rapid test-fix iteration using MTP hot reload (edit code → re-run without rebuilding)

Test generation

Skill	Description
code-testing-agent	Multi-agent pipeline (Research → Plan → Implement → Build → Test → Fix → Lint) that generates tests for any language
writing-mstest-tests	Best practices and modern APIs for writing MSTest 3.x/4.x tests

Test migration

Skill	Description
migrate-mstest-v1v2-to-v3	Upgrade MSTest v1 (assembly refs) or v2 (NuGet 1.x–2.x) to v3
migrate-mstest-v3-to-v4	Upgrade MSTest v3 to v4 — handles all source and behavioral breaking changes
migrate-xunit-to-xunit-v3	Upgrade xUnit.net v2 to v3
migrate-vstest-to-mtp	Migrate from VSTest runner to Microsoft.Testing.Platform

Test quality & analysis (polyglot)

These five skills work across all supported languages by loading a per-language reference file from test-analysis-extensions.

Skill	Description
test-anti-patterns	Quick pragmatic scan for common test quality issues with severity ranking (any language)
test-smell-detection	Deep formal audit using academic test smell taxonomy (19 smell types, any language)
assertion-quality	Measure assertion variety and depth — find shallow tests that barely verify anything (any language)
test-gap-analysis	Pseudo-mutation analysis to find test blind spots that coverage numbers miss (any language)
test-tagging	Tag tests with standardized traits (smoke, regression, boundary, critical-path, etc.); auto-edits where the framework has canonical syntax, report-only otherwise

Coverage & risk (.NET only)

Skill	Description
coverage-analysis	Project-wide code coverage collection with CRAP score computation and risk hotspot reporting
crap-score	Calculate CRAP (Change Risk Anti-Patterns) scores for individual methods, classes, or files

For non-.NET languages, use the native coverage tool: coverage.py/pytest-cov (Python), jest --coverage/c8/nyc/vitest --coverage (JS/TS), JaCoCo (Java), go test -coverprofile (Go), SimpleCov (Ruby), cargo-tarpaulin/cargo-llvm-cov (Rust), xcrun llvm-cov (Swift), Kover (Kotlin), Pester's built-in code coverage (PowerShell), gcov/llvm-cov (C++).

Testability improvement (.NET only)

Skill	Description
detect-static-dependencies	Scan C# code for hard-to-test statics (DateTime.Now, File.*, HttpClient, etc.)
generate-testability-wrappers	Generate wrapper interfaces or guide adoption of built-in abstractions (TimeProvider, IFileSystem)
migrate-static-to-wrapper	Bulk-replace static call sites with injected wrapper calls and add constructor injection

Reference data (loaded by other skills)

Skill	Description
code-testing-extensions	Language-specific guidance loaded by the code-testing pipeline (test generation)
test-analysis-extensions	Language-specific guidance loaded by the polyglot analysis skills (test markers, assertion APIs, sleeps, skips, mystery-guest indicators, integration markers, tag-support capability)
platform-detection (.NET)	Detect VSTest vs MTP and identify the test framework from project files
filter-syntax (.NET)	Test filter syntax reference for VSTest and MTP across all frameworks
dotnet-test-frameworks (.NET)	Framework detection patterns, assertion APIs, skip annotations, and lifecycle methods (kept for backward compatibility with .NET-only skills like `writing-mstest-tests`)

Agents

User-facing agents

These are the entry-point agents you invoke directly:

Agent	Purpose
code-testing-generator	Orchestrates the full test generation pipeline (research → plan → implement → build → test → fix → lint)
test-migration	Auto-detects framework/version and routes to the correct migration skill
test-quality-auditor	Runs multi-skill audit pipelines for comprehensive test suite assessment
testability-migration	End-to-end testability improvement: detect → generate wrappers → migrate call sites

Internal subagents

These are pipeline stages invoked automatically by the agents above (user-invocable: false). You do not need to call them directly:

Agent	Called by	Purpose
code-testing-researcher	code-testing-generator	Analyzes codebase structure, testing patterns, and testability
code-testing-planner	code-testing-generator	Creates phased test implementation plans from research findings
code-testing-implementer	code-testing-generator	Implements one phase from the plan, runs build-test-fix cycles
code-testing-builder	code-testing-implementer	Runs build/compile commands and reports results
code-testing-tester	code-testing-implementer	Runs test commands and reports pass/fail results
code-testing-fixer	code-testing-implementer	Fixes compilation errors in source or test files
code-testing-linter	code-testing-implementer	Runs code formatting and linting

Prerequisites

For polyglot skills and agents

The test-generation pipeline (code-testing-generator and friends) and the five test-analysis skills (test-anti-patterns, test-smell-detection, assertion-quality, test-gap-analysis, test-tagging) plus the test-quality-auditor agent work with any of the supported languages above. You just need a working test runtime for the language you're targeting (e.g., python + pytest, node + npm test, mvn / gradle, go, bundle exec rspec, cargo test, swift test, pwsh + Pester, cmake + your C++ test runner). The skills will detect the framework automatically.

For .NET-only skills and agents

.NET SDK installed (dotnet on PATH)
A project with an existing test framework (MSTest, xUnit, NUnit, or TUnit) for execution, migration, coverage, CRAP, testability, and the experimental dotnet-experimental skills.

7.6 KiB Raw Permalink Blame History Unescape Escape