Amaury Levé
a677ec43d2
* Consolidate dotnet-test-frameworks into test-analysis-extensions Remove the orphaned dotnet-test-frameworks reference skill, whose content was a duplicate subset of test-analysis-extensions/extensions/dotnet.md. Nothing actually loaded it. Redirect the remaining references and drop its eval tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make exp-test-maintainability Step 1 self-contained and fix activation Address PR review: exp-test-maintainability (dotnet-experimental plugin) must not point at test-analysis-extensions (dotnet-test plugin) — inline the .NET framework markers instead. Also strengthen the description/When-to-Use triggers so the 'each new case needs a whole new method / suggest a better structure' scenario reliably activates the skill instead of routing to a test-writing skill. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Revert exp-test-maintainability description broadening; keep self-contained markers The description broadening regressed activation (3/4 -> 0/4 across a variance-dominated single eval run, CV up to 924%). Description is the activation lever, so restore the original wording and keep only the review-comment fix (inlined .NET framework markers in Step 1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix scenario 1 eval: use project fixture so the skill activates Scenario 1 pasted a self-contained snippet inline and asked to 'suggest a better structure', which the base model answers directly without loading the skill -> NOT ACTIVATED with no headroom. Convert it to the project-fixture pattern used by the activating scenarios (3 & 4): move the repetitive InputValidatorTests into a Validation.Tests fixture and ask the model to analyze the project. Mirror the change in eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix scenario 2 eval: use project fixture so the skill activates Scenario 2 also pasted a self-contained snippet inline (well-maintained Auth.Tests) and asked to 'review these tests', which the base model handles without loading the skill. Convert to the project-fixture pattern used by the activating scenarios; add fixtures/well-maintained/Auth.Tests. Mirror in eval.vally.yaml. All four scenarios now reference a project on disk. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add reject_tools to exp-test-maintainability evals to create headroom All four scenarios scored NOT ACTIVATED / no-headroom (#8 in InvestigatingResults.md): baselines are already 4.0-5.0/5, so loading the skill only adds token/tool overhead with no offsetting quality gain. Match the convention of the sibling analysis skill test-anti-patterns: reject bash/edit/create on every scenario. This skill is analysis-only ('do not modify any files'), so the constraint is a no-op for correct behavior but levels the playing field between baseline and skilled runs — scoring focuses on answer quality (weight 0.40+0.30) instead of tool-induced overhead. Applied to both eval.yaml and eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address review: make well-maintained fixture self-contained and deterministic - Import System.Security.Authentication for AuthenticationException - Replace wall-clock assertion (token.ExpiresAt > DateTime.UtcNow) with a deterministic one anchored on a fixed clock (FixedNow), and drive the injected FakeClock from that fixed time. This removes the flakiness and makes the FakeClock injection meaningful — a better exemplar for the 'well-maintained' scenario. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
8.9 KiB
dotnet-test
Skills and agents for running, generating, analyzing, and improving tests. Originally built for .NET (MSTest, xUnit, NUnit, TUnit) and platforms (VSTest, Microsoft.Testing.Platform); the test-generation pipeline and the six test-analysis skills (anti-patterns, smells, assertion quality, gap analysis, tagging, grade tests) plus the test-quality-auditor agent are polyglot and also work with Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/Jasmine/node:test), Java (JUnit 4/5/TestNG), Go (testing/testify), Ruby (RSpec/Minitest), Rust (built-in/proptest), Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest/Boost.Test).
Test framework/platform migration (MSTest/xUnit upgrades, xUnit → MSTest, VSTest → Microsoft.Testing.Platform) lives in the separate
dotnet-test-migrationplugin.
When to use this plugin
- Run tests (.NET only) — execute
dotnet testwith automatic platform/framework detection and filter syntax - Generate tests (polyglot) — scaffold comprehensive unit tests for any language via a multi-agent pipeline
- Migrate tests (.NET only) — see the separate
dotnet-test-migrationplugin (MSTest v1/v2 → v3 → v4, xUnit v2 → v3, xUnit → MSTest, VSTest → Microsoft.Testing.Platform) - Audit test quality (polyglot) — detect anti-patterns, test smells, assertion gaps, and (for .NET) coverage risks
- Improve testability (.NET only) — find static dependencies, generate wrappers, and migrate call sites to injectable abstractions
- Measure coverage (.NET only) — collect code coverage, compute CRAP scores, and surface risk hotspots
Skills
Test execution
| Skill | Description |
|---|---|
| run-tests | Run .NET tests via dotnet test with platform/framework auto-detection and filter support |
| mtp-hot-reload | Rapid test-fix iteration using MTP hot reload (edit code → re-run without rebuilding) |
Test generation
| Skill | Description |
|---|---|
| code-testing-agent | Multi-agent pipeline (Research → Plan → Implement → Build → Test → Fix → Lint) that generates tests for any language |
| writing-mstest-tests | Best practices and modern APIs for writing MSTest 3.x/4.x tests |
Test migration
Moved to the dotnet-test-migration plugin (migrate-mstest-v1v2-to-v3, migrate-mstest-v3-to-v4, migrate-xunit-to-xunit-v3, migrate-xunit-to-mstest, migrate-vstest-to-mtp, and the test-migration orchestrator agent).
Test quality & analysis (polyglot)
These six skills are all polyglot. They work across all supported languages by loading a per-language reference file from test-analysis-extensions. grade-tests additionally embeds its own scoring rubric (sub-grades, weighting, anti-pattern catalog) so the per-test grades stay consistent across calls.
| Skill | Description |
|---|---|
| test-anti-patterns | Quick pragmatic scan for common test quality issues with severity ranking (any language) |
| test-smell-detection | Deep formal audit using academic test smell taxonomy (19 smell types, any language) |
| assertion-quality | Measure assertion variety and depth — find shallow tests that barely verify anything (any language) |
| test-gap-analysis | Pseudo-mutation analysis to find test blind spots that coverage numbers miss (any language) |
| test-tagging | Tag tests with standardized traits (smoke, regression, boundary, critical-path, etc.); auto-edits where the framework has canonical syntax, report-only otherwise |
| grade-tests | Grade a curated list of test methods individually and produce a compact, PR-comment-friendly table of letter grades (A–F), score bands, and one-line notes — designed for per-PR test-quality feedback (any language) |
Coverage & risk (.NET only)
| Skill | Description |
|---|---|
| coverage-analysis | Project-wide code coverage collection with CRAP score computation and risk hotspot reporting |
| crap-score | Calculate CRAP (Change Risk Anti-Patterns) scores for individual methods, classes, or files |
For non-.NET languages, use the native coverage tool: coverage.py/pytest-cov (Python), jest --coverage/c8/nyc/vitest --coverage (JS/TS), JaCoCo (Java), go test -coverprofile (Go), SimpleCov (Ruby), cargo-tarpaulin/cargo-llvm-cov (Rust), xcrun llvm-cov (Swift), Kover (Kotlin), Pester's built-in code coverage (PowerShell), gcov/llvm-cov (C++).
Testability improvement (.NET only)
| Skill | Description |
|---|---|
| detect-static-dependencies | Scan C# code for hard-to-test statics (DateTime.Now, File.*, HttpClient, etc.) |
| generate-testability-wrappers | Generate wrapper interfaces or guide adoption of built-in abstractions (TimeProvider, IFileSystem) |
| migrate-static-to-wrapper | Bulk-replace static call sites with injected wrapper calls and add constructor injection |
Reference data (loaded by other skills)
| Skill | Description |
|---|---|
| code-testing-extensions | Language-specific guidance loaded by the code-testing pipeline (test generation) |
| test-analysis-extensions | Language-specific guidance loaded by the polyglot analysis skills (test markers, assertion APIs, sleeps, skips, mystery-guest indicators, integration markers, tag-support capability) |
| platform-detection (.NET) | Detect VSTest vs MTP and identify the test framework from project files |
| filter-syntax (.NET) | Test filter syntax reference for VSTest and MTP across all frameworks |
Agents
User-facing agents
These are the entry-point agents you invoke directly:
| Agent | Purpose |
|---|---|
| code-testing-generator | Orchestrates the full test generation pipeline (research → plan → implement → build → test → fix → lint) |
| test-quality-auditor | Runs multi-skill audit pipelines for comprehensive test suite assessment |
| testability-migration | End-to-end testability improvement: detect → generate wrappers → migrate call sites |
Test framework/platform migration is handled by the
test-migrationagent in the separatedotnet-test-migrationplugin.
Internal subagents
These are pipeline stages invoked automatically by the agents above (user-invocable: false). You do not need to call them directly:
| Agent | Called by | Purpose |
|---|---|---|
| code-testing-researcher | code-testing-generator | Analyzes codebase structure, testing patterns, and testability |
| code-testing-planner | code-testing-generator | Creates phased test implementation plans from research findings |
| code-testing-implementer | code-testing-generator | Implements one phase from the plan, runs build-test-fix cycles |
| code-testing-builder | code-testing-implementer | Runs build/compile commands and reports results |
| code-testing-tester | code-testing-implementer | Runs test commands and reports pass/fail results |
| code-testing-fixer | code-testing-implementer | Fixes compilation errors in source or test files |
| code-testing-linter | code-testing-implementer | Runs code formatting and linting |
VS Code — enabling full multi-level fan-out: The pipeline delegates in two levels:
code-testing-generator→ researcher / planner / implementer, andcode-testing-implementer→ builder / tester / fixer / linter. VS Code gates nested delegation (a subagent spawning its own subagents) behind a setting that is off by default, so the first level runs out of the box but the second one does not. For large scopes — many files or modules, where parallel build/test/fix/lint workers help — enable it in your VS Code settings:"chat.subagents.allowInvocationsFromSubagents": trueWithout it,
code-testing-implementerstill builds, tests, fixes, and lints — it just does that work inline instead of delegating to the worker subagents, so results are unaffected. The GitHub Copilot CLI has no such gate and always fans out.
Prerequisites
For polyglot skills and agents
The test-generation pipeline (code-testing-generator and friends) and the six test-analysis skills (test-anti-patterns, test-smell-detection, assertion-quality, test-gap-analysis, test-tagging, grade-tests) plus the test-quality-auditor agent work with any of the supported languages above. You just need a working test runtime for the language you're targeting (e.g., python + pytest, node + npm test, mvn / gradle, go, bundle exec rspec, cargo test, swift test, pwsh + Pester, cmake + your C++ test runner). The skills will detect the framework automatically.
For .NET-only skills and agents
- .NET SDK installed (
dotneton PATH) - A project with an existing test framework (MSTest, xUnit, NUnit, or TUnit) for execution, migration, coverage, CRAP, testability, and the experimental
dotnet-experimentalskills.