skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
Files
eval-exec-sonnet46
dotnet__skills/plugins/dotnet-test/README.md
T
Amaury Levé a677ec43d2 Consolidate dotnet-test-frameworks into test-analysis-extensions (#851) 
* Consolidate dotnet-test-frameworks into test-analysis-extensions

Remove the orphaned dotnet-test-frameworks reference skill, whose content
was a duplicate subset of test-analysis-extensions/extensions/dotnet.md.
Nothing actually loaded it. Redirect the remaining references and drop its
eval tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Make exp-test-maintainability Step 1 self-contained and fix activation

Address PR review: exp-test-maintainability (dotnet-experimental plugin)
must not point at test-analysis-extensions (dotnet-test plugin) — inline the
.NET framework markers instead. Also strengthen the description/When-to-Use
triggers so the 'each new case needs a whole new method / suggest a better
structure' scenario reliably activates the skill instead of routing to a
test-writing skill.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Revert exp-test-maintainability description broadening; keep self-contained markers

The description broadening regressed activation (3/4 -> 0/4 across a
variance-dominated single eval run, CV up to 924%). Description is the
activation lever, so restore the original wording and keep only the
review-comment fix (inlined .NET framework markers in Step 1).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fix scenario 1 eval: use project fixture so the skill activates

Scenario 1 pasted a self-contained snippet inline and asked to 'suggest a
better structure', which the base model answers directly without loading the
skill -> NOT ACTIVATED with no headroom. Convert it to the project-fixture
pattern used by the activating scenarios (3 & 4): move the repetitive
InputValidatorTests into a Validation.Tests fixture and ask the model to
analyze the project. Mirror the change in eval.vally.yaml.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Fix scenario 2 eval: use project fixture so the skill activates

Scenario 2 also pasted a self-contained snippet inline (well-maintained
Auth.Tests) and asked to 'review these tests', which the base model handles
without loading the skill. Convert to the project-fixture pattern used by the
activating scenarios; add fixtures/well-maintained/Auth.Tests. Mirror in
eval.vally.yaml. All four scenarios now reference a project on disk.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Add reject_tools to exp-test-maintainability evals to create headroom

All four scenarios scored NOT ACTIVATED / no-headroom (#8 in
InvestigatingResults.md): baselines are already 4.0-5.0/5, so loading the
skill only adds token/tool overhead with no offsetting quality gain.

Match the convention of the sibling analysis skill test-anti-patterns:
reject bash/edit/create on every scenario. This skill is analysis-only
('do not modify any files'), so the constraint is a no-op for correct
behavior but levels the playing field between baseline and skilled runs —
scoring focuses on answer quality (weight 0.40+0.30) instead of tool-induced
overhead. Applied to both eval.yaml and eval.vally.yaml.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

* Address review: make well-maintained fixture self-contained and deterministic

- Import System.Security.Authentication for AuthenticationException
- Replace wall-clock assertion (token.ExpiresAt > DateTime.UtcNow) with a
  deterministic one anchored on a fixed clock (FixedNow), and drive the
  injected FakeClock from that fixed time. This removes the flakiness and
  makes the FakeClock injection meaningful — a better exemplar for the
  'well-maintained' scenario.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
2026-07-01 16:53:02 +02:00

121 lines
8.9 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# dotnet-test
Skills and agents for running, generating, analyzing, and improving tests. Originally built for .NET (MSTest, xUnit, NUnit, TUnit) and platforms (VSTest, Microsoft.Testing.Platform); the test-generation pipeline and the six test-analysis skills (anti-patterns, smells, assertion quality, gap analysis, tagging, grade tests) plus the `test-quality-auditor` agent are **polyglot** and also work with Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/Jasmine/node:test), Java (JUnit 4/5/TestNG), Go (testing/testify), Ruby (RSpec/Minitest), Rust (built-in/proptest), Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest/Boost.Test).
> **Test framework/platform migration** (MSTest/xUnit upgrades, xUnit → MSTest, VSTest → Microsoft.Testing.Platform) lives in the separate [`dotnet-test-migration`](../dotnet-test-migration/) plugin.
## When to use this plugin
- **Run tests** *(.NET only)* — execute `dotnet test` with automatic platform/framework detection and filter syntax
- **Generate tests** *(polyglot)* — scaffold comprehensive unit tests for any language via a multi-agent pipeline
- **Migrate tests** *(.NET only)* — see the separate [`dotnet-test-migration`](../dotnet-test-migration/) plugin (MSTest v1/v2 → v3 → v4, xUnit v2 → v3, xUnit → MSTest, VSTest → Microsoft.Testing.Platform)
- **Audit test quality** *(polyglot)* — detect anti-patterns, test smells, assertion gaps, and (for .NET) coverage risks
- **Improve testability** *(.NET only)* — find static dependencies, generate wrappers, and migrate call sites to injectable abstractions
- **Measure coverage** *(.NET only)* — collect code coverage, compute CRAP scores, and surface risk hotspots
## Skills
### Test execution
| Skill | Description |
|---|---|
| **run-tests** | Run .NET tests via `dotnet test` with platform/framework auto-detection and filter support |
| **mtp-hot-reload** | Rapid test-fix iteration using MTP hot reload (edit code → re-run without rebuilding) |
### Test generation
| Skill | Description |
|---|---|
| **code-testing-agent** | Multi-agent pipeline (Research → Plan → Implement → Build → Test → Fix → Lint) that generates tests for any language |
| **writing-mstest-tests** | Best practices and modern APIs for writing MSTest 3.x/4.x tests |
### Test migration
Moved to the [`dotnet-test-migration`](../dotnet-test-migration/) plugin (`migrate-mstest-v1v2-to-v3`, `migrate-mstest-v3-to-v4`, `migrate-xunit-to-xunit-v3`, `migrate-xunit-to-mstest`, `migrate-vstest-to-mtp`, and the `test-migration` orchestrator agent).
### Test quality & analysis *(polyglot)*
These six skills are all polyglot. They work across all supported languages by loading a per-language reference file from `test-analysis-extensions`. `grade-tests` additionally embeds its own scoring rubric (sub-grades, weighting, anti-pattern catalog) so the per-test grades stay consistent across calls.
| Skill | Description |
|---|---|
| **test-anti-patterns** | Quick pragmatic scan for common test quality issues with severity ranking (any language) |
| **test-smell-detection** | Deep formal audit using academic test smell taxonomy (19 smell types, any language) |
| **assertion-quality** | Measure assertion variety and depth — find shallow tests that barely verify anything (any language) |
| **test-gap-analysis** | Pseudo-mutation analysis to find test blind spots that coverage numbers miss (any language) |
| **test-tagging** | Tag tests with standardized traits (smoke, regression, boundary, critical-path, etc.); auto-edits where the framework has canonical syntax, report-only otherwise |
| **grade-tests** | Grade a curated list of test methods individually and produce a compact, PR-comment-friendly table of letter grades (A–F), score bands, and one-line notes — designed for per-PR test-quality feedback (any language) |
### Coverage & risk *(.NET only)*
| Skill | Description |
|---|---|
| **coverage-analysis** | Project-wide code coverage collection with CRAP score computation and risk hotspot reporting |
| **crap-score** | Calculate CRAP (Change Risk Anti-Patterns) scores for individual methods, classes, or files |
For non-.NET languages, use the native coverage tool: `coverage.py`/`pytest-cov` (Python), `jest --coverage`/`c8`/`nyc`/`vitest --coverage` (JS/TS), JaCoCo (Java), `go test -coverprofile` (Go), SimpleCov (Ruby), `cargo-tarpaulin`/`cargo-llvm-cov` (Rust), `xcrun llvm-cov` (Swift), Kover (Kotlin), Pester's built-in code coverage (PowerShell), `gcov`/`llvm-cov` (C++).
### Testability improvement *(.NET only)*
| Skill | Description |
|---|---|
| **detect-static-dependencies** | Scan C# code for hard-to-test statics (DateTime.Now, File.*, HttpClient, etc.) |
| **generate-testability-wrappers** | Generate wrapper interfaces or guide adoption of built-in abstractions (TimeProvider, IFileSystem) |
| **migrate-static-to-wrapper** | Bulk-replace static call sites with injected wrapper calls and add constructor injection |
### Reference data (loaded by other skills)
| Skill | Description |
|---|---|
| **code-testing-extensions** | Language-specific guidance loaded by the code-testing pipeline (test generation) |
| **test-analysis-extensions** | Language-specific guidance loaded by the polyglot analysis skills (test markers, assertion APIs, sleeps, skips, mystery-guest indicators, integration markers, tag-support capability) |
| **platform-detection** *(.NET)* | Detect VSTest vs MTP and identify the test framework from project files |
| **filter-syntax** *(.NET)* | Test filter syntax reference for VSTest and MTP across all frameworks |
## Agents
### User-facing agents
These are the entry-point agents you invoke directly:
| Agent | Purpose |
|---|---|
| **code-testing-generator** | Orchestrates the full test generation pipeline (research → plan → implement → build → test → fix → lint) |
| **test-quality-auditor** | Runs multi-skill audit pipelines for comprehensive test suite assessment |
| **testability-migration** | End-to-end testability improvement: detect → generate wrappers → migrate call sites |
> **Test framework/platform migration** is handled by the `test-migration` agent in the separate [`dotnet-test-migration`](../dotnet-test-migration/) plugin.
### Internal subagents
These are pipeline stages invoked automatically by the agents above (`user-invocable: false`). You do not need to call them directly:
| Agent | Called by | Purpose |
|---|---|---|
| **code-testing-researcher** | code-testing-generator | Analyzes codebase structure, testing patterns, and testability |
| **code-testing-planner** | code-testing-generator | Creates phased test implementation plans from research findings |
| **code-testing-implementer** | code-testing-generator | Implements one phase from the plan, runs build-test-fix cycles |
| **code-testing-builder** | code-testing-implementer | Runs build/compile commands and reports results |
| **code-testing-tester** | code-testing-implementer | Runs test commands and reports pass/fail results |
| **code-testing-fixer** | code-testing-implementer | Fixes compilation errors in source or test files |
| **code-testing-linter** | code-testing-implementer | Runs code formatting and linting |
> **VS Code — enabling full multi-level fan-out:** The pipeline delegates in two levels: `code-testing-generator` → researcher / planner / implementer, and `code-testing-implementer` → builder / tester / fixer / linter. VS Code gates *nested* delegation (a subagent spawning its own subagents) behind a setting that is **off by default**, so the first level runs out of the box but the second one does not. For large scopes — many files or modules, where parallel build/test/fix/lint workers help — enable it in your VS Code settings:
>
> ```jsonc
> "chat.subagents.allowInvocationsFromSubagents": true
> ```
>
> Without it, `code-testing-implementer` still builds, tests, fixes, and lints — it just does that work inline instead of delegating to the worker subagents, so results are unaffected. The GitHub Copilot CLI has no such gate and always fans out.
## Prerequisites
### For polyglot skills and agents
The test-generation pipeline (`code-testing-generator` and friends) and the six test-analysis skills (`test-anti-patterns`, `test-smell-detection`, `assertion-quality`, `test-gap-analysis`, `test-tagging`, `grade-tests`) plus the `test-quality-auditor` agent work with any of the supported languages above. You just need a working test runtime for the language you're targeting (e.g., `python` + `pytest`, `node` + `npm test`, `mvn` / `gradle`, `go`, `bundle exec rspec`, `cargo test`, `swift test`, `pwsh` + Pester, `cmake` + your C++ test runner). The skills will detect the framework automatically.
### For .NET-only skills and agents
- .NET SDK installed (`dotnet` on PATH)
- A project with an existing test framework (MSTest, xUnit, NUnit, or TUnit) for execution, migration, coverage, CRAP, testability, and the experimental `dotnet-experimental` skills.
Reference in New Issue View Git Blame Copy Permalink