Cover previously uncovered skill aspects:
- [Validation] All .cs files in scope were scanned (check count)
- [Validation] Report includes category totals, top patterns, and affected files
- [Validation] obj/ and bin/ directories were excluded
- [Pitfall] Scanning obj/ or generated code
- [Pitfall] Missing statics inside lambdas/LINQ
- [WorkflowStep] Step 4: Present the report
Three new scenarios with supporting fixtures:
- Structured report with file count, categories, and top patterns
- Exclude obj/bin directories (with Generated.cs decoy in obj/)
- Detect statics inside lambdas and LINQ expressions
* Add eval scenario for filter-syntax [trait] query language pattern
Add a run-tests scenario that exercises the xUnit v3 query filter language
with [traitName=traitValue] syntax (--filter-query), covering the uncovered
[CodePattern] [trait] at line 113 of filter-syntax SKILL.md.
New fixture: mtp-xunit-v3-query with multiple test classes and traits
(Smoke, Regression, Unit) to test combined class-pattern + trait filtering.
* Address review feedback for xUnit v3 query eval coverage
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
The 'Fix Assert.AreEqual object overload errors after v3 upgrade' evaluation
scenario was failing because:
1. The prompt said 'latest major version' without mentioning 'v3' or 'MSTest v3',
missing the skill's activation keywords.
2. The project file showed MSTest 3.8.0 (post-upgrade state) with no v2 context,
while the SKILL.md 'When Not to Use' said 'Project already uses MSTest v3'.
3. The prompt's 'fix compilation issues' phrasing competed with writing-mstest-tests
skill activation.
Changes:
- Updated prompt to explicitly mention upgrading from v2 to v3 and reference
MSTest v3 compatibility issues
- Added XML comment in csproj showing the pre-upgrade v2 package versions
- Clarified SKILL.md 'When Not to Use' to distinguish fully-migrated projects
from projects with active migration errors
Move assertion-quality, test-gap-analysis, test-smell-detection, and
test-tagging from dotnet-experimental to dotnet-test. These skills have
demonstrated sufficient eval coverage, unique value beyond baseline model
capabilities, and are already integrated into the test-quality-auditor
agent pipeline.
Changes:
- Move skills and tests, dropping the exp- prefix
- Update all cross-references in skill descriptions and routing tables
- Update test-quality-auditor agent and eval scenarios
- Update CODEOWNERS for the moved paths
- Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks)
- Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability
Skills held back for further iteration:
- exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios
- exp-test-maintainability: quality gains offset by token overhead
The scenario was timing out on CI because:
- The 180s timeout left insufficient headroom for CI machines
(local runs take 120-135s, CI machines are slower due to load,
network NuGet restores, and parallel execution)
Changes:
- Add 'dotnet restore' setup command to pre-warm NuGet cache before
the agent starts (same pattern used by migrate-mstest-v3-to-v4)
- Increase timeout from 180s to 300s to accommodate CI variability
* Fix dotnet-test-frameworks skill activation for evaluations
The skill had user-invocable: false and a description saying 'DO NOT USE
directly', which prevented the model from activating it during evaluation.
All 6 eval scenarios default to expect_activation: true, causing
SkillNotActivated failures on CI.
- Remove user-invocable: false from both dotnet-test and experimental copies
- Rewrite description with activation keywords matching eval prompts
(assertion mapping, framework identification, skip annotations, exception
assertions, lifecycle conversion, integration test markers)
- Preserve note that other analysis skills also load this as reference data
* Restore user-invocable: false, fix cross-refs, add expect_activation: false
Address review comment: remove exp-* skill references from dotnet-test-
frameworks description since those skills reference exp-dotnet-test-
frameworks instead.
Optimize eval results: this reference data skill's content is knowledge
the model already has (baseline scores 5.0/5 on most scenarios). Direct
activation adds token/time overhead without quality improvement:
- Restore user-invocable: false (correct for reference data skill)
- Shorten description to not trigger activation
- Add expect_activation: false to all 6 scenarios
The skill's value is realized when loaded by consumer skills (e.g.
test-anti-patterns), not via direct user queries.
The 'Migrate MSTest.Sdk v3 project using ManagedType and TestTimeout'
scenario times out on CI because MSTest.Sdk/4.1.0 resolution takes 40+
seconds on cold NuGet cache. Each agent session may call dotnet build
multiple times, easily exceeding the 240s timeout.
Add a 'dotnet restore' setup command that runs before the agent starts
(outside the timeout window), pre-warming the NuGet cache and obj/
directory so subsequent builds complete in ~2s.
* Fix timeout in MSTest.Sdk v3-to-v4 eval scenario 9
The fixture csproj had MSTest.Sdk/3.8.0 while the prompt claimed the user
already upgraded to 4.1.0. This mismatch caused the model to edit the
csproj and run dotnet build, triggering slow MSTest SDK NuGet downloads
that exceeded the 240s timeout on CI.
- Update fixture csproj to MSTest.Sdk/4.1.0 to match the prompt
- Add specific CS error codes to the prompt so the model provides direct
fixes instead of building
- Update rubric item to reflect the project already being on 4.1.0
* Fix CI timeout in migrate-mstest-v3-to-v4 eval scenario
Reduce scope of the 'Fix multiple v4 breaking changes' scenario:
- Remove 3 lower-signal breaking changes from the fixture (AreEqual params,
IsInstanceOfType out param, Properties.Contains) to cut agent work
- Add max_turns: 5 to cap agent iterations
- Lower timeout from 240 to 180 seconds
* Strengthen net6.0 scenario prompt for reliable skill activation
Five scenarios were performing at or below stock Copilot because prompts
gave away answers and rubric items tested knowledge the baseline LLM
can deduce from the code alone.
Changes per scenario:
- Fix DataRow type mismatch: added double-to-float mismatch, vaguer
prompt, rubric tests both mismatch types and latest-3.x knowledge
- Migrate .testsettings to .runsettings: added code coverage data
collector to fixture, rubric tests correct TestTimeout mapping
(<MSTest><TestTimeout> not <TestSessionTimeout>) and collector mapping
- Migrate MSTest v2 NuGet to v3: vaguer prompt (forces model to read
csproj), rubric tests metapackage/MSTest.Sdk recommendation and
Microsoft.NET.Test.Sdk dependency awareness
- Fix Assert.AreEqual overload errors: removed exact error code from
prompt, added mixed-type and interface-reference cases, rubric tests
correct generic type parameter selection
- Handle dropped TFM: changed to multi-target (net5.0;net462;
netcoreapp3.1), rubric tests that model knows which TFMs are still
supported and doesn't upgrade them indiscriminately
The ContosoUniversity test fixture had 17 unused NuGet packages (bootstrap,
jQuery, Modernizr, Antlr4, WebGrease, Microsoft.Build.Tasks.Core, etc.)
that bloated NuGet restore and build time. The multi-agent test generation
pipeline performs multiple build/test cycles, causing cumulative delays
that exceeded the 2400s scenario timeout on CI.
Changes:
- Remove 17 unused PackageReference entries from ContosoUniversity.csproj
- Remove stale ItemGroup entries (Global.asax, Scripts, compile removes)
- Remove NU1701 warning suppression (no longer needed)
- Increase command_timeout from 180s to 300s for dotnet test assertion
(accounts for first-time restore + build + test + coverage collection)
The 5 failing scenarios were timing out because expect_tools: [bash]
forced the agent to execute terminal commands. The hot reload workflow
involves 'dotnet run' which starts a test host that blocks indefinitely
waiting for file changes, causing the scenario to exhaust its timeout.
These scenarios test whether the model provides correct guidance (package
name, env var, commands), not whether it actually executes them. The
output_matches assertions already verify the response content.
Removed expect_tools from:
- Suggest hot reload for failing test in MTP project (SDK 9)
- Suggest hot reload for failing test in MTP project (SDK 10)
- Enable hot reload when package already installed
- Suggest launchSettings.json configuration for hot reload
- Negative: VSTest project cannot use MTP hot reload
Also reduced timeouts since scenarios no longer need to wait for command
execution.
Add rollForward: latestFeature to all global.json entries that pin SDK 9.0.200
so that any 9.0.xxx SDK satisfies the requirement. Without this, machines with
only 9.0.3xx installed fail all dotnet commands during evaluation, causing the
model to burn its entire timeout budget on failed bash retries.
The fixture csproj had MSTest.Sdk/3.8.0 while the prompt claimed the user
already upgraded to 4.1.0. This mismatch caused the model to edit the
csproj and run dotnet build, triggering slow MSTest SDK NuGet downloads
that exceeded the 240s timeout on CI.
- Update fixture csproj to MSTest.Sdk/4.1.0 to match the prompt
- Add specific CS error codes to the prompt so the model provides direct
fixes instead of building
- Update rubric item to reflect the project already being on 4.1.0
* Add testability migration skills and agent for dotnet-test
Introduces three skills and one orchestrator agent to help developers
incrementally replace hard-to-test static dependencies with injectable
abstractions:
- detect-static-dependencies: scans C# code for DateTime.Now, File.*,
Environment.*, HttpClient, Console.*, Process.* and ranks by frequency
- generate-testability-wrappers: generates wrapper interfaces or guides
adoption of built-in abstractions (TimeProvider, IHttpClientFactory,
System.IO.Abstractions)
- migrate-static-to-wrapper: mechanical codemod-style bulk replacement of
static call sites with wrapper calls, scoped per project/namespace
- testability-migration agent: orchestrates the Detect-Generate-Migrate
pipeline end-to-end
Includes eval.yaml tests and fixture projects for all three skills and
the agent.
* Fix activation, token overhead, and eval timeouts
- Expand skill descriptions with more trigger phrases for
migrate-static-to-wrapper and generate-testability-wrappers
to fix NOT ACTIVATED issues
- Collapse verbose replacement mapping tables and constructor
examples to reduce token overhead (~30% fewer tokens)
- Increase generate-testability-wrappers eval timeouts from
120s to 240s to prevent timeout-impacted scoring
- Soften detect-static-dependencies scenario 2 rubric to
reduce run-to-run variance in plugin mode
- Increase migrate-static-to-wrapper non-activation timeout
* Add coverage-analysis skill to dotnet-test plugin
Adds an automated, project-wide code coverage and CRAP score analysis skill
for .NET projects. Complements the existing crap-score skill (which is
manual/targeted for single methods) with an end-to-end pipeline:
- Auto-detects solution structure and test projects
- Detects coverage provider (Microsoft.Testing.Extensions.CodeCoverage or
Coverlet) per test project and selects the correct dotnet test arguments
- Runs coverage collection, generates reports via ReportGenerator
- Calculates CRAP scores per method via bundled PowerShell scripts
- Surfaces risk hotspots ranked by danger (complex + uncovered code)
- Produces actionable recommendations with projected score improvements
Boundary with crap-score: coverage-analysis is project-wide and automated;
crap-score is targeted and LLM-directed. The frontmatter descriptions
cross-reference each other so the skill router disambiguates correctly.
Includes eval.yaml with 3 scenarios (existing data analysis, from-scratch
coverage collection, coverage plateau diagnosis) and shared buildable
fixtures.
Fixes#457
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address PR review: coverage output path and scoped tool install
- Add --coverage-output $rawDir to all MS CodeCoverage dotnet test
commands (unified, .NET 9, .NET 10+, and mixed-project modes) so
Cobertura XML lands in the expected directory for downstream search
- Replace dotnet tool install --global with --tool-path scoped to
the coverage output directory to avoid mutating the user environment
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Cobertura XML fixture line numbers to match StudentService.cs
FindById starts at line 101 (was 88), Search starts at line 109 (was 96).
Updated both partial-coverage and plateau fixtures.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Wrap Get-ChildItem pipelines in @() for single-result array safety
PowerShell returns a scalar FileInfo (not array) when a pipeline yields
exactly one result, making .Count return null. Wrapping in @() ensures
the result is always an array so .Count works reliably for the common
single-test-project case.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove CODEOWNERS change - not needed for this PR
The existing /plugins/dotnet-test/ and /tests/dotnet-test/ wildcards
already route reviews to @dotnet/dotnet-testing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: .csproj fallback, cross-platform paths, .sln header
- Add .csproj fallback discovery in Step 1 when no .sln found (emit
ENTRY_TYPE:Project) to match documented behavior
- Fix ProjectReference backslash to forward slash in test fixture .csproj
- Remove leading blank line from ContosoUniversity.sln fixture
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add plateau/stuck trigger language to frontmatter description
The skill was NOT ACTIVATED for the 'Coverage plateau diagnosis' eval
scenario because the frontmatter description (used by the skill router)
lacked plateau-related keywords. The prompt 'coverage is stuck' and
'what's blocking me' didn't match the existing description text.
Add 'stuck', 'plateaued', and 'blocking improvement' to the frontmatter
Use-when sentence so the skill router activates for plateau diagnosis
prompts.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make CRAP scores mandatory, add existing-data shortcut path
The plateau eval scenario scored 1/5 on CRAP scores across all modes
because the LLM shortcut the workflow when existing Cobertura XML was
provided — it read the XML directly and gave a quick diagnostic answer
without computing CRAP scores or following the output template.
- Add existing-data guidance at top of Workflow section: skip test
execution but still run CRAP score computation (Steps 5-6)
- Add mandatory CRAP rule to guidelines.md: Risk Hotspots table is
required in every output regardless of analysis mode
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add writing-mstest-tests skill with eval scenarios
Add a new skill for writing modern MSTest 3.x/4.x unit tests covering:
- Test class structure (sealed, AAA pattern, naming conventions)
- Modern assertion APIs (Assert.Throws, IsInstanceOfType, collection asserts)
- Data-driven tests (DataRow, DynamicData with ValueTuples)
- Async tests with cancellation and timeout patterns
- Test lifecycle (constructor init, TestContext injection)
- Anti-pattern detection and modernization
Includes 10 eval scenarios across 4 goal categories with fixture files
for both modern MSTest and legacy anti-pattern test projects.
MSTest.Sdk version is recommended via global.json msbuild-sdks instead of
inline in the project file, ensuring all test projects get bumped together.
* Fixes
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Apply suggestion from @Evangelink
* Apply suggestion from @Evangelink
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Fix eval source file resolution to use eval directory
The SetupWorkDir method resolved setup file 'source:' paths relative to the
skill directory (plugins/.../skills/<name>/), but eval.yaml files live under
tests/. The ../../../../ relative paths resolved correctly but were rejected
by the path traversal security check because the resolved path escaped the
skill directory boundary.
Fix by preferring the eval directory as the base for source resolution when
evalPath is available, falling back to skillPath for backward compatibility
with scenarios embedded in SKILL.md files.
Also update run-tests eval.yaml source paths from
'../../../../tests/dotnet-test/run-tests/fixtures/...' to 'fixtures/...'
(relative to the eval directory where the fixtures actually live).
This was causing all 11 source-based run-tests scenarios to get zero fixture
files, making the agent find an empty workspace and fail to evaluate.
* Address review: extract ResolveSourcePath helper, add warning, fix remaining eval.yaml paths
- Extract source-resolution and path-traversal check into internal
ResolveSourcePath helper for testability
- Emit a clear warning when file.Source is set but no base directory
is available (instead of silently continuing)
- Add unit tests covering: evalPath-based resolution, skillPath fallback,
null paths warning, traversal rejection, and evalPath preference
- Fix path traversal patterns in analyzing-dotnet-performance and
clr-activation-debugging eval.yaml files (same issue as run-tests)
Amaury Levé