Several dotnet-test eval scenarios repeatedly hit their wall-clock timeout
in the evaluation dashboard (skilled + plugin arms), producing empty output
and failing all assertions. Increase per-scenario timeouts for the ones that
timed out most often across recent runs:
- writing-mstest-tests: 'Write unit tests for a service class',
'Write data-driven tests for a calculator',
'Use string assertions for format validation' 180->360;
'Use comparison assertions for boundary testing' 120->240
- coverage-analysis: 'Coverage plateau diagnosis',
'Project-wide coverage analysis with existing Cobertura data' 300->480
- test-gap-analysis: 'Decline request to write new tests from scratch' 120->300;
'Acknowledge well-tested code with few surviving mutations' 300->420
- run-tests: 'Filter xUnit v3 tests by trait on MTP',
'Filter xUnit v3 tests by class pattern and trait using query filter language' 120->240
Per eng/skill-validator/src/docs/InvestigatingResults.md, timeouts are the
highest-priority failure mode; code-generation scenarios often need 360s.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Add coverage-analysis skill to dotnet-test plugin
Adds an automated, project-wide code coverage and CRAP score analysis skill
for .NET projects. Complements the existing crap-score skill (which is
manual/targeted for single methods) with an end-to-end pipeline:
- Auto-detects solution structure and test projects
- Detects coverage provider (Microsoft.Testing.Extensions.CodeCoverage or
Coverlet) per test project and selects the correct dotnet test arguments
- Runs coverage collection, generates reports via ReportGenerator
- Calculates CRAP scores per method via bundled PowerShell scripts
- Surfaces risk hotspots ranked by danger (complex + uncovered code)
- Produces actionable recommendations with projected score improvements
Boundary with crap-score: coverage-analysis is project-wide and automated;
crap-score is targeted and LLM-directed. The frontmatter descriptions
cross-reference each other so the skill router disambiguates correctly.
Includes eval.yaml with 3 scenarios (existing data analysis, from-scratch
coverage collection, coverage plateau diagnosis) and shared buildable
fixtures.
Fixes#457
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address PR review: coverage output path and scoped tool install
- Add --coverage-output $rawDir to all MS CodeCoverage dotnet test
commands (unified, .NET 9, .NET 10+, and mixed-project modes) so
Cobertura XML lands in the expected directory for downstream search
- Replace dotnet tool install --global with --tool-path scoped to
the coverage output directory to avoid mutating the user environment
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Cobertura XML fixture line numbers to match StudentService.cs
FindById starts at line 101 (was 88), Search starts at line 109 (was 96).
Updated both partial-coverage and plateau fixtures.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Wrap Get-ChildItem pipelines in @() for single-result array safety
PowerShell returns a scalar FileInfo (not array) when a pipeline yields
exactly one result, making .Count return null. Wrapping in @() ensures
the result is always an array so .Count works reliably for the common
single-test-project case.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove CODEOWNERS change - not needed for this PR
The existing /plugins/dotnet-test/ and /tests/dotnet-test/ wildcards
already route reviews to @dotnet/dotnet-testing.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: .csproj fallback, cross-platform paths, .sln header
- Add .csproj fallback discovery in Step 1 when no .sln found (emit
ENTRY_TYPE:Project) to match documented behavior
- Fix ProjectReference backslash to forward slash in test fixture .csproj
- Remove leading blank line from ContosoUniversity.sln fixture
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add plateau/stuck trigger language to frontmatter description
The skill was NOT ACTIVATED for the 'Coverage plateau diagnosis' eval
scenario because the frontmatter description (used by the skill router)
lacked plateau-related keywords. The prompt 'coverage is stuck' and
'what's blocking me' didn't match the existing description text.
Add 'stuck', 'plateaued', and 'blocking improvement' to the frontmatter
Use-when sentence so the skill router activates for plateau diagnosis
prompts.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make CRAP scores mandatory, add existing-data shortcut path
The plateau eval scenario scored 1/5 on CRAP scores across all modes
because the LLM shortcut the workflow when existing Cobertura XML was
provided — it read the XML directly and gave a quick diagnostic answer
without computing CRAP scores or following the output template.
- Add existing-data guidance at top of Workflow section: skip test
execution but still run CRAP score computation (Steps 5-6)
- Add mandatory CRAP rule to guidelines.md: Risk Hotspots table is
required in every output regardless of analysis mode
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Aditya Mandaleeka