mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Cathy Sullivan
42e29303b4
* Add coverage-analysis skill to dotnet-test plugin Adds an automated, project-wide code coverage and CRAP score analysis skill for .NET projects. Complements the existing crap-score skill (which is manual/targeted for single methods) with an end-to-end pipeline: - Auto-detects solution structure and test projects - Detects coverage provider (Microsoft.Testing.Extensions.CodeCoverage or Coverlet) per test project and selects the correct dotnet test arguments - Runs coverage collection, generates reports via ReportGenerator - Calculates CRAP scores per method via bundled PowerShell scripts - Surfaces risk hotspots ranked by danger (complex + uncovered code) - Produces actionable recommendations with projected score improvements Boundary with crap-score: coverage-analysis is project-wide and automated; crap-score is targeted and LLM-directed. The frontmatter descriptions cross-reference each other so the skill router disambiguates correctly. Includes eval.yaml with 3 scenarios (existing data analysis, from-scratch coverage collection, coverage plateau diagnosis) and shared buildable fixtures. Fixes #457 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR review: coverage output path and scoped tool install - Add --coverage-output $rawDir to all MS CodeCoverage dotnet test commands (unified, .NET 9, .NET 10+, and mixed-project modes) so Cobertura XML lands in the expected directory for downstream search - Replace dotnet tool install --global with --tool-path scoped to the coverage output directory to avoid mutating the user environment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Cobertura XML fixture line numbers to match StudentService.cs FindById starts at line 101 (was 88), Search starts at line 109 (was 96). Updated both partial-coverage and plateau fixtures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wrap Get-ChildItem pipelines in @() for single-result array safety PowerShell returns a scalar FileInfo (not array) when a pipeline yields exactly one result, making .Count return null. Wrapping in @() ensures the result is always an array so .Count works reliably for the common single-test-project case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove CODEOWNERS change - not needed for this PR The existing /plugins/dotnet-test/ and /tests/dotnet-test/ wildcards already route reviews to @dotnet/dotnet-testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: .csproj fallback, cross-platform paths, .sln header - Add .csproj fallback discovery in Step 1 when no .sln found (emit ENTRY_TYPE:Project) to match documented behavior - Fix ProjectReference backslash to forward slash in test fixture .csproj - Remove leading blank line from ContosoUniversity.sln fixture Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add plateau/stuck trigger language to frontmatter description The skill was NOT ACTIVATED for the 'Coverage plateau diagnosis' eval scenario because the frontmatter description (used by the skill router) lacked plateau-related keywords. The prompt 'coverage is stuck' and 'what's blocking me' didn't match the existing description text. Add 'stuck', 'plateaued', and 'blocking improvement' to the frontmatter Use-when sentence so the skill router activates for plateau diagnosis prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make CRAP scores mandatory, add existing-data shortcut path The plateau eval scenario scored 1/5 on CRAP scores across all modes because the LLM shortcut the workflow when existing Cobertura XML was provided — it read the XML directly and gave a quick diagnostic answer without computing CRAP scores or following the output template. - Add existing-data guidance at top of Workflow section: skip test execution but still run CRAP score computation (Steps 5-6) - Add mandatory CRAP rule to guidelines.md: Risk Hotspots table is required in every output regardless of analysis mode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
92 lines
4.9 KiB
YAML
92 lines
4.9 KiB
YAML
scenarios:
|
|
- name: "Project-wide coverage analysis with existing Cobertura data"
|
|
prompt: "Run coverage analysis on this project. Coverage data is already in TestResults/coverage.cobertura.xml."
|
|
setup:
|
|
files:
|
|
- path: "ContosoUniversity.sln"
|
|
source: "fixtures/shared/ContosoUniversity.sln"
|
|
- path: "ContosoUniversity/ContosoUniversity.csproj"
|
|
source: "fixtures/shared/ContosoUniversity/ContosoUniversity.csproj"
|
|
- path: "ContosoUniversity/StudentService.cs"
|
|
source: "fixtures/shared/ContosoUniversity/StudentService.cs"
|
|
- path: "ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
- path: "ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
- path: "TestResults/coverage.cobertura.xml"
|
|
source: "fixtures/partial-coverage/coverage.cobertura.xml"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "CRAP|crap"
|
|
- type: "output_matches"
|
|
pattern: "CalculateGpa|Calculate_?Gpa"
|
|
- type: "output_matches"
|
|
pattern: "hotspot|risk|coverage"
|
|
rubric:
|
|
- "Discovers the solution and identifies the test project"
|
|
- "Parses the existing Cobertura XML to extract per-method coverage data"
|
|
- "Calculates CRAP scores and produces a sorted risk hotspot table"
|
|
- "Identifies CalculateGpa as the highest-risk method (high complexity, low coverage)"
|
|
- "Provides actionable recommendations for the riskiest methods"
|
|
- "Generates or displays a coverage analysis report"
|
|
timeout: 300
|
|
|
|
- name: "Run coverage from scratch without existing data"
|
|
prompt: "Analyze code coverage and risk hotspots for this project. I haven't run any coverage yet."
|
|
setup:
|
|
files:
|
|
- path: "ContosoUniversity.sln"
|
|
source: "fixtures/shared/ContosoUniversity.sln"
|
|
- path: "ContosoUniversity/ContosoUniversity.csproj"
|
|
source: "fixtures/shared/ContosoUniversity/ContosoUniversity.csproj"
|
|
- path: "ContosoUniversity/StudentService.cs"
|
|
source: "fixtures/shared/ContosoUniversity/StudentService.cs"
|
|
- path: "ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
- path: "ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "dotnet test"
|
|
- type: "output_matches"
|
|
pattern: "coverage"
|
|
- type: "output_matches"
|
|
pattern: "CRAP|crap|hotspot|risk"
|
|
rubric:
|
|
- "Detects the coverage provider from the test project .csproj (coverlet.collector)"
|
|
- "Runs dotnet test with coverage collection to generate Cobertura XML"
|
|
- "Extracts per-method coverage from the generated data"
|
|
- "Computes CRAP scores and identifies risk hotspots"
|
|
- "Identifies CalculateGpa as untested or highest risk due to complexity with no coverage"
|
|
- "Provides prioritized recommendations for where to add tests"
|
|
timeout: 360
|
|
|
|
- name: "Coverage plateau diagnosis"
|
|
prompt: "My coverage is stuck at 75% and I can't get it higher. What's blocking me? Coverage data is in TestResults/coverage.cobertura.xml."
|
|
setup:
|
|
files:
|
|
- path: "ContosoUniversity.sln"
|
|
source: "fixtures/shared/ContosoUniversity.sln"
|
|
- path: "ContosoUniversity/ContosoUniversity.csproj"
|
|
source: "fixtures/shared/ContosoUniversity/ContosoUniversity.csproj"
|
|
- path: "ContosoUniversity/StudentService.cs"
|
|
source: "fixtures/shared/ContosoUniversity/StudentService.cs"
|
|
- path: "ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
- path: "ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
- path: "TestResults/coverage.cobertura.xml"
|
|
source: "fixtures/plateau/coverage.cobertura.xml"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "CalculateGpa|Calculate_?Gpa"
|
|
- type: "output_matches"
|
|
pattern: "0%|uncovered|untested|no coverage"
|
|
rubric:
|
|
- "Identifies CalculateGpa as the method blocking coverage improvement (0% coverage, highest complexity)"
|
|
- "Explains that 3 of 4 methods are well-covered but CalculateGpa drags down the average"
|
|
- "Quantifies the impact — covering CalculateGpa would raise overall coverage significantly"
|
|
- "Uses CRAP scores to distinguish the risky gap (CalculateGpa) from trivial ones"
|
|
- "Provides specific test recommendations for the uncovered method"
|
|
timeout: 300
|