mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Amaury Levé
ce75c35569
Several dotnet-test eval scenarios repeatedly hit their wall-clock timeout in the evaluation dashboard (skilled + plugin arms), producing empty output and failing all assertions. Increase per-scenario timeouts for the ones that timed out most often across recent runs: - writing-mstest-tests: 'Write unit tests for a service class', 'Write data-driven tests for a calculator', 'Use string assertions for format validation' 180->360; 'Use comparison assertions for boundary testing' 120->240 - coverage-analysis: 'Coverage plateau diagnosis', 'Project-wide coverage analysis with existing Cobertura data' 300->480 - test-gap-analysis: 'Decline request to write new tests from scratch' 120->300; 'Acknowledge well-tested code with few surviving mutations' 300->420 - run-tests: 'Filter xUnit v3 tests by trait on MTP', 'Filter xUnit v3 tests by class pattern and trait using query filter language' 120->240 Per eng/skill-validator/src/docs/InvestigatingResults.md, timeouts are the highest-priority failure mode; code-generation scenarios often need 360s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
92 lines
4.9 KiB
YAML
92 lines
4.9 KiB
YAML
scenarios:
|
|
- name: "Project-wide coverage analysis with existing Cobertura data"
|
|
prompt: "Run coverage analysis on this project. Coverage data is already in TestResults/coverage.cobertura.xml."
|
|
setup:
|
|
files:
|
|
- path: "ContosoUniversity.sln"
|
|
source: "fixtures/shared/ContosoUniversity.sln"
|
|
- path: "ContosoUniversity/ContosoUniversity.csproj"
|
|
source: "fixtures/shared/ContosoUniversity/ContosoUniversity.csproj"
|
|
- path: "ContosoUniversity/StudentService.cs"
|
|
source: "fixtures/shared/ContosoUniversity/StudentService.cs"
|
|
- path: "ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
- path: "ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
- path: "TestResults/coverage.cobertura.xml"
|
|
source: "fixtures/partial-coverage/coverage.cobertura.xml"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "CRAP|crap"
|
|
- type: "output_matches"
|
|
pattern: "CalculateGpa|Calculate_?Gpa"
|
|
- type: "output_matches"
|
|
pattern: "hotspot|risk|coverage"
|
|
rubric:
|
|
- "Discovers the solution and identifies the test project"
|
|
- "Parses the existing Cobertura XML to extract per-method coverage data"
|
|
- "Calculates CRAP scores and produces a sorted risk hotspot table"
|
|
- "Identifies CalculateGpa as the highest-risk method (high complexity, low coverage)"
|
|
- "Provides actionable recommendations for the riskiest methods"
|
|
- "Generates or displays a coverage analysis report"
|
|
timeout: 480
|
|
|
|
- name: "Run coverage from scratch without existing data"
|
|
prompt: "Analyze code coverage and risk hotspots for this project. I haven't run any coverage yet."
|
|
setup:
|
|
files:
|
|
- path: "ContosoUniversity.sln"
|
|
source: "fixtures/shared/ContosoUniversity.sln"
|
|
- path: "ContosoUniversity/ContosoUniversity.csproj"
|
|
source: "fixtures/shared/ContosoUniversity/ContosoUniversity.csproj"
|
|
- path: "ContosoUniversity/StudentService.cs"
|
|
source: "fixtures/shared/ContosoUniversity/StudentService.cs"
|
|
- path: "ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
- path: "ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "dotnet test"
|
|
- type: "output_matches"
|
|
pattern: "coverage"
|
|
- type: "output_matches"
|
|
pattern: "CRAP|crap|hotspot|risk"
|
|
rubric:
|
|
- "Detects the coverage provider from the test project .csproj (coverlet.collector)"
|
|
- "Runs dotnet test with coverage collection to generate Cobertura XML"
|
|
- "Extracts per-method coverage from the generated data"
|
|
- "Computes CRAP scores and identifies risk hotspots"
|
|
- "Identifies CalculateGpa as untested or highest risk due to complexity with no coverage"
|
|
- "Provides prioritized recommendations for where to add tests"
|
|
timeout: 360
|
|
|
|
- name: "Coverage plateau diagnosis"
|
|
prompt: "My coverage is stuck at 75% and I can't get it higher. What's blocking me? Coverage data is in TestResults/coverage.cobertura.xml."
|
|
setup:
|
|
files:
|
|
- path: "ContosoUniversity.sln"
|
|
source: "fixtures/shared/ContosoUniversity.sln"
|
|
- path: "ContosoUniversity/ContosoUniversity.csproj"
|
|
source: "fixtures/shared/ContosoUniversity/ContosoUniversity.csproj"
|
|
- path: "ContosoUniversity/StudentService.cs"
|
|
source: "fixtures/shared/ContosoUniversity/StudentService.cs"
|
|
- path: "ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/ContosoUniversity.UnitTests.csproj"
|
|
- path: "ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
source: "fixtures/shared/ContosoUniversity.UnitTests/StudentServiceTests.cs"
|
|
- path: "TestResults/coverage.cobertura.xml"
|
|
source: "fixtures/plateau/coverage.cobertura.xml"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "CalculateGpa|Calculate_?Gpa"
|
|
- type: "output_matches"
|
|
pattern: "0%|uncovered|untested|no coverage"
|
|
rubric:
|
|
- "Identifies CalculateGpa as the method blocking coverage improvement (0% coverage, highest complexity)"
|
|
- "Explains that 3 of 4 methods are well-covered but CalculateGpa drags down the average"
|
|
- "Quantifies the impact — covering CalculateGpa would raise overall coverage significantly"
|
|
- "Uses CRAP scores to distinguish the risky gap (CalculateGpa) from trivial ones"
|
|
- "Provides specific test recommendations for the uncovered method"
|
|
timeout: 480
|