skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
473 Commits 131 Branches 2 Tags
dev/jankrivanek/eval-dispatch-plugin-input
Commit Graph

7 Commits

Author SHA1 Message Date
Amaury Levé 04348a46ef tests/dotnet-test/code-testing-agent: fix coverage XML lookup path (#657) 
The grader expected coverage.cobertura.xml under a workspace-root TestResults/ directory, but `dotnet test --collect:"XPlat Code Coverage"` writes the report under <TestProject>/TestResults/<guid>/coverage.cobertura.xml. With the previous path, `find TestResults …` failed with no-such-directory before even searching, and the glob `TestResults/**/coverage.cobertura.xml` could not match the nested location either, so the assertion failed on otherwise-successful runs.

Search recursively from cwd in both eval.vally.yaml (`find . …`) and eval.yaml (`**/TestResults/**/coverage.cobertura.xml`) so the assertion matches the actual output location of XPlat Code Coverage regardless of the test project layout the agent chose.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-05-18 09:26:26 +00:00
Amaury Levé 3d59e44c7e Fix coverage-analysis activation for plateau diagnosis prompts (#647) 
* Fix coverage-analysis activation for plateau diagnosis prompts

coverage-analysis SKILL.md:
- Trim verbose implementation details (provider detection,
  ReportGenerator) that consumed description budget without
  aiding skill activation
- Add explicit USE FOR keywords: coverage stuck, coverage plateau,
  can't increase coverage, what's blocking coverage

code-testing-agent SKILL.md:
- Add 'diagnosing coverage plateaus or CRAP score computation
  (use coverage-analysis)' to DO NOT USE FOR boundary to prevent
  test-generation skill from intercepting diagnostic prompts

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Strengthen code-testing-agent activation; harden coverage-analysis isolated mode

Address eval regressions reported on PR #647 (run 25813728646):

1. code-testing-agent: `Generate tests for ContosoUniversity ASP.NET Core MVC app`
   was NOT ACTIVATED in plugin mode (detectedSkills=[], skillEventCount=0,
   invokedAgents=[]). The model bypassed the skill system entirely.

   - SKILL.md description: restructure to use the proven `Use when user says ...`
     pattern with quoted trigger phrases (matching the run-tests skill that
     consistently activates), make the link to the code-testing-generator
     sub-agent explicit, and tighten DO NOT USE FOR clauses.
   - eval prompt (eval.yaml + eval.vally.yaml): make the request
     pipeline-shaped (`project-wide, multi-file test generation task`,
     `scaffold a new test project`) so the model recognizes it as multi-step
     work that benefits from the orchestrated pipeline. Explicitly request
     coverlet.collector + a Cobertura XML run so rubric criterion 1
     (`high line coverage as reported by the Cobertura XML in TestResults/`)
     becomes achievable without overfitting.

2. code-testing-tester agent + code-testing-extensions/dotnet.md: open a
   scoped exception to the `skip coverage tools` rule. Default behavior
   stays the same, but when the user/harness explicitly asks for a
   Cobertura/XML coverage artifact, the agent may add coverlet.collector
   to the generated test csproj so the harness's coverage command produces
   output. The agent still does not run the coverage command itself.

3. coverage-analysis SKILL.md: add a `User-visible output is mandatory`
   guard at the top of the Workflow section. The latest eval showed isolated
   mode producing literally `(no output)` in 2 of 3 scenarios — the agent
   ran Compute-CrapScores.ps1 / Extract-MethodCoverage.ps1 / ReportGenerator
   in parallel, then the session ended without ever surfacing findings.
   The guard tells the agent to always return a partial summary instead of
   ending silent, and to deprioritize ReportGenerator HTML when budget is
   tight. (Plugin-mode quality is already strong: 4.3 / 4.3 / 5.0 — no
   regression risk there.)

Aggregate dotnet-test plugin description size: 14,925 chars (limit 15,000).
skill-validator check passes (22 skills, 11 agents, 1 plugin); markdownlint
passes for all 4 modified files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Restore MSTest modernization exclusion in code-testing-agent description

* Fix isolated-mode coverage-analysis: emit summary before optional ReportGenerator

The previous workflow encouraged the agent to run `dotnet tool install` for
ReportGenerator in parallel with the CRAP scoring scripts (Phase 2 "Steps
3 and 4 in parallel" + Phase 3 "Steps 5 and 6 in parallel"). In isolated
mode that pattern reliably crashed the session with "Failed to persist
session events: timeout while waiting for mutex to become available"
right after the scripts returned valid data, so the agent never produced
the user-facing summary.

Restructure the workflow into 5 phases:

- Phase 1 (Setup) - unchanged
- Phase 2 (Test execution) - skip when Cobertura XML already exists
- Phase 3 (Analysis) - run only the two PowerShell scripts, no RG
- Phase 4 (User-facing summary) - MANDATORY, must be the next assistant
  response after Phase 3, before any RG work; also save
  coverage-analysis.md as a secondary follow-up
- Phase 5 (ReportGenerator HTML/CSV) - strictly optional, post-summary,
  skipped by default for existing-Cobertura and plateau-diagnosis paths

Also update references/output-format.md so the Reports section marks RG
artifacts as "Not generated (optional - request HTML reports to enable)"
when Phase 5 has not run, and update references/guidelines.md so the
"show and open the markdown report" rule explicitly defers to the
user-facing assistant response.

Targets the isolated-mode regressions in PR #647 eval:
- Project-wide coverage with existing Cobertura: 1.0/5 -> expected 3+
- Coverage plateau diagnosis: 1.0/5 -> expected 3+
- Run coverage from scratch: 2.3/5 -> expected steady or up

Verified: skill-validator check --plugin ./plugins/dotnet-test passes;
markdownlint-cli2 clean on all 3 modified files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR #647 review comments

1. Prerequisites: distinguish the from-scratch path (needs NuGet for the
   coverage-provider package install + optional internet for ReportGenerator)
   from the existing-Cobertura path (needs neither).

2. Add Step 2c "Discover or accept existing Cobertura XML" so the
   existing-data path actually has $coberturaFiles populated before
   Phase 3, instead of relying on Phase 2's discovery (which it
   skips). Also clarify Step 2's destructive Remove-Item only manages
   the skill-owned coverage-analysis/ subdirectory.

3. references/output-format.md: replace the unconditional
   "Reports saved to: <coverageDir>/reports/" line with one that
   always points at <coverageDir>/ (markdown summary + raw Cobertura)
   and only mentions reports/ if Phase 5 ran.

4. Have Compute-CrapScores.ps1 emit OVERALL_LINE_COVERAGE and
   OVERALL_BRANCH_COVERAGE from the Cobertura root attributes, and
   update Phase 4 to read those values directly from the script's
   output. The Phase 4 mandatory-summary rule no longer requires a
   separate XML parse before composing the response.

Verified: skill-validator check --plugin ./plugins/dotnet-test passes;
markdownlint-cli2 clean on all modified files; Compute-CrapScores.ps1
smoke-tested on a synthetic Cobertura XML (emits OVERALL_LINE_COVERAGE:75,
OVERALL_BRANCH_COVERAGE:50 alongside HOTSPOTS).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Apply suggestion from @Evangelink

* Address unresolved coverage-analysis and eval review comments

* Refine follow-up review feedback from validation

* Tighten coverage aggregation fallback notes and counters

* Clarify pre-response save instruction wording

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
 2026-05-14 17:41:31 +00:00
Amaury Levé 0f31228b58 Fix eval timeouts and skill activation for dotnet-test evals (#641) 
* Fix evaluation timeouts and improve prompts for dotnet-test skills

- writing-mstest-tests: increase global vally timeout from 3m to 5m to match
  the 300s eval.yaml timeout for the complex 'Modernize legacy test patterns' stimulus
- code-testing-agent: increase timeout from 40m/2400s to 60m/3600s
- mtp-hot-reload: increase timeout from 6m to 10m and remove overly strict
  expect_tools constraint on launchSettings stimulus
- migrate-mstest-v1v2-to-v3: rephrase prompt for assertion overload errors to
  mention CS1501 and cover AreNotEqual/AreSame alongside AreEqual

* Increase eval timeouts for mtp-hot-reload and migrate-mstest-v1v2-to-v3 scenarios

* Strengthen activation prompts and add missing vally scenarios

- migrate-mstest-v1v2-to-v3: Add explicit 'MSTest v2 to v3' migration
  language in .testsettings and DataRow prompts (both eval.yaml and
  eval.vally.yaml)
- writing-mstest-tests: Add 'MSTest' keyword and skill-specific
  terminology to all prompts that lacked activation signal
- writing-mstest-tests: Add 5 missing vally scenarios (string
  assertions, comparison assertions, collection/null/reference
  assertions, conditional execution, parallelization)
- Add service-registry fixture files for the collection assertions
  scenario

* Improve skill activation via SKILL.md descriptions and prompt alignment

Three-pronged fix for systematic NOT ACTIVATED failures:

1. SKILL.md descriptions: Restructure writing-mstest-tests, migrate-
   mstest-v1v2-to-v3, and code-testing-agent descriptions to use the
   proven 'Use when user says' pattern with quoted trigger phrases
   (matching the working run-tests pattern)

2. Vally prompts: Each prompt now contains at least one exact quoted
   trigger phrase from its SKILL.md description to ensure strong
   description-to-prompt matching

3. Specific fixes:
   - writing-mstest-tests: All 15 prompts now include MSTest-specific
     trigger phrases (Assert.ThrowsExactly, sealed test class, DataRow,
     DynamicData, Assert.HasCount, Assert.IsInstanceOfType, etc.)
   - migrate-mstest-v1v2-to-v3: Add '.NET 5 dropped' and 'framework
     compatibility issues' to trigger phrases; strengthen dropped-TFM
     prompt
   - code-testing-agent: Add 'generate comprehensive unit tests' and
     'achieve high code coverage' to trigger phrases

* Revert SKILL.md description changes that caused activation regression

The description restructuring in 67056d8ec broke skill activation:
- migrate-mstest-v1v2-to-v3: went from 6/9 activating to 0/9
- writing-mstest-tests: lost isolated activation
- code-testing-agent: unnecessary change to working description

Restore all three SKILL.md files to their pre-regression state.
Keep the vally prompt improvements from 259727e.

* Fix writing-mstest-tests plugin-mode activation

- Add MSTest carve-out to code-testing-agent DO NOT USE FOR to prevent
  it from intercepting MSTest-specific prompts
- Add test parallelization/MSTest.Sdk keywords to writing-mstest-tests
  description (scenario 15 failed even in isolated mode)
- Strengthen prompts to be more MSTest-specific:
  - Scenario 1: emphasize MSTest patterns over generic 'write tests'
  - Scenario 4: remove vague 'something seems off', directly ask to
    fix swapped Assert.AreEqual arguments
  - Scenario 5: reframe as modernization request with specific APIs
  - Scenario 13: name specific MSTest assertion APIs in the prompt

* Fix writing-mstest-tests description over 1024 char limit

Shorten the USE FOR section by removing redundant 'help me write
comprehensive tests' and condensing the assertion API list.
Description is now 944 chars (limit: 1024).

* Increase vally job timeout from 60m to 180m

The code-testing-agent eval has a 60m per-stimulus timeout, and vally
runs multiple configurations (baseline, isolated, plugin). The 60m
job-level timeout was causing the job to be cancelled mid-evaluation.

Match the 180m timeout from the evaluate job in evaluation.yml.

* Address PR review comments

- Add IsPackable=false to writing-mstest-tests fixture csproj for
  consistency with other dotnet-test fixture projects
- Fix grammar in migrate-mstest-v1v2-to-v3 eval prompts: add comma
  after 'upgrade' and add 'file' for clarity (both eval.yaml and
  eval.vally.yaml)

* Trim dotnet-test aggregate description size under 15,000 limit

Consolidate duplicate filter keyword mentions in migrate-vstest-to-mtp
and shorten the cross-reference in migrate-xunit-to-xunit-v3 while
keeping the --filter-class/--filter-trait/--filter-query trigger
keywords for skill activation.

* Fix vally skill activation for 9 failing eval scenarios

Improve eval.vally.yaml prompts that were too specific, causing the model
to answer from general knowledge without activating the skill:

- migrate-mstest-v1v2-to-v3: Remove leading question that gives away the
  answer ('Is .NET 5 dropped?'); use multi-target fixture matching eval.yaml
- writing-mstest-tests (7 scenarios): Add 'following MSTest best practices'
  to trigger skill reading; make prompts more open-ended (don't name exact
  APIs like OSCondition, Assert.AreSame); match eval.yaml prompt style
- code-testing-agent: Clarify app complexity to encourage skill pipeline

The pattern: vally prompts that give away exact API names or answers let
the model skip reading the skill. More open-ended prompts referencing
'best practices' trigger skill activation.

* Improve skills and eval prompts for regression scenarios

migrate-xunit-to-xunit-v3 SKILL.md:
- Restructure from 16 steps to 13, merging redundant steps
- Move test platform selection from step 12 to step 4
- Mark steps 6-12 as conditional with prioritization note
- Remove external URL references that caused web_fetch detours

writing-mstest-tests SKILL.md:
- Add Response Guidelines for proportional responses
- Remove redundant Validation checklist and Common Pitfalls table
  (all items already covered in workflow steps)
- Reduces token count from 3,305 to 2,944

migrate-mstest-v1v2-to-v3 eval.vally.yaml:
- Relax DataRow rubric to account for MSTest 3.8.0 fixing the
  16-parameter constructor limit

writing-mstest-tests eval.vally.yaml:
- Make 3 prompts more open-ended by removing API name hints
  (Assert.HasCount, Assert.StartsWith, DynamicData to ValueTuples)
  so baseline model cannot answer confidently without the skill
 2026-05-13 17:49:55 +02:00
Aditya Mandaleeka 9433a31e67 Initial port of evals to Vally (#615) 2026-05-06 10:11:45 -07:00
Amaury Levé 17df9a10eb Fix code-testing-agent eval timeout by removing unused NuGet packages (#551) 
The ContosoUniversity test fixture had 17 unused NuGet packages (bootstrap,
jQuery, Modernizr, Antlr4, WebGrease, Microsoft.Build.Tasks.Core, etc.)
that bloated NuGet restore and build time. The multi-agent test generation
pipeline performs multiple build/test cycles, causing cumulative delays
that exceeded the 2400s scenario timeout on CI.

Changes:
- Remove 17 unused PackageReference entries from ContosoUniversity.csproj
- Remove stale ItemGroup entries (Global.asax, Scripts, compile removes)
- Remove NU1701 warning suppression (no longer needed)
- Increase command_timeout from 180s to 300s for dotnet test assertion
  (accounts for first-time restore + build + test + coverage collection)
 2026-04-20 12:53:45 +00:00
Amaury Levé 400661df09 run-tests: add critical rules, detection guidance, and troubleshooting (#514) 
* run-tests: add critical rules, detection guidance, and troubleshooting

- Add 'Critical Rules' table to prevent cross-platform VSTest/MTP mistakes
- Expand Step 1 detection with explicit file-by-file lookup table
- Add 'dotnet --version' as first detection action
- Add Troubleshooting section with 7 common error patterns
- Expand Common Pitfalls from 3 to 7 entries
- Strengthen negative guidance for SDK version-specific syntax

* Improve description

* Improve evaluation

* Add expected and rejected tools

* Change

* Improve do not use sections

* Improve prompt

* Increase eval timeouts for scenarios hitting the limit

- run-tests: trx reporting MTP SDK 9 (360s -> 480s)
- run-tests: blame-hang MTP SDK 10 (300s -> 420s)
- run-tests: combine filter criteria VSTest (180s -> 300s)
- mtp-hot-reload: hot reload SDK 9 (360s -> 480s)
- mtp-hot-reload: hot reload filter (180s -> 300s)
- code-testing-agent: ContosoUniversity (1800s -> 2400s)
 2026-04-20 10:09:15 +02:00
Jan Krivanek e4670b33a1 [PoC] Code testing agent + tests PoC (#433) 
* Add code testing agent agents + tests PoC

* Remove AssertionEvaluator.cs change (moved to dev/jankrivanek/agents-evals)
 2026-03-30 17:57:08 +02:00