The 'Use comparison assertions for boundary testing' scenario chronically hit its 240s wall-clock timeout. Raise it to 360s to match the other code-generation scenarios in this eval file, per InvestigatingResults.md timeout guidance.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Several dotnet-test eval scenarios repeatedly hit their wall-clock timeout
in the evaluation dashboard (skilled + plugin arms), producing empty output
and failing all assertions. Increase per-scenario timeouts for the ones that
timed out most often across recent runs:
- writing-mstest-tests: 'Write unit tests for a service class',
'Write data-driven tests for a calculator',
'Use string assertions for format validation' 180->360;
'Use comparison assertions for boundary testing' 120->240
- coverage-analysis: 'Coverage plateau diagnosis',
'Project-wide coverage analysis with existing Cobertura data' 300->480
- test-gap-analysis: 'Decline request to write new tests from scratch' 120->300;
'Acknowledge well-tested code with few surviving mutations' 300->420
- run-tests: 'Filter xUnit v3 tests by trait on MTP',
'Filter xUnit v3 tests by class pattern and trait using query filter language' 120->240
Per eng/skill-validator/src/docs/InvestigatingResults.md, timeouts are the
highest-priority failure mode; code-generation scenarios often need 360s.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Add eval coverage for dotnet-test/writing-mstest-tests
Enrich the ServiceRegistry collection/null/reference scenario with deterministic file_contains assertions for the modern MSTest assertion APIs Assert.IsNull, Assert.AreSame, Assert.Contains, Assert.DoesNotContain, Assert.IsEmpty, and Assert.IsNotEmpty.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Reject legacy CollectionAssert/StringAssert helpers in collection scenario
Address PR review: file_contains is a case-sensitive substring check, so the Assert.Contains and Assert.DoesNotContain value checks would also pass for CollectionAssert.* and StringAssert.* helpers. Add file_not_contains guards to keep the scenario a deterministic check for the modern Assert.* API.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix evaluation timeouts and improve prompts for dotnet-test skills
- writing-mstest-tests: increase global vally timeout from 3m to 5m to match
the 300s eval.yaml timeout for the complex 'Modernize legacy test patterns' stimulus
- code-testing-agent: increase timeout from 40m/2400s to 60m/3600s
- mtp-hot-reload: increase timeout from 6m to 10m and remove overly strict
expect_tools constraint on launchSettings stimulus
- migrate-mstest-v1v2-to-v3: rephrase prompt for assertion overload errors to
mention CS1501 and cover AreNotEqual/AreSame alongside AreEqual
* Increase eval timeouts for mtp-hot-reload and migrate-mstest-v1v2-to-v3 scenarios
* Strengthen activation prompts and add missing vally scenarios
- migrate-mstest-v1v2-to-v3: Add explicit 'MSTest v2 to v3' migration
language in .testsettings and DataRow prompts (both eval.yaml and
eval.vally.yaml)
- writing-mstest-tests: Add 'MSTest' keyword and skill-specific
terminology to all prompts that lacked activation signal
- writing-mstest-tests: Add 5 missing vally scenarios (string
assertions, comparison assertions, collection/null/reference
assertions, conditional execution, parallelization)
- Add service-registry fixture files for the collection assertions
scenario
* Improve skill activation via SKILL.md descriptions and prompt alignment
Three-pronged fix for systematic NOT ACTIVATED failures:
1. SKILL.md descriptions: Restructure writing-mstest-tests, migrate-
mstest-v1v2-to-v3, and code-testing-agent descriptions to use the
proven 'Use when user says' pattern with quoted trigger phrases
(matching the working run-tests pattern)
2. Vally prompts: Each prompt now contains at least one exact quoted
trigger phrase from its SKILL.md description to ensure strong
description-to-prompt matching
3. Specific fixes:
- writing-mstest-tests: All 15 prompts now include MSTest-specific
trigger phrases (Assert.ThrowsExactly, sealed test class, DataRow,
DynamicData, Assert.HasCount, Assert.IsInstanceOfType, etc.)
- migrate-mstest-v1v2-to-v3: Add '.NET 5 dropped' and 'framework
compatibility issues' to trigger phrases; strengthen dropped-TFM
prompt
- code-testing-agent: Add 'generate comprehensive unit tests' and
'achieve high code coverage' to trigger phrases
* Revert SKILL.md description changes that caused activation regression
The description restructuring in 67056d8ec broke skill activation:
- migrate-mstest-v1v2-to-v3: went from 6/9 activating to 0/9
- writing-mstest-tests: lost isolated activation
- code-testing-agent: unnecessary change to working description
Restore all three SKILL.md files to their pre-regression state.
Keep the vally prompt improvements from 259727e.
* Fix writing-mstest-tests plugin-mode activation
- Add MSTest carve-out to code-testing-agent DO NOT USE FOR to prevent
it from intercepting MSTest-specific prompts
- Add test parallelization/MSTest.Sdk keywords to writing-mstest-tests
description (scenario 15 failed even in isolated mode)
- Strengthen prompts to be more MSTest-specific:
- Scenario 1: emphasize MSTest patterns over generic 'write tests'
- Scenario 4: remove vague 'something seems off', directly ask to
fix swapped Assert.AreEqual arguments
- Scenario 5: reframe as modernization request with specific APIs
- Scenario 13: name specific MSTest assertion APIs in the prompt
* Fix writing-mstest-tests description over 1024 char limit
Shorten the USE FOR section by removing redundant 'help me write
comprehensive tests' and condensing the assertion API list.
Description is now 944 chars (limit: 1024).
* Increase vally job timeout from 60m to 180m
The code-testing-agent eval has a 60m per-stimulus timeout, and vally
runs multiple configurations (baseline, isolated, plugin). The 60m
job-level timeout was causing the job to be cancelled mid-evaluation.
Match the 180m timeout from the evaluate job in evaluation.yml.
* Address PR review comments
- Add IsPackable=false to writing-mstest-tests fixture csproj for
consistency with other dotnet-test fixture projects
- Fix grammar in migrate-mstest-v1v2-to-v3 eval prompts: add comma
after 'upgrade' and add 'file' for clarity (both eval.yaml and
eval.vally.yaml)
* Trim dotnet-test aggregate description size under 15,000 limit
Consolidate duplicate filter keyword mentions in migrate-vstest-to-mtp
and shorten the cross-reference in migrate-xunit-to-xunit-v3 while
keeping the --filter-class/--filter-trait/--filter-query trigger
keywords for skill activation.
* Fix vally skill activation for 9 failing eval scenarios
Improve eval.vally.yaml prompts that were too specific, causing the model
to answer from general knowledge without activating the skill:
- migrate-mstest-v1v2-to-v3: Remove leading question that gives away the
answer ('Is .NET 5 dropped?'); use multi-target fixture matching eval.yaml
- writing-mstest-tests (7 scenarios): Add 'following MSTest best practices'
to trigger skill reading; make prompts more open-ended (don't name exact
APIs like OSCondition, Assert.AreSame); match eval.yaml prompt style
- code-testing-agent: Clarify app complexity to encourage skill pipeline
The pattern: vally prompts that give away exact API names or answers let
the model skip reading the skill. More open-ended prompts referencing
'best practices' trigger skill activation.
* Improve skills and eval prompts for regression scenarios
migrate-xunit-to-xunit-v3 SKILL.md:
- Restructure from 16 steps to 13, merging redundant steps
- Move test platform selection from step 12 to step 4
- Mark steps 6-12 as conditional with prioritization note
- Remove external URL references that caused web_fetch detours
writing-mstest-tests SKILL.md:
- Add Response Guidelines for proportional responses
- Remove redundant Validation checklist and Common Pitfalls table
(all items already covered in workflow steps)
- Reduces token count from 3,305 to 2,944
migrate-mstest-v1v2-to-v3 eval.vally.yaml:
- Relax DataRow rubric to account for MSTest 3.8.0 fixing the
16-parameter constructor limit
writing-mstest-tests eval.vally.yaml:
- Make 3 prompts more open-ended by removing API name hints
(Assert.HasCount, Assert.StartsWith, DynamicData to ValueTuples)
so baseline model cannot answer confidently without the skill
The scenario was timing out on CI because:
- The 180s timeout left insufficient headroom for CI machines
(local runs take 120-135s, CI machines are slower due to load,
network NuGet restores, and parallel execution)
Changes:
- Add 'dotnet restore' setup command to pre-warm NuGet cache before
the agent starts (same pattern used by migrate-mstest-v3-to-v4)
- Increase timeout from 180s to 300s to accommodate CI variability
* Add writing-mstest-tests skill with eval scenarios
Add a new skill for writing modern MSTest 3.x/4.x unit tests covering:
- Test class structure (sealed, AAA pattern, naming conventions)
- Modern assertion APIs (Assert.Throws, IsInstanceOfType, collection asserts)
- Data-driven tests (DataRow, DynamicData with ValueTuples)
- Async tests with cancellation and timeout patterns
- Test lifecycle (constructor init, TestContext injection)
- Anti-pattern detection and modernization
Includes 10 eval scenarios across 4 goal categories with fixture files
for both modern MSTest and legacy anti-pattern test projects.
MSTest.Sdk version is recommended via global.json msbuild-sdks instead of
inline in the project file, ensuring all test projects get bumped together.
* Fixes
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Apply suggestion from @Evangelink
* Apply suggestion from @Evangelink
---------
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Aditya Mandaleeka