The 'Suggest hot reload for failing test in MTP project (SDK 9)' vally
stimulus forced bash usage on a purely advisory question, triggering a
slow SDK 9.0.200 NuGet restore that exceeded the timeout. Align with the
source-of-truth eval.yaml which does not constrain this scenario.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix evaluation timeouts and improve prompts for dotnet-test skills
- writing-mstest-tests: increase global vally timeout from 3m to 5m to match
the 300s eval.yaml timeout for the complex 'Modernize legacy test patterns' stimulus
- code-testing-agent: increase timeout from 40m/2400s to 60m/3600s
- mtp-hot-reload: increase timeout from 6m to 10m and remove overly strict
expect_tools constraint on launchSettings stimulus
- migrate-mstest-v1v2-to-v3: rephrase prompt for assertion overload errors to
mention CS1501 and cover AreNotEqual/AreSame alongside AreEqual
* Increase eval timeouts for mtp-hot-reload and migrate-mstest-v1v2-to-v3 scenarios
* Strengthen activation prompts and add missing vally scenarios
- migrate-mstest-v1v2-to-v3: Add explicit 'MSTest v2 to v3' migration
language in .testsettings and DataRow prompts (both eval.yaml and
eval.vally.yaml)
- writing-mstest-tests: Add 'MSTest' keyword and skill-specific
terminology to all prompts that lacked activation signal
- writing-mstest-tests: Add 5 missing vally scenarios (string
assertions, comparison assertions, collection/null/reference
assertions, conditional execution, parallelization)
- Add service-registry fixture files for the collection assertions
scenario
* Improve skill activation via SKILL.md descriptions and prompt alignment
Three-pronged fix for systematic NOT ACTIVATED failures:
1. SKILL.md descriptions: Restructure writing-mstest-tests, migrate-
mstest-v1v2-to-v3, and code-testing-agent descriptions to use the
proven 'Use when user says' pattern with quoted trigger phrases
(matching the working run-tests pattern)
2. Vally prompts: Each prompt now contains at least one exact quoted
trigger phrase from its SKILL.md description to ensure strong
description-to-prompt matching
3. Specific fixes:
- writing-mstest-tests: All 15 prompts now include MSTest-specific
trigger phrases (Assert.ThrowsExactly, sealed test class, DataRow,
DynamicData, Assert.HasCount, Assert.IsInstanceOfType, etc.)
- migrate-mstest-v1v2-to-v3: Add '.NET 5 dropped' and 'framework
compatibility issues' to trigger phrases; strengthen dropped-TFM
prompt
- code-testing-agent: Add 'generate comprehensive unit tests' and
'achieve high code coverage' to trigger phrases
* Revert SKILL.md description changes that caused activation regression
The description restructuring in 67056d8ec broke skill activation:
- migrate-mstest-v1v2-to-v3: went from 6/9 activating to 0/9
- writing-mstest-tests: lost isolated activation
- code-testing-agent: unnecessary change to working description
Restore all three SKILL.md files to their pre-regression state.
Keep the vally prompt improvements from 259727e.
* Fix writing-mstest-tests plugin-mode activation
- Add MSTest carve-out to code-testing-agent DO NOT USE FOR to prevent
it from intercepting MSTest-specific prompts
- Add test parallelization/MSTest.Sdk keywords to writing-mstest-tests
description (scenario 15 failed even in isolated mode)
- Strengthen prompts to be more MSTest-specific:
- Scenario 1: emphasize MSTest patterns over generic 'write tests'
- Scenario 4: remove vague 'something seems off', directly ask to
fix swapped Assert.AreEqual arguments
- Scenario 5: reframe as modernization request with specific APIs
- Scenario 13: name specific MSTest assertion APIs in the prompt
* Fix writing-mstest-tests description over 1024 char limit
Shorten the USE FOR section by removing redundant 'help me write
comprehensive tests' and condensing the assertion API list.
Description is now 944 chars (limit: 1024).
* Increase vally job timeout from 60m to 180m
The code-testing-agent eval has a 60m per-stimulus timeout, and vally
runs multiple configurations (baseline, isolated, plugin). The 60m
job-level timeout was causing the job to be cancelled mid-evaluation.
Match the 180m timeout from the evaluate job in evaluation.yml.
* Address PR review comments
- Add IsPackable=false to writing-mstest-tests fixture csproj for
consistency with other dotnet-test fixture projects
- Fix grammar in migrate-mstest-v1v2-to-v3 eval prompts: add comma
after 'upgrade' and add 'file' for clarity (both eval.yaml and
eval.vally.yaml)
* Trim dotnet-test aggregate description size under 15,000 limit
Consolidate duplicate filter keyword mentions in migrate-vstest-to-mtp
and shorten the cross-reference in migrate-xunit-to-xunit-v3 while
keeping the --filter-class/--filter-trait/--filter-query trigger
keywords for skill activation.
* Fix vally skill activation for 9 failing eval scenarios
Improve eval.vally.yaml prompts that were too specific, causing the model
to answer from general knowledge without activating the skill:
- migrate-mstest-v1v2-to-v3: Remove leading question that gives away the
answer ('Is .NET 5 dropped?'); use multi-target fixture matching eval.yaml
- writing-mstest-tests (7 scenarios): Add 'following MSTest best practices'
to trigger skill reading; make prompts more open-ended (don't name exact
APIs like OSCondition, Assert.AreSame); match eval.yaml prompt style
- code-testing-agent: Clarify app complexity to encourage skill pipeline
The pattern: vally prompts that give away exact API names or answers let
the model skip reading the skill. More open-ended prompts referencing
'best practices' trigger skill activation.
* Improve skills and eval prompts for regression scenarios
migrate-xunit-to-xunit-v3 SKILL.md:
- Restructure from 16 steps to 13, merging redundant steps
- Move test platform selection from step 12 to step 4
- Mark steps 6-12 as conditional with prioritization note
- Remove external URL references that caused web_fetch detours
writing-mstest-tests SKILL.md:
- Add Response Guidelines for proportional responses
- Remove redundant Validation checklist and Common Pitfalls table
(all items already covered in workflow steps)
- Reduces token count from 3,305 to 2,944
migrate-mstest-v1v2-to-v3 eval.vally.yaml:
- Relax DataRow rubric to account for MSTest 3.8.0 fixing the
16-parameter constructor limit
writing-mstest-tests eval.vally.yaml:
- Make 3 prompts more open-ended by removing API name hints
(Assert.HasCount, Assert.StartsWith, DynamicData to ValueTuples)
so baseline model cannot answer confidently without the skill
The 5 failing scenarios were timing out because expect_tools: [bash]
forced the agent to execute terminal commands. The hot reload workflow
involves 'dotnet run' which starts a test host that blocks indefinitely
waiting for file changes, causing the scenario to exhaust its timeout.
These scenarios test whether the model provides correct guidance (package
name, env var, commands), not whether it actually executes them. The
output_matches assertions already verify the response content.
Removed expect_tools from:
- Suggest hot reload for failing test in MTP project (SDK 9)
- Suggest hot reload for failing test in MTP project (SDK 10)
- Enable hot reload when package already installed
- Suggest launchSettings.json configuration for hot reload
- Negative: VSTest project cannot use MTP hot reload
Also reduced timeouts since scenarios no longer need to wait for command
execution.
Add rollForward: latestFeature to all global.json entries that pin SDK 9.0.200
so that any 9.0.xxx SDK satisfies the requirement. Without this, machines with
only 9.0.3xx installed fail all dotnet commands during evaluation, causing the
model to burn its entire timeout budget on failed bash retries.
Aditya Mandaleeka