Several dotnet-test eval scenarios repeatedly hit their wall-clock timeout
in the evaluation dashboard (skilled + plugin arms), producing empty output
and failing all assertions. Increase per-scenario timeouts for the ones that
timed out most often across recent runs:
- writing-mstest-tests: 'Write unit tests for a service class',
'Write data-driven tests for a calculator',
'Use string assertions for format validation' 180->360;
'Use comparison assertions for boundary testing' 120->240
- coverage-analysis: 'Coverage plateau diagnosis',
'Project-wide coverage analysis with existing Cobertura data' 300->480
- test-gap-analysis: 'Decline request to write new tests from scratch' 120->300;
'Acknowledge well-tested code with few surviving mutations' 300->420
- run-tests: 'Filter xUnit v3 tests by trait on MTP',
'Filter xUnit v3 tests by class pattern and trait using query filter language' 120->240
Per eng/skill-validator/src/docs/InvestigatingResults.md, timeouts are the
highest-priority failure mode; code-generation scenarios often need 360s.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* run-tests: fix evals (query-filter regex, sibling skills, command observability)
Investigated run-tests eval failures by running the validator locally.
- SKILL.md Step 3: document xUnit v3 --filter-query so the agent stops
answering that complex xUnit v3 filters 'cannot be combined'.
- eval.yaml: fix a broken assertion regex. The query-filter pattern is a
single-quoted YAML scalar using '\\s'/'\\[', which (unlike a double-quoted
scalar) is NOT unescaped, so the regex searched for a literal '\s' and
could never match. Corrected to single backslashes.
- eval.yaml: add additional_required_skills (filter-syntax / platform-detection)
to the filter and detection scenarios, so the isolated arm loads the sibling
reference skills that run-tests explicitly defers to.
- eval.yaml: ask the agent to show the exact command in execute-style prompts.
output_matches only sees the final assistant message; 'run my tests' prompts
make the agent execute and summarize ('tests passed'), so the recommended
command never appears. The assertions still catch wrong commands.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* run-tests: rewrite description for reliable plugin-arm activation
The run-tests skill activated reliably in the isolated eval arm but
unreliably in the plugin arm (all dotnet-test sibling descriptions
loaded), so every scenario's pluginImprovementScore went negative and
dragged min(isolated, plugin) below zero.
Lead the description with natural-language intent triggers that mirror
how the eval prompts phrase requests (run all tests, run a subset via
filters, produce TRX reports, collect crash/hang dumps, run a single
TFM) instead of opening with platform-detection mechanism, and add
explicit DO NOT USE redirects to code-testing-agent / mtp-hot-reload.
Stays under the 1024-char description cap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dotnet-test: fix run-tests plugin-arm activation via skill-menu budget
Root cause (verified against the Copilot CLI SDK skill renderer): the
model-facing skill menu has a 15000-char budget. Skills are listed
alphabetically and emitted with full descriptions only until the budget
is exhausted; the rest collapse to bare names with no description and
effectively cannot be model-activated. With 27 dotnet-test skills,
run-tests (alphabetical position ~20) fell into the name-only overflow,
so it never activated in the plugin eval arm even though it activated
reliably in isolation. This is a real user-facing discoverability bug,
not just an eval artifact.
Fix: hide reference/primitive skills that are never meant to be
model-invoked from the menu via 'disable-model-invocation: true', which
the SDK filters out of the budget entirely:
- filter-syntax, platform-detection, dotnet-test-frameworks,
code-testing-extensions, test-analysis-extensions — already
user-invocable:false reference data ('DO NOT USE directly').
- find-untested-sources, find-untested-sources-polyglot — researcher
primitives invoked by-name from the code-testing-researcher agent
(which has a manual fallback); no standalone evals.
These remain invocable by explicit name (agents/users), only auto-
suggestion is suppressed.
This frees enough budget that run-tests (plus migrate-xunit-to-xunit-v3
and mtp-hot-reload) now receive full descriptions; no previously-visible
skill regresses. Also trimmed the run-tests description so its menu block
fits with margin while keeping all activation triggers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* run-tests: use prompt-shaped multi-TFM keywords for plugin activation
The 'Run tests in a multi-TFM project targeting a specific framework' eval
scenario kept failing plugin-mode activation after #667. The skill description
had 'multi-TFM projects (--framework)' as a brief parenthetical, but the
scenario prompt uses different language: 'This test project targets both
net8.0 and net9.0. I only want to run the tests on net9.0 using dotnet test.'
Per the InvestigatingResults.md guidance for 'skill not activated' failures,
expand that clause to include keywords from the prompt — 'running tests
against a single target framework when a project targets multiple TFMs'
plus a concrete '<TargetFrameworks>net8.0;net9.0</TargetFrameworks>' example
and the '--framework <TFM>' invocation. Per-skill description is 947 chars,
well under the 1024 spec cap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace simple multi-TFM scenario with a harder MTP+SDK 9 variant that requires skill knowledge
The previous 'Run tests in a multi-TFM project targeting a specific
framework' scenario tested a trivial prompt (single --framework flag)
that the model already answers perfectly from training (baseline 4.7/5).
This matches pattern #8 (baseline already good, no headroom) in
InvestigatingResults.md, and explains why two prior attempts to fix
plugin-mode activation via description tweaks (PR #667, #688) failed:
the router rationally skips loading the skill for a trivial question.
This change replaces the scenario with a meaningfully harder one that
combines several signals the model often confuses:
- Multi-TFM project (TargetFrameworks plural)
- MTP runner (TestingPlatformDotnetTestSupport)
- .NET SDK 9 (-- separator required for MTP extension args)
- TRX report (MTP extension flag)
The correct answer requires knowing that --framework is a dotnet
test/MSBuild flag that always goes BEFORE --, while --report-trx is an
MTP extension flag that goes AFTER -- on SDK 9:
dotnet test --framework net9.0 -- --report-trx
The ordering assertion in eval.yaml enforces this on a single line, so
swapping --framework after the separator no longer satisfies the
scenario.
Also fix a documentation bug in SKILL.md surfaced by this work: the
'Common MTP flags > Built-in flags' table previously listed --framework
and --no-build under 'On SDK 8/9, pass after --', which is wrong. Those
are dotnet test/MSBuild flags consumed by dotnet test itself; they
always go before --. The table now lists only flags genuinely consumed
by MTP, and an Important note clarifies the rule.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add eval scenario for filter-syntax [trait] query language pattern
Add a run-tests scenario that exercises the xUnit v3 query filter language
with [traitName=traitValue] syntax (--filter-query), covering the uncovered
[CodePattern] [trait] at line 113 of filter-syntax SKILL.md.
New fixture: mtp-xunit-v3-query with multiple test classes and traits
(Smoke, Regression, Unit) to test combined class-pattern + trait filtering.
* Address review feedback for xUnit v3 query eval coverage
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Add rollForward: latestFeature to all global.json entries that pin SDK 9.0.200
so that any 9.0.xxx SDK satisfies the requirement. Without this, machines with
only 9.0.3xx installed fail all dotnet commands during evaluation, causing the
model to burn its entire timeout budget on failed bash retries.
* Fix eval source file resolution to use eval directory
The SetupWorkDir method resolved setup file 'source:' paths relative to the
skill directory (plugins/.../skills/<name>/), but eval.yaml files live under
tests/. The ../../../../ relative paths resolved correctly but were rejected
by the path traversal security check because the resolved path escaped the
skill directory boundary.
Fix by preferring the eval directory as the base for source resolution when
evalPath is available, falling back to skillPath for backward compatibility
with scenarios embedded in SKILL.md files.
Also update run-tests eval.yaml source paths from
'../../../../tests/dotnet-test/run-tests/fixtures/...' to 'fixtures/...'
(relative to the eval directory where the fixtures actually live).
This was causing all 11 source-based run-tests scenarios to get zero fixture
files, making the agent find an empty workspace and fail to evaluate.
* Address review: extract ResolveSourcePath helper, add warning, fix remaining eval.yaml paths
- Extract source-resolution and path-traversal check into internal
ResolveSourcePath helper for testability
- Emit a clear warning when file.Source is set but no base directory
is available (instead of silently continuing)
- Add unit tests covering: evalPath-based resolution, skillPath fallback,
null paths warning, traversal rejection, and evalPath preference
- Fix path traversal patterns in analyzing-dotnet-performance and
clr-activation-debugging eval.yaml files (same issue as run-tests)
Amaury Levé