mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Amaury Levé
ca21fd61c1
* run-tests: use prompt-shaped multi-TFM keywords for plugin activation The 'Run tests in a multi-TFM project targeting a specific framework' eval scenario kept failing plugin-mode activation after #667. The skill description had 'multi-TFM projects (--framework)' as a brief parenthetical, but the scenario prompt uses different language: 'This test project targets both net8.0 and net9.0. I only want to run the tests on net9.0 using dotnet test.' Per the InvestigatingResults.md guidance for 'skill not activated' failures, expand that clause to include keywords from the prompt — 'running tests against a single target framework when a project targets multiple TFMs' plus a concrete '<TargetFrameworks>net8.0;net9.0</TargetFrameworks>' example and the '--framework <TFM>' invocation. Per-skill description is 947 chars, well under the 1024 spec cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace simple multi-TFM scenario with a harder MTP+SDK 9 variant that requires skill knowledge The previous 'Run tests in a multi-TFM project targeting a specific framework' scenario tested a trivial prompt (single --framework flag) that the model already answers perfectly from training (baseline 4.7/5). This matches pattern #8 (baseline already good, no headroom) in InvestigatingResults.md, and explains why two prior attempts to fix plugin-mode activation via description tweaks (PR #667, #688) failed: the router rationally skips loading the skill for a trivial question. This change replaces the scenario with a meaningfully harder one that combines several signals the model often confuses: - Multi-TFM project (TargetFrameworks plural) - MTP runner (TestingPlatformDotnetTestSupport) - .NET SDK 9 (-- separator required for MTP extension args) - TRX report (MTP extension flag) The correct answer requires knowing that --framework is a dotnet test/MSBuild flag that always goes BEFORE --, while --report-trx is an MTP extension flag that goes AFTER -- on SDK 9: dotnet test --framework net9.0 -- --report-trx The ordering assertion in eval.yaml enforces this on a single line, so swapping --framework after the separator no longer satisfies the scenario. Also fix a documentation bug in SKILL.md surfaced by this work: the 'Common MTP flags > Built-in flags' table previously listed --framework and --no-build under 'On SDK 8/9, pass after --', which is wrong. Those are dotnet test/MSBuild flags consumed by dotnet test itself; they always go before --. The table now lists only flags genuinely consumed by MTP, and an Important note clarifies the rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>