mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
dev/jankrivanek/eval-dispatch-plugin-input
48 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Amaury Levé
|
680292e04c |
Make dotnet-test analysis skills and auditor agent polyglot (#707)
* Make dotnet-test analysis skills and auditor agent polyglot Five test analysis skills and the test-quality-auditor agent in plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/ TUnit cues, even though the underlying techniques (assertion diversity, anti-pattern detection, pseudo-mutation gap analysis, academic smell catalog, trait tagging) apply equally to other test frameworks. This change makes them polyglot across .NET, Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest). Changes ------- * New reference skill `test-analysis-extensions` with per-language extension files (dotnet, python, typescript, java, go, ruby, rust, swift, kotlin, powershell, cpp). Each documents test markers, assertion APIs, sleep / time / random APIs, skip annotations, setup/teardown, mystery-guest indicators, integration markers, and tag-support capability for the corresponding framework. * `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`, `test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter now lists all polyglot frameworks (keeping .NET trigger words for backward compat); bodies reference `test-analysis-extensions` and add per-language calibration (e.g., Go/Rust table-driven loops are idiomatic and not Conditional Test Logic; pytest bare `assert` is canonical; missing-await on async assertions is a new Critical smell). * `test-quality-auditor` agent now runs language detection, gates .NET-only pipeline steps (coverage-analysis, CRAP, detect-static- dependencies, testability migration, experimental dotnet skills), and recommends native tooling for non-.NET projects via an explicit Capability Matrix. * README updated to split sections by polyglot vs .NET-only. * New polyglot eval scenarios added (Python/pytest, TS/Jest, Java/JUnit) alongside existing C# scenarios in `test-anti-patterns`, `assertion-quality`, `test-smell-detection`. The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis, crap-score, detect-static-dependencies, run-tests, filter-syntax, platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are unchanged and continue to activate only for .NET work. Validated with skill-validator `check --plugin ./plugins/dotnet-test`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #707 review feedback * typescript.md: replace xpect.fail() recommendation (Jest has no such API) with hrow new Error(...) / explicit failing assertion. * test-tagging SKILL.md description: clarify that unittest has no canonical tag syntax (pytest markers are the auto-edit mechanism; unittest is report-only). Switch doctest reference to "decorators" rather than implying a Catch2-style [tags] mechanism exists. * test-tagging SKILL.md auto-edit list and table: use `* doctest::test_suite("tag")` decorator chain syntax for doctest in both the bullet list and the per-framework table, matching the example already shown. * cpp.md extension: spell out the Catch2 vs doctest tagging distinction in the capability matrix (Catch2 [tag], doctest * doctest::test_suite("tag") decorator). Validated with `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` (23 skills + 11 agents). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Tom Longhurst
|
36209c3b52 |
TUnit Fixes & Improvements (#677)
* Update target framework to net10.0 and upgrade TUnit package to version 1.45.8
* Fix and expand TUnit coverage in dotnet-test-frameworks reference
Cross-referenced against tunit.dev/llms.txt and the official attributes
comparison page. Corrects the test class marker (TUnit is convention-based,
not [ClassDataSource]), and fills in the previously-missing TUnit rows for
the assertion APIs, setup/teardown methods, exception-handling examples,
and integration test category marker.
* Refine TUnit assertion table accuracy
The Type row's TUnit cell now uses IsAssignableTo<T>() to match the
assignable-type semantics of the MSTest/xUnit/NUnit equivalents, with a
note pointing at IsTypeOf<T>() for exact-type checks. The Skip row now
documents that [Skip] also applies at class and assembly scope and that
Skip.Test("reason") handles dynamic in-test skipping.
Verified against the TUnit migration guides and the type-assertions docs.
* Spell out TUnit assertion alternatives as complete awaited expressions
The Boolean, Null, Exception, and Type rows in the assertions table
previously used a `await Assert.That(x).IsTrue()` / `.IsFalse()` shorthand
for the alternative form. A reader copy-pasting just the second fragment
would end up with an un-awaited assertion that silently passes — which
contradicts the note immediately below the table about always awaiting.
Each alternative is now a complete, independently copy-pasteable expression.
Addresses the inline review on dotnet/skills#677.
---------
Co-authored-by: Amaury Levé <amauryleve@microsoft.com>
|
||
|
Amaury Levé
|
b9db51de9d |
Add TUnit-focused eval scenarios for dotnet-test-frameworks (#699)
* Add TUnit-focused eval scenarios for dotnet-test-frameworks Adds 5 new evaluation scenarios and tightens 2 existing scenarios to exercise the expanded TUnit coverage in dotnet-test-frameworks/SKILL.md (see #677). New scenarios: - Convert cross-framework assertions to TUnit syntax (await / IsEqualTo / IsTrue / IsNull / IsAssignableTo / Contains / Throws<T>) - Diagnose silently-passing TUnit test with missing await (debugging mystery framed without naming the pitfall) - Refactor TUnit try/catch to native exception assertion (Throws<T>() / ThrowsExactly<T>() / WithMessage) - TUnit lifecycle hooks at test / class / assembly / session scope - TUnit skip mechanisms — attribute, assembly-wide [assembly: Skip], and dynamic Skip.Test(...) Updated scenarios: - Identify TUnit framework: rubric now distinguishes [ClassDataSource] as a fixture/data source rather than a class marker (TUnit classes are convention-based, like xUnit) - Identify integration tests: adds a TUnit Project D using live SqlConnection and requires recommending [Category("Integration")] for TUnit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Mirror TUnit eval scenarios into Vally spec Apply the same updated and new scenarios as in eval.yaml to the parallel eval.vally.yaml so both pipelines exercise the new TUnit-focused checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Amaury Levé <amauryleve@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ca21fd61c1 |
run-tests: prompt-shaped multi-TFM keywords for plugin activation (#688)
* run-tests: use prompt-shaped multi-TFM keywords for plugin activation The 'Run tests in a multi-TFM project targeting a specific framework' eval scenario kept failing plugin-mode activation after #667. The skill description had 'multi-TFM projects (--framework)' as a brief parenthetical, but the scenario prompt uses different language: 'This test project targets both net8.0 and net9.0. I only want to run the tests on net9.0 using dotnet test.' Per the InvestigatingResults.md guidance for 'skill not activated' failures, expand that clause to include keywords from the prompt — 'running tests against a single target framework when a project targets multiple TFMs' plus a concrete '<TargetFrameworks>net8.0;net9.0</TargetFrameworks>' example and the '--framework <TFM>' invocation. Per-skill description is 947 chars, well under the 1024 spec cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace simple multi-TFM scenario with a harder MTP+SDK 9 variant that requires skill knowledge The previous 'Run tests in a multi-TFM project targeting a specific framework' scenario tested a trivial prompt (single --framework flag) that the model already answers perfectly from training (baseline 4.7/5). This matches pattern #8 (baseline already good, no headroom) in InvestigatingResults.md, and explains why two prior attempts to fix plugin-mode activation via description tweaks (PR #667, #688) failed: the router rationally skips loading the skill for a trivial question. This change replaces the scenario with a meaningfully harder one that combines several signals the model often confuses: - Multi-TFM project (TargetFrameworks plural) - MTP runner (TestingPlatformDotnetTestSupport) - .NET SDK 9 (-- separator required for MTP extension args) - TRX report (MTP extension flag) The correct answer requires knowing that --framework is a dotnet test/MSBuild flag that always goes BEFORE --, while --report-trx is an MTP extension flag that goes AFTER -- on SDK 9: dotnet test --framework net9.0 -- --report-trx The ordering assertion in eval.yaml enforces this on a single line, so swapping --framework after the separator no longer satisfies the scenario. Also fix a documentation bug in SKILL.md surfaced by this work: the 'Common MTP flags > Built-in flags' table previously listed --framework and --no-build under 'On SDK 8/9, pass after --', which is wrong. Those are dotnet test/MSBuild flags consumed by dotnet test itself; they always go before --. The table now lists only flags genuinely consumed by MTP, and an Important note clarifies the rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
5810d492c9 |
test-anti-patterns / test-smell-detection: narrow-specialist re-pitch (Variant B) (#671)
Both skills suffer plugin-mode activation failures: in plugin runs the model bypasses the skill tool entirely for ~5/6 of test-anti-patterns audit prompts and ~2/3 of test-smell-detection prompts, answering directly from training. Variant B repositions the two skills as umbrella + narrow specialist: * test-anti-patterns becomes THE umbrella audit skill for all pragmatic test-quality reviews (anti-patterns and smell-style prompts alike). Its description explicitly absorbs smell-audit triggers. * test-smell-detection is repositioned as a narrow niche that only fires when the user explicitly asks for the testsmells.org / academic 19-smell catalog with citable smell names from the research literature. Eval prompts for test-smell-detection are rewritten to match the narrow niche (explicit testsmells.org / 19-smell catalog requests) so we test specialist activation rather than the umbrella scenarios. Variant B of a 3-PR experiment to fix activation. Companion PRs: * Variant A = activation-only rewrite, both skills keep overlap; * Variant C = full merge into test-anti-patterns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
362b3f25ec |
Fix plugin-mode skill activation for 8 dotnet-test scenarios (#667)
* migrate-vstest-to-mtp: fix plugin activation for OutputType=Exe scenario The `Set OutputType=Exe only for test projects in Directory.Build.props` scenario was failing plugin-mode skill activation (skillActivationPlugin. activated=false, detectedSkills=[]) in the latest scheduled eval. The model answered from its own knowledge and proposed the wrong solution (`Condition="'$(IsTestProject)' == 'true'"` in Directory.Build.props, which the skill explicitly warns against). Two fixes: 1. SKILL.md description: add an explicit `conditioning OutputType=Exe to test projects when centralizing MTP properties in Directory.Build.props` USE FOR clause so the description surfaces this specific MTP migration pitfall to the skill router. Per-skill description stays at <= 1024 chars (trimmed `configuration` -> `config` and `MSTest version upgrades (use migrate-mstest-* skills)` -> `MSTest version upgrades` to keep within the spec limit). 2. eval.vally.yaml: add the missing scenario (the eval.yaml scenario was added in PR #556 but vally was never synced — vally had 10 stimuli vs eval.yaml's 11). Brings vally and skill-validator coverage to parity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test-anti-patterns: strengthen description for severity-ranked audits The `Detect mixed severity anti-patterns in repository service tests` scenario was failing plugin-mode skill activation (skillActivationPlugin.activated=false, detectedSkills=[]) in the latest scheduled eval, even though the prompt explicitly says `Audit the test class ... for .NET test anti-patterns ... Give me a severity-ranked list (Critical / Warning / Info)`. The previous description only mentioned `USE FOR: audit test quality, review test code, find test anti-patterns, ...`, so the model could not distinguish this skill from sibling test-quality skills based on the `severity-ranked` and `Critical/Warning/Info` framing in the prompt. Surface the catalog shape (severity-ranked output, concrete code-level fixes) and add explicit `severity-ranked test audit`, `self-comparing assertions` and `broad exception types` triggers — these are the exact phrases used by failing scenarios. Description stays at 822 chars (<= 1024 limit). Aggregate dotnet-test description stays at 15,099 chars (<= 20,000 limit). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * run-tests: restructure description for higher-signal plugin activation Six run-tests scenarios were failing plugin-mode skill activation in the latest scheduled eval (skillActivationPlugin.activated=false, detectedSkills=[]) even though isolated activation worked for every scenario: - Run tests with trx reporting on MTP project (SDK 9) - Run tests in a multi-TFM project targeting a specific framework - Filter MSTest tests by category on VSTest - Filter NUnit tests by class name on VSTest - Filter TUnit tests by class using treenode-filter - Negative test: do not use MTP syntax for a VSTest project The previous description piled ~30 quoted trigger phrases (`run tests`, `run my tests`, `run these tests`, `execute tests`, `dotnet test`, `test filter`, `filter by category`, `filter by class`, `combine filters`, `run only specific tests`, `integration tests`, `unit tests`, `tests not running`, `hang timeout`, `blame-hang`, `blame-crash`, `crash dump`, `TRX report`, `TRX`, `test report`, `generate TRX`, `TUnit`, `treenode-filter`, `target framework`, `multi-TFM`, ...). In plugin mode this signal-poor keyword wall made the skill look generic enough that the router preferred to answer directly from model knowledge. Replace the keyword pile with five higher-signal scope clauses grouped by user-visible concern: 1. running, filtering, or troubleshooting `dotnet test` 2. VSTest vs Microsoft.Testing.Platform syntax (incl. `--` separator rules on SDK 8/9 vs 10+) 3. framework-specific filter syntax for MSTest / xUnit / NUnit / TUnit (--filter, --filter-class, --filter-trait, --filter-query, --treenode-filter) 4. TRX/reporting (--report-trx vs --logger trx), blame/hang/crash diagnostics, multi-TFM (--framework) 5. avoiding MTP/VSTest argument mixups These are the same concerns the previous description tried to cover, but framed as decisions the skill helps make (and that distinguish it from sibling test skills) rather than as a list of bare keywords. Per-skill description shrinks from 1,000 to 819 chars (still <= 1,024 limit). Aggregate dotnet-test description stays well under the 20,000-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test-anti-patterns: drop specific severity labels from description PR review (copilot-pull-request-reviewer on #667) caught that `Produces a severity-ranked (Critical / Warning / Info) catalog ...` in the new description contradicts the SKILL.md body, which actually teaches a 4-level Critical / High / Medium / Low taxonomy. The `Critical / Warning / Info` wording came from one eval scenario's prompt, but other prompts ask for different bucket schemes, and the skill itself does not enforce a fixed three-level mapping. Drop the specific labels from the description and keep just `severity-ranked` as the activation signal. The body remains unchanged (it teaches Critical / High / Medium / Low and the agent maps to the prompt's requested buckets at output time). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
b24599aa36 |
test-smell-detection: trim to detailed tier and add prompt-shaped triggers (#653)
Multi-model skill-validator eval (claude-haiku-4.5, claude-opus-4.6, gpt-5.3-codex, gpt-5.4-mini) showed two issues with this skill: 1. Quality regression on stronger models (+0.15 haiku, -0.14 opus, -0.17 codex, -0.29 gpt-5.4-mini): the skill added tokens and tool calls without lifting rubric quality on larger models. 2. Plugin-mode activation loss: the skill activated reliably in isolated mode but was almost never picked in plugin mode where it competed with siblings (code-testing-agent, etc.). Changes: - Trim BPE tokens 2,487 -> 1,472 (still detailed tier, well within the 800-2,500 sweet spot per SkillsBench guidance). Collapsed long smell-by-smell prose, `Why Test Smells Matter`, and pitfalls into a compact taxonomy table plus a short decision procedure / output checklist. Removed bulky examples. Kept taxonomy, calibration, negative guidance, and references intact. - Add prompt-shaped USE FOR triggers drawn from the eval prompts (`review test smells`, `review these tests and tell me if there are` `any problematic patterns`, `check my tests for test design` `problems or anti-patterns`, `give us an objective assessment of` `our integration tests`, plus explicit smell names). - Strengthen DO NOT USE FOR clause: `write new tests / scaffold tests` `from scratch / generate a complete MSTest test suite` now defers cleanly to code-testing-generator (covers the negative eval scenario). Validation: skill-validator check --plugin clean for this skill; build + 545/545 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
04348a46ef |
tests/dotnet-test/code-testing-agent: fix coverage XML lookup path (#657)
The grader expected coverage.cobertura.xml under a workspace-root TestResults/ directory, but `dotnet test --collect:"XPlat Code Coverage"` writes the report under <TestProject>/TestResults/<guid>/coverage.cobertura.xml. With the previous path, `find TestResults …` failed with no-such-directory before even searching, and the glob `TestResults/**/coverage.cobertura.xml` could not match the nested location either, so the assertion failed on otherwise-successful runs. Search recursively from cwd in both eval.vally.yaml (`find . …`) and eval.yaml (`**/TestResults/**/coverage.cobertura.xml`) so the assertion matches the actual output location of XPlat Code Coverage regardless of the test project layout the agent chose. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
3d59e44c7e |
Fix coverage-analysis activation for plateau diagnosis prompts (#647)
* Fix coverage-analysis activation for plateau diagnosis prompts coverage-analysis SKILL.md: - Trim verbose implementation details (provider detection, ReportGenerator) that consumed description budget without aiding skill activation - Add explicit USE FOR keywords: coverage stuck, coverage plateau, can't increase coverage, what's blocking coverage code-testing-agent SKILL.md: - Add 'diagnosing coverage plateaus or CRAP score computation (use coverage-analysis)' to DO NOT USE FOR boundary to prevent test-generation skill from intercepting diagnostic prompts * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Strengthen code-testing-agent activation; harden coverage-analysis isolated mode Address eval regressions reported on PR #647 (run 25813728646): 1. code-testing-agent: `Generate tests for ContosoUniversity ASP.NET Core MVC app` was NOT ACTIVATED in plugin mode (detectedSkills=[], skillEventCount=0, invokedAgents=[]). The model bypassed the skill system entirely. - SKILL.md description: restructure to use the proven `Use when user says ...` pattern with quoted trigger phrases (matching the run-tests skill that consistently activates), make the link to the code-testing-generator sub-agent explicit, and tighten DO NOT USE FOR clauses. - eval prompt (eval.yaml + eval.vally.yaml): make the request pipeline-shaped (`project-wide, multi-file test generation task`, `scaffold a new test project`) so the model recognizes it as multi-step work that benefits from the orchestrated pipeline. Explicitly request coverlet.collector + a Cobertura XML run so rubric criterion 1 (`high line coverage as reported by the Cobertura XML in TestResults/`) becomes achievable without overfitting. 2. code-testing-tester agent + code-testing-extensions/dotnet.md: open a scoped exception to the `skip coverage tools` rule. Default behavior stays the same, but when the user/harness explicitly asks for a Cobertura/XML coverage artifact, the agent may add coverlet.collector to the generated test csproj so the harness's coverage command produces output. The agent still does not run the coverage command itself. 3. coverage-analysis SKILL.md: add a `User-visible output is mandatory` guard at the top of the Workflow section. The latest eval showed isolated mode producing literally `(no output)` in 2 of 3 scenarios — the agent ran Compute-CrapScores.ps1 / Extract-MethodCoverage.ps1 / ReportGenerator in parallel, then the session ended without ever surfacing findings. The guard tells the agent to always return a partial summary instead of ending silent, and to deprioritize ReportGenerator HTML when budget is tight. (Plugin-mode quality is already strong: 4.3 / 4.3 / 5.0 — no regression risk there.) Aggregate dotnet-test plugin description size: 14,925 chars (limit 15,000). skill-validator check passes (22 skills, 11 agents, 1 plugin); markdownlint passes for all 4 modified files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore MSTest modernization exclusion in code-testing-agent description * Fix isolated-mode coverage-analysis: emit summary before optional ReportGenerator The previous workflow encouraged the agent to run `dotnet tool install` for ReportGenerator in parallel with the CRAP scoring scripts (Phase 2 "Steps 3 and 4 in parallel" + Phase 3 "Steps 5 and 6 in parallel"). In isolated mode that pattern reliably crashed the session with "Failed to persist session events: timeout while waiting for mutex to become available" right after the scripts returned valid data, so the agent never produced the user-facing summary. Restructure the workflow into 5 phases: - Phase 1 (Setup) - unchanged - Phase 2 (Test execution) - skip when Cobertura XML already exists - Phase 3 (Analysis) - run only the two PowerShell scripts, no RG - Phase 4 (User-facing summary) - MANDATORY, must be the next assistant response after Phase 3, before any RG work; also save coverage-analysis.md as a secondary follow-up - Phase 5 (ReportGenerator HTML/CSV) - strictly optional, post-summary, skipped by default for existing-Cobertura and plateau-diagnosis paths Also update references/output-format.md so the Reports section marks RG artifacts as "Not generated (optional - request HTML reports to enable)" when Phase 5 has not run, and update references/guidelines.md so the "show and open the markdown report" rule explicitly defers to the user-facing assistant response. Targets the isolated-mode regressions in PR #647 eval: - Project-wide coverage with existing Cobertura: 1.0/5 -> expected 3+ - Coverage plateau diagnosis: 1.0/5 -> expected 3+ - Run coverage from scratch: 2.3/5 -> expected steady or up Verified: skill-validator check --plugin ./plugins/dotnet-test passes; markdownlint-cli2 clean on all 3 modified files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #647 review comments 1. Prerequisites: distinguish the from-scratch path (needs NuGet for the coverage-provider package install + optional internet for ReportGenerator) from the existing-Cobertura path (needs neither). 2. Add Step 2c "Discover or accept existing Cobertura XML" so the existing-data path actually has $coberturaFiles populated before Phase 3, instead of relying on Phase 2's discovery (which it skips). Also clarify Step 2's destructive Remove-Item only manages the skill-owned coverage-analysis/ subdirectory. 3. references/output-format.md: replace the unconditional "Reports saved to: <coverageDir>/reports/" line with one that always points at <coverageDir>/ (markdown summary + raw Cobertura) and only mentions reports/ if Phase 5 ran. 4. Have Compute-CrapScores.ps1 emit OVERALL_LINE_COVERAGE and OVERALL_BRANCH_COVERAGE from the Cobertura root attributes, and update Phase 4 to read those values directly from the script's output. The Phase 4 mandatory-summary rule no longer requires a separate XML parse before composing the response. Verified: skill-validator check --plugin ./plugins/dotnet-test passes; markdownlint-cli2 clean on all modified files; Compute-CrapScores.ps1 smoke-tested on a synthetic Cobertura XML (emits OVERALL_LINE_COVERAGE:75, OVERALL_BRANCH_COVERAGE:50 alongside HOTSPOTS). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestion from @Evangelink * Address unresolved coverage-analysis and eval review comments * Refine follow-up review feedback from validation * Tighten coverage aggregation fallback notes and counters * Clarify pre-response save instruction wording --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f532cf1d94 |
Add coverage-touching and self-referential assertion heuristics (#635)
* Add coverage-touching and self-referential assertion heuristics
- test-anti-patterns: add 'Coverage touching' and 'Self-referential assertion'
as named Critical-severity anti-patterns in the detection catalog
- assertion-quality: add 'Tests with self-referential assertions' metric and
a calibration rule distinguishing tautological from valid round-trip assertions
- Add eval scenarios for both new heuristics in test-anti-patterns (coverage-
touching ProductCatalog facade, self-referential SerializerTests)
- Add eval scenario and fixture for self-referential assertions in
assertion-quality (ConfigTests with identity/round-trip patterns)
* Fix test-anti-patterns activation: rewrite description with USE FOR keyword list
The skill was not activating in any eval scenario because the description
used narrative prose instead of the explicit 'USE FOR:' / 'DO NOT USE FOR:'
keyword list pattern that well-activating sibling skills use (run-tests,
writing-mstest-tests). Restructured the description with trigger phrases
matching common user prompts.
* Fix eval activation, timeouts, and add missing vally scenarios
- test-anti-patterns/eval.vally.yaml: increase timeout 2m→5m, add
coverage-touching and self-referential scenarios with constraints
- assertion-quality/eval.vally.yaml: add self-referential scenario
with environment files, broaden assertion pattern
- test-anti-patterns/eval.yaml: add setup.files to all 6 scenarios
to create workspace files that trigger skill discovery/activation
- assertion-quality/eval.yaml: increase timeouts 120→300 for
self-referential and decline-request scenarios, broaden
self-referential assertion pattern
- Create fixture files for mixed-severity, well-written,
coverage-touching, and self-referential test-anti-patterns scenarios
* Address review comments: fix test names and use source references
- Rename ThrowsArgumentException to ThrowsArgumentOutOfRangeException in
PaymentProcessorTests.cs fixture and inlined code in eval.yaml and
eval.vally.yaml to match the actual exception type being asserted.
- Replace inline csproj content blocks with source references to existing
fixture files for UserRepo, Payments, and CatalogService scenarios in
eval.yaml to avoid drift between fixtures and eval setup.
* Address review comments: pin MSTest versions, use source refs, fix naming
- Pin MSTest to 3.8.2 in all fixture .csproj files, add standard
properties (ImplicitUsings, Nullable, IsPackable)
- Replace inline csproj content blocks with source references to
fixture files to avoid content drift
- Create missing .csproj fixtures for flaky-coupled, duplicate-magic,
and assertion-quality self-referential scenarios
- Fix test method names: ThrowsArgumentException ->
ThrowsArgumentOutOfRangeException to match the actual assertion
* Fix activation, eval timeouts, and per-skill description limits
Address remaining issues from PR #635 eval scoreboard
(run 25677669933) and skill-validator failures:
1. test-anti-patterns activation (was NOT ACTIVATED in BOTH plugin and
isolated mode for ALL 6 scenarios — model answered everything from
inline code without ever needing the skill):
- eval.yaml + eval.vally.yaml: drop the giant inline `csharp` blocks
from every prompt. Each prompt now names the workspace files
explicitly (`Audit `Foo.Tests/FooTests.cs` for .NET test
anti-patterns ...`) and asks for the skill's signature deliverable
(severity-ranked list with concrete code-level fixes). The model
must read the workspace, which gives it a chance to consult the
skill — and the skill's structured-findings format is the actual
value-add over inline analysis.
- All 6 scenarios consistently get reject_tools (read-only review)
and timeout 180s.
- eval.vally.yaml: also added missing environment.files for each
scenario (was prompt-only before, no fixture on disk).
2. assertion-quality `Flag assertion-free tests` plugin-mode activation
(was bouncing to other skills because the prompt was generic `check
whether these tests verify anything useful`):
- eval.yaml + eval.vally.yaml: rewrite the prompt to name the file
(`SmokeTests/ApiEndpointSmokeTests.cs`) and explicitly request
`assertion quality and depth`, `assertion-free tests`, and
`trivial assertions like Assert.IsNotNull` — direct trigger
phrases for assertion-quality.
- assertion-quality SKILL.md: extend description with explicit USE
FOR keywords for `assertion-free tests`, `self-referential`, and
`tautological` so plugin-mode activation isn't won by sibling
skills like test-anti-patterns.
3. test-anti-patterns description size — was 1,301 chars (over the
1,024 per-skill limit) AND pushed dotnet-test plugin aggregate to
15,312 (over the 15,000 limit), failing skill-check on PR #635.
Trimmed to 779 chars; aggregate now at 14,983.
4. Vally job-level timeout (60m → 180m) was already raised on main
in commit
|
||
|
Amaury Levé
|
de0a97c551 |
Fix migrate-vstest-to-mtp activation for Handle exit code 8 scenario (#648)
The `Handle exit code 8 when migrating from VSTest to MTP` scenario was
not activating the migrate-vstest-to-mtp skill in plugin mode in the
latest scheduled eval (skillActivationPlugin.activated=false,
detectedSkills=[], skillEventCount=0, vally row showed `--`), even
though all 10 other scenarios in this eval activated cleanly.
Three layered fixes:
1. Prompt (eval.yaml + eval.vally.yaml): switch from past-tense
`We migrated`/`How do I fix this behavioral difference` to
present-tense migration framing — `We're migrating ... After
enabling MTP ... How do I handle this MTP behavioral difference?`.
The previous past-tense framing read as a generic post-migration CI
troubleshooting question, which the model felt confident answering
without consulting any skill.
2. SKILL.md description: add an explicit `USE FOR:` clause for MTP
behavioral differences (exit code 8, zero tests discovered,
--ignore-exit-code, TESTINGPLATFORM_EXITCODE_IGNORE), matching the
pattern that fixed coverage-analysis activation in
|
||
|
Amaury Levé
|
0f31228b58 |
Fix eval timeouts and skill activation for dotnet-test evals (#641)
* Fix evaluation timeouts and improve prompts for dotnet-test skills
- writing-mstest-tests: increase global vally timeout from 3m to 5m to match
the 300s eval.yaml timeout for the complex 'Modernize legacy test patterns' stimulus
- code-testing-agent: increase timeout from 40m/2400s to 60m/3600s
- mtp-hot-reload: increase timeout from 6m to 10m and remove overly strict
expect_tools constraint on launchSettings stimulus
- migrate-mstest-v1v2-to-v3: rephrase prompt for assertion overload errors to
mention CS1501 and cover AreNotEqual/AreSame alongside AreEqual
* Increase eval timeouts for mtp-hot-reload and migrate-mstest-v1v2-to-v3 scenarios
* Strengthen activation prompts and add missing vally scenarios
- migrate-mstest-v1v2-to-v3: Add explicit 'MSTest v2 to v3' migration
language in .testsettings and DataRow prompts (both eval.yaml and
eval.vally.yaml)
- writing-mstest-tests: Add 'MSTest' keyword and skill-specific
terminology to all prompts that lacked activation signal
- writing-mstest-tests: Add 5 missing vally scenarios (string
assertions, comparison assertions, collection/null/reference
assertions, conditional execution, parallelization)
- Add service-registry fixture files for the collection assertions
scenario
* Improve skill activation via SKILL.md descriptions and prompt alignment
Three-pronged fix for systematic NOT ACTIVATED failures:
1. SKILL.md descriptions: Restructure writing-mstest-tests, migrate-
mstest-v1v2-to-v3, and code-testing-agent descriptions to use the
proven 'Use when user says' pattern with quoted trigger phrases
(matching the working run-tests pattern)
2. Vally prompts: Each prompt now contains at least one exact quoted
trigger phrase from its SKILL.md description to ensure strong
description-to-prompt matching
3. Specific fixes:
- writing-mstest-tests: All 15 prompts now include MSTest-specific
trigger phrases (Assert.ThrowsExactly, sealed test class, DataRow,
DynamicData, Assert.HasCount, Assert.IsInstanceOfType, etc.)
- migrate-mstest-v1v2-to-v3: Add '.NET 5 dropped' and 'framework
compatibility issues' to trigger phrases; strengthen dropped-TFM
prompt
- code-testing-agent: Add 'generate comprehensive unit tests' and
'achieve high code coverage' to trigger phrases
* Revert SKILL.md description changes that caused activation regression
The description restructuring in
|
||
|
Amaury Levé
|
f8cae94781 |
Add test-tagging evals for uncovered skill aspects (#639)
* Add test-tagging evals for uncovered skill aspects Add 3 new scenarios (6-8) to test-tagging eval.yaml: - Scenario 6: Partially-tagged MSTest suite — verifies existing traits are preserved without duplication, only taxonomy values are used, and framework detection works correctly. - Scenario 7: Misleading method names (NUnit) — verifies the agent reads test bodies instead of guessing from method names, uses NUnit [Category] syntax, and classifies security/concurrency/boundary tests accurately. - Scenario 8: Build verification — verifies the project still compiles after tagging changes and only valid taxonomy values are applied. New fixtures: - fixtures/mstest-partially-tagged: OrderService suite with 3 pre-tagged and 12 untagged tests. - fixtures/nunit-misleading-names: TaskManager suite where method names deliberately contradict test body behavior. * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add missing vally stimuli for test-tagging scenarios 6-8 --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
4c40b873f2 |
Add detect-static-dependencies evals for report structure, obj exclusion, and lambda statics (#640)
Cover previously uncovered skill aspects: - [Validation] All .cs files in scope were scanned (check count) - [Validation] Report includes category totals, top patterns, and affected files - [Validation] obj/ and bin/ directories were excluded - [Pitfall] Scanning obj/ or generated code - [Pitfall] Missing statics inside lambdas/LINQ - [WorkflowStep] Step 4: Present the report Three new scenarios with supporting fixtures: - Structured report with file count, categories, and top patterns - Exclude obj/bin directories (with Generated.cs decoy in obj/) - Detect statics inside lambdas and LINQ expressions |
||
|
Amaury Levé
|
e6cf308357 |
Add eval scenario for filter-syntax [trait] query language pattern (#638)
* Add eval scenario for filter-syntax [trait] query language pattern Add a run-tests scenario that exercises the xUnit v3 query filter language with [traitName=traitValue] syntax (--filter-query), covering the uncovered [CodePattern] [trait] at line 113 of filter-syntax SKILL.md. New fixture: mtp-xunit-v3-query with multiple test classes and traits (Smoke, Regression, Unit) to test combined class-pattern + trait filtering. * Address review feedback for xUnit v3 query eval coverage --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d575b1faa8 |
Add eval scenarios for writing-mstest-tests skill coverage (#593)
* Add eval scenarios for writing-mstest-tests skill coverage * Fix invalid YAML and remove build-validation claim from rubric * Fix timeouts and low quality * Improve further * Increase timeout for string assertions scenario from 120s to 180s Agent-Logs-Url: https://github.com/dotnet/skills/sessions/9943a60f-070b-4bea-9912-c994d1bed428 Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
cfc293f8fe |
Fix skill activation for Assert.AreEqual overload eval scenario (#589)
The 'Fix Assert.AreEqual object overload errors after v3 upgrade' evaluation scenario was failing because: 1. The prompt said 'latest major version' without mentioning 'v3' or 'MSTest v3', missing the skill's activation keywords. 2. The project file showed MSTest 3.8.0 (post-upgrade state) with no v2 context, while the SKILL.md 'When Not to Use' said 'Project already uses MSTest v3'. 3. The prompt's 'fix compilation issues' phrasing competed with writing-mstest-tests skill activation. Changes: - Updated prompt to explicitly mention upgrading from v2 to v3 and reference MSTest v3 compatibility issues - Added XML comment in csproj showing the pre-upgrade v2 package versions - Clarified SKILL.md 'When Not to Use' to distinguish fully-migrated projects from projects with active migration errors |
||
|
Amaury Levé
|
9f531203ec |
Promote 4 experimental test skills to dotnet-test plugin (#634)
Move assertion-quality, test-gap-analysis, test-smell-detection, and test-tagging from dotnet-experimental to dotnet-test. These skills have demonstrated sufficient eval coverage, unique value beyond baseline model capabilities, and are already integrated into the test-quality-auditor agent pipeline. Changes: - Move skills and tests, dropping the exp- prefix - Update all cross-references in skill descriptions and routing tables - Update test-quality-auditor agent and eval scenarios - Update CODEOWNERS for the moved paths - Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks) - Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability Skills held back for further iteration: - exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios - exp-test-maintainability: quality gains offset by token overhead |
||
|
Amaury Levé
|
97b23ae750 |
Fix timeout for Modernize legacy test patterns evaluation (#590)
The scenario was timing out on CI because: - The 180s timeout left insufficient headroom for CI machines (local runs take 120-135s, CI machines are slower due to load, network NuGet restores, and parallel execution) Changes: - Add 'dotnet restore' setup command to pre-warm NuGet cache before the agent starts (same pattern used by migrate-mstest-v3-to-v4) - Increase timeout from 180s to 300s to accommodate CI variability |
||
|
Aditya Mandaleeka
|
9433a31e67 | Initial port of evals to Vally (#615) | ||
|
Amaury Levé
|
1b105d1940 |
Fix dotnet-test-frameworks skill activation for evaluations (#580)
* Fix dotnet-test-frameworks skill activation for evaluations The skill had user-invocable: false and a description saying 'DO NOT USE directly', which prevented the model from activating it during evaluation. All 6 eval scenarios default to expect_activation: true, causing SkillNotActivated failures on CI. - Remove user-invocable: false from both dotnet-test and experimental copies - Rewrite description with activation keywords matching eval prompts (assertion mapping, framework identification, skip annotations, exception assertions, lifecycle conversion, integration test markers) - Preserve note that other analysis skills also load this as reference data * Restore user-invocable: false, fix cross-refs, add expect_activation: false Address review comment: remove exp-* skill references from dotnet-test- frameworks description since those skills reference exp-dotnet-test- frameworks instead. Optimize eval results: this reference data skill's content is knowledge the model already has (baseline scores 5.0/5 on most scenarios). Direct activation adds token/time overhead without quality improvement: - Restore user-invocable: false (correct for reference data skill) - Shorten description to not trigger activation - Add expect_activation: false to all 6 scenarios The skill's value is realized when loaded by consumer skills (e.g. test-anti-patterns), not via direct user queries. |
||
|
Amaury Levé
|
dee829b89c |
Fix eval timeout for MSTest.Sdk scenario by pre-warming NuGet cache (#548)
The 'Migrate MSTest.Sdk v3 project using ManagedType and TestTimeout' scenario times out on CI because MSTest.Sdk/4.1.0 resolution takes 40+ seconds on cold NuGet cache. Each agent session may call dotnet build multiple times, easily exceeding the 240s timeout. Add a 'dotnet restore' setup command that runs before the agent starts (outside the timeout window), pre-warming the NuGet cache and obj/ directory so subsequent builds complete in ~2s. |
||
|
Amaury Levé
|
cb7c10bd32 |
fix mstest sdk eval timeout (#558)
* Fix timeout in MSTest.Sdk v3-to-v4 eval scenario 9 The fixture csproj had MSTest.Sdk/3.8.0 while the prompt claimed the user already upgraded to 4.1.0. This mismatch caused the model to edit the csproj and run dotnet build, triggering slow MSTest SDK NuGet downloads that exceeded the 240s timeout on CI. - Update fixture csproj to MSTest.Sdk/4.1.0 to match the prompt - Add specific CS error codes to the prompt so the model provides direct fixes instead of building - Update rubric item to reflect the project already being on 4.1.0 * Fix CI timeout in migrate-mstest-v3-to-v4 eval scenario Reduce scope of the 'Fix multiple v4 breaking changes' scenario: - Remove 3 lower-signal breaking changes from the fixture (AreEqual params, IsInstanceOfType out param, Properties.Contains) to cut agent work - Add max_turns: 5 to cap agent iterations - Lower timeout from 240 to 180 seconds * Strengthen net6.0 scenario prompt for reliable skill activation |
||
|
Copilot
|
c625f09db5 |
Recommend MSBuildProjectName for OutputType conditions in MTP migration skill (#556)
* Initial plan * Improve MTP migration skill guidance for OutputType conditions Agent-Logs-Url: https://github.com/dotnet/skills/sessions/ae2f9ae1-c8c3-4e50-95ab-ccdc08bc388b Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> * Add eval scenario for OutputType conditioning in Directory.Build.props Agent-Logs-Url: https://github.com/dotnet/skills/sessions/734b3736-45a4-4f5d-b83c-3e6ee72fcfae Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> Co-authored-by: Amaury Levé <amauryleve@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7841934a75 |
Add migrate-mstest-v3-to-v4 eval scenarios for uncovered patterns (#564)
* Add migrate-mstest-v3-to-v4 eval scenarios for uncovered patterns
Add 5 new scenarios (Goals 10-14) covering:
- Verified package update with build/test validation
- Sealed custom TestMethodAttribute with Timeout migration
- TestMethodAttribute/TestMethod display name constructor syntax
- Assert.IsInstanceOfType out parameter breaking change
- TreatDiscoveryWarningsAsErrors and behavioral changes review
Add 4 new fixture directories:
- v3-verified-migration: buildable project for end-to-end validation
- v3-sealed-timeout: sealed attr + TestTimeout.Infinite patterns
- v3-testmethod-displayname: [TestMethod/Attribute("name")] patterns
- v3-instanceoftype: Assert.IsInstanceOfType<T> out param pattern
Covers uncovered validation items (lines 449-451, 458) and code
patterns: [TestMethod], [Timeout], sealed, [TestMethodAttribute],
Assert.IsInstanceOfType.
* Improve skill description for activation and bump scenario timeouts
Rewrite migrate-mstest-v3-to-v4 frontmatter description to improve SDK
semantic matching:
- Lead with symptom (fix build errors) instead of action (migrate)
- Add error codes CS0507, CS0103, CS1061, CS1615 as trigger signals
- Add symptom phrase: tests don't compile after upgrading MSTest
- Expand breaking change keywords: sealed, DisplayName syntax,
Assert.IsInstanceOfType out parameter, TestContext.TestName
- Keep under 1024 char limit
Bump timeout 180s -> 240s for scenarios 11 (sealed attr) and 13
(IsInstanceOfType) that hit timeout during evaluation.
* Address PR review comments
- Remove unused 'using System' from OrderServiceTests.cs and TypeCheckTests.cs
- Renumber duplicate Goal 10 to Goal 15 (non-activation scenario)
- Tighten file_contains assertion to check MSTest.TestFramework and
MSTest.TestAdapter individually instead of broad '4.' match
- Tighten Timeout assertion to require int.MaxValue instead of matching
pre-migration symbol names
- Fix error code CS0507 -> CS0115 in scenario 11 prompt to match the
actual MSTest v4 error for missing Execute override
|
||
|
Amaury Levé
|
96489b425e | eval(dotnet-test): add evaluations for dotnet-test-frameworks skill (#563) | ||
|
Amaury Levé
|
7740b2b53c |
Improve migrate-mstest-v1v2-to-v3 evals to differentiate from baseline (#547)
Five scenarios were performing at or below stock Copilot because prompts gave away answers and rubric items tested knowledge the baseline LLM can deduce from the code alone. Changes per scenario: - Fix DataRow type mismatch: added double-to-float mismatch, vaguer prompt, rubric tests both mismatch types and latest-3.x knowledge - Migrate .testsettings to .runsettings: added code coverage data collector to fixture, rubric tests correct TestTimeout mapping (<MSTest><TestTimeout> not <TestSessionTimeout>) and collector mapping - Migrate MSTest v2 NuGet to v3: vaguer prompt (forces model to read csproj), rubric tests metapackage/MSTest.Sdk recommendation and Microsoft.NET.Test.Sdk dependency awareness - Fix Assert.AreEqual overload errors: removed exact error code from prompt, added mixed-type and interface-reference cases, rubric tests correct generic type parameter selection - Handle dropped TFM: changed to multi-target (net5.0;net462; netcoreapp3.1), rubric tests that model knows which TFMs are still supported and doesn't upgrade them indiscriminately |
||
|
Copilot
|
c14d90d28e |
Align MSTest v2→v3 eval assertions/rubric with fixture semantics (#550)
* Initial plan * Align MSTest v2->v3 eval rubric with fixture assertions Agent-Logs-Url: https://github.com/dotnet/skills/sessions/055e6526-4b7d-4457-9004-2c8c99969cd1 Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> |
||
|
Amaury Levé
|
17df9a10eb |
Fix code-testing-agent eval timeout by removing unused NuGet packages (#551)
The ContosoUniversity test fixture had 17 unused NuGet packages (bootstrap, jQuery, Modernizr, Antlr4, WebGrease, Microsoft.Build.Tasks.Core, etc.) that bloated NuGet restore and build time. The multi-agent test generation pipeline performs multiple build/test cycles, causing cumulative delays that exceeded the 2400s scenario timeout on CI. Changes: - Remove 17 unused PackageReference entries from ContosoUniversity.csproj - Remove stale ItemGroup entries (Global.asax, Scripts, compile removes) - Remove NU1701 warning suppression (no longer needed) - Increase command_timeout from 180s to 300s for dotnet test assertion (accounts for first-time restore + build + test + coverage collection) |
||
|
Amaury Levé
|
7a2e91ce94 |
Fix mtp-hot-reload eval timeouts by removing expect_tools: bash (#546)
The 5 failing scenarios were timing out because expect_tools: [bash] forced the agent to execute terminal commands. The hot reload workflow involves 'dotnet run' which starts a test host that blocks indefinitely waiting for file changes, causing the scenario to exhaust its timeout. These scenarios test whether the model provides correct guidance (package name, env var, commands), not whether it actually executes them. The output_matches assertions already verify the response content. Removed expect_tools from: - Suggest hot reload for failing test in MTP project (SDK 9) - Suggest hot reload for failing test in MTP project (SDK 10) - Enable hot reload when package already installed - Suggest launchSettings.json configuration for hot reload - Negative: VSTest project cannot use MTP hot reload Also reduced timeouts since scenarios no longer need to wait for command execution. |
||
|
Amaury Levé
|
400661df09 |
run-tests: add critical rules, detection guidance, and troubleshooting (#514)
* run-tests: add critical rules, detection guidance, and troubleshooting - Add 'Critical Rules' table to prevent cross-platform VSTest/MTP mistakes - Expand Step 1 detection with explicit file-by-file lookup table - Add 'dotnet --version' as first detection action - Add Troubleshooting section with 7 common error patterns - Expand Common Pitfalls from 3 to 7 entries - Strengthen negative guidance for SDK version-specific syntax * Improve description * Improve evaluation * Add expected and rejected tools * Change * Improve do not use sections * Improve prompt * Increase eval timeouts for scenarios hitting the limit - run-tests: trx reporting MTP SDK 9 (360s -> 480s) - run-tests: blame-hang MTP SDK 10 (300s -> 420s) - run-tests: combine filter criteria VSTest (180s -> 300s) - mtp-hot-reload: hot reload SDK 9 (360s -> 480s) - mtp-hot-reload: hot reload filter (180s -> 300s) - code-testing-agent: ContosoUniversity (1800s -> 2400s) |
||
|
Amaury Levé
|
f46275e64e |
Add evaluation tests for test-migration and test-quality-auditor agents (#543)
test-migration eval (4 scenarios): - Auto-detect MSTest v2 on VSTest and recommend migration path - Detect xUnit v2 and route to xUnit v3 migration - Execute targeted MSTest v2 to v3 migration - Non-activation: decline test writing request test-quality-auditor eval (4 scenarios): - Comprehensive audit of weak test suite (multi-skill pipeline) - Targeted anti-pattern review - Assertion quality analysis - Non-activation: decline test generation request Includes fixture projects with intentional quality issues: - MSTest v2 project on VSTest for migration detection - xUnit v2 project for migration detection - ShoppingCart test suite with anti-patterns (assertion-free, tautological, catch-and-swallow, Thread.Sleep, shallow assertions, missing coverage) |
||
|
Amaury Levé
|
d7b80a806b |
Fix SDK 9.0.200 global.json pinning causing eval timeouts (#542)
Add rollForward: latestFeature to all global.json entries that pin SDK 9.0.200 so that any 9.0.xxx SDK satisfies the requirement. Without this, machines with only 9.0.3xx installed fail all dotnet commands during evaluation, causing the model to burn its entire timeout budget on failed bash retries. |
||
|
Amaury Levé
|
c18d4260de |
Fix timeout in MSTest.Sdk v3-to-v4 eval scenario 9 (#544)
The fixture csproj had MSTest.Sdk/3.8.0 while the prompt claimed the user already upgraded to 4.1.0. This mismatch caused the model to edit the csproj and run dotnet build, triggering slow MSTest SDK NuGet downloads that exceeded the 240s timeout on CI. - Update fixture csproj to MSTest.Sdk/4.1.0 to match the prompt - Add specific CS error codes to the prompt so the model provides direct fixes instead of building - Update rubric item to reflect the project already being on 4.1.0 |
||
|
Amaury Levé
|
7e6954f9a9 |
Add testability migration skills and agent for dotnet-test (#518)
* Add testability migration skills and agent for dotnet-test Introduces three skills and one orchestrator agent to help developers incrementally replace hard-to-test static dependencies with injectable abstractions: - detect-static-dependencies: scans C# code for DateTime.Now, File.*, Environment.*, HttpClient, Console.*, Process.* and ranks by frequency - generate-testability-wrappers: generates wrapper interfaces or guides adoption of built-in abstractions (TimeProvider, IHttpClientFactory, System.IO.Abstractions) - migrate-static-to-wrapper: mechanical codemod-style bulk replacement of static call sites with wrapper calls, scoped per project/namespace - testability-migration agent: orchestrates the Detect-Generate-Migrate pipeline end-to-end Includes eval.yaml tests and fixture projects for all three skills and the agent. * Fix activation, token overhead, and eval timeouts - Expand skill descriptions with more trigger phrases for migrate-static-to-wrapper and generate-testability-wrappers to fix NOT ACTIVATED issues - Collapse verbose replacement mapping tables and constructor examples to reduce token overhead (~30% fewer tokens) - Increase generate-testability-wrappers eval timeouts from 120s to 240s to prevent timeout-impacted scoring - Soften detect-static-dependencies scenario 2 rubric to reduce run-to-run variance in plugin mode - Increase migrate-static-to-wrapper non-activation timeout |
||
|
Cathy Sullivan
|
42e29303b4 |
Add coverage-analysis skill to dotnet-test plugin (#499)
* Add coverage-analysis skill to dotnet-test plugin Adds an automated, project-wide code coverage and CRAP score analysis skill for .NET projects. Complements the existing crap-score skill (which is manual/targeted for single methods) with an end-to-end pipeline: - Auto-detects solution structure and test projects - Detects coverage provider (Microsoft.Testing.Extensions.CodeCoverage or Coverlet) per test project and selects the correct dotnet test arguments - Runs coverage collection, generates reports via ReportGenerator - Calculates CRAP scores per method via bundled PowerShell scripts - Surfaces risk hotspots ranked by danger (complex + uncovered code) - Produces actionable recommendations with projected score improvements Boundary with crap-score: coverage-analysis is project-wide and automated; crap-score is targeted and LLM-directed. The frontmatter descriptions cross-reference each other so the skill router disambiguates correctly. Includes eval.yaml with 3 scenarios (existing data analysis, from-scratch coverage collection, coverage plateau diagnosis) and shared buildable fixtures. Fixes #457 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR review: coverage output path and scoped tool install - Add --coverage-output $rawDir to all MS CodeCoverage dotnet test commands (unified, .NET 9, .NET 10+, and mixed-project modes) so Cobertura XML lands in the expected directory for downstream search - Replace dotnet tool install --global with --tool-path scoped to the coverage output directory to avoid mutating the user environment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Cobertura XML fixture line numbers to match StudentService.cs FindById starts at line 101 (was 88), Search starts at line 109 (was 96). Updated both partial-coverage and plateau fixtures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wrap Get-ChildItem pipelines in @() for single-result array safety PowerShell returns a scalar FileInfo (not array) when a pipeline yields exactly one result, making .Count return null. Wrapping in @() ensures the result is always an array so .Count works reliably for the common single-test-project case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove CODEOWNERS change - not needed for this PR The existing /plugins/dotnet-test/ and /tests/dotnet-test/ wildcards already route reviews to @dotnet/dotnet-testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: .csproj fallback, cross-platform paths, .sln header - Add .csproj fallback discovery in Step 1 when no .sln found (emit ENTRY_TYPE:Project) to match documented behavior - Fix ProjectReference backslash to forward slash in test fixture .csproj - Remove leading blank line from ContosoUniversity.sln fixture Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add plateau/stuck trigger language to frontmatter description The skill was NOT ACTIVATED for the 'Coverage plateau diagnosis' eval scenario because the frontmatter description (used by the skill router) lacked plateau-related keywords. The prompt 'coverage is stuck' and 'what's blocking me' didn't match the existing description text. Add 'stuck', 'plateaued', and 'blocking improvement' to the frontmatter Use-when sentence so the skill router activates for plateau diagnosis prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make CRAP scores mandatory, add existing-data shortcut path The plateau eval scenario scored 1/5 on CRAP scores across all modes because the LLM shortcut the workflow when existing Cobertura XML was provided — it read the XML directly and gave a quick diagnostic answer without computing CRAP scores or following the output template. - Add existing-data guidance at top of Workflow section: skip test execution but still run CRAP score computation (Steps 5-6) - Add mandatory CRAP rule to guidelines.md: Risk Hotspots table is required in every output regardless of analysis mode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Youssef Victor
|
ae9e2b4302 |
Add SKILL for xunit.v3 migration (#484)
* Add SKILL for xunit.v3 migration * eval.yaml, to be reviewed * Add run_command_and_assert to eval.yaml for dotnet test validation Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8e1539d5-327e-4216-9a01-5aa062e53608 Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix eval * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix eval.yml * Update eval * Update skill * Increase timeout * Update eval.yaml * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/dotnet-test/migrate-xunit-to-xunit-v3/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
e4670b33a1 |
[PoC] Code testing agent + tests PoC (#433)
* Add code testing agent agents + tests PoC * Remove AssertionEvaluator.cs change (moved to dev/jankrivanek/agents-evals) |
||
|
Amaury Levé
|
51d29a086f | Improve test skills (#428) | ||
|
Amaury Levé
|
17fe01321d | Promote crap-score and test-anti-patterns from experimental to dotnet-test (#434) | ||
|
Amaury Levé
|
e354d43534 |
Improve test skills (#423)
* Increase timeout * Fix activation issues * Improve further |
||
|
Amaury Levé
|
0846e6015d | Improve MSTest migration skills (#387) | ||
|
Amaury Levé
|
ae0b6a1d86 | Add mtp-hot-reload skill for iterative test fixing with MTP hot reload (#389) | ||
|
Amaury Levé
|
c55e4f222b |
Add MSTest migration skills (v1/v2→v3, v3→v4) (#377)
* Add MSTest migration skills (v1/v2→v3, v3→v4) New dotnet-test plugin with two step-by-step migration skills: - migrate-mstest-v1v2-to-v3: covers assembly ref (v1) and NuGet (v2) projects, assertion overload changes, DataRow constructors, .testsettings replacement, timeout unification, and MSTest.Sdk adoption. - migrate-mstest-v3-to-v4: covers Execute→ExecuteAsync, CallerInfo constructors, ClassCleanupBehavior removal, TestContext.Properties generic change, Assert API changes, ExpectedExceptionAttribute removal, dropped <net8.0 TFMs, and behavioral changes (DisableAppDomain, TestCase.Id, TreatDiscoveryWarningsAsErrors). Includes eval.yaml test scenarios and fixtures for both skills, plugin.json, and CODEOWNERS entries. * Fixes * Improve results * Improve * Improve description * Copy fix for path * Bump timeout |
||
|
Amaury Levé
|
92d830ff48 |
Add writing-mstest-tests skill with eval scenarios (#378)
* Add writing-mstest-tests skill with eval scenarios Add a new skill for writing modern MSTest 3.x/4.x unit tests covering: - Test class structure (sealed, AAA pattern, naming conventions) - Modern assertion APIs (Assert.Throws, IsInstanceOfType, collection asserts) - Data-driven tests (DataRow, DynamicData with ValueTuples) - Async tests with cancellation and timeout patterns - Test lifecycle (constructor init, TestContext injection) - Anti-pattern detection and modernization Includes 10 eval scenarios across 4 goal categories with fixture files for both modern MSTest and legacy anti-pattern test projects. MSTest.Sdk version is recommended via global.json msbuild-sdks instead of inline in the project file, ensuring all test projects get bumped together. * Fixes * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Evangelink * Apply suggestion from @Evangelink --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
2a8388dd71 |
Add VSTest to MTP migration skill (#373)
* Add VSTest to MTP migration skill New migrate-vstest-to-mtp skill in the dotnet-test plugin covering: - Framework-specific MTP runner enablement (MSTest, NUnit, xUnit.net) - Directory.Build.props centralization of OutputType, runner properties - MSTest.Sdk UseVSTest removal - NUnit3TestAdapter 5.0.0+ upgrade requirement - xunit v2 to xunit.v3 upgrade requirement - dotnet test CLI argument translation (--logger to --report-trx, --collect to --coverage, --blame-crash to --crashdump, --blame-hang to --hangdump) - xUnit.net --filter syntax migration to --filter-class/--filter-trait/ --filter-query - global.json native MTP mode (.NET 10+) vs TestingPlatformDotnetTestSupport VSTest-bridge mode (.NET 9 and earlier) - Azure DevOps VSTest@3 to DotNetCoreCLI@2 pipeline migration - MTP extension NuGet packages (TrxReport, CrashDump, HangDump, CodeCoverage) - Exit code 8 behavioral difference (--ignore-exit-code 8, TESTINGPLATFORM_EXITCODE_IGNORE) Includes 10 eval scenarios with tiered timeouts (240-360s), 5 fixture directories, plugin.json, marketplace.json entries, CODEOWNERS, and README.md updates. * Fix codeowners * Improve * Improve further * Improve prompt * Improve further the description * Address review comments |
||
|
Amaury Levé
|
ab201ee691 |
Fix eval source file resolution to use eval directory (#374)
* Fix eval source file resolution to use eval directory The SetupWorkDir method resolved setup file 'source:' paths relative to the skill directory (plugins/.../skills/<name>/), but eval.yaml files live under tests/. The ../../../../ relative paths resolved correctly but were rejected by the path traversal security check because the resolved path escaped the skill directory boundary. Fix by preferring the eval directory as the base for source resolution when evalPath is available, falling back to skillPath for backward compatibility with scenarios embedded in SKILL.md files. Also update run-tests eval.yaml source paths from '../../../../tests/dotnet-test/run-tests/fixtures/...' to 'fixtures/...' (relative to the eval directory where the fixtures actually live). This was causing all 11 source-based run-tests scenarios to get zero fixture files, making the agent find an empty workspace and fail to evaluate. * Address review: extract ResolveSourcePath helper, add warning, fix remaining eval.yaml paths - Extract source-resolution and path-traversal check into internal ResolveSourcePath helper for testability - Emit a clear warning when file.Source is set but no base directory is available (instead of silently continuing) - Add unit tests covering: evalPath-based resolution, skillPath fallback, null paths warning, traversal rejection, and evalPath preference - Fix path traversal patterns in analyzing-dotnet-performance and clr-activation-debugging eval.yaml files (same issue as run-tests) |
||
|
Amaury Levé
|
7afa81f831 | Run tests supports platforms and frameworks (#343) |