mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-mai
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aditya Mandaleeka
|
52ba152ed6 | Update to Vally 0.7.0 (#854) | ||
|
Amaury Levé
|
0451fe8d91 |
dotnet-test: raise timeout for assertion-quality decline-write-tests scenario 300->360 (#862)
The 'Decline request to write new tests from scratch' non-activation scenario writes a full MSTest suite from scratch (code generation) and was hitting its 300s wall-clock timeout. Per InvestigatingResults.md, code-generation scenarios often need 360s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f0f42a7918 |
Add eval coverage for dotnet-test/assertion-quality (#819)
Adds outcome-focused rubric items so two previously-uncovered Validation teaching points become covered: - Metrics are computed correctly (counts add up) - If the suite has good diversity, the report acknowledges this Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
e3f0becd4c |
Fix test-gap-analysis skill activation in plugin eval runs (#801)
* Fix test-gap-analysis skill activation in plugin eval runs In plugin-mode eval runs (all dotnet-test skills loaded), test-gap-analysis lost activation: 'Acknowledge well-tested code' was stolen by assertion-quality and 'logic/null-check gaps' loaded no skill at all. Front-load the concrete 'would my tests catch a bug if the code changed' trigger phrasing the prompts use, and add a DO NOT USE -> test-gap-analysis redirect in assertion-quality for mutation-style reasoning. Both descriptions stay within the 1024-char cap; aggregate unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore Jest/TS activation signal for assertion-quality polyglot scenario The earlier trim of assertion-quality's description dropped the Jest/Vitest and toBeTruthy() signals, breaking plugin-mode activation for the 'Polyglot: Jest/TypeScript' scenario (agent answered directly, loading no skill). Restore those polyglot signals, and make the scenario prompt realistic — the prior prompt enumerated the full check catalog and Jest matcher list inline, acting as an answer key that let the agent self-serve without loading the skill. Verified locally: the scenario now activates assertion-quality in plugin mode across runs; all other assertion-quality scenarios still activate; description stays within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
14d727fee6 |
Fix test-anti-patterns skill activation for 5 evals (#786)
* Fix test-anti-patterns skill activation for 5 evals Sibling skills with overlapping descriptions were stealing activation from test-anti-patterns in plugin eval runs (coverage-analysis, assertion-quality, test-smell-detection). Reword descriptions so test-anti-patterns owns the umbrella 'audit my tests for anti-patterns' severity-ranked report, and add DO NOT USE redirects in the metric-focused siblings. Kept all descriptions within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use toBeTruthy() call form in assertion-quality example Addresses review feedback: the Jest matcher example read like a property without parentheses. Description stays within the 1024-char cap (1023). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval validation error and improve activation triggers - assertion-quality eval.yaml/eval.vally.yaml: replace hyphenated 'assertion-quality' (the target skill name) with spaced 'assertion quality' in two scenario prompts, fixing the 'prompt mentions target name' validation error that biased baseline runs. - test-anti-patterns description: add 'what's wrong with my tests' / 'are these tests any good' / 'flaky tests' trigger phrasing to improve organic activation for the flakiness, well-written and polyglot scenarios (which intermittently failed to activate in plugin runs). Stays within the 1024-char description cap (1007). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use Jest matcher call form in assertion-quality polyglot prompt Addresses review feedback: write toBeDefined()/toBeTruthy()/not.toBeNull()/ toBe()/toThrow() in call form in the prompt so they read as matcher calls, consistent with the skill description examples. Regex assertions and rubric left untouched (they match agent output, which may use either form). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Sharpen test-anti-patterns description for flakiness/polyglot activation The flakiness and Python-pytest scenarios failed to activate even in isolated runs (where it's the only candidate skill), because their prompts enumerate the methodology and the description's keywords were too generic. Front-load the concrete trigger keywords those prompts use: Thread.Sleep, DateTime.Now, time.sleep, order-dependent, reflection coupling, and Python/pytest. Stays within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make mixed/flakiness eval prompts realistic to fix plugin activation The mixed-severity and flakiness scenarios consistently failed to activate test-anti-patterns in plugin runs (detected=[] — the agent loaded no skill at all and answered directly). Both prompts enumerated the full anti-pattern catalog inline, acting as an answer key that made the agent self-sufficient. Replace the embedded checklists with realistic user asks while keeping the 'for .NET test anti-patterns' trigger, file references, severity-ranked output format, and read-only constraint. Rubric and output_matches assertions are unchanged — they validate the produced report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
680292e04c |
Make dotnet-test analysis skills and auditor agent polyglot (#707)
* Make dotnet-test analysis skills and auditor agent polyglot Five test analysis skills and the test-quality-auditor agent in plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/ TUnit cues, even though the underlying techniques (assertion diversity, anti-pattern detection, pseudo-mutation gap analysis, academic smell catalog, trait tagging) apply equally to other test frameworks. This change makes them polyglot across .NET, Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest). Changes ------- * New reference skill `test-analysis-extensions` with per-language extension files (dotnet, python, typescript, java, go, ruby, rust, swift, kotlin, powershell, cpp). Each documents test markers, assertion APIs, sleep / time / random APIs, skip annotations, setup/teardown, mystery-guest indicators, integration markers, and tag-support capability for the corresponding framework. * `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`, `test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter now lists all polyglot frameworks (keeping .NET trigger words for backward compat); bodies reference `test-analysis-extensions` and add per-language calibration (e.g., Go/Rust table-driven loops are idiomatic and not Conditional Test Logic; pytest bare `assert` is canonical; missing-await on async assertions is a new Critical smell). * `test-quality-auditor` agent now runs language detection, gates .NET-only pipeline steps (coverage-analysis, CRAP, detect-static- dependencies, testability migration, experimental dotnet skills), and recommends native tooling for non-.NET projects via an explicit Capability Matrix. * README updated to split sections by polyglot vs .NET-only. * New polyglot eval scenarios added (Python/pytest, TS/Jest, Java/JUnit) alongside existing C# scenarios in `test-anti-patterns`, `assertion-quality`, `test-smell-detection`. The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis, crap-score, detect-static-dependencies, run-tests, filter-syntax, platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are unchanged and continue to activate only for .NET work. Validated with skill-validator `check --plugin ./plugins/dotnet-test`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #707 review feedback * typescript.md: replace xpect.fail() recommendation (Jest has no such API) with hrow new Error(...) / explicit failing assertion. * test-tagging SKILL.md description: clarify that unittest has no canonical tag syntax (pytest markers are the auto-edit mechanism; unittest is report-only). Switch doctest reference to "decorators" rather than implying a Catch2-style [tags] mechanism exists. * test-tagging SKILL.md auto-edit list and table: use `* doctest::test_suite("tag")` decorator chain syntax for doctest in both the bullet list and the per-framework table, matching the example already shown. * cpp.md extension: spell out the Catch2 vs doctest tagging distinction in the capability matrix (Catch2 [tag], doctest * doctest::test_suite("tag") decorator). Validated with `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` (23 skills + 11 agents). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f532cf1d94 |
Add coverage-touching and self-referential assertion heuristics (#635)
* Add coverage-touching and self-referential assertion heuristics
- test-anti-patterns: add 'Coverage touching' and 'Self-referential assertion'
as named Critical-severity anti-patterns in the detection catalog
- assertion-quality: add 'Tests with self-referential assertions' metric and
a calibration rule distinguishing tautological from valid round-trip assertions
- Add eval scenarios for both new heuristics in test-anti-patterns (coverage-
touching ProductCatalog facade, self-referential SerializerTests)
- Add eval scenario and fixture for self-referential assertions in
assertion-quality (ConfigTests with identity/round-trip patterns)
* Fix test-anti-patterns activation: rewrite description with USE FOR keyword list
The skill was not activating in any eval scenario because the description
used narrative prose instead of the explicit 'USE FOR:' / 'DO NOT USE FOR:'
keyword list pattern that well-activating sibling skills use (run-tests,
writing-mstest-tests). Restructured the description with trigger phrases
matching common user prompts.
* Fix eval activation, timeouts, and add missing vally scenarios
- test-anti-patterns/eval.vally.yaml: increase timeout 2m→5m, add
coverage-touching and self-referential scenarios with constraints
- assertion-quality/eval.vally.yaml: add self-referential scenario
with environment files, broaden assertion pattern
- test-anti-patterns/eval.yaml: add setup.files to all 6 scenarios
to create workspace files that trigger skill discovery/activation
- assertion-quality/eval.yaml: increase timeouts 120→300 for
self-referential and decline-request scenarios, broaden
self-referential assertion pattern
- Create fixture files for mixed-severity, well-written,
coverage-touching, and self-referential test-anti-patterns scenarios
* Address review comments: fix test names and use source references
- Rename ThrowsArgumentException to ThrowsArgumentOutOfRangeException in
PaymentProcessorTests.cs fixture and inlined code in eval.yaml and
eval.vally.yaml to match the actual exception type being asserted.
- Replace inline csproj content blocks with source references to existing
fixture files for UserRepo, Payments, and CatalogService scenarios in
eval.yaml to avoid drift between fixtures and eval setup.
* Address review comments: pin MSTest versions, use source refs, fix naming
- Pin MSTest to 3.8.2 in all fixture .csproj files, add standard
properties (ImplicitUsings, Nullable, IsPackable)
- Replace inline csproj content blocks with source references to
fixture files to avoid content drift
- Create missing .csproj fixtures for flaky-coupled, duplicate-magic,
and assertion-quality self-referential scenarios
- Fix test method names: ThrowsArgumentException ->
ThrowsArgumentOutOfRangeException to match the actual assertion
* Fix activation, eval timeouts, and per-skill description limits
Address remaining issues from PR #635 eval scoreboard
(run 25677669933) and skill-validator failures:
1. test-anti-patterns activation (was NOT ACTIVATED in BOTH plugin and
isolated mode for ALL 6 scenarios — model answered everything from
inline code without ever needing the skill):
- eval.yaml + eval.vally.yaml: drop the giant inline `csharp` blocks
from every prompt. Each prompt now names the workspace files
explicitly (`Audit `Foo.Tests/FooTests.cs` for .NET test
anti-patterns ...`) and asks for the skill's signature deliverable
(severity-ranked list with concrete code-level fixes). The model
must read the workspace, which gives it a chance to consult the
skill — and the skill's structured-findings format is the actual
value-add over inline analysis.
- All 6 scenarios consistently get reject_tools (read-only review)
and timeout 180s.
- eval.vally.yaml: also added missing environment.files for each
scenario (was prompt-only before, no fixture on disk).
2. assertion-quality `Flag assertion-free tests` plugin-mode activation
(was bouncing to other skills because the prompt was generic `check
whether these tests verify anything useful`):
- eval.yaml + eval.vally.yaml: rewrite the prompt to name the file
(`SmokeTests/ApiEndpointSmokeTests.cs`) and explicitly request
`assertion quality and depth`, `assertion-free tests`, and
`trivial assertions like Assert.IsNotNull` — direct trigger
phrases for assertion-quality.
- assertion-quality SKILL.md: extend description with explicit USE
FOR keywords for `assertion-free tests`, `self-referential`, and
`tautological` so plugin-mode activation isn't won by sibling
skills like test-anti-patterns.
3. test-anti-patterns description size — was 1,301 chars (over the
1,024 per-skill limit) AND pushed dotnet-test plugin aggregate to
15,312 (over the 15,000 limit), failing skill-check on PR #635.
Trimmed to 779 chars; aggregate now at 14,983.
4. Vally job-level timeout (60m → 180m) was already raised on main
in commit
|
||
|
Amaury Levé
|
9f531203ec |
Promote 4 experimental test skills to dotnet-test plugin (#634)
Move assertion-quality, test-gap-analysis, test-smell-detection, and test-tagging from dotnet-experimental to dotnet-test. These skills have demonstrated sufficient eval coverage, unique value beyond baseline model capabilities, and are already integrated into the test-quality-auditor agent pipeline. Changes: - Move skills and tests, dropping the exp- prefix - Update all cross-references in skill descriptions and routing tables - Update test-quality-auditor agent and eval scenarios - Update CODEOWNERS for the moved paths - Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks) - Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability Skills held back for further iteration: - exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios - exp-test-maintainability: quality gains offset by token overhead |