mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-opus48
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aditya Mandaleeka
|
52ba152ed6 | Update to Vally 0.7.0 (#854) | ||
|
Amaury Levé
|
14d727fee6 |
Fix test-anti-patterns skill activation for 5 evals (#786)
* Fix test-anti-patterns skill activation for 5 evals Sibling skills with overlapping descriptions were stealing activation from test-anti-patterns in plugin eval runs (coverage-analysis, assertion-quality, test-smell-detection). Reword descriptions so test-anti-patterns owns the umbrella 'audit my tests for anti-patterns' severity-ranked report, and add DO NOT USE redirects in the metric-focused siblings. Kept all descriptions within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use toBeTruthy() call form in assertion-quality example Addresses review feedback: the Jest matcher example read like a property without parentheses. Description stays within the 1024-char cap (1023). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval validation error and improve activation triggers - assertion-quality eval.yaml/eval.vally.yaml: replace hyphenated 'assertion-quality' (the target skill name) with spaced 'assertion quality' in two scenario prompts, fixing the 'prompt mentions target name' validation error that biased baseline runs. - test-anti-patterns description: add 'what's wrong with my tests' / 'are these tests any good' / 'flaky tests' trigger phrasing to improve organic activation for the flakiness, well-written and polyglot scenarios (which intermittently failed to activate in plugin runs). Stays within the 1024-char description cap (1007). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use Jest matcher call form in assertion-quality polyglot prompt Addresses review feedback: write toBeDefined()/toBeTruthy()/not.toBeNull()/ toBe()/toThrow() in call form in the prompt so they read as matcher calls, consistent with the skill description examples. Regex assertions and rubric left untouched (they match agent output, which may use either form). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Sharpen test-anti-patterns description for flakiness/polyglot activation The flakiness and Python-pytest scenarios failed to activate even in isolated runs (where it's the only candidate skill), because their prompts enumerate the methodology and the description's keywords were too generic. Front-load the concrete trigger keywords those prompts use: Thread.Sleep, DateTime.Now, time.sleep, order-dependent, reflection coupling, and Python/pytest. Stays within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make mixed/flakiness eval prompts realistic to fix plugin activation The mixed-severity and flakiness scenarios consistently failed to activate test-anti-patterns in plugin runs (detected=[] — the agent loaded no skill at all and answered directly). Both prompts enumerated the full anti-pattern catalog inline, acting as an answer key that made the agent self-sufficient. Replace the embedded checklists with realistic user asks while keeping the 'for .NET test anti-patterns' trigger, file references, severity-ranked output format, and read-only constraint. Rubric and output_matches assertions are unchanged — they validate the produced report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f532cf1d94 |
Add coverage-touching and self-referential assertion heuristics (#635)
* Add coverage-touching and self-referential assertion heuristics
- test-anti-patterns: add 'Coverage touching' and 'Self-referential assertion'
as named Critical-severity anti-patterns in the detection catalog
- assertion-quality: add 'Tests with self-referential assertions' metric and
a calibration rule distinguishing tautological from valid round-trip assertions
- Add eval scenarios for both new heuristics in test-anti-patterns (coverage-
touching ProductCatalog facade, self-referential SerializerTests)
- Add eval scenario and fixture for self-referential assertions in
assertion-quality (ConfigTests with identity/round-trip patterns)
* Fix test-anti-patterns activation: rewrite description with USE FOR keyword list
The skill was not activating in any eval scenario because the description
used narrative prose instead of the explicit 'USE FOR:' / 'DO NOT USE FOR:'
keyword list pattern that well-activating sibling skills use (run-tests,
writing-mstest-tests). Restructured the description with trigger phrases
matching common user prompts.
* Fix eval activation, timeouts, and add missing vally scenarios
- test-anti-patterns/eval.vally.yaml: increase timeout 2m→5m, add
coverage-touching and self-referential scenarios with constraints
- assertion-quality/eval.vally.yaml: add self-referential scenario
with environment files, broaden assertion pattern
- test-anti-patterns/eval.yaml: add setup.files to all 6 scenarios
to create workspace files that trigger skill discovery/activation
- assertion-quality/eval.yaml: increase timeouts 120→300 for
self-referential and decline-request scenarios, broaden
self-referential assertion pattern
- Create fixture files for mixed-severity, well-written,
coverage-touching, and self-referential test-anti-patterns scenarios
* Address review comments: fix test names and use source references
- Rename ThrowsArgumentException to ThrowsArgumentOutOfRangeException in
PaymentProcessorTests.cs fixture and inlined code in eval.yaml and
eval.vally.yaml to match the actual exception type being asserted.
- Replace inline csproj content blocks with source references to existing
fixture files for UserRepo, Payments, and CatalogService scenarios in
eval.yaml to avoid drift between fixtures and eval setup.
* Address review comments: pin MSTest versions, use source refs, fix naming
- Pin MSTest to 3.8.2 in all fixture .csproj files, add standard
properties (ImplicitUsings, Nullable, IsPackable)
- Replace inline csproj content blocks with source references to
fixture files to avoid content drift
- Create missing .csproj fixtures for flaky-coupled, duplicate-magic,
and assertion-quality self-referential scenarios
- Fix test method names: ThrowsArgumentException ->
ThrowsArgumentOutOfRangeException to match the actual assertion
* Fix activation, eval timeouts, and per-skill description limits
Address remaining issues from PR #635 eval scoreboard
(run 25677669933) and skill-validator failures:
1. test-anti-patterns activation (was NOT ACTIVATED in BOTH plugin and
isolated mode for ALL 6 scenarios — model answered everything from
inline code without ever needing the skill):
- eval.yaml + eval.vally.yaml: drop the giant inline `csharp` blocks
from every prompt. Each prompt now names the workspace files
explicitly (`Audit `Foo.Tests/FooTests.cs` for .NET test
anti-patterns ...`) and asks for the skill's signature deliverable
(severity-ranked list with concrete code-level fixes). The model
must read the workspace, which gives it a chance to consult the
skill — and the skill's structured-findings format is the actual
value-add over inline analysis.
- All 6 scenarios consistently get reject_tools (read-only review)
and timeout 180s.
- eval.vally.yaml: also added missing environment.files for each
scenario (was prompt-only before, no fixture on disk).
2. assertion-quality `Flag assertion-free tests` plugin-mode activation
(was bouncing to other skills because the prompt was generic `check
whether these tests verify anything useful`):
- eval.yaml + eval.vally.yaml: rewrite the prompt to name the file
(`SmokeTests/ApiEndpointSmokeTests.cs`) and explicitly request
`assertion quality and depth`, `assertion-free tests`, and
`trivial assertions like Assert.IsNotNull` — direct trigger
phrases for assertion-quality.
- assertion-quality SKILL.md: extend description with explicit USE
FOR keywords for `assertion-free tests`, `self-referential`, and
`tautological` so plugin-mode activation isn't won by sibling
skills like test-anti-patterns.
3. test-anti-patterns description size — was 1,301 chars (over the
1,024 per-skill limit) AND pushed dotnet-test plugin aggregate to
15,312 (over the 15,000 limit), failing skill-check on PR #635.
Trimmed to 779 chars; aggregate now at 14,983.
4. Vally job-level timeout (60m → 180m) was already raised on main
in commit
|
||
|
Aditya Mandaleeka
|
9433a31e67 | Initial port of evals to Vally (#615) |