Amaury Levé
948a9a52ef
* Add exp-mock-usage-analysis skill and evaluation New experimental skill that detects unused, redundant, or unnecessary mocks in .NET test suites. Covers Moq, NSubstitute, FakeItEasy, and manual test doubles. - SKILL.md with 6-step workflow: gather code, classify mocks, detect anti-patterns, incorporate runtime data, generate recommendations, report findings - eval.yaml with 10 scenarios: trivial type mocking, unused setups, redundant configs, stable utility mocks, well-placed mocks (no false positives), NSubstitute support, FakeItEasy support, setup sprawl, and 2 non-activation boundary tests - Fixture files for each scenario with production + test code - CODEOWNERS entry for @dotnet/dotnet-testing * Remove no-headroom scenarios, increase NSubstitute timeout Remove 4 scenarios where baseline already scores 5.0/5 (model handles these well without skill guidance): - Detect mocking of DTOs, records, and enums - Recognize well-placed mocks without inventing false positives - Decline when asked about non-mock test anti-patterns - Decline mock framework migration request Increase NSubstitute scenario timeout from 120s to 180s to avoid baseline timeouts impacting scoring. Remaining 6 scenarios all show positive skill impact (3→5, 3→4, etc). * Document areas the model handles well without the skill Add a subsection under 'When Not to Use' listing mock-related tasks where the base model already scores 5.0/5 without skill guidance: - Mocking trivial types (DTOs, records, enums) - Recognizing well-placed mocks (no false positives) * Fix md * Improve
7.6 KiB
Mock Usage Analysis Skill — Design Notes
Evaluation Results (March 2026)
Round 1 — Initial skill (11 scenarios)
| Scenario | Baseline | Isolated | Plugin | Verdict |
|---|---|---|---|---|
| Detect mocking of DTOs, records, and enums | 4.3/5 | 3.3/5 ⏰ | 4.3/5 | ❌ |
| Detect unused and unreachable mock setups | 4.0/5 | 4.0/5 ⏰ | 4.3/5 ⏰ | ❌ ¹ |
| Detect redundant mock configurations | 3.0/5 | 2.3/5 ⏰ | 3.3/5 | ❌ |
| Detect mocking of stable framework types | 3.0/5 | 5.0/5 | 5.0/5 | ✅ |
| Recognize well-placed mocks | 5.0/5 | 5.0/5 | 5.0/5 | ❌ ² |
| Analyze mock usage in NSubstitute tests | 3.7/5 | 5.0/5 | 5.0/5 | ✅ |
| Analyze mock usage in FakeItEasy tests | 5.0/5 | 4.7/5 | 4.7/5 | ❌ |
| Detect excessive mock configuration sprawl | 3.3/5 | 4.0/5 | 3.3/5 | ✅ |
| Decline request to write new tests | 2.0/5 | 2.0/5 | 2.3/5 | ❌ ³ |
| Decline non-mock test anti-patterns | 5.0/5 | 5.0/5 | 5.0/5 | ❌ ² |
| Decline mock framework migration | 4.0/5 | 4.0/5 | 4.0/5 | ❌ ⁴ |
3/11 passed. Overfitting: 0.06 (excellent).
¹ Quality improved in plugin but weighted score -25.2% from token/time overhead. ² Baseline at ceiling — no headroom for skill to add value. ³ Token overhead regression on a non-activation scenario with no quality gain. ⁴ Weighted -1.9% from token/time overhead with no quality delta.
Issues identified:
- Timeouts on scenarios 1-3 (120s too short for fixture-based scenarios)
- Activation failures — scenario 1 not activated in plugin, scenario 3 not activated in either mode (prompts lacked mock-specific keywords)
- Baseline at ceiling — 4 scenarios where the model already scores 5.0/5
Round 2 — Fix timeouts, activation, and no-headroom scenarios
Changes:
- Increased timeouts: 120s → 180s for scenarios with fixture files
- Rewrote prompts with explicit mock terminology for better activation
- Added
reject_tools: ["bash", "edit"]to FakeItEasy and well-placed mocks scenarios - Improved skill description with framework-specific keywords (Mock, Substitute.For, A.Fake)
- Removed "Decline write tests" scenario (token overhead, no value)
| Scenario | Baseline | Isolated | Plugin | Verdict |
|---|---|---|---|---|
| Detect mocking of DTOs, records, and enums | 5.0/5 | 5.0/5 | 5.0/5 | ❌ ⁵ |
| Detect unused and unreachable mock setups | 3.3/5 | 5.0/5 | — | ✅ |
| Detect redundant mock configurations | 3.0/5 | 4.0/5 | — | ✅ |
| Detect mocking of stable framework types | 3.0/5 | 5.0/5 | — | ✅ |
| Recognize well-placed mocks | 5.0/5 | 5.0/5 | 5.0/5 | ❌ ⁵ |
| Analyze mock usage in NSubstitute tests | 3.0/5 ⏰ | 5.0/5 | — | ✅ |
| Analyze mock usage in FakeItEasy tests | 4.3/5 | 4.7/5 | — | ❌ |
| Detect excessive mock configuration sprawl | 3.0/5 | 4.0/5 | — | ✅ |
| Decline non-mock test anti-patterns | 5.0/5 | 3.7/5 ⏰ | — | ❌ |
| Decline mock framework migration | 5.0/5 | 5.0/5 | — | ❌ ⁵ |
⁵ Baseline at ceiling — model handles these well without skill guidance.
Improvements from Round 1:
- DTOs scenario: now activates in both isolated and plugin (was plugin-only failure)
- Redundant mocks: now activates in both modes (was NOT ACTIVATED in either)
- No more timeouts on scenarios 1-3
- NSubstitute baseline still hit timeout at 120s
Round 3 — Remove no-headroom scenarios, fix NSubstitute timeout
Changes:
- Removed 4 scenarios where baseline scores 5.0/5 (see "Decisions" below)
- Increased NSubstitute timeout: 120s → 180s
6 remaining scenarios all show positive skill impact.
Key Insight
The baseline LLM already excels at two mock-related tasks:
- Identifying trivial-type mocking — the model recognizes when
Mock<CustomerDto>should benew CustomerDto(...)without guidance. - Recognizing well-placed mocks — when tests correctly mock external boundaries, the model concludes the approach is sound without inventing false positives.
The skill's unique value is in deep code-path analysis: tracing mock setups through production code to determine whether they are actually invoked at runtime, identifying unreachable setups after early returns or exceptions, and detecting redundant configurations duplicated across tests.
Decisions
Removed: "Detect mocking of DTOs, records, and enums" scenario
Baseline scores 5.0/5 — the model already identifies when DTOs, records, and enums are unnecessarily mocked and recommends real instance construction. No quality delta for the skill to contribute.
Removed: "Recognize well-placed mocks without inventing false positives" scenario
Baseline scores 5.0/5 — the model already correctly concludes that mocking external boundaries (HTTP, DB, email) is appropriate without inflating severity.
Removed: "Decline when asked about non-mock test anti-patterns" scenario
Baseline scores 5.0/5 — non-activation scenario where the model already handles Thread.Sleep/DateTime.Now reviews without the skill. The timeout regression (5.0→3.7 ⏰) in the skilled run was caused by the 60s timeout being too short, not a skill problem.
Removed: "Decline mock framework migration request" scenario
Baseline scores 5.0/5 — the model already handles Moq→NSubstitute migration requests without the skill. Weighted score was -1.2% from time overhead alone.
Round 4 — Drastic skill simplification
Problem: Results degraded significantly. The skill was actively hurting quality — 5/6 scenarios scored worse with the skill than without it. When activated, scores dropped from 2.3-3.3 baseline to 1.0-2.3. Two scenarios showed "NOT ACTIVATED" indicating the skill loaded but the model chose not to use it.
| Scenario | Baseline | With Skill | Verdict |
|---|---|---|---|
| Detect unused and unreachable mock setups | 3.0/5 | 2.3/5 | ❌ |
| Detect redundant mock configurations | 3.3/5 | 1.0/5 (NOT ACTIVATED) | ❌ |
| Detect mocking of stable framework types | 3.0/5 | 2.3/5 (NOT ACTIVATED) | ❌ |
| Analyze mock usage in NSubstitute tests | 2.3/5 | 1.0/5 | ❌ |
| Analyze mock usage in FakeItEasy tests | 3.3/5 | 1.0/5 | ❌ |
| Detect excessive mock configuration sprawl | 2.7/5 | 3.7/5 | ✅ |
Root cause analysis:
- Skill too verbose (~200 lines) — Extensive anti-pattern catalog tables that the model already knows, consuming attention budget that should go to code analysis.
- 6-step workflow too rigid — Model spent effort following the prescribed categorization workflow (classify dependencies as Trivial/Stable/Thin/External/Complex) instead of doing actual code-path tracing.
- Anti-pattern encyclopedia redundant — 4 severity levels × 3-4 patterns each = 15+ anti-patterns listed. The model already knows these; listing them added noise without value.
- Reporting format instructions too prescriptive — "Present findings in this structure: Summary → Critical/High → Medium/Low → Positive → Aggregate" forced a template that didn't match rubric expectations.
Changes:
- Cut skill from ~200 lines to ~90 lines
- Reduced workflow from 6 steps to 4 (read → trace → check replaceable → report)
- Removed the anti-pattern catalog entirely — model already knows common mock anti-patterns
- Removed dependency categorization tables (Trivial/Stable/Thin/External/Complex)
- Removed the runtime data incorporation step (never used in evals)
- Focused Step 2 entirely on code-path tracing — the unique value-add identified in Round 2
- Added explicit guidance on early returns, exceptions, and branch-specific logic as things to trace
- Simplified reporting to: specific location + why unreachable + concrete fix
- Enhanced description with more trigger phrases for better activation