mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Amaury Levé
a677ec43d2
* Consolidate dotnet-test-frameworks into test-analysis-extensions Remove the orphaned dotnet-test-frameworks reference skill, whose content was a duplicate subset of test-analysis-extensions/extensions/dotnet.md. Nothing actually loaded it. Redirect the remaining references and drop its eval tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make exp-test-maintainability Step 1 self-contained and fix activation Address PR review: exp-test-maintainability (dotnet-experimental plugin) must not point at test-analysis-extensions (dotnet-test plugin) — inline the .NET framework markers instead. Also strengthen the description/When-to-Use triggers so the 'each new case needs a whole new method / suggest a better structure' scenario reliably activates the skill instead of routing to a test-writing skill. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Revert exp-test-maintainability description broadening; keep self-contained markers The description broadening regressed activation (3/4 -> 0/4 across a variance-dominated single eval run, CV up to 924%). Description is the activation lever, so restore the original wording and keep only the review-comment fix (inlined .NET framework markers in Step 1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix scenario 1 eval: use project fixture so the skill activates Scenario 1 pasted a self-contained snippet inline and asked to 'suggest a better structure', which the base model answers directly without loading the skill -> NOT ACTIVATED with no headroom. Convert it to the project-fixture pattern used by the activating scenarios (3 & 4): move the repetitive InputValidatorTests into a Validation.Tests fixture and ask the model to analyze the project. Mirror the change in eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix scenario 2 eval: use project fixture so the skill activates Scenario 2 also pasted a self-contained snippet inline (well-maintained Auth.Tests) and asked to 'review these tests', which the base model handles without loading the skill. Convert to the project-fixture pattern used by the activating scenarios; add fixtures/well-maintained/Auth.Tests. Mirror in eval.vally.yaml. All four scenarios now reference a project on disk. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add reject_tools to exp-test-maintainability evals to create headroom All four scenarios scored NOT ACTIVATED / no-headroom (#8 in InvestigatingResults.md): baselines are already 4.0-5.0/5, so loading the skill only adds token/tool overhead with no offsetting quality gain. Match the convention of the sibling analysis skill test-anti-patterns: reject bash/edit/create on every scenario. This skill is analysis-only ('do not modify any files'), so the constraint is a no-op for correct behavior but levels the playing field between baseline and skilled runs — scoring focuses on answer quality (weight 0.40+0.30) instead of tool-induced overhead. Applied to both eval.yaml and eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address review: make well-maintained fixture self-contained and deterministic - Import System.Security.Authentication for AuthenticationException - Replace wall-clock assertion (token.ExpiresAt > DateTime.UtcNow) with a deterministic one anchored on a fixed clock (FixedNow), and drive the injected FakeClock from that fixed time. This removes the flakiness and makes the FakeClock injection meaningful — a better exemplar for the 'well-maintained' scenario. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
107 lines
6.2 KiB
YAML
107 lines
6.2 KiB
YAML
scenarios:
|
|
- name: "Recommend data-driven patterns with display names for unclear parameters"
|
|
prompt: |
|
|
I keep adding test cases to my Validation.Tests project and each new case
|
|
needs a whole new method. Can you analyze the project and suggest a better structure?
|
|
setup:
|
|
files:
|
|
- path: "Validation.Tests/Validation.Tests.csproj"
|
|
source: "fixtures/data-driven-candidate/Validation.Tests/Validation.Tests.csproj"
|
|
- path: "Validation.Tests/InputValidatorTests.cs"
|
|
source: "fixtures/data-driven-candidate/Validation.Tests/InputValidatorTests.cs"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(DataRow|InlineData|TestCase|data.driven|parameteriz)"
|
|
- type: "output_matches"
|
|
pattern: "(DisplayName|display.name)"
|
|
- type: "exit_success"
|
|
rubric:
|
|
- "Identified that the invalid-input tests can be consolidated into a parameterized test with DataRow"
|
|
- "Identified that the valid-input tests can similarly be consolidated"
|
|
- "Recommended labeling parameterized test cases so that test runner output identifies which case failed"
|
|
- "Showed concrete refactored code with clear test case labels"
|
|
- "Acknowledged that the tests are otherwise well-structured (focused, clear assertions, proper naming)"
|
|
reject_tools: ["bash", "edit", "create"]
|
|
timeout: 120
|
|
|
|
- name: "Recognize well-maintained tests that need minimal changes"
|
|
prompt: |
|
|
Can you review my Auth.Tests project for maintainability? I want to make sure
|
|
the tests will be easy to update as the codebase evolves.
|
|
setup:
|
|
files:
|
|
- path: "Auth.Tests/Auth.Tests.csproj"
|
|
source: "fixtures/well-maintained/Auth.Tests/Auth.Tests.csproj"
|
|
- path: "Auth.Tests/TokenServiceTests.cs"
|
|
source: "fixtures/well-maintained/Auth.Tests/TokenServiceTests.cs"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(well.maintained|good shape|clean|solid|maintainable|easy to maintain|no.*(major|significant))"
|
|
- type: "exit_success"
|
|
rubric:
|
|
- "Recognized that these tests are already well-maintained and need no major refactoring"
|
|
- "Acknowledged the existing helper method (CreateTokenService) as a good pattern"
|
|
- "Acknowledged that parameterized tests are well-labeled and effectively structured"
|
|
- "Did not recommend over-engineered patterns disproportionate to the test complexity"
|
|
- "If any suggestions were made, they were minor and clearly labeled as optional polish"
|
|
reject_tools: ["bash", "edit", "create"]
|
|
timeout: 120
|
|
|
|
# ==========================================================================
|
|
# Scenario 3: Heavy boilerplate — repeated construction and assertion patterns
|
|
# ==========================================================================
|
|
|
|
- name: "Detect repeated object construction and setup across test methods"
|
|
prompt: |
|
|
I feel like my test code has a lot of copy-paste. Can you analyze
|
|
my OrderService.Tests project and tell me what patterns are repeated
|
|
and where I could refactor?
|
|
setup:
|
|
files:
|
|
- path: "OrderService.Tests/OrderService.Tests.csproj"
|
|
source: "fixtures/heavy-boilerplate/OrderService.Tests/OrderService.Tests.csproj"
|
|
- path: "OrderService.Tests/OrderProcessorTests.cs"
|
|
source: "fixtures/heavy-boilerplate/OrderService.Tests/OrderProcessorTests.cs"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(factory|helper|extract|CreateProcessor|CreateOrder|shared|reuse|common)"
|
|
- type: "output_matches"
|
|
pattern: "(3|4|5|6|7|8|multiple|every|all|each|repeated)"
|
|
- type: "exit_success"
|
|
rubric:
|
|
- "Identified that the three-line FakeLogger/FakeEmailService/FakeInventoryService construction block is repeated in every OrderProcessorTests test method"
|
|
- "Identified that Order object construction with Items list is repeated across both test classes with very similar structure"
|
|
- "Suggested extracting a helper method or factory for creating the OrderProcessor with its dependencies"
|
|
- "Suggested extracting a helper for creating test Order objects with customizable properties"
|
|
- "Noted that OrderValidatorTests also repeats the same Order construction pattern as OrderProcessorTests, suggesting a cross-class shared helper"
|
|
- "Provided concrete before/after code showing how at least one extraction would look"
|
|
- "Did not recommend removing all duplication indiscriminately — acknowledged trade-offs between DRY and test readability"
|
|
reject_tools: ["bash", "edit", "create"]
|
|
timeout: 120
|
|
|
|
# ==========================================================================
|
|
# Scenario 4: Minimal boilerplate — tests are already well-structured
|
|
# ==========================================================================
|
|
|
|
- name: "Recognize tests with minimal boilerplate that need no refactoring"
|
|
prompt: |
|
|
Can you check my Calculator.Tests project for any repeated code patterns
|
|
that I should clean up? I have two test classes in there.
|
|
setup:
|
|
files:
|
|
- path: "Calculator.Tests/Calculator.Tests.csproj"
|
|
source: "fixtures/minimal-boilerplate/Calculator.Tests/Calculator.Tests.csproj"
|
|
- path: "Calculator.Tests/CalculatorTests.cs"
|
|
source: "fixtures/minimal-boilerplate/Calculator.Tests/CalculatorTests.cs"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(clean|minimal|good|well.structured|no.*(major|significant)|already|little.*(duplicat|boilerplate)|minor)"
|
|
- type: "exit_success"
|
|
rubric:
|
|
- "Recognized that CalculatorTests already uses parameterized data-driven tests and a shared field instance effectively"
|
|
- "Identified that ScientificCalculatorTests has individual test methods that could be consolidated into parameterized tests (e.g., SquareRoot positive/zero cases, Power squared/zero/negative exponent cases)"
|
|
- "Noted that the overall code is well-structured and any suggestions are minor improvements, not critical refactorings"
|
|
- "Did not recommend over-engineered patterns like base classes or shared fixtures for these simple test classes"
|
|
reject_tools: ["bash", "edit", "create"]
|
|
timeout: 120
|