mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Amaury Levé
0451fe8d91
The 'Decline request to write new tests from scratch' non-activation scenario writes a full MSTest suite from scratch (code generation) and was hitting its 300s wall-clock timeout. Per InvestigatingResults.md, code-generation scenarios often need 360s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
207 lines
12 KiB
YAML
207 lines
12 KiB
YAML
scenarios:
|
|
# ==========================================================================
|
|
# Scenario 1: Low assertion diversity — equality-heavy test suite
|
|
# ==========================================================================
|
|
|
|
- name: "Identify low assertion diversity in equality-dominated test suite"
|
|
prompt: |
|
|
I want to understand whether my PaymentService tests are actually
|
|
thorough. They all pass, but I'm not confident they'd catch real bugs.
|
|
Can you analyze the assertion quality and variety?
|
|
setup:
|
|
files:
|
|
- path: "PaymentService.Tests/PaymentService.Tests.csproj"
|
|
source: "fixtures/low-diversity/PaymentService.Tests/PaymentService.Tests.csproj"
|
|
- path: "PaymentService.Tests/PaymentProcessorTests.cs"
|
|
source: "fixtures/low-diversity/PaymentService.Tests/PaymentProcessorTests.cs"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(diversity|variety|spread|shallow|narrow|limited|homogeneous|repetitive)"
|
|
- type: "output_matches"
|
|
pattern: "(AreEqual|equality|equal)"
|
|
- type: "output_matches"
|
|
pattern: "(exception|throw|error|negative|missing)"
|
|
- type: "exit_success"
|
|
rubric:
|
|
- "Identified that the vast majority of assertions are Assert.AreEqual, showing low assertion type diversity"
|
|
- "Noted the absence of exception assertions — no tests verify error handling behavior (e.g., invalid card, negative amounts)"
|
|
- "Identified GetReceipt test as having only a trivial Assert.IsNotNull without checking the receipt's content"
|
|
- "Noted that no tests verify collection contents in GetTransactionHistory — only the count is checked"
|
|
- "Recommended specific additional assertion types that would improve coverage (exception, collection, structural)"
|
|
- "Provided concrete examples of assertions that could be added to specific test methods"
|
|
- "Reported assertion metrics that are computed correctly — the per-category counts add up and are internally consistent with the total number of assertions"
|
|
timeout: 120
|
|
|
|
# ==========================================================================
|
|
# Scenario 2: Assertion-free smoke tests
|
|
# ==========================================================================
|
|
|
|
- name: "Flag assertion-free tests and trivial-only assertions"
|
|
prompt: |
|
|
Our QA team says these API smoke tests give us good coverage but I'm
|
|
suspicious. Analyze the assertion quality and depth in
|
|
`SmokeTests/ApiEndpointSmokeTests.cs` — specifically identify
|
|
assertion-free tests (tests that call methods without verifying
|
|
results) and tests with only trivial assertions like
|
|
`Assert.IsNotNull`. Report which tests are assertion-free, the overall
|
|
assertion quality, and concrete recommendations. Do not edit the
|
|
files.
|
|
setup:
|
|
files:
|
|
- path: "SmokeTests/SmokeTests.csproj"
|
|
source: "fixtures/assertion-free/SmokeTests/SmokeTests.csproj"
|
|
- path: "SmokeTests/ApiEndpointSmokeTests.cs"
|
|
source: "fixtures/assertion-free/SmokeTests/ApiEndpointSmokeTests.cs"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(no assertion|zero assertion|assertion.free|without.*assert|missing.*assert|don.*assert|lack.*assert)"
|
|
- type: "output_matches"
|
|
pattern: "(5|five|62%|most|majority|several)"
|
|
- type: "exit_success"
|
|
rubric:
|
|
- "Identified that 5 out of 8 tests have zero assertions — they only call methods without verifying results"
|
|
- "Identified that the remaining 3 tests only use Assert.IsNotNull, which is a trivial assertion that doesn't verify content"
|
|
- "Explained that these tests give false security — code coverage may look good but correctness is not verified"
|
|
- "Recommended specific meaningful assertions for at least some of the assertion-free tests"
|
|
- "Distinguished between intentional smoke tests (acceptable if labeled as such) and tests masquerading as real verification"
|
|
timeout: 300
|
|
|
|
# ==========================================================================
|
|
# Scenario 3: Good assertion diversity — acknowledge well-tested code
|
|
# ==========================================================================
|
|
|
|
- name: "Recognize well-diversified assertion usage"
|
|
prompt: |
|
|
Can you evaluate the assertion quality in my UserService.Tests project?
|
|
I want to make sure my tests are thorough enough before we ship.
|
|
setup:
|
|
files:
|
|
- path: "UserService.Tests/UserService.Tests.csproj"
|
|
source: "fixtures/good-diversity/UserService.Tests/UserService.Tests.csproj"
|
|
- path: "UserService.Tests/UserManagerTests.cs"
|
|
source: "fixtures/good-diversity/UserService.Tests/UserManagerTests.cs"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(good|strong|solid|diverse|well|thorough|comprehensive|varied)"
|
|
- type: "exit_success"
|
|
rubric:
|
|
- "Recognized that the test suite uses a variety of assertion types including equality, boolean, null, exception, and collection assertions"
|
|
- "Noted positive patterns: tests check multiple properties on returned objects, not just one field"
|
|
- "Acknowledged the negative assertions (Assert.IsNull for non-existent user, Assert.IsFalse for failed delete, Assert.AreNotEqual for role change)"
|
|
- "Acknowledged the exception assertions with specific exception types and parameter name verification"
|
|
- "If any gaps were identified, they were presented as minor enhancement opportunities rather than problems"
|
|
- "Because the suite already has good diversity, the report explicitly acknowledges this strength instead of inventing problems"
|
|
timeout: 300
|
|
|
|
# ==========================================================================
|
|
# Scenario 4: Self-referential assertions — tautological round-trip tests
|
|
# ==========================================================================
|
|
|
|
- name: "Identify self-referential assertions in identity and round-trip tests"
|
|
prompt: |
|
|
I want an assertion quality analysis of my Config tests. They all pass
|
|
but I'm not sure they'd catch real bugs. Analyze the assertion quality
|
|
and depth in `ConfigService.Tests/ConfigTests.cs` — specifically
|
|
identify self-referential or tautological assertions where the test
|
|
asserts the output equals the input on identity / round-trip
|
|
operations. Report assertion quality metrics (how many tests are
|
|
self-referential, which ones), explain why each flagged test fails to
|
|
verify real behavior, and recommend concrete improvements. Do not
|
|
edit the files — this is a read-only assertion quality review.
|
|
setup:
|
|
files:
|
|
- path: "ConfigService.Tests/ConfigService.Tests.csproj"
|
|
source: "fixtures/self-referential/ConfigService.Tests/ConfigService.Tests.csproj"
|
|
- path: "ConfigService.Tests/ConfigTests.cs"
|
|
source: "fixtures/self-referential/ConfigService.Tests/ConfigTests.cs"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(self.referent|tautolog|identity|input.*equals.*output|circular|same.*value|trivial.*round.?trip|round.?trip|set.*get|write.*read|proves nothing|not.*(testing|verif)|meaningless)"
|
|
- type: "output_matches"
|
|
pattern: "(GetTimeout|GetRetries|FormatConnectionString|ValidateKey)"
|
|
- type: "exit_success"
|
|
rubric:
|
|
- "Identified that multiple tests assert the output equals the input — a self-referential pattern that cannot catch transformation bugs"
|
|
- "Flagged GetTimeout_Default_ReturnsDefault and GetRetries_Default_ReturnsDefault as tautological — they assert that reading back a just-written value returns the same value without testing the default/fallback behavior the names suggest"
|
|
- "Flagged FormatConnectionString_RoundTrip as only proving round-trip identity without verifying the formatted string's actual structure or content"
|
|
- "Flagged ValidateKey_Valid_ReturnsSameKey as not exercising the validation logic — it only proves valid input passes through unchanged"
|
|
- "Noted that Serialize_Deserialize round-trip is potentially valid but should be accompanied by tests with edge-case inputs (special characters, nulls, empty strings)"
|
|
- "Recommended concrete improvements: test actual default values, test invalid keys, test formatted string structure, test edge-case serialization inputs"
|
|
reject_tools: ["bash", "edit", "create"]
|
|
timeout: 300
|
|
|
|
# ==========================================================================
|
|
# Scenario 5: Non-activation — user wants to run tests
|
|
# ==========================================================================
|
|
|
|
- name: "Decline request to write new tests from scratch"
|
|
prompt: |
|
|
I need to write unit tests for my InventoryTracker class.
|
|
It handles stock levels, reorder alerts, and warehouse transfers.
|
|
Can you help me write a full MSTest test suite from scratch?
|
|
expect_activation: false
|
|
setup:
|
|
files:
|
|
- path: "InventoryTracker.cs"
|
|
content: |
|
|
namespace Warehouse;
|
|
|
|
public sealed class InventoryTracker
|
|
{
|
|
public int GetStockLevel(string sku) => throw new NotImplementedException();
|
|
public void AddStock(string sku, int quantity) => throw new NotImplementedException();
|
|
public bool NeedsReorder(string sku) => throw new NotImplementedException();
|
|
}
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(TestMethod|TestClass|\\[Fact\\]|test)"
|
|
rubric:
|
|
- "Wrote test methods for the InventoryTracker class"
|
|
- "Covered multiple methods of the class"
|
|
timeout: 360
|
|
|
|
# ==========================================================================
|
|
# Scenario: Polyglot — shallow assertions in a Jest/TypeScript suite
|
|
# ==========================================================================
|
|
|
|
- name: "Polyglot: evaluate shallow assertions in a Jest/TypeScript OrderService suite"
|
|
prompt: |
|
|
Can you audit the assertion quality and depth of my Jest/TypeScript
|
|
test suite? The tests are in `tests/order.test.ts` and the production
|
|
code is in `src/order.ts`. Several tests look like they pass without
|
|
really verifying much — shallow or always-true checks — and I'm not
|
|
sure the async ones actually await their assertions. Please flag the
|
|
weak spots and call out the tests that ARE genuinely well-targeted.
|
|
Read-only review — don't edit the files.
|
|
setup:
|
|
files:
|
|
- path: "package.json"
|
|
source: "fixtures/jest-shallow/package.json"
|
|
- path: "src/order.ts"
|
|
source: "fixtures/jest-shallow/src/order.ts"
|
|
- path: "tests/order.test.ts"
|
|
source: "fixtures/jest-shallow/tests/order.test.ts"
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(shallow|trivial|toBeDefined|toBeTruthy|not\\.toBeNull|weak|low.*depth)"
|
|
- type: "output_matches"
|
|
pattern: "(always.*true|true.*true|tautolog)"
|
|
- type: "output_matches"
|
|
pattern: "(await|missing.*await|\\.resolves|async|silently.*pass)"
|
|
- type: "output_matches"
|
|
pattern: "(identity|same|self.compar|toBe\\(.*\\.id\\)|o\\.id.*o\\.id)"
|
|
- type: "exit_success"
|
|
rubric:
|
|
- "Identified Jest/TS-specific patterns correctly (did NOT mention MSTest/xUnit APIs as if present)"
|
|
- "Flagged the cluster of toBeDefined / toBeTruthy / not.toBeNull / Array.isArray assertions as trivial / low-depth"
|
|
- "Flagged the `expect(true).toBe(true)` always-true assertion"
|
|
- "Flagged the missing `await` on `expect(Promise.resolve(...)).resolves.toBeDefined()` as a silent-pass smell that lets rejections slip through"
|
|
- "Flagged `expect(o.id).toBe(o.id)` as an identity / self-comparing assertion that proves nothing"
|
|
- "Flagged `expect(o.id).not.toBe('totally unrelated string')` as a vacuous negative assertion"
|
|
- "Acknowledged 'legitimate equality on returned customer' (toBe('alice')) as a well-targeted assertion"
|
|
- "Acknowledged 'legitimate exception assertion' (toThrow('customer is required')) as a well-targeted assertion"
|
|
- "Recommended concrete higher-value assertions: verifying status='pending', itemCount, totalCents, id format, list() contents after multiple creates"
|
|
reject_tools: ["bash", "edit", "create"]
|
|
timeout: 180
|
|
|