dotnet__skills/tests/dotnet-test/assertion-quality/eval.yaml

scenarios:
  # ==========================================================================
  # Scenario 1: Low assertion diversity — equality-heavy test suite
  # ==========================================================================

  - name: "Identify low assertion diversity in equality-dominated test suite"
    prompt: |
      I want to understand whether my PaymentService tests are actually
      thorough. They all pass, but I'm not confident they'd catch real bugs.
      Can you analyze the assertion quality and variety?
    setup:
      files:
        - path: "PaymentService.Tests/PaymentService.Tests.csproj"
          source: "fixtures/low-diversity/PaymentService.Tests/PaymentService.Tests.csproj"
        - path: "PaymentService.Tests/PaymentProcessorTests.cs"
          source: "fixtures/low-diversity/PaymentService.Tests/PaymentProcessorTests.cs"
    assertions:
      - type: "output_matches"
        pattern: "(diversity|variety|spread|shallow|narrow|limited|homogeneous|repetitive)"
      - type: "output_matches"
        pattern: "(AreEqual|equality|equal)"
      - type: "output_matches"
        pattern: "(exception|throw|error|negative|missing)"
      - type: "exit_success"
    rubric:
      - "Identified that the vast majority of assertions are Assert.AreEqual, showing low assertion type diversity"
      - "Noted the absence of exception assertions — no tests verify error handling behavior (e.g., invalid card, negative amounts)"
      - "Identified GetReceipt test as having only a trivial Assert.IsNotNull without checking the receipt's content"
      - "Noted that no tests verify collection contents in GetTransactionHistory — only the count is checked"
      - "Recommended specific additional assertion types that would improve coverage (exception, collection, structural)"
      - "Provided concrete examples of assertions that could be added to specific test methods"
      - "Reported assertion metrics that are computed correctly — the per-category counts add up and are internally consistent with the total number of assertions"
    timeout: 120

  # ==========================================================================
  # Scenario 2: Assertion-free smoke tests
  # ==========================================================================

  - name: "Flag assertion-free tests and trivial-only assertions"
    prompt: |
      Our QA team says these API smoke tests give us good coverage but I'm
      suspicious. Analyze the assertion quality and depth in
      `SmokeTests/ApiEndpointSmokeTests.cs` — specifically identify
      assertion-free tests (tests that call methods without verifying
      results) and tests with only trivial assertions like
      `Assert.IsNotNull`. Report which tests are assertion-free, the overall
      assertion quality, and concrete recommendations. Do not edit the
      files.
    setup:
      files:
        - path: "SmokeTests/SmokeTests.csproj"
          source: "fixtures/assertion-free/SmokeTests/SmokeTests.csproj"
        - path: "SmokeTests/ApiEndpointSmokeTests.cs"
          source: "fixtures/assertion-free/SmokeTests/ApiEndpointSmokeTests.cs"
    assertions:
      - type: "output_matches"
        pattern: "(no assertion|zero assertion|assertion.free|without.*assert|missing.*assert|don.*assert|lack.*assert)"
      - type: "output_matches"
        pattern: "(5|five|62%|most|majority|several)"
      - type: "exit_success"
    rubric:
      - "Identified that 5 out of 8 tests have zero assertions — they only call methods without verifying results"
      - "Identified that the remaining 3 tests only use Assert.IsNotNull, which is a trivial assertion that doesn't verify content"
      - "Explained that these tests give false security — code coverage may look good but correctness is not verified"
      - "Recommended specific meaningful assertions for at least some of the assertion-free tests"
      - "Distinguished between intentional smoke tests (acceptable if labeled as such) and tests masquerading as real verification"
    timeout: 300

  # ==========================================================================
  # Scenario 3: Good assertion diversity — acknowledge well-tested code
  # ==========================================================================

  - name: "Recognize well-diversified assertion usage"
    prompt: |
      Can you evaluate the assertion quality in my UserService.Tests project?
      I want to make sure my tests are thorough enough before we ship.
    setup:
      files:
        - path: "UserService.Tests/UserService.Tests.csproj"
          source: "fixtures/good-diversity/UserService.Tests/UserService.Tests.csproj"
        - path: "UserService.Tests/UserManagerTests.cs"
          source: "fixtures/good-diversity/UserService.Tests/UserManagerTests.cs"
    assertions:
      - type: "output_matches"
        pattern: "(good|strong|solid|diverse|well|thorough|comprehensive|varied)"
      - type: "exit_success"
    rubric:
      - "Recognized that the test suite uses a variety of assertion types including equality, boolean, null, exception, and collection assertions"
      - "Noted positive patterns: tests check multiple properties on returned objects, not just one field"
      - "Acknowledged the negative assertions (Assert.IsNull for non-existent user, Assert.IsFalse for failed delete, Assert.AreNotEqual for role change)"
      - "Acknowledged the exception assertions with specific exception types and parameter name verification"
      - "If any gaps were identified, they were presented as minor enhancement opportunities rather than problems"
      - "Because the suite already has good diversity, the report explicitly acknowledges this strength instead of inventing problems"
    timeout: 300

  # ==========================================================================
  # Scenario 4: Self-referential assertions — tautological round-trip tests
  # ==========================================================================

  - name: "Identify self-referential assertions in identity and round-trip tests"
    prompt: |
      I want an assertion quality analysis of my Config tests. They all pass
      but I'm not sure they'd catch real bugs. Analyze the assertion quality
      and depth in `ConfigService.Tests/ConfigTests.cs` — specifically
      identify self-referential or tautological assertions where the test
      asserts the output equals the input on identity / round-trip
      operations. Report assertion quality metrics (how many tests are
      self-referential, which ones), explain why each flagged test fails to
      verify real behavior, and recommend concrete improvements. Do not
      edit the files — this is a read-only assertion quality review.
    setup:
      files:
        - path: "ConfigService.Tests/ConfigService.Tests.csproj"
          source: "fixtures/self-referential/ConfigService.Tests/ConfigService.Tests.csproj"
        - path: "ConfigService.Tests/ConfigTests.cs"
          source: "fixtures/self-referential/ConfigService.Tests/ConfigTests.cs"
    assertions:
      - type: "output_matches"
        pattern: "(self.referent|tautolog|identity|input.*equals.*output|circular|same.*value|trivial.*round.?trip|round.?trip|set.*get|write.*read|proves nothing|not.*(testing|verif)|meaningless)"
      - type: "output_matches"
        pattern: "(GetTimeout|GetRetries|FormatConnectionString|ValidateKey)"
      - type: "exit_success"
    rubric:
      - "Identified that multiple tests assert the output equals the input — a self-referential pattern that cannot catch transformation bugs"
      - "Flagged GetTimeout_Default_ReturnsDefault and GetRetries_Default_ReturnsDefault as tautological — they assert that reading back a just-written value returns the same value without testing the default/fallback behavior the names suggest"
      - "Flagged FormatConnectionString_RoundTrip as only proving round-trip identity without verifying the formatted string's actual structure or content"
      - "Flagged ValidateKey_Valid_ReturnsSameKey as not exercising the validation logic — it only proves valid input passes through unchanged"
      - "Noted that Serialize_Deserialize round-trip is potentially valid but should be accompanied by tests with edge-case inputs (special characters, nulls, empty strings)"
      - "Recommended concrete improvements: test actual default values, test invalid keys, test formatted string structure, test edge-case serialization inputs"
    reject_tools: ["bash", "edit", "create"]
    timeout: 300

  # ==========================================================================
  # Scenario 5: Non-activation — user wants to run tests
  # ==========================================================================

  - name: "Decline request to write new tests from scratch"
    prompt: |
      I need to write unit tests for my InventoryTracker class.
      It handles stock levels, reorder alerts, and warehouse transfers.
      Can you help me write a full MSTest test suite from scratch?
    expect_activation: false
    setup:
      files:
        - path: "InventoryTracker.cs"
          content: |
            namespace Warehouse;

            public sealed class InventoryTracker
            {
                public int GetStockLevel(string sku) => throw new NotImplementedException();
                public void AddStock(string sku, int quantity) => throw new NotImplementedException();
                public bool NeedsReorder(string sku) => throw new NotImplementedException();
            }
    assertions:
      - type: "output_matches"
        pattern: "(TestMethod|TestClass|\\[Fact\\]|test)"
    rubric:
      - "Wrote test methods for the InventoryTracker class"
      - "Covered multiple methods of the class"
    timeout: 360

  # ==========================================================================
  # Scenario: Polyglot — shallow assertions in a Jest/TypeScript suite
  # ==========================================================================

  - name: "Polyglot: evaluate shallow assertions in a Jest/TypeScript OrderService suite"
    prompt: |
      Can you audit the assertion quality and depth of my Jest/TypeScript
      test suite? The tests are in `tests/order.test.ts` and the production
      code is in `src/order.ts`. Several tests look like they pass without
      really verifying much — shallow or always-true checks — and I'm not
      sure the async ones actually await their assertions. Please flag the
      weak spots and call out the tests that ARE genuinely well-targeted.
      Read-only review — don't edit the files.
    setup:
      files:
        - path: "package.json"
          source: "fixtures/jest-shallow/package.json"
        - path: "src/order.ts"
          source: "fixtures/jest-shallow/src/order.ts"
        - path: "tests/order.test.ts"
          source: "fixtures/jest-shallow/tests/order.test.ts"
    assertions:
      - type: "output_matches"
        pattern: "(shallow|trivial|toBeDefined|toBeTruthy|not\\.toBeNull|weak|low.*depth)"
      - type: "output_matches"
        pattern: "(always.*true|true.*true|tautolog)"
      - type: "output_matches"
        pattern: "(await|missing.*await|\\.resolves|async|silently.*pass)"
      - type: "output_matches"
        pattern: "(identity|same|self.compar|toBe\\(.*\\.id\\)|o\\.id.*o\\.id)"
      - type: "exit_success"
    rubric:
      - "Identified Jest/TS-specific patterns correctly (did NOT mention MSTest/xUnit APIs as if present)"
      - "Flagged the cluster of toBeDefined / toBeTruthy / not.toBeNull / Array.isArray assertions as trivial / low-depth"
      - "Flagged the `expect(true).toBe(true)` always-true assertion"
      - "Flagged the missing `await` on `expect(Promise.resolve(...)).resolves.toBeDefined()` as a silent-pass smell that lets rejections slip through"
      - "Flagged `expect(o.id).toBe(o.id)` as an identity / self-comparing assertion that proves nothing"
      - "Flagged `expect(o.id).not.toBe('totally unrelated string')` as a vacuous negative assertion"
      - "Acknowledged 'legitimate equality on returned customer' (toBe('alice')) as a well-targeted assertion"
      - "Acknowledged 'legitimate exception assertion' (toThrow('customer is required')) as a well-targeted assertion"
      - "Recommended concrete higher-value assertions: verifying status='pending', itemCount, totalCents, id format, list() contents after multiple creates"
    reject_tools: ["bash", "edit", "create"]
    timeout: 180