dotnet__skills/tests/dotnet-test/assertion-quality/eval.vally.yaml

name: assertion-quality
description: Evaluates the dotnet-test/assertion-quality skill
type: capability
config:
  timeout: 5m
stimuli:
  - name: Identify low assertion diversity in equality-dominated test suite
    prompt: |
      I want to understand whether my PaymentService tests are actually
      thorough. They all pass, but I'm not confident they'd catch real bugs.
      Can you analyze the assertion quality and variety?
    environment:
      files:
        - src: fixtures/low-diversity/PaymentService.Tests/PaymentService.Tests.csproj
          dest: PaymentService.Tests/PaymentService.Tests.csproj
        - src: fixtures/low-diversity/PaymentService.Tests/PaymentProcessorTests.cs
          dest: PaymentService.Tests/PaymentProcessorTests.cs
    graders:
      - type: output-matches
        config:
          pattern: (diversity|variety|spread|shallow|narrow|limited|homogeneous|repetitive)
      - type: output-matches
        config:
          pattern: (AreEqual|equality|equal)
      - type: output-matches
        config:
          pattern: (exception|throw|error|negative|missing)
      - type: exit-success
      - type: prompt
    rubric:
      - Identified that the vast majority of assertions are Assert.AreEqual, showing low assertion type diversity
      - Noted the absence of exception assertions — no tests verify error handling behavior (e.g., invalid card,
        negative amounts)
      - Identified GetReceipt test as having only a trivial Assert.IsNotNull without checking the receipt's content
      - Noted that no tests verify collection contents in GetTransactionHistory — only the count is checked
      - Recommended specific additional assertion types that would improve coverage (exception, collection, structural)
      - Provided concrete examples of assertions that could be added to specific test methods
  - name: Flag assertion-free tests and trivial-only assertions
    prompt: |
      Our QA team says these API smoke tests give us good coverage but I'm
      suspicious. Analyze the assertion quality and depth in
      `SmokeTests/ApiEndpointSmokeTests.cs` — specifically identify
      assertion-free tests (tests that call methods without verifying
      results) and tests with only trivial assertions like
      `Assert.IsNotNull`. Report which tests are assertion-free, the overall
      assertion quality, and concrete recommendations. Do not edit the
      files.
    environment:
      files:
        - src: fixtures/assertion-free/SmokeTests/SmokeTests.csproj
          dest: SmokeTests/SmokeTests.csproj
        - src: fixtures/assertion-free/SmokeTests/ApiEndpointSmokeTests.cs
          dest: SmokeTests/ApiEndpointSmokeTests.cs
    graders:
      - type: output-matches
        config:
          pattern: (no assertion|zero assertion|assertion.free|without.*assert|missing.*assert|don.*assert|lack.*assert)
      - type: output-matches
        config:
          pattern: (5|five|62%|most|majority|several)
      - type: exit-success
      - type: prompt
    rubric:
      - Identified that 5 out of 8 tests have zero assertions — they only call methods without verifying results
      - Identified that the remaining 3 tests only use Assert.IsNotNull, which is a trivial assertion that doesn't
        verify content
      - Explained that these tests give false security — code coverage may look good but correctness is not verified
      - Recommended specific meaningful assertions for at least some of the assertion-free tests
      - Distinguished between intentional smoke tests (acceptable if labeled as such) and tests masquerading as real
        verification
  - name: Recognize well-diversified assertion usage
    prompt: |
      Can you evaluate the assertion quality in my UserService.Tests project?
      I want to make sure my tests are thorough enough before we ship.
    environment:
      files:
        - src: fixtures/good-diversity/UserService.Tests/UserService.Tests.csproj
          dest: UserService.Tests/UserService.Tests.csproj
        - src: fixtures/good-diversity/UserService.Tests/UserManagerTests.cs
          dest: UserService.Tests/UserManagerTests.cs
    graders:
      - type: output-matches
        config:
          pattern: (good|strong|solid|diverse|well|thorough|comprehensive|varied)
      - type: exit-success
      - type: prompt
    rubric:
      - Recognized that the test suite uses a variety of assertion types including equality, boolean, null, exception,
        and collection assertions
      - "Noted positive patterns: tests check multiple properties on returned objects, not just one field"
      - Acknowledged the negative assertions (Assert.IsNull for non-existent user, Assert.IsFalse for failed delete,
        Assert.AreNotEqual for role change)
      - Acknowledged the exception assertions with specific exception types and parameter name verification
      - If any gaps were identified, they were presented as minor enhancement opportunities rather than problems
  - name: Identify self-referential assertions in identity and round-trip tests
    prompt: |
      I want an assertion quality analysis of my Config tests. They all pass
      but I'm not sure they'd catch real bugs. Analyze the assertion quality
      and depth in `ConfigService.Tests/ConfigTests.cs` — specifically
      identify self-referential or tautological assertions where the test
      asserts the output equals the input on identity / round-trip
      operations. Report assertion quality metrics (how many tests are
      self-referential, which ones), explain why each flagged test fails to
      verify real behavior, and recommend concrete improvements. Do not
      edit the files — this is a read-only assertion quality review.
    environment:
      files:
        - src: fixtures/self-referential/ConfigService.Tests/ConfigTests.cs
          dest: ConfigService.Tests/ConfigTests.cs
        - src: fixtures/self-referential/ConfigService.Tests/ConfigService.Tests.csproj
          dest: ConfigService.Tests/ConfigService.Tests.csproj
    graders:
      - type: output-matches
        config:
          pattern: (self.referent|tautolog|identity|input.*equals.*output|circular|same.*value|trivial.*round.?trip|round.?trip|set.*get|write.*read|proves
            nothing|not.*(testing|verif)|meaningless)
      - type: output-matches
        config:
          pattern: (GetTimeout|GetRetries|FormatConnectionString|ValidateKey)
      - type: exit-success
      - type: prompt
    rubric:
      - Identified that multiple tests assert the output equals the input — a self-referential pattern that cannot catch
        transformation bugs
      - Flagged GetTimeout_Default_ReturnsDefault and GetRetries_Default_ReturnsDefault as tautological — they assert
        that reading back a just-written value returns the same value without testing the default/fallback behavior the
        names suggest
      - Flagged FormatConnectionString_RoundTrip as only proving round-trip identity without verifying the formatted
        string's actual structure or content
      - Flagged ValidateKey_Valid_ReturnsSameKey as not exercising the validation logic — it only proves valid input
        passes through unchanged
      - Noted that Serialize_Deserialize round-trip is potentially valid but should be accompanied by tests with
        edge-case inputs (special characters, nulls, empty strings)
      - "Recommended concrete improvements: test actual default values, test invalid keys, test formatted string
        structure, test edge-case serialization inputs"
    constraints:
      reject_tools:
        - bash
        - edit
        - create
  - name: Decline request to write new tests from scratch
    prompt: |
      I need to write unit tests for my InventoryTracker class.
      It handles stock levels, reorder alerts, and warehouse transfers.
      Can you help me write a full MSTest test suite from scratch?
    environment:
      files:
        - src: ./fixtures/decline-request-to-write-new-tests-from-scratch/InventoryTracker.cs
          dest: InventoryTracker.cs
    graders:
      - type: output-matches
        config:
          pattern: (TestMethod|TestClass|\[Fact\]|test)
      - type: prompt
    rubric:
      - Wrote test methods for the InventoryTracker class
      - Covered multiple methods of the class
    constraints:
      reject_skills:
        - "*"