dotnet__skills/tests/dotnet-test/grade-tests/eval.vally.yaml

name: grade-tests
description: Evaluates the dotnet-test/grade-tests skill
type: capability
stimuli:
  - name: Grade a curated list of MSTest tests with mixed quality
    prompt: |
      Please grade each of the following test methods individually for
      test quality and produce a compact per-test table (one row per
      test) plus a short summary. They live in
      `OrderService.Tests/OrderProcessorTests.cs`. Do not modify any
      files.

      Test methods to grade:
        - OrderService.Tests.OrderProcessorTests.PlaceOrder_ValidItems_AssignsOrderIdAndPersistsOrder
        - OrderService.Tests.OrderProcessorTests.PlaceOrder_EmptyItems_ThrowsArgumentException
        - OrderService.Tests.OrderProcessorTests.GetOrderById_ExistingId_ReturnsOrder
        - OrderService.Tests.OrderProcessorTests.CancelOrder_ExistingOrder_Works
        - OrderService.Tests.OrderProcessorTests.SerializeOrderId_RoundTrip_ReturnsSameId
        - OrderService.Tests.OrderProcessorTests.PlaceOrder_LongRunning_CompletesEventually
        - OrderService.Tests.OrderProcessorTests.Test1
    environment:
      files:
        - src: ./fixtures/mixed-quality/OrderService.Tests/OrderService.Tests.csproj
          dest: OrderService.Tests/OrderService.Tests.csproj
        - src: ./fixtures/mixed-quality/OrderService.Tests/OrderProcessorTests.cs
          dest: OrderService.Tests/OrderProcessorTests.cs
    graders:
      - type: output-matches
        config:
          pattern: (\|\s*Test\s*\|\s*Grade\s*\|)
      - type: output-matches
        config:
          pattern: (PlaceOrder_ValidItems_AssignsOrderIdAndPersistsOrder.*\|\s*A\s*\|)
      - type: output-matches
        config:
          pattern: (PlaceOrder_EmptyItems_ThrowsArgumentException.*\|\s*A\s*\|)
      - type: output-matches
        config:
          pattern: (CancelOrder_ExistingOrder_Works.*\|\s*F\s*\|)
      - type: output-matches
        config:
          pattern: (Test1.*\|\s*F\s*\|)
      - type: exit-success
      - type: prompt
    rubric:
      - Produced a markdown table with columns Test, Grade, Band (or Score Band), and Notes
      - Graded `PlaceOrder_ValidItems_AssignsOrderIdAndPersistsOrder` as A (band 90–100) and acknowledged its AAA structure plus multi-faceted assertions
      - Graded `PlaceOrder_EmptyItems_ThrowsArgumentException` as A (band 90–100) and did NOT penalize it for having few assertions — exception tests are complete on their own
      - Graded `GetOrderById_ExistingId_ReturnsOrder` as C (band 70–79) because the only assertion is a trivial IsNotNull on the returned order
      - Graded `CancelOrder_ExistingOrder_Works` as F (band 0–59) because it has no assertions at all
      - Graded `SerializeOrderId_RoundTrip_ReturnsSameId` as D (band 60–69) and called out the self-referential / tautological round-trip assertion
      - Graded `PlaceOrder_LongRunning_CompletesEventually` as D (band 60–69) and called out the Thread.Sleep flakiness anti-pattern
      - Graded `Test1` as F (band 0–59) and called out the poor name, magic values, and swallowed exception
      - Each grade includes a one-line note explaining the most important reason for that grade
      - The summary leads with the highest-leverage observation (e.g., several tests give false confidence) rather than recapping the table row-by-row
    constraints:
      reject_tools:
        - edit
        - create
  - name: Grade pytest test methods using the same rubric
    prompt: |
      Please grade each of the following pytest test functions
      individually for test quality and produce a compact per-test
      table (one row per test) plus a short summary. They live in
      `tests/test_shopping_cart.py`. Do not modify any files.

      Test methods to grade:
        - test_calculate_total_with_discount_applies_discount_and_returns_rounded_amount
        - test_calculate_total_with_negative_discount_raises_value_error
        - test_get_cart_returns_cart
        - test_clear_cart_works
        - test_async_checkout_completes
    environment:
      files:
        - src: ./fixtures/python-pytest/test_shopping_cart.py
          dest: tests/test_shopping_cart.py
    graders:
      - type: output-matches
        config:
          pattern: (\|\s*Test\s*\|\s*Grade\s*\|)
      - type: output-matches
        config:
          pattern: (test_calculate_total_with_discount_applies_discount_and_returns_rounded_amount.*\|\s*A\s*\|)
      - type: output-matches
        config:
          pattern: (test_calculate_total_with_negative_discount_raises_value_error.*\|\s*A\s*\|)
      - type: output-matches
        config:
          pattern: (test_clear_cart_works.*\|\s*F\s*\|)
      - type: output-matches
        config:
          pattern: (test_async_checkout_completes.*\|\s*F\s*\|)
      - type: exit-success
      - type: prompt
    rubric:
      - Detected pytest as the framework and loaded the python language extension before grading
      - Graded `test_calculate_total_with_discount_applies_discount_and_returns_rounded_amount` as A — recognized the multi-assertion AAA structure with equality + state checks
      - Graded `test_calculate_total_with_negative_discount_raises_value_error` as A — recognized that `pytest.raises(ValueError, match=...)` is a complete exception assertion
      - Graded `test_get_cart_returns_cart` as C — trivial `assert ... is not None` only
      - Graded `test_clear_cart_works` as F — no assertions
      - Graded `test_async_checkout_completes` as F — combines `time.sleep` flakiness with an always-true `assert True`
      - Did NOT flag `pytest.raises(ValueError, match="discount must be non-negative")` as a broad exception assertion — the match argument constrains it
      - Did NOT flag the bare `assert` form as missing-framework — pytest bare `assert` is canonical
    constraints:
      reject_tools:
        - edit
        - create
  - name: Decline to grade the entire workspace when no test list is provided
    prompt: |
      Grade my tests.
    environment:
      files:
        - src: ./fixtures/no-list-provided/Some.Tests/Some.Tests.csproj
          dest: Some.Tests/Some.Tests.csproj
        - src: ./fixtures/no-list-provided/Some.Tests/AccountTests.cs
          dest: Some.Tests/AccountTests.cs
        - src: ./fixtures/no-list-provided/Some.Tests/InvoiceTests.cs
          dest: Some.Tests/InvoiceTests.cs
        - src: ./fixtures/no-list-provided/Some.Tests/ReportingTests.cs
          dest: Some.Tests/ReportingTests.cs
    graders:
      - type: output-matches
        config:
          pattern: (specific|explicit|list|which tests|scope|provide|name|specify|curated)
      - type: exit-success
      - type: prompt
    rubric:
      - Recognized that no specific list of test methods was provided across the three test files (AccountTests, InvoiceTests, ReportingTests)
      - Asked the user to provide an explicit list of test methods, file paths, or a diff hunk to grade
      - Did NOT silently scan and grade all ~21 tests across the workspace
      - Optionally pointed the user at `test-quality-auditor` agent or `test-anti-patterns` skill for full-suite analysis
    constraints:
      reject_tools:
        - edit
        - create
  - name: Grade Go table-driven tests without misreading the loop as branching
    prompt: |
      Please grade each of the following Go test functions individually for
      test quality and produce a compact per-test table (one row per test)
      plus a short summary. They live in `calculator_test.go` and the code
      under test is in `calculator.go`. Do not modify any files.

      Test functions to grade:
        - TestAdd_TableDriven
        - TestDivide_ByZero
        - TestParse_NoError
        - TestReset_NoAssertions
    environment:
      files:
        - src: ./fixtures/go-table-driven/go.mod
          dest: go.mod
        - src: ./fixtures/go-table-driven/calculator.go
          dest: calculator.go
        - src: ./fixtures/go-table-driven/calculator_test.go
          dest: calculator_test.go
    graders:
      - type: output-matches
        config:
          pattern: (\|\s*Test\s*\|\s*Grade\s*\|)
      - type: output-matches
        config:
          pattern: (TestAdd_TableDriven.*\|\s*A\s*\|)
      - type: output-matches
        config:
          pattern: (TestDivide_ByZero.*\|\s*A\s*\|)
      - type: output-matches
        config:
          pattern: (TestParse_NoError.*\|\s*C\s*\|)
      - type: output-matches
        config:
          pattern: (TestReset_NoAssertions.*\|\s*F\s*\|)
      - type: exit-success
      - type: prompt
    rubric:
      - Detected Go and its standard `testing` package and loaded the Go language extension before grading
      - Graded `TestAdd_TableDriven` as A — recognized the idiomatic table-driven subtests driven by `t.Run`
      - Did NOT flag the Go table-driven `for ... range` / `if got != tt.want` loop as conditional logic or branching — it is the idiomatic assertion pattern and incurs no deduction
      - Graded `TestDivide_ByZero` as A — the returned error is checked, which is a complete assertion of the error path
      - Graded `TestParse_NoError` as C — only checks that no error came back and never verifies the parsed value (trivial assertion)
      - Graded `TestReset_NoAssertions` as F — calls the function but never asserts via `t.Error`/`t.Fatal`
      - Justified every grade with at least one observable signal from the captured test body rather than a speculative or hypothetical deduction
      - Did NOT inflate deductions to justify a lower grade — started each test at A and deducted only for observable issues
    constraints:
      reject_tools:
        - edit
        - create
  - name: Grade tests when the production code under test is unavailable
    prompt: |
      Please grade each of the following test methods individually for test
      quality and produce a compact per-test table (one row per test) plus a
      short summary that we can post as a PR comment. They live in
      `Payments.Tests/PaymentGatewayTests.cs`. The production project
      `Payments.Core` is not in this workspace, so its source is unavailable.
      Do not modify any files.

      Test methods to grade:
        - Payments.Tests.PaymentGatewayTests.Charge_ValidCard_ReturnsApprovedResult
        - Payments.Tests.PaymentGatewayTests.Charge_NegativeAmount_ThrowsArgumentOutOfRange
        - Payments.Tests.PaymentGatewayTests.Refund_ExistingCharge_ReturnsReceipt
        - Payments.Tests.PaymentGatewayTests.Settle_PendingBatch_Runs
    environment:
      files:
        - src: ./fixtures/production-unavailable/Payments.Tests/Payments.Tests.csproj
          dest: Payments.Tests/Payments.Tests.csproj
        - src: ./fixtures/production-unavailable/Payments.Tests/PaymentGatewayTests.cs
          dest: Payments.Tests/PaymentGatewayTests.cs
    graders:
      - type: output-matches
        config:
          pattern: (\|\s*Test\s*\|\s*Grade\s*\|)
      - type: output-matches
        config:
          pattern: (Charge_ValidCard_ReturnsApprovedResult.*\|\s*A\s*\|)
      - type: output-matches
        config:
          pattern: (Charge_NegativeAmount_ThrowsArgumentOutOfRange.*\|\s*A\s*\|)
      - type: output-matches
        config:
          pattern: (Refund_ExistingCharge_ReturnsReceipt.*\|\s*C\s*\|)
      - type: output-matches
        config:
          pattern: (Settle_PendingBatch_Runs.*\|\s*F\s*\|)
      - type: output-matches
        config:
          pattern: (?i)(unverified)
      - type: exit-success
      - type: prompt
    rubric:
      - Did NOT penalize the tests because the production code under test (`Payments.Core`) is unavailable — marked behavioral concerns about uncovered behaviors as `Unverified` instead of deducting
      - Graded `Charge_ValidCard_ReturnsApprovedResult` as A on the observable signal of its equality assertions, without inventing weaknesses that would need the unavailable production source to judge
      - Graded `Charge_NegativeAmount_ThrowsArgumentOutOfRange` as A — the exception assertion is complete on its own
      - Graded `Refund_ExistingCharge_ReturnsReceipt` as C — only a trivial `IsNotNull` on the receipt
      - Graded `Settle_PendingBatch_Runs` as F — no assertions at all
      - Kept the report compact and PR-comment-friendly — did not spill a giant (e.g. 500-row) table into the PR comment, and noted that any overflow would collapse into a `<details>` block per the row cap
    constraints:
      reject_tools:
        - edit
        - create