dotnet__skills/tests/dotnet-test/dotnet-test-frameworks/eval.vally.yaml

name: dotnet-test-frameworks
description: Evaluates the dotnet-test/dotnet-test-frameworks skill
type: capability
config:
  timeout: 2m
stimuli:
  - name: Cross-framework assertion equivalence mapping
    expect_activation: false
    prompt: |
      I have these MSTest assertions in my test file and I need to know their equivalents in both xUnit and NUnit. Please provide a complete mapping for all six:

      ```csharp
      Assert.AreEqual(expected, actual);
      Assert.IsTrue(condition);
      Assert.IsNull(obj);
      Assert.IsInstanceOfType(obj, typeof(MyClass));
      CollectionAssert.Contains(list, item);
      StringAssert.Contains(str, "substring");
      ```
    graders:
      - type: output-contains
        config:
          substring: Assert.Equal
      - type: output-matches
        config:
          pattern: Assert\.True|Assert\.False
      - type: output-matches
        config:
          pattern: Assert\.Null|Assert\.NotNull
      - type: output-matches
        config:
          pattern: Assert\.IsAssignableFrom|Assert\.IsType
      - type: output-matches
        config:
          pattern: Assert\.Contains
      - type: output-matches
        config:
          pattern: Assert\.That\(.*Is\.EqualTo
      - type: output-matches
        config:
          pattern: Is\.True|Is\.False
      - type: output-matches
        config:
          pattern: Is\.Null
      - type: output-matches
        config:
          pattern: Has\.Member|Does\.Contain
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - Provided correct xUnit equivalents for all six MSTest assertions
      - Provided correct NUnit equivalents using the constraint model (Assert.That with Is/Has/Does syntax)
      - Covered collection assertion equivalents for both target frameworks
      - Covered string assertion equivalents for both target frameworks

  - name: Identify TUnit framework and its unique attributes
    expect_activation: false
    prompt: |
      I inherited a project with this test file and I'm not sure what testing framework it uses.
      Can you identify it and explain the key attributes — in particular whether the test class
      needs any marker attribute, and what `[ClassDataSource<T>]` actually does?

      ```csharp
      using TUnit.Core;

      [ClassDataSource<DatabaseFixture>]
      public class OrderTests
      {
          private readonly DatabaseFixture _db;

          public OrderTests(DatabaseFixture db)
          {
              _db = db;
          }

          [Test]
          public async Task CreateOrder_WithValidItems_Succeeds()
          {
              var order = new Order { Items = new[] { "Widget" } };
              var result = await _db.OrderService.CreateAsync(order);
              await Assert.That(result.Id).IsNotNull();
          }

          [Test]
          [Skip("Waiting for payment gateway sandbox")]
          public async Task ProcessPayment_ChargesCorrectAmount()
          {
              // TODO: implement
          }
      }
      ```
    graders:
      - type: output-matches
        config:
          pattern: "[Tt][Uu]nit"
      - type: output-matches
        config:
          pattern: ClassDataSource
      - type: output-matches
        config:
          pattern: \[Skip
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - Correctly identified the framework as TUnit
      - Clarified that TUnit test classes do NOT require a class-level marker attribute (convention-based, like xUnit) — `[ClassDataSource<T>]` is a fixture / data source, not the class marker
      - Explained that `[ClassDataSource<DatabaseFixture>]` injects a shared fixture instance into the class constructor
      - Identified `[Test]` as the test method marker
      - Noted the [Skip] attribute with required reason string as TUnit's mechanism for skipping tests
      - Mentioned that TUnit assertions are async and must be awaited (e.g. `await Assert.That(...).IsNotNull()`)

  - name: Replace try-catch with framework-native exception assertions
    expect_activation: false
    prompt: |
      My team lead says these tests should use proper exception assertions instead of try/catch.
      Can you refactor each one using its framework's native approach?

      MSTest test:
      ```csharp
      [TestMethod]
      public void ProcessOrder_EmptyOrder_ThrowsException()
      {
          try
          {
              var processor = new OrderProcessor();
              processor.ProcessOrder(new Order());
              Assert.Fail("Expected exception was not thrown");
          }
          catch (InvalidOperationException ex)
          {
              Assert.AreEqual("Order must contain at least one item", ex.Message);
          }
      }
      ```

      xUnit test:
      ```csharp
      [Fact]
      public void ProcessOrder_EmptyOrder_ThrowsException()
      {
          try
          {
              var processor = new OrderProcessor();
              processor.ProcessOrder(new Order());
              Assert.True(false, "Expected exception was not thrown");
          }
          catch (InvalidOperationException ex)
          {
              Assert.Equal("Order must contain at least one item", ex.Message);
          }
      }
      ```

      NUnit test:
      ```csharp
      [Test]
      public void ProcessOrder_EmptyOrder_ThrowsException()
      {
          try
          {
              var processor = new OrderProcessor();
              processor.ProcessOrder(new Order());
              Assert.Fail("Expected exception was not thrown");
          }
          catch (InvalidOperationException ex)
          {
              Assert.That(ex.Message, Is.EqualTo("Order must contain at least one item"));
          }
      }
      ```
    graders:
      - type: output-matches
        config:
          pattern: \[TestMethod\].*Assert\.Throws(Exactly)?<|Assert\.Throws(Exactly)?<.*AreEqual
      - type: output-matches
        config:
          pattern: \[Fact\].*Assert\.Throws<|Assert\.Throws<.*Assert\.Equal
      - type: output-matches
        config:
          pattern: Throws\.TypeOf<InvalidOperationException>|Assert\.That\(.*Throws
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - Replaced the MSTest try/catch with Assert.ThrowsExactly<T> or Assert.Throws<T>
      - Replaced the xUnit try/catch with Assert.Throws<T>
      - Replaced the NUnit try/catch with the constraint-model pattern or Assert.Throws<T>
      - Preserved exception message verification in all three refactored versions

  - name: Skip annotations across all four frameworks
    expect_activation: false
    prompt: |
      I maintain a solution with test projects using MSTest, xUnit, NUnit, and TUnit.
      I need to temporarily skip some tests in each project with a reason message explaining why.
      What's the correct attribute to use in each framework?
    graders:
      - type: output-matches
        config:
          pattern: MSTest
      - type: output-matches
        config:
          pattern: xUnit
      - type: output-matches
        config:
          pattern: NUnit
      - type: output-matches
        config:
          pattern: TUnit
      - type: output-matches
        config:
          pattern: \[Ignore\(
      - type: output-matches
        config:
          pattern: Skip\s*=\s*"
      - type: output-matches
        config:
          pattern: \[Skip\(
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - Provided the correct skip attribute for MSTest ([Ignore] with optional reason)
      - Provided the correct skip mechanism for xUnit (Skip property on [Fact] or [Theory])
      - Provided the correct skip attribute for NUnit ([Ignore] with required reason)
      - Provided the correct skip attribute for TUnit ([Skip] with required reason)

  - name: Convert NUnit lifecycle methods to xUnit equivalents
    expect_activation: false
    prompt: |
      I'm migrating this NUnit test class to xUnit. How should I convert the setup and teardown methods?

      ```csharp
      [TestFixture]
      public class CustomerRepositoryTests
      {
          private DbContext _context;
          private CustomerRepository _repo;

          [OneTimeSetUp]
          public void ClassSetup()
          {
              Database.Initialize();
          }

          [OneTimeTearDown]
          public void ClassTeardown()
          {
              Database.Cleanup();
          }

          [SetUp]
          public void TestSetup()
          {
              _context = new TestDbContext();
              _repo = new CustomerRepository(_context);
          }

          [TearDown]
          public void TestTeardown()
          {
              _context.Dispose();
          }

          [Test]
          public void FindById_ExistingCustomer_ReturnsCustomer()
          {
              var customer = _repo.FindById(1);
              Assert.That(customer, Is.Not.Null);
              Assert.That(customer.Name, Is.EqualTo("Alice"));
          }
      }
      ```
    graders:
      - type: output-matches
        config:
          pattern: IClassFixture
      - type: output-matches
        config:
          pattern: IDisposable|Dispose
      - type: output-matches
        config:
          pattern: \[Fact\]
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - Converted per-test [SetUp]/[TearDown] to constructor and IDisposable.Dispose
      - Converted class-level [OneTimeSetUp]/[OneTimeTearDown] to an IClassFixture<T> with the fixture's constructor and
        Dispose
      - Removed [TestFixture] and converted [Test] to [Fact]

  - name: Identify integration tests by markers and code patterns
    expect_activation: false
    prompt: |
      I have a test solution with hundreds of tests. I need to separate integration tests from
      unit tests for CI pipeline optimization. Here are samples from three projects — which ones
      are integration tests and what indicators tell you that?

      Project A (MSTest):
      ```csharp
      [TestClass]
      [TestCategory("Integration")]
      public class PaymentGatewayTests
      {
          [TestMethod]
          public async Task ChargeCard_ValidCard_ReturnsSuccess()
          {
              var client = new HttpClient();
              var gateway = new PaymentGateway(client);
              var result = await gateway.ChargeAsync("4111111111111111", 99.99m);
              Assert.IsTrue(result.Success);
          }
      }
      ```

      Project B (xUnit):
      ```csharp
      public class OrderApiTests
      {
          [Fact]
          [Trait("Category", "Integration")]
          public async Task CreateOrder_ReturnsCreatedStatus()
          {
              using var factory = new WebApplicationFactory<Program>();
              var client = factory.CreateClient();
              var response = await client.PostAsJsonAsync("/orders", new { Item = "Widget" });
              Assert.Equal(HttpStatusCode.Created, response.StatusCode);
          }
      }
      ```

      Project C (NUnit):
      ```csharp
      [TestFixture]
      public class DatabaseQueryTests
      {
          [Test]
          public void GetCustomer_ById_ReturnsExpectedName()
          {
              using var conn = new SqlConnection("Server=localhost;...");
              conn.Open();
              var name = conn.QuerySingle<string>("SELECT Name FROM Customers WHERE Id = 1");
              Assert.That(name, Is.EqualTo("Alice"));
          }
      }
      ```

      Project D (TUnit):
      ```csharp
      using TUnit.Core;

      public class InventoryServiceTests
      {
          [Test]
          public async Task ReserveStock_AgainstLiveWarehouse_ReducesAvailableQuantity()
          {
              using var conn = new SqlConnection("Server=warehouse-db;...");
              await conn.OpenAsync();
              var service = new InventoryService(conn);
              var ok = await service.ReserveAsync(sku: "WIDGET-1", quantity: 5);
              await Assert.That(ok).IsTrue();
          }
      }
      ```
    graders:
      - type: output-matches
        config:
          pattern: TestCategory.*Integration|\[TestCategory\("Integration"\)\]
      - type: output-matches
        config:
          pattern: Trait.*Category.*Integration|\[Trait\(
      - type: output-matches
        config:
          pattern: SqlConnection|database
      - type: output-matches
        config:
          pattern: \[Category\("Integration"\)\]|Category.*Integration.*NUnit|NUnit.*Category
      - type: output-matches
        config:
          pattern: "[Tt][Uu]nit"
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - Identified Project A as integration test via the [TestCategory("Integration")] marker and direct HttpClient usage
      - Identified Project B as integration test via the [Trait("Category", "Integration")] marker
      - Identified Project C as integration test based on direct database access (SqlConnection), even without an explicit
        category marker
      - Identified Project D as an integration test based on direct database access against a live warehouse server
      - Recommended adding a framework-appropriate integration category marker to Project C (NUnit `[Category("Integration")]`)
        and Project D (TUnit `[Category("Integration")]`)

  - name: Convert cross-framework assertions to TUnit syntax
    expect_activation: false
    prompt: |
      I'm porting a small test class from MSTest to TUnit. Can you rewrite each of these
      assertions in the TUnit equivalent? Please show me the exact line each one becomes.

      ```csharp
      Assert.AreEqual(expected, actual);
      Assert.IsTrue(order.IsValid);
      Assert.IsNull(result.Error);
      Assert.IsInstanceOfType(handler, typeof(IOrderHandler));
      CollectionAssert.Contains(processedIds, 42);
      var ex = Assert.ThrowsException<InvalidOperationException>(() => processor.Run());
      Assert.AreEqual("Order is empty", ex.Message);
      ```
    graders:
      - type: output-matches
        config:
          pattern: await\s+Assert\.That
      - type: output-matches
        config:
          pattern: \.IsEqualTo\(
      - type: output-matches
        config:
          pattern: \.IsTrue\(\)
      - type: output-matches
        config:
          pattern: \.IsNull\(\)
      - type: output-matches
        config:
          pattern: \.IsAssignableTo<|\.IsTypeOf<
      - type: output-matches
        config:
          pattern: \.Contains\(
      - type: output-matches
        config:
          pattern: \.Throws(Exactly)?<InvalidOperationException>
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - Used `await Assert.That(...)` for every converted assertion (TUnit assertions are async and must be awaited)
      - Mapped equality, boolean, null, type, and collection-contains assertions to the correct TUnit fluent member
        (IsEqualTo / IsTrue / IsNull / IsAssignableTo or IsTypeOf / Contains)
      - Converted the exception assertion to `await Assert.That(() => ...).Throws<T>()` (or `ThrowsExactly<T>()`) and
        preserved the message check (e.g. via `.WithMessage(...)` or a follow-up awaited assertion on the exception)
      - Did not silently drop the `await` on any assertion or leave NUnit/xUnit-style `Assert.Throws<T>` syntax in the
        TUnit output

  - name: Diagnose silently-passing TUnit test with missing await
    expect_activation: false
    prompt: |
      This TUnit test is green in CI, but I'm certain `CalculateTotal` is returning the wrong
      value (I added a deliberate bug that returns 0 instead of the sum). Why is the test still
      passing, and how do I fix it?

      ```csharp
      using TUnit.Core;

      public class CartTests
      {
          [Test]
          public async Task CalculateTotal_TwoItems_ReturnsSum()
          {
              var cart = new Cart();
              cart.Add(new Item(price: 10m));
              cart.Add(new Item(price: 32m));

              var total = cart.CalculateTotal();

              Assert.That(total).IsEqualTo(42m);
          }
      }
      ```
    graders:
      - type: output-matches
        config:
          pattern: await
      - type: output-matches
        config:
          pattern: (never (run|execute|awaited)|not (run|awaited|executed)|silently|discard|fire[- ]and[- ]forget|unobserved)
      - type: output-matches
        config:
          pattern: await\s+Assert\.That\(total\)\.IsEqualTo
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - Correctly identified the missing `await` on the `Assert.That(...).IsEqualTo(...)` expression as the root cause
      - Explained that TUnit assertions are async and produce a task — without `await`, the assertion is never observed
        and the test passes regardless of the actual value
      - "Provided the corrected line: `await Assert.That(total).IsEqualTo(42m);`"
      - Optionally mentioned the built-in TUnit analyzer that warns when an assertion is not awaited, or suggested
        treating that analyzer warning as an error in CI
      - Did NOT misdiagnose the failure as a bug in `Cart.CalculateTotal`, a comparison-precision issue (decimal vs
        double), or a missing test discovery problem

  - name: Refactor TUnit try/catch to native exception assertion
    expect_activation: false
    prompt: |
      My team lead wants this TUnit test to use the framework's native exception assertion
      instead of try/catch, and to verify both the exception type and the message in one
      idiomatic expression. Please refactor it.

      ```csharp
      using TUnit.Core;

      public class OrderProcessorTests
      {
          [Test]
          public async Task ProcessOrder_EmptyOrder_ThrowsInvalidOperation()
          {
              var processor = new OrderProcessor();
              try
              {
                  processor.ProcessOrder(new Order());
                  Assert.Fail("Expected exception was not thrown");
              }
              catch (InvalidOperationException ex)
              {
                  await Assert.That(ex.Message).IsEqualTo("Order must contain at least one item");
              }
          }
      }
      ```
    graders:
      - type: output-matches
        config:
          pattern: await\s+Assert\.That\(
      - type: output-matches
        config:
          pattern: \.Throws(Exactly)?<InvalidOperationException>
      - type: output-matches
        config:
          pattern: (WithMessage|IsEqualTo).*Order must contain at least one item|Order must contain at least one item
      - type: output-not-matches
        config:
          pattern: try\s*\{[\s\S]*catch\s*\(InvalidOperationException
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - Removed the try/catch block entirely and replaced it with a single awaited TUnit exception assertion on the
        throwing delegate
      - Used `Throws<InvalidOperationException>()` (or `ThrowsExactly<InvalidOperationException>()`) to assert the
        exception type
      - Verified the message in an idiomatic way (e.g. `.WithMessage("Order must contain at least one item")` chained on
        the throw assertion, or an awaited follow-up assertion on the captured exception)
      - Kept the `await` on every assertion call in the refactored test

  - name: TUnit lifecycle hooks at test, class, assembly, and session scope
    expect_activation: false
    prompt: |
      In a TUnit test project, I need code that runs:

      1. Before every individual test in a class (e.g. reset a shared in-memory database).
      2. Once before the first test in a class and once after the last (e.g. open / dispose
         a class-scoped fixture).
      3. Once before any test in the assembly starts and once after they all finish
         (e.g. start / stop an in-process WireMock server for the whole assembly).
      4. Once at the very start of the whole test run and once at the very end
         (e.g. apply EF Core migrations and tear down the database for the entire session).

      What attributes / methods does TUnit use for each of these, and where do they go?
      A short C# sketch for each scope would be ideal.
    graders:
      - type: output-matches
        config:
          pattern: \[Before\(Test\)\]
      - type: output-matches
        config:
          pattern: \[After\(Test\)\]
      - type: output-matches
        config:
          pattern: \[Before\(Class\)\]
      - type: output-matches
        config:
          pattern: \[After\(Class\)\]
      - type: output-matches
        config:
          pattern: \[Before\(Assembly\)\]
      - type: output-matches
        config:
          pattern: \[After\(Assembly\)\]
      - type: output-matches
        config:
          pattern: \[Before\(TestSession\)\]
      - type: output-matches
        config:
          pattern: \[After\(TestSession\)\]
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - Provided per-test hooks using `[Before(Test)]` / `[After(Test)]` (or noted the constructor + IAsyncDisposable
        alternative) on instance methods of the test class
      - Provided per-class hooks using `[Before(Class)]` / `[After(Class)]` on static methods of the test class
      - Provided per-assembly hooks using `[Before(Assembly)]` / `[After(Assembly)]` on static methods, noting they
        apply to every test in the assembly
      - Provided per-session hooks using `[Before(TestSession)]` / `[After(TestSession)]` on static methods, noting they
        run exactly once across the whole test run
      - Did NOT confuse TUnit's scoped `[Before(...)]` / `[After(...)]` attributes with NUnit's
        `[SetUp]`/`[TearDown]`/`[OneTimeSetUp]`/`[OneTimeTearDown]` or xUnit's `IClassFixture<T>` / `IAsyncLifetime`
      - Optionally mentioned that hook methods may accept a context object (e.g. `TestContext`, `ClassHookContext`)
        and/or a `CancellationToken`, or that `[BeforeEvery(Test)]` / `[AfterEvery(Test)]` (and the `Class`/`Assembly`
        variants) run for every test/class/assembly across the run

  - name: TUnit skip mechanisms — attribute, assembly-wide, and dynamic
    expect_activation: false
    prompt: |
      In a TUnit test project I need three different ways to skip tests, each with a clear
      reason message:

      1. Skip one specific test method (it's waiting on a payment-gateway sandbox).
      2. Skip every test in an entire assembly when that assembly is built in a special
         "smoke" configuration.
      3. Inside a test method, decide at runtime to skip the test if the current machine
         is not joined to the corporate VPN.

      What does each one look like in TUnit?
    graders:
      - type: output-matches
        config:
          pattern: \[Skip\(
      - type: output-matches
        config:
          pattern: assembly\s*:\s*Skip
      - type: output-matches
        config:
          pattern: Skip\.Test\(
      - type: exit-success
      - type: prompt
      - type: pairwise
    rubric:
      - For the single-test case, used the `[Skip("reason")]` attribute on the test method with a required reason string
      - >-
        For the assembly-wide case, used an assembly-level attribute such as `[assembly: Skip("…")]`
        (or equivalent class-level `[Skip(...)]` on a base class) rather than annotating each test
        individually
      - For the runtime case, used the dynamic `Skip.Test("reason")` call inside the test method, after the VPN check,
        and explained that this is TUnit's equivalent of a runtime skip (distinct from a true "inconclusive" state)
      - Did NOT propose xUnit's `[Fact(Skip = "…")]`, MSTest's `Assert.Inconclusive()`, or NUnit's `Assume.That(...)` as
        TUnit answers