dotnet__skills/tests/dotnet-test/writing-mstest-tests/eval.yaml

scenarios:
  # ============================================================================
  # Goal 1: Writing new MSTest tests from scratch
  # ============================================================================

  - name: "Write unit tests for a service class"
    prompt: "I need to write MSTest unit tests for my OrderService class following MSTest best practices. Help me write comprehensive tests with sealed test classes and Assert.ThrowsExactly."
    setup:
      copy_test_files: true
    assertions:
      - type: output_matches
        pattern: "\\[TestClass\\]"
      - type: output_matches
        pattern: "\\[TestMethod\\]"
      - type: output_matches
        pattern: "\\bsealed\\b"
      - type: output_matches
        pattern: "Assert\\.ThrowsExactly|Assert\\.Throws"
    rubric:
      - "Uses [TestClass] and [TestMethod] attributes correctly"
      - "Seals the test class with the 'sealed' keyword"
      - "Follows Arrange-Act-Assert (AAA) pattern with comments or clear structure"
      - "Names tests descriptively using MethodName_Scenario_ExpectedBehavior or similar convention"
      - "Tests both success and failure paths (e.g., null input, empty items)"
      - "Uses Assert.ThrowsExactly or Assert.Throws for exception testing instead of [ExpectedException]"
      - "Uses Assert.AreEqual with expected as first argument and actual as second"
    timeout: 360

  - name: "Write data-driven tests for a calculator"
    prompt: "I have a PriceCalculator class with CalculateTotal and ApplyDiscount methods. Help me write data-driven MSTest tests using DataRow and DynamicData that cover multiple input combinations following modern MSTest best practices."
    setup:
      copy_test_files: true
    assertions:
      - type: output_matches
        pattern: "\\[DataRow\\]|\\[DynamicData\\]"
      - type: output_matches
        pattern: "\\bsealed\\b"
      - type: output_matches
        pattern: "Assert\\.ThrowsExactly|Assert\\.Throws"
    rubric:
      - "Uses [DataRow] for simple inline test data or [DynamicData] for complex data"
      - "If using DynamicData, prefers ValueTuple return type over IEnumerable<object[]>"
      - "Covers edge cases like zero values, boundary conditions, and invalid inputs"
      - "Seals the test class"
      - "Uses Assert.ThrowsExactly or Assert.Throws for testing invalid argument scenarios"
      - "Does not use manual foreach loops to iterate test data — uses proper data-driven attributes instead"
    timeout: 360

  - name: "Write async tests with cancellation"
    prompt: |
      I have this async service that I need to test:

      ```csharp
      public sealed class DataFetcher
      {
          private readonly HttpClient _client;
          public DataFetcher(HttpClient client) => _client = client;

          public async Task<string> FetchAsync(string url, CancellationToken cancellationToken = default)
          {
              var response = await _client.GetAsync(url, cancellationToken);
              response.EnsureSuccessStatusCode();
              return await response.Content.ReadAsStringAsync(cancellationToken);
          }
      }
      ```

      How should I write MSTest tests for this using modern MSTest patterns for async testing, timeouts, and cancellation token handling?
    setup:
      copy_test_files: true
    assertions:
      - type: output_matches
        pattern: "async Task"
      - type: output_matches
        pattern: "\\[Timeout\\]"
      - type: output_matches
        pattern: "CancellationToken"
    rubric:
      - "Uses async Task test methods (not async void)"
      - "Uses TestContext.CancellationToken or mentions its importance for cooperative timeout"
      - "Uses [Timeout] attribute for tests that could hang"
      - "Uses Assert.ThrowsExactlyAsync or Assert.ThrowsAsync for async exception testing"
      - "Does not use CancellationToken.None when TestContext.CancellationToken is available"
      - "Seals the test class"
    timeout: 120

  # ============================================================================
  # Goal 2: Fixing anti-patterns in existing tests
  # ============================================================================

  - name: "Fix swapped Assert.AreEqual arguments"
    prompt: |
      When these MSTest tests fail, the failure messages look backwards — the value the code actually produced is reported as "Expected", and the literal I wrote is reported as "Actual". Review these assertions and fix whatever is causing that. Don't touch anything that is already correct.

      ```csharp
      [TestClass]
      public class MathTests
      {
          [TestMethod]
          public void Add_TwoNumbers_ReturnsSum()
          {
              var result = Calculator.Add(2, 3);
              Assert.AreEqual(result, 5);
          }

          [TestMethod]
          public void Multiply_ByZero_ReturnsZero()
          {
              var result = Calculator.Multiply(42, 0);
              Assert.AreEqual(result, 0);
          }

          [TestMethod]
          public void Subtract_Numbers_ReturnsDifference()
          {
              var result = Calculator.Subtract(10, 4);
              Assert.AreEqual(6, result);
          }
      }
      ```
    reject_tools: ["bash", "edit", "create"]
    assertions:
      - type: output_matches
        pattern: "Assert\\.AreEqual\\(5,\\s*result"
      - type: output_matches
        pattern: "Assert\\.AreEqual\\(0,\\s*result"
      - type: output_not_matches
        pattern: "Assert\\.AreEqual\\(result,\\s*6\\)"
    rubric:
      - "Diagnoses the root cause unaided (the prompt does not name it): the first two assertions pass arguments in the wrong order — MSTest's signature is `Assert.AreEqual(expected, actual)`, so the literal must come first"
      - "Fixes the two swapped assertions to `Assert.AreEqual(5, result)` and `Assert.AreEqual(0, result)`"
      - "Leaves the already-correct `Assert.AreEqual(6, result)` in Subtract_Numbers_ReturnsDifference unchanged — does not swap a correct assertion"
      - "Explains that argument order drives the 'Expected: X, Actual: Y' failure message, and may reference the MSTEST0017 analyzer that flags this"
    timeout: 120

  - name: "Modernize legacy test patterns"
    prompt: "Can you review these MSTest tests and modernize them to use current MSTest best practices like sealed classes and Assert.ThrowsExactly?"
    setup:
      files:
        - path: "TestProject.csproj"
          source: "fixtures/legacy-antipatterns/TestProject.csproj"
        - path: "PriceCalculatorTests.cs"
          source: "fixtures/legacy-antipatterns/PriceCalculatorTests.cs"
        - path: "PriceCalculator.cs"
          source: "fixtures/legacy-antipatterns/PriceCalculator.cs"
      commands:
        - "dotnet restore"
    assertions:
      - type: output_matches
        pattern: "\\bsealed\\b"
      - type: output_matches
        pattern: "Assert\\.ThrowsExactly|Assert\\.Throws"
      - type: output_matches
        pattern: "IsInstanceOfType"
    rubric:
      - "Seals the test class (adds 'sealed' keyword)"
      - "Replaces [ExpectedException] with Assert.ThrowsExactly or Assert.Throws"
      - "Fixes the swapped Assert.AreEqual arguments (expected first, actual second)"
      - "Moves synchronous setup from [TestInitialize] into the constructor, enabling readonly fields and satisfying nullability analyzers"
      - "Removes nullable annotation from TestContext property or switches to constructor injection"
      - "Improves test naming to be descriptive (replaces 'Test1', 'TestNegativeQuantity')"
      - "Replaces hard cast with Assert.IsInstanceOfType"
      - "Replaces manual foreach test data loop with [DataRow] or [DynamicData] attributes"
    timeout: 300

  - name: "Replace ExpectedException with Assert.Throws"
    prompt: |
      I have this MSTest test that uses [ExpectedException]. How should I modernize it to use Assert.ThrowsExactly instead?

      ```csharp
      [TestMethod]
      [ExpectedException(typeof(ArgumentNullException))]
      public void CreateUser_NullEmail_ThrowsException()
      {
          var service = new UserService();
          service.CreateUser(null, "John");
      }
      ```
    reject_tools: ["bash", "edit", "create"]
    assertions:
      - type: output_matches
        pattern: "Assert\\.ThrowsExactly|Assert\\.Throws"
    rubric:
      - "The modernized test removes the ExpectedException attribute and wraps the throwing call in an inline assertion using Assert.ThrowsExactly or Assert.Throws"
      - "Shows how to capture the thrown exception to verify its properties such as message or parameter name"
      - "Explains when to prefer ThrowsExactly (exact type match) versus Throws (matches derived types too)"
    timeout: 120

  # ============================================================================
  # Goal 3: Using correct assertion methods
  # ============================================================================

  - name: "Use proper collection assertions"
    prompt: |
      I'm writing MSTest tests for a method that returns a list. What MSTest collection assertions should I use instead of Assert.IsTrue? Here's what I have:

      ```csharp
      [TestMethod]
      public void GetActiveUsers_ReturnsOnlyActiveUsers()
      {
          var service = new UserService();
          var users = service.GetActiveUsers();

          Assert.IsTrue(users.Count > 0);
          Assert.IsTrue(users.All(u => u.IsActive));
          Assert.IsTrue(users.Count() == 3);
          var admin = users.Single(u => u.Role == "Admin");
          Assert.IsTrue(admin != null);
      }
      ```
    reject_tools: ["bash", "edit", "create"]
    assertions:
      - type: output_matches
        pattern: "Assert\\.HasCount|Assert\\.AreEqual\\(3"
      - type: output_matches
        pattern: "ContainsSingle|Assert\\.IsNotNull"
    rubric:
      - "The non-empty validation no longer relies on a generic Assert.IsTrue boolean check — it uses a dedicated assertion such as the real MSTest 3.8+ `Assert.IsNotEmpty(collection)` (or `Assert.AreNotEqual(0, x.Count)`) that clearly reports 'collection was empty' on failure. Grading note: `Assert.IsNotEmpty`, `Assert.IsEmpty`, `Assert.HasCount`, `Assert.ContainsSingle`, `Assert.Contains`, and `Assert.DoesNotContain` are real assertion methods introduced in MSTest 3.8+ (Microsoft.Testing) — do not penalize them as non-existent or fabricated."
      - "The exact-count validation uses a count-specific assertion such as the real MSTest 3.8+ `Assert.HasCount(3, collection)` (or `Assert.AreEqual(3, x.Count)`) rather than wrapping the comparison in Assert.IsTrue"
      - "The single-element lookup uses an assertion that produces a meaningful failure message when zero or multiple elements match — such as the real MSTest 3.8+ `Assert.ContainsSingle(...)` — rather than `Single()` followed by Assert.IsTrue, which throws an unrelated exception"
      - "The null check uses a null-specific assertion rather than Assert.IsTrue with a boolean null comparison"
      - "Explains why specialized assertions produce better failure diagnostics than generic Assert.IsTrue"
    timeout: 120

  - name: "Use proper type assertions instead of casts"
    prompt: |
      I'm writing MSTest tests for a factory method. Is there a better MSTest assertion to check the returned type instead of hard casting?

      ```csharp
      [TestMethod]
      public void Create_ReturnsCorrectHandler()
      {
          var factory = new HandlerFactory();
          var result = factory.Create("email");

          var handler = (EmailHandler)result;
          Assert.IsNotNull(handler);
          Assert.AreEqual("email", handler.Type);
      }
      ```
    reject_tools: ["bash", "edit", "create"]
    assertions:
      - type: output_matches
        pattern: "IsInstanceOfType"
    rubric:
      - "Replaces the hard cast with a type-checking assertion that produces a clear failure message showing the actual type when the type does not match"
      - "Shows how to access the strongly-typed result after the type check so further property assertions can be made without a separate cast"
      - "Explains the diagnostic benefit — a type assertion failure message shows expected vs actual type, while a hard cast just throws InvalidCastException"
      - "Covers both MSTest 3.x and 4.x syntax variants for type assertions (out parameter vs direct return)"
    timeout: 120

  # ============================================================================
  # Goal 4: Test lifecycle and setup
  # ============================================================================

  - name: "Set up test lifecycle correctly"
    prompt: |
      I need to set up my MSTest test class with both synchronous and asynchronous initialization using constructor injection for TestContext. What's the recommended way to do this in modern MSTest?
    setup:
      files:
        - path: "TestProject.csproj"
          source: "fixtures/modern-mstest/TestProject.csproj"
    assertions:
      - type: output_matches
        pattern: "constructor|\\bctor\\b"
      - type: output_matches
        pattern: "TestContext"
      - type: output_matches
        pattern: "\\[TestInitialize\\]|TestInitialize"
    rubric:
      - "Always initializes in the constructor — enables readonly fields and satisfies nullability analyzers (fields guaranteed non-null after construction)"
      - "Uses [TestInitialize] only for async initialization, combined with constructor for sync parts"
      - "Injects TestContext via constructor parameter (MSTest 3.6+) rather than property"
      - "Mentions the execution order: Constructor → TestContext property → [TestInitialize] → Test → [TestCleanup] → Dispose"
      - "Seals the test class"
      - "If TestContext property is used, does not add nullable annotation or = null! initializer"
    timeout: 120

  - name: "Use DynamicData with ValueTuples over object arrays"
    prompt: |
      I have this MSTest data-driven test using DynamicData with object arrays. Is there a more type-safe way to write this in modern MSTest?

      ```csharp
      [TestMethod]
      [DynamicData(nameof(TestData))]
      public void Validate_ReturnsExpected(string input, bool expected)
      {
          var result = Validator.IsValid(input);
          Assert.AreEqual(expected, result);
      }

      public static IEnumerable<object[]> TestData =>
      [
          ["valid@email.com", true],
          ["invalid", false],
          ["", false],
      ];
      ```
    assertions:
      - type: output_matches
        pattern: "ValueTuple|\\(string.*bool\\)"
      - type: output_matches
        pattern: "IEnumerable<\\("
    rubric:
      - "Recommends switching from IEnumerable<object[]> to IEnumerable<(string input, bool expected)> ValueTuples"
      - "Explains that ValueTuples provide compile-time type safety"
      - "Shows the correct ValueTuple syntax with named elements"
      - "Mentions TestDataRow<T> as an option when custom DisplayName or metadata is needed"

  # ============================================================================
  # Goal 5: String assertions
  # ============================================================================

  - name: "Use string assertions for format validation"
    prompt: |
      Help me write MSTest tests for this formatting utility. I want to use the right MSTest string assertion methods like Assert.StartsWith and Assert.EndsWith to validate the output formats.

      ```csharp
      namespace Contoso.Text;

      public sealed class StringFormatter
      {
          public string FormatPhoneNumber(string digits)
          {
              ArgumentNullException.ThrowIfNull(digits);
              if (digits.Length != 10)
                  throw new ArgumentException("Must be exactly 10 digits.", nameof(digits));
              return $"({digits[..3]}) {digits[3..6]}-{digits[6..]}";
          }

          public string BuildGreeting(string name, string title = "Mr.")
          {
              ArgumentException.ThrowIfNullOrWhiteSpace(name);
              return $"Dear {title} {name},";
          }

          public string GenerateSlug(string title)
          {
              ArgumentException.ThrowIfNullOrWhiteSpace(title);
              return Regex.Replace(title.Trim().ToLowerInvariant(), @"[^a-z0-9]+", "-").Trim('-');
          }

          public string FormatCurrency(decimal amount, string symbol = "$")
          {
              return $"{symbol}{amount:F2}";
          }
      }
      ```
    reject_tools: ["bash", "edit", "create"]
    assertions:
      - type: output_matches
        pattern: "Assert\\.(StartsWith|EndsWith|MatchesRegex)"
      - type: output_matches
        pattern: "\\bsealed\\b"
    rubric:
      - "Validates string format structure using targeted string assertions like StartsWith, EndsWith, or MatchesRegex rather than only exact equality checks"
      - "Tests FormatPhoneNumber validates the parenthesized area code and hyphen structure"
      - "Tests BuildGreeting validates the output starts with 'Dear' and ends with a comma"
      - "Tests GenerateSlug validates the slug pattern contains only lowercase letters, digits, and hyphens"
      - "Tests exception handling for null or invalid inputs"
    timeout: 360

  # ============================================================================
  # Goal 6: Comparison assertions
  # ============================================================================

  - name: "Use comparison assertions for boundary testing"
    prompt: |
      I have a scoring system and need to write MSTest tests with proper boundary checking. What MSTest comparison assertions like Assert.IsGreaterThan or Assert.IsInRange should I use?

      ```csharp
      namespace Contoso.Scoring;

      public sealed class ScoreCalculator
      {
          public double CalculatePercentage(int correct, int total)
          {
              ArgumentOutOfRangeException.ThrowIfNegativeOrZero(total);
              ArgumentOutOfRangeException.ThrowIfNegative(correct);
              ArgumentOutOfRangeException.ThrowIfGreaterThan(correct, total);
              return (double)correct / total * 100.0;
          }

          public string GetGrade(double percentage) => percentage switch
          {
              >= 90 => "A",
              >= 80 => "B",
              >= 70 => "C",
              >= 60 => "D",
              _ => "F"
          };

          public double NormalizeScore(double rawScore, double maxRaw)
          {
              ArgumentOutOfRangeException.ThrowIfNegative(rawScore);
              ArgumentOutOfRangeException.ThrowIfNegativeOrZero(maxRaw);
              return Math.Min(rawScore / maxRaw * 100.0, 100.0);
          }
      }
      ```
    reject_tools: ["bash", "edit", "create"]
    assertions:
      - type: output_matches
        pattern: "Assert\\.(IsGreaterThan|IsLessThan|IsInRange|IsGreaterThanOrEqualTo|IsLessThanOrEqualTo)"
      - type: output_matches
        pattern: "\\bsealed\\b"
    rubric:
      - "Uses comparison assertions like IsGreaterThan, IsLessThan, or IsInRange for validating numeric bounds instead of Assert.IsTrue with manual comparisons"
      - "Tests that CalculatePercentage returns values in the 0-100 range for valid inputs"
      - "Tests boundary conditions at grade thresholds (e.g., exactly 90 vs 89)"
      - "Tests that NormalizeScore caps the result at 100 when rawScore exceeds maxRaw"
      - "Seals the test class"
    timeout: 360

  # ============================================================================
  # Goal 7: Collection, null, and reference assertions
  # ============================================================================

  - name: "Write tests with collection, null, and reference assertions"
    prompt: "Write MSTest tests for my ServiceRegistry class using the most appropriate assertion for each check. I need tests that verify: registering a service and resolving it returns the exact same instance; resolving an unregistered service returns null; the GetAll list contains a registered service and does not contain it after removal; and GetAll is empty before any registration but not empty afterwards."
    setup:
      files:
        - path: "src/ServiceRegistry.cs"
          content: |
            namespace Contoso.Services;

            public sealed class ServiceRegistry
            {
                private readonly Dictionary<Type, object> _services = new();

                public void Register<T>(T service) where T : class
                {
                    ArgumentNullException.ThrowIfNull(service);
                    _services[typeof(T)] = service;
                }

                public T? Resolve<T>() where T : class =>
                    _services.TryGetValue(typeof(T), out var svc) ? (T)svc : null;

                public T ResolveRequired<T>() where T : class =>
                    Resolve<T>() ?? throw new InvalidOperationException($"{typeof(T).Name} not registered.");

                public IReadOnlyList<object> GetAll() => [.. _services.Values];

                public void Remove<T>() where T : class => _services.Remove(typeof(T));

                public void Clear() => _services.Clear();
            }
        - path: "src/Contoso.Services.csproj"
          content: |
            <Project Sdk="Microsoft.NET.Sdk">
              <PropertyGroup>
                <TargetFramework>net9.0</TargetFramework>
              </PropertyGroup>
            </Project>
        - path: "tests/Contoso.Services.Tests.csproj"
          content: |
            <Project Sdk="Microsoft.NET.Sdk">
              <PropertyGroup>
                <TargetFramework>net9.0</TargetFramework>
              </PropertyGroup>
              <ItemGroup>
                <PackageReference Include="MSTest" Version="3.8.2" />
              </ItemGroup>
              <ItemGroup>
                <ProjectReference Include="../src/Contoso.Services.csproj" />
              </ItemGroup>
            </Project>
    assertions:
      - type: file_exists
        path: "tests/*.cs"
      - type: file_contains
        path: "tests/*.cs"
        value: "[TestClass]"
      - type: file_contains
        path: "tests/*.cs"
        value: "Assert.AreSame"
      - type: file_contains
        path: "tests/*.cs"
        value: "Assert.IsNull"
      - type: file_contains
        path: "tests/*.cs"
        value: "Assert.Contains"
      - type: file_contains
        path: "tests/*.cs"
        value: "Assert.DoesNotContain"
      - type: file_contains
        path: "tests/*.cs"
        value: "Assert.IsEmpty"
      - type: file_contains
        path: "tests/*.cs"
        value: "Assert.IsNotEmpty"
      # Guard the substring-based file_contains checks above against the legacy
      # CollectionAssert/StringAssert helpers (e.g. "CollectionAssert.Contains"
      # also contains the substring "Assert.Contains"). Rejecting them keeps the
      # scenario a deterministic check for the modern Assert.* API.
      - type: file_not_contains
        path: "tests/*.cs"
        value: "CollectionAssert.Contains"
      - type: file_not_contains
        path: "tests/*.cs"
        value: "CollectionAssert.DoesNotContain"
      - type: file_not_contains
        path: "tests/*.cs"
        value: "StringAssert.Contains"
      - type: file_not_contains
        path: "tests/*.cs"
        value: "StringAssert.DoesNotContain"
    rubric:
      - "Verifies reference identity (same object instance, not just value equality) using Assert.AreSame when testing that resolving a registered service returns the exact object that was registered"
      - "Checks that resolving an unregistered service type returns null, using Assert.IsNull rather than a boolean check"
      - "Uses Assert.Contains to verify the GetAll collection includes a registered service, and Assert.DoesNotContain to verify it is absent after Remove"
      - "Asserts the collection is empty with Assert.IsEmpty before registration and not empty with Assert.IsNotEmpty afterwards, rather than checking count manually"
      - "The written tests reference correct class and method names from the provided source code"
    timeout: 180

  # ============================================================================
  # Goal 8: Advanced features — conditional execution, retry, and cleanup
  # ============================================================================

  - name: "Configure conditional execution, retry, and cleanup"
    prompt: |
      I have MSTest tests that:
      1. Only work on Windows because they use the Windows registry
      2. Should be skipped in CI because they require a local GPU
      3. Sometimes fail due to a flaky external weather API
      4. Create temporary files that need to be deleted even if the test fails

      How should I handle all of these using MSTest attributes and patterns?
    reject_tools: ["bash", "edit", "create"]
    assertions:
      - type: output_matches
        pattern: "OSCondition|OsCondition"
      - type: output_matches
        pattern: "CICondition|CiCondition"
      - type: output_matches
        pattern: "Retry"
      - type: output_matches
        pattern: "TestCleanup|IDisposable"
    rubric:
      - "Uses an OS condition attribute to restrict the registry test to Windows only"
      - "Uses a CICondition attribute to exclude the GPU test from CI environments"
      - "Uses a retry attribute for the flaky weather API test with a small retry count"
      - "Uses [TestCleanup] or IDisposable/IAsyncDisposable for deleting temporary files to ensure cleanup runs even on test failure"
      - "Advises that retry should only be used for genuinely flaky external dependencies, not to mask real bugs"
    timeout: 120

  # ============================================================================
  # Goal 9: Test parallelization and MSTest.Sdk project setup
  # ============================================================================

  - name: "Configure test parallelization and MSTest.Sdk project"
    prompt: |
      I'm setting up a new MSTest project from scratch. I want:
      1. The simplest possible project file using MSTest.Sdk
      2. Tests to run in parallel by default
      3. But my database integration tests must run sequentially

      What's the recommended MSTest setup?
    reject_tools: ["bash", "edit", "create"]
    assertions:
      - type: output_matches
        pattern: "MSTest\\.Sdk|MSTest.Sdk"
      - type: output_matches
        pattern: "Parallelize"
      - type: output_matches
        pattern: "DoNotParallelize"
    rubric:
      - "Recommends MSTest.Sdk for the simplest possible project file"
      - "Shows assembly-level Parallelize attribute configuration with Workers and Scope"
      - "Uses [DoNotParallelize] on the database integration test class to opt it out of parallel execution"
      - "Suggests putting the MSTest.Sdk version in global.json so all test projects stay in sync"
    timeout: 120