dotnet__skills/tests/dotnet/dotnet-breaking-changes/eval.vally.yaml

name: dotnet-breaking-changes
description: Evaluates the dotnet/dotnet-breaking-changes skill
type: capability
config:
  timeout: 10m
stimuli:
  - name: Answer a public-API question before consolidating duplicated parsing
    prompt: "Is `AppSettingsHelper` part of the public API anywhere? Its int/bool setting parsing looks duplicated with `ConfigReader`. If it's safe to do, consolidate the setting parsing into one place and update the callers, without changing behavior. The solution is at Fixture.sln."
    environment:
      files:
        - src: src
          dest: src
        - src: tests
          dest: tests
        - src: Fixture.sln
          dest: Fixture.sln
    graders:
      - type: exit-success
      - type: output-matches
        config:
          pattern: (public API|PublicAPI\.Shipped|breaking change|shipped)
      - type: prompt
    rubric:
      - Answers the public-API question by actually inspecting what governs the surface (finds and reads `PublicAPI.Shipped.txt`, which lists the public parsing members) rather than assuming
      - Recognizes that removing or renaming a shipped public member is a breaking change a green test run will not catch, and either preserves the public surface (for example keeps the members as thin forwarders) or explicitly flags the break and updates the API baseline — does not silently break the public API
      - If it consolidates, keeps behavior unchanged and re-runs the build and tests to prove it

  - name: Add behavior to a shipped public member without breaking the contract
    prompt: "Add hex support to `AppSettingsHelper.ParseIntSetting` so callers can parse values like `0x1F` in addition to plain decimals. The solution is at Fixture.sln."
    environment:
      files:
        - src: src
          dest: src
        - src: tests
          dest: tests
        - src: Fixture.sln
          dest: Fixture.sln
    graders:
      - type: exit-success
      - type: output-matches
        config:
          pattern: (public API|PublicAPI|breaking change|shipped|signature|overload)
      - type: prompt
    rubric:
      - Treats this as a feature (adds new hex-parsing behavior), not a behavior-preserving refactor, and does not claim behavior is unchanged
      - Recognizes `AppSettingsHelper.ParseIntSetting` is a shipped public member listed in `PublicAPI.Shipped.txt`, and that changing its signature or return type would be a breaking change a green test run will not catch
      - Adds hex support without breaking the shipped signature — extends the body or adds an overload — and if the public surface changes, updates the API baseline (`PublicAPI.Unshipped.txt`) or explicitly flags the break rather than silently altering the shipped contract
      - Keeps the existing tests green and ideally adds coverage for the hex case

  - name: Rename a helper referenced from every #if branch of a multi-targeted type
    prompt: "The private `Label` helper in `PlatformInfo` is poorly named — rename it to `FormatLabel`. Keep behavior identical. The solution is at Fixture.sln."
    environment:
      files:
        - src: src
          dest: src
        - src: tests
          dest: tests
        - src: Fixture.sln
          dest: Fixture.sln
    graders:
      - type: exit-success
      - type: output-matches
        config:
          pattern: (multi[- ]?target|#if|net8|net10|target framework|TFM|every (branch|target))
      - type: prompt
    rubric:
      - Recognizes the project multi-targets (`net8.0;net10.0`) and that `Label` is referenced inside BOTH `#if` branches, so the rename must be replicated across every `#if` branch and target framework — not just the branch the default build compiles
      - Renames by tracking true binding references, then rebuilds ALL target frameworks and re-runs the tests to prove behavior is preserved
      - Keeps this a single, focused rename without bundling unrelated edits

  - name: Rename a member of a partial type that a generated part references
    prompt: "Rename `Coupons.Redeem` to `Apply` — the name is too generic. Keep behavior identical. The solution is at Fixture.sln."
    environment:
      files:
        - src: src
          dest: src
        - src: tests
          dest: tests
        - src: Fixture.sln
          dest: Fixture.sln
    graders:
      - type: exit-success
      - type: output-matches
        config:
          pattern: (auto[- ]?generated|source generator|generated (file|output|code)|partial|template|\.g\.cs|regenerat)
      - type: prompt
    rubric:
      - Includes EVERY partial declaration of `Coupons` when renaming, and finds the reference in the generated part (which references the renamed member)
      - "Recognizes the generated part is emitted at build time from `Coupons.g.cs.template`: flags that hand-editing the generated output under `obj/` is unsafe (it is overwritten on the next build) and that the correct fix edits the template (the generator input) — not the emitted file"
      - Rebuilds and re-runs the tests, letting the compiler catch any missed reference, to prove behavior is preserved

  - name: Add a new public method that must be correct on every target framework
    prompt: "Add a new public method `PlatformInfo.Tag()` that returns just the short platform tag (for example `net10` or `net8`) without the `platform:` prefix. It must return the correct tag on every target framework. The solution is at Fixture.sln."
    environment:
      files:
        - src: src
          dest: src
        - src: tests
          dest: tests
        - src: Fixture.sln
          dest: Fixture.sln
    graders:
      - type: exit-success
      - type: output-matches
        config:
          pattern: (multi[- ]?target|#if|net8|net10|target framework|TFM|public API|PublicAPI)
      - type: prompt
    rubric:
      - Treats this as a feature addition (a new public method), not a behavior-preserving refactor
      - Recognizes the project multi-targets (`net8.0;net10.0`) and uses `#if` branches, and implements `Tag()` so it returns the correct value on EVERY target framework — building and testing each target, not just the one the default build compiles
      - Recognizes `Tag()` is a NEW public member of a shipped type (public API surface) and updates the API baseline (`PublicAPI.Unshipped.txt`) or flags the surface addition
      - Keeps existing tests green and adds coverage for the new method

  - name: Do not raise breaking-change concerns for a purely internal local rename
    prompt: "Inside `OrderProcessor.DoStuff`, the local variable `subtotal` is a bit terse. Rename that local to `runningTotal` for readability. Keep behavior identical. The solution is at Fixture.sln."
    environment:
      files:
        - src: src
          dest: src
        - src: tests
          dest: tests
        - src: Fixture.sln
          dest: Fixture.sln
    graders:
      - type: exit-success
      - type: prompt
    rubric:
      - Recognizes that renaming a method-local variable touches no public API surface, no `#if`/multi-target branch, and no generated code, so no breaking-change hazard applies
      - Performs the local rename directly without introducing public-API baseline edits, source-generator changes, or other breaking-change ceremony that the change does not warrant
      - Confirms behavior is unchanged by rebuilding and re-running the tests