dotnet__skills/tests/dotnet-template-engine/template-discovery/eval.vally.yaml

name: template-discovery
description: Evaluates the dotnet-template-engine/template-discovery skill
type: capability
config:
  timeout: 3m
stimuli:
  - name: Find template for web API project
    prompt: I need to create a web API project with authentication. What template should I use and what are the
      authentication options?
    graders:
      - type: exit-success
      - type: output-contains
        config:
          substring: webapi
      - type: output-matches
        config:
          pattern: (?i)(individual|windows|none)
      - type: prompt
    rubric:
      - The agent used the template-discovery skill to find suitable templates
      - The agent identified the webapi template as the best match
      - The agent explained available authentication options (Individual, None, Windows, etc.)
      - The agent mentioned how to set the auth parameter
    constraints:
      expect_tools:
        - skill
  - name: Inspect template parameters and compare choices
    prompt: I'm choosing between the 'webapi' and 'webapp' templates. Compare their parameters side by side — which one
      supports authentication? Which one supports AOT? Show me the details.
    graders:
      - type: exit-success
      - type: output-contains
        config:
          substring: Framework
      - type: output-matches
        config:
          pattern: (?i)webapi
      - type: output-matches
        config:
          pattern: (?i)webapp
      - type: prompt
    rubric:
      - The agent used the template-discovery skill to inspect both the webapi and webapp templates
      - The agent compared parameters between the two templates
      - The agent identified which template supports authentication parameters
      - The agent identified AOT-related parameters where available
      - The agent presented a clear comparison rather than just listing parameters
    constraints:
      expect_tools:
        - skill
  - name: Search NuGet for specialized template
    prompt: I need a template for creating gRPC services in .NET. Search for available gRPC templates — both installed
      locally and on NuGet.org. Tell me which package to install.
    graders:
      - type: exit-success
      - type: output-matches
        config:
          pattern: (?i)grpc
      - type: output-matches
        config:
          pattern: (?i)(Microsoft\.Grpc|grpc.*service|ASP\.NET.*gRPC)
      - type: prompt
    rubric:
      - The agent used the template-discovery skill to search for gRPC templates
      - The agent searched both local and NuGet.org sources
      - The agent identified the grpc template or a relevant NuGet package
      - The agent provided the package name and install instructions
    constraints:
      expect_tools:
        - skill
  - name: Resolve ambiguous project intent to multiple candidates
    prompt: I want to build a real-time collaborative dashboard where users can see live updates. What .NET project
      templates could work? I need you to search available templates and show me the top 3 options with their key
      parameters.
    graders:
      - type: exit-success
      - type: output-matches
        config:
          pattern: (?i)blazor
      - type: output-matches
        config:
          pattern: (?i)(server|webassembly|wasm|interactive)
      - type: prompt
    rubric:
      - The agent used the template-discovery skill to find matching templates
      - The agent presented at least 2-3 template options
      - The agent explained the trade-offs between different approaches (Blazor Server vs WebAssembly, SignalR, etc.)
      - The agent showed key parameters for the recommended templates
    constraints:
      expect_tools:
        - skill
  - name: Preview project creation with dry run
    prompt: Before creating anything, show me exactly what files would be generated if I create a Blazor Web App called
      MyBlazorApp in ./test-output with interactive server rendering. Don't actually create the project yet.
    graders:
      - type: exit-success
      - type: output-contains
        config:
          substring: MyBlazorApp
      - type: file-not-exists
        config:
          path: test-output/**/MyBlazorApp.csproj
      - type: output-matches
        config:
          pattern: (?i)(Program\.cs|App\.razor|\.csproj)
      - type: prompt
    rubric:
      - The agent used the template-discovery skill to preview output with dotnet new --dry-run
      - The agent listed the files and directories that would be created
      - The agent did NOT actually create the project
      - The agent explained what key generated files are for
    constraints:
      expect_tools:
        - skill