dotnet__skills/tests/dotnet-ai/technology-selection/eval.vally.yaml

name: technology-selection
description: Evaluates the dotnet-ai/technology-selection skill
type: capability
config:
  timeout: 6m
stimuli:
  - name: ML.NET classification on tabular data
    prompt: "Add a support ticket classifier to this .NET 10 Web API. Tickets have a Subject (string), Description (string),
      Priority (int 1-3), and Category should be predicted as one of: Billing, Technical, General. Use the provided
      training data CSV. Show me the full implementation including model training, evaluation metrics, and a prediction
      endpoint."
    environment:
      files:
        - src: ./fixtures/ml-net-classification-on-tabular-data/TicketClassifier/TicketClassifier.csproj
          dest: TicketClassifier/TicketClassifier.csproj
        - src: ./fixtures/ml-net-classification-on-tabular-data/TicketClassifier/Program.cs
          dest: TicketClassifier/Program.cs
        - src: ./fixtures/ml-net-classification-on-tabular-data/TicketClassifier/Data/tickets.csv
          dest: TicketClassifier/Data/tickets.csv
    graders:
      - type: output-contains
        config:
          substring: MLContext
      - type: output-contains
        config:
          substring: "new MLContext(seed:"
      - type: output-contains
        config:
          substring: TrainTestSplit
      - type: exit-success
      - type: prompt
    rubric:
      - Uses ML.NET (Microsoft.ML) for classification — does NOT use an LLM for this structured data task
      - Sets a random seed on MLContext for reproducibility
      - Splits data into train/test sets and evaluates on held-out test data
      - Logs or reports evaluation metrics appropriate for multiclass classification (e.g., MicroAccuracy,
        MacroAccuracy, LogLoss)
      - Uses PredictionEnginePool in the ASP.NET Core endpoint — not a singleton PredictionEngine
  - name: LLM integration with MEAI abstraction
    prompt: Add a document summarization endpoint to this .NET 10 Web API. The endpoint should accept a document text and
      return a concise summary.
    environment:
      files:
        - src: ./fixtures/llm-integration-with-meai-abstraction/DocSummary/DocSummary.csproj
          dest: DocSummary/DocSummary.csproj
        - src: ./fixtures/llm-integration-with-meai-abstraction/DocSummary/Program.cs
          dest: DocSummary/Program.cs
        - src: ./fixtures/llm-integration-with-meai-abstraction/DocSummary/appsettings.json
          dest: DocSummary/appsettings.json
    graders:
      - type: output-contains
        config:
          substring: IChatClient
      - type: output-contains
        config:
          substring: Temperature
      - type: output-not-contains
        config:
          substring: sk-
      - type: output-not-contains
        config:
          substring: apiKey = "
      - type: exit-success
      - type: prompt
    rubric:
      - Uses Microsoft.Extensions.AI (IChatClient) as the abstraction layer — does not call Azure.AI.OpenAI or OpenAI
        SDK directly in business logic
      - Registers the AI client via dependency injection using AddChatClient
      - Sets Temperature and MaxOutputTokens explicitly in ChatOptions
      - Implements retry logic (e.g., RetryingChatClient or Polly-based)
      - Loads API keys from configuration or environment — not hardcoded in source
      - Pins to a specific dated model version (e.g., gpt-4o-2024-08-06) rather than an unversioned alias
  - name: Reject LLM for tabular classification
    prompt: I have a .NET 10 project with a CSV of customer data (Age, Income, Region, PurchaseHistory columns). I want to
      predict whether a customer will churn (yes/no).
    environment:
      files:
        - src: ./fixtures/reject-llm-for-tabular-classification/ChurnPredictor/ChurnPredictor.csproj
          dest: ChurnPredictor/ChurnPredictor.csproj
        - src: ./fixtures/reject-llm-for-tabular-classification/ChurnPredictor/Program.cs
          dest: ChurnPredictor/Program.cs
        - src: ./fixtures/reject-llm-for-tabular-classification/ChurnPredictor/Data/customers.csv
          dest: ChurnPredictor/Data/customers.csv
    graders:
      - type: output-contains
        config:
          substring: ML.NET
      - type: exit-success
      - type: prompt
    rubric:
      - Redirects the user away from using GPT-4/LLM for this tabular classification task
      - Recommends ML.NET as the appropriate technology with a clear rationale (faster, cheaper, deterministic)
      - Provides an ML.NET-based binary classification implementation instead
      - "Follows the decision tree from the skill: structured tabular data → ML.NET"
  - name: Agentic workflow with guardrails
    prompt: Build a .NET 10 console app that uses an AI agent to research a topic by searching the web, then summarize
      findings. The agent should have access to a web search tool and a note-taking tool.
    environment:
      files:
        - src: ./fixtures/agentic-workflow-with-guardrails/ResearchAgent/ResearchAgent.csproj
          dest: ResearchAgent/ResearchAgent.csproj
        - src: ./fixtures/agentic-workflow-with-guardrails/ResearchAgent/Program.cs
          dest: ResearchAgent/Program.cs
    graders:
      - type: output-contains
        config:
          substring: MaximumIterations
      - type: output-contains
        config:
          substring: Microsoft.Agents.AI
      - type: output-contains
        config:
          substring: Microsoft.Extensions.AI
      - type: exit-success
      - type: prompt
    rubric:
      - Uses Microsoft Agent Framework (Microsoft.Agents.AI) for orchestration — not raw LLM calls in a loop
      - Builds on top of Microsoft.Extensions.AI (IChatClient) as the foundation layer
      - Sets MaximumIterations to cap the agentic loop and prevent runaway execution
      - Defines explicit tool/function schemas with descriptions for each tool
      - Implements observability by logging agent steps (tool selected, input, output)
      - Mentions or implements a cost ceiling or token budget
  - name: Natural-language scenario decomposition — RAG chatbot
    prompt: I want to build a web-based LLM driven chatbot using C# that allows customers to get help with our internal
      product PDF documentation and Blazor. I want to use Postgres for data storage along with just a file folder for
      our documentation. Show me your plan before continuing with the implementation.
    environment:
      files:
        - src: ./fixtures/natural-language-scenario-decomposition-rag-chatbot/HelpChatbot/HelpChatbot.csproj
          dest: HelpChatbot/HelpChatbot.csproj
        - src: ./fixtures/natural-language-scenario-decomposition-rag-chatbot/HelpChatbot/Program.cs
          dest: HelpChatbot/Program.cs
        - src: ./fixtures/natural-language-scenario-decomposition-rag-chatbot/HelpChatbot/Docs/product-guide.pdf
          dest: HelpChatbot/Docs/product-guide.pdf
    graders:
      - type: output-contains
        config:
          substring: Microsoft.Extensions.AI
      - type: exit-success
      - type: prompt
    rubric:
      - Presents a plan or architecture breakdown before diving into code — decomposes the vague request into
        identifiable capability needs (chat, document ingestion, vector search, embedding, UI)
      - Selects Microsoft.Extensions.AI (MEAI) as the abstraction layer and identifies a concrete LLM provider (e.g.,
        OpenAI, Azure.AI.OpenAI, or Azure.AI.Inference)
      - Identifies the need for document ingestion and chunking of PDF files — references
        Microsoft.Extensions.AI.DataIngestion or implements equivalent parsing/chunking
      - Selects Microsoft.Extensions.VectorData.Abstractions with a pgvector connector for PostgreSQL-based vector
        storage
      - Uses Blazor for the web UI as requested — does not substitute a different frontend framework
      - Reads documents from a file folder as specified — does not require a cloud blob store or other infrastructure
        the user did not ask for
      - Does not hallucinate NuGet package names — uses real, existing packages for the selected providers
  - name: RAG pipeline with vector search
    prompt: I need to add a RAG-based Q&A feature to a .NET 10 Web API. The app has a collection of product documentation
      markdown files. Users should be able to ask questions and get answers grounded in the documentation. I need
      document ingestion with chunking and a query endpoint. Show me your plan and architecture only — do not write any
      code.
    environment:
      files:
        - src: ./fixtures/rag-pipeline-with-vector-search/ProductQA/ProductQA.csproj
          dest: ProductQA/ProductQA.csproj
        - src: ./fixtures/rag-pipeline-with-vector-search/ProductQA/Program.cs
          dest: ProductQA/Program.cs
        - src: ./fixtures/rag-pipeline-with-vector-search/ProductQA/Docs/getting-started.md
          dest: ProductQA/Docs/getting-started.md
        - src: ./fixtures/rag-pipeline-with-vector-search/ProductQA/Docs/configuration.md
          dest: ProductQA/Docs/configuration.md
    graders:
      - type: output-contains
        config:
          substring: chunk
      - type: output-contains
        config:
          substring: embedding
      - type: exit-success
      - type: prompt
    rubric:
      - Describes a chunking strategy for documents (semantic or paragraph-based, not naive fixed-size)
      - Selects Microsoft.Extensions.VectorData or a vector database for storing and querying embeddings
      - Mentions setting a minimum similarity score threshold to filter out irrelevant results
      - Includes source attribution in the design — tracks which document chunks contributed to the answer
      - Plans to cache embeddings rather than re-embedding content on every query
      - Selects Microsoft.Extensions.AI (IEmbeddingGenerator) for the embedding abstraction layer