mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
leslierichardson95
7286c31118
* Add C# MCP server skills (create, debug, test, publish) Four new skills for the C# MCP server development lifecycle: - mcp-csharp-create: Scaffolding with dotnet new mcpserver, tools/prompts/resources, transport config - mcp-csharp-debug: MCP Inspector, VS Code integration, breakpoint debugging, logging - mcp-csharp-test: Unit tests with ClientServerTestBase, integration with WebApplicationFactory, evals - mcp-csharp-publish: NuGet packaging, Docker/Azure deployment, MCP Registry publishing Each skill includes SKILL.md with progressive disclosure references/ and eval.yaml tests. * Fix mcp-csharp-create eval: use regex assertions for combined attribute syntax Replace scaffolding-heavy scenarios with implementation-focused ones that test MCP-specific features (resources, prompts, logging). Fix assertion patterns to match combined C# attribute syntax [McpServerTool, Description()] instead of requiring standalone [McpServerTool]. Increase timeouts to 180s to account for skill-reading overhead. Validator result: passed=True, improvement=44.6% (threshold=10%) * Add CODEOWNERS entries for MCP C# skills (create, debug, publish, test) * Update plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-publish/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Remove evaluation step and guide from mcp-csharp-test skill * Add CODEOWNERS entries for dotnet-maui skills * Update CODEOWNERS to add @artl93 as co-owner for mcp-csharp skills * Replace parent-directory file references with backtick skill names Cross-skill references used ../sibling/SKILL.md paths which the validator rejects. Replace with backtick-quoted skill names to match the convention used by other skills in the repo. * Add known domains for MCP server development skills Add domains referenced by the mcp-csharp-* skills: - github.com/modelcontextprotocol/registry - github.com/open-telemetry/semantic-conventions - npmjs.com, xunit.net, fluentassertions.com, nugettest.org Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Docker commands to use placeholder for registry and image names * moved mcp skills and their evals into dotnet-ai plugin * cleaned up moved files to dotnet-ai * cleaned up files * Update .github/CODEOWNERS Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update nuget-packaging.md * Update repository URL format in nuget-packaging.md * Add github.com/ to known-domains to allow truncated placeholder URLs The URL regex truncates at angle brackets (invalid in URIs), so placeholder URLs like https://github.com/<yourusername>/<repo> become https://github.com/ which didn't match any path-scoped entry. Adding github.com/ (with trailing slash) covers only this case without broadly allowing all github.com URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Docker tag command to use placeholder for server name * Update known domains and fix GitHub username placeholders in documentation * Remove McpServerToolType from mcp-csharp-create eval McpServerToolType is only needed with WithToolsFromAssembly(), which is discouraged because it is not Native AOT compatible. Updated eval assertions and rubrics to prefer WithTools<T>() instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve eval resilience for mcp-csharp-create and mcp-csharp-debug - Increase mcp-csharp-create timeouts from 180s (or default 120s) to 360s for all 3 scenarios. These scenarios consistently time out because the model spends time on bash exploration/scaffolding before writing code. - Add explicit timeout to scenario 3 which was relying on the 120s default. - Broaden mcp-csharp-debug Inspector rubric to accept both the concise single CLI command style and the step-by-step UI configuration walkthrough, since both are correct approaches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Broaden debug 'failing tool' rubric to accept multiple debugging approaches The rubric criterion 'Shows how to attach a debugger' was too narrow. The skilled answer correctly focused on the #1 cause (stdout pollution) but scored low because it didn't show a specific 'attach to process' flow. Broadened to accept any valid debugging approach: attaching, Debugger.Launch(), or launch.json configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 4 eval failures in mcp-csharp-debug and mcp-csharp-test skills mcp-csharp-debug: - Trim SKILL.md verbosity (183->160 lines) and mcp-inspector.md (67->54 lines) - Add Diagnosing Tool Errors section covering debugger, output panel, Inspector, common culprits - Rebalance debugging narrative away from stderr-only focus - Move HTTP logging config to ide-config.md reference mcp-csharp-test: - Add eval/evaluations keywords to frontmatter for activation - Add HTTP tool invocation test pattern (tools/call via WebApplicationFactory) - Trim test-patterns.md bloat (remove Test Categories, Coverage, Input Validation) - Create references/evaluations.md with qa_pair format and read-only/deterministic guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
52 lines
2.4 KiB
YAML
52 lines
2.4 KiB
YAML
scenarios:
|
|
- name: "Write unit and integration tests for an MCP server"
|
|
prompt: |
|
|
I have a C# MCP server with tools that call an external REST API.
|
|
I want to write tests that verify the tools work correctly without
|
|
calling the real API. How do I structure the test project?
|
|
assertions:
|
|
- type: "output_contains"
|
|
value: "xunit"
|
|
- type: "output_matches"
|
|
pattern: "(Mock|Fake|mock|fake|HttpMessageHandler)"
|
|
- type: "output_matches"
|
|
pattern: "(CallToolAsync|McpClient)"
|
|
rubric:
|
|
- "Creates a test project with xUnit and references the server project"
|
|
- "Shows how to mock HttpClient or external dependencies"
|
|
- "Demonstrates an integration test using McpClient to call tools in-memory"
|
|
- "Separates unit tests from integration tests using traits or folders"
|
|
timeout: 120
|
|
- name: "Test an HTTP MCP server with WebApplicationFactory"
|
|
prompt: |
|
|
I have an HTTP-based C# MCP server built with ASP.NET Core.
|
|
How do I write integration tests that test the full MCP protocol
|
|
over HTTP without spinning up a real server?
|
|
assertions:
|
|
- type: "output_contains"
|
|
value: "WebApplicationFactory"
|
|
- type: "output_matches"
|
|
pattern: "(initialize|MCP|jsonrpc)"
|
|
rubric:
|
|
- "Uses WebApplicationFactory<Program> for in-process HTTP testing"
|
|
- "Shows how to send an MCP initialize request to verify the server responds"
|
|
- "Mentions InternalsVisibleTo or making Program public for test access"
|
|
- "Tests tool invocation through the HTTP endpoint"
|
|
timeout: 120
|
|
- name: "Create evaluations for an MCP server"
|
|
prompt: |
|
|
I have a C# MCP server that provides tools for querying a product catalog.
|
|
I want to create evaluations to measure how well an LLM uses the tools.
|
|
What format should I use and what makes a good evaluation question?
|
|
assertions:
|
|
- type: "output_matches"
|
|
pattern: "(evaluation|qa_pair|question.*answer)"
|
|
- type: "output_matches"
|
|
pattern: "(read.only|non.destructive|deterministic|verifiable)"
|
|
rubric:
|
|
- "Shows the XML evaluation format with qa_pair elements"
|
|
- "Explains that questions should require multiple tool calls and reasoning"
|
|
- "Emphasizes that questions must be read-only with deterministic answers"
|
|
- "Provides example evaluation questions appropriate for a product catalog"
|
|
timeout: 120
|