Commit Graph

5 Commits

Author SHA1 Message Date
Aditya Mandaleeka 52ba152ed6 Update to Vally 0.7.0 (#854) 2026-07-10 02:19:48 +00:00
Aditya Mandaleeka 9433a31e67 Initial port of evals to Vally (#615) 2026-05-06 10:11:45 -07:00
leslierichardson95 7286c31118 Add skills for C# MCP Server Development (#317)
* Add C# MCP server skills (create, debug, test, publish)

Four new skills for the C# MCP server development lifecycle:
- mcp-csharp-create: Scaffolding with dotnet new mcpserver, tools/prompts/resources, transport config
- mcp-csharp-debug: MCP Inspector, VS Code integration, breakpoint debugging, logging
- mcp-csharp-test: Unit tests with ClientServerTestBase, integration with WebApplicationFactory, evals
- mcp-csharp-publish: NuGet packaging, Docker/Azure deployment, MCP Registry publishing

Each skill includes SKILL.md with progressive disclosure references/ and eval.yaml tests.

* Fix mcp-csharp-create eval: use regex assertions for combined attribute syntax

Replace scaffolding-heavy scenarios with implementation-focused ones that
test MCP-specific features (resources, prompts, logging). Fix assertion
patterns to match combined C# attribute syntax [McpServerTool, Description()]
instead of requiring standalone [McpServerTool]. Increase timeouts to 180s
to account for skill-reading overhead.

Validator result: passed=True, improvement=44.6% (threshold=10%)

* Add CODEOWNERS entries for MCP C# skills (create, debug, publish, test)

* Update plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-publish/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remove evaluation step and guide from mcp-csharp-test skill

* Add CODEOWNERS entries for dotnet-maui skills

* Update CODEOWNERS to add @artl93 as co-owner for mcp-csharp skills

* Replace parent-directory file references with backtick skill names

Cross-skill references used ../sibling/SKILL.md paths which the
validator rejects. Replace with backtick-quoted skill names to match
the convention used by other skills in the repo.

* Add known domains for MCP server development skills

Add domains referenced by the mcp-csharp-* skills:
- github.com/modelcontextprotocol/registry
- github.com/open-telemetry/semantic-conventions
- npmjs.com, xunit.net, fluentassertions.com, nugettest.org

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update Docker commands to use placeholder for registry and image names

* moved mcp skills and their evals into dotnet-ai plugin

* cleaned up moved files to dotnet-ai

* cleaned up files

* Update .github/CODEOWNERS

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update nuget-packaging.md

* Update repository URL format in nuget-packaging.md

* Add github.com/ to known-domains to allow truncated placeholder URLs

The URL regex truncates at angle brackets (invalid in URIs), so
placeholder URLs like https://github.com/<yourusername>/<repo>
become https://github.com/ which didn't match any path-scoped entry.
Adding github.com/ (with trailing slash) covers only this case without
broadly allowing all github.com URLs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update Docker tag command to use placeholder for server name

* Update known domains and fix GitHub username placeholders in documentation

* Remove McpServerToolType from mcp-csharp-create eval

McpServerToolType is only needed with WithToolsFromAssembly(), which is
discouraged because it is not Native AOT compatible. Updated eval assertions
and rubrics to prefer WithTools<T>() instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve eval resilience for mcp-csharp-create and mcp-csharp-debug

- Increase mcp-csharp-create timeouts from 180s (or default 120s) to 360s
  for all 3 scenarios. These scenarios consistently time out because the
  model spends time on bash exploration/scaffolding before writing code.

- Add explicit timeout to scenario 3 which was relying on the 120s default.

- Broaden mcp-csharp-debug Inspector rubric to accept both the concise
  single CLI command style and the step-by-step UI configuration walkthrough,
  since both are correct approaches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update eng/known-domains.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Broaden debug 'failing tool' rubric to accept multiple debugging approaches

The rubric criterion 'Shows how to attach a debugger' was too narrow.
The skilled answer correctly focused on the #1 cause (stdout pollution)
but scored low because it didn't show a specific 'attach to process' flow.
Broadened to accept any valid debugging approach: attaching, Debugger.Launch(),
or launch.json configuration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix 4 eval failures in mcp-csharp-debug and mcp-csharp-test skills

mcp-csharp-debug:
- Trim SKILL.md verbosity (183->160 lines) and mcp-inspector.md (67->54 lines)
- Add Diagnosing Tool Errors section covering debugger, output panel, Inspector, common culprits
- Rebalance debugging narrative away from stderr-only focus
- Move HTTP logging config to ide-config.md reference

mcp-csharp-test:
- Add eval/evaluations keywords to frontmatter for activation
- Add HTTP tool invocation test pattern (tools/call via WebApplicationFactory)
- Trim test-patterns.md bloat (remove Test Categories, Coverage, Input Validation)
- Create references/evaluations.md with qa_pair format and read-only/deterministic guidance

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-25 14:17:57 -06:00
Jan Krivanek 68d83b6ea8 Add parallelism configuration options to eval.yaml (#410) 2026-03-19 19:08:02 +00:00
Jeff Schwartz 551461a926 dotnet-ai-ml (#177)
* moved from private to public fork

* add code owner

* accomodate feedback

* update eval to make more natural language

* address eval feedback

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* update supporting documentation of adding a new plugin

* update claude-plugin

---------

Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
2026-03-11 15:06:04 -07:00