mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-mai
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Abhitej John
|
ca1b47371f |
Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop (#813)
* Add automated per-plugin versioning (NBGV) with /version-bump + weekly backstop WHY Tools that surface skills (Copilot CLI, Claude Code, Codex, Cursor) read a plugin's version directly from its checked-in manifest. With no versioning discipline, a plugin's behavior can change while its advertised version stays flat, so clients never learn to re-pull, and there is no human-readable signal of what changed. We want correct, current versions in the repo with minimal manual work and without bloating the marketplace clone. WHAT - Per-plugin semantic versioning via Nerdbank.GitVersioning (NBGV). Each plugin owns a version.json whose pathFilters exclude the generated manifests and the version.json itself, so version height tracks real content changes only. - The computed version is materialized into the checked-in manifests (plugin.json and .codex-plugin/plugin.json) so every consumer reads a current value with no build step on their side. - eng/version/Sync-PluginVersions.ps1 is the single workhorse. It resolves the set of changed plugins from a git diff, computes each version with nbgv (predicting the squash-merge height for PRs), and either reports or stamps. AUTOMATIONS (two, low-touch by design) - /version-bump: an admin/maintainer comments the command on a PR and the affected plugins are stamped on the PR branch. Gated on collaborator permission (admin/write/maintain); forks are rejected before any privileged step. No other PRs are auto-modified. - weekly-version-sync: a Monday backstop (and workflow_dispatch) that stamps any drift on main, opens/updates a single bot PR, and explains the per-plugin reason. This self-heals anything that merged without a bump. We deliberately did NOT auto-edit contributor PRs or add a noisy advisory comment bot; maintainers stay in control and the signal stays clean. SECURITY (multi-model adversarial review: GPT-5.5 + Gemini 3.1 Pro) - Supply chain (High, both models): dotnet tool restore would have honored a nuget.config authored in the PR tree, letting an attacker remap the nbgv package source to a malicious feed and run code in the privileged contents:write context. Mitigated with a trusted eng/version/nuget.config (clear + nuget.org-only + packageSourceMapping), overlaid from main and used via --configfile so PR-supplied configs are ignored. No nuget.config is tracked in the repo today, so this path was genuinely exploitable. - TOCTOU (Medium): /version-bump now checks out the authorized head SHA rather than the mutable branch name; a racing push fails non-fast-forward, which is the safe outcome. - Injection: Set-ManifestVersion uses a MatchEvaluator (not a replacement string) so a "$"-bearing version cannot re-expand, plus a strict major.minor.patch guard that throws on a malformed base, leaving manifests untouched. - A base-only version.json bump (0.1 -> 0.2) is correctly detected and stamped. VERIFIED End-to-end against a real NBGV git harness: content-scoped predict, base-only bump -> x.y.0, docs-only -> [], weekly drift stamping, malformed-base guard, and --configfile restore (exit 0). actionlint passes on both workflows. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review feedback - Add missing plugins/dotnet-test-migration/version.json so it participates in versioning (it was the only plugin without one; manifests are at 0.1.0). - CONTRIBUTING: the two manifests are not byte-identical; say the version is duplicated across two manifest files instead. - weekly-version-sync: include version.json in commit attribution so a base-only bump is explained rather than showing 'no attributable commits'. - Get-NbgvInfo: capture nbgv stderr and include it in the thrown error so CI failures are diagnosable, while keeping stdout clean for JSON parsing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Abhitej John
|
2ab3501158 |
Add .codex-plugin/plugin.json manifests for Codex CLI plugin install (#726)
The Codex CLI requires .codex-plugin/plugin.json as the plugin manifest entry point. Without it, 'codex plugin add' fails with 'missing plugin.json' even though the marketplace listing works. This adds .codex-plugin/plugin.json to all 14 plugin directories, with paths relative to the plugin root per the Codex docs. Also updates the agents marketplace to use dotnet-aspnetcore (per #711 rename) and adds missing dotnet-blazor and dotnet11 entries. Fixes #578 Fixes #724 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
e7ec5a77a0 | Fix markdownlint CI workflow (startup_failure since inception) (#629) | ||
|
Jan Krivanek
|
05aeb657e6 | Add license to agent files (#568) | ||
|
Dan Moseley
|
165e6439fe |
Tighten known-domains allowlist (#496)
* Tighten known-domains allowlist Narrow broad domain entries to specific paths/subdomains actually used: - aka.ms -> aka.ms/dotnet-trace, aka.ms/perfcollect - nuget.org -> api.nuget.org, www.nuget.org/account/trustedpublishing - npmjs.com/package/@modelcontextprotocol -> pinned to inspector/v/0.21.1 Remove unused domains: fluentassertions.com, speedscope.app Update skill content to match: pin MCP Inspector link, remove FluentAssertions link, remove speedscope hyperlink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * updated mcp skills codeowner * Refine scoped known-domain entries Remove the remaining nugettest.org publish example and drop the nugettest.org allowlist entry. Also normalize path-scoped entries to omit a hardcoded www prefix so both host forms match the existing ReferenceScanner logic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Leslie Richardson <lerich@microsoft.com> |
||
|
leslierichardson95
|
7286c31118 |
Add skills for C# MCP Server Development (#317)
* Add C# MCP server skills (create, debug, test, publish) Four new skills for the C# MCP server development lifecycle: - mcp-csharp-create: Scaffolding with dotnet new mcpserver, tools/prompts/resources, transport config - mcp-csharp-debug: MCP Inspector, VS Code integration, breakpoint debugging, logging - mcp-csharp-test: Unit tests with ClientServerTestBase, integration with WebApplicationFactory, evals - mcp-csharp-publish: NuGet packaging, Docker/Azure deployment, MCP Registry publishing Each skill includes SKILL.md with progressive disclosure references/ and eval.yaml tests. * Fix mcp-csharp-create eval: use regex assertions for combined attribute syntax Replace scaffolding-heavy scenarios with implementation-focused ones that test MCP-specific features (resources, prompts, logging). Fix assertion patterns to match combined C# attribute syntax [McpServerTool, Description()] instead of requiring standalone [McpServerTool]. Increase timeouts to 180s to account for skill-reading overhead. Validator result: passed=True, improvement=44.6% (threshold=10%) * Add CODEOWNERS entries for MCP C# skills (create, debug, publish, test) * Update plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-publish/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Remove evaluation step and guide from mcp-csharp-test skill * Add CODEOWNERS entries for dotnet-maui skills * Update CODEOWNERS to add @artl93 as co-owner for mcp-csharp skills * Replace parent-directory file references with backtick skill names Cross-skill references used ../sibling/SKILL.md paths which the validator rejects. Replace with backtick-quoted skill names to match the convention used by other skills in the repo. * Add known domains for MCP server development skills Add domains referenced by the mcp-csharp-* skills: - github.com/modelcontextprotocol/registry - github.com/open-telemetry/semantic-conventions - npmjs.com, xunit.net, fluentassertions.com, nugettest.org Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Docker commands to use placeholder for registry and image names * moved mcp skills and their evals into dotnet-ai plugin * cleaned up moved files to dotnet-ai * cleaned up files * Update .github/CODEOWNERS Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update nuget-packaging.md * Update repository URL format in nuget-packaging.md * Add github.com/ to known-domains to allow truncated placeholder URLs The URL regex truncates at angle brackets (invalid in URIs), so placeholder URLs like https://github.com/<yourusername>/<repo> become https://github.com/ which didn't match any path-scoped entry. Adding github.com/ (with trailing slash) covers only this case without broadly allowing all github.com URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Docker tag command to use placeholder for server name * Update known domains and fix GitHub username placeholders in documentation * Remove McpServerToolType from mcp-csharp-create eval McpServerToolType is only needed with WithToolsFromAssembly(), which is discouraged because it is not Native AOT compatible. Updated eval assertions and rubrics to prefer WithTools<T>() instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve eval resilience for mcp-csharp-create and mcp-csharp-debug - Increase mcp-csharp-create timeouts from 180s (or default 120s) to 360s for all 3 scenarios. These scenarios consistently time out because the model spends time on bash exploration/scaffolding before writing code. - Add explicit timeout to scenario 3 which was relying on the 120s default. - Broaden mcp-csharp-debug Inspector rubric to accept both the concise single CLI command style and the step-by-step UI configuration walkthrough, since both are correct approaches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Broaden debug 'failing tool' rubric to accept multiple debugging approaches The rubric criterion 'Shows how to attach a debugger' was too narrow. The skilled answer correctly focused on the #1 cause (stdout pollution) but scored low because it didn't show a specific 'attach to process' flow. Broadened to accept any valid debugging approach: attaching, Debugger.Launch(), or launch.json configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 4 eval failures in mcp-csharp-debug and mcp-csharp-test skills mcp-csharp-debug: - Trim SKILL.md verbosity (183->160 lines) and mcp-inspector.md (67->54 lines) - Add Diagnosing Tool Errors section covering debugger, output panel, Inspector, common culprits - Rebalance debugging narrative away from stderr-only focus - Move HTTP logging config to ide-config.md reference mcp-csharp-test: - Add eval/evaluations keywords to frontmatter for activation - Add HTTP tool invocation test pattern (tools/call via WebApplicationFactory) - Trim test-patterns.md bloat (remove Test Categories, Coverage, Input Validation) - Create references/evaluations.md with qa_pair format and read-only/deterministic guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Copilot
|
6face98ff6 |
Change agents and skills fields in plugin.json to array format for Claude Code CLI compatibility (#407)
* Initial plan * Remove agents string field from plugin.json and handle array format in parser Claude Code CLI validates `agents` as array-of-strings per its plugin manifest schema. Our plugin.json files had `"agents": "./agents/"` (a string), which caused "agents: Invalid input" validation errors on install. Remove the field from the 3 affected plugin.json files — agents are still discovered by convention from the `agents/` directory. Also make ParsePluginJson resilient to both string and array formats for forward-compatibility. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Remove `agents` string field from plugin.json for Claude Code CLI compatibility Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Change agents field to array format in plugin.json, update validator to read array-first with string fallback Plugin.json files now declare agents as an array of file paths: "agents": ["./agents/foo.agent.md", "./agents/bar.agent.md"] ParsePluginJson reads the array first (preferred, Claude Code schema), falls back to a string path (legacy). PluginInfo gains an AgentPaths field. Validator and discovery use the array when present, else fall back to directory-based convention. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Simplify agents to directory-path array form ["./agents/"], handle directories in array entries Per Copilot CLI spec, agents field values are "Path(s) to agent directories" — no need to list individual files. Use ["./agents/"] to mirror how skills uses "./skills/". Updated validator and discovery to handle both directory and file entries in the array. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Delete .nuget/nuget.exe * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Convert skills field to array format in plugin.json, update validator to handle both forms Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Align AgentPaths null checks to use { Count: > 0 } pattern for consistency Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Change `agents` and `skills` fields in plugin.json to array format for Claude Code CLI compatibility Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Delete .nuget/nuget.exe * Consolidate SkillsPath/SkillPaths and AgentsPath/AgentPaths into single array fields Remove dual string/array fields from PluginInfo. The parser now normalizes legacy string values into single-element arrays, so all downstream code uses only SkillPaths and AgentPaths (IReadOnlyList<string>). This eliminates all array-vs-string fallback branching. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jeff Schwartz
|
551461a926 |
dotnet-ai-ml (#177)
* moved from private to public fork * add code owner * accomodate feedback * update eval to make more natural language * address eval feedback * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * update supporting documentation of adding a new plugin * update claude-plugin --------- Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |