mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
fix-dotnet-test-analysis-scripts
3 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 YuliiaKovalova
|
74c58505e5 |
Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills (#745)
* Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills Fix inaccurate reserved-shortName guidance, consolidate validation rules into a single skill, expand discovery mappings, add explicit CPM/version steps, and introduce template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .codex-plugin manifest indentation; add evals for new skills Make .codex-plugin/plugin.json byte-consistent with plugin.json (2-space indent on the agents line). Add eval.yaml + eval.vally.yaml capability evals for the new template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: source reserved list from dotnet new --help, note workload/package availability, tighten version-refresh - Clarify the reserved shortName set is the current dotnet new subcommands (authoritative source: dotnet new --help); create is verified as a real subcommand (alias behind dotnet new <template>). - template-discovery: note that some mapped short names (maui, winui3, aspire, func, orleans) need workloads/template packages, with fallback to dotnet new list/search. - template-instantiation: keep template versions by default; if refreshing, use dotnet list package --outdated + user confirmation and constrain to same major/minor rather than always latest stable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address 2nd review round: enforce eval negatives, split combined assertion, reframe reserved list - smart-defaults evals: enforce --no-https absence (auth scenario), absence of minimal-API flag (controllers scenario), and no newer --framework TFM when net8.0 is explicitly required, using output_not_contains/output_not_matches. - comparison eval: split the combined (auth|aot|docker|controllers) check into four separate output_matches assertions so partial comparisons fail. - template-validation/authoring: reframe the reserved shortName list as non-exhaustive examples and source the authoritative set from dotnet new --help; drop the specific create-alias assertion in favor of parsing-ambiguity wording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make template-comparison evals robust to CI SDK currency Switch the Blazor comparison scenario from blazorserver (absent in the CI SDK) to blazor (Blazor Web App) vs blazorwasm, both reliably present in .NET 8+, and instruct the agent to inspect each via --help. Update the SKILL.md example reference for currency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CLI-failure resilience guidance to discovery/comparison skills The isolated eval runs failed because the agent ran 'dotnet new <t> --help', hit the template engine's global-mutex/persistence error (common when the command runs concurrently in a sandbox), and then returned no answer at all. Instruct both skills to run 'dotnet new' calls sequentially, retry once on a transient mutex/persistence error, and fall back to the intent/parameter mapping so a concrete answer is always produced instead of empty output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: accurate flags + robust negative-assertion prompts - template-smart-defaults SKILL.md: drop the non-existent --publish-aot flag. Clarify --aot is a dotnet new flag only on templates that expose it (console/worker/grpc, not webapi) and that publish-time AOT is the MSBuild PublishAot=true property, not a dotnet new flag. - template-discovery SKILL.md: replace the hardcoded --enable-docker mapping (not a real flag on common templates) with generic 'confirm with --help'. - smart-defaults evals: tighten the negative-assertion prompts to output only the command line and not mention unused flags, so a negated explanation can't trip output_not_contains/output_not_matches. Switch the AOT scenario from webapi to worker (which actually supports --aot). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 3: scope negative checks, de-emphasize stale lists - smart-defaults evals: anchor the negative assertions to the 'dotnet new' command line (same-line regex) instead of whole-output substring/regex, so a flag mentioned only in prose can't fail the test. - template-validation / template-authoring: mark the dotnet new subcommand examples as illustrative/version-dependent and tell readers not to hardcode them; the live 'dotnet new --help' output is canonical. - template-comparison: fix the example table's AOT row — webapi/webapp do not expose a --aot template flag; native AOT is publish-time via PublishAot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
05aeb657e6 | Add license to agent files (#568) | ||
|
YuliiaKovalova
|
dcba4f3d8e |
Add dotnet-template-engine plugin (#341)
* Add dotnet-template-engine plugin Add template-engine skills for .NET project scaffolding powered by the DotnetTemplateMCP MCP server (v1.1.0). Includes three skills and one agent: Skills: - template-discovery: find, inspect, and compare templates with intent resolution (70+ keyword mappings) - template-instantiation: create projects with CPM adaptation, latest NuGet versions, parameter validation, and multi-template composition - template-authoring: create templates from existing projects and validate template.json for authoring issues Agent: - template-engine: expert agent that routes to the appropriate skill based on user intent Also adds CODEOWNERS entries, marketplace.json registration, eval tests, and .mcp.json for DotnetTemplateMCP server configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: mcpServers in plugin.json, expect_tools, path fix - Move MCP server config into plugin.json under mcpServers so SkillValidator discovers the template_* tools during evaluation runs - Add expect_tools constraints to all eval scenarios to verify actual MCP tool invocation instead of relying on rubric text alone - Fix frontmatter path: template.config → .template.config (leading dot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update DotnetTemplateMCP to v1.2.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Include 'dotnet new' in agent description for discoverability Address review feedback from danmoseley: the frontmatter description is how the caller decides whether to load the agent tokens, so it needs to mention 'dotnet new' as a trigger keyword. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove MCP server configuration and references - Delete .mcp.json MCP server config file - Remove mcpServers section from plugin.json - Remove MCP/DotnetTemplateMCP references from agent, skills, and marketplace descriptions - Skills and eval files remain unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove ghost tool references and add CLI fallback guidance - Remove references to 4 non-existent tools: template_compose, template_suggest_parameters, template_validate, solution_analyze - Replace with real alternatives (template_inspect, manual review, sequential template_instantiate, direct file inspection) - Add CLI fallback guidance so LLMs degrade gracefully when template tools are unavailable - Fix authoring eval to not expect template_validate tool Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: CODEOWNERS naming, claude marketplace, README - Rename CODEOWNERS header to dotnet-template-engine for consistency - Mirror plugin entry in .claude-plugin/marketplace.json - Add plugin to README What's Included table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PR review comments: align eval rubric with expect_tools and sync marketplace.json - Updated template-discovery eval.yaml rubric to match expect_tools (removed 'or template_search' since expect_tools enforces template_from_intent) - Made .claude-plugin/marketplace.json byte-for-byte identical to .github/plugin/marketplace.json (removed extra trailing newline) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments: strengthen eval assertions and fix rubrics - Add exit_success assertions to all eval scenarios - Add file_exists and file_contains assertions for template-instantiation - Add output_contains for shortName and datatype in template-authoring - Fix rubrics to reference actual tools (skill + dotnet new CLI) instead of non-existent MCP tools (template_from_intent, template_inspect, template_instantiate) which caused the judge to penalize quality scores - Update expect_tools from non-existent MCP tools to 'skill' - Change plugin version from 1.2.0 to 0.1.0 for consistency with other plugins - Remove extra blank line in README.md after plugin table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rewrite skills and agent to use dotnet new CLI instead of phantom MCP tools The SKILL.md files and agent.md referenced MCP tools (template_from_intent, template_inspect, template_instantiate, template_dry_run, etc.) that are not available at runtime since no MCP server is declared in plugin.json. This caused the agent to waste tokens attempting to find these tools before falling back to dotnet new CLI commands. Changes: - template-discovery/SKILL.md: Replace MCP tool calls with dotnet new search, dotnet new list, dotnet new --help, and dotnet new --dry-run. Add intent-to-template keyword mapping table. - template-instantiation/SKILL.md: Replace template_instantiate with dotnet new CLI commands. Add explicit CPM adaptation steps and multi-project composition with dotnet sln/dotnet add reference. - template-authoring/SKILL.md: Replace template_create_from_existing with manual template.json creation guidance including example JSON. Replace template_dry_run/template_instantiate with CLI equivalents. - template-engine.agent.md: Replace Available Tools table with CLI Commands Reference. Update all workflow steps to use CLI commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix file assertion paths: remove ./ prefix for glob matcher compatibility Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |