mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
YuliiaKovalova
74c58505e5
* Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills Fix inaccurate reserved-shortName guidance, consolidate validation rules into a single skill, expand discovery mappings, add explicit CPM/version steps, and introduce template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .codex-plugin manifest indentation; add evals for new skills Make .codex-plugin/plugin.json byte-consistent with plugin.json (2-space indent on the agents line). Add eval.yaml + eval.vally.yaml capability evals for the new template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: source reserved list from dotnet new --help, note workload/package availability, tighten version-refresh - Clarify the reserved shortName set is the current dotnet new subcommands (authoritative source: dotnet new --help); create is verified as a real subcommand (alias behind dotnet new <template>). - template-discovery: note that some mapped short names (maui, winui3, aspire, func, orleans) need workloads/template packages, with fallback to dotnet new list/search. - template-instantiation: keep template versions by default; if refreshing, use dotnet list package --outdated + user confirmation and constrain to same major/minor rather than always latest stable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address 2nd review round: enforce eval negatives, split combined assertion, reframe reserved list - smart-defaults evals: enforce --no-https absence (auth scenario), absence of minimal-API flag (controllers scenario), and no newer --framework TFM when net8.0 is explicitly required, using output_not_contains/output_not_matches. - comparison eval: split the combined (auth|aot|docker|controllers) check into four separate output_matches assertions so partial comparisons fail. - template-validation/authoring: reframe the reserved shortName list as non-exhaustive examples and source the authoritative set from dotnet new --help; drop the specific create-alias assertion in favor of parsing-ambiguity wording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make template-comparison evals robust to CI SDK currency Switch the Blazor comparison scenario from blazorserver (absent in the CI SDK) to blazor (Blazor Web App) vs blazorwasm, both reliably present in .NET 8+, and instruct the agent to inspect each via --help. Update the SKILL.md example reference for currency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CLI-failure resilience guidance to discovery/comparison skills The isolated eval runs failed because the agent ran 'dotnet new <t> --help', hit the template engine's global-mutex/persistence error (common when the command runs concurrently in a sandbox), and then returned no answer at all. Instruct both skills to run 'dotnet new' calls sequentially, retry once on a transient mutex/persistence error, and fall back to the intent/parameter mapping so a concrete answer is always produced instead of empty output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: accurate flags + robust negative-assertion prompts - template-smart-defaults SKILL.md: drop the non-existent --publish-aot flag. Clarify --aot is a dotnet new flag only on templates that expose it (console/worker/grpc, not webapi) and that publish-time AOT is the MSBuild PublishAot=true property, not a dotnet new flag. - template-discovery SKILL.md: replace the hardcoded --enable-docker mapping (not a real flag on common templates) with generic 'confirm with --help'. - smart-defaults evals: tighten the negative-assertion prompts to output only the command line and not mention unused flags, so a negated explanation can't trip output_not_contains/output_not_matches. Switch the AOT scenario from webapi to worker (which actually supports --aot). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 3: scope negative checks, de-emphasize stale lists - smart-defaults evals: anchor the negative assertions to the 'dotnet new' command line (same-line regex) instead of whole-output substring/regex, so a flag mentioned only in prose can't fail the test. - template-validation / template-authoring: mark the dotnet new subcommand examples as illustrative/version-dependent and tell readers not to hardcode them; the live 'dotnet new --help' output is canonical. - template-comparison: fix the example table's AOT row — webapi/webapp do not expose a --aot template flag; native AOT is publish-time via PublishAot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
61 lines
2.9 KiB
YAML
61 lines
2.9 KiB
YAML
scenarios:
|
|
- name: "Compare webapi vs webapp side by side"
|
|
prompt: "I'm deciding between the 'webapi' and 'webapp' templates for a new .NET project. Compare them side by side — show parameters, which supports authentication, AOT, Docker, and controllers, and the available frameworks. Then recommend one for a JSON REST backend."
|
|
assertions:
|
|
- type: "exit_success"
|
|
- type: "output_matches"
|
|
pattern: "(?i)webapi"
|
|
- type: "output_matches"
|
|
pattern: "(?i)webapp"
|
|
- type: "output_matches"
|
|
pattern: "(?i)auth"
|
|
- type: "output_matches"
|
|
pattern: "(?i)aot"
|
|
- type: "output_matches"
|
|
pattern: "(?i)docker"
|
|
- type: "output_matches"
|
|
pattern: "(?i)controllers"
|
|
rubric:
|
|
- "The agent used the template-comparison skill to compare the two templates"
|
|
- "The agent inspected each template with `dotnet new <template> --help`"
|
|
- "The agent produced a side-by-side comparison table (not just two separate lists)"
|
|
- "The comparison covered parameters, feature support (auth/AOT/Docker/controllers), and frameworks"
|
|
- "The agent gave a clear recommendation for a JSON REST backend"
|
|
expect_tools:
|
|
- skill
|
|
timeout: 180
|
|
|
|
- name: "Choose between blazor and blazorwasm"
|
|
prompt: "Which template should I use for an app that needs offline support: 'blazor' (Blazor Web App) or 'blazorwasm' (Blazor WebAssembly Standalone)? Inspect both with `dotnet new <template> --help`, compare the interactivity/hosting trade-offs, and recommend one."
|
|
assertions:
|
|
- type: "exit_success"
|
|
- type: "output_matches"
|
|
pattern: "(?i)\\bblazor\\b"
|
|
- type: "output_matches"
|
|
pattern: "(?i)blazorwasm"
|
|
rubric:
|
|
- "The agent used the template-comparison skill to compare both Blazor templates"
|
|
- "The agent inspected each template's parameters via `dotnet new <template> --help`"
|
|
- "The agent explained the hosting/interactivity trade-offs (server-rendered vs WebAssembly)"
|
|
- "The agent connected the offline-support requirement to a concrete recommendation"
|
|
expect_tools:
|
|
- skill
|
|
timeout: 180
|
|
|
|
- name: "Decide which template fits a background processing scenario"
|
|
prompt: "I need to run a long-lived background job that processes a queue. Which template should I use — 'worker' or 'console'? Compare them and justify the choice."
|
|
assertions:
|
|
- type: "exit_success"
|
|
- type: "output_matches"
|
|
pattern: "(?i)worker"
|
|
- type: "output_matches"
|
|
pattern: "(?i)console"
|
|
rubric:
|
|
- "The agent used the template-comparison skill rather than just creating a project"
|
|
- "The agent compared worker and console templates side by side"
|
|
- "The agent explained why the worker template fits a long-lived queue processor (hosted service lifecycle)"
|
|
- "The agent did NOT create any project files"
|
|
expect_tools:
|
|
- skill
|
|
timeout: 180
|