mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
YuliiaKovalova
74c58505e5
* Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills Fix inaccurate reserved-shortName guidance, consolidate validation rules into a single skill, expand discovery mappings, add explicit CPM/version steps, and introduce template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .codex-plugin manifest indentation; add evals for new skills Make .codex-plugin/plugin.json byte-consistent with plugin.json (2-space indent on the agents line). Add eval.yaml + eval.vally.yaml capability evals for the new template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: source reserved list from dotnet new --help, note workload/package availability, tighten version-refresh - Clarify the reserved shortName set is the current dotnet new subcommands (authoritative source: dotnet new --help); create is verified as a real subcommand (alias behind dotnet new <template>). - template-discovery: note that some mapped short names (maui, winui3, aspire, func, orleans) need workloads/template packages, with fallback to dotnet new list/search. - template-instantiation: keep template versions by default; if refreshing, use dotnet list package --outdated + user confirmation and constrain to same major/minor rather than always latest stable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address 2nd review round: enforce eval negatives, split combined assertion, reframe reserved list - smart-defaults evals: enforce --no-https absence (auth scenario), absence of minimal-API flag (controllers scenario), and no newer --framework TFM when net8.0 is explicitly required, using output_not_contains/output_not_matches. - comparison eval: split the combined (auth|aot|docker|controllers) check into four separate output_matches assertions so partial comparisons fail. - template-validation/authoring: reframe the reserved shortName list as non-exhaustive examples and source the authoritative set from dotnet new --help; drop the specific create-alias assertion in favor of parsing-ambiguity wording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make template-comparison evals robust to CI SDK currency Switch the Blazor comparison scenario from blazorserver (absent in the CI SDK) to blazor (Blazor Web App) vs blazorwasm, both reliably present in .NET 8+, and instruct the agent to inspect each via --help. Update the SKILL.md example reference for currency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CLI-failure resilience guidance to discovery/comparison skills The isolated eval runs failed because the agent ran 'dotnet new <t> --help', hit the template engine's global-mutex/persistence error (common when the command runs concurrently in a sandbox), and then returned no answer at all. Instruct both skills to run 'dotnet new' calls sequentially, retry once on a transient mutex/persistence error, and fall back to the intent/parameter mapping so a concrete answer is always produced instead of empty output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: accurate flags + robust negative-assertion prompts - template-smart-defaults SKILL.md: drop the non-existent --publish-aot flag. Clarify --aot is a dotnet new flag only on templates that expose it (console/worker/grpc, not webapi) and that publish-time AOT is the MSBuild PublishAot=true property, not a dotnet new flag. - template-discovery SKILL.md: replace the hardcoded --enable-docker mapping (not a real flag on common templates) with generic 'confirm with --help'. - smart-defaults evals: tighten the negative-assertion prompts to output only the command line and not mention unused flags, so a negated explanation can't trip output_not_contains/output_not_matches. Switch the AOT scenario from webapi to worker (which actually supports --aot). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 3: scope negative checks, de-emphasize stale lists - smart-defaults evals: anchor the negative assertions to the 'dotnet new' command line (same-line regex) instead of whole-output substring/regex, so a flag mentioned only in prose can't fail the test. - template-validation / template-authoring: mark the dotnet new subcommand examples as illustrative/version-dependent and tell readers not to hardcode them; the live 'dotnet new --help' output is canonical. - template-comparison: fix the example table's AOT row — webapi/webapp do not expose a --aot template flag; native AOT is publish-time via PublishAot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
70 lines
3.6 KiB
YAML
70 lines
3.6 KiB
YAML
scenarios:
|
|
- name: "AOT implies a compatible framework"
|
|
prompt: "Create a native AOT worker service. I haven't picked a target framework — choose an appropriate one for me and explain why. Don't write any files yet, just tell me the parameters you'd use."
|
|
assertions:
|
|
- type: "exit_success"
|
|
- type: "output_matches"
|
|
pattern: "(?i)(aot)"
|
|
- type: "output_matches"
|
|
pattern: "(?i)net(8|9|10)\\.0"
|
|
rubric:
|
|
- "The agent used the template-smart-defaults skill to fill the cross-parameter gap"
|
|
- "Because AOT was requested, the agent also selected a target framework"
|
|
- "The agent chose the latest AOT-compatible framework available rather than an arbitrary one"
|
|
- "The agent explained the rationale (native AOT needs a recent AOT-capable TFM)"
|
|
- "The agent confirmed it would verify parameter names against `dotnet new <template> --help`"
|
|
expect_tools:
|
|
- skill
|
|
timeout: 180
|
|
|
|
- name: "Auth implies HTTPS stays enabled"
|
|
prompt: "I want to create a webapi with Individual authentication. First output the exact single `dotnet new` command line you would run — include only the flags you will actually pass, and do not mention any flags you are not using. Then give a brief explanation. Don't create anything yet."
|
|
assertions:
|
|
- type: "exit_success"
|
|
- type: "output_matches"
|
|
pattern: "(?i)individual"
|
|
- type: "output_not_matches"
|
|
pattern: "(?i)dotnet new[^\\n]*--no-https"
|
|
rubric:
|
|
- "The agent used the template-smart-defaults skill"
|
|
- "Because auth is not None, the agent did NOT add a --no-https flag"
|
|
- "The agent explained that authentication flows require HTTPS"
|
|
- "The agent only filled gaps and logged each applied default"
|
|
expect_tools:
|
|
- skill
|
|
timeout: 180
|
|
|
|
- name: "Controllers exclude the minimal-API flag"
|
|
prompt: "Create a web API that uses controllers. First output the exact single `dotnet new` command line you would run — include only the flags you will actually pass, and do not mention any flags you are not using. Then explain. Do not create files."
|
|
assertions:
|
|
- type: "exit_success"
|
|
- type: "output_matches"
|
|
pattern: "(?i)(use-controllers|controllers)"
|
|
- type: "output_not_matches"
|
|
pattern: "(?i)dotnet new[^\\n]*--[a-z-]*minimal"
|
|
rubric:
|
|
- "The agent used the template-smart-defaults skill"
|
|
- "The agent passed the controllers flag"
|
|
- "The agent did NOT also pass a minimal-API flag (the two are mutually exclusive)"
|
|
- "The agent explained the reasoning"
|
|
expect_tools:
|
|
- skill
|
|
timeout: 180
|
|
|
|
- name: "Never override an explicit user value"
|
|
prompt: "Create a native AOT console app, but I specifically want it targeting net8.0 — keep it on net8.0 no matter what. First output the exact single `dotnet new` command line you would run — include only the flags and framework value you will actually pass, and do not mention any flags or framework versions you are not using. Then tell me the parameters."
|
|
assertions:
|
|
- type: "exit_success"
|
|
- type: "output_contains"
|
|
value: "net8.0"
|
|
- type: "output_not_matches"
|
|
pattern: "(?i)dotnet new[^\\n]*--framework[ =]+net(9|10)\\.0"
|
|
rubric:
|
|
- "The agent used the template-smart-defaults skill"
|
|
- "The agent respected the explicit net8.0 framework the user set"
|
|
- "The agent did NOT override the user value with a newer framework despite AOT being requested"
|
|
- "The agent explained that smart defaults only fill gaps and never override explicit user choices"
|
|
expect_tools:
|
|
- skill
|
|
timeout: 180
|