mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-gpt55
156 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Abhitej John
|
1ca10c7b7e | held-out grid (gpt55): replace eval.vally.yaml with 5 fresh untuned scenarios | ||
|
Abhitej John
|
5429c2f621 |
Add 2 fresh mixed-intent boundary fixtures; revert executor to opus-4.6
Closes the 'refactoring-only endpoint too narrow' risk flagged by the
cross-family rubber-duck. Adds two boundary stimuli never used in trim tuning:
9. Separate a smuggled behavior change from a requested rename (rename +
smuggled 10%->12% discount change) -> should separate/flag, not bundle.
10. Recognize a behavior-changing simplification is not a refactor (remove
free-shipping-over-\ rule under a 'cleanup' framing) -> should not
silently change observable behavior.
Executor reverted to claude-opus-4.6 / judge gpt-5.5 (re-judge) to match the
stringent Confirm A family where boundary behavior looked weakest.
Eval-branch only; not on PR #873.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 3c9f9823-7f2f-4f7d-9d1b-f2b9e7a20c60
|
||
|
Abhitej John
|
c6b09b071b |
Add vally evals + cost-trim for dotnet refactoring/breaking-change skills
CI-only eval branch (NOT part of PR #873). Carries the uncommitted cost-trim + PR-comment fix on both SKILL.md, the eval.yaml rubric re-alignment, and two new eval.vally.yaml specs so the vally CI can evaluate whether the current skill content helps before merging. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 3c9f9823-7f2f-4f7d-9d1b-f2b9e7a20c60 |
||
|
Abhitej John
|
a5265d8600 | Merge branch 'main' into add-dotnet-refactoring-skills | ||
|
Aditya Mandaleeka
|
52ba152ed6 | Update to Vally 0.7.0 (#854) | ||
|
Abhitej John
|
1b3a5bd13d |
Add eval tests for csharp-refactoring and dotnet-breaking-changes skills
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
1270722c95 |
Fix migrate-static-to-wrapper skill activation on evals (#864)
Sharpen the description boundary between migrate-static-to-wrapper and generate-testability-wrappers so prompts that replace call sites when the abstraction is already registered in DI activate the migration skill. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0451fe8d91 |
dotnet-test: raise timeout for assertion-quality decline-write-tests scenario 300->360 (#862)
The 'Decline request to write new tests from scratch' non-activation scenario writes a full MSTest suite from scratch (code generation) and was hitting its 300s wall-clock timeout. Per InvestigatingResults.md, code-generation scenarios often need 360s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
e558acb8e8 |
dotnet-test: raise timeout for boundary comparison assertions eval scenario (#863)
The 'Use comparison assertions for boundary testing' scenario chronically hit its 240s wall-clock timeout. Raise it to 360s to match the other code-generation scenarios in this eval file, per InvestigatingResults.md timeout guidance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
a60d989504 |
Fix mtp-hot-reload SDK 9 eval timeout by removing expect_tools bash (#861)
The 'Suggest hot reload for failing test in MTP project (SDK 9)' vally stimulus forced bash usage on a purely advisory question, triggering a slow SDK 9.0.200 NuGet restore that exceeded the timeout. Align with the source-of-truth eval.yaml which does not constrain this scenario. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ce75c35569 |
dotnet-test: raise timeouts for chronic-timeout eval scenarios (#850)
Several dotnet-test eval scenarios repeatedly hit their wall-clock timeout in the evaluation dashboard (skilled + plugin arms), producing empty output and failing all assertions. Increase per-scenario timeouts for the ones that timed out most often across recent runs: - writing-mstest-tests: 'Write unit tests for a service class', 'Write data-driven tests for a calculator', 'Use string assertions for format validation' 180->360; 'Use comparison assertions for boundary testing' 120->240 - coverage-analysis: 'Coverage plateau diagnosis', 'Project-wide coverage analysis with existing Cobertura data' 300->480 - test-gap-analysis: 'Decline request to write new tests from scratch' 120->300; 'Acknowledge well-tested code with few surviving mutations' 300->420 - run-tests: 'Filter xUnit v3 tests by trait on MTP', 'Filter xUnit v3 tests by class pattern and trait using query filter language' 120->240 Per eng/skill-validator/src/docs/InvestigatingResults.md, timeouts are the highest-priority failure mode; code-generation scenarios often need 360s. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
a677ec43d2 |
Consolidate dotnet-test-frameworks into test-analysis-extensions (#851)
* Consolidate dotnet-test-frameworks into test-analysis-extensions Remove the orphaned dotnet-test-frameworks reference skill, whose content was a duplicate subset of test-analysis-extensions/extensions/dotnet.md. Nothing actually loaded it. Redirect the remaining references and drop its eval tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Make exp-test-maintainability Step 1 self-contained and fix activation Address PR review: exp-test-maintainability (dotnet-experimental plugin) must not point at test-analysis-extensions (dotnet-test plugin) — inline the .NET framework markers instead. Also strengthen the description/When-to-Use triggers so the 'each new case needs a whole new method / suggest a better structure' scenario reliably activates the skill instead of routing to a test-writing skill. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Revert exp-test-maintainability description broadening; keep self-contained markers The description broadening regressed activation (3/4 -> 0/4 across a variance-dominated single eval run, CV up to 924%). Description is the activation lever, so restore the original wording and keep only the review-comment fix (inlined .NET framework markers in Step 1). Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix scenario 1 eval: use project fixture so the skill activates Scenario 1 pasted a self-contained snippet inline and asked to 'suggest a better structure', which the base model answers directly without loading the skill -> NOT ACTIVATED with no headroom. Convert it to the project-fixture pattern used by the activating scenarios (3 & 4): move the repetitive InputValidatorTests into a Validation.Tests fixture and ask the model to analyze the project. Mirror the change in eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Fix scenario 2 eval: use project fixture so the skill activates Scenario 2 also pasted a self-contained snippet inline (well-maintained Auth.Tests) and asked to 'review these tests', which the base model handles without loading the skill. Convert to the project-fixture pattern used by the activating scenarios; add fixtures/well-maintained/Auth.Tests. Mirror in eval.vally.yaml. All four scenarios now reference a project on disk. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Add reject_tools to exp-test-maintainability evals to create headroom All four scenarios scored NOT ACTIVATED / no-headroom (#8 in InvestigatingResults.md): baselines are already 4.0-5.0/5, so loading the skill only adds token/tool overhead with no offsetting quality gain. Match the convention of the sibling analysis skill test-anti-patterns: reject bash/edit/create on every scenario. This skill is analysis-only ('do not modify any files'), so the constraint is a no-op for correct behavior but levels the playing field between baseline and skilled runs — scoring focuses on answer quality (weight 0.40+0.30) instead of tool-induced overhead. Applied to both eval.yaml and eval.vally.yaml. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * Address review: make well-maintained fixture self-contained and deterministic - Import System.Security.Authentication for AuthenticationException - Replace wall-clock assertion (token.ExpiresAt > DateTime.UtcNow) with a deterministic one anchored on a fixed clock (FixedNow), and drive the injected FakeClock from that fixed time. This removes the flakiness and makes the FakeClock injection meaningful — a better exemplar for the 'well-maintained' scenario. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> |
||
|
Wendy Breiding
|
4975793af0 |
Moving niche skills to dotnet-advanced (#807)
* Add csharp-development skill and update CODEOWNERS and README.md - Introduced a new skill for C# development guidance. - Updated CODEOWNERS to include reviewers for the new skill. - Modified README.md to list the new csharp-development skill. - Added evaluation scenarios for the csharp-development skill. * Refactor C# evaluation scenarios to enhance production code review and validation guidance * Refactor production code review scenarios to improve clarity and focus on async error handling * Update CODEOWNERS to reflect current ownership and streamline reviewer assignments * Remove csharp-development skill documentation and evaluation tests * Remove csharp-development skill from README * Fix formatting in Constants.g.cs by adding missing semicolon * Revert Constants.g.cs to main * Remove csharp-development skill from CODEOWNERS * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Update CODEOWNERS to include additional reviewers for dotnet-ai skills * Move dotnet skills to dotnet-util to keep dotnet plugin focused and targeted. * Add CODEOWNERS, SKILL.md, package-types.md, and publish-workflow.md for NuGet trusted publishing; implement eval.yaml for dotnet-pinvoke and dotnet-util skills * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Update README files to enhance plugin descriptions and remove outdated LSP section * Rename repository title from 'dotnet' to 'dotnet-util' in README.md for clarity. * Refine README.md to clarify included .NET skills and remove redundant descriptions * Add 'dotnet-util' plugin for utility skills in .NET development * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add 'dotnet-util' plugin for utility skills in .NET development * Add dotnet-util to Codex marketplace manifest * Align dotnet-util marketplace descriptions * Refactor CODEOWNERS and update README for nuget-trusted-publishing; add evaluation scenarios and workflow template for trusted publishing * Add P/Invoke diagnostics and type mapping references; implement NuGet trusted publishing setup - Created diagnostics.md for common pitfalls and failure modes in P/Invoke. - Added type-mapping.md for native-to-.NET type mappings. - Introduced nuget-trusted-publishing skill for setting up OIDC-based NuGet publishing. - Added package-types.md detailing structural requirements for various NuGet package types. - Created publish-workflow.md as a template for GitHub Actions workflows for NuGet publishing. * Add evaluation YAML files for csharp-scripts, dotnet-pinvoke, and nuget-trusted-publishing skills * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Complete nuget-trusted-publishing move: remove from dotnet, add to dotnet-specialized README * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Remove eval.vally.yaml and eval.yaml files for nuget-trusted-publishing skill * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add NuGet trusted publishing skill and related references - Introduced `nuget-trusted-publishing` skill for setting up OIDC-based trusted publishing on GitHub Actions. - Added detailed documentation in SKILL.md covering prerequisites, process, safety rules, and troubleshooting. - Created `package-types.md` to outline structural requirements for various NuGet package types. - Developed `publish-workflow.md` as a template for GitHub Actions workflows for NuGet publishing. - Removed obsolete `dotnet-specialized` plugin and updated `dotnet` plugin description. - Added evaluation tests for `nuget-trusted-publishing` skill to ensure proper guidance for users. - Implemented evaluation tests for `dotnet-pinvoke` and `csharp-scripts` skills to validate functionality. * Update dotnet plugin description to include high-level .NET development skills * Add NuGet trusted publishing and P/Invoke documentation - Introduced type mapping reference for native-to-.NET types in dotnet-pinvoke. - Added a comprehensive guide for setting up NuGet trusted publishing using OIDC in GitHub Actions. - Created a reference document detailing structural requirements for various NuGet package types. - Provided a template for GitHub Actions workflow for publishing NuGet packages with trusted publishing. - Implemented evaluation scenarios for testing the dotnet-pinvoke and nuget-trusted-publishing skills, including prompts and assertions for expected outputs. * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add CODEOWNERS entries for setup-local-sdk * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Update dotnet README to list setup-local-sdk skill * Add setup-local-sdk to skills section in README * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Javier Calvarro Nelson
|
9ed853bffc |
Port dotnet-blazor evals to Vally format (#843)
Adds eval.vally.yaml for all 9 dotnet-blazor skills, converted from the legacy skill-validator eval.yaml specs, and registers a dotnet-blazor suite in .vally.yaml so the shadow Vally pipeline discovers them. Conversion preserves prompts, assertions, and rubric verbatim. Legacy setup blocks map to Vally environment config: 'dotnet new' scaffolding commands become environment.commands. Verified end-to-end by running the author-component eval (baseline + skilled + adapt), which passed the shadow threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
26f01b5d5c |
Add eval coverage for dotnet-test/writing-mstest-tests (#828)
* Add eval coverage for dotnet-test/writing-mstest-tests Enrich the ServiceRegistry collection/null/reference scenario with deterministic file_contains assertions for the modern MSTest assertion APIs Assert.IsNull, Assert.AreSame, Assert.Contains, Assert.DoesNotContain, Assert.IsEmpty, and Assert.IsNotEmpty. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reject legacy CollectionAssert/StringAssert helpers in collection scenario Address PR review: file_contains is a case-sensitive substring check, so the Assert.Contains and Assert.DoesNotContain value checks would also pass for CollectionAssert.* and StringAssert.* helpers. Add file_not_contains guards to keep the scenario a deterministic check for the modern Assert.* API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
dc9261f483 |
Add eval coverage for dotnet-test/run-tests (#822)
* Add eval coverage for dotnet-test/run-tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Consolidate redundant rubric bullets in run-tests eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
988f5d24ee |
Add eval coverage for dotnet-test/generate-testability-wrappers (#821)
* Add eval coverage for dotnet-test/generate-testability-wrappers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reword ambient-context prompt to avoid DO-NOT-USE activation trigger Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7b02af3fcf |
Add eval coverage for dotnet-test/filter-syntax (#824)
* Add eval coverage for dotnet-test/filter-syntax Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: align xUnit v3 fixtures and tighten filter-syntax assertions Match known-good MTP xUnit v3 fixtures, enforce the real Trait annotation and SDK 9 separator, and tighten the query-filter scenario with a realistic fixture. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
32ec973eeb |
Add eval coverage for dotnet-test/test-smell-detection (#826)
* Add eval coverage for dotnet-test/test-smell-detection Adds a scenario plus fixture covering concrete fix examples, contextually-obvious-number calibration, justified severity levels, and reasoned vs bare skip distinction. Skill coverage now 100%. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: tighten Sleepy regex and block bash for read-only scenario Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
29f09cc6e4 |
Add eval coverage for dotnet-test/grade-tests (#825)
* Add eval coverage for dotnet-test/grade-tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add missing per-test grade assertions to grade-tests evals Address PR review: verify TestParse_NoError=C in scenario 4, and Charge_NegativeAmount=A / Refund_ExistingCharge=C in scenario 5 so the scenarios cannot pass on misgraded output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
91d68cbd5e |
Add eval coverage for dotnet-test/test-gap-analysis (#829)
* Add eval coverage for dotnet-test/test-gap-analysis Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Mark InvoiceProcessor as partial to match generated part Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d16bad471c |
Add eval coverage for dotnet-test/dotnet-test-frameworks (#820)
* Add eval coverage for dotnet-test/dotnet-test-frameworks Add a deterministic output_matches assertion for Assert.AreEqual in the MSTest try/catch refactor scenario, covering the previously-uncovered MSTest assertion CodePattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten Assert.AreEqual assertion to require message comparison Address review feedback: the output_matches pattern now requires the exception message comparison (Assert.AreEqual(..., ex.Message)) rather than matching Assert.AreEqual anywhere. The CodePattern remains covered via the rubric evidence. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
5e264c36fa |
Add eval coverage for dotnet-test/test-tagging (#830)
Enrich the NUnit misleading-names scenario with rubric items and an output_matches assertion covering the [Category] code pattern, reading the test body before classifying, and classifying each test method. Add a Go standard-testing scenario (report-only) with a fixture to cover leaving source files unmodified for report-only frameworks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ddb2487e7b |
Add eval coverage for dotnet-test/test-anti-patterns (#823)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
4ca17dc05b |
Add eval coverage for dotnet-test/migrate-static-to-wrapper (#827)
* Add eval coverage for dotnet-test/migrate-static-to-wrapper Extend eval.yaml with rubric items and two scenarios covering: required using directives, build-succeeds-after-migration, test-double updates, no-behavioral-changes, not breaking static classes, and incremental migration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review on migrate-static-to-wrapper eval Restore expect_activation: false on the decline scenario and add missing using BillingApp; to the test fixture so it starts buildable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f0f42a7918 |
Add eval coverage for dotnet-test/assertion-quality (#819)
Adds outcome-focused rubric items so two previously-uncovered Validation teaching points become covered: - Metrics are computed correctly (counts add up) - If the suite has good diversity, the report acknowledges this Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Gerald Versluis
|
72c0d2c7b7 |
Add setup-local-sdk skill for global.json paths feature (#508)
* Rebase setup-local-sdk skill Replay PR #508 onto current main and address the remaining setup-local-sdk review feedback. The install examples now use fail-fast temporary downloads, the team scripts preserve existing global.json content, and eval assertions cover the critical paths/version/workload behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address setup-local-sdk review comments Add the missing skill license metadata and fix the PowerShell .gitignore regex examples so they correctly detect an existing .dotnet/ entry. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Stabilize setup-local-sdk evals Reduce overfit-prone assertions, make incompatible host handling explicit, configure base SDK resolution before slow workload installs, and tighten exact-version/team-script validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address setup-local-sdk follow-up review Include the shared dotnet skill reviewer team in CODEOWNERS, make .gitignore appends newline-safe, and tighten the incompatible-host eval to reward stopping before local install commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten exact local SDK eval Measure project-level SDK resolution and exact roll-forward behavior using global.json artifact assertions so the scenario distinguishes PATH-based local installs from SDK paths setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address install script review feedback Use the documented lowercase dotnet-install quality value and call out both bash and PowerShell official install script URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Increase team script eval timeout Give the team install script scenario the same realistic long-running timeout as other scenarios that install preview SDK bits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add Windows local SDK check Document the PowerShell equivalent for detecting an existing local SDK and give exact SDK installs enough time to complete in eval. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Stabilize team script eval Focus the team script scenario on script/config generation, create reproducible setup guidance before long downloads, and avoid brittle transcript assertions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make mktemp usage portable Use BSD/macOS-compatible mktemp templates in setup-local-sdk bash snippets and generated team install script. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
35ace775b3 |
Add dotnet-test-migration plugin and move migration skills there (#808)
Move the .NET test framework/platform migration skills (migrate-mstest-v1v2-to-v3, migrate-mstest-v3-to-v4, migrate-vstest-to-mtp, migrate-xunit-to-mstest, migrate-xunit-to-xunit-v3) and the test-migration orchestrator agent out of dotnet-test into a new dedicated dotnet-test-migration plugin, along with their evals. Update all marketplace manifests, READMEs, CODEOWNERS, and .vally.yaml accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
33110eeba8 |
code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric (#806)
* code-testing-agent: fix workspace-integrity activation + stabilize Contoso rubric
Workspace-integrity scenario was NOT ACTIVATED in plugin mode: the prompt
("Generate unit tests for its core module") was terse and small-scoped, so
the runtime did not route to the code-testing-agent skill. Reframe it with
high-level test-generation language ("comprehensive pytest test suite",
"scaffold", "thorough unit tests") that matches the skill description,
while preserving the guardrail anchors: it still points at the on-disk module
without naming it and never implies restoring the gutted tree.
ContosoUniversity rubric item #5 (find-untested-sources) was conditional on
that skill being loaded — only true in plugin mode. In isolated runs the
agent cannot satisfy it, so the judge penalized it asymmetrically, injecting
isolated-vs-plugin variance. Make the conditional deterministic: explicitly
N/A when the skill is not loaded, without lowering the bar when it is.
Applied to both eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* code-testing-agent: make find-untested-sources rubric decidable from session timeline
Address review feedback: the 'treat as satisfied (N/A) when the skill is
not loaded' clause is not verifiable from the judge's inputs (which show
which tools were called, not which were available). Rewrite the criterion
to be decidable from the session timeline by requiring a source-to-test
pairing map recorded in .testagent/research.md that either cites
find-untested-sources output or documents the equivalent manual approach.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
29bfae5f5b |
run-tests: fix evals (query-filter regex, sibling skills, command observability) (#800)
* run-tests: fix evals (query-filter regex, sibling skills, command observability)
Investigated run-tests eval failures by running the validator locally.
- SKILL.md Step 3: document xUnit v3 --filter-query so the agent stops
answering that complex xUnit v3 filters 'cannot be combined'.
- eval.yaml: fix a broken assertion regex. The query-filter pattern is a
single-quoted YAML scalar using '\\s'/'\\[', which (unlike a double-quoted
scalar) is NOT unescaped, so the regex searched for a literal '\s' and
could never match. Corrected to single backslashes.
- eval.yaml: add additional_required_skills (filter-syntax / platform-detection)
to the filter and detection scenarios, so the isolated arm loads the sibling
reference skills that run-tests explicitly defers to.
- eval.yaml: ask the agent to show the exact command in execute-style prompts.
output_matches only sees the final assistant message; 'run my tests' prompts
make the agent execute and summarize ('tests passed'), so the recommended
command never appears. The assertions still catch wrong commands.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* run-tests: rewrite description for reliable plugin-arm activation
The run-tests skill activated reliably in the isolated eval arm but
unreliably in the plugin arm (all dotnet-test sibling descriptions
loaded), so every scenario's pluginImprovementScore went negative and
dragged min(isolated, plugin) below zero.
Lead the description with natural-language intent triggers that mirror
how the eval prompts phrase requests (run all tests, run a subset via
filters, produce TRX reports, collect crash/hang dumps, run a single
TFM) instead of opening with platform-detection mechanism, and add
explicit DO NOT USE redirects to code-testing-agent / mtp-hot-reload.
Stays under the 1024-char description cap.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* dotnet-test: fix run-tests plugin-arm activation via skill-menu budget
Root cause (verified against the Copilot CLI SDK skill renderer): the
model-facing skill menu has a 15000-char budget. Skills are listed
alphabetically and emitted with full descriptions only until the budget
is exhausted; the rest collapse to bare names with no description and
effectively cannot be model-activated. With 27 dotnet-test skills,
run-tests (alphabetical position ~20) fell into the name-only overflow,
so it never activated in the plugin eval arm even though it activated
reliably in isolation. This is a real user-facing discoverability bug,
not just an eval artifact.
Fix: hide reference/primitive skills that are never meant to be
model-invoked from the menu via 'disable-model-invocation: true', which
the SDK filters out of the budget entirely:
- filter-syntax, platform-detection, dotnet-test-frameworks,
code-testing-extensions, test-analysis-extensions — already
user-invocable:false reference data ('DO NOT USE directly').
- find-untested-sources, find-untested-sources-polyglot — researcher
primitives invoked by-name from the code-testing-researcher agent
(which has a manual fallback); no standalone evals.
These remain invocable by explicit name (agents/users), only auto-
suggestion is suppressed.
This frees enough budget that run-tests (plus migrate-xunit-to-xunit-v3
and mtp-hot-reload) now receive full descriptions; no previously-visible
skill regresses. Also trimmed the run-tests description so its menu block
fits with margin while keeping all activation triggers.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
102663d337 |
Fix code-testing-agent activation for the Flask pytest scenario (#802)
* Fix code-testing-agent activation for the Flask pytest scenario The 'Generate pytest tests for the Flask tasks API' scenario failed to activate code-testing-agent in BOTH isolated and plugin mode: its prompt enumerated, file by file, exactly what to mock/inject/test (TaskService with repo mocked + clock injected, queries.apply_query over fixed lists, both repositories, the blueprint via test_client), acting as an answer key that let the base agent generate tests directly with edit tools instead of routing to the skill's research-plan-implement pipeline. Rewrite the prompt to a realistic, high-level ask (mirroring the ContosoUniversity scenario that does activate): describe the app at a layer level, keep the 'no tests yet', project-wide multi-file framing and the 80% coverage floor, and drop the per-module test checklist. Assertions, rubric and timeout are unchanged. Verified locally that the skill now activates in both isolated and plugin mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use python3 in Flask prompt to match the grader Review feedback: the prompt told the agent to run \python -m ...\ but the grader (and the vally command) invoke \python3\. On Linux runners that may lack a \python\ shim the agent could hit command-not-found. Align the prompt to \python3\ in both eval.yaml and eval.vally.yaml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
e3f0becd4c |
Fix test-gap-analysis skill activation in plugin eval runs (#801)
* Fix test-gap-analysis skill activation in plugin eval runs In plugin-mode eval runs (all dotnet-test skills loaded), test-gap-analysis lost activation: 'Acknowledge well-tested code' was stolen by assertion-quality and 'logic/null-check gaps' loaded no skill at all. Front-load the concrete 'would my tests catch a bug if the code changed' trigger phrasing the prompts use, and add a DO NOT USE -> test-gap-analysis redirect in assertion-quality for mutation-style reasoning. Both descriptions stay within the 1024-char cap; aggregate unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore Jest/TS activation signal for assertion-quality polyglot scenario The earlier trim of assertion-quality's description dropped the Jest/Vitest and toBeTruthy() signals, breaking plugin-mode activation for the 'Polyglot: Jest/TypeScript' scenario (agent answered directly, loading no skill). Restore those polyglot signals, and make the scenario prompt realistic — the prior prompt enumerated the full check catalog and Jest matcher list inline, acting as an answer key that let the agent self-serve without loading the skill. Verified locally: the scenario now activates assertion-quality in plugin mode across runs; all other assertion-quality scenarios still activate; description stays within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
14d727fee6 |
Fix test-anti-patterns skill activation for 5 evals (#786)
* Fix test-anti-patterns skill activation for 5 evals Sibling skills with overlapping descriptions were stealing activation from test-anti-patterns in plugin eval runs (coverage-analysis, assertion-quality, test-smell-detection). Reword descriptions so test-anti-patterns owns the umbrella 'audit my tests for anti-patterns' severity-ranked report, and add DO NOT USE redirects in the metric-focused siblings. Kept all descriptions within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use toBeTruthy() call form in assertion-quality example Addresses review feedback: the Jest matcher example read like a property without parentheses. Description stays within the 1024-char cap (1023). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval validation error and improve activation triggers - assertion-quality eval.yaml/eval.vally.yaml: replace hyphenated 'assertion-quality' (the target skill name) with spaced 'assertion quality' in two scenario prompts, fixing the 'prompt mentions target name' validation error that biased baseline runs. - test-anti-patterns description: add 'what's wrong with my tests' / 'are these tests any good' / 'flaky tests' trigger phrasing to improve organic activation for the flakiness, well-written and polyglot scenarios (which intermittently failed to activate in plugin runs). Stays within the 1024-char description cap (1007). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use Jest matcher call form in assertion-quality polyglot prompt Addresses review feedback: write toBeDefined()/toBeTruthy()/not.toBeNull()/ toBe()/toThrow() in call form in the prompt so they read as matcher calls, consistent with the skill description examples. Regex assertions and rubric left untouched (they match agent output, which may use either form). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Sharpen test-anti-patterns description for flakiness/polyglot activation The flakiness and Python-pytest scenarios failed to activate even in isolated runs (where it's the only candidate skill), because their prompts enumerate the methodology and the description's keywords were too generic. Front-load the concrete trigger keywords those prompts use: Thread.Sleep, DateTime.Now, time.sleep, order-dependent, reflection coupling, and Python/pytest. Stays within the 1024-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make mixed/flakiness eval prompts realistic to fix plugin activation The mixed-severity and flakiness scenarios consistently failed to activate test-anti-patterns in plugin runs (detected=[] — the agent loaded no skill at all and answered directly). Both prompts enumerated the full anti-pattern catalog inline, acting as an answer key that made the agent self-sufficient. Replace the embedded checklists with realistic user asks while keeping the 'for .NET test anti-patterns' trigger, file references, severity-ranked output format, and read-only constraint. Rubric and output_matches assertions are unchanged — they validate the produced report. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
cd1a42dfa5 |
Fix skill activation for 5 MSTest v3-to-v4 eval scenarios (#784)
Five migrate-mstest-v3-to-v4 eval scenarios were answered from general model knowledge without loading the skill (NOT ACTIVATED): Migrate custom TestMethodAttribute from Execute to ExecuteAsync, Replace ExpectedExceptionAttribute with Assert.ThrowsExactly, Fix multiple v4 breaking changes, Fix TestMethodAttribute display name constructor, and Fix Assert.IsInstanceOfType out parameter removal. Reworded each prompt in both eval.yaml and eval.vally.yaml into an explicit 'I upgraded my test project to MSTest v4' fix request embedding the exact SKILL.md trigger phrase 'Help me fix this MSTest v4 breaking change' plus the matching USE FOR entry (Execute to ExecuteAsync, ExpectedExceptionAttribute removal, CallerInfo constructor on TestMethodAttribute, Assert.IsInstanceOfType out parameter removal). Graders and rubrics are unchanged; both YAML files parse cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
9bc2eafec3 |
Fix skill activation for MSTest.Sdk/MTP v4 migration eval scenario (#782)
Reword the 'Handle MSTest.Sdk and MTP changes in v4' prompt in both eval.yaml and eval.vally.yaml from a passive 'What changed?' question into an explicit 'fix this MSTest v4 breaking change' request embedding exact SKILL.md trigger phrases, while still requiring the skill's knowledge to answer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
231592396c |
Fix skill activation for dropped-TFM v3 migration eval scenario (#781)
Reword the 'Handle dropped target framework during v3 migration' prompt in both eval.yaml and eval.vally.yaml from a passive yes/no question into an explicit migration request. The prior phrasing was answerable from general knowledge without loading the skill (NOT ACTIVATED pattern). The new prompt embeds exact SKILL.md trigger phrases and names the fixture TFMs without revealing which one is dropped. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f389af86c7 |
code-testing-generator: mandatory pre-completion self-review gate (#768)
* code-testing-generator: mandatory pre-completion self-review gate Replaces the prose `Verify tests are implementation-specific'' bullet in Step 7 of the code-testing-generator agent with a mandatory pre-completion self-review gate that invokes two existing plugin skills: 1. `test-gap-analysis'' (pseudo-mutation check) against the source files tested and the produced test files 2. `assertion-quality'' (trivial/tautological assertion check) against the produced test files Both skills already ship in plugins/dotnet-test; this PR only wires them into the generator's workflow as a mandatory gate before declaring a run complete. The two skills' `When to Use'' sections are extended to list `called by code-testing-generator as a pre-completion self-review step'' as a recognised use case so the model does not refuse the invocation. The skill descriptions are unchanged (frontmatter is already at the 1024-char limit). Rule 11 of the generator agent is updated to list the gate alongside final build, final test, and coverage-gap review as mandatory for ALL strategies including Direct. A matching rubric item is added to the ContosoUniversity scenario in the code-testing-agent eval (yaml and vally) so the LLM judge can verify the gate was actually invoked on the trajectory. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
754011b5ee |
Add workspace-integrity guardrail to code-testing agents (#773)
The code-testing-generator/implementer agents could treat an unusual or
scaffolded workspace (e.g. a gutted repo with an injected synthetic module)
as corruption and "repair" it with git checkout/restore/reset/clean or rm,
restoring deleted tracked files and testing the wrong code.
- Replace generator Rule 5 ("Clean git first - stash changes") with an
explicit "Treat the workspace as delivered" rule, and add a "Never mutate
version control" rule. Output must be purely additive test files.
- Add a no-revert/no-clean invariant to the implementer's edit boundaries.
- Add a 'workspace integrity' eval to the code-testing-agent suite
(eval.yaml + eval.vally.yaml). The fixture looks gutted: a metricsd project
whose real core/io modules are committed at HEAD but deleted from the
working tree, leaving only a synthetic 'synthstr' decoy. A git restore would
resurrect the deleted sentinel files; graders fail if they reappear and
require passing pytest tests for the module as delivered.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
a98fb44e52 |
code-testing-agent: pin down behavior in generated tests (#767)
Adds a new `Write Tests That Pin Down Behavior'' section to `unit-test-generation.prompt.md'' covering five universal test-craft principles: 1. Mutation thinking - each assertion would fail under a plausible bug 2. Property intersections - test combinations, not only coordinate axes 3. Behavior radius - assert on at least one secondary observable 4. Fixture realism - never set the parameter under test to a degenerate value 5. Quick self-review before declaring a test method done Mirrors the same depth requirements in `code-testing-implementer.agent.md'' Step 4 as a cross-language invariant block alongside the existing `Edit boundaries'' rules. Extends the existing `code-testing-agent'' eval rubrics (yaml and vally) with one or two depth-oriented bullets per scenario: * ContosoUniversity: minimal IsNotNull-only assertions + secondary observable check on controller actions * python-flask-tasks: minimal `is not None''-only assertions + at least one combined-property TaskService validation test * typescript-vitest-cart: minimal `toBeDefined''/`toBeTruthy''-only assertions + at least one intersection test (discount + tax + shipping together) Rationale and prior-art references in PR description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Daniel Plaisted
|
8a437b1d4f |
Add anti-pattern for forking a project instance via <MSBuild> with path-neutral global properties (#770)
Adds msbuild-antipatterns AP-22 (in references/additional-antipatterns.md): using
the <MSBuild> task to build/publish a project with an extra global property that
does not change the output path forks a distinct instance sharing the same
OutputPath/IntermediateOutputPath as the instance the graph already builds,
causing duplicate builds and parallel-write races on PDBs, *.sourcelink, and
project.assets.json. Covers both shapes:
(a) a project re-invoking itself (publish-on-build), and
(b) project A invoking Build/Publish on another project B it consumes
(e.g. a test/layout project publishing a tool) — the more common trigger.
GOOD fixes: (a) static _IsPublishing + DependsOnTargets with a cycle guard;
(b) let the producer publish in its own build and have the consumer sequence it
with a non-output ProjectReference instead of re-publishing it. Also updates the
SKILL.md pointer (AP-16 through AP-22) and the quick-reference checklist.
Extends check-bin-obj-clash with the matching cause (self or cross-project
<MSBuild> Build/Publish with a path-neutral global property), binlog-based
detection via the evaluations/global-properties/double_writes tools, the two
fixes, and an _IsPublishing row in the global-properties table.
Adds eval coverage: a publish-on-build smell in the msbuild-antipatterns review
fixture (LibA.csproj) plus a matching rubric line in eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Amaury Levé
|
05597a329c |
migrate-xunit-to-mstest: switch file-state graders from output to file (no false negatives) (#761)
The eval was tripping false-negative graders: the skill activates and the agent EXPLAINS what it converted (e.g., "I removed using Xunit;", "Converting [Theory] to [TestMethod]"), so output-not-matches on patterns like `using Xunit;`, `[Theory]`, `Trait\(`, `IClassFixture`, `TestContext.Current.CancellationToken`, etc. fired on the agent's prose even when the migrated file was correct. Conversely output-matches on `Microsoft\.VisualStudio\.TestTools\.UnitTesting`, `[TestClass]`, `[DataRow(`, etc. could match the agent's explanation without verifying the file. Convert every grader that is really checking FILE state into file-contains / file-not-contains (with the relevant glob: **/*.cs or **/*.csproj). Keep the prose-checking graders where the rubric genuinely wants the agent to talk about something (unsupported-TFM explanation, scope-decision narrative, already-on-MSTest "no migration needed" message). The two CancellationToken positive graders stay on prose because they use regex alternation that file-contains (literal only) can't express; the file is still indirectly checked by the dotnet test run-command. Mirrored across both eval.vally.yaml (vally runner) and eval.yaml (skill-validator). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d9c1e7a801 |
code-testing-agent: mention find-untested-sources for C# discovery (#734)
* code-testing-agent: mention find-untested-sources for C# discovery Adds a conditional pointer (gated on 'when available') to the find-untested-sources skill in two places: - SKILL.md Step 3 (Research Phase): high-level note for C# / .NET multi-file scopes — prefer the helper over manual find/grep/glob walks. - code-testing-researcher.agent.md Section 7 (Discover Preexisting Tests): directive instruction telling the researcher to invoke the helper before manually pairing source <-> test files, and to use its source_to_tests / untested output to fill the research document. Both callouts are phrased as 'when available', so installations without the find-untested-sources skill continue to work via manual discovery. Adds no behavior for non-C# repos. Context: in a 5x136-instance internal experiment on the msbench .NET test bench, adding equivalent pointers to the routed code-testing-agent yielded a 15.67% input-token reduction at neutral pass rate — the model trusted the documented pairing heuristics and skipped its own discovery walk. The helper itself was not invoked in those runs (the Copilot CLI router did not auto-load the sibling skill), so the measured win comes from the doc text causing the model to short-circuit its manual exploration, not from the helper executing. Depends on dotnet/skills#733 (which adds the find-untested-sources skill itself). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * code-testing-agent eval: grade find-untested-sources usage on C# scenario PR #734 adds a doc pointer steering the researcher toward the `find-untested-sources` skill on .NET / C# multi-file tasks. The existing ContosoUniversity scenario is the natural fit: it's the only C# multi-file scenario in this eval, and the new pointer is .NET- scoped. Add one rubric item to that scenario (in both eval.vally.yaml and the legacy eval.yaml) asking the grader to verify the researcher actually leveraged the helper — either by citing its `source_to_tests` / `untested` JSON output in `.testagent/research.md`, or by executing `scripts/Find-UntestedSources.cs` — instead of falling back to manual `find` / `grep` / `glob` walks. Gated on `when available in the workspace` to match the SKILL.md wording, so installs without find-untested-sources are not penalized. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
74c58505e5 |
Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills (#745)
* Improve dotnet-template-engine plugin: accuracy, dedup, and two new skills Fix inaccurate reserved-shortName guidance, consolidate validation rules into a single skill, expand discovery mappings, add explicit CPM/version steps, and introduce template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .codex-plugin manifest indentation; add evals for new skills Make .codex-plugin/plugin.json byte-consistent with plugin.json (2-space indent on the agents line). Add eval.yaml + eval.vally.yaml capability evals for the new template-comparison and template-smart-defaults skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: source reserved list from dotnet new --help, note workload/package availability, tighten version-refresh - Clarify the reserved shortName set is the current dotnet new subcommands (authoritative source: dotnet new --help); create is verified as a real subcommand (alias behind dotnet new <template>). - template-discovery: note that some mapped short names (maui, winui3, aspire, func, orleans) need workloads/template packages, with fallback to dotnet new list/search. - template-instantiation: keep template versions by default; if refreshing, use dotnet list package --outdated + user confirmation and constrain to same major/minor rather than always latest stable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address 2nd review round: enforce eval negatives, split combined assertion, reframe reserved list - smart-defaults evals: enforce --no-https absence (auth scenario), absence of minimal-API flag (controllers scenario), and no newer --framework TFM when net8.0 is explicitly required, using output_not_contains/output_not_matches. - comparison eval: split the combined (auth|aot|docker|controllers) check into four separate output_matches assertions so partial comparisons fail. - template-validation/authoring: reframe the reserved shortName list as non-exhaustive examples and source the authoritative set from dotnet new --help; drop the specific create-alias assertion in favor of parsing-ambiguity wording. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make template-comparison evals robust to CI SDK currency Switch the Blazor comparison scenario from blazorserver (absent in the CI SDK) to blazor (Blazor Web App) vs blazorwasm, both reliably present in .NET 8+, and instruct the agent to inspect each via --help. Update the SKILL.md example reference for currency. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add CLI-failure resilience guidance to discovery/comparison skills The isolated eval runs failed because the agent ran 'dotnet new <t> --help', hit the template engine's global-mutex/persistence error (common when the command runs concurrently in a sandbox), and then returned no answer at all. Instruct both skills to run 'dotnet new' calls sequentially, retry once on a transient mutex/persistence error, and fall back to the intent/parameter mapping so a concrete answer is always produced instead of empty output. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: accurate flags + robust negative-assertion prompts - template-smart-defaults SKILL.md: drop the non-existent --publish-aot flag. Clarify --aot is a dotnet new flag only on templates that expose it (console/worker/grpc, not webapi) and that publish-time AOT is the MSBuild PublishAot=true property, not a dotnet new flag. - template-discovery SKILL.md: replace the hardcoded --enable-docker mapping (not a real flag on common templates) with generic 'confirm with --help'. - smart-defaults evals: tighten the negative-assertion prompts to output only the command line and not mention unused flags, so a negated explanation can't trip output_not_contains/output_not_matches. Switch the AOT scenario from webapi to worker (which actually supports --aot). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review round 3: scope negative checks, de-emphasize stale lists - smart-defaults evals: anchor the negative assertions to the 'dotnet new' command line (same-line regex) instead of whole-output substring/regex, so a flag mentioned only in prose can't fail the test. - template-validation / template-authoring: mark the dotnet new subcommand examples as illustrative/version-dependent and tell readers not to hardcode them; the live 'dotnet new --help' output is canonical. - template-comparison: fix the example table's AOT row — webapi/webapp do not expose a --aot template flag; native AOT is publish-time via PublishAot. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Copilot
|
2d163bb600 |
Rename ASP.NET Core plugin from dotnet-aspnet to dotnet-aspnetcore (#711)
* Initial plan * Rename dotnet-aspnet plugin to dotnet-aspnetcore --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
803b91f167 |
Add grade-tests skill for per-test PR-comment quality grading (#727)
* Add grade-tests skill for per-test quality grading Adds a new polyglot skill that grades a provided list of test methods on a letter scale (A-F) with a score band and 1-line notes, suitable for posting in a PR comment table. Complements existing audit skills (assertion-quality, test-anti-patterns, test-smell-detection) which produce suite-wide reports. The skill: - Takes an explicit list of tests (refuses to grade a whole workspace silently) - Uses a 3-dimension rubric (Assertion strength, Structure & focus, Anti-pattern hygiene) - Combines via 0.45/0.30/0.25 weighting with safety caps - Starts at A and only deducts for observable issues - Does not flag idiomatic patterns (table-driven sub-tests, bare assert, etc.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback - OrderProcessorTests.cs: fix 4 invocations that mixed named arg customerId: with subsequent positional args -- invalid C# syntax (Copilot review on PR #727) - README.md: update 'five test-analysis skills' to 'six' in three places now that grade-tests is added to the polyglot list, and clarify that only the original 5 load from test-analysis-extensions while grade-tests has an inline rubric Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * eval: drop 'grade-tests skill' wording from scenario prompts The skill-validator's evaluate pre-flight check ValidateEvalPrompts rejects any prompt that names the target skill, because mentioning the skill biases baseline (skill-not-loaded) runs and inflates measured lift. The two scenarios that asked the agent to 'use the grade-tests skill' now just ask it to grade test methods individually -- the agent is still expected to pick the right skill organically. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix rubric consistency per review feedback - README: drop incorrect 'inline rubric instead' claim (grade-tests also loads test-analysis-extensions for language-specific guidance) - Assertion sub-grade: literal always-true assertions are now F (was D), matching the eval expectation that 'assert True' produces overall F - Anti-pattern catalog: split always-true literals (now -> F) from self-referential assertions (still -> D) — they were conflated - Combination rule: overall grade is now capped at the worst sub-grade (was capped at D), so an F in any one dimension yields overall F --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
dependabot[bot]
|
faca87d669 |
Bump vite, @vitest/coverage-v8 and vitest (#731)
Bumps [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite) to 8.0.16 and updates ancestor dependencies [vite](https://github.com/vitejs/vite/tree/HEAD/packages/vite), [@vitest/coverage-v8](https://github.com/vitest-dev/vitest/tree/HEAD/packages/coverage-v8) and [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest). These dependencies need to be updated together. Updates `vite` from 5.4.21 to 8.0.16 - [Release notes](https://github.com/vitejs/vite/releases) - [Changelog](https://github.com/vitejs/vite/blob/main/packages/vite/CHANGELOG.md) - [Commits](https://github.com/vitejs/vite/commits/v8.0.16/packages/vite) Updates `@vitest/coverage-v8` from 2.1.9 to 4.1.8 - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md) - [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.8/packages/coverage-v8) Updates `vitest` from 2.1.9 to 4.1.8 - [Release notes](https://github.com/vitest-dev/vitest/releases) - [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md) - [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.8/packages/vitest) --- updated-dependencies: - dependency-name: vite dependency-version: 8.0.16 dependency-type: indirect - dependency-name: "@vitest/coverage-v8" dependency-version: 4.1.8 dependency-type: direct:development - dependency-name: vitest dependency-version: 4.1.8 dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Amaury Levé
|
812565f8d1 |
Add polyglot evals for code-testing-agent (Python Flask + TypeScript Vitest) (#709)
* Add polyglot evals for code-testing-agent (Python Flask + TypeScript Vitest) Adds two non-.NET evaluation scenarios to the dotnet-test/code-testing-agent eval to measure how well the agent generates idiomatic tests in Python and TypeScript — the languages exercised by the msbench top5-* benchmarks. New fixtures (no pre-existing tests, so passing pytest/vitest proves the agent actually generated them): - fixtures/python-flask-tasks/ — small Flask app with TaskService (injected clock), TaskRepository Protocol + InMemoryTaskRepository, and a /tasks blueprint with HTTP 201/200/400/404/409 surfaces. Validated locally with `python -m pip install -e ".[test]" && python -m pytest`. - fixtures/typescript-vitest-cart/ — small shopping-cart library with an injectable DiscountPolicy seam (No/Percentage policies) and a Cart class with non-trivial merge / clamping semantics. Validated locally with `npm ci && npx vitest run`. package-lock.json committed so `npm ci` is reproducible in CI. New scenarios (added to both eval.yaml and eval.vally.yaml): - "Generate pytest tests for the Flask tasks API (Python polyglot)" — asserts that `python3 -m pip install -e '.[test]' && python3 -m pytest` exits 0 and that at least one `tests/**/test_*.py` file was produced. Rubric rewards using Flask's `test_client()`, mocking `TaskRepository` for service-level tests, asserting HTTP status + JSON body, and covering the empty-title / >200-char / not-found / already-done error paths. - "Generate Vitest tests for the shopping-cart library (TypeScript polyglot)" — asserts that `npm ci && npx vitest run` exits 0 and that at least one `tests/**/*.test.ts` file was produced. Rubric rewards mocking `DiscountPolicy` via `vi.fn()`, covering Cart merge semantics, `updateQuantity` zero-removes, `totals()` discount-clamping, and the `PercentageDiscountPolicy` constructor boundary. Companion to #708 (polyglot examples + sub-agent generification); these scenarios exercise the per-language guidance that PR adds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback for #709 - routes.py: guard `create_task` against non-dict JSON payloads (e.g. list) by returning 400 instead of letting `payload.get(...)` raise AttributeError and surface as a 500. This keeps the API behavior stable and makes it easier for generated tests to assert 400 on invalid input shapes. - tsconfig.json: drop `vitest/globals` from `compilerOptions.types`. The fixture's vitest.config.ts sets `globals: false`, so allowing TypeScript to assume globals (describe/it/expect) only enables tests that compile but fail at runtime. Removing it keeps the fixture aligned with the non-global Vitest API the eval is exercising. Both fixtures re-smoke-tested locally: vitest 1/1, pytest 2/2 (including a new test confirming the non-dict body returns 400). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill activation for polyglot scenarios in PR #709 Both polyglot scenarios (pytest + Flask, Vitest + TS) scored 5.0/5 quality but reported `⚠️ NOT ACTIVATED` because the SDK's skill router did not load `code-testing-agent` into the agent's context: `activated: false, detectedSkills: []` in skillActivationIsolated / skillActivationPlugin. The .NET ContosoUniversity scenario activates fine because its prompt has stronger `project-wide, multi-file ... high coverage` framing and SKILL.md already covers .NET keywords. Fixes per `eng/skill-validator/src/docs/InvestigatingResults.md` section "Skill not activated": - SKILL.md description: add framework-specific keywords (pytest, Flask/Django, Vitest, Jest, Mocha, JUnit, Node libraries, API, package, project-wide, multi-file) so the router has explicit hooks for polyglot prompts. Trimmed verbose `DO NOT USE FOR` clauses to stay under the 1,024-char skill spec limit (now 1,019 chars). - eval.yaml + eval.vally.yaml: rewrite the two polyglot prompts to mirror the .NET scenario's framing — add `project-wide, multi-file test generation task across the ... layers` and `achieve high coverage`, soften the library-prescriptive bullets (Mock(spec=...), vi.fn()) into capability-level requirements (`mocked or stubbed`, `mock or hand-written stub`). Rubric items unchanged — they remain flexible enough to grade either mock style. Validated locally: `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` => ✅ All checks passed (23 skill(s), 11 agent(s), 1 plugin(s)). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #709 review + sharpen polyglot skill activation **Activation fix for polyglot scenarios (Python + TypeScript)** The previous attempt ( |
||
|
Amaury Levé
|
7fc6d472c3 |
Add migrate-xunit-to-mstest skill (#718)
* Add migrate-xunit-to-mstest skill Adds a skill to migrate .NET test projects from xUnit.net (v2 or v3) to MSTest v4. Covers package replacement, attribute/assertion/fixture/lifecycle translation, ITestOutputHelper -> TestContext, [Trait] -> [TestCategory]/ [TestProperty], and xUnit v3 TestContext.Current.CancellationToken. Front-loads parallelization handling because xUnit parallelizes test classes by default while MSTest serializes — the single largest source of post- migration regressions. Step 11 enumerates three explicit choices and how to translate CollectionBehavior/MaxParallelThreads/[Collection] settings. Preserves the existing test platform (VSTest stays VSTest; MTP stays MTP) by default — bundling a platform migration would violate the test-migration agent's 'never mix migration steps' rule. - plugins/dotnet-test/skills/migrate-xunit-to-mstest/SKILL.md (workflow) - plugins/dotnet-test/skills/migrate-xunit-to-mstest/references/mapping-cheatsheet.md - tests/dotnet-test/migrate-xunit-to-mstest/eval.yaml (12 scenarios) - tests/dotnet-test/migrate-xunit-to-mstest/eval.vally.yaml (parity with sibling skill) - plugins/dotnet-test/README.md (table entry) - plugins/dotnet-test/agents/test-migration.agent.md (triage routing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix step cross-references flagged in PR review - Step 1.5 'Custom DataAttribute' bullet pointed to Step 6 (assertions); custom DataAttribute mapping lives in Step 5 (data-driven tests). - Step 7 lifecycle table 'Constructor' row pointed to Step 8 for ITestOutputHelper; ITestOutputHelper conversion is in Step 9 (output). - Commit-strategy callout said 'after Step 5 (asserts fixed)' and 'after Step 9 (fixtures/lifecycle rewritten)'; asserts are in Step 6 and fixtures/lifecycle complete after Step 8. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Map [assembly: Xunit.Trait] to [assembly: TestCategory]/[TestProperty] Both TestCategoryAttribute and TestPropertyAttribute target Assembly, Class, and Method, so an xUnit [assembly: Trait] does not need to be demoted to per-class/per-method MSTest attributes — assembly scope is preserved 1:1. - references/mapping-cheatsheet.md §8 now maps [assembly: Trait] to [assembly: TestCategory] / [assembly: TestProperty] instead of 'Remove'. - SKILL.md Step 12 promoted to 'Convert' (was 'Remove'); the same mapping is documented inline. - Notes after the Step 4 and §1 trait tables now disclose the assembly target so users do not assume class/method-only scope. Addresses PR feedback from @Evangelink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address second round of PR review feedback From @Evangelink: - [Ignore]/[Timeout] do NOT discover a test on their own — they are modifiers. SKILL.md Step 4 + cheatsheet §1 now show [Fact(Skip)] -> [TestMethod] + [Ignore(...)] (and same for [Timeout]). Eval scenario 9 rubric updated to require both attributes. - TestProperty IS filterable (was incorrectly described as 'metadata that does not need filter integration'). Notes after the trait tables in both SKILL.md and cheatsheet §1 now show both filter syntaxes. - For environmental gates (OS-/CI-/arch-conditional), point at MSTest 3.10+'s condition attributes ([OSCondition], [CICondition], [ArchitectureCondition], [NonParallelizableCondition]) as the preferred alternative to overloading [TestCategory] or scattering Assert.Inconclusive through test bodies. Strengthened in cheatsheet §3.9, SKILL.md Step 6, Step 10 SkippableFact row, and the Common Pitfalls table. - TestDataRow<T> is also a supported MSTest data source (strongly typed with per-row DisplayName/Ignore metadata). Added to both SKILL.md Step 5 and cheatsheet §2 with link to the docs. - MSTest.Sdk + <UseVSTest>true</UseVSTest> pulls in Microsoft.NET.Test.Sdk automatically — the manual PackageReference was incorrect. Removed it from both files (Step 2, Step 3 'common mistakes', and Common Pitfalls). - Added the 'pin MSTest.Sdk in global.json msbuild-sdks' alternative as the recommended pattern for multi-project solutions. - Xunit.Combinatorial -> Combinatorial.MSTest (Youssef1313 community port). Updated both SKILL.md Step 10 and cheatsheet §10. From Copilot reviewer: - MSTest.Sdk example in SKILL.md Step 2 Option B hardcoded <TargetFramework>net9.0</TargetFramework>, contradicting the preserve-TFM rule. Replaced with a placeholder + explicit comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve xUnit-to-MSTest skill effectiveness based on eval feedback Eval results on PR #718 showed regressions on three rubric items where the agent did the migration correctly but did not COMMUNICATE the judgement-call decisions, so the judge could not verify the choice was deliberate: - ICollectionFixture scope decision (rubric: 'Explicitly explains the scope decision instead of silently widening' scored 1/5 because the agent said 'Let me convert everything' with zero scope discussion). Step 8 now requires a templated explanation of (source scope, target scope, what is shared, what is serialized, why the rejected alternative was rejected) before applying. - Parallelization 'MSTest default is serial' messaging (rubric scored 1.7/5 because the agent only implicitly conveyed it). Step 11 Choice A now demands a templated sentence explaining that the [assembly: Parallelize] is REQUIRED to match xUnit, otherwise the suite silently regresses to serial. - Response Guidelines now lists the three judgement-call decisions that MUST be communicated up-front: fixture scope, parallelization model, and the Throws/ThrowsExactly semantic flip. Additional fixes from this round: - Step 2: explicit guidance to default to Option A (MSTest metapackage) when the user says 'preserve VSTest', so the safer PackageReference path is picked over MSTest.Sdk + UseVSTest. - Step 4: pulled the '[Ignore]/[Timeout] are modifiers, need [TestMethod]' callout out of a table footer into a visible note above the table. - known-domains.txt: allowlist github.com/Youssef1313/Combinatorial.MSTest (referenced from SKILL.md Step 10 and cheatsheet §10 after the recent Xunit.Combinatorial -> Combinatorial.MSTest switch). Validator: green (24 skills, 11 agents). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add concrete TestContext.Current.CancellationToken migration example The CancellationToken scenario was failing/low-scoring because the skill only contained a one-line mapping with no concrete example. The agent often missed the requirement to: - Add constructor TestContext injection (even when ITestOutputHelper is absent) - Map Assert.False -> Assert.IsFalse alongside the rename - NOT fabricate a CancellationTokenSource as a replacement Add a full xUnit v3 -> MSTest worked example next to the TestContext.Current mapping (Step 9), and reinforce the CancellationTokenSource warning in the cheatsheet. The example covers all 3 rubric points for the scenario. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address third round of PR review feedback - Remove fictional MSTest condition attributes [ArchitectureCondition] and [NonParallelizableCondition] from SKILL.md and cheatsheet (only [OSCondition] and [CICondition] exist in MSTest 3.10+; for non-parallel intent use [DoNotParallelize], for architecture gating fall back to runtime Assert.Inconclusive). (Copilot) - TestPropertyAttribute targets only [Class|Method] (no AttributeTargets.Assembly), so [assembly: TestProperty(...)] does not compile. Update the trait notes in SKILL.md Step 4, cheatsheet section 1, Step 12, and cheatsheet section 8 to collapse a non-category xUnit [assembly: Trait] to [assembly: TestCategory] or push it down to per-class [TestProperty]. (@Evangelink) - Step 1 platform detection: stop inlining a buggy VSTest-vs-MTP matrix; delegate to the platform-detection skill. Note explicitly that <UseMicrosoftTestingPlatformRunner> only affects 'dotnet run' and is not a reliable runner signal. (@Youssef1313) - Step 2 Option A: add MTP code-coverage caveat -- Microsoft.NET.Test.Sdk pulls VSTest's Microsoft.CodeCoverage transitively, which can interfere with MTP's collector. Prefer Option B (MSTest.Sdk without UseVSTest) for MTP projects. (@Youssef1313) - Step 3: remove fictional MSBuild properties (<CaptureConsoleOutput>, <UseRoslynCompilers>); restate the bullet about xunit.runner.json -> Step 11 port instead. Restate the <UseMicrosoftTestingPlatformRunner> bullet to make clear it is not a runner switch. (@Youssef1313) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 4 - mapping-cheatsheet.md (Assert.Contains predicate): the suggested fallback Assert.Contains(collection.First(x => predicate), collection) is buggy -- First() throws InvalidOperationException on no-match (changing the failure mode), and feeding the result back into Assert.Contains is circular. Keep only the correct translation: Assert.IsTrue(collection.Any(x => predicate)). - SKILL.md Step 4 / cheatsheet table: stop instructing 'mark every class sealed'. xUnit projects commonly use base/derived test classes (shared setup, generic base fixtures); mechanically sealing them would break compilation. Sealing is now framed as an optional follow-up handled by writing-mstest-tests, not part of the mechanical migration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
680292e04c |
Make dotnet-test analysis skills and auditor agent polyglot (#707)
* Make dotnet-test analysis skills and auditor agent polyglot Five test analysis skills and the test-quality-auditor agent in plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/ TUnit cues, even though the underlying techniques (assertion diversity, anti-pattern detection, pseudo-mutation gap analysis, academic smell catalog, trait tagging) apply equally to other test frameworks. This change makes them polyglot across .NET, Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest). Changes ------- * New reference skill `test-analysis-extensions` with per-language extension files (dotnet, python, typescript, java, go, ruby, rust, swift, kotlin, powershell, cpp). Each documents test markers, assertion APIs, sleep / time / random APIs, skip annotations, setup/teardown, mystery-guest indicators, integration markers, and tag-support capability for the corresponding framework. * `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`, `test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter now lists all polyglot frameworks (keeping .NET trigger words for backward compat); bodies reference `test-analysis-extensions` and add per-language calibration (e.g., Go/Rust table-driven loops are idiomatic and not Conditional Test Logic; pytest bare `assert` is canonical; missing-await on async assertions is a new Critical smell). * `test-quality-auditor` agent now runs language detection, gates .NET-only pipeline steps (coverage-analysis, CRAP, detect-static- dependencies, testability migration, experimental dotnet skills), and recommends native tooling for non-.NET projects via an explicit Capability Matrix. * README updated to split sections by polyglot vs .NET-only. * New polyglot eval scenarios added (Python/pytest, TS/Jest, Java/JUnit) alongside existing C# scenarios in `test-anti-patterns`, `assertion-quality`, `test-smell-detection`. The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis, crap-score, detect-static-dependencies, run-tests, filter-syntax, platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are unchanged and continue to activate only for .NET work. Validated with skill-validator `check --plugin ./plugins/dotnet-test`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #707 review feedback * typescript.md: replace xpect.fail() recommendation (Jest has no such API) with hrow new Error(...) / explicit failing assertion. * test-tagging SKILL.md description: clarify that unittest has no canonical tag syntax (pytest markers are the auto-edit mechanism; unittest is report-only). Switch doctest reference to "decorators" rather than implying a Catch2-style [tags] mechanism exists. * test-tagging SKILL.md auto-edit list and table: use `* doctest::test_suite("tag")` decorator chain syntax for doctest in both the bullet list and the per-framework table, matching the example already shown. * cpp.md extension: spell out the Catch2 vs doctest tagging distinction in the capability matrix (Catch2 [tag], doctest * doctest::test_suite("tag") decorator). Validated with `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` (23 skills + 11 agents). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Tom Longhurst
|
36209c3b52 |
TUnit Fixes & Improvements (#677)
* Update target framework to net10.0 and upgrade TUnit package to version 1.45.8
* Fix and expand TUnit coverage in dotnet-test-frameworks reference
Cross-referenced against tunit.dev/llms.txt and the official attributes
comparison page. Corrects the test class marker (TUnit is convention-based,
not [ClassDataSource]), and fills in the previously-missing TUnit rows for
the assertion APIs, setup/teardown methods, exception-handling examples,
and integration test category marker.
* Refine TUnit assertion table accuracy
The Type row's TUnit cell now uses IsAssignableTo<T>() to match the
assignable-type semantics of the MSTest/xUnit/NUnit equivalents, with a
note pointing at IsTypeOf<T>() for exact-type checks. The Skip row now
documents that [Skip] also applies at class and assembly scope and that
Skip.Test("reason") handles dynamic in-test skipping.
Verified against the TUnit migration guides and the type-assertions docs.
* Spell out TUnit assertion alternatives as complete awaited expressions
The Boolean, Null, Exception, and Type rows in the assertions table
previously used a `await Assert.That(x).IsTrue()` / `.IsFalse()` shorthand
for the alternative form. A reader copy-pasting just the second fragment
would end up with an un-awaited assertion that silently passes — which
contradicts the note immediately below the table about always awaiting.
Each alternative is now a complete, independently copy-pasteable expression.
Addresses the inline review on dotnet/skills#677.
---------
Co-authored-by: Amaury Levé <amauryleve@microsoft.com>
|