mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
fix-dotnet-test-analysis-scripts
354 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Amaury Levé
|
cfc293f8fe |
Fix skill activation for Assert.AreEqual overload eval scenario (#589)
The 'Fix Assert.AreEqual object overload errors after v3 upgrade' evaluation scenario was failing because: 1. The prompt said 'latest major version' without mentioning 'v3' or 'MSTest v3', missing the skill's activation keywords. 2. The project file showed MSTest 3.8.0 (post-upgrade state) with no v2 context, while the SKILL.md 'When Not to Use' said 'Project already uses MSTest v3'. 3. The prompt's 'fix compilation issues' phrasing competed with writing-mstest-tests skill activation. Changes: - Updated prompt to explicitly mention upgrading from v2 to v3 and reference MSTest v3 compatibility issues - Added XML comment in csproj showing the pre-upgrade v2 package versions - Clarified SKILL.md 'When Not to Use' to distinguish fully-migrated projects from projects with active migration errors |
||
|
Amaury Levé
|
9f531203ec |
Promote 4 experimental test skills to dotnet-test plugin (#634)
Move assertion-quality, test-gap-analysis, test-smell-detection, and test-tagging from dotnet-experimental to dotnet-test. These skills have demonstrated sufficient eval coverage, unique value beyond baseline model capabilities, and are already integrated into the test-quality-auditor agent pipeline. Changes: - Move skills and tests, dropping the exp- prefix - Update all cross-references in skill descriptions and routing tables - Update test-quality-auditor agent and eval scenarios - Update CODEOWNERS for the moved paths - Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks) - Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability Skills held back for further iteration: - exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios - exp-test-maintainability: quality gains offset by token overhead |
||
|
Amaury Levé
|
97b23ae750 |
Fix timeout for Modernize legacy test patterns evaluation (#590)
The scenario was timing out on CI because: - The 180s timeout left insufficient headroom for CI machines (local runs take 120-135s, CI machines are slower due to load, network NuGet restores, and parallel execution) Changes: - Add 'dotnet restore' setup command to pre-warm NuGet cache before the agent starts (same pattern used by migrate-mstest-v3-to-v4) - Increase timeout from 180s to 300s to accommodate CI variability |
||
|
Jan Jones
|
3aefa0f10a |
Update file-based app skill to mention features from 10.0.300 (#602)
Co-authored-by: Copilot <copilot@github.com> |
||
|
Dan Moseley
|
e7ec5a77a0 | Fix markdownlint CI workflow (startup_failure since inception) (#629) | ||
|
Copilot
|
2f9922e073 |
Remove LangVersion=latest from skill examples and test fixtures (#608)
* Initial plan * Remove LangVersion=latest from skill docs and test fixtures Per official C# docs, LangVersion should not be set to `latest` as it causes builds to silently vary across machines with different SDKs. Best practice is to omit LangVersion entirely and let the TFM determine the correct C# version automatically. - Update msbuild-modernization/SKILL.md: remove LangVersion=latest from example and add explicit guidance to omit LangVersion - Update directory-build-organization/SKILL.md: remove from example - Update directory-build-organization/references/multi-level-examples.md: remove from all 4 examples - Update msbuild-antipatterns/SKILL.md: remove from AP-08 examples - Remove LangVersion=latest from 11 test fixture project files Agent-Logs-Url: https://github.com/dotnet/skills/sessions/7a3d2722-770b-4010-a6e8-70c92668ad4c Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Fix LangVersion guidance and stale eval rubric reference Agent-Logs-Url: https://github.com/dotnet/skills/sessions/e89afcd8-7227-424b-9c8c-9ec0ce7113c2 Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Roman Głogowski
|
e03f2f2b38 |
Add Claude Code-shaped .lsp.json for the dotnet plugin
Claude Code auto-discovers `.lsp.json` at plugin root and expects a schema that differs from Copilot CLI's `lsp.json`: - No outer `lspServers` wrapper in the external file (the wrapper is only used when defining LSP servers inline in `plugin.json`). - Required field is `extensionToLanguage`, not `fileExtensions`. Without this file, Claude Code silently rejects the existing `lsp.json` (Copilot CLI shape) and the Roslyn language server is never registered — `/reload-plugins` reports `0 plugin LSP servers`. After adding `.lsp.json`, the count becomes `1 plugin LSP server` and `findReferences`, `goToImplementation`, etc. work end-to-end. The existing `lsp.json` is left untouched so Copilot CLI continues to work; the two hosts read different filenames and never see each other's config. References: - https://code.claude.com/docs/en/plugins-reference (lspServers, external `.lsp.json` format, required fields) - https://docs.github.com/en/copilot/how-tos/copilot-cli/set-up-copilot-cli/add-lsp-servers (Copilot CLI uses `lsp.json` with `lspServers` wrapper and `fileExtensions`) Signed-off-by: Roman Głogowski <roman.glogowski@volue.com> |
||
|
Manish Jayaswal
|
2633d3d482 |
add a new dotnet11 plugin and a new system text json skill to it. (#535)
* add a new dotnet11 plugin and a new system text json skill to it. * Update plugins/dotnet11/skills/system-text-json-net11/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * added the option to run dotnet11 skills. by default they don't get included * Update tests/dotnet11/system-text-json-net11/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add JSON output assertions to system-text-json-net11 eval.yaml Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8e302f3f-4328-44a0-845d-4eb0c930af63 Co-authored-by: ManishJayaswal <9527491+ManishJayaswal@users.noreply.github.com> * addressed CR comments * addressed more CR comments * fixed agument to pass while enabling dotnet11 skills * Remove dotnet11-specific filtering from skill-validator and evaluation workflow Now that .NET 11 is the default SDK for the repo (global.json updated on main), there is no need for special dotnet11 exclusion logic. The dotnet11 plugin skills are treated like any other plugin and included in all evaluation runs. Changes: - Remove --include-dotnet11 CLI option and filtering logic from EvaluateCommand.cs - Remove IncludeDotnet11 from ValidatorConfig model - Remove --dotnet11 flag parsing from evaluation workflow gate job - Remove dotnet11 exclusion logic from discover job - Remove conditional .NET 11 preview SDK install step (global.json handles it) - Remove dotnet11 from excludeFromSchedule list - Remove all dotnet11-specific warning/status messages - Remove section 9 from InvestigatingResults.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * address code review comments after rebase * added the plugin to cursor and added MIT license to skill --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: [Manish Jayaswal] <[manishj@microsoft.com]> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ManishJayaswal <9527491+ManishJayaswal@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
809d0b180c |
Improve test generation quality based on SWE Atlas benchmark analysis (#599)
* Improve test quality for benchmark performance Based on SWE Atlas benchmark analysis (48.8% vanilla vs 19-34% CTA): 1. Default to Direct strategy for single-task requests — reduces multi-agent overhead that costs tokens without improving results 2. Run tests immediately in Direct strategy — catches assertion errors early instead of accumulating failures 3. Read source thoroughly before writing tests — trace actual logic and return values, not just function signatures 4. Quality over Quantity guidelines — cover stated requirements first, fewer focused tests beat many shallow ones 5. Verify tests are implementation-specific — tests that pass with an empty function body aren't testing anything useful Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Combine redundant implementation-specificity bullets into one Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Tomas Grosup
|
a55227f030 |
Fold F# .fsproj ordering into msbuild-antipatterns AP-05 (#577)
* Add fsharp-project-structure skill for .fsproj file ordering and .fsi files Teaches agents F# compilation order rules, signature file placement, and common FS0039/FS0010/FS0034 fixes. Includes eval with three scenarios (add module, fix broken order, add signature file) and test fixtures. Validated +64.6% improvement on gpt-4.1 (significant, CI [+17.2%, +63.0%]). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add fsharp-project-structure skill for .fsproj file ordering and .fsi files Teaches agents F# compilation order rules, signature file placement, and common FS0039/FS0010/FS0034 fixes. Includes eval with three scenarios (add module, fix broken order, add signature file) and test fixtures. Validated +64.6% improvement on gpt-4.1 (significant, CI [+17.2%, +63.0%]). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/fsharp-project-structure/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fold F# project-structure into msbuild-antipatterns AP-05 Scraps the standalone fsharp-project-structure skill and instead adds a short F# exception to AP-05 (Manual File Listing in SDK-Style Projects): F# compilation is order-dependent, so .fsproj files must list Compile items in dependency order (utilities first, entry point last), and .fsi signature files must appear immediately before their companion .fs. The three F# eval scenarios (add module, fix broken order, add signature file) are ported into tests/dotnet-msbuild/msbuild-antipatterns/eval.yaml with fixtures isolated under a fsharp/ subfolder (with empty Directory.Build.{props,targets} to prevent the existing C#-antipattern fixture files from leaking into the F# builds). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
81946d2a38 | Add proxy for CTA extension files (#583) | ||
|
Amaury Levé
|
1b105d1940 |
Fix dotnet-test-frameworks skill activation for evaluations (#580)
* Fix dotnet-test-frameworks skill activation for evaluations The skill had user-invocable: false and a description saying 'DO NOT USE directly', which prevented the model from activating it during evaluation. All 6 eval scenarios default to expect_activation: true, causing SkillNotActivated failures on CI. - Remove user-invocable: false from both dotnet-test and experimental copies - Rewrite description with activation keywords matching eval prompts (assertion mapping, framework identification, skip annotations, exception assertions, lifecycle conversion, integration test markers) - Preserve note that other analysis skills also load this as reference data * Restore user-invocable: false, fix cross-refs, add expect_activation: false Address review comment: remove exp-* skill references from dotnet-test- frameworks description since those skills reference exp-dotnet-test- frameworks instead. Optimize eval results: this reference data skill's content is knowledge the model already has (baseline scores 5.0/5 on most scenarios). Direct activation adds token/time overhead without quality improvement: - Restore user-invocable: false (correct for reference data skill) - Shorten description to not trigger activation - Add expect_activation: false to all 6 scenarios The skill's value is realized when loaded by consumer skills (e.g. test-anti-patterns), not via direct user queries. |
||
|
Jan Krivanek
|
4ace5459f8 |
Improve agent based on C++ testing feedback (#571)
* Improve agent based on C++ testing feedback * Rename Cpp.md to cpp.md |
||
|
Amaury Levé
|
f8e8191268 |
Add concrete pipeline examples for code-testing-agent (#572)
Add end-to-end input/output examples to anchor expected behavior for the LLM, addressing the Example Quality (2/5) review feedback. Changes: - New extensions/dotnet-examples.md with .NET-specific examples: sample source code, research output, plan output, generated test file, fix cycle walkthroughs, and final report - SKILL.md: replace one-liner examples with strategy selection table, pipeline walkthrough, and pointer to language-specific extensions - Generator agent: add strategy decision examples and sample final report - Researcher/Planner/Implementer agents: add references to extension examples for concrete output shapes Architecture keeps core agents language-agnostic — all language-specific examples live in the extensions/ folder. |
||
|
Amaury Levé
|
8651264f12 |
Fix skill activation for exit code 8 scenario (#567)
* Initial plan * Improve MTP migration skill guidance for OutputType conditions Agent-Logs-Url: https://github.com/dotnet/skills/sessions/ae2f9ae1-c8c3-4e50-95ab-ccdc08bc388b Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> * Add eval scenario for OutputType conditioning in Directory.Build.props Agent-Logs-Url: https://github.com/dotnet/skills/sessions/734b3736-45a4-4f5d-b83c-3e6ee72fcfae Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> * Fix skill activation for exit code 8 scenario Add 'zero tests discovered', 'MTP behavioral differences', '--ignore-exit-code', and 'TESTINGPLATFORM_EXITCODE_IGNORE' to skill description to improve activation for the exit code 8 eval scenario. Consolidate filter syntax mention to stay within 1024-char limit. --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
05aeb657e6 | Add license to agent files (#568) | ||
|
Artur Spychaj
|
270ef910ec |
Ensure validation runs for all strategies including Direct (#570)
Harden the test generator agent to ensure final build validation, test validation, and git commit always run — even when using the Direct strategy for small requests. Changes: - Step 1: Mandate reading language-specific extensions (e.g., dotnet.md) before writing any code. The extension contains critical project registration (dotnet sln add) and build validation guidance. - Step 2: Direct strategy now skips only Steps 3-5 (research, plan, implement sub-agents), NOT Steps 6-9. Added explicit note that Steps 6-9 are mandatory for ALL strategies. - Step 6: Clarify full-solution build must use no --framework flag to catch multi-target framework issues (e.g., net472 errors). - Step 7: Require fresh build for final test validation (no --no-build). - New Step 9: Explicit git commit with verification that the diff is non-empty. - New rules 9-11: Read extensions first, always validate+commit, preserve existing tests. Motivated by MSBench investigation showing: - Agent used Direct strategy and skipped final validation entirely - Agent never read extensions/dotnet.md (missing dotnet sln add) - Agent forgot to git commit (300 tests created, Changes +0 -0) - Agent deleted pre-existing tests while adding new ones Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Copilot
|
c625f09db5 |
Recommend MSBuildProjectName for OutputType conditions in MTP migration skill (#556)
* Initial plan * Improve MTP migration skill guidance for OutputType conditions Agent-Logs-Url: https://github.com/dotnet/skills/sessions/ae2f9ae1-c8c3-4e50-95ab-ccdc08bc388b Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> * Add eval scenario for OutputType conditioning in Directory.Build.props Agent-Logs-Url: https://github.com/dotnet/skills/sessions/734b3736-45a4-4f5d-b83c-3e6ee72fcfae Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> Co-authored-by: Amaury Levé <amauryleve@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
5826c421b4 |
Fix test-anti-patterns plugin activation by disambiguating competing descriptions (#552)
The test-anti-patterns skill was not reliably activated in plugin mode because sibling descriptions overlapped: - test-quality-auditor agent: narrowed to multi-skill audit pipelines only, explicitly excluding single-file reviews - writing-mstest-tests skill: removed 'review' and 'anti-patterns' from description, added explicit exclusion for test quality auditing - test-anti-patterns skill: expanded trigger phrases to better match review prompts (flaky tests, duplication, tests that pass without verifying) |
||
|
Amaury Levé
|
7841934a75 |
Add migrate-mstest-v3-to-v4 eval scenarios for uncovered patterns (#564)
* Add migrate-mstest-v3-to-v4 eval scenarios for uncovered patterns
Add 5 new scenarios (Goals 10-14) covering:
- Verified package update with build/test validation
- Sealed custom TestMethodAttribute with Timeout migration
- TestMethodAttribute/TestMethod display name constructor syntax
- Assert.IsInstanceOfType out parameter breaking change
- TreatDiscoveryWarningsAsErrors and behavioral changes review
Add 4 new fixture directories:
- v3-verified-migration: buildable project for end-to-end validation
- v3-sealed-timeout: sealed attr + TestTimeout.Infinite patterns
- v3-testmethod-displayname: [TestMethod/Attribute("name")] patterns
- v3-instanceoftype: Assert.IsInstanceOfType<T> out param pattern
Covers uncovered validation items (lines 449-451, 458) and code
patterns: [TestMethod], [Timeout], sealed, [TestMethodAttribute],
Assert.IsInstanceOfType.
* Improve skill description for activation and bump scenario timeouts
Rewrite migrate-mstest-v3-to-v4 frontmatter description to improve SDK
semantic matching:
- Lead with symptom (fix build errors) instead of action (migrate)
- Add error codes CS0507, CS0103, CS1061, CS1615 as trigger signals
- Add symptom phrase: tests don't compile after upgrading MSTest
- Expand breaking change keywords: sealed, DisplayName syntax,
Assert.IsInstanceOfType out parameter, TestContext.TestName
- Keep under 1024 char limit
Bump timeout 180s -> 240s for scenarios 11 (sealed attr) and 13
(IsInstanceOfType) that hit timeout during evaluation.
* Address PR review comments
- Remove unused 'using System' from OrderServiceTests.cs and TypeCheckTests.cs
- Renumber duplicate Goal 10 to Goal 15 (non-activation scenario)
- Tighten file_contains assertion to check MSTest.TestFramework and
MSTest.TestAdapter individually instead of broad '4.' match
- Tighten Timeout assertion to require int.MaxValue instead of matching
pre-migration symbol names
- Fix error code CS0507 -> CS0115 in scenario 11 prompt to match the
actual MSTest v4 error for missing Execute override
|
||
|
Amaury Levé
|
1125fe7864 |
improve(dotnet-test): enhance agent descriptions for subagent discovery (#560)
- Add 'Use when:' trigger phrases to all 7 sub-agent descriptions so parent agents can reliably discover and delegate to them - Add .testagent/ cleanup rule to generator agent to prevent ephemeral pipeline state from being committed |
||
|
Amaury Levé
|
9ab90d6400 |
feat(dotnet-test): add handoffs between orchestrator agents (#561)
- test-quality-auditor -> code-testing-generator (generate missing tests) - test-quality-auditor -> testability-migration (fix untestable code) - testability-migration -> code-testing-generator (test migrated code) - test-migration -> test-quality-auditor (audit after migration) Enables guided sequential workflows between agents with suggested next steps, per VS Code custom agent handoff pattern. |
||
|
Copilot
|
a4927f1b79 |
Align writing-mstest-tests skill routing with anti-pattern disambiguation (#553)
* Initial plan * fix(dotnet-test): align writing-mstest-tests routing with anti-pattern skill Agent-Logs-Url: https://github.com/dotnet/skills/sessions/eb70209e-8e7a-4b49-820d-65e0e3fcaedc Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> |
||
|
Amaury Levé
|
400661df09 |
run-tests: add critical rules, detection guidance, and troubleshooting (#514)
* run-tests: add critical rules, detection guidance, and troubleshooting - Add 'Critical Rules' table to prevent cross-platform VSTest/MTP mistakes - Expand Step 1 detection with explicit file-by-file lookup table - Add 'dotnet --version' as first detection action - Add Troubleshooting section with 7 common error patterns - Expand Common Pitfalls from 3 to 7 entries - Strengthen negative guidance for SDK version-specific syntax * Improve description * Improve evaluation * Add expected and rejected tools * Change * Improve do not use sections * Improve prompt * Increase eval timeouts for scenarios hitting the limit - run-tests: trx reporting MTP SDK 9 (360s -> 480s) - run-tests: blame-hang MTP SDK 10 (300s -> 420s) - run-tests: combine filter criteria VSTest (180s -> 300s) - mtp-hot-reload: hot reload SDK 9 (360s -> 480s) - mtp-hot-reload: hot reload filter (180s -> 300s) - code-testing-agent: ContosoUniversity (1800s -> 2400s) |
||
|
Amaury Levé
|
7791735b60 |
Add test-quality-auditor hub agent for .NET test suite analysis (#541)
* Add test-quality-auditor hub agent for .NET test suite analysis Introduces a new test-quality-auditor agent that triages test quality concerns to specialized analysis skills: assertion quality, test smells, anti-patterns, mock usage, test gaps, maintainability, coverage risk, and test tagging. Includes a comprehensive audit pipeline that chains multiple skills for broad health checks. * Address Copilot review: clarify diagnostic role and coverage routing - Reword intro to acknowledge file-modifying workflows (test tagging) are only used with explicit user confirmation - Clarify coverage-analysis vs crap-score: prefer coverage-analysis for project-wide analysis, reserve crap-score for targeted method/class scope |
||
|
Amaury Levé
|
f9d7faf0ee |
Add test-migration hub agent for .NET test framework and platform migrations (#540)
* Add test-migration hub agent for .NET test framework and platform migrations Introduces a new test-migration agent that auto-detects the current test framework, version, and platform, then routes to the appropriate migration skill (MSTest v1/v2→v3, v3→v4, xUnit v2→v3, VSTest→MTP). Handles multi-step migrations with commit boundaries between steps. * Align framework detection with platform-detection skill Address Copilot review: MSTest requires metapackage, Sdk, or both TestFramework+TestAdapter (not either alone). Add missing xUnit MTP package variants. |
||
|
Artur Spychaj
|
3dbe832796 |
Add 'dotnet sln add' step to code-testing-implementer (#522)
* Add 'dotnet sln add' step to code-testing-implementer When a new test project is created, register it with the solution file using 'dotnet sln add' so that 'dotnet test <solution>' discovers the tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: improve dotnet sln add guidance - Use solution identified in research/plan rather than searching - Account for .slnf solution filter scenarios - Prefer 'dotnet test --solution' over 'dotnet test <solution>' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move dotnet sln add guidance to dotnet.md extension The implementer is polyglot, so 'dotnet sln add' doesn't belong there. Replace with a generic 'register with build system' step that defers to extensions/, and add the .NET-specific detail (sln/slnx/slnf handling) to extensions/dotnet.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: fix solution registration guidance in dotnet.md - Include .slnf in 'don't substitute' warning alongside .sln/.slnx - Qualify '--solution' flag as SDK 10+/MTP-only; fall back to positional form for older SDKs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7e6954f9a9 |
Add testability migration skills and agent for dotnet-test (#518)
* Add testability migration skills and agent for dotnet-test Introduces three skills and one orchestrator agent to help developers incrementally replace hard-to-test static dependencies with injectable abstractions: - detect-static-dependencies: scans C# code for DateTime.Now, File.*, Environment.*, HttpClient, Console.*, Process.* and ranks by frequency - generate-testability-wrappers: generates wrapper interfaces or guides adoption of built-in abstractions (TimeProvider, IHttpClientFactory, System.IO.Abstractions) - migrate-static-to-wrapper: mechanical codemod-style bulk replacement of static call sites with wrapper calls, scoped per project/namespace - testability-migration agent: orchestrates the Detect-Generate-Migrate pipeline end-to-end Includes eval.yaml tests and fixture projects for all three skills and the agent. * Fix activation, token overhead, and eval timeouts - Expand skill descriptions with more trigger phrases for migrate-static-to-wrapper and generate-testability-wrappers to fix NOT ACTIVATED issues - Collapse verbose replacement mapping tables and constructor examples to reduce token overhead (~30% fewer tokens) - Increase generate-testability-wrappers eval timeouts from 120s to 240s to prevent timeout-impacted scoring - Soften detect-static-dependencies scenario 2 rubric to reduce run-to-run variance in plugin mode - Increase migrate-static-to-wrapper non-activation timeout |
||
|
Copilot
|
7ab8006c18 |
Fix markdownlint violations in dotnet-test plugin (#513)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com> |
||
|
Milos Kotlar
|
038dd4f087 |
Add Apple crash symbolication skill (#201)
* Add SKILL.md for iOS crash symbolication process Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/ios-crash-symbolication/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add automation script, tests, and review fixes for ios-crash-symbolication - Add Symbolicate-Crash.ps1 (664 lines): parses .ips JSON, searches local dSYMs (SDK packs, NuGet cache, user paths), verifies UUIDs via dwarfdump, batch-symbolicates with atos, identifies runtime version - Update SKILL.md: add Automation Script and Runtime Version Identification sections, fix atos -o to point inside dSYM bundle, add dwarfdump and MAUI to frontmatter, wrap steps in Workflow heading for consistency with Android sibling, fix misleading 'rebuild' guidance per review feedback (Rolf) - Add eval.yaml with 7 test scenarios: Mono crash, CoreCLR crash, no .NET frames, NativeAOT, multi-library UUIDs, ASI field priority, and Android tombstone rejection - Add 5 .ips test fixture files (two-part JSON format) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add iOS crash log and evaluation scenarios for symbolication tests * Remove unused file * Add CODEOWNERS entry for iOS crash symbolication * Rename ios-crash-symbolication to apple-crash-symbolication Address PR feedback to support all Apple platforms (tvOS, Mac Catalyst, macOS) not just iOS: - Add $appleRids array covering ios, tvos, maccatalyst, and osx RIDs - Refactor Find-Dsym and Find-RuntimeVersion to search all platform packs - Rewrite SKILL.md for orchestration focus and reduced token budget - Extract domain knowledge to references/ips-crash-format.md - Tune eval.yaml: outcome-based rubrics, broad assertions, overfit 0.26 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix script robustness: base address parsing and null-safe property access - Simplify base address cast to [uint64] without manual hex prefix stripping - Use PSObject.Properties check before accessing lastExceptionBacktrace Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dSYM-path rubric items to CoreCLR and ASI scenarios Add rubric items testing for skill-specific domain knowledge (SDK pack paths, NuGet cache directories) that baseline agents cannot provide. This creates the quality delta needed to pass the 10% improvement threshold while keeping overfit at 0.12 (Low). 3-run validation: 30.1% improvement, 5/7 scenarios positive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor code structure for improved readability and maintainability * Fix crash symbolication script for real .ips format and improve eval scenarios - Fix Symbolicate-Crash.ps1 strict-mode bugs with real .ips crash logs: safe property access for image name/path (sentinel entries), thread name (most threads unnamed), and single-element array unwrapping (.Count) - Add SKILL.md efficiency guidance: resolve script path from skill directory (no find /), start with -ParseOnly, don't run broad filesystem searches - Reframe eval scenarios for platform-independent evaluation (parse/analyze instead of requiring macOS-only atos/dwarfdump), tighten rubric to test skill-specific knowledge (NuGet package name, all .NET binary images) Validation results (3 runs, claude-opus-4.6): Scenario 1 (parse frames): 3.3 → 4.0 (+0.7) ✅ Scenario 2 (investigate crash): 3.3 → 4.0 (+0.7) ✅ Scenario 3 (reject Android): 3.3 → 4.0 (+0.7) ✅ Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Auto-fallback to parse-only output when atos is unavailable On Linux (CI), atos and xcrun don't exist. Previously the script would error out after completing all parsing, losing the results. Now it detects the missing tool and falls back to ParseOnly output automatically, ensuring the agent always gets structured parse data regardless of platform. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move apple-crash-symbolication to dotnet-diag plugin Relocate skill from plugins/dotnet to plugins/dotnet-diag and tests from tests/dotnet to tests/dotnet-diag. Update CODEOWNERS accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve apple-crash-symbolication: macOS symbols, bug fixes, training log - Fix JSON case-conflict parsing (vmRegionInfo/vmregioninfo duplicate keys) - Fix strict-mode safe access for optional asi field - Expand Step 4 with macOS-specific symbol package guidance (.symbols NuGet) - Add .dwarf to .dSYM conversion instructions - Add src/coreclr/ to validation paths - Soften stop signals to allow crash analysis and deeper investigation - Add macOS Symbol Packages and JSON Parsing Gotchas to reference doc - Create training log documenting session findings - Add .github/skills/ project-local copy for CLI testing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: automated version extraction and symbol acquisition Script improvements: - Preserve full image paths from crash log (previously discarded by GetFileName) - Add Get-RuntimeVersionFromPath: extracts .NET version from image paths (e.g., .../shared/Microsoft.NETCore.App/10.0.4/libcoreclr.dylib) - Add Get-RidFromPath: infers RID from path or crash metadata (OS/CPU) - Path-based version detection as fast primary method, UUID matching as fallback - Emit copy-pasteable symbol acquisition commands when dSYMs are missing - Show .NET version in ParseOnly library listing SKILL.md updates: - Step 2: document automated version detection and acquisition commands - Step 4: script now prints ready-to-run download/conversion commands Training log: record session 2 findings (5 issues, script + SKILL.md changes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: add symbol server anti-pattern dotnet-symbol and msdl.microsoft.com do not serve macOS dSYM/DWARF symbols — only Windows PDBs and Linux ELF debug info. NuGet packages are the only public source. Added anti-pattern to SKILL.md (both copies) and reference doc to prevent wasted tool calls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: fix symbol server guidance — dotnet-symbol works for macOS dotnet-symbol --symbols <binary> successfully downloads .dwarf debug symbols for macOS Mach-O binaries from msdl.microsoft.com. Previous commit incorrectly claimed this didn't work. Replaced anti-pattern with positive guidance in SKILL.md (both copies) and reference doc. Added training log entry documenting the correction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Promote dotnet-symbol as preferred macOS symbol acquisition method - Script: macOS acquisition block now shows Option A (dotnet-symbol, preferred) and Option B (.symbols NuGet, fallback). Fixed .dwarf filename doubling bug in cp command. - SKILL.md (both copies): Step 2 updated for dotnet-symbol preference, Step 4 reordered with dotnet-symbol as #2 and NuGet symbols as #3. - Reference doc: Restructured macOS Symbol Packages section with Preferred/Fallback subsections and shared .dwarf→.dSYM conversion. - Training log: Session 4 entry documenting the promotion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add automated symbol server download for macOS crash symbolication - Add Get-DebugSymbols function: downloads .dwarf files from Microsoft symbol server using Mach-O UUID (mirrors android tombstone approach) - Add Convert-DwarfToDsym function: creates .dSYM bundle from .dwarf with UUID verification via dwarfdump - New params: -SymbolCacheDir, -SymbolServerUrl, -SkipSymbolDownload - Wire download+conversion into main flow after local dSYM search - Refactor manual acquisition guidance as fallback-only - Update both SKILL.md copies: frontmatter, Step 4, new flags - Update ips-crash-format.md: automated download as primary method - Add training log session 5 URL pattern: https://msdl.microsoft.com/download/symbols/_.dwarf/mach-uuid-sym-{UUID}/_.dwarf Verified end-to-end: 391/391 .NET frames symbolicated with clean cache. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: add triage order guidance to Step 3 Step 3 now instructs the agent to explain the faulting mechanism (frames #0-#1) before examining cross-thread context. This addresses a misdiagnosis where GC activity on neighboring threads was mistaken for causation when the actual root cause (_sigtramp -> NULL signal handler) was visible in the crashing thread's first two frames. Training log updated with session retrospective and corrected the original crash description from GC race to NULL signal handler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove stale .github/skills/apple-crash-symbolication remnant Skill lives in plugins/dotnet-diag/skills/apple-crash-symbolication/ since the plugin restructuring. The old .github/skills/ copy was a leftover. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add libimobiledevice.org to known-domains.txt The reference scanner CI check fails because the apple-crash-symbolication skill references https://libimobiledevice.org/ which is not in the allowed domains list. This domain hosts the libimobiledevice project, a community library for communicating with iOS devices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Delete eng/reference-scanner/known-domains.txt * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add libimobiledevice.org to known-domains allowlist The apple-crash-symbolication SKILL.md references libimobiledevice.org for the idevicecrashreport tool. Add the domain to the known-domains file to fix the skill-check CI reference validation failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restructure apple-crash-symbolication for analysis-first approach The skill was getting ❌ verdicts on all eval scenarios because it directed the LLM to 'run the script' instead of teaching crash analysis domain knowledge. This mirrors the problem where quality decreased from 3.0 to 2.7 in the parse scenario. Restructure to follow the Android sibling's proven pattern: - Lead with parsing (.ips two-part JSON format, key fields) - Teach .NET library identification (inline library table) - Teach crash interpretation (asi, faulting thread, exception) - Teach atos command construction (concrete examples) - Teach dSYM search paths (ordered list with commands) - Move automation script to optional section at the end - Move crash log retrieval to a separate section (not Step 1) Trim the reference doc to avoid duplication, keeping only macOS symbol distribution differences and supported RID list. Token budget: ~2,100 tokens (within 800-2,500 optimal range). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refine prompts and assertions in eval.yaml for iOS crash symbolication scenarios * Add format verification guard and fix overfitting in reject scenario - SKILL.md: Add explicit format check at start of Step 1 to verify .ips JSON before proceeding; add 'Wrong file format' stop signal - eval.yaml: Remove direct skill reference from scenario 3 prompt to reduce overfitting (was triggering skill activation in plugin mode) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix scenario 3: provide actual Android tombstone file for rejection test - eval.yaml: Replace invalid 'extra_files' with correct 'files' syntax (extra_files was silently ignored, so crash_android.txt was never copied) - eval.yaml: Remove copy_test_files for scenario 3 to avoid copying ios_crash.ips which tempts the agent in plugin mode - Add tombstone_sample.txt to test directory (copy from android sibling) - SKILL.md: Mention ndk-stack/addr2line in wrong-format stop signal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update tests/dotnet-diag/apple-crash-symbolication/tombstone_sample.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedback: simulator RIDs, perf, safety, accuracy fixes - Add simulator RIDs (iossimulator, tvossimulator) to script search list - Build hashtable for O(1) image lookups in Get-ThreadFrames - Remove -UseBasicParsing (unnecessary in pwsh 7+) - Include UUID in Convert-DwarfToDsym cache path; sanitize library name - Make version regex greedy to capture pre-release suffixes - Improve format detection error message for non-.ips files - Guard xcrun fallback with Get-Command check - Add .dwarf-to-.dSYM conversion guidance for macOS manual fallback - Escape .ips in eval.yaml regex assertion - Fix UUID note in reference doc (normalize, not assume lowercase) - Fix symbol server example to use crash-log image name Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 2: case-sensitive replace, dedup, path fix, redact PII - Use -creplace with guard for vmregioninfo duplicate key handling - Use Sort-Object -Unique for proper library deduplication - Fix macOS fallback to create .dSYM bundles under symbols-out/ - Redact device identifiers in test fixture .ips file Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Prevent dSYM cache poisoning: validate UUID on cache hit, remove on mismatch - Convert-DwarfToDsym now verifies cached dSYM UUID before reusing - On UUID mismatch during download, remove the bad cached bundle so subsequent runs can retry cleanly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cache validation: normalize UUID comparison, clean up dwarf + bundle on mismatch - Use Format-Uuid to normalize dwarfdump output before comparing to already-normalized \ in Convert-DwarfToDsym cache check - On UUID mismatch after download, remove the entire .dSYM bundle directory and the cached .dwarf file to prevent repeat failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resolve-Frames null handling: use List[object] to preserve null entries PowerShell array += \ silently drops the element, breaking index alignment between results and input addresses. Switch to List[object].Add() which correctly preserves null entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dSYM bundle root traversal, qualify version-from-path note - Walk up directory tree until *.dSYM is found instead of going up only 2 levels (which lands at Contents/Resources, not the bundle) - SKILL.md: note that version-in-path only works on macOS shared framework installs; iOS paths don't embed the runtime version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove incorrect .github/skills path reference from training log The .github/skills/ directory doesn't exist in this repo. The file was already listed under its actual plugins/dotnet-diag/ path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Detect simulator for RID inference in manual fallback guidance Detect CoreSimulator in image paths to use iossimulator-/tvossimulator- RIDs for simulator crashes, avoiding UUID mismatches from wrong runtime packs. Also handle arm64e CPU type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resolve-Frames return type and scope issue in simulator detection - Return \.ToArray() instead of ,\ to avoid wrapping the list in a single-element array - Use \.usedImages instead of undefined \ in RID inference block Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify ParseOnly .NET Libraries section shows only frame-relevant images Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix YAML escape in assertion pattern (use single quotes for regex backslash) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten stop signal to prevent Android symbolication spillover The apple-crash-symbolication skill's stop signal for wrong file formats was naming specific Android tools (ndk-stack, addr2line) and the android-tombstone-symbolication skill. This caused models to learn about Android symbolication from the Apple skill and then actually execute it, resulting in 4-5x token bloat and completion regression on the Android rejection eval scenario. Remove tool suggestions from the stop signal — just say 'stop, don't symbolicate.' The Android skill handles its own routing when loaded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify scenario 3 rubric to focus on positive skill knowledge Replace 4 rubric items (including 2 'Did NOT' items) with 2 items that test positive knowledge the skill provides. The 'Did NOT' items gave credit to baseline responses that also don't attempt iOS workflow (because they don't know about it), minimizing the quality delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix nuget.org domain reference: drop www. prefix to match known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge upstream main and use api.nuget.org v3 endpoint Main changed known-domains.txt to use path-scoped entries (nuget.org/account/trustedpublishing instead of bare nuget.org). Switch the symbols download URL to the v3 flatcontainer endpoint on api.nuget.org which is in the allowlist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .gitignore merge conflict Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Steve Pfister <stpfiste@microsoft.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |
||
|
Amaury Levé
|
93dc33da86 |
Stabilize and unify some test skills (#501)
* Deduplicate test skill references and clarify skill boundaries - Move platform-detection.md and filter-syntax.md to plugins/dotnet-test/shared/, removing 3 identical copies of each from run-tests, mtp-hot-reload, and migrate-vstest-to-mtp reference directories. - Move dotnet.md from exp-test-smell-detection/extensions/ to shared/ as dotnet-test-frameworks.md in both dotnet-test and dotnet-experimental plugins. Update exp-assertion-quality, exp-test-boilerplate-detection, exp-test-tagging, and test-anti-patterns to reference the shared file instead of inlining framework detection tables. - Differentiate test-anti-patterns (quick pragmatic review) from exp-test-smell-detection (deep formal audit with academic taxonomy) by updating descriptions and cross-referencing each other in When Not to Use sections. - Update skill-validator to allow ../../shared/ file references while still blocking other parent-directory traversals. Add tests for the new rule. * Switch from shared/ directories to hidden reference skills Replace the plugin-level shared/ directories with non-invocable reference skills (user-invocable: false) that other skills reference by name. - Create platform-detection, filter-syntax, and dotnet-test-frameworks as hidden skills under plugins/dotnet-test/skills/. These contain the detection tables and syntax references previously duplicated across run-tests, mtp-hot-reload, and migrate-vstest-to-mtp. - Create exp-dotnet-test-frameworks as a hidden skill under plugins/dotnet-experimental/skills/ for the experimental test analysis skills (exp-test-smell-detection, exp-assertion-quality, etc.). - Update all consuming skills to reference these by skill name in backtick notation instead of file links. - Revert the skill-validator ../../shared/ exception — no longer needed since all references now use the standard skill name mechanism. * Merge exp-test-boilerplate-detection into exp-test-maintainability exp-test-maintainability was only 6 calibration rules with no workflow. exp-test-boilerplate-detection had the full 5-category detection workflow, examples, calibration, and validation. Merge the boilerplate content into exp-test-maintainability (the broader, more user-facing name) and add the two unique maintainability rules (DisplayName guidance, DataRow vs DynamicData preference) to Category 3. - Replace exp-test-maintainability SKILL.md with the merged content - Move test fixtures from exp-test-boilerplate-detection to exp-test-maintainability - Merge eval.yaml scenarios (4 total: 2 from each original skill) - Delete exp-test-boilerplate-detection skill and tests - Update all cross-references in exp-test-smell-detection, dotnet-test-frameworks, exp-dotnet-test-frameworks, and CODEOWNERS * Add cross-references to test-anti-patterns for deep mock and duplication analysis Point users to exp-mock-usage-analysis from the Over-mocking entry and to exp-test-maintainability from the Duplicate tests entry. * Add exp-dotnet-test-frameworks to CODEOWNERS * Improve run-tests SDK 10 MTP detection for blame-hang scenario Inline the critical SDK 10 detection signal (global.json test.runner) directly in Step 1 instead of deferring entirely to the platform-detection skill. This makes the distinction between SDK 10 (no -- separator) and SDK 8/9 (requires -- separator) more prominent. Add a quick detection summary table, strengthen the Common Pitfalls entry for SDK 10 with a blame-hang-timeout example, and keep the platform-detection skill reference for the full detection logic. * Improve skill activation keywords in descriptions - exp-test-maintainability: Add 'suggest a better test structure', 'consolidate similar test methods', 'convert copy-paste tests to data-driven parameterized tests' to match prompts like 'each new case needs a whole new method, suggest a better structure'. - test-anti-patterns: Add 'review tests', 'find test problems', 'check test quality', 'audit tests for common mistakes' to match review-style prompts that don't use the word 'anti-pattern'. - run-tests: Add 'hang timeout', 'blame-hang', 'blame-crash', 'TUnit' to match SDK 10 blame scenarios and TUnit filter scenarios that were intermittently not activating. |
||
|
Cathy Sullivan
|
42e29303b4 |
Add coverage-analysis skill to dotnet-test plugin (#499)
* Add coverage-analysis skill to dotnet-test plugin Adds an automated, project-wide code coverage and CRAP score analysis skill for .NET projects. Complements the existing crap-score skill (which is manual/targeted for single methods) with an end-to-end pipeline: - Auto-detects solution structure and test projects - Detects coverage provider (Microsoft.Testing.Extensions.CodeCoverage or Coverlet) per test project and selects the correct dotnet test arguments - Runs coverage collection, generates reports via ReportGenerator - Calculates CRAP scores per method via bundled PowerShell scripts - Surfaces risk hotspots ranked by danger (complex + uncovered code) - Produces actionable recommendations with projected score improvements Boundary with crap-score: coverage-analysis is project-wide and automated; crap-score is targeted and LLM-directed. The frontmatter descriptions cross-reference each other so the skill router disambiguates correctly. Includes eval.yaml with 3 scenarios (existing data analysis, from-scratch coverage collection, coverage plateau diagnosis) and shared buildable fixtures. Fixes #457 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR review: coverage output path and scoped tool install - Add --coverage-output $rawDir to all MS CodeCoverage dotnet test commands (unified, .NET 9, .NET 10+, and mixed-project modes) so Cobertura XML lands in the expected directory for downstream search - Replace dotnet tool install --global with --tool-path scoped to the coverage output directory to avoid mutating the user environment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Cobertura XML fixture line numbers to match StudentService.cs FindById starts at line 101 (was 88), Search starts at line 109 (was 96). Updated both partial-coverage and plateau fixtures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wrap Get-ChildItem pipelines in @() for single-result array safety PowerShell returns a scalar FileInfo (not array) when a pipeline yields exactly one result, making .Count return null. Wrapping in @() ensures the result is always an array so .Count works reliably for the common single-test-project case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove CODEOWNERS change - not needed for this PR The existing /plugins/dotnet-test/ and /tests/dotnet-test/ wildcards already route reviews to @dotnet/dotnet-testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: .csproj fallback, cross-platform paths, .sln header - Add .csproj fallback discovery in Step 1 when no .sln found (emit ENTRY_TYPE:Project) to match documented behavior - Fix ProjectReference backslash to forward slash in test fixture .csproj - Remove leading blank line from ContosoUniversity.sln fixture Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add plateau/stuck trigger language to frontmatter description The skill was NOT ACTIVATED for the 'Coverage plateau diagnosis' eval scenario because the frontmatter description (used by the skill router) lacked plateau-related keywords. The prompt 'coverage is stuck' and 'what's blocking me' didn't match the existing description text. Add 'stuck', 'plateaued', and 'blocking improvement' to the frontmatter Use-when sentence so the skill router activates for plateau diagnosis prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make CRAP scores mandatory, add existing-data shortcut path The plateau eval scenario scored 1/5 on CRAP scores across all modes because the LLM shortcut the workflow when existing Cobertura XML was provided — it read the XML directly and gave a quick diagnostic answer without computing CRAP scores or following the output template. - Add existing-data guidance at top of Workflow section: skip test execution but still run CRAP score computation (Steps 5-6) - Add mandatory CRAP rule to guidelines.md: Risk Hotspots table is required in every output regardless of analysis mode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Mukund Raghav Sharma (Moko)
|
665c39e957 |
Add configuring-opentelemetry-dotnet skill (#268)
* Add configuring-opentelemetry-dotnet skill (+8.1% eval, near-miss) Teaches OpenTelemetry SDK configuration in ASP.NET Core: package selection, AddOpenTelemetry with tracing/metrics/logging, custom ActivitySource and Meter creation, OTLP exporter setup, and context propagation. Eval results: +8.1% improvement (threshold: 10%, near-miss) Includes eval.yaml with OTel setup scenario + negative test. * Sharpen eval.yaml: add custom-spans-invisible pain-point and name-matching rubric emphasis * Migrate configuring-opentelemetry-dotnet to plugins/ directory structure * Move OpenTelemetry skill to new dotnet-aspnet plugin, address review feedback - Move skill from plugins/dotnet-diag to new plugins/dotnet-aspnet plugin - Create dotnet-aspnet plugin.json - Fix SKILL.md frontmatter: remove code fence wrapper so YAML metadata parses - Fix undefined 'order' variable in ProcessOrderAsync example - Move SqlClient/Runtime instrumentation to clearly marked optional sections - Add missing using directives (System.Diagnostics, OpenTelemetry.Context.Propagation) - Handle null Activity.Current in context propagation snippet - Use IMeterFactory via DI instead of static Meter per official guidance - Add OpenTelemetry.Instrumentation.Http to eval rubric package list - Simplify eval prompt to be more natural/generalized - Update CODEOWNERS: add dotnet-aspnet plugin entries, remove old dotnet-diag entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Configure metrics OTLP exporter endpoint explicitly to match tracing config Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address round 2 review: logging endpoint, marketplace, RecordException using, ActivitySource in propagation, HttpClient pitfall - Configure logging OTLP exporter endpoint explicitly for consistency - Add dotnet-aspnet to both marketplace.json files - Add 'using OpenTelemetry.Trace' for RecordException extension method - Declare ActivitySource in context propagation snippet - Fix HttpClient instrumentation pitfall (works with new HttpClient() too) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify logging package source and add ActivitySource registration reminder Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add @dotnet/aspnet team to skill CODEOWNERS entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dotnet-aspnet to README table, make propagation ActivitySource static Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add expect_activation:false, IMeterFactory using, clarify OTLP exporter scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OtlpExporter using directive, clarify OTLP package covers logging Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Prometheus from OTLP claim, add OTLP exporter assertion to eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify Jaeger is traces-only, document logging package provenance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval: increase timeout, reject tools, rewrite prompt as explanation-only - Scenario 1: rewrite prompt to ask for code explanation (not file modification), add reject_tools to prevent bash/edit overhead, increase timeout 120s->360s, update rubric wording - Scenario 2: add reject_tools to eliminate tool-overhead noise Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve OTel skill evals and address review feedback - Remove redundant per-skill CODEOWNERS lines (ViktorHofer feedback) - Add SetResourceBuilder + ParseStateValues to SKILL.md Step 3 (logging provider gap) - Rewrite eval scenarios: 2 targeted positive scenarios testing real skill gaps - Remove noisy negative activation scenario (high variance, -21.5% drag) - Eval results: +14.2% improvement (threshold: 10%), overfitting 0.10 (low) Scenario 1: 3.0/5 -> 5.0/5, Scenario 2: 3.8/5 -> 5.0/5 * Add propagate-trace-context scenario to OTel evals (+25.6%) Third scenario tests trace context propagation across message queues, where the base model scores 3.2/5 and skill-enhanced scores 5.0/5. Removed metrics and debug scenarios that baseline already aced (5.0/5) which only added token cost without measurable improvement. Final: 3 scenarios, +25.6% improvement, overfitting 0.09 (low) * Remove duplicate dotnet-aspnet CODEOWNERS entries (L69-70 shadowed by L86-87) * Use @dotnet/aspnet team alias in CODEOWNERS for wider reach * Addressed feedback --------- Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
David Ortinau
|
c4b6819dda |
Add 7 core .NET MAUI development skills to dotnet-maui plugin (#456)
* Add 7 core .NET MAUI development skills to dotnet-maui plugin Add maui-shell-navigation, maui-collectionview, maui-data-binding, maui-dependency-injection, maui-app-lifecycle, maui-safe-area, and maui-theming skills with eval.yaml tests and reference docs. Update plugin.json, marketplace.json (both copies), and CODEOWNERS. Resolves #455 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cross-skill references to non-existent skills Replace 10 dangling references to skills not in this repo (maui-deep-linking, maui-platform-invoke, maui-unit-testing, maui-maps, maui-animations, maui-accessibility, maui-app-icons-splash, maui-bootstrap-theme, maui-localization) with plain-English guidance or references to skills that exist in this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix code correctness bugs and remaining cross-references Address PR review feedback and skeptic review findings: Code correctness: - Fix BackgroundColor="Primary" (XAML crash) → use {StaticResource Primary} - Fix MergedDictionaries.Clear() (nukes all styles) → remove-then-add pattern - Fix HorizontalStackLayout Fill (no effect) → use Grid with ColumnDefinitions - Fix AncestorType with ViewModel type → use ContentPage (visual element) - Fix Frame usage → replace with Border (recommended control) - Remove redundant MultiBinding StringFormat="{}{0}" - Fix OnStart reference → use CreateWindow for theme restore - Soften lifecycle OnStart/OnSleep characterization (compat shims, not absent) - Qualify ResourceDictionary code-behind requirement (only for x:Class) Cross-references: - Remove all remaining body-text refs to non-existent skills - Replace with Microsoft Learn doc links or generic guidance Eval: - Relax SafeAreaRegions assertion to accept SafeAreaEdges or SafeAreaRegions - Relax theming rubric for ResourceDictionary code-behind Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore Clear() theme pattern with clarifying comment The Clear()+Add() pattern is intentional — theme dictionaries are designed to be the only merged dictionaries in this architecture. Added comment explaining the assumption and what to do if non-theme dictionaries exist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix second-pass review findings: API accuracy and eval robustness Code correctness: - ScrollView SafeAreaEdges="All" → "Container" (ScrollView only honors Container and None; All silently fails for keyboard avoidance) - DI CreateWindow example: add comment noting AppShell must be registered - AddHttpClient: add comment noting Microsoft.Extensions.Http NuGet required - MultiBinding: remove redundant StringFormat from reference doc (matches SKILL.md) - RelativeSource Self table: Path=Width → Path=WidthRequest (Width is read-only -1 before layout; WidthRequest is the correct bindable property) - CollectionView StackLayout warning: clarify it refers to BindableLayout on StackLayout vs CollectionView ItemsLayout types - SafeAreaEdges: fix "attached property" → "property" (it's a regular property) - ios:NavigationPage example: add xmlns comment for ios namespace Eval assertions: - DI Scoped regex: accept "doesn't"/"lacks"/"without" not just "no" - CollectionView ViewCell: match negation in either direction around ViewCell - Lifecycle events: require 3+ distinct events instead of any single one Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix final PR review comments - safe-area eval: tighten Container assertion to require SafeAreaEdges context - DI: merge #else fallback into single compilable #if block - lifecycle: behaviour → behavior (American English consistency) - theming: macOS Catalyst minimum 10.14 → 10.15 (Catalina required) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
948a9a52ef |
Add exp-mock-usage-analysis skill and evaluation (#483)
* Add exp-mock-usage-analysis skill and evaluation New experimental skill that detects unused, redundant, or unnecessary mocks in .NET test suites. Covers Moq, NSubstitute, FakeItEasy, and manual test doubles. - SKILL.md with 6-step workflow: gather code, classify mocks, detect anti-patterns, incorporate runtime data, generate recommendations, report findings - eval.yaml with 10 scenarios: trivial type mocking, unused setups, redundant configs, stable utility mocks, well-placed mocks (no false positives), NSubstitute support, FakeItEasy support, setup sprawl, and 2 non-activation boundary tests - Fixture files for each scenario with production + test code - CODEOWNERS entry for @dotnet/dotnet-testing * Remove no-headroom scenarios, increase NSubstitute timeout Remove 4 scenarios where baseline already scores 5.0/5 (model handles these well without skill guidance): - Detect mocking of DTOs, records, and enums - Recognize well-placed mocks without inventing false positives - Decline when asked about non-mock test anti-patterns - Decline mock framework migration request Increase NSubstitute scenario timeout from 120s to 180s to avoid baseline timeouts impacting scoring. Remaining 6 scenarios all show positive skill impact (3→5, 3→4, etc). * Document areas the model handles well without the skill Add a subsection under 'When Not to Use' listing mock-related tasks where the base model already scores 5.0/5 without skill guidance: - Mocking trivial types (DTOs, records, enums) - Recognizing well-placed mocks (no false positives) * Fix md * Improve |
||
|
Amaury Levé
|
d5ca4c6c01 |
Add exp-test-smell-detection skill and evaluation (#491)
* Add exp-test-smell-detection skill and evaluation - Language-agnostic core skill detecting 10 test smells (conditional logic, mystery guest, sleepy test, assertion-free, eager test, magic numbers, sensitive equality, exception handling, general fixture, ignored tests) - .NET extension with framework-specific detection tables (MSTest/xUnit/NUnit/TUnit) - Reference catalog with 19 smells from testsmells.org research - Evaluation with 4 scenarios: smelly suite, clean suite, integration tests, non-activation - CODEOWNERS entry for @dotnet/dotnet-testing * Improve integration test fixture and non-activation timeout - Add real smells to integration test fixture (Thread.Sleep, conditional logic, assertion-free test) mixed with legitimate integration patterns - Update rubric to test nuanced distinction between real smells and acceptable integration test patterns - Increase non-activation scenario timeout from 120s to 300s to prevent timeout-related scoring penalties * Fix test projects * Add domains for testsmells * Remove extra domains |
||
|
Dan Moseley
|
165e6439fe |
Tighten known-domains allowlist (#496)
* Tighten known-domains allowlist Narrow broad domain entries to specific paths/subdomains actually used: - aka.ms -> aka.ms/dotnet-trace, aka.ms/perfcollect - nuget.org -> api.nuget.org, www.nuget.org/account/trustedpublishing - npmjs.com/package/@modelcontextprotocol -> pinned to inspector/v/0.21.1 Remove unused domains: fluentassertions.com, speedscope.app Update skill content to match: pin MCP Inspector link, remove FluentAssertions link, remove speedscope hyperlink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * updated mcp skills codeowner * Refine scoped known-domain entries Remove the remaining nugettest.org publish example and drop the nugettest.org allowlist entry. Also normalize path-scoped entries to omit a hardcoded www prefix so both host forms match the existing ReferenceScanner logic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Leslie Richardson <lerich@microsoft.com> |
||
|
Amaury Levé
|
eb0f98eba6 |
Add exp-test-gap-analysis skill for pseudo-mutation test gap detection (#490)
New experimental skill that performs pseudo-mutation analysis on .NET production code to find gaps in existing test suites. Covers boundary, boolean/logic, return value, exception removal, arithmetic, and null-check mutation categories. Includes eval.yaml with 4 scenarios (boundary gaps, logic gaps, well-tested code recognition, non-activation) and fixture projects. |
||
|
Youssef Victor
|
ae9e2b4302 |
Add SKILL for xunit.v3 migration (#484)
* Add SKILL for xunit.v3 migration * eval.yaml, to be reviewed * Add run_command_and_assert to eval.yaml for dotnet test validation Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8e1539d5-327e-4216-9a01-5aa062e53608 Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix eval * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix eval.yml * Update eval * Update skill * Increase timeout * Update eval.yaml * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/dotnet-test/migrate-xunit-to-xunit-v3/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
59d4485e14 |
Improve dotnet-template-engine skills and eval suite, add template-validation skill (#480)
* Improve dotnet-template-engine eval quality and plugin activation - Replace useless 'Create project with specific framework' test (vanilla=10/10 in all 30 runs, zero skill delta) with harder scenarios that vanilla cannot trivially solve - Expand eval suite from 5 to 10 scenarios to reduce overfitting signal and improve coverage of untested capabilities (dry-run, search, install, template creation from existing project, ambiguous intent resolution) - Add file_not_exists assertion to dry-run scenario to verify projects are not actually created during preview - Add output_matches assertions with specific patterns (e.g. 'individual', 'blazor') instead of relying solely on output_contains - Increase all timeouts from 120s to 180s to reduce MCP cold-start timeouts - Enrich plugin.json description with activation keywords (console app, class library, web API, Blazor, MAUI, framework versions) to fix Plugin mode activation failures (19/21 not-activated events were from one scenario) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add template-validation skill with eval scenarios New skill encoding template.json validation rules: required fields, identity format, shortName conflicts with CLI commands, symbol type/datatype validation, choice parameter default checking, computed/generated symbol completeness, parameter prefix collisions, source condition syntax, post-action requirements, constraint configuration, and tag recommendations. Two eval scenarios: - Validate a template with 8+ deliberate errors (missing identity, CLI- conflicting shortName, invalid defaults, incomplete symbols, missing actionId) - Validate a correct template and confirm it's ready to publish Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments - Add ./test-output to dry-run prompt so file_not_exists assertion checks the correct location - Add missing expect_tools: skill to template-authoring validate scenario - Add 'using Xunit;' to seeded test file so the setup project is compilable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten assertions and replace low-delta scenario - Replace low-delta 'worker service search' (vanilla solves trivially) with 'create multi-project solution' (two projects + project reference — vanilla struggles with multi-step orchestration) - Fix gRPC search assertion: replace trivial output_contains 'template' with specific pattern matching gRPC service/package names - Fix validation restore assertion: require 'shortName' context near 'restore' to prevent false pass from post-action description - Fix correct-template assertion: replace fragile output_not_contains with positive assertion that agent mentioned specific fields it checked - Fix create-from-existing assertion: replace pass-anything pattern with assertion requiring template.json / sourceName / .template.config mention Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 2) - Change datatype 'integer' to 'int' in eval fixture and SKILL.md to align with actual template engine schema - Tighten correct-template assertion: require unambiguous success signal (e.g. 'no errors', 'template is valid') instead of permissive pattern that matches 'not ready' or 'pass' in other contexts - Replace output_not_contains 'missing required' (fails on 'no missing required fields') with positive assertion checking field names - Remove int/integer dual listing from valid datatypes in SKILL.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval failures from CI run 4 scenarios failed; fixes: - Web API + AOT: increase timeout 180s -> 300s (auth scaffolding + NuGet restore needs more time) - Dry-run preview: rephrase prompt to explicitly mention 'template' for better skill activation matching - Multi-project solution: replace with simpler 'create xUnit test project with net8.0' — the multi-project scenario gave vanilla a HIGHER score (4.7) than skilled (4.3) because the skill added overhead without helping - Create-from-existing: increase timeout 180s -> 300s (template generation burned 256K tokens and timed out) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dishonest eval adjustments - Revert dry-run prompt to natural language — don't coach activation by injecting 'template' keyword. If skill doesn't activate, that's a real signal about activation quality. - Replace recycled 'xUnit + net8.0' test (same ceiling problem as the classlib + net9.0 test we removed) with 'Blazor + interactive rendering' which tests parameter knowledge vanilla is less likely to have (render mode selection, blazor vs blazorserver vs blazorwasm template choice). - Keep timeout increases (300s for web API + auth, 300s for create-from- existing) as these address real infrastructure constraints. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move dry-run scenario from template-instantiation to template-discovery The dry-run scenario failed with 'NOT ACTIVATED' because the platform couldn't match a preview/dry-run prompt to the template-instantiation skill. This is correct behavior — previewing what files a template produces is a discovery operation, not instantiation. The template-discovery SKILL.md explicitly covers dry-run in Step 4: 'Use dotnet new <template> --dry-run to show what files a template would create without writing anything to disk.' This is an objective reclassification, not prompt tuning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill content causing eval failures template-instantiation: Add parameter reference table with common parameter combinations (auth values, interactivity modes, AOT flag) for webapi, blazor, grpc, worker templates. The web API + AOT scenario scored 2.0 for both vanilla and skilled because the skill mentioned --auth and --aot but never explained what values they accept. Now the agent gets actionable knowledge. template-authoring: Trim verbose content to reduce token overhead. The create-from-existing scenario burned 256K tokens (vs 185K vanilla) and timed out because the agent read packaging/distribution steps it didn't need. Condensed Step 1 example, removed Step 5 (packaging), compressed Step 4 (testing). Reduces skill size by ~30%. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 3) - Web API+AOT: replace output_matches for AOT with file_contains checking for PublishAot in the generated .csproj — proves the parameter was actually passed, not just mentioned in text - Validation multi-error: add assertions for bool default (enableLogging/ yes), int default (maxRetries/abc), computed missing value, generated missing generator — now enforces 7 of 8 rubric items via assertions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 4) - Fix YAML double-quote escaping for regex word boundaries - Fix worker template example to include --aot flag - Fix webapi example to show --auth Individual --aot together - Fix grammar: Available Symbol Generators Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove web API+AOT and Blazor eval scenarios Web API+AOT consistently degraded quality (run 1: 2.0->2.0, run 2: 3.0->2.7). The skill adds reading overhead on execution tasks without improving the result. Blazor hit ceiling (5.0->5.0) — vanilla already creates Blazor apps perfectly, so the scenario contributes zero signal. template-instantiation now has 1 honest scenario (console app, +1.0 delta). The skill's value is modest for execution tasks; the PR's strength is in discovery (+2.3 to +3.7), validation (+1.7 to +2.0), and authoring (+2.0) where knowledge matters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 5) - Tighten bool/int validation assertions to require explicit error wording (invalid, not valid, must be) alongside field/value — prevents passing when agent just echoes the JSON without diagnosing the issue - Add actionId assertion to enforce detection of missing postAction actionId (was in rubric but not in assertions) - Use single-quoted YAML for patterns with backslash sequences Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Mukund Raghav Sharma (Moko)
|
c92faa7800 |
Add minimal-api-file-upload skill (#264)
* Move minimal-api-file-upload skill to aspnetcore plugin with 3 eval scenarios Per repo restructuring feedback, ASP.NET Core specific skills should be under the aspnetcore plugin rather than the dotnet plugin. * Increase eval timeouts to 180s to avoid timeout-related scoring penalties * Generalized and addressed feedback * Improve eval scenarios: remove noisy tests, add dual-limit and security review scenarios * Add CODEOWNERS for aspnetcore plugin * Fix eval: enrich prompts for skill activation, bump security review timeout to 240s * Drop size-limit scenario (baseline already 5.0), keep 3 high-signal scenarios * Reframe security review as file upload fix to improve skill activation * Review * Contribution guide followed * Updated name * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-aspnet/skills/minimal-api-file-upload/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR #264 review feedback from Brennan and Copilot - Rename plugin from aspnetcore to dotnet-aspnet per repo conventions - Fix streaming filename: remove user-controlled extension, use bare GUID - Remove explicit sanitization code (Option B) per Brennan's feedback - Add case-insensitive MIME type comparison - Validate ContentType matches magic bytes (reject mismatches) - Fix stream rewind + CopyToAsync: copy from already-opened stream - Reword streaming section to accurately describe buffering vs chunks - Update marketplace.json, README.md, CODEOWNERS for dotnet-aspnet rename --------- Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
e4670b33a1 |
[PoC] Code testing agent + tests PoC (#433)
* Add code testing agent agents + tests PoC * Remove AssertionEvaluator.cs change (moved to dev/jankrivanek/agents-evals) |
||
|
Amaury Levé
|
51d29a086f | Improve test skills (#428) | ||
|
Amaury Levé
|
b527b41573 |
Add exp-test-boilerplate-detection and exp-assertion-quality skills (#468)
Two new experimental skills for dotnet-experimental plugin: - exp-test-boilerplate-detection: Detects duplicate boilerplate patterns across .NET test suites and identifies refactoring opportunities. Categories: repeated construction, assertion patterns, copy-paste methods, duplicated setup/teardown, repeated infrastructure. - exp-assertion-quality: Analyzes assertion variety and depth across .NET test suites. Produces metrics dashboard including assertion count, type spread, trivial %, assertion-free tests, and negative assertion coverage. Both skills include eval.yaml with 3-4 scenarios each (positive, negative, and non-activation) plus test fixtures. |
||
|
Jan Jones
|
873c2e4381 |
Mention more directives in the FBA skill (#307)
* Mention more directives in the FBA skill * Address feedback * Mention shebang Co-authored-by: Damian Edwards <damian@damianedwards.com> --------- Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Damian Edwards <damian@damianedwards.com> |
||
|
Julien Couvreur
|
c6be495147 |
Create README.md for dotnet plugin (#460)
I expect this will fix https://github.com/dotnet/skills/issues/458 |
||
|
Amaury Levé
|
a220f36256 |
Add exp-test-maintainability skill for test structure review (#418)
New experimental skill that assesses test maintainability: duplication, test size, data-driven patterns, display names, builder/helper extraction, and shared setup. |
||
|
leslierichardson95
|
7286c31118 |
Add skills for C# MCP Server Development (#317)
* Add C# MCP server skills (create, debug, test, publish) Four new skills for the C# MCP server development lifecycle: - mcp-csharp-create: Scaffolding with dotnet new mcpserver, tools/prompts/resources, transport config - mcp-csharp-debug: MCP Inspector, VS Code integration, breakpoint debugging, logging - mcp-csharp-test: Unit tests with ClientServerTestBase, integration with WebApplicationFactory, evals - mcp-csharp-publish: NuGet packaging, Docker/Azure deployment, MCP Registry publishing Each skill includes SKILL.md with progressive disclosure references/ and eval.yaml tests. * Fix mcp-csharp-create eval: use regex assertions for combined attribute syntax Replace scaffolding-heavy scenarios with implementation-focused ones that test MCP-specific features (resources, prompts, logging). Fix assertion patterns to match combined C# attribute syntax [McpServerTool, Description()] instead of requiring standalone [McpServerTool]. Increase timeouts to 180s to account for skill-reading overhead. Validator result: passed=True, improvement=44.6% (threshold=10%) * Add CODEOWNERS entries for MCP C# skills (create, debug, publish, test) * Update plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-publish/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Remove evaluation step and guide from mcp-csharp-test skill * Add CODEOWNERS entries for dotnet-maui skills * Update CODEOWNERS to add @artl93 as co-owner for mcp-csharp skills * Replace parent-directory file references with backtick skill names Cross-skill references used ../sibling/SKILL.md paths which the validator rejects. Replace with backtick-quoted skill names to match the convention used by other skills in the repo. * Add known domains for MCP server development skills Add domains referenced by the mcp-csharp-* skills: - github.com/modelcontextprotocol/registry - github.com/open-telemetry/semantic-conventions - npmjs.com, xunit.net, fluentassertions.com, nugettest.org Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Docker commands to use placeholder for registry and image names * moved mcp skills and their evals into dotnet-ai plugin * cleaned up moved files to dotnet-ai * cleaned up files * Update .github/CODEOWNERS Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update nuget-packaging.md * Update repository URL format in nuget-packaging.md * Add github.com/ to known-domains to allow truncated placeholder URLs The URL regex truncates at angle brackets (invalid in URIs), so placeholder URLs like https://github.com/<yourusername>/<repo> become https://github.com/ which didn't match any path-scoped entry. Adding github.com/ (with trailing slash) covers only this case without broadly allowing all github.com URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Docker tag command to use placeholder for server name * Update known domains and fix GitHub username placeholders in documentation * Remove McpServerToolType from mcp-csharp-create eval McpServerToolType is only needed with WithToolsFromAssembly(), which is discouraged because it is not Native AOT compatible. Updated eval assertions and rubrics to prefer WithTools<T>() instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve eval resilience for mcp-csharp-create and mcp-csharp-debug - Increase mcp-csharp-create timeouts from 180s (or default 120s) to 360s for all 3 scenarios. These scenarios consistently time out because the model spends time on bash exploration/scaffolding before writing code. - Add explicit timeout to scenario 3 which was relying on the 120s default. - Broaden mcp-csharp-debug Inspector rubric to accept both the concise single CLI command style and the step-by-step UI configuration walkthrough, since both are correct approaches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Broaden debug 'failing tool' rubric to accept multiple debugging approaches The rubric criterion 'Shows how to attach a debugger' was too narrow. The skilled answer correctly focused on the #1 cause (stdout pollution) but scored low because it didn't show a specific 'attach to process' flow. Broadened to accept any valid debugging approach: attaching, Debugger.Launch(), or launch.json configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 4 eval failures in mcp-csharp-debug and mcp-csharp-test skills mcp-csharp-debug: - Trim SKILL.md verbosity (183->160 lines) and mcp-inspector.md (67->54 lines) - Add Diagnosing Tool Errors section covering debugger, output panel, Inspector, common culprits - Rebalance debugging narrative away from stderr-only focus - Move HTTP logging config to ide-config.md reference mcp-csharp-test: - Add eval/evaluations keywords to frontmatter for activation - Add HTTP tool invocation test pattern (tools/call via WebApplicationFactory) - Trim test-patterns.md bloat (remove Test Categories, Coverage, Input Validation) - Create references/evaluations.md with qa_pair format and read-only/deterministic guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jeff Schwartz
|
55a7d80527 |
adding SIMD/TensorPrimitives to dotent-diag/analyzing-dotnet-performance (#330)
* incorporated feedback from previous pr, iterated with 100+ scalar loop tests to come to this skill, moved from skill to reference in the performance skill * Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedback on SIMD vectorization reference - Check Span<T>/MemoryExtensions before TensorPrimitives (no extra dependency) - Add all supported types (sbyte, ushort, uint, ulong, nint, nuint, char via ushort) - TensorPrimitives: add constraint and applicable types columns (not just float/double) - Add FusedMultiplyAdd, clarify AddMultiply vs MultiplyAdd distinction - Prefer portable APIs over platform-specific intrinsics (allow when perf justifies) - Fix dispatch pattern to use if/else if to avoid pessimizing small inputs - Remove LINQ references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * addressing more feedback * move to the experimental plugin * move the simd tests to exprimental * add codeowners --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
1d56634150 | Create experimental plugin (#446) |