mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
v1.0.0
87 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Copilot
|
c625f09db5 |
Recommend MSBuildProjectName for OutputType conditions in MTP migration skill (#556)
* Initial plan * Improve MTP migration skill guidance for OutputType conditions Agent-Logs-Url: https://github.com/dotnet/skills/sessions/ae2f9ae1-c8c3-4e50-95ab-ccdc08bc388b Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> * Add eval scenario for OutputType conditioning in Directory.Build.props Agent-Logs-Url: https://github.com/dotnet/skills/sessions/734b3736-45a4-4f5d-b83c-3e6ee72fcfae Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> Co-authored-by: Amaury Levé <amauryleve@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
5826c421b4 |
Fix test-anti-patterns plugin activation by disambiguating competing descriptions (#552)
The test-anti-patterns skill was not reliably activated in plugin mode because sibling descriptions overlapped: - test-quality-auditor agent: narrowed to multi-skill audit pipelines only, explicitly excluding single-file reviews - writing-mstest-tests skill: removed 'review' and 'anti-patterns' from description, added explicit exclusion for test quality auditing - test-anti-patterns skill: expanded trigger phrases to better match review prompts (flaky tests, duplication, tests that pass without verifying) |
||
|
Amaury Levé
|
7841934a75 |
Add migrate-mstest-v3-to-v4 eval scenarios for uncovered patterns (#564)
* Add migrate-mstest-v3-to-v4 eval scenarios for uncovered patterns
Add 5 new scenarios (Goals 10-14) covering:
- Verified package update with build/test validation
- Sealed custom TestMethodAttribute with Timeout migration
- TestMethodAttribute/TestMethod display name constructor syntax
- Assert.IsInstanceOfType out parameter breaking change
- TreatDiscoveryWarningsAsErrors and behavioral changes review
Add 4 new fixture directories:
- v3-verified-migration: buildable project for end-to-end validation
- v3-sealed-timeout: sealed attr + TestTimeout.Infinite patterns
- v3-testmethod-displayname: [TestMethod/Attribute("name")] patterns
- v3-instanceoftype: Assert.IsInstanceOfType<T> out param pattern
Covers uncovered validation items (lines 449-451, 458) and code
patterns: [TestMethod], [Timeout], sealed, [TestMethodAttribute],
Assert.IsInstanceOfType.
* Improve skill description for activation and bump scenario timeouts
Rewrite migrate-mstest-v3-to-v4 frontmatter description to improve SDK
semantic matching:
- Lead with symptom (fix build errors) instead of action (migrate)
- Add error codes CS0507, CS0103, CS1061, CS1615 as trigger signals
- Add symptom phrase: tests don't compile after upgrading MSTest
- Expand breaking change keywords: sealed, DisplayName syntax,
Assert.IsInstanceOfType out parameter, TestContext.TestName
- Keep under 1024 char limit
Bump timeout 180s -> 240s for scenarios 11 (sealed attr) and 13
(IsInstanceOfType) that hit timeout during evaluation.
* Address PR review comments
- Remove unused 'using System' from OrderServiceTests.cs and TypeCheckTests.cs
- Renumber duplicate Goal 10 to Goal 15 (non-activation scenario)
- Tighten file_contains assertion to check MSTest.TestFramework and
MSTest.TestAdapter individually instead of broad '4.' match
- Tighten Timeout assertion to require int.MaxValue instead of matching
pre-migration symbol names
- Fix error code CS0507 -> CS0115 in scenario 11 prompt to match the
actual MSTest v4 error for missing Execute override
|
||
|
Amaury Levé
|
1125fe7864 |
improve(dotnet-test): enhance agent descriptions for subagent discovery (#560)
- Add 'Use when:' trigger phrases to all 7 sub-agent descriptions so parent agents can reliably discover and delegate to them - Add .testagent/ cleanup rule to generator agent to prevent ephemeral pipeline state from being committed |
||
|
Amaury Levé
|
9ab90d6400 |
feat(dotnet-test): add handoffs between orchestrator agents (#561)
- test-quality-auditor -> code-testing-generator (generate missing tests) - test-quality-auditor -> testability-migration (fix untestable code) - testability-migration -> code-testing-generator (test migrated code) - test-migration -> test-quality-auditor (audit after migration) Enables guided sequential workflows between agents with suggested next steps, per VS Code custom agent handoff pattern. |
||
|
Copilot
|
a4927f1b79 |
Align writing-mstest-tests skill routing with anti-pattern disambiguation (#553)
* Initial plan * fix(dotnet-test): align writing-mstest-tests routing with anti-pattern skill Agent-Logs-Url: https://github.com/dotnet/skills/sessions/eb70209e-8e7a-4b49-820d-65e0e3fcaedc Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> |
||
|
Amaury Levé
|
400661df09 |
run-tests: add critical rules, detection guidance, and troubleshooting (#514)
* run-tests: add critical rules, detection guidance, and troubleshooting - Add 'Critical Rules' table to prevent cross-platform VSTest/MTP mistakes - Expand Step 1 detection with explicit file-by-file lookup table - Add 'dotnet --version' as first detection action - Add Troubleshooting section with 7 common error patterns - Expand Common Pitfalls from 3 to 7 entries - Strengthen negative guidance for SDK version-specific syntax * Improve description * Improve evaluation * Add expected and rejected tools * Change * Improve do not use sections * Improve prompt * Increase eval timeouts for scenarios hitting the limit - run-tests: trx reporting MTP SDK 9 (360s -> 480s) - run-tests: blame-hang MTP SDK 10 (300s -> 420s) - run-tests: combine filter criteria VSTest (180s -> 300s) - mtp-hot-reload: hot reload SDK 9 (360s -> 480s) - mtp-hot-reload: hot reload filter (180s -> 300s) - code-testing-agent: ContosoUniversity (1800s -> 2400s) |
||
|
Amaury Levé
|
7791735b60 |
Add test-quality-auditor hub agent for .NET test suite analysis (#541)
* Add test-quality-auditor hub agent for .NET test suite analysis Introduces a new test-quality-auditor agent that triages test quality concerns to specialized analysis skills: assertion quality, test smells, anti-patterns, mock usage, test gaps, maintainability, coverage risk, and test tagging. Includes a comprehensive audit pipeline that chains multiple skills for broad health checks. * Address Copilot review: clarify diagnostic role and coverage routing - Reword intro to acknowledge file-modifying workflows (test tagging) are only used with explicit user confirmation - Clarify coverage-analysis vs crap-score: prefer coverage-analysis for project-wide analysis, reserve crap-score for targeted method/class scope |
||
|
Amaury Levé
|
f9d7faf0ee |
Add test-migration hub agent for .NET test framework and platform migrations (#540)
* Add test-migration hub agent for .NET test framework and platform migrations Introduces a new test-migration agent that auto-detects the current test framework, version, and platform, then routes to the appropriate migration skill (MSTest v1/v2→v3, v3→v4, xUnit v2→v3, VSTest→MTP). Handles multi-step migrations with commit boundaries between steps. * Align framework detection with platform-detection skill Address Copilot review: MSTest requires metapackage, Sdk, or both TestFramework+TestAdapter (not either alone). Add missing xUnit MTP package variants. |
||
|
Artur Spychaj
|
3dbe832796 |
Add 'dotnet sln add' step to code-testing-implementer (#522)
* Add 'dotnet sln add' step to code-testing-implementer When a new test project is created, register it with the solution file using 'dotnet sln add' so that 'dotnet test <solution>' discovers the tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: improve dotnet sln add guidance - Use solution identified in research/plan rather than searching - Account for .slnf solution filter scenarios - Prefer 'dotnet test --solution' over 'dotnet test <solution>' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move dotnet sln add guidance to dotnet.md extension The implementer is polyglot, so 'dotnet sln add' doesn't belong there. Replace with a generic 'register with build system' step that defers to extensions/, and add the .NET-specific detail (sln/slnx/slnf handling) to extensions/dotnet.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: fix solution registration guidance in dotnet.md - Include .slnf in 'don't substitute' warning alongside .sln/.slnx - Qualify '--solution' flag as SDK 10+/MTP-only; fall back to positional form for older SDKs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7e6954f9a9 |
Add testability migration skills and agent for dotnet-test (#518)
* Add testability migration skills and agent for dotnet-test Introduces three skills and one orchestrator agent to help developers incrementally replace hard-to-test static dependencies with injectable abstractions: - detect-static-dependencies: scans C# code for DateTime.Now, File.*, Environment.*, HttpClient, Console.*, Process.* and ranks by frequency - generate-testability-wrappers: generates wrapper interfaces or guides adoption of built-in abstractions (TimeProvider, IHttpClientFactory, System.IO.Abstractions) - migrate-static-to-wrapper: mechanical codemod-style bulk replacement of static call sites with wrapper calls, scoped per project/namespace - testability-migration agent: orchestrates the Detect-Generate-Migrate pipeline end-to-end Includes eval.yaml tests and fixture projects for all three skills and the agent. * Fix activation, token overhead, and eval timeouts - Expand skill descriptions with more trigger phrases for migrate-static-to-wrapper and generate-testability-wrappers to fix NOT ACTIVATED issues - Collapse verbose replacement mapping tables and constructor examples to reduce token overhead (~30% fewer tokens) - Increase generate-testability-wrappers eval timeouts from 120s to 240s to prevent timeout-impacted scoring - Soften detect-static-dependencies scenario 2 rubric to reduce run-to-run variance in plugin mode - Increase migrate-static-to-wrapper non-activation timeout |
||
|
Copilot
|
7ab8006c18 |
Fix markdownlint violations in dotnet-test plugin (#513)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com> |
||
|
Milos Kotlar
|
038dd4f087 |
Add Apple crash symbolication skill (#201)
* Add SKILL.md for iOS crash symbolication process Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/ios-crash-symbolication/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add automation script, tests, and review fixes for ios-crash-symbolication - Add Symbolicate-Crash.ps1 (664 lines): parses .ips JSON, searches local dSYMs (SDK packs, NuGet cache, user paths), verifies UUIDs via dwarfdump, batch-symbolicates with atos, identifies runtime version - Update SKILL.md: add Automation Script and Runtime Version Identification sections, fix atos -o to point inside dSYM bundle, add dwarfdump and MAUI to frontmatter, wrap steps in Workflow heading for consistency with Android sibling, fix misleading 'rebuild' guidance per review feedback (Rolf) - Add eval.yaml with 7 test scenarios: Mono crash, CoreCLR crash, no .NET frames, NativeAOT, multi-library UUIDs, ASI field priority, and Android tombstone rejection - Add 5 .ips test fixture files (two-part JSON format) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add iOS crash log and evaluation scenarios for symbolication tests * Remove unused file * Add CODEOWNERS entry for iOS crash symbolication * Rename ios-crash-symbolication to apple-crash-symbolication Address PR feedback to support all Apple platforms (tvOS, Mac Catalyst, macOS) not just iOS: - Add $appleRids array covering ios, tvos, maccatalyst, and osx RIDs - Refactor Find-Dsym and Find-RuntimeVersion to search all platform packs - Rewrite SKILL.md for orchestration focus and reduced token budget - Extract domain knowledge to references/ips-crash-format.md - Tune eval.yaml: outcome-based rubrics, broad assertions, overfit 0.26 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix script robustness: base address parsing and null-safe property access - Simplify base address cast to [uint64] without manual hex prefix stripping - Use PSObject.Properties check before accessing lastExceptionBacktrace Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dSYM-path rubric items to CoreCLR and ASI scenarios Add rubric items testing for skill-specific domain knowledge (SDK pack paths, NuGet cache directories) that baseline agents cannot provide. This creates the quality delta needed to pass the 10% improvement threshold while keeping overfit at 0.12 (Low). 3-run validation: 30.1% improvement, 5/7 scenarios positive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor code structure for improved readability and maintainability * Fix crash symbolication script for real .ips format and improve eval scenarios - Fix Symbolicate-Crash.ps1 strict-mode bugs with real .ips crash logs: safe property access for image name/path (sentinel entries), thread name (most threads unnamed), and single-element array unwrapping (.Count) - Add SKILL.md efficiency guidance: resolve script path from skill directory (no find /), start with -ParseOnly, don't run broad filesystem searches - Reframe eval scenarios for platform-independent evaluation (parse/analyze instead of requiring macOS-only atos/dwarfdump), tighten rubric to test skill-specific knowledge (NuGet package name, all .NET binary images) Validation results (3 runs, claude-opus-4.6): Scenario 1 (parse frames): 3.3 → 4.0 (+0.7) ✅ Scenario 2 (investigate crash): 3.3 → 4.0 (+0.7) ✅ Scenario 3 (reject Android): 3.3 → 4.0 (+0.7) ✅ Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Auto-fallback to parse-only output when atos is unavailable On Linux (CI), atos and xcrun don't exist. Previously the script would error out after completing all parsing, losing the results. Now it detects the missing tool and falls back to ParseOnly output automatically, ensuring the agent always gets structured parse data regardless of platform. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move apple-crash-symbolication to dotnet-diag plugin Relocate skill from plugins/dotnet to plugins/dotnet-diag and tests from tests/dotnet to tests/dotnet-diag. Update CODEOWNERS accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve apple-crash-symbolication: macOS symbols, bug fixes, training log - Fix JSON case-conflict parsing (vmRegionInfo/vmregioninfo duplicate keys) - Fix strict-mode safe access for optional asi field - Expand Step 4 with macOS-specific symbol package guidance (.symbols NuGet) - Add .dwarf to .dSYM conversion instructions - Add src/coreclr/ to validation paths - Soften stop signals to allow crash analysis and deeper investigation - Add macOS Symbol Packages and JSON Parsing Gotchas to reference doc - Create training log documenting session findings - Add .github/skills/ project-local copy for CLI testing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: automated version extraction and symbol acquisition Script improvements: - Preserve full image paths from crash log (previously discarded by GetFileName) - Add Get-RuntimeVersionFromPath: extracts .NET version from image paths (e.g., .../shared/Microsoft.NETCore.App/10.0.4/libcoreclr.dylib) - Add Get-RidFromPath: infers RID from path or crash metadata (OS/CPU) - Path-based version detection as fast primary method, UUID matching as fallback - Emit copy-pasteable symbol acquisition commands when dSYMs are missing - Show .NET version in ParseOnly library listing SKILL.md updates: - Step 2: document automated version detection and acquisition commands - Step 4: script now prints ready-to-run download/conversion commands Training log: record session 2 findings (5 issues, script + SKILL.md changes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: add symbol server anti-pattern dotnet-symbol and msdl.microsoft.com do not serve macOS dSYM/DWARF symbols — only Windows PDBs and Linux ELF debug info. NuGet packages are the only public source. Added anti-pattern to SKILL.md (both copies) and reference doc to prevent wasted tool calls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: fix symbol server guidance — dotnet-symbol works for macOS dotnet-symbol --symbols <binary> successfully downloads .dwarf debug symbols for macOS Mach-O binaries from msdl.microsoft.com. Previous commit incorrectly claimed this didn't work. Replaced anti-pattern with positive guidance in SKILL.md (both copies) and reference doc. Added training log entry documenting the correction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Promote dotnet-symbol as preferred macOS symbol acquisition method - Script: macOS acquisition block now shows Option A (dotnet-symbol, preferred) and Option B (.symbols NuGet, fallback). Fixed .dwarf filename doubling bug in cp command. - SKILL.md (both copies): Step 2 updated for dotnet-symbol preference, Step 4 reordered with dotnet-symbol as #2 and NuGet symbols as #3. - Reference doc: Restructured macOS Symbol Packages section with Preferred/Fallback subsections and shared .dwarf→.dSYM conversion. - Training log: Session 4 entry documenting the promotion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add automated symbol server download for macOS crash symbolication - Add Get-DebugSymbols function: downloads .dwarf files from Microsoft symbol server using Mach-O UUID (mirrors android tombstone approach) - Add Convert-DwarfToDsym function: creates .dSYM bundle from .dwarf with UUID verification via dwarfdump - New params: -SymbolCacheDir, -SymbolServerUrl, -SkipSymbolDownload - Wire download+conversion into main flow after local dSYM search - Refactor manual acquisition guidance as fallback-only - Update both SKILL.md copies: frontmatter, Step 4, new flags - Update ips-crash-format.md: automated download as primary method - Add training log session 5 URL pattern: https://msdl.microsoft.com/download/symbols/_.dwarf/mach-uuid-sym-{UUID}/_.dwarf Verified end-to-end: 391/391 .NET frames symbolicated with clean cache. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: add triage order guidance to Step 3 Step 3 now instructs the agent to explain the faulting mechanism (frames #0-#1) before examining cross-thread context. This addresses a misdiagnosis where GC activity on neighboring threads was mistaken for causation when the actual root cause (_sigtramp -> NULL signal handler) was visible in the crashing thread's first two frames. Training log updated with session retrospective and corrected the original crash description from GC race to NULL signal handler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove stale .github/skills/apple-crash-symbolication remnant Skill lives in plugins/dotnet-diag/skills/apple-crash-symbolication/ since the plugin restructuring. The old .github/skills/ copy was a leftover. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add libimobiledevice.org to known-domains.txt The reference scanner CI check fails because the apple-crash-symbolication skill references https://libimobiledevice.org/ which is not in the allowed domains list. This domain hosts the libimobiledevice project, a community library for communicating with iOS devices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Delete eng/reference-scanner/known-domains.txt * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add libimobiledevice.org to known-domains allowlist The apple-crash-symbolication SKILL.md references libimobiledevice.org for the idevicecrashreport tool. Add the domain to the known-domains file to fix the skill-check CI reference validation failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restructure apple-crash-symbolication for analysis-first approach The skill was getting ❌ verdicts on all eval scenarios because it directed the LLM to 'run the script' instead of teaching crash analysis domain knowledge. This mirrors the problem where quality decreased from 3.0 to 2.7 in the parse scenario. Restructure to follow the Android sibling's proven pattern: - Lead with parsing (.ips two-part JSON format, key fields) - Teach .NET library identification (inline library table) - Teach crash interpretation (asi, faulting thread, exception) - Teach atos command construction (concrete examples) - Teach dSYM search paths (ordered list with commands) - Move automation script to optional section at the end - Move crash log retrieval to a separate section (not Step 1) Trim the reference doc to avoid duplication, keeping only macOS symbol distribution differences and supported RID list. Token budget: ~2,100 tokens (within 800-2,500 optimal range). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refine prompts and assertions in eval.yaml for iOS crash symbolication scenarios * Add format verification guard and fix overfitting in reject scenario - SKILL.md: Add explicit format check at start of Step 1 to verify .ips JSON before proceeding; add 'Wrong file format' stop signal - eval.yaml: Remove direct skill reference from scenario 3 prompt to reduce overfitting (was triggering skill activation in plugin mode) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix scenario 3: provide actual Android tombstone file for rejection test - eval.yaml: Replace invalid 'extra_files' with correct 'files' syntax (extra_files was silently ignored, so crash_android.txt was never copied) - eval.yaml: Remove copy_test_files for scenario 3 to avoid copying ios_crash.ips which tempts the agent in plugin mode - Add tombstone_sample.txt to test directory (copy from android sibling) - SKILL.md: Mention ndk-stack/addr2line in wrong-format stop signal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update tests/dotnet-diag/apple-crash-symbolication/tombstone_sample.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedback: simulator RIDs, perf, safety, accuracy fixes - Add simulator RIDs (iossimulator, tvossimulator) to script search list - Build hashtable for O(1) image lookups in Get-ThreadFrames - Remove -UseBasicParsing (unnecessary in pwsh 7+) - Include UUID in Convert-DwarfToDsym cache path; sanitize library name - Make version regex greedy to capture pre-release suffixes - Improve format detection error message for non-.ips files - Guard xcrun fallback with Get-Command check - Add .dwarf-to-.dSYM conversion guidance for macOS manual fallback - Escape .ips in eval.yaml regex assertion - Fix UUID note in reference doc (normalize, not assume lowercase) - Fix symbol server example to use crash-log image name Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 2: case-sensitive replace, dedup, path fix, redact PII - Use -creplace with guard for vmregioninfo duplicate key handling - Use Sort-Object -Unique for proper library deduplication - Fix macOS fallback to create .dSYM bundles under symbols-out/ - Redact device identifiers in test fixture .ips file Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Prevent dSYM cache poisoning: validate UUID on cache hit, remove on mismatch - Convert-DwarfToDsym now verifies cached dSYM UUID before reusing - On UUID mismatch during download, remove the bad cached bundle so subsequent runs can retry cleanly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cache validation: normalize UUID comparison, clean up dwarf + bundle on mismatch - Use Format-Uuid to normalize dwarfdump output before comparing to already-normalized \ in Convert-DwarfToDsym cache check - On UUID mismatch after download, remove the entire .dSYM bundle directory and the cached .dwarf file to prevent repeat failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resolve-Frames null handling: use List[object] to preserve null entries PowerShell array += \ silently drops the element, breaking index alignment between results and input addresses. Switch to List[object].Add() which correctly preserves null entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dSYM bundle root traversal, qualify version-from-path note - Walk up directory tree until *.dSYM is found instead of going up only 2 levels (which lands at Contents/Resources, not the bundle) - SKILL.md: note that version-in-path only works on macOS shared framework installs; iOS paths don't embed the runtime version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove incorrect .github/skills path reference from training log The .github/skills/ directory doesn't exist in this repo. The file was already listed under its actual plugins/dotnet-diag/ path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Detect simulator for RID inference in manual fallback guidance Detect CoreSimulator in image paths to use iossimulator-/tvossimulator- RIDs for simulator crashes, avoiding UUID mismatches from wrong runtime packs. Also handle arm64e CPU type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resolve-Frames return type and scope issue in simulator detection - Return \.ToArray() instead of ,\ to avoid wrapping the list in a single-element array - Use \.usedImages instead of undefined \ in RID inference block Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify ParseOnly .NET Libraries section shows only frame-relevant images Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix YAML escape in assertion pattern (use single quotes for regex backslash) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten stop signal to prevent Android symbolication spillover The apple-crash-symbolication skill's stop signal for wrong file formats was naming specific Android tools (ndk-stack, addr2line) and the android-tombstone-symbolication skill. This caused models to learn about Android symbolication from the Apple skill and then actually execute it, resulting in 4-5x token bloat and completion regression on the Android rejection eval scenario. Remove tool suggestions from the stop signal — just say 'stop, don't symbolicate.' The Android skill handles its own routing when loaded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify scenario 3 rubric to focus on positive skill knowledge Replace 4 rubric items (including 2 'Did NOT' items) with 2 items that test positive knowledge the skill provides. The 'Did NOT' items gave credit to baseline responses that also don't attempt iOS workflow (because they don't know about it), minimizing the quality delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix nuget.org domain reference: drop www. prefix to match known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge upstream main and use api.nuget.org v3 endpoint Main changed known-domains.txt to use path-scoped entries (nuget.org/account/trustedpublishing instead of bare nuget.org). Switch the symbols download URL to the v3 flatcontainer endpoint on api.nuget.org which is in the allowlist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .gitignore merge conflict Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Steve Pfister <stpfiste@microsoft.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |
||
|
Amaury Levé
|
93dc33da86 |
Stabilize and unify some test skills (#501)
* Deduplicate test skill references and clarify skill boundaries - Move platform-detection.md and filter-syntax.md to plugins/dotnet-test/shared/, removing 3 identical copies of each from run-tests, mtp-hot-reload, and migrate-vstest-to-mtp reference directories. - Move dotnet.md from exp-test-smell-detection/extensions/ to shared/ as dotnet-test-frameworks.md in both dotnet-test and dotnet-experimental plugins. Update exp-assertion-quality, exp-test-boilerplate-detection, exp-test-tagging, and test-anti-patterns to reference the shared file instead of inlining framework detection tables. - Differentiate test-anti-patterns (quick pragmatic review) from exp-test-smell-detection (deep formal audit with academic taxonomy) by updating descriptions and cross-referencing each other in When Not to Use sections. - Update skill-validator to allow ../../shared/ file references while still blocking other parent-directory traversals. Add tests for the new rule. * Switch from shared/ directories to hidden reference skills Replace the plugin-level shared/ directories with non-invocable reference skills (user-invocable: false) that other skills reference by name. - Create platform-detection, filter-syntax, and dotnet-test-frameworks as hidden skills under plugins/dotnet-test/skills/. These contain the detection tables and syntax references previously duplicated across run-tests, mtp-hot-reload, and migrate-vstest-to-mtp. - Create exp-dotnet-test-frameworks as a hidden skill under plugins/dotnet-experimental/skills/ for the experimental test analysis skills (exp-test-smell-detection, exp-assertion-quality, etc.). - Update all consuming skills to reference these by skill name in backtick notation instead of file links. - Revert the skill-validator ../../shared/ exception — no longer needed since all references now use the standard skill name mechanism. * Merge exp-test-boilerplate-detection into exp-test-maintainability exp-test-maintainability was only 6 calibration rules with no workflow. exp-test-boilerplate-detection had the full 5-category detection workflow, examples, calibration, and validation. Merge the boilerplate content into exp-test-maintainability (the broader, more user-facing name) and add the two unique maintainability rules (DisplayName guidance, DataRow vs DynamicData preference) to Category 3. - Replace exp-test-maintainability SKILL.md with the merged content - Move test fixtures from exp-test-boilerplate-detection to exp-test-maintainability - Merge eval.yaml scenarios (4 total: 2 from each original skill) - Delete exp-test-boilerplate-detection skill and tests - Update all cross-references in exp-test-smell-detection, dotnet-test-frameworks, exp-dotnet-test-frameworks, and CODEOWNERS * Add cross-references to test-anti-patterns for deep mock and duplication analysis Point users to exp-mock-usage-analysis from the Over-mocking entry and to exp-test-maintainability from the Duplicate tests entry. * Add exp-dotnet-test-frameworks to CODEOWNERS * Improve run-tests SDK 10 MTP detection for blame-hang scenario Inline the critical SDK 10 detection signal (global.json test.runner) directly in Step 1 instead of deferring entirely to the platform-detection skill. This makes the distinction between SDK 10 (no -- separator) and SDK 8/9 (requires -- separator) more prominent. Add a quick detection summary table, strengthen the Common Pitfalls entry for SDK 10 with a blame-hang-timeout example, and keep the platform-detection skill reference for the full detection logic. * Improve skill activation keywords in descriptions - exp-test-maintainability: Add 'suggest a better test structure', 'consolidate similar test methods', 'convert copy-paste tests to data-driven parameterized tests' to match prompts like 'each new case needs a whole new method, suggest a better structure'. - test-anti-patterns: Add 'review tests', 'find test problems', 'check test quality', 'audit tests for common mistakes' to match review-style prompts that don't use the word 'anti-pattern'. - run-tests: Add 'hang timeout', 'blame-hang', 'blame-crash', 'TUnit' to match SDK 10 blame scenarios and TUnit filter scenarios that were intermittently not activating. |
||
|
Cathy Sullivan
|
42e29303b4 |
Add coverage-analysis skill to dotnet-test plugin (#499)
* Add coverage-analysis skill to dotnet-test plugin Adds an automated, project-wide code coverage and CRAP score analysis skill for .NET projects. Complements the existing crap-score skill (which is manual/targeted for single methods) with an end-to-end pipeline: - Auto-detects solution structure and test projects - Detects coverage provider (Microsoft.Testing.Extensions.CodeCoverage or Coverlet) per test project and selects the correct dotnet test arguments - Runs coverage collection, generates reports via ReportGenerator - Calculates CRAP scores per method via bundled PowerShell scripts - Surfaces risk hotspots ranked by danger (complex + uncovered code) - Produces actionable recommendations with projected score improvements Boundary with crap-score: coverage-analysis is project-wide and automated; crap-score is targeted and LLM-directed. The frontmatter descriptions cross-reference each other so the skill router disambiguates correctly. Includes eval.yaml with 3 scenarios (existing data analysis, from-scratch coverage collection, coverage plateau diagnosis) and shared buildable fixtures. Fixes #457 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR review: coverage output path and scoped tool install - Add --coverage-output $rawDir to all MS CodeCoverage dotnet test commands (unified, .NET 9, .NET 10+, and mixed-project modes) so Cobertura XML lands in the expected directory for downstream search - Replace dotnet tool install --global with --tool-path scoped to the coverage output directory to avoid mutating the user environment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Cobertura XML fixture line numbers to match StudentService.cs FindById starts at line 101 (was 88), Search starts at line 109 (was 96). Updated both partial-coverage and plateau fixtures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wrap Get-ChildItem pipelines in @() for single-result array safety PowerShell returns a scalar FileInfo (not array) when a pipeline yields exactly one result, making .Count return null. Wrapping in @() ensures the result is always an array so .Count works reliably for the common single-test-project case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove CODEOWNERS change - not needed for this PR The existing /plugins/dotnet-test/ and /tests/dotnet-test/ wildcards already route reviews to @dotnet/dotnet-testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: .csproj fallback, cross-platform paths, .sln header - Add .csproj fallback discovery in Step 1 when no .sln found (emit ENTRY_TYPE:Project) to match documented behavior - Fix ProjectReference backslash to forward slash in test fixture .csproj - Remove leading blank line from ContosoUniversity.sln fixture Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add plateau/stuck trigger language to frontmatter description The skill was NOT ACTIVATED for the 'Coverage plateau diagnosis' eval scenario because the frontmatter description (used by the skill router) lacked plateau-related keywords. The prompt 'coverage is stuck' and 'what's blocking me' didn't match the existing description text. Add 'stuck', 'plateaued', and 'blocking improvement' to the frontmatter Use-when sentence so the skill router activates for plateau diagnosis prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make CRAP scores mandatory, add existing-data shortcut path The plateau eval scenario scored 1/5 on CRAP scores across all modes because the LLM shortcut the workflow when existing Cobertura XML was provided — it read the XML directly and gave a quick diagnostic answer without computing CRAP scores or following the output template. - Add existing-data guidance at top of Workflow section: skip test execution but still run CRAP score computation (Steps 5-6) - Add mandatory CRAP rule to guidelines.md: Risk Hotspots table is required in every output regardless of analysis mode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Mukund Raghav Sharma (Moko)
|
665c39e957 |
Add configuring-opentelemetry-dotnet skill (#268)
* Add configuring-opentelemetry-dotnet skill (+8.1% eval, near-miss) Teaches OpenTelemetry SDK configuration in ASP.NET Core: package selection, AddOpenTelemetry with tracing/metrics/logging, custom ActivitySource and Meter creation, OTLP exporter setup, and context propagation. Eval results: +8.1% improvement (threshold: 10%, near-miss) Includes eval.yaml with OTel setup scenario + negative test. * Sharpen eval.yaml: add custom-spans-invisible pain-point and name-matching rubric emphasis * Migrate configuring-opentelemetry-dotnet to plugins/ directory structure * Move OpenTelemetry skill to new dotnet-aspnet plugin, address review feedback - Move skill from plugins/dotnet-diag to new plugins/dotnet-aspnet plugin - Create dotnet-aspnet plugin.json - Fix SKILL.md frontmatter: remove code fence wrapper so YAML metadata parses - Fix undefined 'order' variable in ProcessOrderAsync example - Move SqlClient/Runtime instrumentation to clearly marked optional sections - Add missing using directives (System.Diagnostics, OpenTelemetry.Context.Propagation) - Handle null Activity.Current in context propagation snippet - Use IMeterFactory via DI instead of static Meter per official guidance - Add OpenTelemetry.Instrumentation.Http to eval rubric package list - Simplify eval prompt to be more natural/generalized - Update CODEOWNERS: add dotnet-aspnet plugin entries, remove old dotnet-diag entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Configure metrics OTLP exporter endpoint explicitly to match tracing config Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address round 2 review: logging endpoint, marketplace, RecordException using, ActivitySource in propagation, HttpClient pitfall - Configure logging OTLP exporter endpoint explicitly for consistency - Add dotnet-aspnet to both marketplace.json files - Add 'using OpenTelemetry.Trace' for RecordException extension method - Declare ActivitySource in context propagation snippet - Fix HttpClient instrumentation pitfall (works with new HttpClient() too) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify logging package source and add ActivitySource registration reminder Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add @dotnet/aspnet team to skill CODEOWNERS entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dotnet-aspnet to README table, make propagation ActivitySource static Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add expect_activation:false, IMeterFactory using, clarify OTLP exporter scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OtlpExporter using directive, clarify OTLP package covers logging Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Prometheus from OTLP claim, add OTLP exporter assertion to eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify Jaeger is traces-only, document logging package provenance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval: increase timeout, reject tools, rewrite prompt as explanation-only - Scenario 1: rewrite prompt to ask for code explanation (not file modification), add reject_tools to prevent bash/edit overhead, increase timeout 120s->360s, update rubric wording - Scenario 2: add reject_tools to eliminate tool-overhead noise Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve OTel skill evals and address review feedback - Remove redundant per-skill CODEOWNERS lines (ViktorHofer feedback) - Add SetResourceBuilder + ParseStateValues to SKILL.md Step 3 (logging provider gap) - Rewrite eval scenarios: 2 targeted positive scenarios testing real skill gaps - Remove noisy negative activation scenario (high variance, -21.5% drag) - Eval results: +14.2% improvement (threshold: 10%), overfitting 0.10 (low) Scenario 1: 3.0/5 -> 5.0/5, Scenario 2: 3.8/5 -> 5.0/5 * Add propagate-trace-context scenario to OTel evals (+25.6%) Third scenario tests trace context propagation across message queues, where the base model scores 3.2/5 and skill-enhanced scores 5.0/5. Removed metrics and debug scenarios that baseline already aced (5.0/5) which only added token cost without measurable improvement. Final: 3 scenarios, +25.6% improvement, overfitting 0.09 (low) * Remove duplicate dotnet-aspnet CODEOWNERS entries (L69-70 shadowed by L86-87) * Use @dotnet/aspnet team alias in CODEOWNERS for wider reach * Addressed feedback --------- Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
David Ortinau
|
c4b6819dda |
Add 7 core .NET MAUI development skills to dotnet-maui plugin (#456)
* Add 7 core .NET MAUI development skills to dotnet-maui plugin Add maui-shell-navigation, maui-collectionview, maui-data-binding, maui-dependency-injection, maui-app-lifecycle, maui-safe-area, and maui-theming skills with eval.yaml tests and reference docs. Update plugin.json, marketplace.json (both copies), and CODEOWNERS. Resolves #455 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cross-skill references to non-existent skills Replace 10 dangling references to skills not in this repo (maui-deep-linking, maui-platform-invoke, maui-unit-testing, maui-maps, maui-animations, maui-accessibility, maui-app-icons-splash, maui-bootstrap-theme, maui-localization) with plain-English guidance or references to skills that exist in this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix code correctness bugs and remaining cross-references Address PR review feedback and skeptic review findings: Code correctness: - Fix BackgroundColor="Primary" (XAML crash) → use {StaticResource Primary} - Fix MergedDictionaries.Clear() (nukes all styles) → remove-then-add pattern - Fix HorizontalStackLayout Fill (no effect) → use Grid with ColumnDefinitions - Fix AncestorType with ViewModel type → use ContentPage (visual element) - Fix Frame usage → replace with Border (recommended control) - Remove redundant MultiBinding StringFormat="{}{0}" - Fix OnStart reference → use CreateWindow for theme restore - Soften lifecycle OnStart/OnSleep characterization (compat shims, not absent) - Qualify ResourceDictionary code-behind requirement (only for x:Class) Cross-references: - Remove all remaining body-text refs to non-existent skills - Replace with Microsoft Learn doc links or generic guidance Eval: - Relax SafeAreaRegions assertion to accept SafeAreaEdges or SafeAreaRegions - Relax theming rubric for ResourceDictionary code-behind Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore Clear() theme pattern with clarifying comment The Clear()+Add() pattern is intentional — theme dictionaries are designed to be the only merged dictionaries in this architecture. Added comment explaining the assumption and what to do if non-theme dictionaries exist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix second-pass review findings: API accuracy and eval robustness Code correctness: - ScrollView SafeAreaEdges="All" → "Container" (ScrollView only honors Container and None; All silently fails for keyboard avoidance) - DI CreateWindow example: add comment noting AppShell must be registered - AddHttpClient: add comment noting Microsoft.Extensions.Http NuGet required - MultiBinding: remove redundant StringFormat from reference doc (matches SKILL.md) - RelativeSource Self table: Path=Width → Path=WidthRequest (Width is read-only -1 before layout; WidthRequest is the correct bindable property) - CollectionView StackLayout warning: clarify it refers to BindableLayout on StackLayout vs CollectionView ItemsLayout types - SafeAreaEdges: fix "attached property" → "property" (it's a regular property) - ios:NavigationPage example: add xmlns comment for ios namespace Eval assertions: - DI Scoped regex: accept "doesn't"/"lacks"/"without" not just "no" - CollectionView ViewCell: match negation in either direction around ViewCell - Lifecycle events: require 3+ distinct events instead of any single one Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix final PR review comments - safe-area eval: tighten Container assertion to require SafeAreaEdges context - DI: merge #else fallback into single compilable #if block - lifecycle: behaviour → behavior (American English consistency) - theming: macOS Catalyst minimum 10.14 → 10.15 (Catalina required) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
948a9a52ef |
Add exp-mock-usage-analysis skill and evaluation (#483)
* Add exp-mock-usage-analysis skill and evaluation New experimental skill that detects unused, redundant, or unnecessary mocks in .NET test suites. Covers Moq, NSubstitute, FakeItEasy, and manual test doubles. - SKILL.md with 6-step workflow: gather code, classify mocks, detect anti-patterns, incorporate runtime data, generate recommendations, report findings - eval.yaml with 10 scenarios: trivial type mocking, unused setups, redundant configs, stable utility mocks, well-placed mocks (no false positives), NSubstitute support, FakeItEasy support, setup sprawl, and 2 non-activation boundary tests - Fixture files for each scenario with production + test code - CODEOWNERS entry for @dotnet/dotnet-testing * Remove no-headroom scenarios, increase NSubstitute timeout Remove 4 scenarios where baseline already scores 5.0/5 (model handles these well without skill guidance): - Detect mocking of DTOs, records, and enums - Recognize well-placed mocks without inventing false positives - Decline when asked about non-mock test anti-patterns - Decline mock framework migration request Increase NSubstitute scenario timeout from 120s to 180s to avoid baseline timeouts impacting scoring. Remaining 6 scenarios all show positive skill impact (3→5, 3→4, etc). * Document areas the model handles well without the skill Add a subsection under 'When Not to Use' listing mock-related tasks where the base model already scores 5.0/5 without skill guidance: - Mocking trivial types (DTOs, records, enums) - Recognizing well-placed mocks (no false positives) * Fix md * Improve |
||
|
Amaury Levé
|
d5ca4c6c01 |
Add exp-test-smell-detection skill and evaluation (#491)
* Add exp-test-smell-detection skill and evaluation - Language-agnostic core skill detecting 10 test smells (conditional logic, mystery guest, sleepy test, assertion-free, eager test, magic numbers, sensitive equality, exception handling, general fixture, ignored tests) - .NET extension with framework-specific detection tables (MSTest/xUnit/NUnit/TUnit) - Reference catalog with 19 smells from testsmells.org research - Evaluation with 4 scenarios: smelly suite, clean suite, integration tests, non-activation - CODEOWNERS entry for @dotnet/dotnet-testing * Improve integration test fixture and non-activation timeout - Add real smells to integration test fixture (Thread.Sleep, conditional logic, assertion-free test) mixed with legitimate integration patterns - Update rubric to test nuanced distinction between real smells and acceptable integration test patterns - Increase non-activation scenario timeout from 120s to 300s to prevent timeout-related scoring penalties * Fix test projects * Add domains for testsmells * Remove extra domains |
||
|
Dan Moseley
|
165e6439fe |
Tighten known-domains allowlist (#496)
* Tighten known-domains allowlist Narrow broad domain entries to specific paths/subdomains actually used: - aka.ms -> aka.ms/dotnet-trace, aka.ms/perfcollect - nuget.org -> api.nuget.org, www.nuget.org/account/trustedpublishing - npmjs.com/package/@modelcontextprotocol -> pinned to inspector/v/0.21.1 Remove unused domains: fluentassertions.com, speedscope.app Update skill content to match: pin MCP Inspector link, remove FluentAssertions link, remove speedscope hyperlink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * updated mcp skills codeowner * Refine scoped known-domain entries Remove the remaining nugettest.org publish example and drop the nugettest.org allowlist entry. Also normalize path-scoped entries to omit a hardcoded www prefix so both host forms match the existing ReferenceScanner logic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Leslie Richardson <lerich@microsoft.com> |
||
|
Amaury Levé
|
eb0f98eba6 |
Add exp-test-gap-analysis skill for pseudo-mutation test gap detection (#490)
New experimental skill that performs pseudo-mutation analysis on .NET production code to find gaps in existing test suites. Covers boundary, boolean/logic, return value, exception removal, arithmetic, and null-check mutation categories. Includes eval.yaml with 4 scenarios (boundary gaps, logic gaps, well-tested code recognition, non-activation) and fixture projects. |
||
|
Youssef Victor
|
ae9e2b4302 |
Add SKILL for xunit.v3 migration (#484)
* Add SKILL for xunit.v3 migration * eval.yaml, to be reviewed * Add run_command_and_assert to eval.yaml for dotnet test validation Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8e1539d5-327e-4216-9a01-5aa062e53608 Co-authored-by: Youssef1313 <31348972+Youssef1313@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix eval * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix eval.yml * Update eval * Update skill * Increase timeout * Update eval.yaml * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/migrate-xunit-to-xunit-v3/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/dotnet-test/migrate-xunit-to-xunit-v3/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
59d4485e14 |
Improve dotnet-template-engine skills and eval suite, add template-validation skill (#480)
* Improve dotnet-template-engine eval quality and plugin activation - Replace useless 'Create project with specific framework' test (vanilla=10/10 in all 30 runs, zero skill delta) with harder scenarios that vanilla cannot trivially solve - Expand eval suite from 5 to 10 scenarios to reduce overfitting signal and improve coverage of untested capabilities (dry-run, search, install, template creation from existing project, ambiguous intent resolution) - Add file_not_exists assertion to dry-run scenario to verify projects are not actually created during preview - Add output_matches assertions with specific patterns (e.g. 'individual', 'blazor') instead of relying solely on output_contains - Increase all timeouts from 120s to 180s to reduce MCP cold-start timeouts - Enrich plugin.json description with activation keywords (console app, class library, web API, Blazor, MAUI, framework versions) to fix Plugin mode activation failures (19/21 not-activated events were from one scenario) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add template-validation skill with eval scenarios New skill encoding template.json validation rules: required fields, identity format, shortName conflicts with CLI commands, symbol type/datatype validation, choice parameter default checking, computed/generated symbol completeness, parameter prefix collisions, source condition syntax, post-action requirements, constraint configuration, and tag recommendations. Two eval scenarios: - Validate a template with 8+ deliberate errors (missing identity, CLI- conflicting shortName, invalid defaults, incomplete symbols, missing actionId) - Validate a correct template and confirm it's ready to publish Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments - Add ./test-output to dry-run prompt so file_not_exists assertion checks the correct location - Add missing expect_tools: skill to template-authoring validate scenario - Add 'using Xunit;' to seeded test file so the setup project is compilable Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten assertions and replace low-delta scenario - Replace low-delta 'worker service search' (vanilla solves trivially) with 'create multi-project solution' (two projects + project reference — vanilla struggles with multi-step orchestration) - Fix gRPC search assertion: replace trivial output_contains 'template' with specific pattern matching gRPC service/package names - Fix validation restore assertion: require 'shortName' context near 'restore' to prevent false pass from post-action description - Fix correct-template assertion: replace fragile output_not_contains with positive assertion that agent mentioned specific fields it checked - Fix create-from-existing assertion: replace pass-anything pattern with assertion requiring template.json / sourceName / .template.config mention Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 2) - Change datatype 'integer' to 'int' in eval fixture and SKILL.md to align with actual template engine schema - Tighten correct-template assertion: require unambiguous success signal (e.g. 'no errors', 'template is valid') instead of permissive pattern that matches 'not ready' or 'pass' in other contexts - Replace output_not_contains 'missing required' (fails on 'no missing required fields') with positive assertion checking field names - Remove int/integer dual listing from valid datatypes in SKILL.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval failures from CI run 4 scenarios failed; fixes: - Web API + AOT: increase timeout 180s -> 300s (auth scaffolding + NuGet restore needs more time) - Dry-run preview: rephrase prompt to explicitly mention 'template' for better skill activation matching - Multi-project solution: replace with simpler 'create xUnit test project with net8.0' — the multi-project scenario gave vanilla a HIGHER score (4.7) than skilled (4.3) because the skill added overhead without helping - Create-from-existing: increase timeout 180s -> 300s (template generation burned 256K tokens and timed out) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dishonest eval adjustments - Revert dry-run prompt to natural language — don't coach activation by injecting 'template' keyword. If skill doesn't activate, that's a real signal about activation quality. - Replace recycled 'xUnit + net8.0' test (same ceiling problem as the classlib + net9.0 test we removed) with 'Blazor + interactive rendering' which tests parameter knowledge vanilla is less likely to have (render mode selection, blazor vs blazorserver vs blazorwasm template choice). - Keep timeout increases (300s for web API + auth, 300s for create-from- existing) as these address real infrastructure constraints. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move dry-run scenario from template-instantiation to template-discovery The dry-run scenario failed with 'NOT ACTIVATED' because the platform couldn't match a preview/dry-run prompt to the template-instantiation skill. This is correct behavior — previewing what files a template produces is a discovery operation, not instantiation. The template-discovery SKILL.md explicitly covers dry-run in Step 4: 'Use dotnet new <template> --dry-run to show what files a template would create without writing anything to disk.' This is an objective reclassification, not prompt tuning. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill content causing eval failures template-instantiation: Add parameter reference table with common parameter combinations (auth values, interactivity modes, AOT flag) for webapi, blazor, grpc, worker templates. The web API + AOT scenario scored 2.0 for both vanilla and skilled because the skill mentioned --auth and --aot but never explained what values they accept. Now the agent gets actionable knowledge. template-authoring: Trim verbose content to reduce token overhead. The create-from-existing scenario burned 256K tokens (vs 185K vanilla) and timed out because the agent read packaging/distribution steps it didn't need. Condensed Step 1 example, removed Step 5 (packaging), compressed Step 4 (testing). Reduces skill size by ~30%. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 3) - Web API+AOT: replace output_matches for AOT with file_contains checking for PublishAot in the generated .csproj — proves the parameter was actually passed, not just mentioned in text - Validation multi-error: add assertions for bool default (enableLogging/ yes), int default (maxRetries/abc), computed missing value, generated missing generator — now enforces 7 of 8 rubric items via assertions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 4) - Fix YAML double-quote escaping for regex word boundaries - Fix worker template example to include --aot flag - Fix webapi example to show --auth Individual --aot together - Fix grammar: Available Symbol Generators Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove web API+AOT and Blazor eval scenarios Web API+AOT consistently degraded quality (run 1: 2.0->2.0, run 2: 3.0->2.7). The skill adds reading overhead on execution tasks without improving the result. Blazor hit ceiling (5.0->5.0) — vanilla already creates Blazor apps perfectly, so the scenario contributes zero signal. template-instantiation now has 1 honest scenario (console app, +1.0 delta). The skill's value is modest for execution tasks; the PR's strength is in discovery (+2.3 to +3.7), validation (+1.7 to +2.0), and authoring (+2.0) where knowledge matters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments (round 5) - Tighten bool/int validation assertions to require explicit error wording (invalid, not valid, must be) alongside field/value — prevents passing when agent just echoes the JSON without diagnosing the issue - Add actionId assertion to enforce detection of missing postAction actionId (was in rubric but not in assertions) - Use single-quoted YAML for patterns with backslash sequences Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Mukund Raghav Sharma (Moko)
|
c92faa7800 |
Add minimal-api-file-upload skill (#264)
* Move minimal-api-file-upload skill to aspnetcore plugin with 3 eval scenarios Per repo restructuring feedback, ASP.NET Core specific skills should be under the aspnetcore plugin rather than the dotnet plugin. * Increase eval timeouts to 180s to avoid timeout-related scoring penalties * Generalized and addressed feedback * Improve eval scenarios: remove noisy tests, add dual-limit and security review scenarios * Add CODEOWNERS for aspnetcore plugin * Fix eval: enrich prompts for skill activation, bump security review timeout to 240s * Drop size-limit scenario (baseline already 5.0), keep 3 high-signal scenarios * Reframe security review as file upload fix to improve skill activation * Review * Contribution guide followed * Updated name * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-aspnet/skills/minimal-api-file-upload/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR #264 review feedback from Brennan and Copilot - Rename plugin from aspnetcore to dotnet-aspnet per repo conventions - Fix streaming filename: remove user-controlled extension, use bare GUID - Remove explicit sanitization code (Option B) per Brennan's feedback - Add case-insensitive MIME type comparison - Validate ContentType matches magic bytes (reject mismatches) - Fix stream rewind + CopyToAsync: copy from already-opened stream - Reword streaming section to accurately describe buffering vs chunks - Update marketplace.json, README.md, CODEOWNERS for dotnet-aspnet rename --------- Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
e4670b33a1 |
[PoC] Code testing agent + tests PoC (#433)
* Add code testing agent agents + tests PoC * Remove AssertionEvaluator.cs change (moved to dev/jankrivanek/agents-evals) |
||
|
Amaury Levé
|
51d29a086f | Improve test skills (#428) | ||
|
Amaury Levé
|
b527b41573 |
Add exp-test-boilerplate-detection and exp-assertion-quality skills (#468)
Two new experimental skills for dotnet-experimental plugin: - exp-test-boilerplate-detection: Detects duplicate boilerplate patterns across .NET test suites and identifies refactoring opportunities. Categories: repeated construction, assertion patterns, copy-paste methods, duplicated setup/teardown, repeated infrastructure. - exp-assertion-quality: Analyzes assertion variety and depth across .NET test suites. Produces metrics dashboard including assertion count, type spread, trivial %, assertion-free tests, and negative assertion coverage. Both skills include eval.yaml with 3-4 scenarios each (positive, negative, and non-activation) plus test fixtures. |
||
|
Jan Jones
|
873c2e4381 |
Mention more directives in the FBA skill (#307)
* Mention more directives in the FBA skill * Address feedback * Mention shebang Co-authored-by: Damian Edwards <damian@damianedwards.com> --------- Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Damian Edwards <damian@damianedwards.com> |
||
|
Julien Couvreur
|
c6be495147 |
Create README.md for dotnet plugin (#460)
I expect this will fix https://github.com/dotnet/skills/issues/458 |
||
|
Amaury Levé
|
a220f36256 |
Add exp-test-maintainability skill for test structure review (#418)
New experimental skill that assesses test maintainability: duplication, test size, data-driven patterns, display names, builder/helper extraction, and shared setup. |
||
|
leslierichardson95
|
7286c31118 |
Add skills for C# MCP Server Development (#317)
* Add C# MCP server skills (create, debug, test, publish) Four new skills for the C# MCP server development lifecycle: - mcp-csharp-create: Scaffolding with dotnet new mcpserver, tools/prompts/resources, transport config - mcp-csharp-debug: MCP Inspector, VS Code integration, breakpoint debugging, logging - mcp-csharp-test: Unit tests with ClientServerTestBase, integration with WebApplicationFactory, evals - mcp-csharp-publish: NuGet packaging, Docker/Azure deployment, MCP Registry publishing Each skill includes SKILL.md with progressive disclosure references/ and eval.yaml tests. * Fix mcp-csharp-create eval: use regex assertions for combined attribute syntax Replace scaffolding-heavy scenarios with implementation-focused ones that test MCP-specific features (resources, prompts, logging). Fix assertion patterns to match combined C# attribute syntax [McpServerTool, Description()] instead of requiring standalone [McpServerTool]. Increase timeouts to 180s to account for skill-reading overhead. Validator result: passed=True, improvement=44.6% (threshold=10%) * Add CODEOWNERS entries for MCP C# skills (create, debug, publish, test) * Update plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-publish/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Remove evaluation step and guide from mcp-csharp-test skill * Add CODEOWNERS entries for dotnet-maui skills * Update CODEOWNERS to add @artl93 as co-owner for mcp-csharp skills * Replace parent-directory file references with backtick skill names Cross-skill references used ../sibling/SKILL.md paths which the validator rejects. Replace with backtick-quoted skill names to match the convention used by other skills in the repo. * Add known domains for MCP server development skills Add domains referenced by the mcp-csharp-* skills: - github.com/modelcontextprotocol/registry - github.com/open-telemetry/semantic-conventions - npmjs.com, xunit.net, fluentassertions.com, nugettest.org Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Docker commands to use placeholder for registry and image names * moved mcp skills and their evals into dotnet-ai plugin * cleaned up moved files to dotnet-ai * cleaned up files * Update .github/CODEOWNERS Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update nuget-packaging.md * Update repository URL format in nuget-packaging.md * Add github.com/ to known-domains to allow truncated placeholder URLs The URL regex truncates at angle brackets (invalid in URIs), so placeholder URLs like https://github.com/<yourusername>/<repo> become https://github.com/ which didn't match any path-scoped entry. Adding github.com/ (with trailing slash) covers only this case without broadly allowing all github.com URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Docker tag command to use placeholder for server name * Update known domains and fix GitHub username placeholders in documentation * Remove McpServerToolType from mcp-csharp-create eval McpServerToolType is only needed with WithToolsFromAssembly(), which is discouraged because it is not Native AOT compatible. Updated eval assertions and rubrics to prefer WithTools<T>() instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve eval resilience for mcp-csharp-create and mcp-csharp-debug - Increase mcp-csharp-create timeouts from 180s (or default 120s) to 360s for all 3 scenarios. These scenarios consistently time out because the model spends time on bash exploration/scaffolding before writing code. - Add explicit timeout to scenario 3 which was relying on the 120s default. - Broaden mcp-csharp-debug Inspector rubric to accept both the concise single CLI command style and the step-by-step UI configuration walkthrough, since both are correct approaches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Broaden debug 'failing tool' rubric to accept multiple debugging approaches The rubric criterion 'Shows how to attach a debugger' was too narrow. The skilled answer correctly focused on the #1 cause (stdout pollution) but scored low because it didn't show a specific 'attach to process' flow. Broadened to accept any valid debugging approach: attaching, Debugger.Launch(), or launch.json configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 4 eval failures in mcp-csharp-debug and mcp-csharp-test skills mcp-csharp-debug: - Trim SKILL.md verbosity (183->160 lines) and mcp-inspector.md (67->54 lines) - Add Diagnosing Tool Errors section covering debugger, output panel, Inspector, common culprits - Rebalance debugging narrative away from stderr-only focus - Move HTTP logging config to ide-config.md reference mcp-csharp-test: - Add eval/evaluations keywords to frontmatter for activation - Add HTTP tool invocation test pattern (tools/call via WebApplicationFactory) - Trim test-patterns.md bloat (remove Test Categories, Coverage, Input Validation) - Create references/evaluations.md with qa_pair format and read-only/deterministic guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jeff Schwartz
|
55a7d80527 |
adding SIMD/TensorPrimitives to dotent-diag/analyzing-dotnet-performance (#330)
* incorporated feedback from previous pr, iterated with 100+ scalar loop tests to come to this skill, moved from skill to reference in the performance skill * Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedback on SIMD vectorization reference - Check Span<T>/MemoryExtensions before TensorPrimitives (no extra dependency) - Add all supported types (sbyte, ushort, uint, ulong, nint, nuint, char via ushort) - TensorPrimitives: add constraint and applicable types columns (not just float/double) - Add FusedMultiplyAdd, clarify AddMultiply vs MultiplyAdd distinction - Prefer portable APIs over platform-specific intrinsics (allow when perf justifies) - Fix dispatch pattern to use if/else if to avoid pessimizing small inputs - Remove LINQ references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * addressing more feedback * move to the experimental plugin * move the simd tests to exprimental * add codeowners --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
1d56634150 | Create experimental plugin (#446) | ||
|
Amaury Levé
|
17fe01321d | Promote crap-score and test-anti-patterns from experimental to dotnet-test (#434) | ||
|
Amaury Levé
|
e354d43534 |
Improve test skills (#423)
* Increase timeout * Fix activation issues * Improve further |
||
|
Viktor Hofer
|
a895c226b3 |
Fix agents field in plugin.json to use explicit file paths (#419)
* Fix agents field in plugin.json to use explicit file paths Claude Code's manifest validator rejects directory paths in the agents field. Replace directory references with explicit .agent.md file paths in all affected plugins (dotnet-diag, dotnet-msbuild, dotnet-template-engine). Fixes https://github.com/dotnet/skills/issues/416 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Validate that agents paths are explicit files, not directories Claude Code's manifest validator rejects directory paths in the agents field. Update PluginValidator to error when an agent path resolves to a directory, guiding authors to use explicit .agent.md file paths instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
0846e6015d | Improve MSTest migration skills (#387) | ||
|
Amaury Levé
|
9adebf86a8 | Add experimental CRAP score skill for dotnet-test (#413) | ||
|
Amaury Levé
|
00c1e82006 |
Add exp-test-tagging skill to dotnet-test-experimental (#417)
* Add exp-test-tagging skill and evaluation Experimental skill for analyzing and tagging test suites with a standardized trait taxonomy (positive, negative, boundary, critical-path, smoke, regression, integration, end-to-end, performance, security, concurrency, resilience, destructive, configuration, flaky). Supports MSTest, xUnit, and NUnit with framework-specific attribute syntax. Includes 5 eval scenarios: tag MSTest/xUnit/NUnit, audit-only, non-activation. * Move exp-test-tagging to dotnet-test-experimental plugin |
||
|
Amaury Levé
|
6489fbee39 |
Add exp-test-anti-patterns skill for test quality review (#415)
New experimental skill that detects anti-patterns and code smells in .NET test suites. Covers assertion quality, test isolation, naming, flakiness indicators, over-mocking, and structural problems across MSTest, xUnit, NUnit, and TUnit. Key features: - 4-tier severity classification (Critical/High/Medium/Low) - False-positive avoidance: calibrates severity honestly and recognizes well-written tests without inventing problems - Positive observations alongside findings Evaluation results (3 runs, claude-opus-4.6): - Flakiness/coupling detection: 3.0 -> 5.0 (isolated), 4.0 (plugin) - Duplicates/magic values: 3.0 -> 5.0 (isolated), 4.7 (plugin) - Well-written test recognition: 2.0 -> 5.0 (both modes) - Mixed severity detection: 5.0 baseline maintained - Overfitting score: 0.06 across all scenarios |
||
|
Amaury Levé
|
e0e1a91d90 |
Extract shared references and add cross-refs for dotnet-test skills (#414)
* Extract shared references and add cross-refs for dotnet-test skills - Extract platform detection logic into run-tests/references/platform-detection.md (shared by run-tests, mtp-hot-reload, migrate-vstest-to-mtp) - Extract filter syntax reference into run-tests/references/filter-syntax.md (shared by run-tests, mtp-hot-reload, migrate-vstest-to-mtp) - Add cross-references between skills in When Not to Use / Related Skills sections - run-tests SKILL.md reduced from 364 to 191 lines * Fixes |
||
|
Copilot
|
6face98ff6 |
Change agents and skills fields in plugin.json to array format for Claude Code CLI compatibility (#407)
* Initial plan * Remove agents string field from plugin.json and handle array format in parser Claude Code CLI validates `agents` as array-of-strings per its plugin manifest schema. Our plugin.json files had `"agents": "./agents/"` (a string), which caused "agents: Invalid input" validation errors on install. Remove the field from the 3 affected plugin.json files — agents are still discovered by convention from the `agents/` directory. Also make ParsePluginJson resilient to both string and array formats for forward-compatibility. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Remove `agents` string field from plugin.json for Claude Code CLI compatibility Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Change agents field to array format in plugin.json, update validator to read array-first with string fallback Plugin.json files now declare agents as an array of file paths: "agents": ["./agents/foo.agent.md", "./agents/bar.agent.md"] ParsePluginJson reads the array first (preferred, Claude Code schema), falls back to a string path (legacy). PluginInfo gains an AgentPaths field. Validator and discovery use the array when present, else fall back to directory-based convention. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Simplify agents to directory-path array form ["./agents/"], handle directories in array entries Per Copilot CLI spec, agents field values are "Path(s) to agent directories" — no need to list individual files. Use ["./agents/"] to mirror how skills uses "./skills/". Updated validator and discovery to handle both directory and file entries in the array. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Delete .nuget/nuget.exe * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Convert skills field to array format in plugin.json, update validator to handle both forms Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Align AgentPaths null checks to use { Count: > 0 } pattern for consistency Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Change `agents` and `skills` fields in plugin.json to array format for Claude Code CLI compatibility Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Delete .nuget/nuget.exe * Consolidate SkillsPath/SkillPaths and AgentsPath/AgentPaths into single array fields Remove dual string/array fields from PluginInfo. The parser now normalizes legacy string values into single-element arrays, so all downstream code uses only SkillPaths and AgentPaths (IReadOnlyList<string>). This eliminates all array-vs-string fallback branching. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
dcba4f3d8e |
Add dotnet-template-engine plugin (#341)
* Add dotnet-template-engine plugin Add template-engine skills for .NET project scaffolding powered by the DotnetTemplateMCP MCP server (v1.1.0). Includes three skills and one agent: Skills: - template-discovery: find, inspect, and compare templates with intent resolution (70+ keyword mappings) - template-instantiation: create projects with CPM adaptation, latest NuGet versions, parameter validation, and multi-template composition - template-authoring: create templates from existing projects and validate template.json for authoring issues Agent: - template-engine: expert agent that routes to the appropriate skill based on user intent Also adds CODEOWNERS entries, marketplace.json registration, eval tests, and .mcp.json for DotnetTemplateMCP server configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: mcpServers in plugin.json, expect_tools, path fix - Move MCP server config into plugin.json under mcpServers so SkillValidator discovers the template_* tools during evaluation runs - Add expect_tools constraints to all eval scenarios to verify actual MCP tool invocation instead of relying on rubric text alone - Fix frontmatter path: template.config → .template.config (leading dot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update DotnetTemplateMCP to v1.2.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Include 'dotnet new' in agent description for discoverability Address review feedback from danmoseley: the frontmatter description is how the caller decides whether to load the agent tokens, so it needs to mention 'dotnet new' as a trigger keyword. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove MCP server configuration and references - Delete .mcp.json MCP server config file - Remove mcpServers section from plugin.json - Remove MCP/DotnetTemplateMCP references from agent, skills, and marketplace descriptions - Skills and eval files remain unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove ghost tool references and add CLI fallback guidance - Remove references to 4 non-existent tools: template_compose, template_suggest_parameters, template_validate, solution_analyze - Replace with real alternatives (template_inspect, manual review, sequential template_instantiate, direct file inspection) - Add CLI fallback guidance so LLMs degrade gracefully when template tools are unavailable - Fix authoring eval to not expect template_validate tool Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: CODEOWNERS naming, claude marketplace, README - Rename CODEOWNERS header to dotnet-template-engine for consistency - Mirror plugin entry in .claude-plugin/marketplace.json - Add plugin to README What's Included table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PR review comments: align eval rubric with expect_tools and sync marketplace.json - Updated template-discovery eval.yaml rubric to match expect_tools (removed 'or template_search' since expect_tools enforces template_from_intent) - Made .claude-plugin/marketplace.json byte-for-byte identical to .github/plugin/marketplace.json (removed extra trailing newline) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments: strengthen eval assertions and fix rubrics - Add exit_success assertions to all eval scenarios - Add file_exists and file_contains assertions for template-instantiation - Add output_contains for shortName and datatype in template-authoring - Fix rubrics to reference actual tools (skill + dotnet new CLI) instead of non-existent MCP tools (template_from_intent, template_inspect, template_instantiate) which caused the judge to penalize quality scores - Update expect_tools from non-existent MCP tools to 'skill' - Change plugin version from 1.2.0 to 0.1.0 for consistency with other plugins - Remove extra blank line in README.md after plugin table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rewrite skills and agent to use dotnet new CLI instead of phantom MCP tools The SKILL.md files and agent.md referenced MCP tools (template_from_intent, template_inspect, template_instantiate, template_dry_run, etc.) that are not available at runtime since no MCP server is declared in plugin.json. This caused the agent to waste tokens attempting to find these tools before falling back to dotnet new CLI commands. Changes: - template-discovery/SKILL.md: Replace MCP tool calls with dotnet new search, dotnet new list, dotnet new --help, and dotnet new --dry-run. Add intent-to-template keyword mapping table. - template-instantiation/SKILL.md: Replace template_instantiate with dotnet new CLI commands. Add explicit CPM adaptation steps and multi-project composition with dotnet sln/dotnet add reference. - template-authoring/SKILL.md: Replace template_create_from_existing with manual template.json creation guidance including example JSON. Replace template_dry_run/template_instantiate with CLI equivalents. - template-engine.agent.md: Replace Available Tools table with CLI Commands Reference. Update all workflow steps to use CLI commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix file assertion paths: remove ./ prefix for glob matcher compatibility Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |
||
|
Jeff Handley
|
646b5d93b0 |
Add Skill: Convert to Central Package Management (convert-to-cpm) (#75)
* Add convert-to-cpm skill for NuGet Central Package Management conversion Authoring process: - Researched the Agent Skills standard at agentskills.io and reviewed repo conventions - Studied the existing csharp-scripts skill and test structure as a reference implementation - Studied Microsoft's CPM documentation at learn.microsoft.com - Authored SKILL.md with a 9-step workflow covering scope determination through validation - Created 5 test scenarios across simple, moderate, and advanced tiers - Performed dry-run evaluation against all scenarios, identifying 15 findings - Applied all 15 fixes (4 gaps, 6 ambiguities, 5 improvements) to SKILL.md - Re-evaluated: 50 OK, 0 gaps, 0 ambiguities, 1 test fixture improvement - Updated test fixture to resolve the remaining finding - Final validation: 254 lines, 0 errors, 0 warnings Skill implementation (skills/convert-to-cpm/SKILL.md): - 9-step workflow: scope, CPM check, audit, create/update props, update projects, handle properties, restore/validate, cleanup, summary - Handles single projects, solutions, and full repositories - Audits packages with per-project detail table including version conflicts and security advisories - Flags CVE/security vulnerabilities and recommends patched versions - Supports MSBuild property-based versioning with inline vs keep decisions - Covers conditional PackageReference items across target frameworks - Discovers PackageReference in imported .props/.targets via Import chain scanning - Provides cross-platform commands (Unix and Windows/PowerShell) - Documents import order requirements for property references in Directory.Packages.props - Mentions dotnet new packagesprops as preferred creation method (.NET 8+) - Includes validation checklist and common pitfalls table Test scenarios (tests/convert-to-cpm/): - simple-single-project: 1 project, 3 packages, no conflicts - simple-solution: 3 projects via .sln, 5 packages, PrivateAssets preservation - moderate-version-conflicts: 4 projects, System.Text.Json version conflicts with CVE-2024-43485 advisory, AutoMapper major version conflict with VersionOverride - moderate-msbuild-properties: 2 projects using MSBuild properties for versioning in Directory.Build.props, inline vs keep decisions, property cleanup verification - advanced-multi-complexity: 5 projects combining version conflicts, MSBuild properties, conditional TFM PackageReference, shared Common.props imports, VersionOverride, and security advisory handling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor SKILL.md with references, baseline build, and package comparison Condensed SKILL.md using progressive disclosure: - Reduced from 254 lines / ~3937 tokens to 120 lines / ~1778 tokens - Extracted detailed procedural instructions into 5 focused reference files - Main file retains workflow decision logic with links to references loaded on demand - Follows agentskills.io progressive disclosure pattern: metadata -> instructions -> resources Added baseline build verification (new Step 2): - Requires clean build (dotnet clean && dotnet build) before any changes - Captures baseline.binlog for structured build artifact - Captures baseline-packages.json via dotnet list package --format json - Blocks conversion if baseline build fails Added post-conversion baseline comparison (Steps 8 and 10): - Step 8 produces after-cpm.binlog and after-cpm-packages.json from clean build - Step 10 compares baseline vs post-conversion package lists per project - Changes table shows version bumps, VersionOverride entries, added/removed packages - Unchanged table confirms version-neutral conversion for unmodified packages - Binlog files produced as artifacts for user review in MSBuild Structured Log Viewer Reference files created: - references/audit-complexities.md: 7 complexity categories for package auditing - references/baseline-comparison.md: dotnet list package comparison procedure, binlog production, artifact guidance, table formats for changes and unchanged packages - references/directory-packages-props.md: File creation (dotnet new packagesprops), placement rules, conditional versions, VersionOverride patterns - references/msbuild-property-handling.md: Property usage search, inline vs keep decisions, import order requirements, cleanup verification - references/validation-and-errors.md: NuGet error codes (NU1008, NU1010, NU1507), clean build validation, multi-TFM guidance, common pitfalls Workflow is now 10 steps (was 9): 1. Determine scope 2. Establish baseline build (new) 3. Check for existing CPM 4. Audit package references 5. Create or update Directory.Packages.props 6. Update project files 7. Handle MSBuild version properties 8. Restore and validate (now with binlog + package list capture) 9. Clean up obsolete properties 10. Summary and baseline comparison (expanded with diff tables) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace third-party packages with Microsoft equivalents, add baseline/comparison steps, add .gitignore for test solutions Package replacements across all skill references and test scenarios: - Serilog.AspNetCore -> OpenTelemetry.Extensions.Hosting 1.15.0 - Serilog (standalone) -> OpenTelemetry.Extensions.Hosting 1.15.0 - Swashbuckle.AspNetCore -> Microsoft.AspNetCore.OpenApi 8.0.24 - SerilogVersion MSBuild property -> OTelVersion - DIVersion updated from 8.0.1 to 9.0.0 (DI/Abstractions alignment) - Microsoft.AspNetCore.OpenApi uses 8.0.24 (not 9.x) for net8.0 compat Test fixture improvements: - Added Step 2 (baseline build with binlog) to all 5 with-skill.md files - Added Step 10 (package comparison table + binlog artifact links) to all 5 with-skill.md files - Expanded simple-solution without-skill.md with more realistic agent behavior - Standardized security fix terminology: table markers cite CVEs, prose references CVE IDs - Removed space-padded table formatting in favor of compact markdown with emojis Test solution infrastructure: - Added tests/convert-to-cpm/.gitignore to exclude src/ build artifacts - Added generation prompt to test README for recreating buildable solutions - All 5 test solutions verified to restore and build successfully Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback on terminology, placement, and reference accuracy - Rename Step 10 and validation checklist from 'binlog comparison' to 'package list comparison' - Rewrite baseline-comparison.md: title to 'Baseline Comparison', lead with dotnet list package as primary comparison mechanism, position binlogs as supplementary artifacts for manual review/troubleshooting - Add brief definition of binlogs before the artifacts section - Fix Directory.Packages.props placement guidance: use first common ancestor of all .NET projects instead of assuming repository root (.git), since many repos nest source under src/ or similar - Update SKILL.md scope input and Step 1 to reference directory-scoped conversion - Standardize security fix terminology: table markers cite CVEs, prose uses 'security fix for CVE-...' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review findings: sort PackageVersion, add packages.config scenario, fix commands Review findings addressed: 1. Fixed BlobsVersion grep step description in advanced scenario 2. Sorted PackageVersion entries alphabetically in all Directory.Packages.props outputs 3. Consolidated baseline-comparison.md to single-table format 4. Added solution file arguments and multi-line commands in test fixtures 5. Added Directory.Packages.props placement question for single-project scenario 6. Created simple-packages-config test scenario (packages.config rejection) 7. Use dotnet new packagesprops in simple-single-project workflow 8. Fixed More Info link text to match reference file title Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test scenario accuracy: sln/slnx mix, package counts, ImplicitUsings - Use mix of .sln and .slnx across scenarios (simple-solution and moderate-msbuild-properties use .sln; moderate-version-conflicts and advanced-multi-complexity use .slnx) - Fix simple-solution package count from 5 to 6 - Fix advanced user response #2 to reference Azure.Storage.Blobs version - Add ImplicitUsings to advanced Directory.Build.props in audit and cleanup - Complete the advanced unchanged table with all expected package entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add skill evaluation section to test README Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Inventory.sln references and make evaluation opt-in - Fix 4 occurrences of Inventory.sln to Inventory.slnx in moderate-version-conflicts/with-skill.md for consistency with the scenario README and test solution - Update test README to only run skill evaluation when the user explicitly requests it Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Condense SKILL.md with progressive disclosure - Remove duplicated bash code blocks from Steps 2 and 8, linking to baseline-comparison.md instead - Merge Steps 7 (MSBuild property decisions) and 9 (cleanup) into a single step since both reference msbuild-property-handling.md - Condense Step 10 summary to reference baseline-comparison.md for procedure details - Reduces from 120 to 97 lines (10 steps to 9 steps) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Enterprise.sln references to Enterprise.slnx in with-skill.md Address code review feedback: fix 4 occurrences of Enterprise.sln to Enterprise.slnx in advanced-multi-complexity/with-skill.md for consistency with the scenario README and test solution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve convert-to-cpm skill: preserve whitespace, save summary report, fix links - Instruct Step 6 to retain each file's existing indentation style and blank lines to keep diffs minimal - Instruct Step 9 to save the full summary as convert-to-cpm.md alongside binlog and JSON artifacts, suggesting its contents for a PR description - Strip en-us/ from learn.microsoft.com links in SKILL.md and validation-and-errors.md to make them culture-agnostic - Update all with-skill.md test expectations to include the convert-to-cpm.md create step and artifact mention Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Indicate that the binlog viewer is for the user; not the tool/skill * Move convert-to-cpm skill and tests to src/dotnet-msbuild layout Move skill from skills/convert-to-cpm/ to src/dotnet-msbuild/skills/convert-to-cpm/ and tests from tests/convert-to-cpm/ to src/dotnet-msbuild/tests/convert-to-cpm/ to match the repository layout established on main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Convert tests from with/without-skill format to eval.yaml Replace manual test approach (README.md + with-skill.md + without-skill.md) with automated eval.yaml scenarios containing assertions and rubric items. Add committed test fixture files (.csproj, .sln, .slnx, .props) for each scenario with copy_test_files setup, replacing the git-ignored generated solutions. 6 scenarios converted: - simple-single-project: basic single project CPM conversion - simple-solution: multi-project solution conversion - simple-packages-config: negative test (packages.config rejection) - moderate-version-conflicts: version conflict resolution - moderate-msbuild-properties: MSBuild property version handling - advanced-multi-complexity: combined complexities (conflicts, properties, conditionals, shared props) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Consolidate eval.yaml files into single root eval.yaml Move all 6 scenario definitions from individual subdirectory eval.yaml files into a single src/dotnet-msbuild/tests/convert-to-cpm/eval.yaml file, matching the skill-validator discovery convention that expects {testsDir}/{skillname}/eval.yaml. Scenario prompts updated to reference subdirectory-prefixed paths since copy_test_files now copies all fixture subdirectories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reorganize convert-to-cpm into plugins/tests folders * Add post-conversion reporting rubrics to CPM eval scenarios Add rubric items for baseline-vs-result comparison reporting, binlog capture and guidance, version reconciliation, and risk/reassurance messaging across all conversion scenarios. Increase complex scenario timeout to 360s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve post-conversion reporting, conflict resolution, and skill activation SKILL.md: - Surface binlog capture commands explicitly in Steps 2 and 8 - Expand Step 4 with detailed version conflict presentation and resolution workflow - Restructure Step 9 into 5-section post-conversion report template: overview, conflict resolutions, baseline comparison, risk assessment, and artifacts/next steps with binlog education - Broaden activation triggers in description and When to Use to match update/bump/upgrade/align/sync terminology from ecosystem PRs - Add activation for out-of-sync/conflicting/mismatched version scenarios audit-complexities.md: - Expand version conflict section with per-conflict presentation format, 4 resolution options with trade-offs, and risk-level guidance eval.yaml: - Pre-specify conflict resolution strategy in prompts to avoid blocking on unanswered user prompts during single-turn evaluation - Add 3 new scenarios testing CPM recommendation when user asks to update/align packages without mentioning CPM - Update rubrics to match directed resolution strategy - Increase complex scenario timeout to 360s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback from kartheekp-ms - Use GitHub markdown URL for CPM docs reference (SKILL.md) - Use dotnet package list (renamed in .NET 10) with backward compat note for dotnet list package (SKILL.md, baseline-comparison.md) - Use dotnet package list --format json for auditing (SKILL.md Step 4) - Document Directory.Packages.props sub-folder support for test projects with different dependencies (directory-packages-props.md) - Remove security advisory version bumping from CPM conversion scope; instead surface advisories as follow-up items in the post-conversion report (audit-complexities.md, SKILL.md Step 9 Section 5) - Add Follow-up items section to report template covering security advisories, deprecated packages, version alignment opportunities, and test validation (SKILL.md Step 9) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve convert-to-cpm skill activation, artifact preservation, and eval assertions - Broaden skill description to trigger on version drift/sync scenarios and MSBuild property contexts even when CPM is not explicitly mentioned - Add explicit instructions to preserve binlog and JSON artifacts (not delete) - Make convert-to-cpm.md report a mandatory deliverable, not optional - Replace fragile output_contains assertions (ManagePackageVersionsCentrally, Directory.Packages.props) with file_contains checks on the actual generated files across 5 conversion scenarios Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extend timeouts and add CODEOWNERS entry * Move convert-to-cpm skill into a new dotnet-nuget plugin * Fix erroneous plugin msbuild/nuget plugin split change. Add dotnet-nuget to marketplace. * Add NuGet package management skills to README * Address PR feedback: clarify CPM sub-folder semantics, remove security-fix example, require explicit conflict strategy - directory-packages-props.md: Clarify that sub-folder Directory.Packages.props files are independent and replace (not extend) the parent; require explicit MSBuild Import to chain files - baseline-comparison.md: Replace security-fix upgrade example with version conflict alignment example, consistent with audit guidance that security advisories are follow-up items - SKILL.md: Remove default 'use highest version' conflict strategy; require explicit user choice or confirmation when conflicts are detected Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extend eval timeouts for scenarios hitting time limits - Convert single project: 180s -> 270s - Recommend CPM (version conflicts): 180s -> 270s - Recommend CPM (complex repo): 360s -> 540s - Convert complex repository: 360s -> 540s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add missing CODEOWNERS for dotnet-nuget plugin. Reorder convert-to-cpm evals and expand timeouts. * Remove containing skill mentions from reference files; fix ordering * Replace output_matches with deterministic file assertions in eval.yaml Convert 6 non-deterministic output_matches assertions to 8 file-based assertions (file_contains, file_not_contains) across conversion scenarios: - Scenario 5: Verify PrivateAssets preserved in Tests.csproj - Scenario 6: Verify OTelVersion/DIVersion removed from Directory.Build.props - Scenario 7: Verify System.Text.Json and Azure.Identity in Directory.Packages.props - Scenario 8: Verify System.Text.Json in Directory.Packages.props and BlobsVersion/HostingVersion removed from Directory.Build.props Recommendation-only scenarios and conversational assertions (conflict detection, conditionals, Common.props awareness) remain as output_matches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reorganize skill description with USE FOR / DO NOT USE FOR markers Restructure the description to improve activation reliability by adopting the USE FOR / DO NOT USE FOR pattern used by other skills in the repo (thread-abort-migration, binlog-failure-analysis). Key changes: - Front-load activation cues before capability list - Add explicit USE FOR / DO NOT USE FOR section markers - Add .sln/.slnx file extensions for solution-path matching - Move DO NOT USE exclusions into description (was only in body) - Use YAML folded scalar for multi-line readability All existing activation keywords preserved. Length reduced from 882 to 849 chars (max 1024). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix issues for external link references * Increase timeouts * Handle packages.config non-activation in convert-to-cpm skill Set expect_activation: false on the packages.config eval scenario so non-activation is reported as expected rather than flagged as a warning. Add a packages.config guard in Step 1 of the skill workflow so that if the skill is activated for a packages.config project, it stops early and recommends migrating to PackageReference first. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove leading blank lines from .sln test files Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use imperative 'Read' instead of 'See' for reference file links Ensures the agent explicitly loads and reads reference files rather than treating them as optional 'see also' links. This aligns with the repo convention of explicitly instructing agents to load reference content. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify dotnet-nuget CODEOWNERS * Simplify packages.config guard * Add minimal Program.cs entrypoints to Web/Worker test fixtures Test fixture projects using Microsoft.NET.Sdk.Web and Microsoft.NET.Sdk.Worker require a Program.cs entrypoint for dotnet build to succeed. The skill workflow instructs the agent to run dotnet restore and dotnet build for validation, so these fixtures must be buildable. Adds minimal entrypoints (with explicit using directives since ImplicitUsings is not enabled in these projects) to 6 projects: - simple-solution/Web - moderate-version-conflicts/Api, Worker - moderate-msbuild-properties/Api - advanced-multi-complexity/Api, Web Also adds Microsoft.Extensions.Hosting package reference to the Worker project, matching what the dotnet new worker template produces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
Amaury Levé
|
ae0b6a1d86 | Add mtp-hot-reload skill for iterative test fixing with MTP hot reload (#389) | ||
|
YuliiaKovalova
|
14346ff3b2 |
Enrich build-perf-diagnostics skill with MSBuild team and community insights (#342)
* Enrich build-perf-diagnostics skill with MSBuild team and community insights Additions (pure knowledge, no MCP dependency): - RAR: dotnet/msbuild#2015 context, DisableTransitiveProjectReferences, ReferenceOutputAssembly - Copy tasks: SkipCopyUnchangedFiles, --artifacts-path, Copy batching (dotnet/msbuild#12884) - Dev Drive (ReFS): copy-on-write + reduced Defender scans, OrchardCore 7257 Copy tasks benchmark - Evaluation: NuGetSdkResolver 180-400ms overhead (dotnet/msbuild#4025), EnableDefaultItems - NuGet: RestoreUseStaticGraphEvaluation (20s+ savings in large builds) - Project graph shape: wide vs deep (40% faster clean, 20% faster incremental) - MSBuild Server: MSBUILDUSESERVER=1 for CLI caching - Inline task overhead: 1s vs 3ms (RoslynCodeTaskFactory vs compiled) - Misleading ResolveProjectReferences time (dotnet/msbuild#3135) - Incrementality anti-patterns (dotnet/msbuild#13206) - Impact categorization (HIGH/MEDIUM/QUICK WINS) - Quick Wins: expanded to 15 items (from 7) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix minor issues from review comments - Fix missing closing XML tag for CreateHardLinksForCopyFilesToOutputDirectoryIfPossible in Quick Wins checklist - Clarify confusing Copy task batching wording per reviewer feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Trim skill to reduce token overhead Remove standalone sections (MSBuild Server, Inline Tasks, ResolveProjectReferences, Incrementality Anti-patterns) that add ~1400 extra tokens without benefiting the evaluated scenario. Consolidate Quick Wins checklist accordingly. These sections can be contributed as separate skills in follow-up PRs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add separate skills for MSBuild server, inline tasks, and ResolveProjectReferences Extract content removed from build-perf-diagnostics into standalone skills to keep each skill focused and avoid token budget bloat: - msbuild-server: MSBUILDUSESERVER=1 for CLI incremental caching - inline-task-overhead: RoslynCodeTaskFactory ~1s overhead vs ~3ms compiled - resolve-project-references: misleading Target Performance Summary time (dotnet/msbuild#3135), guides users to Task Performance Summary instead Incrementality anti-patterns content was not extracted as a separate skill because the existing incremental-build skill already covers that area. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add github.com/dotnet/msbuild to known-domains.txt Required by reference scanner for MSBuild issue links in build-perf-diagnostics and resolve-project-references skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace full dotnet/msbuild URLs with shorthand notation The reference scanner reads known-domains.txt from upstream main, not from the PR branch. Use shorthand 'dotnet/msbuild#NNNN' format (consistent with the rest of the file) instead of full URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments and fix reference scanner errors - Remove https://aka.ms/devdrive URLs that trigger reference scanner - Soften unsourced benchmark claims (OrchardCore Copy tasks, 20s+ restore savings, 40%/20% graph shape improvements) per reviewer feedback to use workload-dependent language instead Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add eval tests for new skills; remove inline-task-overhead Add eval.yaml and test fixtures for msbuild-server and resolve-project-references skills: - msbuild-server: tests CLI build perf scenario with developer report - resolve-project-references: tests misleading target perf summary with multi-project setup and perf report Remove inline-task-overhead skill: failed 3 evaluation iterations (avg +2.7%) because the baseline LLM already knows about RoslynCodeTaskFactory overhead — the skill doesn't add enough unique value to meet the 10% improvement threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address review comments - Add caveat that DisableTransitiveProjectReferences may require adding direct references for consumed types - Fix escaped quotes in Quick Wins RunAnalyzers checklist item - Add OutputType=Exe and Main entry point to msbuild-server fixture - Change 'solution' to 'project' in resolve-project-references report - Trim trailing spaces in developer-report.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore accidentally removed known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix remaining review comments - Quote -flp argument in resolve-project-references bash example to handle semicolons in POSIX shells - Format LibA/Class1.cs and LibB/Class1.cs across multiple lines for consistency with other test fixtures - Rename msbuild-server/Program.cs to Calculator.cs to match content Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
7e7dabd343 |
[WIP] Improve msbuild skills descriptions (#384)
* Improve msbuild skills descriptions * Improve descriptions |
||
|
Amaury Levé
|
c55e4f222b |
Add MSTest migration skills (v1/v2→v3, v3→v4) (#377)
* Add MSTest migration skills (v1/v2→v3, v3→v4) New dotnet-test plugin with two step-by-step migration skills: - migrate-mstest-v1v2-to-v3: covers assembly ref (v1) and NuGet (v2) projects, assertion overload changes, DataRow constructors, .testsettings replacement, timeout unification, and MSTest.Sdk adoption. - migrate-mstest-v3-to-v4: covers Execute→ExecuteAsync, CallerInfo constructors, ClassCleanupBehavior removal, TestContext.Properties generic change, Assert API changes, ExpectedExceptionAttribute removal, dropped <net8.0 TFMs, and behavioral changes (DisableAppDomain, TestCase.Id, TreatDiscoveryWarningsAsErrors). Includes eval.yaml test scenarios and fixtures for both skills, plugin.json, and CODEOWNERS entries. * Fixes * Improve results * Improve * Improve description * Copy fix for path * Bump timeout |
||
|
Amaury Levé
|
92d830ff48 |
Add writing-mstest-tests skill with eval scenarios (#378)
* Add writing-mstest-tests skill with eval scenarios Add a new skill for writing modern MSTest 3.x/4.x unit tests covering: - Test class structure (sealed, AAA pattern, naming conventions) - Modern assertion APIs (Assert.Throws, IsInstanceOfType, collection asserts) - Data-driven tests (DataRow, DynamicData with ValueTuples) - Async tests with cancellation and timeout patterns - Test lifecycle (constructor init, TestContext injection) - Anti-pattern detection and modernization Includes 10 eval scenarios across 4 goal categories with fixture files for both modern MSTest and legacy anti-pattern test projects. MSTest.Sdk version is recommended via global.json msbuild-sdks instead of inline in the project file, ensuring all test projects get bumped together. * Fixes * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Evangelink * Apply suggestion from @Evangelink --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
2a8388dd71 |
Add VSTest to MTP migration skill (#373)
* Add VSTest to MTP migration skill New migrate-vstest-to-mtp skill in the dotnet-test plugin covering: - Framework-specific MTP runner enablement (MSTest, NUnit, xUnit.net) - Directory.Build.props centralization of OutputType, runner properties - MSTest.Sdk UseVSTest removal - NUnit3TestAdapter 5.0.0+ upgrade requirement - xunit v2 to xunit.v3 upgrade requirement - dotnet test CLI argument translation (--logger to --report-trx, --collect to --coverage, --blame-crash to --crashdump, --blame-hang to --hangdump) - xUnit.net --filter syntax migration to --filter-class/--filter-trait/ --filter-query - global.json native MTP mode (.NET 10+) vs TestingPlatformDotnetTestSupport VSTest-bridge mode (.NET 9 and earlier) - Azure DevOps VSTest@3 to DotNetCoreCLI@2 pipeline migration - MTP extension NuGet packages (TrxReport, CrashDump, HangDump, CodeCoverage) - Exit code 8 behavioral difference (--ignore-exit-code 8, TESTINGPLATFORM_EXITCODE_IGNORE) Includes 10 eval scenarios with tiered timeouts (240-360s), 5 fixture directories, plugin.json, marketplace.json entries, CODEOWNERS, and README.md updates. * Fix codeowners * Improve * Improve further * Improve prompt * Improve further the description * Address review comments |