mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
dev/jankrivanek/eval-dispatch-plugin-input
108 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Amaury Levé
|
680292e04c |
Make dotnet-test analysis skills and auditor agent polyglot (#707)
* Make dotnet-test analysis skills and auditor agent polyglot Five test analysis skills and the test-quality-auditor agent in plugins/dotnet-test previously activated only on .NET MSTest/xUnit/NUnit/ TUnit cues, even though the underlying techniques (assertion diversity, anti-pattern detection, pseudo-mutation gap analysis, academic smell catalog, trait tagging) apply equally to other test frameworks. This change makes them polyglot across .NET, Python (pytest/unittest), TypeScript/JavaScript (Jest/Vitest/Mocha/node:test), Java (JUnit/TestNG), Go, Ruby (RSpec/Minitest), Rust, Swift (XCTest/Swift Testing), Kotlin (JUnit/Kotest), PowerShell (Pester), and C++ (GoogleTest/Catch2/doctest). Changes ------- * New reference skill `test-analysis-extensions` with per-language extension files (dotnet, python, typescript, java, go, ruby, rust, swift, kotlin, powershell, cpp). Each documents test markers, assertion APIs, sleep / time / random APIs, skip annotations, setup/teardown, mystery-guest indicators, integration markers, and tag-support capability for the corresponding framework. * `assertion-quality`, `test-anti-patterns`, `test-gap-analysis`, `test-smell-detection`, `test-tagging` SKILL.md updated: frontmatter now lists all polyglot frameworks (keeping .NET trigger words for backward compat); bodies reference `test-analysis-extensions` and add per-language calibration (e.g., Go/Rust table-driven loops are idiomatic and not Conditional Test Logic; pytest bare `assert` is canonical; missing-await on async assertions is a new Critical smell). * `test-quality-auditor` agent now runs language detection, gates .NET-only pipeline steps (coverage-analysis, CRAP, detect-static- dependencies, testability migration, experimental dotnet skills), and recommends native tooling for non-.NET projects via an explicit Capability Matrix. * README updated to split sections by polyglot vs .NET-only. * New polyglot eval scenarios added (Python/pytest, TS/Jest, Java/JUnit) alongside existing C# scenarios in `test-anti-patterns`, `assertion-quality`, `test-smell-detection`. The .NET-only skills (writing-mstest-tests, migrate-*, coverage-analysis, crap-score, detect-static-dependencies, run-tests, filter-syntax, platform-detection, dotnet-test-frameworks, mtp-hot-reload, exp-*) are unchanged and continue to activate only for .NET work. Validated with skill-validator `check --plugin ./plugins/dotnet-test`. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #707 review feedback * typescript.md: replace xpect.fail() recommendation (Jest has no such API) with hrow new Error(...) / explicit failing assertion. * test-tagging SKILL.md description: clarify that unittest has no canonical tag syntax (pytest markers are the auto-edit mechanism; unittest is report-only). Switch doctest reference to "decorators" rather than implying a Catch2-style [tags] mechanism exists. * test-tagging SKILL.md auto-edit list and table: use `* doctest::test_suite("tag")` decorator chain syntax for doctest in both the bullet list and the per-framework table, matching the example already shown. * cpp.md extension: spell out the Catch2 vs doctest tagging distinction in the capability matrix (Catch2 [tag], doctest * doctest::test_suite("tag") decorator). Validated with `dotnet run --project eng/skill-validator/src -- check --plugin ./plugins/dotnet-test` (23 skills + 11 agents). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Tom Longhurst
|
36209c3b52 |
TUnit Fixes & Improvements (#677)
* Update target framework to net10.0 and upgrade TUnit package to version 1.45.8
* Fix and expand TUnit coverage in dotnet-test-frameworks reference
Cross-referenced against tunit.dev/llms.txt and the official attributes
comparison page. Corrects the test class marker (TUnit is convention-based,
not [ClassDataSource]), and fills in the previously-missing TUnit rows for
the assertion APIs, setup/teardown methods, exception-handling examples,
and integration test category marker.
* Refine TUnit assertion table accuracy
The Type row's TUnit cell now uses IsAssignableTo<T>() to match the
assignable-type semantics of the MSTest/xUnit/NUnit equivalents, with a
note pointing at IsTypeOf<T>() for exact-type checks. The Skip row now
documents that [Skip] also applies at class and assembly scope and that
Skip.Test("reason") handles dynamic in-test skipping.
Verified against the TUnit migration guides and the type-assertions docs.
* Spell out TUnit assertion alternatives as complete awaited expressions
The Boolean, Null, Exception, and Type rows in the assertions table
previously used a `await Assert.That(x).IsTrue()` / `.IsFalse()` shorthand
for the alternative form. A reader copy-pasting just the second fragment
would end up with an un-awaited assertion that silently passes — which
contradicts the note immediately below the table about always awaiting.
Each alternative is now a complete, independently copy-pasteable expression.
Addresses the inline review on dotnet/skills#677.
---------
Co-authored-by: Amaury Levé <amauryleve@microsoft.com>
|
||
|
Amaury Levé
|
b9db51de9d |
Add TUnit-focused eval scenarios for dotnet-test-frameworks (#699)
* Add TUnit-focused eval scenarios for dotnet-test-frameworks Adds 5 new evaluation scenarios and tightens 2 existing scenarios to exercise the expanded TUnit coverage in dotnet-test-frameworks/SKILL.md (see #677). New scenarios: - Convert cross-framework assertions to TUnit syntax (await / IsEqualTo / IsTrue / IsNull / IsAssignableTo / Contains / Throws<T>) - Diagnose silently-passing TUnit test with missing await (debugging mystery framed without naming the pitfall) - Refactor TUnit try/catch to native exception assertion (Throws<T>() / ThrowsExactly<T>() / WithMessage) - TUnit lifecycle hooks at test / class / assembly / session scope - TUnit skip mechanisms — attribute, assembly-wide [assembly: Skip], and dynamic Skip.Test(...) Updated scenarios: - Identify TUnit framework: rubric now distinguishes [ClassDataSource] as a fixture/data source rather than a class marker (TUnit classes are convention-based, like xUnit) - Identify integration tests: adds a TUnit Project D using live SqlConnection and requires recommending [Category("Integration")] for TUnit Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Mirror TUnit eval scenarios into Vally spec Apply the same updated and new scenarios as in eval.yaml to the parallel eval.vally.yaml so both pipelines exercise the new TUnit-focused checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Amaury Levé <amauryleve@github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
14ee3c90f9 |
Rework evaluation to allow sharding of resource intensive evaluations (#692)
* Shard plugin evaluation via executionShard tag in eval.yaml MCP-heavy plugins like dotnet-msbuild reliably exhaust the GitHub-hosted runner host after ~87 min of cumulative MCP traffic (Copilot CLI plus dotnet dnx Microsoft.AITools.BinlogMcp child processes), which has produced no dashboard data for dotnet-msbuild scheduled runs since 2026-05-20. Cutting per-runner concurrency alone was insufficient: at the default 18-way concurrency the runner OOMs in 17-30 min; at 3-way it survives longer but still hits a host-level termination at ~87 min. Two mitigations, both required: 1. Sharding (this change). The discover step reads an optional top-level 'executionShard:' string from each tests/<plugin>/<skill>/eval.yaml and groups skills by tag, emitting one matrix entry per shard so each shard runs on a fresh runner. Untagged skills fall into a 'default' bucket. Plugins where all skills resolve to a single bucket emit one entry (unchanged behavior). The validator already calls IgnoreUnmatchedProperties so no Models.cs change is needed. Shard names follow '<plugin>--shard-<tag>' so the existing 'skill-validator-results-<plugin>--*' artifact glob keeps working. 2. Per-MCP-plugin concurrency caps (already in place). When plugin.json declares mcpServers, parallelism is forced to 1x3x1 with --judge-timeout 600 to avoid acute OOM within a shard. Also adds a workflow_dispatch 'plugin' input for targeted re-runs of a single plugin, and tags 10 dotnet-msbuild eval.yamls (heavy/medium); the remaining 8 fall into 'default'. Empirically the three shards complete in parallel in ~30-40 min each, well inside the 180-min job timeout, and successfully upload dashboard data. * Address PR review: lenient shard regex and validate dispatch plugin input - executionShard regex now tolerates optional leading whitespace, so an accidentally indented key still groups correctly instead of silently falling back to the default bucket. - workflow_dispatch 'plugin' input is now validated against ^[a-zA-Z0-9._-]+$ before any Join-Path/Test-Path use, preventing path-traversal style values (e.g. '../foo') from being injected into filesystem paths. |
||
|
Javier Calvarro Nelson
|
198e58c983 |
Add blazor skills to dotnet-blazor plugin (#357)
* Add dotnet-blazor plugin and convert-blazor-server-to-webapp skill Add the dotnet-blazor plugin with 9 skills covering Blazor Web App development: - plan-ui-change: Plan and scaffold UI features in Blazor Web Apps - create-blazor-project: Create new Blazor projects with proper render mode setup - author-component: Author Razor components with parameters, events, lifecycle - coordinate-components: Share state across components using CascadingValueSource/scoped services - use-js-interop: Call JavaScript from Blazor and vice versa - fetch-and-send-data: HTTP data access with proper service patterns - configure-auth: Set up ASP.NET Core Identity and authorization in Blazor - collect-user-input: Build forms with EditForm, validation, and file uploads - support-prerendering: Handle prerendering lifecycle and state persistence Add convert-blazor-server-to-webapp skill to the dotnet-aspnet plugin for migrating .NET 7 Blazor Server apps to .NET 8+ Blazor Web App architecture. All 10 skills pass evaluation with quality improvements ranging from 12-50% and overfitting scores within acceptable thresholds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eng/allowed-external-deps.txt * Update .gitignore --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ca21fd61c1 |
run-tests: prompt-shaped multi-TFM keywords for plugin activation (#688)
* run-tests: use prompt-shaped multi-TFM keywords for plugin activation The 'Run tests in a multi-TFM project targeting a specific framework' eval scenario kept failing plugin-mode activation after #667. The skill description had 'multi-TFM projects (--framework)' as a brief parenthetical, but the scenario prompt uses different language: 'This test project targets both net8.0 and net9.0. I only want to run the tests on net9.0 using dotnet test.' Per the InvestigatingResults.md guidance for 'skill not activated' failures, expand that clause to include keywords from the prompt — 'running tests against a single target framework when a project targets multiple TFMs' plus a concrete '<TargetFrameworks>net8.0;net9.0</TargetFrameworks>' example and the '--framework <TFM>' invocation. Per-skill description is 947 chars, well under the 1024 spec cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace simple multi-TFM scenario with a harder MTP+SDK 9 variant that requires skill knowledge The previous 'Run tests in a multi-TFM project targeting a specific framework' scenario tested a trivial prompt (single --framework flag) that the model already answers perfectly from training (baseline 4.7/5). This matches pattern #8 (baseline already good, no headroom) in InvestigatingResults.md, and explains why two prior attempts to fix plugin-mode activation via description tweaks (PR #667, #688) failed: the router rationally skips loading the skill for a trivial question. This change replaces the scenario with a meaningfully harder one that combines several signals the model often confuses: - Multi-TFM project (TargetFrameworks plural) - MTP runner (TestingPlatformDotnetTestSupport) - .NET SDK 9 (-- separator required for MTP extension args) - TRX report (MTP extension flag) The correct answer requires knowing that --framework is a dotnet test/MSBuild flag that always goes BEFORE --, while --report-trx is an MTP extension flag that goes AFTER -- on SDK 9: dotnet test --framework net9.0 -- --report-trx The ordering assertion in eval.yaml enforces this on a single line, so swapping --framework after the separator no longer satisfies the scenario. Also fix a documentation bug in SKILL.md surfaced by this work: the 'Common MTP flags > Built-in flags' table previously listed --framework and --no-build under 'On SDK 8/9, pass after --', which is wrong. Those are dotnet test/MSBuild flags consumed by dotnet test itself; they always go before --. The table now lists only flags genuinely consumed by MTP, and an Important note clarifies the rule. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
5810d492c9 |
test-anti-patterns / test-smell-detection: narrow-specialist re-pitch (Variant B) (#671)
Both skills suffer plugin-mode activation failures: in plugin runs the model bypasses the skill tool entirely for ~5/6 of test-anti-patterns audit prompts and ~2/3 of test-smell-detection prompts, answering directly from training. Variant B repositions the two skills as umbrella + narrow specialist: * test-anti-patterns becomes THE umbrella audit skill for all pragmatic test-quality reviews (anti-patterns and smell-style prompts alike). Its description explicitly absorbs smell-audit triggers. * test-smell-detection is repositioned as a narrow niche that only fires when the user explicitly asks for the testsmells.org / academic 19-smell catalog with citable smell names from the research literature. Eval prompts for test-smell-detection are rewritten to match the narrow niche (explicit testsmells.org / 19-smell catalog requests) so we test specialist activation rather than the umbrella scenarios. Variant B of a 3-PR experiment to fix activation. Companion PRs: * Variant A = activation-only rewrite, both skills keep overlap; * Variant C = full merge into test-anti-patterns. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
362b3f25ec |
Fix plugin-mode skill activation for 8 dotnet-test scenarios (#667)
* migrate-vstest-to-mtp: fix plugin activation for OutputType=Exe scenario The `Set OutputType=Exe only for test projects in Directory.Build.props` scenario was failing plugin-mode skill activation (skillActivationPlugin. activated=false, detectedSkills=[]) in the latest scheduled eval. The model answered from its own knowledge and proposed the wrong solution (`Condition="'$(IsTestProject)' == 'true'"` in Directory.Build.props, which the skill explicitly warns against). Two fixes: 1. SKILL.md description: add an explicit `conditioning OutputType=Exe to test projects when centralizing MTP properties in Directory.Build.props` USE FOR clause so the description surfaces this specific MTP migration pitfall to the skill router. Per-skill description stays at <= 1024 chars (trimmed `configuration` -> `config` and `MSTest version upgrades (use migrate-mstest-* skills)` -> `MSTest version upgrades` to keep within the spec limit). 2. eval.vally.yaml: add the missing scenario (the eval.yaml scenario was added in PR #556 but vally was never synced — vally had 10 stimuli vs eval.yaml's 11). Brings vally and skill-validator coverage to parity. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test-anti-patterns: strengthen description for severity-ranked audits The `Detect mixed severity anti-patterns in repository service tests` scenario was failing plugin-mode skill activation (skillActivationPlugin.activated=false, detectedSkills=[]) in the latest scheduled eval, even though the prompt explicitly says `Audit the test class ... for .NET test anti-patterns ... Give me a severity-ranked list (Critical / Warning / Info)`. The previous description only mentioned `USE FOR: audit test quality, review test code, find test anti-patterns, ...`, so the model could not distinguish this skill from sibling test-quality skills based on the `severity-ranked` and `Critical/Warning/Info` framing in the prompt. Surface the catalog shape (severity-ranked output, concrete code-level fixes) and add explicit `severity-ranked test audit`, `self-comparing assertions` and `broad exception types` triggers — these are the exact phrases used by failing scenarios. Description stays at 822 chars (<= 1024 limit). Aggregate dotnet-test description stays at 15,099 chars (<= 20,000 limit). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * run-tests: restructure description for higher-signal plugin activation Six run-tests scenarios were failing plugin-mode skill activation in the latest scheduled eval (skillActivationPlugin.activated=false, detectedSkills=[]) even though isolated activation worked for every scenario: - Run tests with trx reporting on MTP project (SDK 9) - Run tests in a multi-TFM project targeting a specific framework - Filter MSTest tests by category on VSTest - Filter NUnit tests by class name on VSTest - Filter TUnit tests by class using treenode-filter - Negative test: do not use MTP syntax for a VSTest project The previous description piled ~30 quoted trigger phrases (`run tests`, `run my tests`, `run these tests`, `execute tests`, `dotnet test`, `test filter`, `filter by category`, `filter by class`, `combine filters`, `run only specific tests`, `integration tests`, `unit tests`, `tests not running`, `hang timeout`, `blame-hang`, `blame-crash`, `crash dump`, `TRX report`, `TRX`, `test report`, `generate TRX`, `TUnit`, `treenode-filter`, `target framework`, `multi-TFM`, ...). In plugin mode this signal-poor keyword wall made the skill look generic enough that the router preferred to answer directly from model knowledge. Replace the keyword pile with five higher-signal scope clauses grouped by user-visible concern: 1. running, filtering, or troubleshooting `dotnet test` 2. VSTest vs Microsoft.Testing.Platform syntax (incl. `--` separator rules on SDK 8/9 vs 10+) 3. framework-specific filter syntax for MSTest / xUnit / NUnit / TUnit (--filter, --filter-class, --filter-trait, --filter-query, --treenode-filter) 4. TRX/reporting (--report-trx vs --logger trx), blame/hang/crash diagnostics, multi-TFM (--framework) 5. avoiding MTP/VSTest argument mixups These are the same concerns the previous description tried to cover, but framed as decisions the skill helps make (and that distinguish it from sibling test skills) rather than as a list of bare keywords. Per-skill description shrinks from 1,000 to 819 chars (still <= 1,024 limit). Aggregate dotnet-test description stays well under the 20,000-char cap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * test-anti-patterns: drop specific severity labels from description PR review (copilot-pull-request-reviewer on #667) caught that `Produces a severity-ranked (Critical / Warning / Info) catalog ...` in the new description contradicts the SKILL.md body, which actually teaches a 4-level Critical / High / Medium / Low taxonomy. The `Critical / Warning / Info` wording came from one eval scenario's prompt, but other prompts ask for different bucket schemes, and the skill itself does not enforce a fixed three-level mapping. Drop the specific labels from the description and keep just `severity-ranked` as the activation signal. The body remains unchanged (it teaches Critical / High / Medium / Low and the agent maps to the prompt's requested buckets at output time). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
b24599aa36 |
test-smell-detection: trim to detailed tier and add prompt-shaped triggers (#653)
Multi-model skill-validator eval (claude-haiku-4.5, claude-opus-4.6, gpt-5.3-codex, gpt-5.4-mini) showed two issues with this skill: 1. Quality regression on stronger models (+0.15 haiku, -0.14 opus, -0.17 codex, -0.29 gpt-5.4-mini): the skill added tokens and tool calls without lifting rubric quality on larger models. 2. Plugin-mode activation loss: the skill activated reliably in isolated mode but was almost never picked in plugin mode where it competed with siblings (code-testing-agent, etc.). Changes: - Trim BPE tokens 2,487 -> 1,472 (still detailed tier, well within the 800-2,500 sweet spot per SkillsBench guidance). Collapsed long smell-by-smell prose, `Why Test Smells Matter`, and pitfalls into a compact taxonomy table plus a short decision procedure / output checklist. Removed bulky examples. Kept taxonomy, calibration, negative guidance, and references intact. - Add prompt-shaped USE FOR triggers drawn from the eval prompts (`review test smells`, `review these tests and tell me if there are` `any problematic patterns`, `check my tests for test design` `problems or anti-patterns`, `give us an objective assessment of` `our integration tests`, plus explicit smell names). - Strengthen DO NOT USE FOR clause: `write new tests / scaffold tests` `from scratch / generate a complete MSTest test suite` now defers cleanly to code-testing-generator (covers the negative eval scenario). Validation: skill-validator check --plugin clean for this skill; build + 545/545 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
5f1170152b |
Add MSBuild target authoring skills (target-authoring, property-patterns, item-management, extension-points) (#669)
* Add 4 MSBuild target authoring skills for dotnet-msbuild plugin New skills: - target-authoring: three-level target chain, DependsOn extension, naming conventions - property-patterns: conditional defaults, composition, path normalization, TFM helpers - item-management: Include/Remove/Update, batching, transforms, FileWrites registration - extension-points: CustomBefore/After hooks, import gating, NuGet build extensions Each skill includes eval.yaml tests with anti-pattern fixtures. Closes #668 * Address Copilot review feedback - target-authoring: Reword DO NOT USE clause to exclude only deep incremental-build diagnostics, not basic Inputs/Outputs usage - property-patterns: Close unclosed XML elements in String Functions snippet (PropertyGroup, TargetFrameworkMoniker) - extension-points: Add missing CustomAfterMySDK property definition to match the import at bottom of example * Remove accidental pr-body.md file * Fix wildcard import placeholder path and quote NormalizePath args - extension-points: Use full MSBuildExtensionsPath expression and consistent Exists() casing in wildcard import example - property-patterns: Quote property arguments in NormalizePath call * Tune eval prompts to reduce overfitting Rewrite all 4 eval prompts as natural developer problem descriptions instead of skill-aligned checklists. Softens technique-prescriptive rubric items. Result: overfitting scores drop from 0.15-0.31 to 0.06-0.08 (all green). property-patterns now passes eval. * Fix Copilot review round 3: quote property function args, fix fixture - property-patterns: Quote in IsPathRooted call - extension-points: Quote in GetDirectoryNameOfFileAbove - item-management fixture: Use literal semicolon instead of %3B in WriteLinesToFile * Add harder multi-file eval scenarios for all 4 skills Each skill now has a second scenario with multi-file setups containing interacting bugs that require cross-file analysis: - target-authoring: SDK .targets + Directory.Build.targets with 3 bugs (target redefinition, fragile BeforeTargets, duplicate DependsOn chain) - property-patterns: nested Directory.Build.props with 5 bugs (import order, unconditional overwrite, unquoted condition, missing trailing slash, NoWarn overwrite) - item-management: csproj with 5 interacting bugs (Include vs Update, cross-product batching, eval-time FileWrites, too-broad Remove glob) - extension-points: Directory.Build.props/targets + NuGet package with 6 bugs (inverted guard, late ImportByWildcard, CustomBefore overwrite, missing Exists guard, target name collision, internal target hook) Timeouts increased to 180s for all scenarios. * Fix review round 4: revert unintended README change, quote intrinsic call in condition * Improve eval quality for 4 new MSBuild authoring skills Eval improvements across target-authoring, property-patterns, item-management, and extension-points skills: - Rewrite prompts to describe observable symptoms without naming MSBuild concepts, inspired by real dotnet/msbuild and dotnet/sdk issues (msbuild#2470, #12894, #13056, #4109, sdk#43908) - Broaden assertion regex patterns to test for diagnostic outcomes rather than skill-specific vocabulary - Improve rubric items to test understanding and diagnosis flow (diagnosis -> root cause -> fix) instead of skill terminology - Add task-completion 'fix it' scenario to each skill with file_contains assertions that verify the agent applied correct patterns (CompileDependsOn append, FileWrites, Exists guard, etc.) - Remove leading 'Anti-pattern:', 'BUG N:' comments from fixture files that were giving away answers to the baseline agent, making scenarios more realistic and better at measuring skill value Each skill now has 3 scenarios: basic review, hard multi-file review, and a task-completion fix scenario. Total: 12 scenarios across 4 skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix review comments: quote TFM intrinsic, fix PrivateAssets example - property-patterns SKILL.md: Quote the GetTargetFrameworkIdentifier intrinsic call result in the Condition to match the 'always quote both sides' guidance and avoid brittle parse when empty - item-management SKILL.md: Replace PackageReference Update on Microsoft.NETCore.App (not a real user-added item) with Include on Microsoft.CodeAnalysis.NetAnalyzers (a concrete analyzer package) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval activation and timeouts based on eval run results - Add MSBuild domain terms to prompts (e.g., 'MSBuild targets', 'MSBuild property patterns', 'MSBuild item groups', 'MSBuild extension points') to improve skill activation rates — skills were NOT ACTIVATED in plugin mode due to prompts being too symptom-focused without domain keywords - Increase timeouts: review scenarios 180s -> 240s, fix-it scenarios 180s -> 300s to avoid timeout-related scoring penalties - Keep prompts symptom-driven but include enough domain context for the skill router to activate the correct skill Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PR review comments and improve eval quality Review comment fixes: - TargetAuthoring.csproj: Replace %22/%3B URL encoding with XML escaping ("/;) so generated C# code is valid - target-authoring eval: Fix prompt referencing 'schema files' when fixture has endpoint definitions — now says 'endpoint list' - property-patterns eval: Reword LangVersion rubric — unconditional assignment in .props IS overridable by csproj but prevents override from earlier imports and command-line properties - extension-points eval: Add stronger activation terms to Fix scenario prompt (explicit mention of CustomBeforeMicrosoftCommonTargets) Eval quality improvements based on CI results: - Shorten target-authoring rubric items to reduce overfitting (was 0.30) - Remove 'Focus on target authoring patterns' instruction from hard scenario prompt (was directing agent too specifically) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve eval: increase timeout, add activation keywords, explicit fix instructions - property-patterns 'Diagnose multi-level': timeout 240s -> 360s (was hitting limit at 249s) - target-authoring 'Diagnose custom target': add CompileDependsOn/incremental build keywords for better skill activation - property-patterns 'Fix': add 'edit ... directly' to prompt to encourage file edits - target-authoring 'Fix': add 'edit ... directly' to prompt - item-management 'Fix': add 'edit ... directly' to prompt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix target-authoring skill description and eval prompts for plugin activation The skill description said 'DO NOT USE FOR: deep incremental-build diagnostics' which caused the plugin agent to skip it for scenarios about build regressions. Clarified that the skill IS for diagnosing target authoring mistakes (missing Inputs/Outputs, broken dependency chains) that cause full rebuilds. Also updated eval prompts to use 'target authoring mistakes' framing and increased Fix scenario timeout to 360s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Increase eval timeouts: extension-points NuGet (240->360s), item-management cascading (240->360s) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 3 fresh PR review comments + update eval prompts for target-authoring - property-patterns/SKILL.md: add warning that TargetFramework-conditioned PropertyGroups must be in .targets not .props (empty in single-target .props) - extension-points/SKILL.md: add Exists() guard to GetPathOfFileAbove Import to prevent build failure when no parent file is found - target-authoring/SKILL.md: fix comment mismatch — validation target runs via dependency chain, not BeforeTargets; update comment accordingly - target-authoring/eval.yaml: reframe 'Diagnose' and 'Fix' prompts to use 'dependency chain' / 'authoring patterns' language (avoids routing to incremental-build skill which also claims Inputs/Outputs diagnostics) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve skill descriptions for reliable plugin-mode activation Add 'Only activate in MSBuild/.NET build context.' prefix to all 4 skills for consistency with existing skills in the plugin. Add explicit 'diagnosing and fixing' and 'reviewing' keywords to USE FOR sections so the SDK selects these skills over the broader msbuild-antipatterns skill when prompts ask to fix or review specific item/extension/property/target patterns. Add 'general MSBuild anti-pattern catalog (use msbuild-antipatterns)' to DO NOT USE FOR sections to help the SDK disambiguate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill descriptions: shorten to under 1024-char SDK limit All 4 new skills had descriptions exceeding the 1024-character maximum enforced by the skill-validator check command. The Copilot SDK silently ignores skills with over-length descriptions, causing 0% activation in both isolated and plugin evaluation modes. Shortened all 4 descriptions while preserving key activation keywords (USE FOR / DO NOT USE FOR / 'Only activate in MSBuild/.NET build context'). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments - target-authoring eval.yaml: fix prompt to say 'Outputs attribute' (not 'Returns') since the fixture uses Outputs on a query target, which is the actual bug that causes MSBuild to skip re-execution - property-patterns eval.yaml: reword prompt and rubric — the real issue is unconditional assignment breaking parent-child Directory.Build.props inheritance, not project files being unable to override properties - extension-points SKILL.md: fix GetPathOfFileAbove example to factor the path into a property so Project= and Condition= use the same value consistently (avoiding the ..\\ vs ..\ discrepancy) - CustomSdk.targets: add WriteLinesToFile to CoreCodeGen so it actually creates the .g.cs output files (prevents compilation failure in fixture) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix property-patterns fixtures to match rubric claims Scenario 1 (path merge bug): - PropertyPatterns.csproj: add OutputPath using CustomOutputDir without a trailing separator, making the merge bug observable (artifacts\binPropertyPatterns\ instead of artifacts\bin\PropertyPatterns\) Scenario 2 (multi-level hierarchy bugs): - hard/Directory.Build.props: make LangVersion unconditional so it overwrites the child src/Directory.Build.props LangVersion=preview, matching the rubric claim about parent unconditional assignments - hard/Directory.Build.props: add NoWarn=NU1702 (conditional default) so parent suppressions are visible but lost when the child's unconditional <NoWarn>CS1591;IDE0005</NoWarn> overwrites them after import Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Sayed Ibrahim Hashimi
|
3b0ece94ad |
Add web api skill and tests (#613)
* adding web api skill and tests * fixing link * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
dfc8aa786d |
dotnet-msbuild: register AITools.BinlogMcp and use it from binlog-failure-analysis (#673)
* dotnet-msbuild: register AITools.BinlogMcp and use it from binlog-failure-analysis Adds the AITools.BinlogMcp MCP server (pulled via dnx from the dotnet-eng public NuGet feed, prerelease, no version pin) to the dotnet-msbuild plugin and rewrites the binlog-failure-analysis SKILL.md to lead with the MCP tools (binlog_overview, binlog_errors, binlog_search, binlog_items, binlog_properties, binlog_imports, binlog_nuget). The text-log replay workflow is kept as a fallback for environments without the MCP. * Improve skills * Reflect PR comments |
||
|
Amaury Levé
|
04348a46ef |
tests/dotnet-test/code-testing-agent: fix coverage XML lookup path (#657)
The grader expected coverage.cobertura.xml under a workspace-root TestResults/ directory, but `dotnet test --collect:"XPlat Code Coverage"` writes the report under <TestProject>/TestResults/<guid>/coverage.cobertura.xml. With the previous path, `find TestResults …` failed with no-such-directory before even searching, and the glob `TestResults/**/coverage.cobertura.xml` could not match the nested location either, so the assertion failed on otherwise-successful runs. Search recursively from cwd in both eval.vally.yaml (`find . …`) and eval.yaml (`**/TestResults/**/coverage.cobertura.xml`) so the assertion matches the actual output location of XPlat Code Coverage regardless of the test project layout the agent chose. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
3d59e44c7e |
Fix coverage-analysis activation for plateau diagnosis prompts (#647)
* Fix coverage-analysis activation for plateau diagnosis prompts coverage-analysis SKILL.md: - Trim verbose implementation details (provider detection, ReportGenerator) that consumed description budget without aiding skill activation - Add explicit USE FOR keywords: coverage stuck, coverage plateau, can't increase coverage, what's blocking coverage code-testing-agent SKILL.md: - Add 'diagnosing coverage plateaus or CRAP score computation (use coverage-analysis)' to DO NOT USE FOR boundary to prevent test-generation skill from intercepting diagnostic prompts * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Strengthen code-testing-agent activation; harden coverage-analysis isolated mode Address eval regressions reported on PR #647 (run 25813728646): 1. code-testing-agent: `Generate tests for ContosoUniversity ASP.NET Core MVC app` was NOT ACTIVATED in plugin mode (detectedSkills=[], skillEventCount=0, invokedAgents=[]). The model bypassed the skill system entirely. - SKILL.md description: restructure to use the proven `Use when user says ...` pattern with quoted trigger phrases (matching the run-tests skill that consistently activates), make the link to the code-testing-generator sub-agent explicit, and tighten DO NOT USE FOR clauses. - eval prompt (eval.yaml + eval.vally.yaml): make the request pipeline-shaped (`project-wide, multi-file test generation task`, `scaffold a new test project`) so the model recognizes it as multi-step work that benefits from the orchestrated pipeline. Explicitly request coverlet.collector + a Cobertura XML run so rubric criterion 1 (`high line coverage as reported by the Cobertura XML in TestResults/`) becomes achievable without overfitting. 2. code-testing-tester agent + code-testing-extensions/dotnet.md: open a scoped exception to the `skip coverage tools` rule. Default behavior stays the same, but when the user/harness explicitly asks for a Cobertura/XML coverage artifact, the agent may add coverlet.collector to the generated test csproj so the harness's coverage command produces output. The agent still does not run the coverage command itself. 3. coverage-analysis SKILL.md: add a `User-visible output is mandatory` guard at the top of the Workflow section. The latest eval showed isolated mode producing literally `(no output)` in 2 of 3 scenarios — the agent ran Compute-CrapScores.ps1 / Extract-MethodCoverage.ps1 / ReportGenerator in parallel, then the session ended without ever surfacing findings. The guard tells the agent to always return a partial summary instead of ending silent, and to deprioritize ReportGenerator HTML when budget is tight. (Plugin-mode quality is already strong: 4.3 / 4.3 / 5.0 — no regression risk there.) Aggregate dotnet-test plugin description size: 14,925 chars (limit 15,000). skill-validator check passes (22 skills, 11 agents, 1 plugin); markdownlint passes for all 4 modified files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore MSTest modernization exclusion in code-testing-agent description * Fix isolated-mode coverage-analysis: emit summary before optional ReportGenerator The previous workflow encouraged the agent to run `dotnet tool install` for ReportGenerator in parallel with the CRAP scoring scripts (Phase 2 "Steps 3 and 4 in parallel" + Phase 3 "Steps 5 and 6 in parallel"). In isolated mode that pattern reliably crashed the session with "Failed to persist session events: timeout while waiting for mutex to become available" right after the scripts returned valid data, so the agent never produced the user-facing summary. Restructure the workflow into 5 phases: - Phase 1 (Setup) - unchanged - Phase 2 (Test execution) - skip when Cobertura XML already exists - Phase 3 (Analysis) - run only the two PowerShell scripts, no RG - Phase 4 (User-facing summary) - MANDATORY, must be the next assistant response after Phase 3, before any RG work; also save coverage-analysis.md as a secondary follow-up - Phase 5 (ReportGenerator HTML/CSV) - strictly optional, post-summary, skipped by default for existing-Cobertura and plateau-diagnosis paths Also update references/output-format.md so the Reports section marks RG artifacts as "Not generated (optional - request HTML reports to enable)" when Phase 5 has not run, and update references/guidelines.md so the "show and open the markdown report" rule explicitly defers to the user-facing assistant response. Targets the isolated-mode regressions in PR #647 eval: - Project-wide coverage with existing Cobertura: 1.0/5 -> expected 3+ - Coverage plateau diagnosis: 1.0/5 -> expected 3+ - Run coverage from scratch: 2.3/5 -> expected steady or up Verified: skill-validator check --plugin ./plugins/dotnet-test passes; markdownlint-cli2 clean on all 3 modified files. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #647 review comments 1. Prerequisites: distinguish the from-scratch path (needs NuGet for the coverage-provider package install + optional internet for ReportGenerator) from the existing-Cobertura path (needs neither). 2. Add Step 2c "Discover or accept existing Cobertura XML" so the existing-data path actually has $coberturaFiles populated before Phase 3, instead of relying on Phase 2's discovery (which it skips). Also clarify Step 2's destructive Remove-Item only manages the skill-owned coverage-analysis/ subdirectory. 3. references/output-format.md: replace the unconditional "Reports saved to: <coverageDir>/reports/" line with one that always points at <coverageDir>/ (markdown summary + raw Cobertura) and only mentions reports/ if Phase 5 ran. 4. Have Compute-CrapScores.ps1 emit OVERALL_LINE_COVERAGE and OVERALL_BRANCH_COVERAGE from the Cobertura root attributes, and update Phase 4 to read those values directly from the script's output. The Phase 4 mandatory-summary rule no longer requires a separate XML parse before composing the response. Verified: skill-validator check --plugin ./plugins/dotnet-test passes; markdownlint-cli2 clean on all modified files; Compute-CrapScores.ps1 smoke-tested on a synthetic Cobertura XML (emits OVERALL_LINE_COVERAGE:75, OVERALL_BRANCH_COVERAGE:50 alongside HOTSPOTS). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestion from @Evangelink * Address unresolved coverage-analysis and eval review comments * Refine follow-up review feedback from validation * Tighten coverage aggregation fallback notes and counters * Clarify pre-response save instruction wording --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
f532cf1d94 |
Add coverage-touching and self-referential assertion heuristics (#635)
* Add coverage-touching and self-referential assertion heuristics
- test-anti-patterns: add 'Coverage touching' and 'Self-referential assertion'
as named Critical-severity anti-patterns in the detection catalog
- assertion-quality: add 'Tests with self-referential assertions' metric and
a calibration rule distinguishing tautological from valid round-trip assertions
- Add eval scenarios for both new heuristics in test-anti-patterns (coverage-
touching ProductCatalog facade, self-referential SerializerTests)
- Add eval scenario and fixture for self-referential assertions in
assertion-quality (ConfigTests with identity/round-trip patterns)
* Fix test-anti-patterns activation: rewrite description with USE FOR keyword list
The skill was not activating in any eval scenario because the description
used narrative prose instead of the explicit 'USE FOR:' / 'DO NOT USE FOR:'
keyword list pattern that well-activating sibling skills use (run-tests,
writing-mstest-tests). Restructured the description with trigger phrases
matching common user prompts.
* Fix eval activation, timeouts, and add missing vally scenarios
- test-anti-patterns/eval.vally.yaml: increase timeout 2m→5m, add
coverage-touching and self-referential scenarios with constraints
- assertion-quality/eval.vally.yaml: add self-referential scenario
with environment files, broaden assertion pattern
- test-anti-patterns/eval.yaml: add setup.files to all 6 scenarios
to create workspace files that trigger skill discovery/activation
- assertion-quality/eval.yaml: increase timeouts 120→300 for
self-referential and decline-request scenarios, broaden
self-referential assertion pattern
- Create fixture files for mixed-severity, well-written,
coverage-touching, and self-referential test-anti-patterns scenarios
* Address review comments: fix test names and use source references
- Rename ThrowsArgumentException to ThrowsArgumentOutOfRangeException in
PaymentProcessorTests.cs fixture and inlined code in eval.yaml and
eval.vally.yaml to match the actual exception type being asserted.
- Replace inline csproj content blocks with source references to existing
fixture files for UserRepo, Payments, and CatalogService scenarios in
eval.yaml to avoid drift between fixtures and eval setup.
* Address review comments: pin MSTest versions, use source refs, fix naming
- Pin MSTest to 3.8.2 in all fixture .csproj files, add standard
properties (ImplicitUsings, Nullable, IsPackable)
- Replace inline csproj content blocks with source references to
fixture files to avoid content drift
- Create missing .csproj fixtures for flaky-coupled, duplicate-magic,
and assertion-quality self-referential scenarios
- Fix test method names: ThrowsArgumentException ->
ThrowsArgumentOutOfRangeException to match the actual assertion
* Fix activation, eval timeouts, and per-skill description limits
Address remaining issues from PR #635 eval scoreboard
(run 25677669933) and skill-validator failures:
1. test-anti-patterns activation (was NOT ACTIVATED in BOTH plugin and
isolated mode for ALL 6 scenarios — model answered everything from
inline code without ever needing the skill):
- eval.yaml + eval.vally.yaml: drop the giant inline `csharp` blocks
from every prompt. Each prompt now names the workspace files
explicitly (`Audit `Foo.Tests/FooTests.cs` for .NET test
anti-patterns ...`) and asks for the skill's signature deliverable
(severity-ranked list with concrete code-level fixes). The model
must read the workspace, which gives it a chance to consult the
skill — and the skill's structured-findings format is the actual
value-add over inline analysis.
- All 6 scenarios consistently get reject_tools (read-only review)
and timeout 180s.
- eval.vally.yaml: also added missing environment.files for each
scenario (was prompt-only before, no fixture on disk).
2. assertion-quality `Flag assertion-free tests` plugin-mode activation
(was bouncing to other skills because the prompt was generic `check
whether these tests verify anything useful`):
- eval.yaml + eval.vally.yaml: rewrite the prompt to name the file
(`SmokeTests/ApiEndpointSmokeTests.cs`) and explicitly request
`assertion quality and depth`, `assertion-free tests`, and
`trivial assertions like Assert.IsNotNull` — direct trigger
phrases for assertion-quality.
- assertion-quality SKILL.md: extend description with explicit USE
FOR keywords for `assertion-free tests`, `self-referential`, and
`tautological` so plugin-mode activation isn't won by sibling
skills like test-anti-patterns.
3. test-anti-patterns description size — was 1,301 chars (over the
1,024 per-skill limit) AND pushed dotnet-test plugin aggregate to
15,312 (over the 15,000 limit), failing skill-check on PR #635.
Trimmed to 779 chars; aggregate now at 14,983.
4. Vally job-level timeout (60m → 180m) was already raised on main
in commit
|
||
|
Amaury Levé
|
de0a97c551 |
Fix migrate-vstest-to-mtp activation for Handle exit code 8 scenario (#648)
The `Handle exit code 8 when migrating from VSTest to MTP` scenario was
not activating the migrate-vstest-to-mtp skill in plugin mode in the
latest scheduled eval (skillActivationPlugin.activated=false,
detectedSkills=[], skillEventCount=0, vally row showed `--`), even
though all 10 other scenarios in this eval activated cleanly.
Three layered fixes:
1. Prompt (eval.yaml + eval.vally.yaml): switch from past-tense
`We migrated`/`How do I fix this behavioral difference` to
present-tense migration framing — `We're migrating ... After
enabling MTP ... How do I handle this MTP behavioral difference?`.
The previous past-tense framing read as a generic post-migration CI
troubleshooting question, which the model felt confident answering
without consulting any skill.
2. SKILL.md description: add an explicit `USE FOR:` clause for MTP
behavioral differences (exit code 8, zero tests discovered,
--ignore-exit-code, TESTINGPLATFORM_EXITCODE_IGNORE), matching the
pattern that fixed coverage-analysis activation in
|
||
|
Amaury Levé
|
0f31228b58 |
Fix eval timeouts and skill activation for dotnet-test evals (#641)
* Fix evaluation timeouts and improve prompts for dotnet-test skills
- writing-mstest-tests: increase global vally timeout from 3m to 5m to match
the 300s eval.yaml timeout for the complex 'Modernize legacy test patterns' stimulus
- code-testing-agent: increase timeout from 40m/2400s to 60m/3600s
- mtp-hot-reload: increase timeout from 6m to 10m and remove overly strict
expect_tools constraint on launchSettings stimulus
- migrate-mstest-v1v2-to-v3: rephrase prompt for assertion overload errors to
mention CS1501 and cover AreNotEqual/AreSame alongside AreEqual
* Increase eval timeouts for mtp-hot-reload and migrate-mstest-v1v2-to-v3 scenarios
* Strengthen activation prompts and add missing vally scenarios
- migrate-mstest-v1v2-to-v3: Add explicit 'MSTest v2 to v3' migration
language in .testsettings and DataRow prompts (both eval.yaml and
eval.vally.yaml)
- writing-mstest-tests: Add 'MSTest' keyword and skill-specific
terminology to all prompts that lacked activation signal
- writing-mstest-tests: Add 5 missing vally scenarios (string
assertions, comparison assertions, collection/null/reference
assertions, conditional execution, parallelization)
- Add service-registry fixture files for the collection assertions
scenario
* Improve skill activation via SKILL.md descriptions and prompt alignment
Three-pronged fix for systematic NOT ACTIVATED failures:
1. SKILL.md descriptions: Restructure writing-mstest-tests, migrate-
mstest-v1v2-to-v3, and code-testing-agent descriptions to use the
proven 'Use when user says' pattern with quoted trigger phrases
(matching the working run-tests pattern)
2. Vally prompts: Each prompt now contains at least one exact quoted
trigger phrase from its SKILL.md description to ensure strong
description-to-prompt matching
3. Specific fixes:
- writing-mstest-tests: All 15 prompts now include MSTest-specific
trigger phrases (Assert.ThrowsExactly, sealed test class, DataRow,
DynamicData, Assert.HasCount, Assert.IsInstanceOfType, etc.)
- migrate-mstest-v1v2-to-v3: Add '.NET 5 dropped' and 'framework
compatibility issues' to trigger phrases; strengthen dropped-TFM
prompt
- code-testing-agent: Add 'generate comprehensive unit tests' and
'achieve high code coverage' to trigger phrases
* Revert SKILL.md description changes that caused activation regression
The description restructuring in
|
||
|
Amaury Levé
|
f8cae94781 |
Add test-tagging evals for uncovered skill aspects (#639)
* Add test-tagging evals for uncovered skill aspects Add 3 new scenarios (6-8) to test-tagging eval.yaml: - Scenario 6: Partially-tagged MSTest suite — verifies existing traits are preserved without duplication, only taxonomy values are used, and framework detection works correctly. - Scenario 7: Misleading method names (NUnit) — verifies the agent reads test bodies instead of guessing from method names, uses NUnit [Category] syntax, and classifies security/concurrency/boundary tests accurately. - Scenario 8: Build verification — verifies the project still compiles after tagging changes and only valid taxonomy values are applied. New fixtures: - fixtures/mstest-partially-tagged: OrderService suite with 3 pre-tagged and 12 untagged tests. - fixtures/nunit-misleading-names: TaskManager suite where method names deliberately contradict test body behavior. * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add missing vally stimuli for test-tagging scenarios 6-8 --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
4c40b873f2 |
Add detect-static-dependencies evals for report structure, obj exclusion, and lambda statics (#640)
Cover previously uncovered skill aspects: - [Validation] All .cs files in scope were scanned (check count) - [Validation] Report includes category totals, top patterns, and affected files - [Validation] obj/ and bin/ directories were excluded - [Pitfall] Scanning obj/ or generated code - [Pitfall] Missing statics inside lambdas/LINQ - [WorkflowStep] Step 4: Present the report Three new scenarios with supporting fixtures: - Structured report with file count, categories, and top patterns - Exclude obj/bin directories (with Generated.cs decoy in obj/) - Detect statics inside lambdas and LINQ expressions |
||
|
Dan Moseley
|
fe11dec69d |
Update .NET 10→11 migration skill for Preview 2 and 3 (#628)
* Update .NET 10→11 migration skill for Preview 2 and 3 Add breaking changes from .NET 11 Preview 2 and Preview 3 to the migrate-dotnet10-to-dotnet11 skill. The skill previously covered only Preview 1. Changes: - Add 15 new breaking changes across all reference files - Create new aspnetcore-dotnet10to11.md reference (OpenAPI v3, Blazor) - Update SKILL.md workflow steps, assessment checklist, and patterns - Add 4 new eval scenarios covering high-impact P2/P3 changes - Add github.com/dotnet/aspnetcore to known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback - Replace prompt-echoable 'corrupt' in ZipArchive eval regex with 'checksum' - Update EF Core reference note to say 'Preview 1 through Preview 3' - Add missing Source link for Zstandard entry Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: tighten eval regex, fix raw HTML - Tighten OverscanCount grader regex to require 'OverscanCount' with a number, instead of matching bare '3' anywhere - Wrap Blazor component snippet in backticks so it renders as code Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
e6cf308357 |
Add eval scenario for filter-syntax [trait] query language pattern (#638)
* Add eval scenario for filter-syntax [trait] query language pattern Add a run-tests scenario that exercises the xUnit v3 query filter language with [traitName=traitValue] syntax (--filter-query), covering the uncovered [CodePattern] [trait] at line 113 of filter-syntax SKILL.md. New fixture: mtp-xunit-v3-query with multiple test classes and traits (Smoke, Regression, Unit) to test combined class-pattern + trait filtering. * Address review feedback for xUnit v3 query eval coverage --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d575b1faa8 |
Add eval scenarios for writing-mstest-tests skill coverage (#593)
* Add eval scenarios for writing-mstest-tests skill coverage * Fix invalid YAML and remove build-validation claim from rubric * Fix timeouts and low quality * Improve further * Increase timeout for string assertions scenario from 120s to 180s Agent-Logs-Url: https://github.com/dotnet/skills/sessions/9943a60f-070b-4bea-9912-c994d1bed428 Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
cfc293f8fe |
Fix skill activation for Assert.AreEqual overload eval scenario (#589)
The 'Fix Assert.AreEqual object overload errors after v3 upgrade' evaluation scenario was failing because: 1. The prompt said 'latest major version' without mentioning 'v3' or 'MSTest v3', missing the skill's activation keywords. 2. The project file showed MSTest 3.8.0 (post-upgrade state) with no v2 context, while the SKILL.md 'When Not to Use' said 'Project already uses MSTest v3'. 3. The prompt's 'fix compilation issues' phrasing competed with writing-mstest-tests skill activation. Changes: - Updated prompt to explicitly mention upgrading from v2 to v3 and reference MSTest v3 compatibility issues - Added XML comment in csproj showing the pre-upgrade v2 package versions - Clarified SKILL.md 'When Not to Use' to distinguish fully-migrated projects from projects with active migration errors |
||
|
Amaury Levé
|
9f531203ec |
Promote 4 experimental test skills to dotnet-test plugin (#634)
Move assertion-quality, test-gap-analysis, test-smell-detection, and test-tagging from dotnet-experimental to dotnet-test. These skills have demonstrated sufficient eval coverage, unique value beyond baseline model capabilities, and are already integrated into the test-quality-auditor agent pipeline. Changes: - Move skills and tests, dropping the exp- prefix - Update all cross-references in skill descriptions and routing tables - Update test-quality-auditor agent and eval scenarios - Update CODEOWNERS for the moved paths - Delete duplicate exp-dotnet-test-frameworks (identical to dotnet-test-frameworks) - Fix stale exp-dotnet-test-frameworks reference in exp-test-maintainability Skills held back for further iteration: - exp-mock-usage-analysis: baseline model scores 5/5 on key scenarios - exp-test-maintainability: quality gains offset by token overhead |
||
|
Amaury Levé
|
97b23ae750 |
Fix timeout for Modernize legacy test patterns evaluation (#590)
The scenario was timing out on CI because: - The 180s timeout left insufficient headroom for CI machines (local runs take 120-135s, CI machines are slower due to load, network NuGet restores, and parallel execution) Changes: - Add 'dotnet restore' setup command to pre-warm NuGet cache before the agent starts (same pattern used by migrate-mstest-v3-to-v4) - Increase timeout from 180s to 300s to accommodate CI variability |
||
|
Jan Jones
|
3aefa0f10a |
Update file-based app skill to mention features from 10.0.300 (#602)
Co-authored-by: Copilot <copilot@github.com> |
||
|
Aditya Mandaleeka
|
9433a31e67 | Initial port of evals to Vally (#615) | ||
|
Copilot
|
2f9922e073 |
Remove LangVersion=latest from skill examples and test fixtures (#608)
* Initial plan * Remove LangVersion=latest from skill docs and test fixtures Per official C# docs, LangVersion should not be set to `latest` as it causes builds to silently vary across machines with different SDKs. Best practice is to omit LangVersion entirely and let the TFM determine the correct C# version automatically. - Update msbuild-modernization/SKILL.md: remove LangVersion=latest from example and add explicit guidance to omit LangVersion - Update directory-build-organization/SKILL.md: remove from example - Update directory-build-organization/references/multi-level-examples.md: remove from all 4 examples - Update msbuild-antipatterns/SKILL.md: remove from AP-08 examples - Remove LangVersion=latest from 11 test fixture project files Agent-Logs-Url: https://github.com/dotnet/skills/sessions/7a3d2722-770b-4010-a6e8-70c92668ad4c Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Fix LangVersion guidance and stale eval rubric reference Agent-Logs-Url: https://github.com/dotnet/skills/sessions/e89afcd8-7227-424b-9c8c-9ec0ce7113c2 Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Manish Jayaswal
|
2633d3d482 |
add a new dotnet11 plugin and a new system text json skill to it. (#535)
* add a new dotnet11 plugin and a new system text json skill to it. * Update plugins/dotnet11/skills/system-text-json-net11/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * added the option to run dotnet11 skills. by default they don't get included * Update tests/dotnet11/system-text-json-net11/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add JSON output assertions to system-text-json-net11 eval.yaml Agent-Logs-Url: https://github.com/dotnet/skills/sessions/8e302f3f-4328-44a0-845d-4eb0c930af63 Co-authored-by: ManishJayaswal <9527491+ManishJayaswal@users.noreply.github.com> * addressed CR comments * addressed more CR comments * fixed agument to pass while enabling dotnet11 skills * Remove dotnet11-specific filtering from skill-validator and evaluation workflow Now that .NET 11 is the default SDK for the repo (global.json updated on main), there is no need for special dotnet11 exclusion logic. The dotnet11 plugin skills are treated like any other plugin and included in all evaluation runs. Changes: - Remove --include-dotnet11 CLI option and filtering logic from EvaluateCommand.cs - Remove IncludeDotnet11 from ValidatorConfig model - Remove --dotnet11 flag parsing from evaluation workflow gate job - Remove dotnet11 exclusion logic from discover job - Remove conditional .NET 11 preview SDK install step (global.json handles it) - Remove dotnet11 from excludeFromSchedule list - Remove all dotnet11-specific warning/status messages - Remove section 9 from InvestigatingResults.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * address code review comments after rebase * added the plugin to cursor and added MIT license to skill --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: [Manish Jayaswal] <[manishj@microsoft.com]> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ManishJayaswal <9527491+ManishJayaswal@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Tomas Grosup
|
a55227f030 |
Fold F# .fsproj ordering into msbuild-antipatterns AP-05 (#577)
* Add fsharp-project-structure skill for .fsproj file ordering and .fsi files Teaches agents F# compilation order rules, signature file placement, and common FS0039/FS0010/FS0034 fixes. Includes eval with three scenarios (add module, fix broken order, add signature file) and test fixtures. Validated +64.6% improvement on gpt-4.1 (significant, CI [+17.2%, +63.0%]). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add fsharp-project-structure skill for .fsproj file ordering and .fsi files Teaches agents F# compilation order rules, signature file placement, and common FS0039/FS0010/FS0034 fixes. Includes eval with three scenarios (add module, fix broken order, add signature file) and test fixtures. Validated +64.6% improvement on gpt-4.1 (significant, CI [+17.2%, +63.0%]). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/fsharp-project-structure/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fold F# project-structure into msbuild-antipatterns AP-05 Scraps the standalone fsharp-project-structure skill and instead adds a short F# exception to AP-05 (Manual File Listing in SDK-Style Projects): F# compilation is order-dependent, so .fsproj files must list Compile items in dependency order (utilities first, entry point last), and .fsi signature files must appear immediately before their companion .fs. The three F# eval scenarios (add module, fix broken order, add signature file) are ported into tests/dotnet-msbuild/msbuild-antipatterns/eval.yaml with fixtures isolated under a fsharp/ subfolder (with empty Directory.Build.{props,targets} to prevent the existing C#-antipattern fixture files from leaking into the F# builds). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
1b105d1940 |
Fix dotnet-test-frameworks skill activation for evaluations (#580)
* Fix dotnet-test-frameworks skill activation for evaluations The skill had user-invocable: false and a description saying 'DO NOT USE directly', which prevented the model from activating it during evaluation. All 6 eval scenarios default to expect_activation: true, causing SkillNotActivated failures on CI. - Remove user-invocable: false from both dotnet-test and experimental copies - Rewrite description with activation keywords matching eval prompts (assertion mapping, framework identification, skip annotations, exception assertions, lifecycle conversion, integration test markers) - Preserve note that other analysis skills also load this as reference data * Restore user-invocable: false, fix cross-refs, add expect_activation: false Address review comment: remove exp-* skill references from dotnet-test- frameworks description since those skills reference exp-dotnet-test- frameworks instead. Optimize eval results: this reference data skill's content is knowledge the model already has (baseline scores 5.0/5 on most scenarios). Direct activation adds token/time overhead without quality improvement: - Restore user-invocable: false (correct for reference data skill) - Shorten description to not trigger activation - Add expect_activation: false to all 6 scenarios The skill's value is realized when loaded by consumer skills (e.g. test-anti-patterns), not via direct user queries. |
||
|
Amaury Levé
|
dee829b89c |
Fix eval timeout for MSTest.Sdk scenario by pre-warming NuGet cache (#548)
The 'Migrate MSTest.Sdk v3 project using ManagedType and TestTimeout' scenario times out on CI because MSTest.Sdk/4.1.0 resolution takes 40+ seconds on cold NuGet cache. Each agent session may call dotnet build multiple times, easily exceeding the 240s timeout. Add a 'dotnet restore' setup command that runs before the agent starts (outside the timeout window), pre-warming the NuGet cache and obj/ directory so subsequent builds complete in ~2s. |
||
|
Amaury Levé
|
cb7c10bd32 |
fix mstest sdk eval timeout (#558)
* Fix timeout in MSTest.Sdk v3-to-v4 eval scenario 9 The fixture csproj had MSTest.Sdk/3.8.0 while the prompt claimed the user already upgraded to 4.1.0. This mismatch caused the model to edit the csproj and run dotnet build, triggering slow MSTest SDK NuGet downloads that exceeded the 240s timeout on CI. - Update fixture csproj to MSTest.Sdk/4.1.0 to match the prompt - Add specific CS error codes to the prompt so the model provides direct fixes instead of building - Update rubric item to reflect the project already being on 4.1.0 * Fix CI timeout in migrate-mstest-v3-to-v4 eval scenario Reduce scope of the 'Fix multiple v4 breaking changes' scenario: - Remove 3 lower-signal breaking changes from the fixture (AreEqual params, IsInstanceOfType out param, Properties.Contains) to cut agent work - Add max_turns: 5 to cap agent iterations - Lower timeout from 240 to 180 seconds * Strengthen net6.0 scenario prompt for reliable skill activation |
||
|
Copilot
|
c625f09db5 |
Recommend MSBuildProjectName for OutputType conditions in MTP migration skill (#556)
* Initial plan * Improve MTP migration skill guidance for OutputType conditions Agent-Logs-Url: https://github.com/dotnet/skills/sessions/ae2f9ae1-c8c3-4e50-95ab-ccdc08bc388b Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> * Add eval scenario for OutputType conditioning in Directory.Build.props Agent-Logs-Url: https://github.com/dotnet/skills/sessions/734b3736-45a4-4f5d-b83c-3e6ee72fcfae Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> Co-authored-by: Amaury Levé <amauryleve@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7841934a75 |
Add migrate-mstest-v3-to-v4 eval scenarios for uncovered patterns (#564)
* Add migrate-mstest-v3-to-v4 eval scenarios for uncovered patterns
Add 5 new scenarios (Goals 10-14) covering:
- Verified package update with build/test validation
- Sealed custom TestMethodAttribute with Timeout migration
- TestMethodAttribute/TestMethod display name constructor syntax
- Assert.IsInstanceOfType out parameter breaking change
- TreatDiscoveryWarningsAsErrors and behavioral changes review
Add 4 new fixture directories:
- v3-verified-migration: buildable project for end-to-end validation
- v3-sealed-timeout: sealed attr + TestTimeout.Infinite patterns
- v3-testmethod-displayname: [TestMethod/Attribute("name")] patterns
- v3-instanceoftype: Assert.IsInstanceOfType<T> out param pattern
Covers uncovered validation items (lines 449-451, 458) and code
patterns: [TestMethod], [Timeout], sealed, [TestMethodAttribute],
Assert.IsInstanceOfType.
* Improve skill description for activation and bump scenario timeouts
Rewrite migrate-mstest-v3-to-v4 frontmatter description to improve SDK
semantic matching:
- Lead with symptom (fix build errors) instead of action (migrate)
- Add error codes CS0507, CS0103, CS1061, CS1615 as trigger signals
- Add symptom phrase: tests don't compile after upgrading MSTest
- Expand breaking change keywords: sealed, DisplayName syntax,
Assert.IsInstanceOfType out parameter, TestContext.TestName
- Keep under 1024 char limit
Bump timeout 180s -> 240s for scenarios 11 (sealed attr) and 13
(IsInstanceOfType) that hit timeout during evaluation.
* Address PR review comments
- Remove unused 'using System' from OrderServiceTests.cs and TypeCheckTests.cs
- Renumber duplicate Goal 10 to Goal 15 (non-activation scenario)
- Tighten file_contains assertion to check MSTest.TestFramework and
MSTest.TestAdapter individually instead of broad '4.' match
- Tighten Timeout assertion to require int.MaxValue instead of matching
pre-migration symbol names
- Fix error code CS0507 -> CS0115 in scenario 11 prompt to match the
actual MSTest v4 error for missing Execute override
|
||
|
Amaury Levé
|
96489b425e | eval(dotnet-test): add evaluations for dotnet-test-frameworks skill (#563) | ||
|
Amaury Levé
|
7740b2b53c |
Improve migrate-mstest-v1v2-to-v3 evals to differentiate from baseline (#547)
Five scenarios were performing at or below stock Copilot because prompts gave away answers and rubric items tested knowledge the baseline LLM can deduce from the code alone. Changes per scenario: - Fix DataRow type mismatch: added double-to-float mismatch, vaguer prompt, rubric tests both mismatch types and latest-3.x knowledge - Migrate .testsettings to .runsettings: added code coverage data collector to fixture, rubric tests correct TestTimeout mapping (<MSTest><TestTimeout> not <TestSessionTimeout>) and collector mapping - Migrate MSTest v2 NuGet to v3: vaguer prompt (forces model to read csproj), rubric tests metapackage/MSTest.Sdk recommendation and Microsoft.NET.Test.Sdk dependency awareness - Fix Assert.AreEqual overload errors: removed exact error code from prompt, added mixed-type and interface-reference cases, rubric tests correct generic type parameter selection - Handle dropped TFM: changed to multi-target (net5.0;net462; netcoreapp3.1), rubric tests that model knows which TFMs are still supported and doesn't upgrade them indiscriminately |
||
|
Copilot
|
c14d90d28e |
Align MSTest v2→v3 eval assertions/rubric with fixture semantics (#550)
* Initial plan * Align MSTest v2->v3 eval rubric with fixture assertions Agent-Logs-Url: https://github.com/dotnet/skills/sessions/055e6526-4b7d-4457-9004-2c8c99969cd1 Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com> |
||
|
Amaury Levé
|
17df9a10eb |
Fix code-testing-agent eval timeout by removing unused NuGet packages (#551)
The ContosoUniversity test fixture had 17 unused NuGet packages (bootstrap, jQuery, Modernizr, Antlr4, WebGrease, Microsoft.Build.Tasks.Core, etc.) that bloated NuGet restore and build time. The multi-agent test generation pipeline performs multiple build/test cycles, causing cumulative delays that exceeded the 2400s scenario timeout on CI. Changes: - Remove 17 unused PackageReference entries from ContosoUniversity.csproj - Remove stale ItemGroup entries (Global.asax, Scripts, compile removes) - Remove NU1701 warning suppression (no longer needed) - Increase command_timeout from 180s to 300s for dotnet test assertion (accounts for first-time restore + build + test + coverage collection) |
||
|
Amaury Levé
|
7a2e91ce94 |
Fix mtp-hot-reload eval timeouts by removing expect_tools: bash (#546)
The 5 failing scenarios were timing out because expect_tools: [bash] forced the agent to execute terminal commands. The hot reload workflow involves 'dotnet run' which starts a test host that blocks indefinitely waiting for file changes, causing the scenario to exhaust its timeout. These scenarios test whether the model provides correct guidance (package name, env var, commands), not whether it actually executes them. The output_matches assertions already verify the response content. Removed expect_tools from: - Suggest hot reload for failing test in MTP project (SDK 9) - Suggest hot reload for failing test in MTP project (SDK 10) - Enable hot reload when package already installed - Suggest launchSettings.json configuration for hot reload - Negative: VSTest project cannot use MTP hot reload Also reduced timeouts since scenarios no longer need to wait for command execution. |
||
|
Amaury Levé
|
400661df09 |
run-tests: add critical rules, detection guidance, and troubleshooting (#514)
* run-tests: add critical rules, detection guidance, and troubleshooting - Add 'Critical Rules' table to prevent cross-platform VSTest/MTP mistakes - Expand Step 1 detection with explicit file-by-file lookup table - Add 'dotnet --version' as first detection action - Add Troubleshooting section with 7 common error patterns - Expand Common Pitfalls from 3 to 7 entries - Strengthen negative guidance for SDK version-specific syntax * Improve description * Improve evaluation * Add expected and rejected tools * Change * Improve do not use sections * Improve prompt * Increase eval timeouts for scenarios hitting the limit - run-tests: trx reporting MTP SDK 9 (360s -> 480s) - run-tests: blame-hang MTP SDK 10 (300s -> 420s) - run-tests: combine filter criteria VSTest (180s -> 300s) - mtp-hot-reload: hot reload SDK 9 (360s -> 480s) - mtp-hot-reload: hot reload filter (180s -> 300s) - code-testing-agent: ContosoUniversity (1800s -> 2400s) |
||
|
Amaury Levé
|
f46275e64e |
Add evaluation tests for test-migration and test-quality-auditor agents (#543)
test-migration eval (4 scenarios): - Auto-detect MSTest v2 on VSTest and recommend migration path - Detect xUnit v2 and route to xUnit v3 migration - Execute targeted MSTest v2 to v3 migration - Non-activation: decline test writing request test-quality-auditor eval (4 scenarios): - Comprehensive audit of weak test suite (multi-skill pipeline) - Targeted anti-pattern review - Assertion quality analysis - Non-activation: decline test generation request Includes fixture projects with intentional quality issues: - MSTest v2 project on VSTest for migration detection - xUnit v2 project for migration detection - ShoppingCart test suite with anti-patterns (assertion-free, tautological, catch-and-swallow, Thread.Sleep, shallow assertions, missing coverage) |
||
|
Amaury Levé
|
d7b80a806b |
Fix SDK 9.0.200 global.json pinning causing eval timeouts (#542)
Add rollForward: latestFeature to all global.json entries that pin SDK 9.0.200 so that any 9.0.xxx SDK satisfies the requirement. Without this, machines with only 9.0.3xx installed fail all dotnet commands during evaluation, causing the model to burn its entire timeout budget on failed bash retries. |
||
|
Amaury Levé
|
c18d4260de |
Fix timeout in MSTest.Sdk v3-to-v4 eval scenario 9 (#544)
The fixture csproj had MSTest.Sdk/3.8.0 while the prompt claimed the user already upgraded to 4.1.0. This mismatch caused the model to edit the csproj and run dotnet build, triggering slow MSTest SDK NuGet downloads that exceeded the 240s timeout on CI. - Update fixture csproj to MSTest.Sdk/4.1.0 to match the prompt - Add specific CS error codes to the prompt so the model provides direct fixes instead of building - Update rubric item to reflect the project already being on 4.1.0 |
||
|
Amaury Levé
|
7e6954f9a9 |
Add testability migration skills and agent for dotnet-test (#518)
* Add testability migration skills and agent for dotnet-test Introduces three skills and one orchestrator agent to help developers incrementally replace hard-to-test static dependencies with injectable abstractions: - detect-static-dependencies: scans C# code for DateTime.Now, File.*, Environment.*, HttpClient, Console.*, Process.* and ranks by frequency - generate-testability-wrappers: generates wrapper interfaces or guides adoption of built-in abstractions (TimeProvider, IHttpClientFactory, System.IO.Abstractions) - migrate-static-to-wrapper: mechanical codemod-style bulk replacement of static call sites with wrapper calls, scoped per project/namespace - testability-migration agent: orchestrates the Detect-Generate-Migrate pipeline end-to-end Includes eval.yaml tests and fixture projects for all three skills and the agent. * Fix activation, token overhead, and eval timeouts - Expand skill descriptions with more trigger phrases for migrate-static-to-wrapper and generate-testability-wrappers to fix NOT ACTIVATED issues - Collapse verbose replacement mapping tables and constructor examples to reduce token overhead (~30% fewer tokens) - Increase generate-testability-wrappers eval timeouts from 120s to 240s to prevent timeout-impacted scoring - Soften detect-static-dependencies scenario 2 rubric to reduce run-to-run variance in plugin mode - Increase migrate-static-to-wrapper non-activation timeout |
||
|
Milos Kotlar
|
038dd4f087 |
Add Apple crash symbolication skill (#201)
* Add SKILL.md for iOS crash symbolication process Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/ios-crash-symbolication/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add automation script, tests, and review fixes for ios-crash-symbolication - Add Symbolicate-Crash.ps1 (664 lines): parses .ips JSON, searches local dSYMs (SDK packs, NuGet cache, user paths), verifies UUIDs via dwarfdump, batch-symbolicates with atos, identifies runtime version - Update SKILL.md: add Automation Script and Runtime Version Identification sections, fix atos -o to point inside dSYM bundle, add dwarfdump and MAUI to frontmatter, wrap steps in Workflow heading for consistency with Android sibling, fix misleading 'rebuild' guidance per review feedback (Rolf) - Add eval.yaml with 7 test scenarios: Mono crash, CoreCLR crash, no .NET frames, NativeAOT, multi-library UUIDs, ASI field priority, and Android tombstone rejection - Add 5 .ips test fixture files (two-part JSON format) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add iOS crash log and evaluation scenarios for symbolication tests * Remove unused file * Add CODEOWNERS entry for iOS crash symbolication * Rename ios-crash-symbolication to apple-crash-symbolication Address PR feedback to support all Apple platforms (tvOS, Mac Catalyst, macOS) not just iOS: - Add $appleRids array covering ios, tvos, maccatalyst, and osx RIDs - Refactor Find-Dsym and Find-RuntimeVersion to search all platform packs - Rewrite SKILL.md for orchestration focus and reduced token budget - Extract domain knowledge to references/ips-crash-format.md - Tune eval.yaml: outcome-based rubrics, broad assertions, overfit 0.26 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix script robustness: base address parsing and null-safe property access - Simplify base address cast to [uint64] without manual hex prefix stripping - Use PSObject.Properties check before accessing lastExceptionBacktrace Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dSYM-path rubric items to CoreCLR and ASI scenarios Add rubric items testing for skill-specific domain knowledge (SDK pack paths, NuGet cache directories) that baseline agents cannot provide. This creates the quality delta needed to pass the 10% improvement threshold while keeping overfit at 0.12 (Low). 3-run validation: 30.1% improvement, 5/7 scenarios positive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor code structure for improved readability and maintainability * Fix crash symbolication script for real .ips format and improve eval scenarios - Fix Symbolicate-Crash.ps1 strict-mode bugs with real .ips crash logs: safe property access for image name/path (sentinel entries), thread name (most threads unnamed), and single-element array unwrapping (.Count) - Add SKILL.md efficiency guidance: resolve script path from skill directory (no find /), start with -ParseOnly, don't run broad filesystem searches - Reframe eval scenarios for platform-independent evaluation (parse/analyze instead of requiring macOS-only atos/dwarfdump), tighten rubric to test skill-specific knowledge (NuGet package name, all .NET binary images) Validation results (3 runs, claude-opus-4.6): Scenario 1 (parse frames): 3.3 → 4.0 (+0.7) ✅ Scenario 2 (investigate crash): 3.3 → 4.0 (+0.7) ✅ Scenario 3 (reject Android): 3.3 → 4.0 (+0.7) ✅ Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Auto-fallback to parse-only output when atos is unavailable On Linux (CI), atos and xcrun don't exist. Previously the script would error out after completing all parsing, losing the results. Now it detects the missing tool and falls back to ParseOnly output automatically, ensuring the agent always gets structured parse data regardless of platform. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move apple-crash-symbolication to dotnet-diag plugin Relocate skill from plugins/dotnet to plugins/dotnet-diag and tests from tests/dotnet to tests/dotnet-diag. Update CODEOWNERS accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve apple-crash-symbolication: macOS symbols, bug fixes, training log - Fix JSON case-conflict parsing (vmRegionInfo/vmregioninfo duplicate keys) - Fix strict-mode safe access for optional asi field - Expand Step 4 with macOS-specific symbol package guidance (.symbols NuGet) - Add .dwarf to .dSYM conversion instructions - Add src/coreclr/ to validation paths - Soften stop signals to allow crash analysis and deeper investigation - Add macOS Symbol Packages and JSON Parsing Gotchas to reference doc - Create training log documenting session findings - Add .github/skills/ project-local copy for CLI testing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: automated version extraction and symbol acquisition Script improvements: - Preserve full image paths from crash log (previously discarded by GetFileName) - Add Get-RuntimeVersionFromPath: extracts .NET version from image paths (e.g., .../shared/Microsoft.NETCore.App/10.0.4/libcoreclr.dylib) - Add Get-RidFromPath: infers RID from path or crash metadata (OS/CPU) - Path-based version detection as fast primary method, UUID matching as fallback - Emit copy-pasteable symbol acquisition commands when dSYMs are missing - Show .NET version in ParseOnly library listing SKILL.md updates: - Step 2: document automated version detection and acquisition commands - Step 4: script now prints ready-to-run download/conversion commands Training log: record session 2 findings (5 issues, script + SKILL.md changes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: add symbol server anti-pattern dotnet-symbol and msdl.microsoft.com do not serve macOS dSYM/DWARF symbols — only Windows PDBs and Linux ELF debug info. NuGet packages are the only public source. Added anti-pattern to SKILL.md (both copies) and reference doc to prevent wasted tool calls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: fix symbol server guidance — dotnet-symbol works for macOS dotnet-symbol --symbols <binary> successfully downloads .dwarf debug symbols for macOS Mach-O binaries from msdl.microsoft.com. Previous commit incorrectly claimed this didn't work. Replaced anti-pattern with positive guidance in SKILL.md (both copies) and reference doc. Added training log entry documenting the correction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Promote dotnet-symbol as preferred macOS symbol acquisition method - Script: macOS acquisition block now shows Option A (dotnet-symbol, preferred) and Option B (.symbols NuGet, fallback). Fixed .dwarf filename doubling bug in cp command. - SKILL.md (both copies): Step 2 updated for dotnet-symbol preference, Step 4 reordered with dotnet-symbol as #2 and NuGet symbols as #3. - Reference doc: Restructured macOS Symbol Packages section with Preferred/Fallback subsections and shared .dwarf→.dSYM conversion. - Training log: Session 4 entry documenting the promotion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add automated symbol server download for macOS crash symbolication - Add Get-DebugSymbols function: downloads .dwarf files from Microsoft symbol server using Mach-O UUID (mirrors android tombstone approach) - Add Convert-DwarfToDsym function: creates .dSYM bundle from .dwarf with UUID verification via dwarfdump - New params: -SymbolCacheDir, -SymbolServerUrl, -SkipSymbolDownload - Wire download+conversion into main flow after local dSYM search - Refactor manual acquisition guidance as fallback-only - Update both SKILL.md copies: frontmatter, Step 4, new flags - Update ips-crash-format.md: automated download as primary method - Add training log session 5 URL pattern: https://msdl.microsoft.com/download/symbols/_.dwarf/mach-uuid-sym-{UUID}/_.dwarf Verified end-to-end: 391/391 .NET frames symbolicated with clean cache. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: add triage order guidance to Step 3 Step 3 now instructs the agent to explain the faulting mechanism (frames #0-#1) before examining cross-thread context. This addresses a misdiagnosis where GC activity on neighboring threads was mistaken for causation when the actual root cause (_sigtramp -> NULL signal handler) was visible in the crashing thread's first two frames. Training log updated with session retrospective and corrected the original crash description from GC race to NULL signal handler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove stale .github/skills/apple-crash-symbolication remnant Skill lives in plugins/dotnet-diag/skills/apple-crash-symbolication/ since the plugin restructuring. The old .github/skills/ copy was a leftover. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add libimobiledevice.org to known-domains.txt The reference scanner CI check fails because the apple-crash-symbolication skill references https://libimobiledevice.org/ which is not in the allowed domains list. This domain hosts the libimobiledevice project, a community library for communicating with iOS devices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Delete eng/reference-scanner/known-domains.txt * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add libimobiledevice.org to known-domains allowlist The apple-crash-symbolication SKILL.md references libimobiledevice.org for the idevicecrashreport tool. Add the domain to the known-domains file to fix the skill-check CI reference validation failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restructure apple-crash-symbolication for analysis-first approach The skill was getting ❌ verdicts on all eval scenarios because it directed the LLM to 'run the script' instead of teaching crash analysis domain knowledge. This mirrors the problem where quality decreased from 3.0 to 2.7 in the parse scenario. Restructure to follow the Android sibling's proven pattern: - Lead with parsing (.ips two-part JSON format, key fields) - Teach .NET library identification (inline library table) - Teach crash interpretation (asi, faulting thread, exception) - Teach atos command construction (concrete examples) - Teach dSYM search paths (ordered list with commands) - Move automation script to optional section at the end - Move crash log retrieval to a separate section (not Step 1) Trim the reference doc to avoid duplication, keeping only macOS symbol distribution differences and supported RID list. Token budget: ~2,100 tokens (within 800-2,500 optimal range). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refine prompts and assertions in eval.yaml for iOS crash symbolication scenarios * Add format verification guard and fix overfitting in reject scenario - SKILL.md: Add explicit format check at start of Step 1 to verify .ips JSON before proceeding; add 'Wrong file format' stop signal - eval.yaml: Remove direct skill reference from scenario 3 prompt to reduce overfitting (was triggering skill activation in plugin mode) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix scenario 3: provide actual Android tombstone file for rejection test - eval.yaml: Replace invalid 'extra_files' with correct 'files' syntax (extra_files was silently ignored, so crash_android.txt was never copied) - eval.yaml: Remove copy_test_files for scenario 3 to avoid copying ios_crash.ips which tempts the agent in plugin mode - Add tombstone_sample.txt to test directory (copy from android sibling) - SKILL.md: Mention ndk-stack/addr2line in wrong-format stop signal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update tests/dotnet-diag/apple-crash-symbolication/tombstone_sample.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedback: simulator RIDs, perf, safety, accuracy fixes - Add simulator RIDs (iossimulator, tvossimulator) to script search list - Build hashtable for O(1) image lookups in Get-ThreadFrames - Remove -UseBasicParsing (unnecessary in pwsh 7+) - Include UUID in Convert-DwarfToDsym cache path; sanitize library name - Make version regex greedy to capture pre-release suffixes - Improve format detection error message for non-.ips files - Guard xcrun fallback with Get-Command check - Add .dwarf-to-.dSYM conversion guidance for macOS manual fallback - Escape .ips in eval.yaml regex assertion - Fix UUID note in reference doc (normalize, not assume lowercase) - Fix symbol server example to use crash-log image name Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 2: case-sensitive replace, dedup, path fix, redact PII - Use -creplace with guard for vmregioninfo duplicate key handling - Use Sort-Object -Unique for proper library deduplication - Fix macOS fallback to create .dSYM bundles under symbols-out/ - Redact device identifiers in test fixture .ips file Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Prevent dSYM cache poisoning: validate UUID on cache hit, remove on mismatch - Convert-DwarfToDsym now verifies cached dSYM UUID before reusing - On UUID mismatch during download, remove the bad cached bundle so subsequent runs can retry cleanly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cache validation: normalize UUID comparison, clean up dwarf + bundle on mismatch - Use Format-Uuid to normalize dwarfdump output before comparing to already-normalized \ in Convert-DwarfToDsym cache check - On UUID mismatch after download, remove the entire .dSYM bundle directory and the cached .dwarf file to prevent repeat failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resolve-Frames null handling: use List[object] to preserve null entries PowerShell array += \ silently drops the element, breaking index alignment between results and input addresses. Switch to List[object].Add() which correctly preserves null entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dSYM bundle root traversal, qualify version-from-path note - Walk up directory tree until *.dSYM is found instead of going up only 2 levels (which lands at Contents/Resources, not the bundle) - SKILL.md: note that version-in-path only works on macOS shared framework installs; iOS paths don't embed the runtime version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove incorrect .github/skills path reference from training log The .github/skills/ directory doesn't exist in this repo. The file was already listed under its actual plugins/dotnet-diag/ path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Detect simulator for RID inference in manual fallback guidance Detect CoreSimulator in image paths to use iossimulator-/tvossimulator- RIDs for simulator crashes, avoiding UUID mismatches from wrong runtime packs. Also handle arm64e CPU type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resolve-Frames return type and scope issue in simulator detection - Return \.ToArray() instead of ,\ to avoid wrapping the list in a single-element array - Use \.usedImages instead of undefined \ in RID inference block Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify ParseOnly .NET Libraries section shows only frame-relevant images Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix YAML escape in assertion pattern (use single quotes for regex backslash) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten stop signal to prevent Android symbolication spillover The apple-crash-symbolication skill's stop signal for wrong file formats was naming specific Android tools (ndk-stack, addr2line) and the android-tombstone-symbolication skill. This caused models to learn about Android symbolication from the Apple skill and then actually execute it, resulting in 4-5x token bloat and completion regression on the Android rejection eval scenario. Remove tool suggestions from the stop signal — just say 'stop, don't symbolicate.' The Android skill handles its own routing when loaded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify scenario 3 rubric to focus on positive skill knowledge Replace 4 rubric items (including 2 'Did NOT' items) with 2 items that test positive knowledge the skill provides. The 'Did NOT' items gave credit to baseline responses that also don't attempt iOS workflow (because they don't know about it), minimizing the quality delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix nuget.org domain reference: drop www. prefix to match known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge upstream main and use api.nuget.org v3 endpoint Main changed known-domains.txt to use path-scoped entries (nuget.org/account/trustedpublishing instead of bare nuget.org). Switch the symbols download URL to the v3 flatcontainer endpoint on api.nuget.org which is in the allowlist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .gitignore merge conflict Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Steve Pfister <stpfiste@microsoft.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |
||
|
Amaury Levé
|
93dc33da86 |
Stabilize and unify some test skills (#501)
* Deduplicate test skill references and clarify skill boundaries - Move platform-detection.md and filter-syntax.md to plugins/dotnet-test/shared/, removing 3 identical copies of each from run-tests, mtp-hot-reload, and migrate-vstest-to-mtp reference directories. - Move dotnet.md from exp-test-smell-detection/extensions/ to shared/ as dotnet-test-frameworks.md in both dotnet-test and dotnet-experimental plugins. Update exp-assertion-quality, exp-test-boilerplate-detection, exp-test-tagging, and test-anti-patterns to reference the shared file instead of inlining framework detection tables. - Differentiate test-anti-patterns (quick pragmatic review) from exp-test-smell-detection (deep formal audit with academic taxonomy) by updating descriptions and cross-referencing each other in When Not to Use sections. - Update skill-validator to allow ../../shared/ file references while still blocking other parent-directory traversals. Add tests for the new rule. * Switch from shared/ directories to hidden reference skills Replace the plugin-level shared/ directories with non-invocable reference skills (user-invocable: false) that other skills reference by name. - Create platform-detection, filter-syntax, and dotnet-test-frameworks as hidden skills under plugins/dotnet-test/skills/. These contain the detection tables and syntax references previously duplicated across run-tests, mtp-hot-reload, and migrate-vstest-to-mtp. - Create exp-dotnet-test-frameworks as a hidden skill under plugins/dotnet-experimental/skills/ for the experimental test analysis skills (exp-test-smell-detection, exp-assertion-quality, etc.). - Update all consuming skills to reference these by skill name in backtick notation instead of file links. - Revert the skill-validator ../../shared/ exception — no longer needed since all references now use the standard skill name mechanism. * Merge exp-test-boilerplate-detection into exp-test-maintainability exp-test-maintainability was only 6 calibration rules with no workflow. exp-test-boilerplate-detection had the full 5-category detection workflow, examples, calibration, and validation. Merge the boilerplate content into exp-test-maintainability (the broader, more user-facing name) and add the two unique maintainability rules (DisplayName guidance, DataRow vs DynamicData preference) to Category 3. - Replace exp-test-maintainability SKILL.md with the merged content - Move test fixtures from exp-test-boilerplate-detection to exp-test-maintainability - Merge eval.yaml scenarios (4 total: 2 from each original skill) - Delete exp-test-boilerplate-detection skill and tests - Update all cross-references in exp-test-smell-detection, dotnet-test-frameworks, exp-dotnet-test-frameworks, and CODEOWNERS * Add cross-references to test-anti-patterns for deep mock and duplication analysis Point users to exp-mock-usage-analysis from the Over-mocking entry and to exp-test-maintainability from the Duplicate tests entry. * Add exp-dotnet-test-frameworks to CODEOWNERS * Improve run-tests SDK 10 MTP detection for blame-hang scenario Inline the critical SDK 10 detection signal (global.json test.runner) directly in Step 1 instead of deferring entirely to the platform-detection skill. This makes the distinction between SDK 10 (no -- separator) and SDK 8/9 (requires -- separator) more prominent. Add a quick detection summary table, strengthen the Common Pitfalls entry for SDK 10 with a blame-hang-timeout example, and keep the platform-detection skill reference for the full detection logic. * Improve skill activation keywords in descriptions - exp-test-maintainability: Add 'suggest a better test structure', 'consolidate similar test methods', 'convert copy-paste tests to data-driven parameterized tests' to match prompts like 'each new case needs a whole new method, suggest a better structure'. - test-anti-patterns: Add 'review tests', 'find test problems', 'check test quality', 'audit tests for common mistakes' to match review-style prompts that don't use the word 'anti-pattern'. - run-tests: Add 'hang timeout', 'blame-hang', 'blame-crash', 'TUnit' to match SDK 10 blame scenarios and TUnit filter scenarios that were intermittently not activating. |
||
|
Cathy Sullivan
|
42e29303b4 |
Add coverage-analysis skill to dotnet-test plugin (#499)
* Add coverage-analysis skill to dotnet-test plugin Adds an automated, project-wide code coverage and CRAP score analysis skill for .NET projects. Complements the existing crap-score skill (which is manual/targeted for single methods) with an end-to-end pipeline: - Auto-detects solution structure and test projects - Detects coverage provider (Microsoft.Testing.Extensions.CodeCoverage or Coverlet) per test project and selects the correct dotnet test arguments - Runs coverage collection, generates reports via ReportGenerator - Calculates CRAP scores per method via bundled PowerShell scripts - Surfaces risk hotspots ranked by danger (complex + uncovered code) - Produces actionable recommendations with projected score improvements Boundary with crap-score: coverage-analysis is project-wide and automated; crap-score is targeted and LLM-directed. The frontmatter descriptions cross-reference each other so the skill router disambiguates correctly. Includes eval.yaml with 3 scenarios (existing data analysis, from-scratch coverage collection, coverage plateau diagnosis) and shared buildable fixtures. Fixes #457 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR review: coverage output path and scoped tool install - Add --coverage-output $rawDir to all MS CodeCoverage dotnet test commands (unified, .NET 9, .NET 10+, and mixed-project modes) so Cobertura XML lands in the expected directory for downstream search - Replace dotnet tool install --global with --tool-path scoped to the coverage output directory to avoid mutating the user environment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Cobertura XML fixture line numbers to match StudentService.cs FindById starts at line 101 (was 88), Search starts at line 109 (was 96). Updated both partial-coverage and plateau fixtures. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Wrap Get-ChildItem pipelines in @() for single-result array safety PowerShell returns a scalar FileInfo (not array) when a pipeline yields exactly one result, making .Count return null. Wrapping in @() ensures the result is always an array so .Count works reliably for the common single-test-project case. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove CODEOWNERS change - not needed for this PR The existing /plugins/dotnet-test/ and /tests/dotnet-test/ wildcards already route reviews to @dotnet/dotnet-testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: .csproj fallback, cross-platform paths, .sln header - Add .csproj fallback discovery in Step 1 when no .sln found (emit ENTRY_TYPE:Project) to match documented behavior - Fix ProjectReference backslash to forward slash in test fixture .csproj - Remove leading blank line from ContosoUniversity.sln fixture Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add plateau/stuck trigger language to frontmatter description The skill was NOT ACTIVATED for the 'Coverage plateau diagnosis' eval scenario because the frontmatter description (used by the skill router) lacked plateau-related keywords. The prompt 'coverage is stuck' and 'what's blocking me' didn't match the existing description text. Add 'stuck', 'plateaued', and 'blocking improvement' to the frontmatter Use-when sentence so the skill router activates for plateau diagnosis prompts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make CRAP scores mandatory, add existing-data shortcut path The plateau eval scenario scored 1/5 on CRAP scores across all modes because the LLM shortcut the workflow when existing Cobertura XML was provided — it read the XML directly and gave a quick diagnostic answer without computing CRAP scores or following the output template. - Add existing-data guidance at top of Workflow section: skip test execution but still run CRAP score computation (Steps 5-6) - Add mandatory CRAP rule to guidelines.md: Risk Hotspots table is required in every output regardless of analysis mode Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-test/skills/coverage-analysis/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Mukund Raghav Sharma (Moko)
|
665c39e957 |
Add configuring-opentelemetry-dotnet skill (#268)
* Add configuring-opentelemetry-dotnet skill (+8.1% eval, near-miss) Teaches OpenTelemetry SDK configuration in ASP.NET Core: package selection, AddOpenTelemetry with tracing/metrics/logging, custom ActivitySource and Meter creation, OTLP exporter setup, and context propagation. Eval results: +8.1% improvement (threshold: 10%, near-miss) Includes eval.yaml with OTel setup scenario + negative test. * Sharpen eval.yaml: add custom-spans-invisible pain-point and name-matching rubric emphasis * Migrate configuring-opentelemetry-dotnet to plugins/ directory structure * Move OpenTelemetry skill to new dotnet-aspnet plugin, address review feedback - Move skill from plugins/dotnet-diag to new plugins/dotnet-aspnet plugin - Create dotnet-aspnet plugin.json - Fix SKILL.md frontmatter: remove code fence wrapper so YAML metadata parses - Fix undefined 'order' variable in ProcessOrderAsync example - Move SqlClient/Runtime instrumentation to clearly marked optional sections - Add missing using directives (System.Diagnostics, OpenTelemetry.Context.Propagation) - Handle null Activity.Current in context propagation snippet - Use IMeterFactory via DI instead of static Meter per official guidance - Add OpenTelemetry.Instrumentation.Http to eval rubric package list - Simplify eval prompt to be more natural/generalized - Update CODEOWNERS: add dotnet-aspnet plugin entries, remove old dotnet-diag entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Configure metrics OTLP exporter endpoint explicitly to match tracing config Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address round 2 review: logging endpoint, marketplace, RecordException using, ActivitySource in propagation, HttpClient pitfall - Configure logging OTLP exporter endpoint explicitly for consistency - Add dotnet-aspnet to both marketplace.json files - Add 'using OpenTelemetry.Trace' for RecordException extension method - Declare ActivitySource in context propagation snippet - Fix HttpClient instrumentation pitfall (works with new HttpClient() too) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify logging package source and add ActivitySource registration reminder Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add @dotnet/aspnet team to skill CODEOWNERS entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dotnet-aspnet to README table, make propagation ActivitySource static Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add expect_activation:false, IMeterFactory using, clarify OTLP exporter scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OtlpExporter using directive, clarify OTLP package covers logging Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Prometheus from OTLP claim, add OTLP exporter assertion to eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify Jaeger is traces-only, document logging package provenance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval: increase timeout, reject tools, rewrite prompt as explanation-only - Scenario 1: rewrite prompt to ask for code explanation (not file modification), add reject_tools to prevent bash/edit overhead, increase timeout 120s->360s, update rubric wording - Scenario 2: add reject_tools to eliminate tool-overhead noise Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve OTel skill evals and address review feedback - Remove redundant per-skill CODEOWNERS lines (ViktorHofer feedback) - Add SetResourceBuilder + ParseStateValues to SKILL.md Step 3 (logging provider gap) - Rewrite eval scenarios: 2 targeted positive scenarios testing real skill gaps - Remove noisy negative activation scenario (high variance, -21.5% drag) - Eval results: +14.2% improvement (threshold: 10%), overfitting 0.10 (low) Scenario 1: 3.0/5 -> 5.0/5, Scenario 2: 3.8/5 -> 5.0/5 * Add propagate-trace-context scenario to OTel evals (+25.6%) Third scenario tests trace context propagation across message queues, where the base model scores 3.2/5 and skill-enhanced scores 5.0/5. Removed metrics and debug scenarios that baseline already aced (5.0/5) which only added token cost without measurable improvement. Final: 3 scenarios, +25.6% improvement, overfitting 0.09 (low) * Remove duplicate dotnet-aspnet CODEOWNERS entries (L69-70 shadowed by L86-87) * Use @dotnet/aspnet team alias in CODEOWNERS for wider reach * Addressed feedback --------- Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
David Ortinau
|
c4b6819dda |
Add 7 core .NET MAUI development skills to dotnet-maui plugin (#456)
* Add 7 core .NET MAUI development skills to dotnet-maui plugin Add maui-shell-navigation, maui-collectionview, maui-data-binding, maui-dependency-injection, maui-app-lifecycle, maui-safe-area, and maui-theming skills with eval.yaml tests and reference docs. Update plugin.json, marketplace.json (both copies), and CODEOWNERS. Resolves #455 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cross-skill references to non-existent skills Replace 10 dangling references to skills not in this repo (maui-deep-linking, maui-platform-invoke, maui-unit-testing, maui-maps, maui-animations, maui-accessibility, maui-app-icons-splash, maui-bootstrap-theme, maui-localization) with plain-English guidance or references to skills that exist in this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix code correctness bugs and remaining cross-references Address PR review feedback and skeptic review findings: Code correctness: - Fix BackgroundColor="Primary" (XAML crash) → use {StaticResource Primary} - Fix MergedDictionaries.Clear() (nukes all styles) → remove-then-add pattern - Fix HorizontalStackLayout Fill (no effect) → use Grid with ColumnDefinitions - Fix AncestorType with ViewModel type → use ContentPage (visual element) - Fix Frame usage → replace with Border (recommended control) - Remove redundant MultiBinding StringFormat="{}{0}" - Fix OnStart reference → use CreateWindow for theme restore - Soften lifecycle OnStart/OnSleep characterization (compat shims, not absent) - Qualify ResourceDictionary code-behind requirement (only for x:Class) Cross-references: - Remove all remaining body-text refs to non-existent skills - Replace with Microsoft Learn doc links or generic guidance Eval: - Relax SafeAreaRegions assertion to accept SafeAreaEdges or SafeAreaRegions - Relax theming rubric for ResourceDictionary code-behind Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore Clear() theme pattern with clarifying comment The Clear()+Add() pattern is intentional — theme dictionaries are designed to be the only merged dictionaries in this architecture. Added comment explaining the assumption and what to do if non-theme dictionaries exist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix second-pass review findings: API accuracy and eval robustness Code correctness: - ScrollView SafeAreaEdges="All" → "Container" (ScrollView only honors Container and None; All silently fails for keyboard avoidance) - DI CreateWindow example: add comment noting AppShell must be registered - AddHttpClient: add comment noting Microsoft.Extensions.Http NuGet required - MultiBinding: remove redundant StringFormat from reference doc (matches SKILL.md) - RelativeSource Self table: Path=Width → Path=WidthRequest (Width is read-only -1 before layout; WidthRequest is the correct bindable property) - CollectionView StackLayout warning: clarify it refers to BindableLayout on StackLayout vs CollectionView ItemsLayout types - SafeAreaEdges: fix "attached property" → "property" (it's a regular property) - ios:NavigationPage example: add xmlns comment for ios namespace Eval assertions: - DI Scoped regex: accept "doesn't"/"lacks"/"without" not just "no" - CollectionView ViewCell: match negation in either direction around ViewCell - Lifecycle events: require 3+ distinct events instead of any single one Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix final PR review comments - safe-area eval: tighten Container assertion to require SafeAreaEdges context - DI: merge #else fallback into single compilable #if block - lifecycle: behaviour → behavior (American English consistency) - theming: macOS Catalyst minimum 10.14 → 10.15 (Catalina required) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |