mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-sonnet46
14 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aditya Mandaleeka
|
52ba152ed6 | Update to Vally 0.7.0 (#854) | ||
|
Daniel Plaisted
|
8a437b1d4f |
Add anti-pattern for forking a project instance via <MSBuild> with path-neutral global properties (#770)
Adds msbuild-antipatterns AP-22 (in references/additional-antipatterns.md): using
the <MSBuild> task to build/publish a project with an extra global property that
does not change the output path forks a distinct instance sharing the same
OutputPath/IntermediateOutputPath as the instance the graph already builds,
causing duplicate builds and parallel-write races on PDBs, *.sourcelink, and
project.assets.json. Covers both shapes:
(a) a project re-invoking itself (publish-on-build), and
(b) project A invoking Build/Publish on another project B it consumes
(e.g. a test/layout project publishing a tool) — the more common trigger.
GOOD fixes: (a) static _IsPublishing + DependsOnTargets with a cycle guard;
(b) let the producer publish in its own build and have the consumer sequence it
with a non-output ProjectReference instead of re-publishing it. Also updates the
SKILL.md pointer (AP-16 through AP-22) and the quick-reference checklist.
Extends check-bin-obj-clash with the matching cause (self or cross-project
<MSBuild> Build/Publish with a path-neutral global property), binlog-based
detection via the evaluations/global-properties/double_writes tools, the two
fixes, and an _IsPublishing row in the global-properties table.
Adds eval coverage: a publish-on-build smell in the msbuild-antipatterns review
fixture (LibA.csproj) plus a matching rubric line in eval.yaml and eval.vally.yaml.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Jan Krivanek
|
14ee3c90f9 |
Rework evaluation to allow sharding of resource intensive evaluations (#692)
* Shard plugin evaluation via executionShard tag in eval.yaml MCP-heavy plugins like dotnet-msbuild reliably exhaust the GitHub-hosted runner host after ~87 min of cumulative MCP traffic (Copilot CLI plus dotnet dnx Microsoft.AITools.BinlogMcp child processes), which has produced no dashboard data for dotnet-msbuild scheduled runs since 2026-05-20. Cutting per-runner concurrency alone was insufficient: at the default 18-way concurrency the runner OOMs in 17-30 min; at 3-way it survives longer but still hits a host-level termination at ~87 min. Two mitigations, both required: 1. Sharding (this change). The discover step reads an optional top-level 'executionShard:' string from each tests/<plugin>/<skill>/eval.yaml and groups skills by tag, emitting one matrix entry per shard so each shard runs on a fresh runner. Untagged skills fall into a 'default' bucket. Plugins where all skills resolve to a single bucket emit one entry (unchanged behavior). The validator already calls IgnoreUnmatchedProperties so no Models.cs change is needed. Shard names follow '<plugin>--shard-<tag>' so the existing 'skill-validator-results-<plugin>--*' artifact glob keeps working. 2. Per-MCP-plugin concurrency caps (already in place). When plugin.json declares mcpServers, parallelism is forced to 1x3x1 with --judge-timeout 600 to avoid acute OOM within a shard. Also adds a workflow_dispatch 'plugin' input for targeted re-runs of a single plugin, and tags 10 dotnet-msbuild eval.yamls (heavy/medium); the remaining 8 fall into 'default'. Empirically the three shards complete in parallel in ~30-40 min each, well inside the 180-min job timeout, and successfully upload dashboard data. * Address PR review: lenient shard regex and validate dispatch plugin input - executionShard regex now tolerates optional leading whitespace, so an accidentally indented key still groups correctly instead of silently falling back to the default bucket. - workflow_dispatch 'plugin' input is now validated against ^[a-zA-Z0-9._-]+$ before any Join-Path/Test-Path use, preventing path-traversal style values (e.g. '../foo') from being injected into filesystem paths. |
||
|
YuliiaKovalova
|
5f1170152b |
Add MSBuild target authoring skills (target-authoring, property-patterns, item-management, extension-points) (#669)
* Add 4 MSBuild target authoring skills for dotnet-msbuild plugin New skills: - target-authoring: three-level target chain, DependsOn extension, naming conventions - property-patterns: conditional defaults, composition, path normalization, TFM helpers - item-management: Include/Remove/Update, batching, transforms, FileWrites registration - extension-points: CustomBefore/After hooks, import gating, NuGet build extensions Each skill includes eval.yaml tests with anti-pattern fixtures. Closes #668 * Address Copilot review feedback - target-authoring: Reword DO NOT USE clause to exclude only deep incremental-build diagnostics, not basic Inputs/Outputs usage - property-patterns: Close unclosed XML elements in String Functions snippet (PropertyGroup, TargetFrameworkMoniker) - extension-points: Add missing CustomAfterMySDK property definition to match the import at bottom of example * Remove accidental pr-body.md file * Fix wildcard import placeholder path and quote NormalizePath args - extension-points: Use full MSBuildExtensionsPath expression and consistent Exists() casing in wildcard import example - property-patterns: Quote property arguments in NormalizePath call * Tune eval prompts to reduce overfitting Rewrite all 4 eval prompts as natural developer problem descriptions instead of skill-aligned checklists. Softens technique-prescriptive rubric items. Result: overfitting scores drop from 0.15-0.31 to 0.06-0.08 (all green). property-patterns now passes eval. * Fix Copilot review round 3: quote property function args, fix fixture - property-patterns: Quote in IsPathRooted call - extension-points: Quote in GetDirectoryNameOfFileAbove - item-management fixture: Use literal semicolon instead of %3B in WriteLinesToFile * Add harder multi-file eval scenarios for all 4 skills Each skill now has a second scenario with multi-file setups containing interacting bugs that require cross-file analysis: - target-authoring: SDK .targets + Directory.Build.targets with 3 bugs (target redefinition, fragile BeforeTargets, duplicate DependsOn chain) - property-patterns: nested Directory.Build.props with 5 bugs (import order, unconditional overwrite, unquoted condition, missing trailing slash, NoWarn overwrite) - item-management: csproj with 5 interacting bugs (Include vs Update, cross-product batching, eval-time FileWrites, too-broad Remove glob) - extension-points: Directory.Build.props/targets + NuGet package with 6 bugs (inverted guard, late ImportByWildcard, CustomBefore overwrite, missing Exists guard, target name collision, internal target hook) Timeouts increased to 180s for all scenarios. * Fix review round 4: revert unintended README change, quote intrinsic call in condition * Improve eval quality for 4 new MSBuild authoring skills Eval improvements across target-authoring, property-patterns, item-management, and extension-points skills: - Rewrite prompts to describe observable symptoms without naming MSBuild concepts, inspired by real dotnet/msbuild and dotnet/sdk issues (msbuild#2470, #12894, #13056, #4109, sdk#43908) - Broaden assertion regex patterns to test for diagnostic outcomes rather than skill-specific vocabulary - Improve rubric items to test understanding and diagnosis flow (diagnosis -> root cause -> fix) instead of skill terminology - Add task-completion 'fix it' scenario to each skill with file_contains assertions that verify the agent applied correct patterns (CompileDependsOn append, FileWrites, Exists guard, etc.) - Remove leading 'Anti-pattern:', 'BUG N:' comments from fixture files that were giving away answers to the baseline agent, making scenarios more realistic and better at measuring skill value Each skill now has 3 scenarios: basic review, hard multi-file review, and a task-completion fix scenario. Total: 12 scenarios across 4 skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix review comments: quote TFM intrinsic, fix PrivateAssets example - property-patterns SKILL.md: Quote the GetTargetFrameworkIdentifier intrinsic call result in the Condition to match the 'always quote both sides' guidance and avoid brittle parse when empty - item-management SKILL.md: Replace PackageReference Update on Microsoft.NETCore.App (not a real user-added item) with Include on Microsoft.CodeAnalysis.NetAnalyzers (a concrete analyzer package) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval activation and timeouts based on eval run results - Add MSBuild domain terms to prompts (e.g., 'MSBuild targets', 'MSBuild property patterns', 'MSBuild item groups', 'MSBuild extension points') to improve skill activation rates — skills were NOT ACTIVATED in plugin mode due to prompts being too symptom-focused without domain keywords - Increase timeouts: review scenarios 180s -> 240s, fix-it scenarios 180s -> 300s to avoid timeout-related scoring penalties - Keep prompts symptom-driven but include enough domain context for the skill router to activate the correct skill Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PR review comments and improve eval quality Review comment fixes: - TargetAuthoring.csproj: Replace %22/%3B URL encoding with XML escaping ("/;) so generated C# code is valid - target-authoring eval: Fix prompt referencing 'schema files' when fixture has endpoint definitions — now says 'endpoint list' - property-patterns eval: Reword LangVersion rubric — unconditional assignment in .props IS overridable by csproj but prevents override from earlier imports and command-line properties - extension-points eval: Add stronger activation terms to Fix scenario prompt (explicit mention of CustomBeforeMicrosoftCommonTargets) Eval quality improvements based on CI results: - Shorten target-authoring rubric items to reduce overfitting (was 0.30) - Remove 'Focus on target authoring patterns' instruction from hard scenario prompt (was directing agent too specifically) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve eval: increase timeout, add activation keywords, explicit fix instructions - property-patterns 'Diagnose multi-level': timeout 240s -> 360s (was hitting limit at 249s) - target-authoring 'Diagnose custom target': add CompileDependsOn/incremental build keywords for better skill activation - property-patterns 'Fix': add 'edit ... directly' to prompt to encourage file edits - target-authoring 'Fix': add 'edit ... directly' to prompt - item-management 'Fix': add 'edit ... directly' to prompt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix target-authoring skill description and eval prompts for plugin activation The skill description said 'DO NOT USE FOR: deep incremental-build diagnostics' which caused the plugin agent to skip it for scenarios about build regressions. Clarified that the skill IS for diagnosing target authoring mistakes (missing Inputs/Outputs, broken dependency chains) that cause full rebuilds. Also updated eval prompts to use 'target authoring mistakes' framing and increased Fix scenario timeout to 360s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Increase eval timeouts: extension-points NuGet (240->360s), item-management cascading (240->360s) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 3 fresh PR review comments + update eval prompts for target-authoring - property-patterns/SKILL.md: add warning that TargetFramework-conditioned PropertyGroups must be in .targets not .props (empty in single-target .props) - extension-points/SKILL.md: add Exists() guard to GetPathOfFileAbove Import to prevent build failure when no parent file is found - target-authoring/SKILL.md: fix comment mismatch — validation target runs via dependency chain, not BeforeTargets; update comment accordingly - target-authoring/eval.yaml: reframe 'Diagnose' and 'Fix' prompts to use 'dependency chain' / 'authoring patterns' language (avoids routing to incremental-build skill which also claims Inputs/Outputs diagnostics) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve skill descriptions for reliable plugin-mode activation Add 'Only activate in MSBuild/.NET build context.' prefix to all 4 skills for consistency with existing skills in the plugin. Add explicit 'diagnosing and fixing' and 'reviewing' keywords to USE FOR sections so the SDK selects these skills over the broader msbuild-antipatterns skill when prompts ask to fix or review specific item/extension/property/target patterns. Add 'general MSBuild anti-pattern catalog (use msbuild-antipatterns)' to DO NOT USE FOR sections to help the SDK disambiguate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix skill descriptions: shorten to under 1024-char SDK limit All 4 new skills had descriptions exceeding the 1024-character maximum enforced by the skill-validator check command. The Copilot SDK silently ignores skills with over-length descriptions, causing 0% activation in both isolated and plugin evaluation modes. Shortened all 4 descriptions while preserving key activation keywords (USE FOR / DO NOT USE FOR / 'Only activate in MSBuild/.NET build context'). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments - target-authoring eval.yaml: fix prompt to say 'Outputs attribute' (not 'Returns') since the fixture uses Outputs on a query target, which is the actual bug that causes MSBuild to skip re-execution - property-patterns eval.yaml: reword prompt and rubric — the real issue is unconditional assignment breaking parent-child Directory.Build.props inheritance, not project files being unable to override properties - extension-points SKILL.md: fix GetPathOfFileAbove example to factor the path into a property so Project= and Condition= use the same value consistently (avoiding the ..\\ vs ..\ discrepancy) - CustomSdk.targets: add WriteLinesToFile to CoreCodeGen so it actually creates the .g.cs output files (prevents compilation failure in fixture) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix property-patterns fixtures to match rubric claims Scenario 1 (path merge bug): - PropertyPatterns.csproj: add OutputPath using CustomOutputDir without a trailing separator, making the merge bug observable (artifacts\binPropertyPatterns\ instead of artifacts\bin\PropertyPatterns\) Scenario 2 (multi-level hierarchy bugs): - hard/Directory.Build.props: make LangVersion unconditional so it overwrites the child src/Directory.Build.props LangVersion=preview, matching the rubric claim about parent unconditional assignments - hard/Directory.Build.props: add NoWarn=NU1702 (conditional default) so parent suppressions are visible but lost when the child's unconditional <NoWarn>CS1591;IDE0005</NoWarn> overwrites them after import Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
dfc8aa786d |
dotnet-msbuild: register AITools.BinlogMcp and use it from binlog-failure-analysis (#673)
* dotnet-msbuild: register AITools.BinlogMcp and use it from binlog-failure-analysis Adds the AITools.BinlogMcp MCP server (pulled via dnx from the dotnet-eng public NuGet feed, prerelease, no version pin) to the dotnet-msbuild plugin and rewrites the binlog-failure-analysis SKILL.md to lead with the MCP tools (binlog_overview, binlog_errors, binlog_search, binlog_items, binlog_properties, binlog_imports, binlog_nuget). The text-log replay workflow is kept as a fallback for environments without the MCP. * Improve skills * Reflect PR comments |
||
|
Aditya Mandaleeka
|
9433a31e67 | Initial port of evals to Vally (#615) | ||
|
Copilot
|
2f9922e073 |
Remove LangVersion=latest from skill examples and test fixtures (#608)
* Initial plan * Remove LangVersion=latest from skill docs and test fixtures Per official C# docs, LangVersion should not be set to `latest` as it causes builds to silently vary across machines with different SDKs. Best practice is to omit LangVersion entirely and let the TFM determine the correct C# version automatically. - Update msbuild-modernization/SKILL.md: remove LangVersion=latest from example and add explicit guidance to omit LangVersion - Update directory-build-organization/SKILL.md: remove from example - Update directory-build-organization/references/multi-level-examples.md: remove from all 4 examples - Update msbuild-antipatterns/SKILL.md: remove from AP-08 examples - Remove LangVersion=latest from 11 test fixture project files Agent-Logs-Url: https://github.com/dotnet/skills/sessions/7a3d2722-770b-4010-a6e8-70c92668ad4c Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Fix LangVersion guidance and stale eval rubric reference Agent-Logs-Url: https://github.com/dotnet/skills/sessions/e89afcd8-7227-424b-9c8c-9ec0ce7113c2 Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Tomas Grosup
|
a55227f030 |
Fold F# .fsproj ordering into msbuild-antipatterns AP-05 (#577)
* Add fsharp-project-structure skill for .fsproj file ordering and .fsi files Teaches agents F# compilation order rules, signature file placement, and common FS0039/FS0010/FS0034 fixes. Includes eval with three scenarios (add module, fix broken order, add signature file) and test fixtures. Validated +64.6% improvement on gpt-4.1 (significant, CI [+17.2%, +63.0%]). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add fsharp-project-structure skill for .fsproj file ordering and .fsi files Teaches agents F# compilation order rules, signature file placement, and common FS0039/FS0010/FS0034 fixes. Includes eval with three scenarios (add module, fix broken order, add signature file) and test fixtures. Validated +64.6% improvement on gpt-4.1 (significant, CI [+17.2%, +63.0%]). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/fsharp-project-structure/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fold F# project-structure into msbuild-antipatterns AP-05 Scraps the standalone fsharp-project-structure skill and instead adds a short F# exception to AP-05 (Manual File Listing in SDK-Style Projects): F# compilation is order-dependent, so .fsproj files must list Compile items in dependency order (utilities first, entry point last), and .fsi signature files must appear immediately before their companion .fs. The three F# eval scenarios (add module, fix broken order, add signature file) are ported into tests/dotnet-msbuild/msbuild-antipatterns/eval.yaml with fixtures isolated under a fsharp/ subfolder (with empty Directory.Build.{props,targets} to prevent the existing C#-antipattern fixture files from leaking into the F# builds). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
c94bc5cdc3 |
Add Ability to author agent evals (#436)
* Add ability to author agent evals * Adjust |
||
|
Jan Krivanek
|
101a151e98 | Improve the build-perf-diag scenario test (#404) | ||
|
YuliiaKovalova
|
14346ff3b2 |
Enrich build-perf-diagnostics skill with MSBuild team and community insights (#342)
* Enrich build-perf-diagnostics skill with MSBuild team and community insights Additions (pure knowledge, no MCP dependency): - RAR: dotnet/msbuild#2015 context, DisableTransitiveProjectReferences, ReferenceOutputAssembly - Copy tasks: SkipCopyUnchangedFiles, --artifacts-path, Copy batching (dotnet/msbuild#12884) - Dev Drive (ReFS): copy-on-write + reduced Defender scans, OrchardCore 7257 Copy tasks benchmark - Evaluation: NuGetSdkResolver 180-400ms overhead (dotnet/msbuild#4025), EnableDefaultItems - NuGet: RestoreUseStaticGraphEvaluation (20s+ savings in large builds) - Project graph shape: wide vs deep (40% faster clean, 20% faster incremental) - MSBuild Server: MSBUILDUSESERVER=1 for CLI caching - Inline task overhead: 1s vs 3ms (RoslynCodeTaskFactory vs compiled) - Misleading ResolveProjectReferences time (dotnet/msbuild#3135) - Incrementality anti-patterns (dotnet/msbuild#13206) - Impact categorization (HIGH/MEDIUM/QUICK WINS) - Quick Wins: expanded to 15 items (from 7) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix minor issues from review comments - Fix missing closing XML tag for CreateHardLinksForCopyFilesToOutputDirectoryIfPossible in Quick Wins checklist - Clarify confusing Copy task batching wording per reviewer feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Trim skill to reduce token overhead Remove standalone sections (MSBuild Server, Inline Tasks, ResolveProjectReferences, Incrementality Anti-patterns) that add ~1400 extra tokens without benefiting the evaluated scenario. Consolidate Quick Wins checklist accordingly. These sections can be contributed as separate skills in follow-up PRs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add separate skills for MSBuild server, inline tasks, and ResolveProjectReferences Extract content removed from build-perf-diagnostics into standalone skills to keep each skill focused and avoid token budget bloat: - msbuild-server: MSBUILDUSESERVER=1 for CLI incremental caching - inline-task-overhead: RoslynCodeTaskFactory ~1s overhead vs ~3ms compiled - resolve-project-references: misleading Target Performance Summary time (dotnet/msbuild#3135), guides users to Task Performance Summary instead Incrementality anti-patterns content was not extracted as a separate skill because the existing incremental-build skill already covers that area. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add github.com/dotnet/msbuild to known-domains.txt Required by reference scanner for MSBuild issue links in build-perf-diagnostics and resolve-project-references skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace full dotnet/msbuild URLs with shorthand notation The reference scanner reads known-domains.txt from upstream main, not from the PR branch. Use shorthand 'dotnet/msbuild#NNNN' format (consistent with the rest of the file) instead of full URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments and fix reference scanner errors - Remove https://aka.ms/devdrive URLs that trigger reference scanner - Soften unsourced benchmark claims (OrchardCore Copy tasks, 20s+ restore savings, 40%/20% graph shape improvements) per reviewer feedback to use workload-dependent language instead Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add eval tests for new skills; remove inline-task-overhead Add eval.yaml and test fixtures for msbuild-server and resolve-project-references skills: - msbuild-server: tests CLI build perf scenario with developer report - resolve-project-references: tests misleading target perf summary with multi-project setup and perf report Remove inline-task-overhead skill: failed 3 evaluation iterations (avg +2.7%) because the baseline LLM already knows about RoslynCodeTaskFactory overhead — the skill doesn't add enough unique value to meet the 10% improvement threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address review comments - Add caveat that DisableTransitiveProjectReferences may require adding direct references for consumed types - Fix escaped quotes in Quick Wins RunAnalyzers checklist item - Add OutputType=Exe and Main entry point to msbuild-server fixture - Change 'solution' to 'project' in resolve-project-references report - Trim trailing spaces in developer-report.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore accidentally removed known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix remaining review comments - Quote -flp argument in resolve-project-references bash example to handle semicolons in POSIX shells - Format LibA/Class1.cs and LibB/Class1.cs across multiple lines for consistency with other test fixtures - Rename msbuild-server/Program.cs to Calculator.cs to match content Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
0f72bde887 | Update STJ in samples to avoid security vulnerability reports (#313) | ||
|
Jared Parsons
|
4793aa27e9 |
Replace binlog mcp with replay to textlog (#162)
Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com> |
||
|
Viktor Hofer
|
6e2ad53e88 | Move plugin tests out of plugins folder into a root tests/ folder |