mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
dev/jankrivanek/eval-dispatch-plugin-input
473 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Viktor Hofer
|
a895c226b3 |
Fix agents field in plugin.json to use explicit file paths (#419)
* Fix agents field in plugin.json to use explicit file paths Claude Code's manifest validator rejects directory paths in the agents field. Replace directory references with explicit .agent.md file paths in all affected plugins (dotnet-diag, dotnet-msbuild, dotnet-template-engine). Fixes https://github.com/dotnet/skills/issues/416 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Validate that agents paths are explicit files, not directories Claude Code's manifest validator rejects directory paths in the agents field. Update PluginValidator to error when an agent path resolves to a directory, guiding authors to use explicit .agent.md file paths instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Youssef Victor
|
c4937f8f68 |
Add 'run_command_and_assert' assertion (#401)
* Add 'run_command_and_assert' assertion * Rename for clarity * Add tests * Fix race condition in command output handling * Fix async output race condition in RunCommandAndAssert WaitForExit(TimeSpan) does not guarantee that async output event handlers have completed. Add parameterless WaitForExit() call after the timed wait to ensure stdout/stderr are fully captured before checking their contents. Also handle the timeout case explicitly by killing the process. * Address PR review: extract CommandAssertionArgs, add regex matching, configurable timeout - Extract command-specific fields from Assertion into a dedicated CommandAssertionArgs record type for cleaner separation of concerns - Add ExpectedStdOutMatches/ExpectedStdErrorMatches for regex-based stdout/stderr assertions (matches type asserts) - Make command timeout configurable via assertion-level 'command_timeout' field, falling back to the scenario timeout when not specified - Show actual timeout value in timeout error messages - Update EvalSchema parsing, EvaluateCommand call sites, and all tests - Add new tests for regex matching and timeout configuration --------- Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com> |
||
|
Amaury Levé
|
0846e6015d | Improve MSTest migration skills (#387) | ||
|
Amaury Levé
|
9adebf86a8 | Add experimental CRAP score skill for dotnet-test (#413) | ||
|
Amaury Levé
|
00c1e82006 |
Add exp-test-tagging skill to dotnet-test-experimental (#417)
* Add exp-test-tagging skill and evaluation Experimental skill for analyzing and tagging test suites with a standardized trait taxonomy (positive, negative, boundary, critical-path, smoke, regression, integration, end-to-end, performance, security, concurrency, resilience, destructive, configuration, flaky). Supports MSTest, xUnit, and NUnit with framework-specific attribute syntax. Includes 5 eval scenarios: tag MSTest/xUnit/NUnit, audit-only, non-activation. * Move exp-test-tagging to dotnet-test-experimental plugin |
||
|
Amaury Levé
|
6489fbee39 |
Add exp-test-anti-patterns skill for test quality review (#415)
New experimental skill that detects anti-patterns and code smells in .NET test suites. Covers assertion quality, test isolation, naming, flakiness indicators, over-mocking, and structural problems across MSTest, xUnit, NUnit, and TUnit. Key features: - 4-tier severity classification (Critical/High/Medium/Low) - False-positive avoidance: calibrates severity honestly and recognizes well-written tests without inventing problems - Positive observations alongside findings Evaluation results (3 runs, claude-opus-4.6): - Flakiness/coupling detection: 3.0 -> 5.0 (isolated), 4.0 (plugin) - Duplicates/magic values: 3.0 -> 5.0 (isolated), 4.7 (plugin) - Well-written test recognition: 2.0 -> 5.0 (both modes) - Mixed severity detection: 5.0 baseline maintained - Overfitting score: 0.06 across all scenarios |
||
|
Amaury Levé
|
e0e1a91d90 |
Extract shared references and add cross-refs for dotnet-test skills (#414)
* Extract shared references and add cross-refs for dotnet-test skills - Extract platform detection logic into run-tests/references/platform-detection.md (shared by run-tests, mtp-hot-reload, migrate-vstest-to-mtp) - Extract filter syntax reference into run-tests/references/filter-syntax.md (shared by run-tests, mtp-hot-reload, migrate-vstest-to-mtp) - Add cross-references between skills in When Not to Use / Related Skills sections - run-tests SKILL.md reduced from 364 to 191 lines * Fixes |
||
|
Copilot
|
a68fa8da18 |
Add Codex CLI installation docs to README (#408)
* Initial plan * Add Codex CLI installation section to README and allow developers.openai.com domain Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Remove Codex CLI example per review feedback Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Copilot
|
6face98ff6 |
Change agents and skills fields in plugin.json to array format for Claude Code CLI compatibility (#407)
* Initial plan * Remove agents string field from plugin.json and handle array format in parser Claude Code CLI validates `agents` as array-of-strings per its plugin manifest schema. Our plugin.json files had `"agents": "./agents/"` (a string), which caused "agents: Invalid input" validation errors on install. Remove the field from the 3 affected plugin.json files — agents are still discovered by convention from the `agents/` directory. Also make ParsePluginJson resilient to both string and array formats for forward-compatibility. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Remove `agents` string field from plugin.json for Claude Code CLI compatibility Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Change agents field to array format in plugin.json, update validator to read array-first with string fallback Plugin.json files now declare agents as an array of file paths: "agents": ["./agents/foo.agent.md", "./agents/bar.agent.md"] ParsePluginJson reads the array first (preferred, Claude Code schema), falls back to a string path (legacy). PluginInfo gains an AgentPaths field. Validator and discovery use the array when present, else fall back to directory-based convention. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Simplify agents to directory-path array form ["./agents/"], handle directories in array entries Per Copilot CLI spec, agents field values are "Path(s) to agent directories" — no need to list individual files. Use ["./agents/"] to mirror how skills uses "./skills/". Updated validator and discovery to handle both directory and file entries in the array. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Delete .nuget/nuget.exe * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Convert skills field to array format in plugin.json, update validator to handle both forms Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Align AgentPaths null checks to use { Count: > 0 } pattern for consistency Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Change `agents` and `skills` fields in plugin.json to array format for Claude Code CLI compatibility Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Delete .nuget/nuget.exe * Consolidate SkillsPath/SkillPaths and AgentsPath/AgentPaths into single array fields Remove dual string/array fields from PluginInfo. The parser now normalizes legacy string values into single-element arrays, so all downstream code uses only SkillPaths and AgentPaths (IReadOnlyList<string>). This eliminates all array-vs-string fallback branching. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
29a9dd1202 | Add eval test authoring skill (#409) | ||
|
Jan Krivanek
|
68d83b6ea8 | Add parallelism configuration options to eval.yaml (#410) | ||
|
Jan Krivanek
|
101a151e98 | Improve the build-perf-diag scenario test (#404) | ||
|
Jan Krivanek
|
cc8010f357 | Include judge tokens, fix summaries (#405) | ||
|
YuliiaKovalova
|
dcba4f3d8e |
Add dotnet-template-engine plugin (#341)
* Add dotnet-template-engine plugin Add template-engine skills for .NET project scaffolding powered by the DotnetTemplateMCP MCP server (v1.1.0). Includes three skills and one agent: Skills: - template-discovery: find, inspect, and compare templates with intent resolution (70+ keyword mappings) - template-instantiation: create projects with CPM adaptation, latest NuGet versions, parameter validation, and multi-template composition - template-authoring: create templates from existing projects and validate template.json for authoring issues Agent: - template-engine: expert agent that routes to the appropriate skill based on user intent Also adds CODEOWNERS entries, marketplace.json registration, eval tests, and .mcp.json for DotnetTemplateMCP server configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: mcpServers in plugin.json, expect_tools, path fix - Move MCP server config into plugin.json under mcpServers so SkillValidator discovers the template_* tools during evaluation runs - Add expect_tools constraints to all eval scenarios to verify actual MCP tool invocation instead of relying on rubric text alone - Fix frontmatter path: template.config → .template.config (leading dot) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update DotnetTemplateMCP to v1.2.0 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Include 'dotnet new' in agent description for discoverability Address review feedback from danmoseley: the frontmatter description is how the caller decides whether to load the agent tokens, so it needs to mention 'dotnet new' as a trigger keyword. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove MCP server configuration and references - Delete .mcp.json MCP server config file - Remove mcpServers section from plugin.json - Remove MCP/DotnetTemplateMCP references from agent, skills, and marketplace descriptions - Skills and eval files remain unchanged Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove ghost tool references and add CLI fallback guidance - Remove references to 4 non-existent tools: template_compose, template_suggest_parameters, template_validate, solution_analyze - Replace with real alternatives (template_inspect, manual review, sequential template_instantiate, direct file inspection) - Add CLI fallback guidance so LLMs degrade gracefully when template tools are unavailable - Fix authoring eval to not expect template_validate tool Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: CODEOWNERS naming, claude marketplace, README - Rename CODEOWNERS header to dotnet-template-engine for consistency - Mirror plugin entry in .claude-plugin/marketplace.json - Add plugin to README What's Included table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix PR review comments: align eval rubric with expect_tools and sync marketplace.json - Updated template-discovery eval.yaml rubric to match expect_tools (removed 'or template_search' since expect_tools enforces template_from_intent) - Made .claude-plugin/marketplace.json byte-for-byte identical to .github/plugin/marketplace.json (removed extra trailing newline) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments: strengthen eval assertions and fix rubrics - Add exit_success assertions to all eval scenarios - Add file_exists and file_contains assertions for template-instantiation - Add output_contains for shortName and datatype in template-authoring - Fix rubrics to reference actual tools (skill + dotnet new CLI) instead of non-existent MCP tools (template_from_intent, template_inspect, template_instantiate) which caused the judge to penalize quality scores - Update expect_tools from non-existent MCP tools to 'skill' - Change plugin version from 1.2.0 to 0.1.0 for consistency with other plugins - Remove extra blank line in README.md after plugin table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rewrite skills and agent to use dotnet new CLI instead of phantom MCP tools The SKILL.md files and agent.md referenced MCP tools (template_from_intent, template_inspect, template_instantiate, template_dry_run, etc.) that are not available at runtime since no MCP server is declared in plugin.json. This caused the agent to waste tokens attempting to find these tools before falling back to dotnet new CLI commands. Changes: - template-discovery/SKILL.md: Replace MCP tool calls with dotnet new search, dotnet new list, dotnet new --help, and dotnet new --dry-run. Add intent-to-template keyword mapping table. - template-instantiation/SKILL.md: Replace template_instantiate with dotnet new CLI commands. Add explicit CPM adaptation steps and multi-project composition with dotnet sln/dotnet add reference. - template-authoring/SKILL.md: Replace template_create_from_existing with manual template.json creation guidance including example JSON. Replace template_dry_run/template_instantiate with CLI equivalents. - template-engine.agent.md: Replace Available Tools table with CLI Commands Reference. Update all workflow steps to use CLI commands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix file assertion paths: remove ./ prefix for glob matcher compatibility Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |
||
|
Jan Krivanek
|
defd58af6a |
Add proper skills isolation (copying and allowlisting) (#402)
* Add proper skills isolation (copying and allowlisting) * Address PR review: harden isolation, skip symlinks, precompute allowlist - Pass null skillPath in isolated mode so the original skill tree (including siblings) is not accessible; only staged copies are allowed. - Harden CopyDirectory to skip symlinks/reparse points and verify that recursed paths stay within the source root. - Precompute additionalAllowedDirs once at session creation instead of allocating on every permission request. - Add IsolatedStagingDoesNotExposeOriginalOrSiblingSkills test that verifies both directory-level and permission-level isolation. * Fix unit test |
||
|
Jan Krivanek
|
30c19eae13 | Add tokens usage dashboard and data collection (#385) | ||
|
dependabot[bot]
|
243f35a4bd |
Bump the microsoft-and-system group with 4 updates (#397)
Bumps Microsoft.Bcl.Memory from 9.0.14 to 10.0.5 Bumps Microsoft.Data.Sqlite from 10.0.0 to 10.0.5 Bumps Microsoft.Extensions.FileSystemGlobbing from 10.0.4 to 10.0.5 Bumps System.CommandLine from 2.0.4 to 2.0.5 --- updated-dependencies: - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-major dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Bcl.Memory dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-major dependency-group: microsoft-and-system - dependency-name: Microsoft.Data.Sqlite dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
dependabot[bot]
|
2531f0f374 |
Bump Azure.Identity from 1.10.0 to 1.11.4 (#399)
--- updated-dependencies: - dependency-name: Azure.Identity dependency-version: 1.11.4 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
dependabot[bot]
|
8e27c14ea4 |
Bump actions/download-artifact in the github-actions-dependencies group (#396)
Bumps the github-actions-dependencies group with 1 update: [actions/download-artifact](https://github.com/actions/download-artifact). Updates `actions/download-artifact` from 8.0.0 to 8.0.1 - [Release notes](https://github.com/actions/download-artifact/releases) - [Commits](https://github.com/actions/download-artifact/compare/70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c) --- updated-dependencies: - dependency-name: actions/download-artifact dependency-version: 8.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
Youssef Victor
|
f75fd8840c | Create SECURITY.md (#400) | ||
|
Viktor Hofer
|
c5435dd3da |
Make evaluate a subcommand with tests-dir as required argument (#398)
* Make evaluate a subcommand with tests-dir as required argument
Change EvaluateCommand from RootCommand to a regular Command('evaluate')
so it is a subcommand like check, consolidate, and rejudge. Convert
--tests-dir from an optional Option to a required positional Argument
placed after the skill paths.
Update .csproj RunArguments, evaluation-run workflow, README, and
CONTRIBUTING.md to reflect the new CLI syntax.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Default --parallel-skills/scenarios/runs to 3
Update the default from 1 to 3 for all three parallelism options and
remove the now-redundant overrides from RunArguments in the .csproj.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Replace RunArguments with launchSettings.json, publish in skill-check
Add Properties/launchSettings.json with default evaluate args for local
development. Remove RunArguments from .csproj. Update skill-check.yml to
publish the validator and invoke the binary directly instead of using
dotnet run.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Keep --tests-dir as a required option, not positional
A variadic paths argument greedily consumes all positional values,
making a trailing positional tests-dir impossible to parse. Keep
--tests-dir as a named required option which matches the previous UX.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move testsDirOpt after resultsDirOpt, drop evaluate from CONTRIBUTING examples
Reorder testsDirOpt declaration and registration to follow resultsDirOpt.
Remove the evaluate subcommand from CONTRIBUTING.md dotnet run examples
since the launch profile already provides it.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
||
|
Viktor Hofer
|
bd965cbd55 |
Require /evaluate command for PR evaluation (#394)
* Require /evaluate command for PR evaluation Update evaluation.yml to require an /evaluate command to trigger evaluation, matching the behavior in evaluation-fork-pr.yml. This change: - Adds pr-status job that posts initial commit status on PR open/sync: - 'success' if no skills changed (PR not blocked) - 'pending' if skills changed (maintainer must post /evaluate) - Adds gate job to validate /evaluate command (permissions, same-repo check) - Replaces evaluation-status check-run with report-status using commit status API - Scheduled runs continue to work unchanged (daily evaluation of all plugins) The evaluation-status required check still works because: - PRs without skill changes get immediate success status - PRs with skill changes get pending status until /evaluate is posted Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge evaluation and evaluation-fork-pr into single workflow Consolidate evaluation.yml and evaluation-fork-pr.yml into one unified workflow that handles both same-repo and fork PRs via conditionals: - pr-status: posts initial commit status for same-repo PRs (pull_request) - fork-pr-status: posts initial commit status for fork PRs (pull_request_target) - gate: unified /evaluate command validation with is_fork output - discover: unified skill discovery with conditional worktree for forks - report-status: single job for posting final commit status - Scheduled runs and publish-benchmark unchanged Also fixes: - run-evaluation now uses always() so scheduled runs aren't skipped - report-status scoped to issue_comment events only - Removed stale evaluation-fork-pr.yml reference from infra detection regex - Updated evaluation-run.yml comment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback - Fetch PR head and use worktree for both fork and same-repo PRs in discover - Always pass checkout-ref (PR head SHA) to evaluation-run for /evaluate - Fix runs count: use event_name == 'schedule' instead of github.ref Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Require /evaluate for infra changes too Expand pr-status and fork-pr-status discovery to detect evaluation workflow and skill-validator changes in addition to skill/test changes. Both now post pending status requiring /evaluate when infra files are modified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Build skill-validator from PR branch for non-fork PRs Add is-fork input to evaluation-run.yml. When false (same-repo PRs), the validator is built from checkout-ref so skill-validator changes are tested. When true (fork PRs), the validator is built from the base branch to prevent untrusted code from modifying tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review: fix security comment, add issues:write - Update security model comment to reflect validator-ref behavior - Add issues: write permission to gate and report-status jobs for reaction and comment API calls Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add prominent note about issue_comment trigger limitation Changes to this workflow file in a PR won't take effect until merged because issue_comment always runs from the default branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Don't require /evaluate for workflow YAML changes Workflow YAML changes can't be tested via /evaluate anyway (issue_comment always runs from main). Only eng/skill-validator/ changes now require /evaluate in the pr-status jobs. The discover job still detects workflow changes to trigger all-plugin evaluation when /evaluate is run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jeff Handley
|
646b5d93b0 |
Add Skill: Convert to Central Package Management (convert-to-cpm) (#75)
* Add convert-to-cpm skill for NuGet Central Package Management conversion Authoring process: - Researched the Agent Skills standard at agentskills.io and reviewed repo conventions - Studied the existing csharp-scripts skill and test structure as a reference implementation - Studied Microsoft's CPM documentation at learn.microsoft.com - Authored SKILL.md with a 9-step workflow covering scope determination through validation - Created 5 test scenarios across simple, moderate, and advanced tiers - Performed dry-run evaluation against all scenarios, identifying 15 findings - Applied all 15 fixes (4 gaps, 6 ambiguities, 5 improvements) to SKILL.md - Re-evaluated: 50 OK, 0 gaps, 0 ambiguities, 1 test fixture improvement - Updated test fixture to resolve the remaining finding - Final validation: 254 lines, 0 errors, 0 warnings Skill implementation (skills/convert-to-cpm/SKILL.md): - 9-step workflow: scope, CPM check, audit, create/update props, update projects, handle properties, restore/validate, cleanup, summary - Handles single projects, solutions, and full repositories - Audits packages with per-project detail table including version conflicts and security advisories - Flags CVE/security vulnerabilities and recommends patched versions - Supports MSBuild property-based versioning with inline vs keep decisions - Covers conditional PackageReference items across target frameworks - Discovers PackageReference in imported .props/.targets via Import chain scanning - Provides cross-platform commands (Unix and Windows/PowerShell) - Documents import order requirements for property references in Directory.Packages.props - Mentions dotnet new packagesprops as preferred creation method (.NET 8+) - Includes validation checklist and common pitfalls table Test scenarios (tests/convert-to-cpm/): - simple-single-project: 1 project, 3 packages, no conflicts - simple-solution: 3 projects via .sln, 5 packages, PrivateAssets preservation - moderate-version-conflicts: 4 projects, System.Text.Json version conflicts with CVE-2024-43485 advisory, AutoMapper major version conflict with VersionOverride - moderate-msbuild-properties: 2 projects using MSBuild properties for versioning in Directory.Build.props, inline vs keep decisions, property cleanup verification - advanced-multi-complexity: 5 projects combining version conflicts, MSBuild properties, conditional TFM PackageReference, shared Common.props imports, VersionOverride, and security advisory handling Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor SKILL.md with references, baseline build, and package comparison Condensed SKILL.md using progressive disclosure: - Reduced from 254 lines / ~3937 tokens to 120 lines / ~1778 tokens - Extracted detailed procedural instructions into 5 focused reference files - Main file retains workflow decision logic with links to references loaded on demand - Follows agentskills.io progressive disclosure pattern: metadata -> instructions -> resources Added baseline build verification (new Step 2): - Requires clean build (dotnet clean && dotnet build) before any changes - Captures baseline.binlog for structured build artifact - Captures baseline-packages.json via dotnet list package --format json - Blocks conversion if baseline build fails Added post-conversion baseline comparison (Steps 8 and 10): - Step 8 produces after-cpm.binlog and after-cpm-packages.json from clean build - Step 10 compares baseline vs post-conversion package lists per project - Changes table shows version bumps, VersionOverride entries, added/removed packages - Unchanged table confirms version-neutral conversion for unmodified packages - Binlog files produced as artifacts for user review in MSBuild Structured Log Viewer Reference files created: - references/audit-complexities.md: 7 complexity categories for package auditing - references/baseline-comparison.md: dotnet list package comparison procedure, binlog production, artifact guidance, table formats for changes and unchanged packages - references/directory-packages-props.md: File creation (dotnet new packagesprops), placement rules, conditional versions, VersionOverride patterns - references/msbuild-property-handling.md: Property usage search, inline vs keep decisions, import order requirements, cleanup verification - references/validation-and-errors.md: NuGet error codes (NU1008, NU1010, NU1507), clean build validation, multi-TFM guidance, common pitfalls Workflow is now 10 steps (was 9): 1. Determine scope 2. Establish baseline build (new) 3. Check for existing CPM 4. Audit package references 5. Create or update Directory.Packages.props 6. Update project files 7. Handle MSBuild version properties 8. Restore and validate (now with binlog + package list capture) 9. Clean up obsolete properties 10. Summary and baseline comparison (expanded with diff tables) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace third-party packages with Microsoft equivalents, add baseline/comparison steps, add .gitignore for test solutions Package replacements across all skill references and test scenarios: - Serilog.AspNetCore -> OpenTelemetry.Extensions.Hosting 1.15.0 - Serilog (standalone) -> OpenTelemetry.Extensions.Hosting 1.15.0 - Swashbuckle.AspNetCore -> Microsoft.AspNetCore.OpenApi 8.0.24 - SerilogVersion MSBuild property -> OTelVersion - DIVersion updated from 8.0.1 to 9.0.0 (DI/Abstractions alignment) - Microsoft.AspNetCore.OpenApi uses 8.0.24 (not 9.x) for net8.0 compat Test fixture improvements: - Added Step 2 (baseline build with binlog) to all 5 with-skill.md files - Added Step 10 (package comparison table + binlog artifact links) to all 5 with-skill.md files - Expanded simple-solution without-skill.md with more realistic agent behavior - Standardized security fix terminology: table markers cite CVEs, prose references CVE IDs - Removed space-padded table formatting in favor of compact markdown with emojis Test solution infrastructure: - Added tests/convert-to-cpm/.gitignore to exclude src/ build artifacts - Added generation prompt to test README for recreating buildable solutions - All 5 test solutions verified to restore and build successfully Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback on terminology, placement, and reference accuracy - Rename Step 10 and validation checklist from 'binlog comparison' to 'package list comparison' - Rewrite baseline-comparison.md: title to 'Baseline Comparison', lead with dotnet list package as primary comparison mechanism, position binlogs as supplementary artifacts for manual review/troubleshooting - Add brief definition of binlogs before the artifacts section - Fix Directory.Packages.props placement guidance: use first common ancestor of all .NET projects instead of assuming repository root (.git), since many repos nest source under src/ or similar - Update SKILL.md scope input and Step 1 to reference directory-scoped conversion - Standardize security fix terminology: table markers cite CVEs, prose uses 'security fix for CVE-...' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review findings: sort PackageVersion, add packages.config scenario, fix commands Review findings addressed: 1. Fixed BlobsVersion grep step description in advanced scenario 2. Sorted PackageVersion entries alphabetically in all Directory.Packages.props outputs 3. Consolidated baseline-comparison.md to single-table format 4. Added solution file arguments and multi-line commands in test fixtures 5. Added Directory.Packages.props placement question for single-project scenario 6. Created simple-packages-config test scenario (packages.config rejection) 7. Use dotnet new packagesprops in simple-single-project workflow 8. Fixed More Info link text to match reference file title Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix test scenario accuracy: sln/slnx mix, package counts, ImplicitUsings - Use mix of .sln and .slnx across scenarios (simple-solution and moderate-msbuild-properties use .sln; moderate-version-conflicts and advanced-multi-complexity use .slnx) - Fix simple-solution package count from 5 to 6 - Fix advanced user response #2 to reference Azure.Storage.Blobs version - Add ImplicitUsings to advanced Directory.Build.props in audit and cleanup - Complete the advanced unchanged table with all expected package entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add skill evaluation section to test README Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Inventory.sln references and make evaluation opt-in - Fix 4 occurrences of Inventory.sln to Inventory.slnx in moderate-version-conflicts/with-skill.md for consistency with the scenario README and test solution - Update test README to only run skill evaluation when the user explicitly requests it Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Condense SKILL.md with progressive disclosure - Remove duplicated bash code blocks from Steps 2 and 8, linking to baseline-comparison.md instead - Merge Steps 7 (MSBuild property decisions) and 9 (cleanup) into a single step since both reference msbuild-property-handling.md - Condense Step 10 summary to reference baseline-comparison.md for procedure details - Reduces from 120 to 97 lines (10 steps to 9 steps) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Enterprise.sln references to Enterprise.slnx in with-skill.md Address code review feedback: fix 4 occurrences of Enterprise.sln to Enterprise.slnx in advanced-multi-complexity/with-skill.md for consistency with the scenario README and test solution. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve convert-to-cpm skill: preserve whitespace, save summary report, fix links - Instruct Step 6 to retain each file's existing indentation style and blank lines to keep diffs minimal - Instruct Step 9 to save the full summary as convert-to-cpm.md alongside binlog and JSON artifacts, suggesting its contents for a PR description - Strip en-us/ from learn.microsoft.com links in SKILL.md and validation-and-errors.md to make them culture-agnostic - Update all with-skill.md test expectations to include the convert-to-cpm.md create step and artifact mention Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Indicate that the binlog viewer is for the user; not the tool/skill * Move convert-to-cpm skill and tests to src/dotnet-msbuild layout Move skill from skills/convert-to-cpm/ to src/dotnet-msbuild/skills/convert-to-cpm/ and tests from tests/convert-to-cpm/ to src/dotnet-msbuild/tests/convert-to-cpm/ to match the repository layout established on main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Convert tests from with/without-skill format to eval.yaml Replace manual test approach (README.md + with-skill.md + without-skill.md) with automated eval.yaml scenarios containing assertions and rubric items. Add committed test fixture files (.csproj, .sln, .slnx, .props) for each scenario with copy_test_files setup, replacing the git-ignored generated solutions. 6 scenarios converted: - simple-single-project: basic single project CPM conversion - simple-solution: multi-project solution conversion - simple-packages-config: negative test (packages.config rejection) - moderate-version-conflicts: version conflict resolution - moderate-msbuild-properties: MSBuild property version handling - advanced-multi-complexity: combined complexities (conflicts, properties, conditionals, shared props) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Consolidate eval.yaml files into single root eval.yaml Move all 6 scenario definitions from individual subdirectory eval.yaml files into a single src/dotnet-msbuild/tests/convert-to-cpm/eval.yaml file, matching the skill-validator discovery convention that expects {testsDir}/{skillname}/eval.yaml. Scenario prompts updated to reference subdirectory-prefixed paths since copy_test_files now copies all fixture subdirectories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reorganize convert-to-cpm into plugins/tests folders * Add post-conversion reporting rubrics to CPM eval scenarios Add rubric items for baseline-vs-result comparison reporting, binlog capture and guidance, version reconciliation, and risk/reassurance messaging across all conversion scenarios. Increase complex scenario timeout to 360s. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve post-conversion reporting, conflict resolution, and skill activation SKILL.md: - Surface binlog capture commands explicitly in Steps 2 and 8 - Expand Step 4 with detailed version conflict presentation and resolution workflow - Restructure Step 9 into 5-section post-conversion report template: overview, conflict resolutions, baseline comparison, risk assessment, and artifacts/next steps with binlog education - Broaden activation triggers in description and When to Use to match update/bump/upgrade/align/sync terminology from ecosystem PRs - Add activation for out-of-sync/conflicting/mismatched version scenarios audit-complexities.md: - Expand version conflict section with per-conflict presentation format, 4 resolution options with trade-offs, and risk-level guidance eval.yaml: - Pre-specify conflict resolution strategy in prompts to avoid blocking on unanswered user prompts during single-turn evaluation - Add 3 new scenarios testing CPM recommendation when user asks to update/align packages without mentioning CPM - Update rubrics to match directed resolution strategy - Increase complex scenario timeout to 360s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback from kartheekp-ms - Use GitHub markdown URL for CPM docs reference (SKILL.md) - Use dotnet package list (renamed in .NET 10) with backward compat note for dotnet list package (SKILL.md, baseline-comparison.md) - Use dotnet package list --format json for auditing (SKILL.md Step 4) - Document Directory.Packages.props sub-folder support for test projects with different dependencies (directory-packages-props.md) - Remove security advisory version bumping from CPM conversion scope; instead surface advisories as follow-up items in the post-conversion report (audit-complexities.md, SKILL.md Step 9 Section 5) - Add Follow-up items section to report template covering security advisories, deprecated packages, version alignment opportunities, and test validation (SKILL.md Step 9) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve convert-to-cpm skill activation, artifact preservation, and eval assertions - Broaden skill description to trigger on version drift/sync scenarios and MSBuild property contexts even when CPM is not explicitly mentioned - Add explicit instructions to preserve binlog and JSON artifacts (not delete) - Make convert-to-cpm.md report a mandatory deliverable, not optional - Replace fragile output_contains assertions (ManagePackageVersionsCentrally, Directory.Packages.props) with file_contains checks on the actual generated files across 5 conversion scenarios Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extend timeouts and add CODEOWNERS entry * Move convert-to-cpm skill into a new dotnet-nuget plugin * Fix erroneous plugin msbuild/nuget plugin split change. Add dotnet-nuget to marketplace. * Add NuGet package management skills to README * Address PR feedback: clarify CPM sub-folder semantics, remove security-fix example, require explicit conflict strategy - directory-packages-props.md: Clarify that sub-folder Directory.Packages.props files are independent and replace (not extend) the parent; require explicit MSBuild Import to chain files - baseline-comparison.md: Replace security-fix upgrade example with version conflict alignment example, consistent with audit guidance that security advisories are follow-up items - SKILL.md: Remove default 'use highest version' conflict strategy; require explicit user choice or confirmation when conflicts are detected Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extend eval timeouts for scenarios hitting time limits - Convert single project: 180s -> 270s - Recommend CPM (version conflicts): 180s -> 270s - Recommend CPM (complex repo): 360s -> 540s - Convert complex repository: 360s -> 540s Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add missing CODEOWNERS for dotnet-nuget plugin. Reorder convert-to-cpm evals and expand timeouts. * Remove containing skill mentions from reference files; fix ordering * Replace output_matches with deterministic file assertions in eval.yaml Convert 6 non-deterministic output_matches assertions to 8 file-based assertions (file_contains, file_not_contains) across conversion scenarios: - Scenario 5: Verify PrivateAssets preserved in Tests.csproj - Scenario 6: Verify OTelVersion/DIVersion removed from Directory.Build.props - Scenario 7: Verify System.Text.Json and Azure.Identity in Directory.Packages.props - Scenario 8: Verify System.Text.Json in Directory.Packages.props and BlobsVersion/HostingVersion removed from Directory.Build.props Recommendation-only scenarios and conversational assertions (conflict detection, conditionals, Common.props awareness) remain as output_matches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reorganize skill description with USE FOR / DO NOT USE FOR markers Restructure the description to improve activation reliability by adopting the USE FOR / DO NOT USE FOR pattern used by other skills in the repo (thread-abort-migration, binlog-failure-analysis). Key changes: - Front-load activation cues before capability list - Add explicit USE FOR / DO NOT USE FOR section markers - Add .sln/.slnx file extensions for solution-path matching - Move DO NOT USE exclusions into description (was only in body) - Use YAML folded scalar for multi-line readability All existing activation keywords preserved. Length reduced from 882 to 849 chars (max 1024). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix issues for external link references * Increase timeouts * Handle packages.config non-activation in convert-to-cpm skill Set expect_activation: false on the packages.config eval scenario so non-activation is reported as expected rather than flagged as a warning. Add a packages.config guard in Step 1 of the skill workflow so that if the skill is activated for a packages.config project, it stops early and recommends migrating to PackageReference first. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove leading blank lines from .sln test files Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use imperative 'Read' instead of 'See' for reference file links Ensures the agent explicitly loads and reads reference files rather than treating them as optional 'see also' links. This aligns with the repo convention of explicitly instructing agents to load reference content. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify dotnet-nuget CODEOWNERS * Simplify packages.config guard * Add minimal Program.cs entrypoints to Web/Worker test fixtures Test fixture projects using Microsoft.NET.Sdk.Web and Microsoft.NET.Sdk.Worker require a Program.cs entrypoint for dotnet build to succeed. The skill workflow instructs the agent to run dotnet restore and dotnet build for validation, so these fixtures must be buildable. Adds minimal entrypoints (with explicit using directives since ImplicitUsings is not enabled in these projects) to 6 projects: - simple-solution/Web - moderate-version-conflicts/Api, Worker - moderate-msbuild-properties/Api - advanced-multi-complexity/Api, Web Also adds Microsoft.Extensions.Hosting package reference to the Worker project, matching what the dotnet new worker template produces. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
Copilot
|
0bb3606942 |
Fix release job skipped by transitive dependency chain from discover (#395)
* Initial plan * Fix release job being skipped due to tainted dependency chain from discover The release job was always skipped for schedule and workflow_dispatch events because the discover job (a transitive dependency via build) is skipped for non-PR events, which taints the dependency chain. Without always() in its if condition, GitHub skips release without evaluating the expression. Add always() so the if expression is always evaluated, and add an explicit needs.build.result == 'success' guard so the release only runs when the build actually passed. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Viktor Hofer
|
1a20e39a8d |
Refactor skill-validator: add check command, rename validate to evaluate (#388)
* Refactor skill-validator: add check command, rename validate to evaluate - Add new 'check' command for CI skill validation (frontmatter, structure) - Rename ValidateCommand to EvaluateCommand to clarify its purpose - Add skill-check.yml GitHub Actions workflow - Refactor SkillDiscovery for improved skill detection - Simplify OverfittingJudge and SkillProfiler - Update tests to match refactored APIs Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix check subcommand: use EvaluateCommand as root command System.CommandLine 2.0.4 requires the primary command to be the RootCommand. Restores the upstream pattern where the evaluate/validate command is the root and check/consolidate/rejudge are subcommands. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Pass all plugins in single check invocation Build plugin args dynamically and pass them all at once instead of looping per plugin. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments - Remove 'evaluate' subcommand token from README, evaluation-run.yml, and RunArguments in csproj (evaluate is now the root command) - Wrap ParsePluginJson in try/catch for JsonException with clean error - Add CheckCommandTests with aggregate description limit coverage (under/at/over limit and multi-plugin cases) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
ae0b6a1d86 | Add mtp-hot-reload skill for iterative test fixing with MTP hot reload (#389) | ||
|
YuliiaKovalova
|
14346ff3b2 |
Enrich build-perf-diagnostics skill with MSBuild team and community insights (#342)
* Enrich build-perf-diagnostics skill with MSBuild team and community insights Additions (pure knowledge, no MCP dependency): - RAR: dotnet/msbuild#2015 context, DisableTransitiveProjectReferences, ReferenceOutputAssembly - Copy tasks: SkipCopyUnchangedFiles, --artifacts-path, Copy batching (dotnet/msbuild#12884) - Dev Drive (ReFS): copy-on-write + reduced Defender scans, OrchardCore 7257 Copy tasks benchmark - Evaluation: NuGetSdkResolver 180-400ms overhead (dotnet/msbuild#4025), EnableDefaultItems - NuGet: RestoreUseStaticGraphEvaluation (20s+ savings in large builds) - Project graph shape: wide vs deep (40% faster clean, 20% faster incremental) - MSBuild Server: MSBUILDUSESERVER=1 for CLI caching - Inline task overhead: 1s vs 3ms (RoslynCodeTaskFactory vs compiled) - Misleading ResolveProjectReferences time (dotnet/msbuild#3135) - Incrementality anti-patterns (dotnet/msbuild#13206) - Impact categorization (HIGH/MEDIUM/QUICK WINS) - Quick Wins: expanded to 15 items (from 7) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix minor issues from review comments - Fix missing closing XML tag for CreateHardLinksForCopyFilesToOutputDirectoryIfPossible in Quick Wins checklist - Clarify confusing Copy task batching wording per reviewer feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Trim skill to reduce token overhead Remove standalone sections (MSBuild Server, Inline Tasks, ResolveProjectReferences, Incrementality Anti-patterns) that add ~1400 extra tokens without benefiting the evaluated scenario. Consolidate Quick Wins checklist accordingly. These sections can be contributed as separate skills in follow-up PRs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add separate skills for MSBuild server, inline tasks, and ResolveProjectReferences Extract content removed from build-perf-diagnostics into standalone skills to keep each skill focused and avoid token budget bloat: - msbuild-server: MSBUILDUSESERVER=1 for CLI incremental caching - inline-task-overhead: RoslynCodeTaskFactory ~1s overhead vs ~3ms compiled - resolve-project-references: misleading Target Performance Summary time (dotnet/msbuild#3135), guides users to Task Performance Summary instead Incrementality anti-patterns content was not extracted as a separate skill because the existing incremental-build skill already covers that area. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add github.com/dotnet/msbuild to known-domains.txt Required by reference scanner for MSBuild issue links in build-perf-diagnostics and resolve-project-references skills. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace full dotnet/msbuild URLs with shorthand notation The reference scanner reads known-domains.txt from upstream main, not from the PR branch. Use shorthand 'dotnet/msbuild#NNNN' format (consistent with the rest of the file) instead of full URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review comments and fix reference scanner errors - Remove https://aka.ms/devdrive URLs that trigger reference scanner - Soften unsourced benchmark claims (OrchardCore Copy tasks, 20s+ restore savings, 40%/20% graph shape improvements) per reviewer feedback to use workload-dependent language instead Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add eval tests for new skills; remove inline-task-overhead Add eval.yaml and test fixtures for msbuild-server and resolve-project-references skills: - msbuild-server: tests CLI build perf scenario with developer report - resolve-project-references: tests misleading target perf summary with multi-project setup and perf report Remove inline-task-overhead skill: failed 3 evaluation iterations (avg +2.7%) because the baseline LLM already knows about RoslynCodeTaskFactory overhead — the skill doesn't add enough unique value to meet the 10% improvement threshold. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Address review comments - Add caveat that DisableTransitiveProjectReferences may require adding direct references for consumed types - Fix escaped quotes in Quick Wins RunAnalyzers checklist item - Add OutputType=Exe and Main entry point to msbuild-server fixture - Change 'solution' to 'project' in resolve-project-references report - Trim trailing spaces in developer-report.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore accidentally removed known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix remaining review comments - Quote -flp argument in resolve-project-references bash example to handle semicolons in POSIX shells - Format LibA/Class1.cs and LibB/Class1.cs across multiple lines for consistency with other test fixtures - Rename msbuild-server/Program.cs to Calculator.cs to match content Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
7e7dabd343 |
[WIP] Improve msbuild skills descriptions (#384)
* Improve msbuild skills descriptions * Improve descriptions |
||
|
Amaury Levé
|
c55e4f222b |
Add MSTest migration skills (v1/v2→v3, v3→v4) (#377)
* Add MSTest migration skills (v1/v2→v3, v3→v4) New dotnet-test plugin with two step-by-step migration skills: - migrate-mstest-v1v2-to-v3: covers assembly ref (v1) and NuGet (v2) projects, assertion overload changes, DataRow constructors, .testsettings replacement, timeout unification, and MSTest.Sdk adoption. - migrate-mstest-v3-to-v4: covers Execute→ExecuteAsync, CallerInfo constructors, ClassCleanupBehavior removal, TestContext.Properties generic change, Assert API changes, ExpectedExceptionAttribute removal, dropped <net8.0 TFMs, and behavioral changes (DisableAppDomain, TestCase.Id, TreatDiscoveryWarningsAsErrors). Includes eval.yaml test scenarios and fixtures for both skills, plugin.json, and CODEOWNERS entries. * Fixes * Improve results * Improve * Improve description * Copy fix for path * Bump timeout |
||
|
Amaury Levé
|
92d830ff48 |
Add writing-mstest-tests skill with eval scenarios (#378)
* Add writing-mstest-tests skill with eval scenarios Add a new skill for writing modern MSTest 3.x/4.x unit tests covering: - Test class structure (sealed, AAA pattern, naming conventions) - Modern assertion APIs (Assert.Throws, IsInstanceOfType, collection asserts) - Data-driven tests (DataRow, DynamicData with ValueTuples) - Async tests with cancellation and timeout patterns - Test lifecycle (constructor init, TestContext injection) - Anti-pattern detection and modernization Includes 10 eval scenarios across 4 goal categories with fixture files for both modern MSTest and legacy anti-pattern test projects. MSTest.Sdk version is recommended via global.json msbuild-sdks instead of inline in the project file, ensuring all test projects get bumped together. * Fixes * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Evangelink * Apply suggestion from @Evangelink --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
d5d2136157 | Update code owners to use team alias (#382) | ||
|
Larry Ewing
|
1f127d4239 |
Add --keep-sessions flag and rejudge command to skill-validator (#149)
* Add --keep-sessions flag and rejudge command - Add --keep-sessions flag to preserve agent session data in SQLite DB - Create SessionDatabase service with WAL mode and thread-safe writes - Update AgentRunner to separate config/work dir lifecycle - Track session metadata, RunMetrics, judge and pairwise results - Add rejudge command to re-run judges on saved sessions - Add 8 tests for SessionDatabase (CRUD, concurrency, isolation) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add prompt and skill_content columns to sessions.db Store the eval scenario prompt and SKILL.md content in the sessions table so they are available for browsing and replay without needing the original skill directory. Both columns are optional (nullable) for backward compatibility. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Replace skill_content with skill_sha for change detection Compute a SHA-256 hash (truncated to 12 hex chars) over all files in the skill directory, sorted by relative path for determinism. This gives lightweight change detection without storing potentially large skill content in the database. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback: relative config_dir, schema_info table - Use session ID as config dir folder name (sessions/<id>) so the DB record links directly to its files on disk - Store config_dir as relative path for portable results directories - Add schema_info table with type='skill-validator' and version='1' for easy DB detection by external tools like dotnet-replay - Add GetSchemaInfo() method for consumers - Extract GetSessions(whereClause) for reuse - Add tests for schema_info and relative config_dir Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix: share timestamped results dir between sessions and reporters When --keep-sessions is active, pass the already-created timestamped directory to Reporter.ReportResults so sessions.db, session files, and report outputs (results.json, summary.md) all live in the same folder instead of creating two separate timestamped directories. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Adapt rejudge for current validator model Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR #149 review feedback Persist the judge model for rejudge, use fresh judge workdirs, tighten session DB hashing/foreign key behavior, and cache skill SHA calculation during validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix rejudge rubric persistence Persist scenario rubrics in sessions.db so rejudge can reconstruct the original EvalScenario rubric, self-migrate older databases to the new schema, and validate --judge-mode inputs explicitly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Guard SessionDatabase reads with semaphore Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix validator session cleanup and reporting Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
2a8388dd71 |
Add VSTest to MTP migration skill (#373)
* Add VSTest to MTP migration skill New migrate-vstest-to-mtp skill in the dotnet-test plugin covering: - Framework-specific MTP runner enablement (MSTest, NUnit, xUnit.net) - Directory.Build.props centralization of OutputType, runner properties - MSTest.Sdk UseVSTest removal - NUnit3TestAdapter 5.0.0+ upgrade requirement - xunit v2 to xunit.v3 upgrade requirement - dotnet test CLI argument translation (--logger to --report-trx, --collect to --coverage, --blame-crash to --crashdump, --blame-hang to --hangdump) - xUnit.net --filter syntax migration to --filter-class/--filter-trait/ --filter-query - global.json native MTP mode (.NET 10+) vs TestingPlatformDotnetTestSupport VSTest-bridge mode (.NET 9 and earlier) - Azure DevOps VSTest@3 to DotNetCoreCLI@2 pipeline migration - MTP extension NuGet packages (TrxReport, CrashDump, HangDump, CodeCoverage) - Exit code 8 behavioral difference (--ignore-exit-code 8, TESTINGPLATFORM_EXITCODE_IGNORE) Includes 10 eval scenarios with tiered timeouts (240-360s), 5 fixture directories, plugin.json, marketplace.json entries, CODEOWNERS, and README.md updates. * Fix codeowners * Improve * Improve further * Improve prompt * Improve further the description * Address review comments |
||
|
Amaury Levé
|
ab201ee691 |
Fix eval source file resolution to use eval directory (#374)
* Fix eval source file resolution to use eval directory The SetupWorkDir method resolved setup file 'source:' paths relative to the skill directory (plugins/.../skills/<name>/), but eval.yaml files live under tests/. The ../../../../ relative paths resolved correctly but were rejected by the path traversal security check because the resolved path escaped the skill directory boundary. Fix by preferring the eval directory as the base for source resolution when evalPath is available, falling back to skillPath for backward compatibility with scenarios embedded in SKILL.md files. Also update run-tests eval.yaml source paths from '../../../../tests/dotnet-test/run-tests/fixtures/...' to 'fixtures/...' (relative to the eval directory where the fixtures actually live). This was causing all 11 source-based run-tests scenarios to get zero fixture files, making the agent find an empty workspace and fail to evaluate. * Address review: extract ResolveSourcePath helper, add warning, fix remaining eval.yaml paths - Extract source-resolution and path-traversal check into internal ResolveSourcePath helper for testability - Emit a clear warning when file.Source is set but no base directory is available (instead of silently continuing) - Add unit tests covering: evalPath-based resolution, skillPath fallback, null paths warning, traversal rejection, and evalPath preference - Fix path traversal patterns in analyzing-dotnet-performance and clr-activation-debugging eval.yaml files (same issue as run-tests) |
||
|
Manish Jayaswal
|
9cf10eebd2 |
Update CONTRIBUTING.md with new guidelines (#319)
* Update CONTRIBUTING.md with new guidelines Added clarification on contribution expectations and skill proposals. * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
Amaury Levé
|
45c66f87d4 | Add prompt to evaluate skills description (#371) | ||
|
Amaury Levé
|
8cb692082a | Improve run-tests skill description (#372) | ||
|
Jan Krivanek
|
93ccead1e7 | Adjust for timeout issues (#354) | ||
|
Amaury Levé
|
1b8f2aa1e4 |
Address Copilot review comments on test agent/skills (#351)
Follow-up from PR #343 where copilot added review after the PR was merged |
||
|
Dan Moseley
|
a0f77070f5 |
Reduce false positives in reference scanner (#360)
- Skip HTTP-not-HTTPS inside fenced code blocks (code examples use http:// listen addresses). EXTERNAL-DOMAIN checks still run inside fences since agents see raw text, not rendered markdown. - CommonMark-compliant fence tracking (type, length, closing-only whitespace, max 3-space indent). - Skip placeholder/template URLs by checking the host portion only (not the full URL) to prevent bypass via path components. - Case-insensitive placeholder matching. - Generalize path-scoped domain matching to all domains (not just github.com), enabling entries like npmjs.com/package/@org. - Treat wildcard listen addresses (http://+:port, http://*:port) as local URLs. - Add xunit.net to known-domains.txt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
1ccd28b1ba |
Make publish-benchmark resilient to individual plugin eval failures (#365)
* Make publish-benchmark resilient to individual plugin eval failures When any single evaluate matrix leg fails (e.g. OOM kill), the entire run-evaluation reusable workflow reports failure. Previously this caused publish-benchmark to be skipped, blocking dashboard updates for ALL plugins -- even the ones that succeeded. Changes: - publish-benchmark: use !cancelled() so it runs when some legs fail - download-artifact: continue-on-error so missing artifacts don't abort - The existing Generate Benchmark Data step already handles missing plugin results with Write-Warning and continue Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make continue-on-error conditional on evaluation result Only tolerate artifact download failures when run-evaluation didn't fully succeed. When all legs passed, a download failure is a real error that should fail the job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
492ab1d174 |
Reduce scheduled evaluation from every 3h to daily (#362)
* Reduce scheduled evaluation from every 3h to daily, skip if no changes The evaluation schedule was consuming ~/5 days in Copilot premium requests from scheduled runs alone (~/month pace). This change: 1. Reduces the cron schedule from every 3 hours (8 runs/day) to once daily at 08:00 UTC. This alone saves ~87% of scheduled evaluation cost. 2. Adds a check-for-changes guard that skips the scheduled run entirely if there have been no new commits to main since the last successful scheduled evaluation. This avoids wasting tokens re-evaluating identical code. The PR trigger and all PR evaluation behavior is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: harden skip-if-no-changes guard - Add actions: read permission so the workflow run lookup succeeds - Use status=completed with jq conclusion filter instead of status=success - Make all gh api calls non-fatal (fallback to has_changes=true on error) - Make compare API call best-effort for logging only Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Scope actions:read to discover job, increase per_page to 100 - Move actions:read from workflow-level to discover job permissions for least-privilege (only the schedule guard needs it) - Increase per_page from 10 to 100 so the skip guard finds the last successful run even after long failure streaks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Key skip guard off most recent completed run, not just successful Only skip when the most recent completed scheduled run was successful AND at the same SHA. If it failed/cancelled at the current SHA, proceed to retry. This prevents the guard from skipping indefinitely after a transient failure. Also simplifies to per_page=1 since we only inspect the latest run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix jq filter to yield empty when no prior runs exist Use // empty to guard against null when workflow_runs array is empty, preventing 'null null' from leaking into LAST_SHA/LAST_CONCLUSION. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bypass skip guard on manual reruns (run_attempt > 1) When a maintainer clicks 'Re-run jobs' on a scheduled run, the event is still schedule with the same SHA. Without this check, the guard would skip the rerun if the previous attempt succeeded. Now reruns always proceed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
74c4a1c7ac |
Fix PowerShell parsing of three-dot diff range in reference scanner (#359)
When pwsh parses git diff --name-only "origin/main"...HEAD, the partially-quoted argument is tokenized incorrectly, causing git to receive malformed arguments and print its usage message. Build the range string in a variable first so it is passed to git as a single token. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
06d2921a5b |
Add workflow to warn on fork PRs touching infrastructure (#356)
Adds a pull_request_target workflow that posts an advisory comment when a fork PR is opened that modifies eng/ or .github/ files, suggesting the author recreate the PR from an upstream branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
b966ac4dec |
Add structural external dependency checker (#355)
* Add structural external dependency checker for skills, agents, and plugins Add ExternalDependencyChecker to the skill-validator that produces advisory warnings for structural dependencies that need human review: - Script files in skill scripts/ directories (.ps1, .sh, .py, .bat, .cmd, .bash) - INVOKES patterns referencing external scripts in skill descriptions - #tool: references to non-built-in tools in skill/agent content - Non-built-in tools declared in agent frontmatter tools array - MCP server declarations in plugin.json files URL scanning is intentionally excluded — it is handled by the reference scanner (eng/reference-scanner/scan.ps1) with its known-domains.txt allowlist. Also adds Tools property to AgentInfo record and RawAgentFrontmatter to support parsing the tools array from agent YAML frontmatter. Based on dotnet/skills#337 by ManishJayaswal, trimmed to structural checks only (URL scanning superseded by #336). Co-authored-by: ManishJayaswal <ManishJayaswal@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add allowlist support for external dependency checker Introduce an allowlist file (eng/skill-validator/allowed-external-deps.txt) so known external dependencies can be pre-approved. Each finding generates a deterministic key (type:name:detail) that is checked against the list. Allowed items are silently suppressed; unrecognized entries produce warnings. The allowlist is pre-seeded with the 5 findings currently on main so the checker runs clean out of the box. Over time, entries can be removed as issues are resolved, and checks can be promoted to hard errors. Adds 7 tests covering allowlist loading and filtering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: naming, dedup, test robustness - Rename 'errors' variables to 'findings' in all Check* methods to reflect advisory (warning) semantics, not hard failures. - Rename test methods from FlagsError to FlagsWarning, NoError to NoWarning for consistency. - De-duplicate #tool: reference warnings so repeated mentions of the same tool in a skill/agent produce only one finding. - Use GUID-based temp path instead of hardcoded '/nonexistent/path.txt' in LoadAllowList_MissingFile test. - Add 2 dedup tests (skill and agent) bringing total to 379. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments: tool refs scan full content, not just body Update comments in CheckSkill and CheckAgent to accurately reflect that #tool: references are scanned in the full markdown content (including frontmatter), not just the body. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: ManishJayaswal <ManishJayaswal@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
5a199da6f6 | Fix the compilation issues with devops agentic workflows (#353) | ||
|
Dan Moseley
|
3f510a8c55 |
Fix minor review feedback on migrate-dotnet9-to-dotnet10 (#180) (#335)
* Fix post-merge review feedback on #180 - sdk-msbuild: Fix incorrect 'use single quotes' guidance for #r/#load directives — the correct fix is to remove the quotes entirely - serialization: Split dual Animal class declaration into separate before/after code blocks so each is valid C# Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-upgrade/skills/migrate-dotnet9-to-dotnet10/references/serialization-networking-dotnet9to10.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
6666c9e5e8 |
[Infra] Add validation with whole plugin loaded (#316)
* Add plugin-aware skill validation with activation detection * Address Copilot review comments (round 4) - DESIGN doc: mark Targeted Skill Activation TODO as implemented - ValidateCommand: fix misleading --plugin-dir comment - Comparator: set PluginScore to null when no plugin runs exist - ComparatorTests: copy missing fields in rehydrated ScenarioComparison - AgentRunner: fix GetPluginClient docstring, route RunAgentCore through it - AgentRunner: stage isolated skill to temp dir for true single-skill isolation - dashboard.js: fix triple chart comment to match actual series names - RunnerTests: update assertions for new isolated skill staging behavior * Remove implementation doc |
||
|
Dan Moseley
|
144b129995 |
Fix reference scanner silently passing when git diff fails (#348)
The PR-mode code path used --depth=1 when fetching the base branch, which prevented git from finding the merge-base for the three-dot diff. The git diff failure was silently swallowed because its stderr output was piped through Where-Object and filtered out, causing the scanner to report success without scanning anything. Fix: - Remove --depth=1 from git fetch (checkout already uses fetch-depth: 0) - Check LASTEXITCODE after both git fetch and git diff - Separate git diff from the Where-Object filter so exit code is captured Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7afa81f831 | Run tests supports platforms and frameworks (#343) |