skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
589 Commits 131 Branches 2 Tags
eval-exec-opus48
Commit Graph

235 Commits

Author SHA1 Message Date
leslierichardson95 7286c31118 Add skills for C# MCP Server Development (#317) 
* Add C# MCP server skills (create, debug, test, publish)

Four new skills for the C# MCP server development lifecycle:
- mcp-csharp-create: Scaffolding with dotnet new mcpserver, tools/prompts/resources, transport config
- mcp-csharp-debug: MCP Inspector, VS Code integration, breakpoint debugging, logging
- mcp-csharp-test: Unit tests with ClientServerTestBase, integration with WebApplicationFactory, evals
- mcp-csharp-publish: NuGet packaging, Docker/Azure deployment, MCP Registry publishing

Each skill includes SKILL.md with progressive disclosure references/ and eval.yaml tests.

* Fix mcp-csharp-create eval: use regex assertions for combined attribute syntax

Replace scaffolding-heavy scenarios with implementation-focused ones that
test MCP-specific features (resources, prompts, logging). Fix assertion
patterns to match combined C# attribute syntax [McpServerTool, Description()]
instead of requiring standalone [McpServerTool]. Increase timeouts to 180s
to account for skill-reading overhead.

Validator result: passed=True, improvement=44.6% (threshold=10%)

* Add CODEOWNERS entries for MCP C# skills (create, debug, publish, test)

* Update plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-publish/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remove evaluation step and guide from mcp-csharp-test skill

* Add CODEOWNERS entries for dotnet-maui skills

* Update CODEOWNERS to add @artl93 as co-owner for mcp-csharp skills

* Replace parent-directory file references with backtick skill names

Cross-skill references used ../sibling/SKILL.md paths which the
validator rejects. Replace with backtick-quoted skill names to match
the convention used by other skills in the repo.

* Add known domains for MCP server development skills

Add domains referenced by the mcp-csharp-* skills:
- github.com/modelcontextprotocol/registry
- github.com/open-telemetry/semantic-conventions
- npmjs.com, xunit.net, fluentassertions.com, nugettest.org

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update Docker commands to use placeholder for registry and image names

* moved mcp skills and their evals into dotnet-ai plugin

* cleaned up moved files to dotnet-ai

* cleaned up files

* Update .github/CODEOWNERS

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update nuget-packaging.md

* Update repository URL format in nuget-packaging.md

* Add github.com/ to known-domains to allow truncated placeholder URLs

The URL regex truncates at angle brackets (invalid in URIs), so
placeholder URLs like https://github.com/<yourusername>/<repo>
become https://github.com/ which didn't match any path-scoped entry.
Adding github.com/ (with trailing slash) covers only this case without
broadly allowing all github.com URLs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update Docker tag command to use placeholder for server name

* Update known domains and fix GitHub username placeholders in documentation

* Remove McpServerToolType from mcp-csharp-create eval

McpServerToolType is only needed with WithToolsFromAssembly(), which is
discouraged because it is not Native AOT compatible. Updated eval assertions
and rubrics to prefer WithTools<T>() instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve eval resilience for mcp-csharp-create and mcp-csharp-debug

- Increase mcp-csharp-create timeouts from 180s (or default 120s) to 360s
  for all 3 scenarios. These scenarios consistently time out because the
  model spends time on bash exploration/scaffolding before writing code.

- Add explicit timeout to scenario 3 which was relying on the 120s default.

- Broaden mcp-csharp-debug Inspector rubric to accept both the concise
  single CLI command style and the step-by-step UI configuration walkthrough,
  since both are correct approaches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update eng/known-domains.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Broaden debug 'failing tool' rubric to accept multiple debugging approaches

The rubric criterion 'Shows how to attach a debugger' was too narrow.
The skilled answer correctly focused on the #1 cause (stdout pollution)
but scored low because it didn't show a specific 'attach to process' flow.
Broadened to accept any valid debugging approach: attaching, Debugger.Launch(),
or launch.json configuration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix 4 eval failures in mcp-csharp-debug and mcp-csharp-test skills

mcp-csharp-debug:
- Trim SKILL.md verbosity (183->160 lines) and mcp-inspector.md (67->54 lines)
- Add Diagnosing Tool Errors section covering debugger, output panel, Inspector, common culprits
- Rebalance debugging narrative away from stderr-only focus
- Move HTTP logging config to ide-config.md reference

mcp-csharp-test:
- Add eval/evaluations keywords to frontmatter for activation
- Add HTTP tool invocation test pattern (tools/call via WebApplicationFactory)
- Trim test-patterns.md bloat (remove Test Categories, Coverage, Input Validation)
- Create references/evaluations.md with qa_pair format and read-only/deterministic guidance

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-25 14:17:57 -06:00
Copilot 97832189cf Simplify evaluation table in PR comments; keep full table in workflow summary (#448) 
* Initial plan

* Simplify evaluation workflow table: remove Quality (Plugin) and Agents Invoked columns from PR comment

- Add `simplified` parameter to GenerateMarkdownSummary in Reporter.cs
- When simplified=true, exclude Quality (Plugin) and Agents Invoked columns
- Update ConsolidateCommand with --summary-output for full table
- evaluation.yml: simplified table for PR comment, full table for step summary
- Update Full results link to indicate additional columns are available
- Add 3 tests for simplified table behavior

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/d2590b7a-023a-45a5-928b-9e658f40b58e

* Simplify evaluation table in PR comments; keep full table in workflow summary

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/d2590b7a-023a-45a5-928b-9e658f40b58e

* Delete .nuget/nuget.exe

* Move table simplification to workflow post-processing instead of skill-validator

Revert all C# changes (Reporter.cs, ConsolidateCommand.cs, tests). The table
simplification is now a post-processing step in evaluation.yml using a Python
script that strips "Quality (Plugin)" and "Agents Invoked" columns from the
markdown before posting as a PR comment. The full table goes to the workflow
step summary.

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/c8e0acc6-1128-4bf5-8170-382b278500e5

* Apply review suggestions: use heredoc, remove unused import, full step summary

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/be2dd88e-e30b-4c50-905c-1cff9cd21422

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
 2026-03-25 19:04:12 +01:00
Jeff Schwartz 55a7d80527 adding SIMD/TensorPrimitives to dotent-diag/analyzing-dotnet-performance (#330) 
* incorporated feedback from previous pr, iterated with 100+ scalar loop tests to come to this skill, moved from skill to reference in the performance skill

* Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Address review feedback on SIMD vectorization reference

- Check Span<T>/MemoryExtensions before TensorPrimitives (no extra dependency)
- Add all supported types (sbyte, ushort, uint, ulong, nint, nuint, char via ushort)
- TensorPrimitives: add constraint and applicable types columns (not just float/double)
- Add FusedMultiplyAdd, clarify AddMultiply vs MultiplyAdd distinction
- Prefer portable APIs over platform-specific intrinsics (allow when perf justifies)
- Fix dispatch pattern to use if/else if to avoid pessimizing small inputs
- Remove LINQ references

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* addressing more feedback

* move to the experimental plugin

* move the simd tests to exprimental

* add codeowners

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-25 10:51:28 -06:00
Amaury Levé 1d56634150 Create experimental plugin (#446) 2026-03-25 16:11:07 +00:00
Dan Moseley b39fe8541b Add evaluation troubleshooting guide for AI agents (#441) 
* Add evaluation troubleshooting guide and link from PR comments

Add InvestigatingResults.md with:
- How to find and download evaluation artifacts from workflow runs
- Full results.json schema documentation
- Common failure patterns (timeouts, empty output, variance, rubric
  issues, skill not activated) with diagnosis steps and fixes
- Example analysis script for AI agents to parse results

Link the doc from:
- Reporter.cs: show a link in the PR comment when any skill fails
- skill-validator README.md: Output section
- CONTRIBUTING.md: CI evaluation section

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update InvestigatingResults.md

* Address review feedback: fix hard-coded URLs and optional plugin run docs

- Reporter.cs: Use repo-relative link instead of hard-coded GitHub URL
- evaluation.yml: Use github.sha instead of hard-coded 'main' branch
- InvestigatingResults.md: Clarify skilledPlugin run is optional

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* THROWAWAY: add trailing newline to trigger eval workflow

This change is intentionally trivial and will be reverted before merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR #441 review feedback (round 2)

- Reporter.cs: Remove leading / from relative link
- evaluation.yml: Use jq on results.json instead of grepping for emoji
- evaluation.yml: Use needs.gate.outputs.head_sha for revision-pinned URL
- InvestigatingResults.md: Mark pluginImprovementScore/pluginBreakdown as optional
- InvestigatingResults.md: Fix skill-not-activated symptoms to use
  skillActivation fields instead of turnCount

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Use specific skillActivation field names and fix timestamp description

- Use skillActivationIsolated/skillActivationPlugin instead of generic skillActivation
- Clarify timestamp is when results were written, not run start

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix timestamp field description (was lost in prior commit)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Revert "THROWAWAY: add trailing newline to trigger eval workflow"

This reverts commit 83bd7f7601.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-25 16:10:17 +00:00
Viktor Hofer 1762a78fed Add stale PRs agentic workflow (#435) 
* Add stale PRs agentic workflow

* Add fork-guard to close-stale-prs workflow

Skip scheduled runs on forked repositories to avoid consuming fork
owner minutes and failing due to missing secrets/context, matching
the pattern used by other scheduled workflows in this repo.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-25 13:00:10 +01:00
Jan Krivanek 19b6cadddf Move token usage data into separate branch (#431) 
* Move token usage data into separate branch

* Address PR review: add help doc, fix title injection, add token-usage fallback

* Add -SkipBenchmarkData switch, clean stale files on branch push
 2026-03-24 13:19:47 +01:00
Viktor Hofer c14e63ba6c Combine evaluation.yml and evaluation-run.yml into single workflow (#430) 
* Combine evaluation.yml and evaluation-run.yml into single workflow

Inline the three jobs from the reusable evaluation-run.yml workflow
(build-validator, evaluate, comment-on-pr) directly into evaluation.yml
and delete the now-unnecessary reusable workflow file.

Changes:
- Inline build-validator, evaluate, and comment-on-pr jobs
- Eliminate 8-secret forwarding block (direct secrets.* access)
- Replace inputs.* with direct needs.* references
- Add MODEL/JUDGE_MODEL workflow-level env vars
- Fix --verbose duplication in evaluate step
- Simplify infra-change detection regex
- Update dependency chains (report-status, publish-benchmark)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR feedback: fix consolidate subcommand, improve error reporting

- Fix consolidate invocation: use 'evaluate consolidate' subcommand
  (was incorrectly changed to just 'consolidate' during inlining)
- Add build-validator to report-status needs so build failures surface
  as a deterministic failure state instead of generic error
- Add discover to comment-on-pr needs and gate on has_entries so
  /evaluate on a no-op PR doesn't post a misleading failure comment

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix matrix fallback and header comment

- Use fromJson(... || '[]') fallback for matrix entries to prevent
  errors when discover is skipped on non-evaluation events
- Update header comment to accurately describe the is_fork mechanism

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-24 11:11:51 +01:00
Viktor Hofer b92dde1854 Refactor skill-validator to vertical slice architecture (#429) 
Reorganize the skill-validator project from layer-based grouping
(Commands/, Services/, Models/, Utilities/) to feature-based slices
(Check/, Evaluate/, Consolidate/, Shared/).

Key changes:
- Check/: CheckCommand, SkillProfiler, AgentProfiler, PluginValidator,
  ExternalDependencyChecker, ReferenceScanner
- Evaluate/: EvaluateCommand, RejudgeCommand (now a subcommand of
  evaluate), AgentRunner, Judge, PairwiseJudge, and other eval services
- Consolidate/: ConsolidateCommand
- Shared/: Models, SkillDiscovery, PluginParser (extracted from
  PluginValidator), Reporter, and utilities

Split PluginValidator: parsing helpers (ParsePluginJson,
TryGetSafeSubdirectory) moved to Shared/PluginParser.cs; validation
logic stays in Check/PluginValidator.cs.

Move RawFrontmatter and RawAgentFrontmatter types from EvalSchema into
SkillDiscovery since they are only consumed by discovery.

Wire RejudgeCommand as subcommand: skill-validator evaluate rejudge.

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-24 09:41:55 +00:00
Viktor Hofer c4c26175e2 Integrate reference scanner into skill-validator CheckCommand (#422) 
* Integrate reference scanner into skill-validator CheckCommand

- Port reference scanner logic from scan.ps1 to C# ReferenceScanner service
- Add --known-domains option to CheckCommand for reference scanning
- Move known-domains.txt and allowed-external-deps.txt to eng/
- Remove standalone scan.ps1 and reference-validation.yml workflow
- Reference scanning now runs as part of skill-check.yml via skill-validator
- Add 34 unit tests for ReferenceScanner (all 463 tests pass)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR feedback: repoRoot derivation, early validation, .agents scanning, SRI policy

- Derive repoRoot from --known-domains file path instead of cwd
- Fail early with clear error when known-domains file is missing
- Add .agents/**/*.md scanning to DiscoverFiles (regression fix)
- Fix SRI policy loophole: SRI suppresses SCRIPT-NO-SRI only, domain
  and HTTPS checks still apply
- Use actual known-domains file path in error messages instead of
  hardcoded eng/known-domains.txt
- Update doc comments to reflect full set of scanned paths
- Add tests for .agents discovery and SRI domain enforcement

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove repo-specific paths from ReferenceScanner

DiscoverFiles now only scans directories it is given -- no hardcoded
knowledge of .agents, agentic-workflows, eng, or README.md. The tool
is repo-agnostic; callers control what gets scanned via --plugin dirs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Drop repoRoot from ReferenceScanner API

ScanFile/ScanFiles no longer take a repoRoot parameter. Finding paths
use the file path as provided by the caller, keeping the scanner
completely decoupled from any repo layout assumptions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Prune stale domains and add json.schemastore.org to known-domains.txt

Removed 10 entries no longer referenced by any plugin content:
agentskills.io, visualstudio.com, mcr.microsoft.com, dotnet.github.io,
github.com/dotnet/docs, github.com/dotnet/dotnet-api-docs,
github.com/dotnet/skills, github.com/github/gh-aw,
github.com/modelcontextprotocol/csharp-sdk, developers.openai.com.

Added json.schemastore.org (referenced by dotnet-template-engine).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix case-insensitive regex, update docs, add tests

- Make UrlRegex and PipeToShellRegex case-insensitive to match the
  original PowerShell behavior (HTTP://, CURL, Wget all now detected)
- Update DiscoverFiles doc comment to remove stale 'individual files' claim
- Narrow CONTRIBUTING.md scope description to match actual plugin-only scanning
- Add tests for mixed/upper-case URL and pipe-to-shell detection

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove IPv6 loopback support, fix pipe-to-shell allowlist case sensitivity

- Remove Ipv6LoopbackRegex and IPv6 test data (not needed)
- Add case-insensitive + culture-invariant matching for pipe-to-shell allowlist
- Add mixed-case allowlist test

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-23 13:05:08 +00:00
Amaury Levé 6489fbee39 Add exp-test-anti-patterns skill for test quality review (#415) 
New experimental skill that detects anti-patterns and code smells in .NET
test suites. Covers assertion quality, test isolation, naming, flakiness
indicators, over-mocking, and structural problems across MSTest, xUnit,
NUnit, and TUnit.

Key features:
- 4-tier severity classification (Critical/High/Medium/Low)
- False-positive avoidance: calibrates severity honestly and recognizes
  well-written tests without inventing problems
- Positive observations alongside findings

Evaluation results (3 runs, claude-opus-4.6):
- Flakiness/coupling detection: 3.0 -> 5.0 (isolated), 4.0 (plugin)
- Duplicates/magic values: 3.0 -> 5.0 (isolated), 4.7 (plugin)
- Well-written test recognition: 2.0 -> 5.0 (both modes)
- Mixed severity detection: 5.0 baseline maintained
- Overfitting score: 0.06 across all scenarios
 2026-03-20 14:04:18 +00:00
YuliiaKovalova dcba4f3d8e Add dotnet-template-engine plugin (#341) 
* Add dotnet-template-engine plugin

Add template-engine skills for .NET project scaffolding powered by the
DotnetTemplateMCP MCP server (v1.1.0). Includes three skills and one agent:

Skills:
- template-discovery: find, inspect, and compare templates with intent
  resolution (70+ keyword mappings)
- template-instantiation: create projects with CPM adaptation, latest NuGet
  versions, parameter validation, and multi-template composition
- template-authoring: create templates from existing projects and validate
  template.json for authoring issues

Agent:
- template-engine: expert agent that routes to the appropriate skill based
  on user intent

Also adds CODEOWNERS entries, marketplace.json registration, eval tests,
and .mcp.json for DotnetTemplateMCP server configuration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: mcpServers in plugin.json, expect_tools, path fix

- Move MCP server config into plugin.json under mcpServers so SkillValidator
  discovers the template_* tools during evaluation runs
- Add expect_tools constraints to all eval scenarios to verify actual MCP
  tool invocation instead of relying on rubric text alone
- Fix frontmatter path: template.config → .template.config (leading dot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update DotnetTemplateMCP to v1.2.0

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Include 'dotnet new' in agent description for discoverability

Address review feedback from danmoseley: the frontmatter description
is how the caller decides whether to load the agent tokens, so it
needs to mention 'dotnet new' as a trigger keyword.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove MCP server configuration and references

- Delete .mcp.json MCP server config file
- Remove mcpServers section from plugin.json
- Remove MCP/DotnetTemplateMCP references from agent, skills, and marketplace descriptions
- Skills and eval files remain unchanged

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove ghost tool references and add CLI fallback guidance

- Remove references to 4 non-existent tools: template_compose,
  template_suggest_parameters, template_validate, solution_analyze
- Replace with real alternatives (template_inspect, manual review,
  sequential template_instantiate, direct file inspection)
- Add CLI fallback guidance so LLMs degrade gracefully when
  template tools are unavailable
- Fix authoring eval to not expect template_validate tool

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: CODEOWNERS naming, claude marketplace, README

- Rename CODEOWNERS header to dotnet-template-engine for consistency
- Mirror plugin entry in .claude-plugin/marketplace.json
- Add plugin to README What's Included table

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix PR review comments: align eval rubric with expect_tools and sync marketplace.json

- Updated template-discovery eval.yaml rubric to match expect_tools
  (removed 'or template_search' since expect_tools enforces template_from_intent)
- Made .claude-plugin/marketplace.json byte-for-byte identical to
  .github/plugin/marketplace.json (removed extra trailing newline)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review comments: strengthen eval assertions and fix rubrics

- Add exit_success assertions to all eval scenarios
- Add file_exists and file_contains assertions for template-instantiation
- Add output_contains for shortName and datatype in template-authoring
- Fix rubrics to reference actual tools (skill + dotnet new CLI) instead
  of non-existent MCP tools (template_from_intent, template_inspect,
  template_instantiate) which caused the judge to penalize quality scores
- Update expect_tools from non-existent MCP tools to 'skill'
- Change plugin version from 1.2.0 to 0.1.0 for consistency with other plugins
- Remove extra blank line in README.md after plugin table

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Rewrite skills and agent to use dotnet new CLI instead of phantom MCP tools

The SKILL.md files and agent.md referenced MCP tools (template_from_intent,
template_inspect, template_instantiate, template_dry_run, etc.) that are
not available at runtime since no MCP server is declared in plugin.json.
This caused the agent to waste tokens attempting to find these tools before
falling back to dotnet new CLI commands.

Changes:
- template-discovery/SKILL.md: Replace MCP tool calls with dotnet new
  search, dotnet new list, dotnet new --help, and dotnet new --dry-run.
  Add intent-to-template keyword mapping table.
- template-instantiation/SKILL.md: Replace template_instantiate with
  dotnet new CLI commands. Add explicit CPM adaptation steps and
  multi-project composition with dotnet sln/dotnet add reference.
- template-authoring/SKILL.md: Replace template_create_from_existing
  with manual template.json creation guidance including example JSON.
  Replace template_dry_run/template_instantiate with CLI equivalents.
- template-engine.agent.md: Replace Available Tools table with CLI
  Commands Reference. Update all workflow steps to use CLI commands.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix file assertion paths: remove ./ prefix for glob matcher compatibility

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
 2026-03-18 22:51:20 +01:00
Jan Krivanek 30c19eae13 Add tokens usage dashboard and data collection (#385) 2026-03-18 20:04:46 +00:00
dependabot[bot] 8e27c14ea4 Bump actions/download-artifact in the github-actions-dependencies group (#396) 
Bumps the github-actions-dependencies group with 1 update: [actions/download-artifact](https://github.com/actions/download-artifact).


Updates `actions/download-artifact` from 8.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
 2026-03-18 09:58:46 +00:00
Viktor Hofer c5435dd3da Make evaluate a subcommand with tests-dir as required argument (#398) 
* Make evaluate a subcommand with tests-dir as required argument

Change EvaluateCommand from RootCommand to a regular Command('evaluate')
so it is a subcommand like check, consolidate, and rejudge. Convert
--tests-dir from an optional Option to a required positional Argument
placed after the skill paths.

Update .csproj RunArguments, evaluation-run workflow, README, and
CONTRIBUTING.md to reflect the new CLI syntax.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Default --parallel-skills/scenarios/runs to 3

Update the default from 1 to 3 for all three parallelism options and
remove the now-redundant overrides from RunArguments in the .csproj.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Replace RunArguments with launchSettings.json, publish in skill-check

Add Properties/launchSettings.json with default evaluate args for local
development. Remove RunArguments from .csproj. Update skill-check.yml to
publish the validator and invoke the binary directly instead of using
dotnet run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Keep --tests-dir as a required option, not positional

A variadic paths argument greedily consumes all positional values,
making a trailing positional tests-dir impossible to parse. Keep
--tests-dir as a named required option which matches the previous UX.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move testsDirOpt after resultsDirOpt, drop evaluate from CONTRIBUTING examples

Reorder testsDirOpt declaration and registration to follow resultsDirOpt.
Remove the evaluate subcommand from CONTRIBUTING.md dotnet run examples
since the launch profile already provides it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-18 08:59:03 +00:00
Viktor Hofer bd965cbd55 Require /evaluate command for PR evaluation (#394) 
* Require /evaluate command for PR evaluation

Update evaluation.yml to require an /evaluate command to trigger evaluation,
matching the behavior in evaluation-fork-pr.yml. This change:

- Adds pr-status job that posts initial commit status on PR open/sync:
  - 'success' if no skills changed (PR not blocked)
  - 'pending' if skills changed (maintainer must post /evaluate)
- Adds gate job to validate /evaluate command (permissions, same-repo check)
- Replaces evaluation-status check-run with report-status using commit status API
- Scheduled runs continue to work unchanged (daily evaluation of all plugins)

The evaluation-status required check still works because:
- PRs without skill changes get immediate success status
- PRs with skill changes get pending status until /evaluate is posted

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Merge evaluation and evaluation-fork-pr into single workflow

Consolidate evaluation.yml and evaluation-fork-pr.yml into one unified
workflow that handles both same-repo and fork PRs via conditionals:

- pr-status: posts initial commit status for same-repo PRs (pull_request)
- fork-pr-status: posts initial commit status for fork PRs (pull_request_target)
- gate: unified /evaluate command validation with is_fork output
- discover: unified skill discovery with conditional worktree for forks
- report-status: single job for posting final commit status
- Scheduled runs and publish-benchmark unchanged

Also fixes:
- run-evaluation now uses always() so scheduled runs aren't skipped
- report-status scoped to issue_comment events only
- Removed stale evaluation-fork-pr.yml reference from infra detection regex
- Updated evaluation-run.yml comment

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback

- Fetch PR head and use worktree for both fork and same-repo PRs in discover
- Always pass checkout-ref (PR head SHA) to evaluation-run for /evaluate
- Fix runs count: use event_name == 'schedule' instead of github.ref

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Require /evaluate for infra changes too

Expand pr-status and fork-pr-status discovery to detect evaluation workflow
and skill-validator changes in addition to skill/test changes. Both now post
pending status requiring /evaluate when infra files are modified.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Build skill-validator from PR branch for non-fork PRs

Add is-fork input to evaluation-run.yml. When false (same-repo PRs), the
validator is built from checkout-ref so skill-validator changes are tested.
When true (fork PRs), the validator is built from the base branch to prevent
untrusted code from modifying tooling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review: fix security comment, add issues:write

- Update security model comment to reflect validator-ref behavior
- Add issues: write permission to gate and report-status jobs for
  reaction and comment API calls

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Add prominent note about issue_comment trigger limitation

Changes to this workflow file in a PR won't take effect until merged
because issue_comment always runs from the default branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Don't require /evaluate for workflow YAML changes

Workflow YAML changes can't be tested via /evaluate anyway (issue_comment
always runs from main). Only eng/skill-validator/ changes now require
/evaluate in the pr-status jobs. The discover job still detects workflow
changes to trigger all-plugin evaluation when /evaluate is run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-03-18 09:15:42 +01:00
Jeff Handley 646b5d93b0 Add Skill: Convert to Central Package Management (convert-to-cpm) (#75) 
* Add convert-to-cpm skill for NuGet Central Package Management conversion

Authoring process:
- Researched the Agent Skills standard at agentskills.io and reviewed repo conventions
- Studied the existing csharp-scripts skill and test structure as a reference implementation
- Studied Microsoft's CPM documentation at learn.microsoft.com
- Authored SKILL.md with a 9-step workflow covering scope determination through validation
- Created 5 test scenarios across simple, moderate, and advanced tiers
- Performed dry-run evaluation against all scenarios, identifying 15 findings
- Applied all 15 fixes (4 gaps, 6 ambiguities, 5 improvements) to SKILL.md
- Re-evaluated: 50 OK, 0 gaps, 0 ambiguities, 1 test fixture improvement
- Updated test fixture to resolve the remaining finding
- Final validation: 254 lines, 0 errors, 0 warnings

Skill implementation (skills/convert-to-cpm/SKILL.md):
- 9-step workflow: scope, CPM check, audit, create/update props, update projects, handle properties, restore/validate, cleanup, summary
- Handles single projects, solutions, and full repositories
- Audits packages with per-project detail table including version conflicts and security advisories
- Flags CVE/security vulnerabilities and recommends patched versions
- Supports MSBuild property-based versioning with inline vs keep decisions
- Covers conditional PackageReference items across target frameworks
- Discovers PackageReference in imported .props/.targets via Import chain scanning
- Provides cross-platform commands (Unix and Windows/PowerShell)
- Documents import order requirements for property references in Directory.Packages.props
- Mentions dotnet new packagesprops as preferred creation method (.NET 8+)
- Includes validation checklist and common pitfalls table

Test scenarios (tests/convert-to-cpm/):
- simple-single-project: 1 project, 3 packages, no conflicts
- simple-solution: 3 projects via .sln, 5 packages, PrivateAssets preservation
- moderate-version-conflicts: 4 projects, System.Text.Json version conflicts with CVE-2024-43485 advisory, AutoMapper major version conflict with VersionOverride
- moderate-msbuild-properties: 2 projects using MSBuild properties for versioning in Directory.Build.props, inline vs keep decisions, property cleanup verification
- advanced-multi-complexity: 5 projects combining version conflicts, MSBuild properties, conditional TFM PackageReference, shared Common.props imports, VersionOverride, and security advisory handling

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refactor SKILL.md with references, baseline build, and package comparison

Condensed SKILL.md using progressive disclosure:
- Reduced from 254 lines / ~3937 tokens to 120 lines / ~1778 tokens
- Extracted detailed procedural instructions into 5 focused reference files
- Main file retains workflow decision logic with links to references loaded on demand
- Follows agentskills.io progressive disclosure pattern: metadata -> instructions -> resources

Added baseline build verification (new Step 2):
- Requires clean build (dotnet clean && dotnet build) before any changes
- Captures baseline.binlog for structured build artifact
- Captures baseline-packages.json via dotnet list package --format json
- Blocks conversion if baseline build fails

Added post-conversion baseline comparison (Steps 8 and 10):
- Step 8 produces after-cpm.binlog and after-cpm-packages.json from clean build
- Step 10 compares baseline vs post-conversion package lists per project
- Changes table shows version bumps, VersionOverride entries, added/removed packages
- Unchanged table confirms version-neutral conversion for unmodified packages
- Binlog files produced as artifacts for user review in MSBuild Structured Log Viewer

Reference files created:
- references/audit-complexities.md: 7 complexity categories for package auditing
- references/baseline-comparison.md: dotnet list package comparison procedure, binlog production, artifact guidance, table formats for changes and unchanged packages
- references/directory-packages-props.md: File creation (dotnet new packagesprops), placement rules, conditional versions, VersionOverride patterns
- references/msbuild-property-handling.md: Property usage search, inline vs keep decisions, import order requirements, cleanup verification
- references/validation-and-errors.md: NuGet error codes (NU1008, NU1010, NU1507), clean build validation, multi-TFM guidance, common pitfalls

Workflow is now 10 steps (was 9):
1. Determine scope
2. Establish baseline build (new)
3. Check for existing CPM
4. Audit package references
5. Create or update Directory.Packages.props
6. Update project files
7. Handle MSBuild version properties
8. Restore and validate (now with binlog + package list capture)
9. Clean up obsolete properties
10. Summary and baseline comparison (expanded with diff tables)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Replace third-party packages with Microsoft equivalents, add baseline/comparison steps, add .gitignore for test solutions

Package replacements across all skill references and test scenarios:
- Serilog.AspNetCore -> OpenTelemetry.Extensions.Hosting 1.15.0
- Serilog (standalone) -> OpenTelemetry.Extensions.Hosting 1.15.0
- Swashbuckle.AspNetCore -> Microsoft.AspNetCore.OpenApi 8.0.24
- SerilogVersion MSBuild property -> OTelVersion
- DIVersion updated from 8.0.1 to 9.0.0 (DI/Abstractions alignment)
- Microsoft.AspNetCore.OpenApi uses 8.0.24 (not 9.x) for net8.0 compat

Test fixture improvements:
- Added Step 2 (baseline build with binlog) to all 5 with-skill.md files
- Added Step 10 (package comparison table + binlog artifact links) to all 5 with-skill.md files
- Expanded simple-solution without-skill.md with more realistic agent behavior
- Standardized security fix terminology: table markers cite CVEs, prose references CVE IDs
- Removed space-padded table formatting in favor of compact markdown with emojis

Test solution infrastructure:
- Added tests/convert-to-cpm/.gitignore to exclude src/ build artifacts
- Added generation prompt to test README for recreating buildable solutions
- All 5 test solutions verified to restore and build successfully

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback on terminology, placement, and reference accuracy

- Rename Step 10 and validation checklist from 'binlog comparison' to 'package list comparison'
- Rewrite baseline-comparison.md: title to 'Baseline Comparison', lead with dotnet list package as primary comparison mechanism, position binlogs as supplementary artifacts for manual review/troubleshooting
- Add brief definition of binlogs before the artifacts section
- Fix Directory.Packages.props placement guidance: use first common ancestor of all .NET projects instead of assuming repository root (.git), since many repos nest source under src/ or similar
- Update SKILL.md scope input and Step 1 to reference directory-scoped conversion
- Standardize security fix terminology: table markers cite CVEs, prose uses 'security fix for CVE-...'

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review findings: sort PackageVersion, add packages.config scenario, fix commands

Review findings addressed:
1. Fixed BlobsVersion grep step description in advanced scenario
2. Sorted PackageVersion entries alphabetically in all Directory.Packages.props outputs
3. Consolidated baseline-comparison.md to single-table format
4. Added solution file arguments and multi-line commands in test fixtures
5. Added Directory.Packages.props placement question for single-project scenario
6. Created simple-packages-config test scenario (packages.config rejection)
7. Use dotnet new packagesprops in simple-single-project workflow
8. Fixed More Info link text to match reference file title

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix test scenario accuracy: sln/slnx mix, package counts, ImplicitUsings

- Use mix of .sln and .slnx across scenarios (simple-solution and
  moderate-msbuild-properties use .sln; moderate-version-conflicts and
  advanced-multi-complexity use .slnx)
- Fix simple-solution package count from 5 to 6
- Fix advanced user response #2 to reference Azure.Storage.Blobs version
- Add ImplicitUsings to advanced Directory.Build.props in audit and cleanup
- Complete the advanced unchanged table with all expected package entries

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add skill evaluation section to test README

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Inventory.sln references and make evaluation opt-in

- Fix 4 occurrences of Inventory.sln to Inventory.slnx in
  moderate-version-conflicts/with-skill.md for consistency with the
  scenario README and test solution
- Update test README to only run skill evaluation when the user
  explicitly requests it

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Condense SKILL.md with progressive disclosure

- Remove duplicated bash code blocks from Steps 2 and 8, linking to
  baseline-comparison.md instead
- Merge Steps 7 (MSBuild property decisions) and 9 (cleanup) into a
  single step since both reference msbuild-property-handling.md
- Condense Step 10 summary to reference baseline-comparison.md for
  procedure details
- Reduces from 120 to 97 lines (10 steps to 9 steps)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Enterprise.sln references to Enterprise.slnx in with-skill.md

Address code review feedback: fix 4 occurrences of Enterprise.sln to
Enterprise.slnx in advanced-multi-complexity/with-skill.md for
consistency with the scenario README and test solution.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve convert-to-cpm skill: preserve whitespace, save summary report, fix links

- Instruct Step 6 to retain each file's existing indentation style and
  blank lines to keep diffs minimal
- Instruct Step 9 to save the full summary as convert-to-cpm.md alongside
  binlog and JSON artifacts, suggesting its contents for a PR description
- Strip en-us/ from learn.microsoft.com links in SKILL.md and
  validation-and-errors.md to make them culture-agnostic
- Update all with-skill.md test expectations to include the convert-to-cpm.md
  create step and artifact mention

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Indicate that the binlog viewer is for the user; not the tool/skill

* Move convert-to-cpm skill and tests to src/dotnet-msbuild layout

Move skill from skills/convert-to-cpm/ to src/dotnet-msbuild/skills/convert-to-cpm/
and tests from tests/convert-to-cpm/ to src/dotnet-msbuild/tests/convert-to-cpm/
to match the repository layout established on main.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Convert tests from with/without-skill format to eval.yaml

Replace manual test approach (README.md + with-skill.md + without-skill.md)
with automated eval.yaml scenarios containing assertions and rubric items.
Add committed test fixture files (.csproj, .sln, .slnx, .props) for each
scenario with copy_test_files setup, replacing the git-ignored generated
solutions.

6 scenarios converted:
- simple-single-project: basic single project CPM conversion
- simple-solution: multi-project solution conversion
- simple-packages-config: negative test (packages.config rejection)
- moderate-version-conflicts: version conflict resolution
- moderate-msbuild-properties: MSBuild property version handling
- advanced-multi-complexity: combined complexities (conflicts, properties,
  conditionals, shared props)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Consolidate eval.yaml files into single root eval.yaml

Move all 6 scenario definitions from individual subdirectory eval.yaml
files into a single src/dotnet-msbuild/tests/convert-to-cpm/eval.yaml
file, matching the skill-validator discovery convention that expects
{testsDir}/{skillname}/eval.yaml. Scenario prompts updated to reference
subdirectory-prefixed paths since copy_test_files now copies all fixture
subdirectories.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reorganize convert-to-cpm into plugins/tests folders

* Add post-conversion reporting rubrics to CPM eval scenarios

Add rubric items for baseline-vs-result comparison reporting, binlog
capture and guidance, version reconciliation, and risk/reassurance
messaging across all conversion scenarios. Increase complex scenario
timeout to 360s.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve post-conversion reporting, conflict resolution, and skill activation

SKILL.md:
- Surface binlog capture commands explicitly in Steps 2 and 8
- Expand Step 4 with detailed version conflict presentation and
  resolution workflow
- Restructure Step 9 into 5-section post-conversion report template:
  overview, conflict resolutions, baseline comparison, risk assessment,
  and artifacts/next steps with binlog education
- Broaden activation triggers in description and When to Use to match
  update/bump/upgrade/align/sync terminology from ecosystem PRs
- Add activation for out-of-sync/conflicting/mismatched version scenarios

audit-complexities.md:
- Expand version conflict section with per-conflict presentation format,
  4 resolution options with trade-offs, and risk-level guidance

eval.yaml:
- Pre-specify conflict resolution strategy in prompts to avoid blocking
  on unanswered user prompts during single-turn evaluation
- Add 3 new scenarios testing CPM recommendation when user asks to
  update/align packages without mentioning CPM
- Update rubrics to match directed resolution strategy
- Increase complex scenario timeout to 360s

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR feedback from kartheekp-ms

- Use GitHub markdown URL for CPM docs reference (SKILL.md)
- Use dotnet package list (renamed in .NET 10) with backward compat
  note for dotnet list package (SKILL.md, baseline-comparison.md)
- Use dotnet package list --format json for auditing (SKILL.md Step 4)
- Document Directory.Packages.props sub-folder support for test
  projects with different dependencies (directory-packages-props.md)
- Remove security advisory version bumping from CPM conversion scope;
  instead surface advisories as follow-up items in the post-conversion
  report (audit-complexities.md, SKILL.md Step 9 Section 5)
- Add Follow-up items section to report template covering security
  advisories, deprecated packages, version alignment opportunities,
  and test validation (SKILL.md Step 9)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve convert-to-cpm skill activation, artifact preservation, and eval assertions

- Broaden skill description to trigger on version drift/sync scenarios and
  MSBuild property contexts even when CPM is not explicitly mentioned
- Add explicit instructions to preserve binlog and JSON artifacts (not delete)
- Make convert-to-cpm.md report a mandatory deliverable, not optional
- Replace fragile output_contains assertions (ManagePackageVersionsCentrally,
  Directory.Packages.props) with file_contains checks on the actual generated
  files across 5 conversion scenarios

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Extend timeouts and add CODEOWNERS entry

* Move convert-to-cpm skill into a new dotnet-nuget plugin

* Fix erroneous plugin msbuild/nuget plugin split change. Add dotnet-nuget to marketplace.

* Add NuGet package management skills to README

* Address PR feedback: clarify CPM sub-folder semantics, remove security-fix example, require explicit conflict strategy

- directory-packages-props.md: Clarify that sub-folder Directory.Packages.props
  files are independent and replace (not extend) the parent; require explicit
  MSBuild Import to chain files
- baseline-comparison.md: Replace security-fix upgrade example with version
  conflict alignment example, consistent with audit guidance that security
  advisories are follow-up items
- SKILL.md: Remove default 'use highest version' conflict strategy; require
  explicit user choice or confirmation when conflicts are detected

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Extend eval timeouts for scenarios hitting time limits

- Convert single project: 180s -> 270s
- Recommend CPM (version conflicts): 180s -> 270s
- Recommend CPM (complex repo): 360s -> 540s
- Convert complex repository: 360s -> 540s

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add missing CODEOWNERS for dotnet-nuget plugin. Reorder convert-to-cpm evals and expand timeouts.

* Remove containing skill mentions from reference files; fix ordering

* Replace output_matches with deterministic file assertions in eval.yaml

Convert 6 non-deterministic output_matches assertions to 8 file-based
assertions (file_contains, file_not_contains) across conversion scenarios:

- Scenario 5: Verify PrivateAssets preserved in Tests.csproj
- Scenario 6: Verify OTelVersion/DIVersion removed from Directory.Build.props
- Scenario 7: Verify System.Text.Json and Azure.Identity in Directory.Packages.props
- Scenario 8: Verify System.Text.Json in Directory.Packages.props and
  BlobsVersion/HostingVersion removed from Directory.Build.props

Recommendation-only scenarios and conversational assertions (conflict
detection, conditionals, Common.props awareness) remain as output_matches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reorganize skill description with USE FOR / DO NOT USE FOR markers

Restructure the description to improve activation reliability by
adopting the USE FOR / DO NOT USE FOR pattern used by other skills
in the repo (thread-abort-migration, binlog-failure-analysis).

Key changes:
- Front-load activation cues before capability list
- Add explicit USE FOR / DO NOT USE FOR section markers
- Add .sln/.slnx file extensions for solution-path matching
- Move DO NOT USE exclusions into description (was only in body)
- Use YAML folded scalar for multi-line readability

All existing activation keywords preserved. Length reduced from
882 to 849 chars (max 1024).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix issues for external link references

* Increase timeouts

* Handle packages.config non-activation in convert-to-cpm skill

Set expect_activation: false on the packages.config eval scenario so
non-activation is reported as expected rather than flagged as a warning.

Add a packages.config guard in Step 1 of the skill workflow so that if
the skill is activated for a packages.config project, it stops early and
recommends migrating to PackageReference first.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove leading blank lines from .sln test files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Use imperative 'Read' instead of 'See' for reference file links

Ensures the agent explicitly loads and reads reference files rather than
treating them as optional 'see also' links. This aligns with the repo
convention of explicitly instructing agents to load reference content.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Simplify dotnet-nuget CODEOWNERS

* Simplify packages.config guard

* Add minimal Program.cs entrypoints to Web/Worker test fixtures

Test fixture projects using Microsoft.NET.Sdk.Web and
Microsoft.NET.Sdk.Worker require a Program.cs entrypoint for dotnet build
to succeed. The skill workflow instructs the agent to run dotnet restore
and dotnet build for validation, so these fixtures must be buildable.

Adds minimal entrypoints (with explicit using directives since
ImplicitUsings is not enabled in these projects) to 6 projects:
- simple-solution/Web
- moderate-version-conflicts/Api, Worker
- moderate-msbuild-properties/Api
- advanced-multi-complexity/Api, Web

Also adds Microsoft.Extensions.Hosting package reference to the Worker
project, matching what the dotnet new worker template produces.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
 2026-03-18 08:41:57 +01:00
Copilot 0bb3606942 Fix release job skipped by transitive dependency chain from discover (#395) 
* Initial plan

* Fix release job being skipped due to tainted dependency chain from discover

The release job was always skipped for schedule and workflow_dispatch
events because the discover job (a transitive dependency via build) is
skipped for non-PR events, which taints the dependency chain. Without
always() in its if condition, GitHub skips release without evaluating
the expression.

Add always() so the if expression is always evaluated, and add an
explicit needs.build.result == 'success' guard so the release only
runs when the build actually passed.

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
 2026-03-17 13:30:30 -07:00
Viktor Hofer 1a20e39a8d Refactor skill-validator: add check command, rename validate to evaluate (#388) 
* Refactor skill-validator: add check command, rename validate to evaluate

- Add new 'check' command for CI skill validation (frontmatter, structure)
- Rename ValidateCommand to EvaluateCommand to clarify its purpose
- Add skill-check.yml GitHub Actions workflow
- Refactor SkillDiscovery for improved skill detection
- Simplify OverfittingJudge and SkillProfiler
- Update tests to match refactored APIs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix check subcommand: use EvaluateCommand as root command

System.CommandLine 2.0.4 requires the primary command to be the
RootCommand. Restores the upstream pattern where the evaluate/validate
command is the root and check/consolidate/rejudge are subcommands.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Pass all plugins in single check invocation

Build plugin args dynamically and pass them all at once instead of
looping per plugin.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review comments

- Remove 'evaluate' subcommand token from README, evaluation-run.yml,
  and RunArguments in csproj (evaluate is now the root command)
- Wrap ParsePluginJson in try/catch for JsonException with clean error
- Add CheckCommandTests with aggregate description limit coverage
  (under/at/over limit and multi-plugin cases)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-17 17:00:47 +01:00
Amaury Levé c55e4f222b Add MSTest migration skills (v1/v2→v3, v3→v4) (#377) 
* Add MSTest migration skills (v1/v2→v3, v3→v4)

New dotnet-test plugin with two step-by-step migration skills:

- migrate-mstest-v1v2-to-v3: covers assembly ref (v1) and NuGet (v2) projects,
  assertion overload changes, DataRow constructors, .testsettings replacement,
  timeout unification, and MSTest.Sdk adoption.
- migrate-mstest-v3-to-v4: covers Execute→ExecuteAsync, CallerInfo constructors,
  ClassCleanupBehavior removal, TestContext.Properties generic change, Assert API
  changes, ExpectedExceptionAttribute removal, dropped <net8.0 TFMs, and
  behavioral changes (DisableAppDomain, TestCase.Id, TreatDiscoveryWarningsAsErrors).

Includes eval.yaml test scenarios and fixtures for both skills, plugin.json,
and CODEOWNERS entries.

* Fixes

* Improve results

* Improve

* Improve description

* Copy fix for path

* Bump timeout
 2026-03-17 14:25:45 +01:00
Amaury Levé d5d2136157 Update code owners to use team alias (#382) 2026-03-17 08:13:48 +01:00
Amaury Levé 2a8388dd71 Add VSTest to MTP migration skill (#373) 
* Add VSTest to MTP migration skill

New migrate-vstest-to-mtp skill in the dotnet-test plugin covering:

- Framework-specific MTP runner enablement (MSTest, NUnit, xUnit.net)
- Directory.Build.props centralization of OutputType, runner properties
- MSTest.Sdk UseVSTest removal
- NUnit3TestAdapter 5.0.0+ upgrade requirement
- xunit v2 to xunit.v3 upgrade requirement
- dotnet test CLI argument translation (--logger to --report-trx, --collect
  to --coverage, --blame-crash to --crashdump, --blame-hang to --hangdump)
- xUnit.net --filter syntax migration to --filter-class/--filter-trait/
  --filter-query
- global.json native MTP mode (.NET 10+) vs TestingPlatformDotnetTestSupport
  VSTest-bridge mode (.NET 9 and earlier)
- Azure DevOps VSTest@3 to DotNetCoreCLI@2 pipeline migration
- MTP extension NuGet packages (TrxReport, CrashDump, HangDump, CodeCoverage)
- Exit code 8 behavioral difference (--ignore-exit-code 8,
  TESTINGPLATFORM_EXITCODE_IGNORE)

Includes 10 eval scenarios with tiered timeouts (240-360s), 5 fixture
directories, plugin.json, marketplace.json entries, CODEOWNERS, and
README.md updates.

* Fix codeowners

* Improve

* Improve further

* Improve prompt

* Improve further the description

* Address review comments
 2026-03-16 23:56:48 +01:00
Amaury Levé 45c66f87d4 Add prompt to evaluate skills description (#371) 2026-03-16 13:14:55 +01:00
Jan Krivanek 93ccead1e7 Adjust for timeout issues (#354) 2026-03-15 21:08:51 +00:00
Amaury Levé 1b8f2aa1e4 Address Copilot review comments on test agent/skills (#351) 
Follow-up from PR #343 where copilot added review after the PR was merged
 2026-03-15 06:45:40 +00:00
Dan Moseley 1ccd28b1ba Make publish-benchmark resilient to individual plugin eval failures (#365) 
* Make publish-benchmark resilient to individual plugin eval failures

When any single evaluate matrix leg fails (e.g. OOM kill), the entire
run-evaluation reusable workflow reports failure. Previously this caused
publish-benchmark to be skipped, blocking dashboard updates for ALL
plugins -- even the ones that succeeded.

Changes:
- publish-benchmark: use !cancelled() so it runs when some legs fail
- download-artifact: continue-on-error so missing artifacts don't abort
- The existing Generate Benchmark Data step already handles missing
  plugin results with Write-Warning and continue

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Make continue-on-error conditional on evaluation result

Only tolerate artifact download failures when run-evaluation didn't
fully succeed. When all legs passed, a download failure is a real error
that should fail the job.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-14 07:46:40 +01:00
Dan Moseley 492ab1d174 Reduce scheduled evaluation from every 3h to daily (#362) 
* Reduce scheduled evaluation from every 3h to daily, skip if no changes

The evaluation schedule was consuming ~/5 days in Copilot premium
requests from scheduled runs alone (~/month pace). This change:

1. Reduces the cron schedule from every 3 hours (8 runs/day) to once
   daily at 08:00 UTC. This alone saves ~87% of scheduled evaluation
   cost.

2. Adds a check-for-changes guard that skips the scheduled run entirely
   if there have been no new commits to main since the last successful
   scheduled evaluation. This avoids wasting tokens re-evaluating
   identical code.

The PR trigger and all PR evaluation behavior is unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: harden skip-if-no-changes guard

- Add actions: read permission so the workflow run lookup succeeds
- Use status=completed with jq conclusion filter instead of status=success
- Make all gh api calls non-fatal (fallback to has_changes=true on error)
- Make compare API call best-effort for logging only

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Scope actions:read to discover job, increase per_page to 100

- Move actions:read from workflow-level to discover job permissions
  for least-privilege (only the schedule guard needs it)
- Increase per_page from 10 to 100 so the skip guard finds the last
  successful run even after long failure streaks

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Key skip guard off most recent completed run, not just successful

Only skip when the most recent completed scheduled run was successful
AND at the same SHA. If it failed/cancelled at the current SHA, proceed
to retry. This prevents the guard from skipping indefinitely after a
transient failure.

Also simplifies to per_page=1 since we only inspect the latest run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix jq filter to yield empty when no prior runs exist

Use // empty to guard against null when workflow_runs array is empty,
preventing 'null null' from leaking into LAST_SHA/LAST_CONCLUSION.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Bypass skip guard on manual reruns (run_attempt > 1)

When a maintainer clicks 'Re-run jobs' on a scheduled run, the event
is still schedule with the same SHA. Without this check, the guard
would skip the rerun if the previous attempt succeeded. Now reruns
always proceed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-14 02:24:47 +00:00
Dan Moseley 74c4a1c7ac Fix PowerShell parsing of three-dot diff range in reference scanner (#359) 
When pwsh parses git diff --name-only "origin/main"...HEAD, the
partially-quoted argument is tokenized incorrectly, causing git to
receive malformed arguments and print its usage message.

Build the range string in a variable first so it is passed to git as
a single token.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-13 12:13:34 -06:00
Dan Moseley 06d2921a5b Add workflow to warn on fork PRs touching infrastructure (#356) 
Adds a pull_request_target workflow that posts an advisory comment
when a fork PR is opened that modifies eng/ or .github/ files,
suggesting the author recreate the PR from an upstream branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-13 10:28:35 -06:00
Jan Krivanek 5a199da6f6 Fix the compilation issues with devops agentic workflows (#353) 2026-03-13 10:04:28 +01:00
Dan Moseley 144b129995 Fix reference scanner silently passing when git diff fails (#348) 
The PR-mode code path used --depth=1 when fetching the base branch,
which prevented git from finding the merge-base for the three-dot diff.
The git diff failure was silently swallowed because its stderr output
was piped through Where-Object and filtered out, causing the scanner
to report success without scanning anything.

Fix:
- Remove --depth=1 from git fetch (checkout already uses fetch-depth: 0)
- Check LASTEXITCODE after both git fetch and git diff
- Separate git diff from the Where-Object filter so exit code is captured

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-12 16:34:56 -06:00
Amaury Levé 7afa81f831 Run tests supports platforms and frameworks (#343) 2026-03-12 21:29:37 +01:00
Abhitej John bb974f7360 Security Update (#331) 
* Security Update

Identified and remediated 15 vulnerabilities across the skill-validator
runtime, dashboard, and CI/CD pipelines.

Changes:
- Runtime (AgentRunner.cs): path traversal prevention, deny-by-default
  permissions, env scrubbing, MCP server hardening
- Regex (AssertionEvaluator.cs): ReDoS timeout
- Logging (SkillDiscovery.cs): exception logging
- Dashboard (dashboard.html, dashboard.js): SRI hash, XSS escaping
- CI/CD (5 workflow files): SHA pinning, persist-credentials: false
- Tests (RunnerTests.cs): 45+ new security test cases

MCP Server Defense-in-Depth (5 layers):
1. Command allowlist: only dotnet, node, npx, python, python3, uvx
2. Path rejection: no /, \, or .. in command names
3. Args validation: per-runtime dangerous flag blocklist
4. Env sanitization: strips PATH, LD_PRELOAD, NODE_OPTIONS,
   DOTNET_STARTUP_HOOKS, etc.
5. Cwd dropped: custom working directories not passed to SDK

Subprocess Isolation:
- Sensitive env vars + prefix patterns scrubbed from child processes
- 120-second timeout with Kill(entireProcessTree: true)
- Deny-by-default permission model

Supply Chain:
- All GitHub Action references pinned to full commit SHAs
- All checkout steps set persist-credentials: false
- Chart.js CDN pinned to v4.4.7 with SRI integrity hash

* Update .github/workflows/skill-validator.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update eng/skill-validator/src/Services/AgentRunner.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update eng/skill-validator/src/Services/AgentRunner.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests to use fullCommandText field name

Update ExtractsCommandProperty and PrefersPathOverCommand tests to use
the fullCommandText property instead of command, matching the updated
permission request schema in AgentRunner.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add 'dnx' to allowed MCP commands

* Address PR review feedback: security hardening fixes

- CheckPermission: normalize paths with TrimEndingDirectorySeparator to
  prevent double-separator false denials when dirs have trailing slash
- SanitizeMcpArgs: reject prefix forms (-econsole.log) and --flag=value
  variants that bypass exact-match blocklist checks
- Setup file path traversal: normalize canonicalWorkDir and
  canonicalSkillPath to avoid double-separator in StartsWith checks
- IsAllowedMcpCommand: use GetFileName instead of
  GetFileNameWithoutExtension to block dotnet.bat/node.cmd; only strip
  .exe on Windows

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* prune

* fix

* Fix concurrency setting

* Add binlog generation and upload to skill-validator workflow

Produce binary logs for build, test, and pack steps using /bl: flag.
Upload binlogs as artifacts (with if: always()) for build diagnostics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve CheckPermission diagnostics and fix AOT build error

Log allowed directories alongside denied paths to help diagnose
permission mismatches. Replace JsonSerializer.Serialize with manual
string building to fix IL3050 AOT compilation error.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Extract fileName from permission requests

Some permission requests use 'fileName' instead of 'path'. Handle
this property to avoid false denials for file-level operations.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* nownar nu1903 temporarily

* Log config dir in verbose mode alongside work dir

Include skilled/baseline label for consistency with the work dir log.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add skilled/baseline label to permission denial logs

Helps identify which run type triggered the denial when debugging.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix empty runLabel brackets and add labels to judge permission checks

Conditionally format the label suffix to avoid empty () when no
runLabel is supplied. Pass 'judge' and 'pairwise-judge' labels from
the respective judge call sites.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Allow pairwise judge to read from skilled work directory

The pairwise judge compares baseline and skilled runs in one session,
but previously only allowed reads from the baseline work directory.
Add additionalAllowedDirs parameter to CheckPermission and pass the
skilled work directory through PairwiseJudgeOptions.SkilledWorkDir.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix tests to match allow-by-default for unrecognized permission requests

CheckPermission was changed to allow requests with no extractable
path/command (allow-by-default), but four tests still expected the
old deny-by-default behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove NoWarn

* Pin discover job checkout action to full commit SHA

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
 2026-03-12 18:00:55 +00:00
Dan Moseley 64464e87e3 Add external reference scanner for skill/agent content (#336) 
* Add external reference scanner for skill/agent content

- PowerShell scanner (eng/reference-scanner/scan.ps1) checks for:
  - http:// URLs (blocked except localhost and XML namespace URIs)
  - External <script> tags without SRI integrity attribute
  - Pipe-to-shell patterns (except dotnet install scripts)
  - URLs referencing domains not in .github/known-domains.txt
- CI workflow runs on PRs and pushes touching skill/agent paths
- CONTRIBUTING.md updated with external references policy
- README.md: fixed http:// to https:// for agentskills.io

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move known-domains.txt to eng/reference-scanner/, fix comments, add ollama.com

- Moved from .github/ to eng/reference-scanner/ (co-located with scanner)
- Updated header comments: errors not informational notices, removed misleading org-scoping language
- Added ollama.com (pre-existing reference in dotnet-ai skill)
- Updated all references in scan.ps1, CI workflow, CONTRIBUTING.md

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback

- Add ? and # as valid URL terminators in path-scoped domain matching
- Strip # from host extraction in Test-KnownDomain and Test-HttpNotHttps
- Make script/SRI regex patterns case-insensitive
- Emit error finding on file read failure instead of silent skip
- Fix help comment (two levels up, not three)
- Unify CONTRIBUTING.md error language (all checks are CI-blocking)
- Add eng/reference-scanner/** to workflow path triggers

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add cdn.jsdelivr.net to known-domains (used by dashboard after #331 adds SRI)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove cdn.jsdelivr.net from known-domains

jsdelivr serves arbitrary npm packages -- keeping it off the allowlist
means any use requires both SRI and reviewer approval (two gates).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Skip domain check for scripts with SRI, fix single-line file handling

Scripts with integrity attributes are secured by hash -- the domain
is irrelevant. This means cdn.jsdelivr.net does not need to be in
known-domains.txt as long as SRI is present.

Also wrap Get-Content in @() to handle single-line files correctly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address round-2 review feedback

- Multi-line <script> tag detection: join file content so SRI attributes
  spanning lines are correctly detected
- Safe relative path: use GetRelativePath() with fallback instead of
  bare Substring that throws for out-of-repo files
- Remove dead notice plumbing: all findings are errors now

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Scan only changed files on PRs, full scan on push to main

PRs scan only changed .md/.html files so pre-existing issues on main
don't block unrelated PRs. Push to main runs the full -All scan.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-12 10:43:19 -06:00
Jason Malinowski 59241eabcf Add the C# LSP configuration to the 'dotnet' plugin (#298) 2026-03-11 17:23:11 -07:00
Jeff Schwartz 551461a926 dotnet-ai-ml (#177) 
* moved from private to public fork

* add code owner

* accomodate feedback

* update eval to make more natural language

* address eval feedback

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* Update plugins/dotnet-ai/skills/technology-selection/SKILL.md

Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>

* update supporting documentation of adding a new plugin

* update claude-plugin

---------

Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-11 15:06:04 -07:00
Dan Moseley 3923c078b9 Add migrate-dotnet9-to-dotnet10 skill (#180) 
* Add migrate-dotnet9-to-dotnet10 skill

Adds a new migration skill that guides upgrading .NET 9 projects to .NET 10,
covering TFM updates, NuGet package upgrades, and source-breaking changes
across the full .NET 10 surface area.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback: fix eval/reference consistency and CODEOWNERS

- Fix ExecuteUpdateAsync rubric: use 'regular lambda/delegate instead of
  expression tree' instead of incorrect 'Func instead of Expression'
- Fix config null binding eval prompt: .NET 9 bound null as empty string,
  not preserving constructor default
- Clarify .NET 10 null binding overwrites constructor default in reference
- Add @ViktorHofer as CODEOWNER for migrate-dotnet9-to-dotnet10

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Suffix reference filenames with -dotnet9to10 to avoid collisions

Rename all reference files under migrate-dotnet9-to-dotnet10/references/
from e.g. efcore.md to efcore-dotnet9to10.md so they don't collide with
reference files from other version migration skills. Update all
references in SKILL.md to match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address BrennanConroy review: ForwardedHeaders KnownIpNetworks example and IApiEndpointMetadata guidance

- Show KnownIpNetworks property with UseForwardedHeaders context in example
- Add IApiEndpointMetadata explanation for cookie login redirect behavior
- Link to aspnetcore#62883 for details on influencing the behavior

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Shorten skill description to fit 1024-char limit

Reduce description from 1,370 to 1,017 characters by trimming
redundant phrasing while preserving all key search terms.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update CODEOWNERS

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-11 14:06:00 -06:00
Dan Moseley db33912b98 Add migrate-dotnet10-to-dotnet11 skill (#181) 
* Add migrate-dotnet10-to-dotnet11 skill (Preview 1 coverage)

Adds a new migration skill that guides upgrading .NET 10 projects to .NET 11,
covering TFM updates, NuGet package upgrades, and breaking changes across
the .NET 11 Preview 1 surface area.

Sources:
- https://learn.microsoft.com/en-us/dotnet/core/compatibility/11
- https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-11.0/breaking-changes
- https://learn.microsoft.com/en-us/dotnet/csharp/whats-new/breaking-changes/compiler%20breaking%20changes%20-%20dotnet%2011
- https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotnet-11/overview

Skill coverage:
- C# 15 compiler: Span safe-context, ref readonly InAttribute, nameof(this.),
  with() parsing, dynamic &&/|| with interfaces
- Core libraries: DeflateStream/GZipStream empty payloads, MemoryStream capacity,
  TAR checksum validation, ZipArchive.CreateAsync eager loading,
  Environment.TickCount consistency
- EF Core 11: Cosmos DB sync I/O fully removed
- Cryptography: DSA removed from macOS
- Runtime/JIT: x86-64-v2 minimum, Arm64 LSE requirement on Windows
- SDK: Mono launch target removed for .NET Framework apps

6 reference files loaded based on project type + 7 eval scenarios.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add SYSLIB0063 NamedPipeClientStream breaking change to 10→11 skill

Add documentation for the NamedPipeClientStream constructor obsoletion
(SYSLIB0063) which was merged in dotnet/runtime#120328 for .NET 11 but
not yet documented on learn.microsoft.com. The isConnected parameter
overload is obsoleted, and projects with TreatWarningsAsErrors will
fail to build.

Changes:
- core-libraries.md: Add Obsoleted APIs section with SYSLIB0063
- SKILL.md: Add to description, Step 1 assessment, Step 3 patterns

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Tune skill description and eval prompts for activation

- Add explicit migration framing to skill description to differentiate
  from csharp-scripts skill (SDK was routing to wrong skill)
- Add 'Use the migrate-dotnet10-to-dotnet11 skill' to eval prompts,
  matching real-world usage where users explicitly invoke migration skills

Eval results (3 runs each, explicit skill invocation):

  Haiku:  6/7 activated, best +1.3 (compression/TAR scenario)
  Sonnet: 7/7 activated, best +0.3 (EF Core, TFM update scenarios)

Skill activates reliably now. Quality gains are modest because the
skill covers only Preview 1 breaking changes (training data already
includes MS Learn docs). Expect larger gains as content grows with
later previews.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/migrate-dotnet10-to-dotnet11/references/csharp-compiler.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Split nameof(this.) example into before/after code blocks

Separate the breaking and fixed code into distinct blocks to avoid
ambiguity about which line to use vs remove.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback: fix code examples and global.json guidance

- SKILL.md: Show global.json as a diff edit preserving existing keys
  (rollForward, other settings) instead of replacing the entire file
- csharp-compiler.md: Fix ref readonly lambda example to use a static
  field instead of a captured local (which won't compile in a lambda)
- csharp-compiler.md: Declare missing local variable x in ref readonly
  local function example
- eval.yaml: Fix ref readonly lambda snippet to use a class with a
  static field, matching the reference file fix

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR feedback: add second CODEOWNER, add preview version to EF Core

- Add @ViktorHofer as second owner for migrate-dotnet10-to-dotnet11 and
  migrate-nullable-references paths per CONTRIBUTING.md requirements
- Add Preview 1 annotation to EF Core reference file per reviewer feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Suffix reference filenames with -dotnet10to11 to avoid collisions

Rename e.g. efcore.md to efcore-dotnet10to11.md so names won't collide
with reference files from other version migration skills. Update all
paths in SKILL.md to match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove explicit skill name from eval prompts to reduce overfitting

The eval prompts were explicitly saying 'Use the migrate-dotnet10-to-dotnet11
skill' which biases baseline runs — the agent wastes time searching for a
nonexistent skill, guaranteeing worse performance. Rephrased all 7 prompts
to describe the migration task naturally without naming the skill.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: fix with/when label, add exit_success assertions

- Rename 'with()' to 'when' in switch-expression-arm parsing label
  (SKILL.md and csharp-compiler reference) to match the actual breaking change
- Add exit_success assertion to all 7 eval scenarios to match repo convention

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move Japanese Calendar breaking change to core-libraries reference

The globalization change was in the runtime-jit reference, which is only
conditionally loaded for older-hardware deployments. Since this affects
all projects regardless of hardware, move it to core-libraries (always
loaded) so it is not missed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clarify workflow note: allow local repo inspection, forbid only external sources

The previous wording ("Do not search the filesystem") was too broad —
the agent needs to read .csproj, solution, and source files to determine
which breaking changes apply. Reword to explicitly allow local repository
inspection while forbidding web/external sources for breaking change info.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Shorten skill description to fit 1,024-char limit (762 chars)

Remove granular breaking change names (covered in body/references),
drop LOADS REFERENCES (implementation detail), remove unsupported
ASP.NET Core 11 claim (no reference docs shipped), and tighten
DO NOT USE FOR section.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update CODEOWNERS to use @dotnet/compat team instead of individual

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move 'when' parsing change from Step 4 to Step 3

The when switch-expression-arm parsing change is a source/compilation
breaking change, not a runtime behavioral change. It belongs in Step 3
(source-breaking changes) alongside other C# 15 compiler changes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-11 14:05:03 -06:00
Dan Moseley e9370fe663 Add migrate-dotnet8-to-dotnet9 skill (#182) 
* Add migrate-dotnet8-to-dotnet9 skill with references and eval scenarios

Mirrors the structure and approach of the existing migrate-dotnet9-to-dotnet10 skill.

Covers all .NET 8→9 breaking changes organized by technology area:
- C# 13 compiler (InlineArray records, iterator safe context, collection expressions)
- Core libraries (params span overloads, BinaryFormatter removal, BigInteger, TimeSpan)
- SDK/MSBuild (Terminal Logger, version requirements, .NET 7/Standard warnings)
- ASP.NET Core (HostBuilder validation, forwarded headers, middleware constructors)
- EF Core (pending migrations, explicit transactions, Cosmos DB changes)
- Cryptography (X509Certificate2 SYSLIB0057, SafeEvpPKeyHandle)
- Serialization/Networking (BinaryFormatter, JsonDocument null, HttpClientFactory)
- WinForms/WPF (security analyzers, StatusStrip, PictureBox, XmlNamespaceMaps)
- Containers/Interop (zlib removal, CET support)
- Deployment/Runtime (env var precedence, FP-to-int saturation, MonoVM deprecated)

12 eval scenarios covering all technology areas with assertions and rubrics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* BinaryFormatter: stop and ask user for replacement strategy

BinaryFormatter migration is a major architectural decision affecting data
format, backward compatibility, and stored data migration. The skill should
present options and let the user decide rather than picking a serializer.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Trim SKILL.md to delegate BinaryFormatter detail to reference

Keep SKILL.md as concise index; full guidance lives in
serialization-networking.md reference.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Trim SKILL.md: delegate fix details to references, remove BinaryFormatter duplicate

- Remove BinaryFormatter duplicate from Step 4 (already covered in Step 3)
- Delegate inline fix details for EF Core, FP saturation, HttpClientFactory to references
- Collapse niche SYSLIB0055/0056 into one-liner pointing to reference
- Shorten params span and String.Trim entries

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve eval scenarios: setup files, task-oriented prompts, expect_activation

- Add setup files (csproj, .cs, Dockerfile, etc.) to 10/12 scenarios so agent
  works with real code instead of pure knowledge questions
- Make prompts task-oriented ('Migrate this app to .NET 9') instead of
  knowledge-oriented ('What breaks?')
- Set expect_activation: false on all scenarios — this is a pure knowledge/
  reference skill with no MCP tools, so activation via SkillInvokedEvent is
  unreliable
- Increase timeouts to 420s to avoid penalizing skilled runs that do more work

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refocus eval scenarios on Opus blind spots

Replace easy scenarios (BinaryFormatter/X509) where Opus always ties with
harder ones targeting areas where the skill adds measurable value:
- Empty env var behavior on Unix (null vs empty string)
- ZipArchiveEntry UTF-8 bit flag mechanism
- FromKeyedServices no-fallback behavior
- Cosmos DB id format change (discriminator no longer embedded)
- Env var precedence reversal with concrete GC/ThreadPool impact

Keep scenarios where skill consistently helps: Cosmos DB id format,
container zlib removal, CI pipeline net7.0 warnings, WinForms PictureBox.

Increase timeouts to 420s to avoid penalizing skilled runs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add forward-chaining guidance to migrate-dotnet8-to-dotnet9 skill

Tell the agent about the migrate-dotnet9-to-dotnet10 skill so it knows
to continue the chain after completing the 8-to-9 migration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix EF Core eval scenarios to start with 8.0.0 package versions

The eval scenarios claim to test a .NET 8 to .NET 9 migration but the
starting csproj files already reference EF Core 9.0.0 packages. This
reduces test signal since the package upgrade step is already done.
Change all EF Core package versions to 8.0.0 in scenarios 4, 5, and 10.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix InlineArray eval to actually exceed 1 MiB size limit

The rubric expects the model to warn about the InlineArray 1 MiB
runtime size limit, but the example was only 8 KiB (InlineArray(1024)
with byte element). Increase to 1,048,577 bytes so the breaking change
is concretely applicable.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix non-compiling ConfigurePrimaryHttpMessageHandler in eval

The eval used a non-existent overload of ConfigurePrimaryHttpMessageHandler
with a (handler, _) => {} lambda that doesn't return a handler. Replace
with ConfigureHttpMessageHandlerBuilder which correctly casts
PrimaryHandler to HttpClientHandler -- the pattern that actually breaks
when the default switches to SocketsHttpHandler in .NET 9.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix non-compiling HttpClientFactory example in reference doc

The breaking change example used a non-existent overload of
ConfigurePrimaryHttpMessageHandler. Replace with
ConfigureHttpMessageHandlerBuilder which is the actual API that casts
PrimaryHandler and would break when the default changes to
SocketsHttpHandler. Also fix Option 2 to use the same correct API.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove unsafe KnownProxies.Clear() guidance from aspnet-core reference

Clearing KnownProxies and KnownNetworks accepts forwarded headers from
any source, enabling attackers to spoof X-Forwarded-For/Proto/Host
headers. Replace with a warning against this pattern and steer toward
registering only trusted proxy addresses.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove unsafe RedactLoggedHeaders guidance from reference doc

Disabling header redaction globally with RedactLoggedHeaders(_ => false)
logs Authorization tokens, cookies, and other credentials in cleartext.
Replace with guidance to whitelist only specific non-sensitive headers
and add an explicit warning against disabling redaction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove insecure BinaryFormatter re-enablement option from guidance

Replace the unsupported NuGet package option with a clear security
warning that BinaryFormatter deserialization enables remote code
execution and should not be re-enabled.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add security warning against suppressing WFO1000 analyzer

WFO1000 guards against insecure deserialization of WinForms control
properties. Replace the suppression option with a warning explaining
the security risk and recommend the attribute-based fix instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add security warning to CET opt-out guidance

Warn that disabling CET removes hardware-enforced control-flow
integrity (protection against ROP/JOP exploits) and should only
be done when a specific native library is confirmed incompatible.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Label EF Core warning suppressions as temporary workarounds

Add context that suppressing PendingModelChangesWarning risks silent
schema drift and that MigrationsUserTransactionWarning suppression
has transaction safety implications. Both are temporary workarounds.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Prioritize behavioral changes and add global.json rollForward guidance

Mark behavioral changes that can cause runtime exceptions with a
warning indicator and move them to the top of the list so agents
address them first. Add rollForward policy guidance to the
global.json update step to prevent SDK resolution issues.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Split BinaryFormatter code snippet into separate serialize/deserialize blocks

The combined snippet used obj before declaration, which was misleading.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix SYSLIB heading to include full range SYSLIB0054-SYSLIB0057

Previously listed only SYSLIB0054 and SYSLIB0057, omitting 0055 and 0056.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix eval rubric to match setup code: Migrate/MigrateAsync

Setup code uses synchronous Migrate(), not MigrateAsync().

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Use distinct variable names in X509CertificateLoader examples

Duplicate 'var cert' declarations would cause compilation errors if pasted as-is.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove misleading 'backported from .NET 8' from ForwardedHeaders section

The claim is confusing in an 8-to-9 migration guide context.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clarify GetXmlNamespaceMaps InvalidCastException explanation

Previous wording was logically inconsistent about the direction of the type change.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/migrate-dotnet8-to-dotnet9/references/efcore.md

Co-authored-by: Andriy Svyryd <AndriySvyryd@users.noreply.github.com>

* Fix ReadOnlySpan<char> String.Trim example and eval scenario

The code example incorrectly used ReadOnlySpan<char> as a static field,
which is impossible (ref struct). The eval scenario used a local
ReadOnlySpan<char> variable which actually compiles fine on .NET 9 GA,
so it didn't demonstrate the breaking change. Updated both to correctly
show the RC2-to-GA overload removal issue.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add -dotnet8to9 suffix to reference filenames to avoid cross-skill collisions

Other version migration skills (e.g., 9-to-10) have similarly named
reference files. Adding the suffix disambiguates them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Shorten skill description to fit 1024-char limit

Remove LOADS REFERENCES section (implementation detail already covered by
the reference loading table in the body) and tighten wording. All USE FOR
activation keywords and DO NOT USE FOR rejection keywords preserved.
Description is now 1006 characters.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback

- CODEOWNERS: Replace @ViktorHofer with @jeffschwMSFT for migrate-dotnet8-to-dotnet9
- SKILL.md: Shorten description from 1,400 to 671 chars (max 1,024)
- sdk-msbuild-dotnet8to9.md: Clarify VS 17.11 SDK targeting phrasing
- core-libraries-dotnet8to9.md: Fix cross-reference to serialization-networking-dotnet8to9.md
- serialization-networking-dotnet8to9.md: Replace 'whitelist' with 'allowlist'
- winforms-wpf-dotnet8to9.md: Rename heading to cover both Get/SetXmlNamespaceMaps
- eval.yaml: Use ReadOnlySpan<char> to properly demonstrate String.Trim RC2 breakage

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* CODEOWNERS: Use solo @danmoseley (consistent with migrate-nullable-references)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update CODEOWNERS to use @dotnet/compat team instead of individual

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: fix factual errors in references and eval

- Fix URL with %20 spaces to use hyphens (csharp-compiler ref)
- Fix DOTNET_GCConcurrent -> DOTNET_gcConcurrent casing (eval.yaml)
- Remove invalid [StructLayout] combined with [InlineArray] (eval.yaml)
- Move JSON comment outside code fence (deployment-runtime ref)
- Remove confusing preview-era sentence (winforms-wpf ref)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Revert URL change: %20 encoding is correct for this MS Learn page

The original URL with percent-encoded spaces is the actual working URL;
the hyphenated version returns 404.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve EF Core reference doc to help failing eval scenarios

- Promote HasIndex from Low to Medium Impact with crash detail and
  removal guidance (Scenario 5 missed this causing quality regression)
- Expand id format change section with data migration implications
  and HasRootDiscriminatorInJsonId mitigation (Scenario 5 rubric)
- Add concrete DateTime.UtcNow in HasData before/after example
  (Scenario 4 missed connecting HasData to pending changes exception)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add explicit HasData search action for pending model changes

The agent reads the reference doc but doesn't connect DateTime.UtcNow
in HasData to the pending changes exception. Make the SKILL.md workflow
step prescriptive: search for non-deterministic values and replace them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Andriy Svyryd <AndriySvyryd@users.noreply.github.com>
 2026-03-11 14:03:42 -06:00
Jan Krivanek 4a4306f563 Disable scheduled workflows on forks (#318) 
* Disable schedules on forks

* Reflect PR comments
 2026-03-11 10:17:01 +01:00
Viktor Hofer ea6f594295 Enable "evaluation-status" as a required status check (#320) 
* Enable evaluation as a required status check

- Remove path filters from evaluation.yml pull_request trigger so the
  workflow runs on every PR
- Add statuses: write permission and a report-status job that posts a
  commit status with context 'evaluation' for non-fork PRs
- Change evaluation-fork-pr.yml commit status context from
  'evaluation (fork PRs)' to 'evaluation' so both workflows share the
  same status context
- The 'evaluation' status can now be configured as a required check
  in the GitHub ruleset

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: handle discover failure, least-privilege perms

- Remove workflow-level statuses: write (job already declares it)
- Broaden report-status condition to handle discover failure/cancellation
  instead of leaving PRs stuck with no status
- Exclude fork PRs (discover skipped) to avoid conflicting with
  evaluation-fork-pr.yml

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Handle fork PRs in evaluation.yml for required status check

- Remove fork guard from discover so it runs for all PRs
- Add is_fork output to detect fork PRs
- Skip run-evaluation for fork PRs (no access to secrets)
- Post 'pending' status for fork PRs with skill changes, prompting
  the author to use /evaluate
- Post 'success' for fork PRs with no skill changes
- This ensures every PR gets an 'evaluation' commit status, making
  it viable as a required check without fork PRs getting stuck

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add statuses: write at workflow level (required for job-level grant)

Job-level permissions cannot exceed the workflow-level permissions.
The report-status job needs statuses: write but was getting read-only
because the workflow level didn't include it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Use pull_request_target for fork PR status posting

Fork PRs need an 'evaluation' commit status but pull_request triggers
have restricted GITHUB_TOKEN permissions for forks. Use
pull_request_target in evaluation-fork-pr.yml instead, which runs in
the base repo context with full permissions.

The new fork-status job:
- Triggers on pull_request_target for fork PRs only
- Checks if the PR changes any skills (git diff only, no code execution)
- Posts 'pending' status if skills changed (needs /evaluate)
- Posts 'success' if no skills to evaluate

evaluation.yml reverts to the original fork guard on discover (skips
fork PRs entirely). report-status excludes skipped discover results
so it only posts for non-fork PRs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review: fix header comment, use shallow checkout

- Correct security model comment: pull_request_target runs from the
  base branch, not necessarily the default branch
- Use default shallow checkout (depth 1) instead of fetch-depth: 0
- Diff against HEAD instead of base SHA since checkout is the base

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Simplify evaluation.yml: use plain check-run job instead of commit status API

Replace report-status job (which called gh api to post commit statuses)
with a plain evaluation job whose check run can be required directly.
Removes the need for statuses: write permission in this workflow.

Fork PRs are handled by evaluation-fork-pr.yml which still uses commit
statuses (necessary because issue_comment check runs are not associated
with the PR head SHA).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix fork-status: pin checkout to base SHA, use merge-base for diff

- Pin actions/checkout to the PR base SHA to avoid checking out the
  merge ref which includes untrusted fork content
- Use explicit base/head SHAs with merge-base for reliable diff,
  consistent with the non-fork workflow's discovery logic

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Rename evaluation job and context to evaluation-status

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review: least-privilege permissions, full clone for merge-base

- Add permissions: {} to evaluation-status job (evaluation.yml)
- Use fetch-depth: 0 for fork-status checkout so merge-base works
- Remove --depth=1 from PR head fetch
- Keep statuses: write at workflow level (needed by reusable workflow)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix header comment: evaluation → evaluation-status

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-11 09:48:50 +01:00
Viktor Hofer 7be01578bd Rename status job to skill-validator-status 2026-03-10 23:59:45 +01:00
Viktor Hofer 8fecbba151 Rename status job to skill-validator-status (#322) 
* Rename status job to skill-validator-status

* Update .github/workflows/skill-validator.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-10 23:58:23 +01:00
Viktor Hofer 7bd004ee9f Enable skill-validator as a required status check (#321) 
* Enable skill-validator as a required status check

- Remove path filters from pull_request trigger
- Add discover job to skip the expensive 5-platform build matrix when
  no skill-validator files changed
- Add report-status job that posts commit status 'skill-validator':
  - No relevant changes → success
  - Build passed → success
  - Build failed → failure
- Schedule and workflow_dispatch triggers unchanged (discover is
  skipped, build runs unconditionally)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Handle fork PRs with pull_request_target for status posting

Fork PRs have restricted GITHUB_TOKEN and can't post commit statuses
via pull_request trigger. Add pull_request_target trigger with a
fork-status job that posts success for fork PRs (they should never
change skill-validator).

- Add pull_request_target trigger
- Gate build to skip for pull_request_target events
- Gate report-status to non-fork PRs only
- Add fork-status job for fork PRs (always posts success)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Enable skill-validator as a required check

- Remove path filters from pull_request trigger so the workflow runs
  on every PR
- Add discover job to skip the expensive 5-platform build matrix when
  no skill-validator files changed
- Add status job that can be required in the GitHub ruleset: succeeds
  when no relevant changes or when the build passes, fails otherwise
- No commit status API calls or pull_request_target needed — just a
  plain job whose check run is required

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-10 23:48:34 +01:00
Viktor Hofer c735d84f9d Improve CODEOWNERS validation: require sufficient owners and enable as required check (#314) 
* Improve CODEOWNERS validation workflow

- Remove path filters from pull_request trigger so it can be a required check
- Add owner sufficiency validation: each skill/test folder must have either
  2+ individual owners or 1+ team in CODEOWNERS
- Update issue creation to report both missing entries and insufficient owners
- Update failure step to trigger on either validation failure

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update CODEOWNERS to add additional code owners

Added @agocke and @hoyosjs as code owners for migrate-nullable-references and dotnet-trace-collect skills.

* Update CODEOWNERS for dotnet-trace-collect

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-10 21:16:41 +01:00
Dan Moseley 70ea8cf225 Include error details in evaluation failure PR comments (#297) 
* Include error details in evaluation failure PR comments

When evaluation fails due to validation errors (e.g., description too long),
the PR comment previously only said 'Evaluation failed. View workflow run',
requiring a click-through to discover the actual error.

Two fixes:
- Reporter.cs: GenerateMarkdownSummary() now renders an Errors section for
  verdicts with FailureKind set, so the marker comment shows what went wrong.
- evaluation-fork-pr.yml: The report-status job now downloads result artifacts
  and extracts error reasons from results.json, including them directly in the
  failure comment posted to the PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* TEST: inflate build-perf-diagnostics description to trigger validation error

Temporarily makes the description 1221 chars (limit is 1024) to verify that
the new error reporting shows the validation failure in the PR comment.

This commit should be reverted after verifying the error display.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Tighten error filter to pre-eval failures and add tests

Address review feedback:
- Restrict the Errors section in markdown summary to verdicts with no
  scenarios (pre-eval failures like spec_conformance_failure), avoiding
  duplication with threshold/regression failures already shown in the table.
- Add three tests: errors section appears for pre-eval failures, is omitted
  when all verdicts pass, and is omitted for failures that have scenarios.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix error details in evaluation-run.yml post-completion-comment

The failure comment was posted by evaluation-run.yml's post-completion-comment
job (shared by both same-repo and fork PR workflows), not just by the fork PR
workflow. Apply the same artifact download + error extraction to this job so
error details appear for same-repo PRs too.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Promote eval prompt skill-name mention from warning to error

When an eval scenario prompt explicitly references the skill name,
it biases baseline runs and forces activation instead of testing
organic discovery. Move the check from warnings to errors so it
blocks evaluation via the existing profile.Errors fail-fast path.
No existing evals on main have this pattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Sanitize @ in error output to prevent mention injection

Neutralize @ characters with a zero-width space in all three locations
where error text from PR-controlled content is rendered into PR comments:
Reporter.cs, evaluation-run.yml, and evaluation-fork-pr.yml.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Distinguish errors vs warnings in evaluation failure comments

Pre-eval structural errors (spec conformance, missing eval) get a
red X icon, while eval verdict failures (below threshold, completion
regression) get a warning icon. This makes it clear which failures
are blocking issues vs expected score variance.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Only show pre-eval errors in failure comment, link to results table

The failure comment no longer repeats verdict warnings (below
threshold, completion regression) that are already visible in the
results table. It only shows pre-eval structural errors (spec
conformance failures) and links to the results table for full details.

Three cases:
- Pre-eval errors: show them + link to results table
- No errors but failed (verdict): just link to results table
- No results table: fallback with workflow run link

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove duplicate failure comment from fork workflow

The reusable evaluation-run.yml already posts failure comments when
pr-number is provided. The fork workflow was duplicating this by
posting its own failure comment, resulting in two comments per failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Revert "TEST: inflate build-perf-diagnostics description to trigger validation error"

This reverts commit 5f9056695a.

* Address review feedback: boundary-aware match, markdown escaping, safe find loop

- SkillProfiler: use boundary-aware regex for skill-name-in-prompt check
  to avoid false positives on short skill names (e.g. 'go')
- Reporter: wrap skill name in inline code and escape backticks/newlines
  to prevent markdown injection from PR-controlled content
- evaluation-run.yml: use find -print0 with while read loop to handle
  paths with spaces; match Reporter inline-code format in jq output

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clean up comments in evaluation-fork-pr.yml

Removed redundant comments regarding failure comments for fork PRs.

* Refine comments on eval prompts and skill discovery

Clarify comment regarding eval prompts and skill discovery errors.

* Update eng/skill-validator/src/Services/Reporter.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update .github/workflows/evaluation-run.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Reapply "TEST: inflate build-perf-diagnostics description to trigger validation error"

This reverts commit 331ed870c5.

* Fix test assertion to match updated markdown error format

The Reporter.GenerateMarkdownSummary method was updated in 95ede45 to wrap
both skill name and reason in a single backtick pair for better markdown
injection prevention, but the test assertion was not updated to match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update eng/skill-validator/src/Services/Reporter.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update .github/workflows/evaluation-run.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Link to newest results-table comment instead of oldest

The GitHub comments API returns oldest-first, so head -1 was linking
the failure comment to the oldest results-table marker comment rather
than the most recent run. Use tail -1 to select the newest match.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Consolidate PR comment jobs into a single comment-on-pr job

Merge comment-on-pr and post-completion-comment into one job that always
posts a new comment with the full results summary. When no result
artifacts exist (evaluate skipped or failed entirely), a short failure
message is posted instead.

This removes the find-and-update-existing-comment pattern, the separate
error extraction via jq, the PR_COMMENT_MARKER, and the comment_action
workflow output.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Guard consolidation step against missing validator binary

When result artifacts download succeeds but the validator binary
download/extract fails, the consolidate command would fail silently
(step is continue-on-error) and no PR comment would be posted. Check
that the validator executable exists before attempting consolidation,
falling through to the generic failure message otherwise.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Skip results table header when there are no scenario rows

Only emit the markdown table when at least one scenario row exists,
so error-only summaries show just the heading and errors section
without an empty table.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Display errors as bullet points in markdown summary

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Show validation errors with status emoji in heading

Use '### ❌ Skill validation errors:' heading for the errors section
to make the failure status immediately visible.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Revert "Reapply "TEST: inflate build-perf-diagnostics description to trigger validation error""

This reverts commit 59d5d8dfd4.

* Update eng/skill-validator/tests/OverfittingJudgeTests.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update .github/workflows/evaluation-run.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-10 12:51:12 +01:00
Jan Krivanek b93d6c7d15 Prevent rep health description duplication (#306) 2026-03-10 12:07:20 +01:00
Viktor Hofer b6bc33c926 Purge dashboard entries older than 14 days (#304) 
* Purge dashboard entries older than 14 days

When updating plugin JSON files, filter out Quality and Efficiency
entries whose date is older than 14 days. If all entries are purged,
delete the JSON file so it is excluded from the components.json
manifest and no longer appears in the dashboard.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move stale file cleanup to workflow level

Address review feedback: the delete-if-empty branch in the script was
unreachable because fresh entries are always appended before purging.
Instead, add a workflow step that scans all existing data files after
per-plugin generation and removes any whose entries are entirely older
than 14 days.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move stale file purge logic into the script

Add a -PurgeStaleFiles/-DataDir parameter set so the workflow calls
the script instead of duplicating the retention logic inline. The
retention window is defined once in the script.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Make retention days configurable via workflow env var

Add -RetentionDays parameter to the script (mandatory for
-PurgeStaleFiles, optional for generate mode) and define
DASHBOARD_RETENTION_DAYS env var at the workflow level so the value
is controlled in one place.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: fix comment and add entries guard

- Fix workflow comment to accurately describe the purge behavior
- Guard against missing 'entries' key in purge mode to avoid silently
  skipping malformed plugin JSON files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update eng/dashboard/generate-benchmark-data.ps1

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update .github/workflows/evaluation.yml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-10 11:00:19 +01:00
Gerald Versluis ebe71e2f06 Add dotnet-maui plugin with dotnet-maui-doctor skill (#199) 
* Add dotnet-maui-doctor skill and refs

Add a new .agents/skills/dotnet-maui-doctor skill bundle. Includes SKILL.md describing an autonomous workflow to detect and remediate .NET MAUI environment issues and platform-specific reference docs for installation commands, platform requirements, troubleshooting, Microsoft OpenJDK guidance, and a WorkloadDependencies discovery guide that pulls authoritative versions from NuGet. Documents validation/remediation steps for .NET SDK, workloads, JDK, Android SDK, Xcode, and Windows SDK (macOS/Windows/Linux) and includes a temporary recommendation to prefer Microsoft OpenJDK (JDK 21) until manifests are updated.

* MAUI doesn't quite yet support dotnet run

* Add dotnet-maui plugin and dotnet-maui-doctor skill

Introduce the dotnet-maui plugin and move the dotnet-maui-doctor skill under plugins/ layout. Update marketplace, CODEOWNERS, and README accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add comprehensive eval scenarios for dotnet-maui-doctor

10 scenarios covering: macOS/Linux/Windows setup, JDK vendor enforcement,
workload update/repair guardrails, dynamic version discovery, JAVA_HOME
misconception, Android SDK diagnosis, out-of-scope decline, and stale
workload remediation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refine eval scenarios: remove weak scenarios, fix overfitting

Remove 'Resist hardcoding' (penalizes skill — baseline gives more useful
direct answer) and 'Decline runtime bug' (skill makes agent refuse to
help). Rework Android SDK scenario to test CI-oriented package discovery
where baseline lacks knowledge. Replace vocabulary-level NuGet/API
assertions with outcome-focused rubric items across all scenarios.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove hardcoded JDK version workarounds in favor of dynamic discovery

Replace temporary JDK 21 workaround notes with guidance to use the
version from WorkloadDependencies.json. Replace hardcoded JDK paths
in troubleshooting.md with {VERSION} placeholders.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve eval scenarios: fix JAVA_HOME timeout, macOS tie, reduce overfitting

- JAVA_HOME: rewrite prompt to request conceptual answer (prevents tool timeout)
- macOS Xcode: add installation source guidance, remove equal-penalty assertion
- Remove irrelevant workload update/repair assertions from 3 off-topic scenarios
- Keep workload assertions in 5 scenarios where workload management is the topic

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reframe JAVA_HOME scenario as prevent-harmful-action pattern

Rewrite prompt from conceptual yes/no question (which the agent answered
from wrong general knowledge without activating the skill) to a scenario
where the user is about to apply incorrect Stack Overflow advice. This
mirrors the successful workload guardrail pattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reduce overfitting: rephrase assertions and rubric to be outcome-focused

- Replace literal '--version' assertions with broader 'version|pin' pattern
- Rephrase Xcode rubric from skill-specific source to version control risk
- Rephrase version discovery rubric from 'NuGet API' to outcome-focused
- Rephrase workload install rubric from '--version flag' to 'pinned version'

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Inline critical JAVA_HOME auto-detection guidance into SKILL.md

The JAVA_HOME eval scenario scored poorly because the agent couldn't read
the reference file (references/microsoft-openjdk.md) and fell back on
general knowledge, which incorrectly says JAVA_HOME must be set.

Changes:
- Task 5: Add inline JAVA_HOME guidance with decision table showing that
  JAVA_HOME is NOT required and MAUI auto-detects JDK installations
- Common Pitfalls: Add 'Unnecessary JAVA_HOME' entry with unset commands

This ensures the agent knows JAVA_HOME is unnecessary even when reference
files can't be loaded. File stays at 210 lines (limit: 500).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Inline JAVA_HOME auto-detection knowledge into SKILL.md

Add JAVA_HOME guidance directly to Task 5, Task 9, and Common Pitfalls
so the agent knows JAVA_HOME is not required even when reference files
can't be read. Based on Redth's original microsoft-openjdk.md reference.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address code review feedback from Copilot

- Fix cmd.exe syntax in installation-commands.md (use PowerShell $env:)
- Replace hardcoded simulator device/runtime with placeholders
- Add System.IO.Compression.FileSystem assembly load for PS 5.1 compat
- Clarify workload reinstall example is macOS-specific (omit ios on Linux)
- Replace hardcoded android-35 with $API_LEVEL placeholder

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address jonathanpeppers review feedback

- Add Apple Developer login/2FA note for Xcode downloads
- Fix Windows SDK detection: extract KitsRoot10 and version subkeys
  instead of dumping full registry (was outputting noise)
- Replace manual NuGet API approach with dotnet workload search version
  CLI command for discovering workload versions
- Reframe JAVA_HOME guidance: report as anomaly rather than prescribing
  unset/set commands (per jonathanpeppers recommendation)
- Use $env:ProgramFiles instead of hardcoded C:\Program Files paths
  in PowerShell commands
- Update SKILL.md to match: softer JAVA_HOME guidance, improved
  Windows SDK detection command

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address Redth review feedback

- Remove Windows SDK detection commands — installed automatically
  via MAUI workload/VS Installer (per Redth recommendation)
- Use dotnet workload search version --format json --take 1 for
  workload version discovery instead of NuGet search API (per both
  Redth and jonathanpeppers)
- Update workload-dependencies-discovery.md Step 2 to match
- Expand Android SDK detection with known filesystem paths
  instead of relying solely on environment variables

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Align eval rubric with reviewer feedback to reduce overfitting

Replace technique-specific rubric items with outcome-focused criteria:
- Replace PowerShell API query check with dotnet CLI discovery check
- Replace manifest extraction check with workload-version-based discovery
- Replace prescriptive version discovery with CLI/docs verification
- Convert workload update/repair regex assertions to rubric criteria
- Remove Invoke-RestMethod assertion (no longer using API queries)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Align JAVA_HOME guidance with reviewer consensus, add Xcode download expectations

- Consistently use ⚠️ anomaly approach for non-Microsoft JAVA_HOME across
  microsoft-openjdk.md and troubleshooting.md (matching SKILL.md)
- Remove contradictory Note in microsoft-openjdk.md that conflicted with table
- Reframe unset commands as user's choice, not automatic action
- Add Xcode download size (~12GB) and time expectations per Redth's feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update plugins/dotnet-maui/skills/dotnet-maui-doctor/references/workload-dependencies-discovery.md

Co-authored-by: Jonathan Peppers <jonathan.peppers@gmail.com>

* Update JDK example to version 21 per jonathanpeppers feedback

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: redth <jondick@gmail.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Jonathan Peppers <jonathan.peppers@gmail.com>
 2026-03-09 13:34:19 -07:00
Jan Krivanek d709e0b55a Add checking for orphaned skills/plugins (#291) 2026-03-09 14:23:07 +01:00