skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
473 Commits 131 Branches 2 Tags
dev/jankrivanek/eval-dispatch-plugin-input
Commit Graph

209 Commits

Author SHA1 Message Date
dependabot[bot] 25e3b5fd0c Bump the github-actions-dependencies group with 2 updates (#503) 
Bumps the github-actions-dependencies group with 2 updates: [actions/cache](https://github.com/actions/cache) and [actions/setup-node](https://github.com/actions/setup-node).


Updates `actions/cache` from 4.2.3 to 5.0.4
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v4.2.3...668228422ae6a00e4ad889ee87cd7109ec5666a7)

Updates `actions/setup-node` from 4.4.0 to 6.3.0
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/49933ea5288caeca8642d1e84afbd3f7d6820020...53b83947a5a98c8d113130e565377fae1a50d02f)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: 5.0.4
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-dependencies
- dependency-name: actions/setup-node
  dependency-version: 6.3.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-04-08 06:45:17 +00:00
Jan Krivanek 070f8b4648 Add pats rotation to agentic workflows (#500) 
* Copy PATs rotation from dotnet/core

* Recompile agentic workflows with gh-aw v0.67.1

Remove unsupported 'failure' safe-output (dropped in gh-aw v0.67.1).
Regenerate lock files with the new compiler version.
No functional changes.

* Add Copilot PAT pool rotation to agentic workflows

Use the select-copilot-pat action to randomly select from a pool of
COPILOT_GITHUB_TOKEN / COPILOT_GITHUB_TOKEN_2..8 secrets, distributing
Copilot API load across multiple PATs.

This is a stop-gap until organization-level billing is available for
Agentic Workflows. See .github/actions/select-copilot-pat/README.md.
 2026-04-07 12:37:30 +02:00
Mukund Raghav Sharma (Moko) 665c39e957 Add configuring-opentelemetry-dotnet skill (#268) 
* Add configuring-opentelemetry-dotnet skill (+8.1% eval, near-miss)

Teaches OpenTelemetry SDK configuration in ASP.NET Core: package selection,
AddOpenTelemetry with tracing/metrics/logging, custom ActivitySource and
Meter creation, OTLP exporter setup, and context propagation.

Eval results: +8.1% improvement (threshold: 10%, near-miss)
Includes eval.yaml with OTel setup scenario + negative test.

* Sharpen eval.yaml: add custom-spans-invisible pain-point and name-matching rubric emphasis

* Migrate configuring-opentelemetry-dotnet to plugins/ directory structure

* Move OpenTelemetry skill to new dotnet-aspnet plugin, address review feedback

- Move skill from plugins/dotnet-diag to new plugins/dotnet-aspnet plugin
- Create dotnet-aspnet plugin.json
- Fix SKILL.md frontmatter: remove code fence wrapper so YAML metadata parses
- Fix undefined 'order' variable in ProcessOrderAsync example
- Move SqlClient/Runtime instrumentation to clearly marked optional sections
- Add missing using directives (System.Diagnostics, OpenTelemetry.Context.Propagation)
- Handle null Activity.Current in context propagation snippet
- Use IMeterFactory via DI instead of static Meter per official guidance
- Add OpenTelemetry.Instrumentation.Http to eval rubric package list
- Simplify eval prompt to be more natural/generalized
- Update CODEOWNERS: add dotnet-aspnet plugin entries, remove old dotnet-diag entries

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Configure metrics OTLP exporter endpoint explicitly to match tracing config

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address round 2 review: logging endpoint, marketplace, RecordException using, ActivitySource in propagation, HttpClient pitfall

- Configure logging OTLP exporter endpoint explicitly for consistency
- Add dotnet-aspnet to both marketplace.json files
- Add 'using OpenTelemetry.Trace' for RecordException extension method
- Declare ActivitySource in context propagation snippet
- Fix HttpClient instrumentation pitfall (works with new HttpClient() too)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clarify logging package source and add ActivitySource registration reminder

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add @dotnet/aspnet team to skill CODEOWNERS entries

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add dotnet-aspnet to README table, make propagation ActivitySource static

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add expect_activation:false, IMeterFactory using, clarify OTLP exporter scope

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add OtlpExporter using directive, clarify OTLP package covers logging

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove Prometheus from OTLP claim, add OTLP exporter assertion to eval

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clarify Jaeger is traces-only, document logging package provenance

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix eval: increase timeout, reject tools, rewrite prompt as explanation-only

- Scenario 1: rewrite prompt to ask for code explanation (not file
  modification), add reject_tools to prevent bash/edit overhead,
  increase timeout 120s->360s, update rubric wording
- Scenario 2: add reject_tools to eliminate tool-overhead noise

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve OTel skill evals and address review feedback

- Remove redundant per-skill CODEOWNERS lines (ViktorHofer feedback)
- Add SetResourceBuilder + ParseStateValues to SKILL.md Step 3 (logging provider gap)
- Rewrite eval scenarios: 2 targeted positive scenarios testing real skill gaps
- Remove noisy negative activation scenario (high variance, -21.5% drag)
- Eval results: +14.2% improvement (threshold: 10%), overfitting 0.10 (low)
  Scenario 1: 3.0/5 -> 5.0/5, Scenario 2: 3.8/5 -> 5.0/5

* Add propagate-trace-context scenario to OTel evals (+25.6%)

Third scenario tests trace context propagation across message queues,
where the base model scores 3.2/5 and skill-enhanced scores 5.0/5.

Removed metrics and debug scenarios that baseline already aced (5.0/5)
which only added token cost without measurable improvement.

Final: 3 scenarios, +25.6% improvement, overfitting 0.09 (low)

* Remove duplicate dotnet-aspnet CODEOWNERS entries (L69-70 shadowed by L86-87)

* Use @dotnet/aspnet team alias in CODEOWNERS for wider reach

* Addressed feedback

---------

Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com>
 2026-04-07 03:32:05 -07:00
Jan Krivanek d861bbf62e Add AGENTVIZ session replay integration (#494) 
* Add AGENTVIZ session replay integration

- Add workflow_dispatch trigger to evaluation.yml
- Add --keep-sessions to skill-validator evaluate step
- Add publish-session-data job (mirrors publish-token-data)
- Add replay link to PR comments (comment-on-pr)
- Add AGENTVIZ SPA build/deploy to deploy-dashboard job
- Add setup-node step to deploy-dashboard
- Add per-plugin Sessions Visualisation links to dashboard.js
- Create build-replay-sessions.ps1 (manifest generation from sessions.db)
- Create purge-replay-sessions.ps1 (7-day retention management)

* Address code review feedback on PR #494

- Use [IO.Path]::PathSeparator instead of hardcoded ';' for cross-platform PATH
- Compute cutoff date in UTC for correct retention comparisons
- Precompute ID HashSet before merge loop to avoid O(n^2)
- Pin actions/setup-node to commit SHA (49933ea5...#v4)
- Pin AGENTVIZ clone to commit SHA with verification
- Skip npm ci + build when deployed commit SHA matches pinned SHA

* Remove hardcoded AGENTVIZ SHA; use cache + zero-clone checks

- Resolve AGENTVIZ target SHA via git ls-remote (no clone)
- Read deployed SHA via curl from raw.githubusercontent.com (no clone)
- Cache build output keyed by commit SHA (skip npm ci+build on hit)
- Only clone AGENTVIZ repo on cache miss when SPA needs rebuild

* Address round 2 code review feedback on PR #494

- Compare scheduled dir dates at day granularity (dirDate.Date vs cutoffDate.Date)
- Fix useTempDir null-safe path comparison via GetFullPath + null guard
- Use UTC for dateTag in build-replay-sessions.ps1 (consistent with purge)
- Fix cache/deploy path mismatch: deploy from /tmp/agentviz-dist (cache path)
- Deterministic clone: full clone + git checkout TARGET_SHA (fail on mismatch)
- URL-encode manifest param in PR comment link (jq @uri)
- Derive manifest generated timestamp from newest session mtime to avoid churn
 2026-04-02 17:39:59 +02:00
David Ortinau c4b6819dda Add 7 core .NET MAUI development skills to dotnet-maui plugin (#456) 
* Add 7 core .NET MAUI development skills to dotnet-maui plugin

Add maui-shell-navigation, maui-collectionview, maui-data-binding,
maui-dependency-injection, maui-app-lifecycle, maui-safe-area, and
maui-theming skills with eval.yaml tests and reference docs.

Update plugin.json, marketplace.json (both copies), and CODEOWNERS.

Resolves #455

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix cross-skill references to non-existent skills

Replace 10 dangling references to skills not in this repo
(maui-deep-linking, maui-platform-invoke, maui-unit-testing,
maui-maps, maui-animations, maui-accessibility, maui-app-icons-splash,
maui-bootstrap-theme, maui-localization) with plain-English guidance
or references to skills that exist in this PR.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix code correctness bugs and remaining cross-references

Address PR review feedback and skeptic review findings:

Code correctness:
- Fix BackgroundColor="Primary" (XAML crash) → use {StaticResource Primary}
- Fix MergedDictionaries.Clear() (nukes all styles) → remove-then-add pattern
- Fix HorizontalStackLayout Fill (no effect) → use Grid with ColumnDefinitions
- Fix AncestorType with ViewModel type → use ContentPage (visual element)
- Fix Frame usage → replace with Border (recommended control)
- Remove redundant MultiBinding StringFormat="{}{0}"
- Fix OnStart reference → use CreateWindow for theme restore
- Soften lifecycle OnStart/OnSleep characterization (compat shims, not absent)
- Qualify ResourceDictionary code-behind requirement (only for x:Class)

Cross-references:
- Remove all remaining body-text refs to non-existent skills
- Replace with Microsoft Learn doc links or generic guidance

Eval:
- Relax SafeAreaRegions assertion to accept SafeAreaEdges or SafeAreaRegions
- Relax theming rubric for ResourceDictionary code-behind

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Restore Clear() theme pattern with clarifying comment

The Clear()+Add() pattern is intentional — theme dictionaries are designed
to be the only merged dictionaries in this architecture. Added comment
explaining the assumption and what to do if non-theme dictionaries exist.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix second-pass review findings: API accuracy and eval robustness

Code correctness:
- ScrollView SafeAreaEdges="All" → "Container" (ScrollView only honors
  Container and None; All silently fails for keyboard avoidance)
- DI CreateWindow example: add comment noting AppShell must be registered
- AddHttpClient: add comment noting Microsoft.Extensions.Http NuGet required
- MultiBinding: remove redundant StringFormat from reference doc (matches SKILL.md)
- RelativeSource Self table: Path=Width → Path=WidthRequest (Width is
  read-only -1 before layout; WidthRequest is the correct bindable property)
- CollectionView StackLayout warning: clarify it refers to BindableLayout
  on StackLayout vs CollectionView ItemsLayout types
- SafeAreaEdges: fix "attached property" → "property" (it's a regular property)
- ios:NavigationPage example: add xmlns comment for ios namespace

Eval assertions:
- DI Scoped regex: accept "doesn't"/"lacks"/"without" not just "no"
- CollectionView ViewCell: match negation in either direction around ViewCell
- Lifecycle events: require 3+ distinct events instead of any single one

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix final PR review comments

- safe-area eval: tighten Container assertion to require SafeAreaEdges context
- DI: merge #else fallback into single compilable #if block
- lifecycle: behaviour → behavior (American English consistency)
- theming: macOS Catalyst minimum 10.14 → 10.15 (Catalina required)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-04-02 15:48:30 +02:00
Amaury Levé 948a9a52ef Add exp-mock-usage-analysis skill and evaluation (#483) 
* Add exp-mock-usage-analysis skill and evaluation

New experimental skill that detects unused, redundant, or unnecessary
mocks in .NET test suites. Covers Moq, NSubstitute, FakeItEasy, and
manual test doubles.

- SKILL.md with 6-step workflow: gather code, classify mocks, detect
  anti-patterns, incorporate runtime data, generate recommendations,
  report findings
- eval.yaml with 10 scenarios: trivial type mocking, unused setups,
  redundant configs, stable utility mocks, well-placed mocks (no false
  positives), NSubstitute support, FakeItEasy support, setup sprawl,
  and 2 non-activation boundary tests
- Fixture files for each scenario with production + test code
- CODEOWNERS entry for @dotnet/dotnet-testing

* Remove no-headroom scenarios, increase NSubstitute timeout

Remove 4 scenarios where baseline already scores 5.0/5 (model handles
these well without skill guidance):
- Detect mocking of DTOs, records, and enums
- Recognize well-placed mocks without inventing false positives
- Decline when asked about non-mock test anti-patterns
- Decline mock framework migration request

Increase NSubstitute scenario timeout from 120s to 180s to avoid
baseline timeouts impacting scoring.

Remaining 6 scenarios all show positive skill impact (3→5, 3→4, etc).

* Document areas the model handles well without the skill

Add a subsection under 'When Not to Use' listing mock-related tasks
where the base model already scores 5.0/5 without skill guidance:
- Mocking trivial types (DTOs, records, enums)
- Recognizing well-placed mocks (no false positives)

* Fix md

* Improve
 2026-04-02 10:03:17 +00:00
Amaury Levé d5ca4c6c01 Add exp-test-smell-detection skill and evaluation (#491) 
* Add exp-test-smell-detection skill and evaluation

- Language-agnostic core skill detecting 10 test smells (conditional logic,
  mystery guest, sleepy test, assertion-free, eager test, magic numbers,
  sensitive equality, exception handling, general fixture, ignored tests)
- .NET extension with framework-specific detection tables (MSTest/xUnit/NUnit/TUnit)
- Reference catalog with 19 smells from testsmells.org research
- Evaluation with 4 scenarios: smelly suite, clean suite, integration tests, non-activation
- CODEOWNERS entry for @dotnet/dotnet-testing

* Improve integration test fixture and non-activation timeout

- Add real smells to integration test fixture (Thread.Sleep, conditional
  logic, assertion-free test) mixed with legitimate integration patterns
- Update rubric to test nuanced distinction between real smells and
  acceptable integration test patterns
- Increase non-activation scenario timeout from 120s to 300s to prevent
  timeout-related scoring penalties

* Fix test projects

* Add domains for testsmells

* Remove extra domains
 2026-04-02 11:51:19 +02:00
Dan Moseley 165e6439fe Tighten known-domains allowlist (#496) 
* Tighten known-domains allowlist

Narrow broad domain entries to specific paths/subdomains actually used:
- aka.ms -> aka.ms/dotnet-trace, aka.ms/perfcollect
- nuget.org -> api.nuget.org, www.nuget.org/account/trustedpublishing
- npmjs.com/package/@modelcontextprotocol -> pinned to inspector/v/0.21.1

Remove unused domains: fluentassertions.com, speedscope.app
Update skill content to match: pin MCP Inspector link, remove
FluentAssertions link, remove speedscope hyperlink.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* updated mcp skills codeowner

* Refine scoped known-domain entries

Remove the remaining nugettest.org publish example and drop the nugettest.org allowlist entry. Also normalize path-scoped entries to omit a hardcoded www prefix so both host forms match the existing ReferenceScanner logic.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Leslie Richardson <lerich@microsoft.com>
 2026-04-02 03:35:54 +00:00
Amaury Levé eb0f98eba6 Add exp-test-gap-analysis skill for pseudo-mutation test gap detection (#490) 
New experimental skill that performs pseudo-mutation analysis on .NET
production code to find gaps in existing test suites. Covers boundary,
boolean/logic, return value, exception removal, arithmetic, and
null-check mutation categories.

Includes eval.yaml with 4 scenarios (boundary gaps, logic gaps,
well-tested code recognition, non-activation) and fixture projects.
 2026-04-01 21:19:36 +02:00
dependabot[bot] 0417c8ec62 Bump actions/cache in the github-actions-dependencies group (#492) 
Bumps the github-actions-dependencies group with 1 update: [actions/cache](https://github.com/actions/cache).


Updates `actions/cache` from 4.2.3 to 5.0.4
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/5a3ec84eff668545956fd18022155c47e93e2684...668228422ae6a00e4ad889ee87cd7109ec5666a7)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-version: 5.0.4
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-04-01 10:20:01 +00:00
Jan Krivanek 498755bc0d Make the infra eval gate faster (#489) 
* Make the infra eval gate faster

* Address PR feedback: handle infra+skill overlap, detect evaluation.yml changes

- When both infra and skill changes exist in a PR, evaluate the changed
  skills normally instead of the random subset (fixes Copilot review).
- Expand infra detection to include .github/workflows/evaluation.yml
  changes in all three detection sites (pr-status, fork-pr-status, discover).
 2026-03-31 19:50:49 +02:00
Viktor Hofer 167b7c3772 Fix evaluation cache key: compute explicitly instead of cache-primary-key (#488) 
The pinned actions/cache SHA (5a3ec84, v4) does not expose the
cache-primary-key output. This caused the evaluate job to fail with
'Input required and not supplied: key' because the build-validator
job output was always empty.

Fix: compute the cache key in a dedicated step and pass it as a job
output, instead of relying on an output that doesn't exist at this
action version.

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-31 17:31:52 +02:00
Viktor Hofer 3fff1f2573 Skill-validator: perf, cancellation, and reliability improvements (#482) 
* Skill-validator: perf, cancellation, and reliability improvements

Performance:
- Cache skill-validator-dist.tar.gz in CI (evaluation.yml and skill-check.yml)
  to skip .NET SDK setup, build, and tar on cache hit. Evaluate and comment-on-pr
  jobs restore from cache with artifact download fallback.
- Replace SemaphoreSlim with Lock in SessionDatabase for lighter-weight
  synchronous locking.

Cancellation:
- Thread CancellationToken from System.CommandLine entry point through the
  entire evaluate call chain: Run -> EvaluateTarget -> EvaluateSkill/EvaluateAgent
  -> ExecuteScenario/ExecuteAgentScenario -> ExecuteRun/ExecuteAgentRun, plus
  EvaluateSkillNoise -> ExecuteNoiseTest.
- Add CancellationToken parameter to AgentRunner.RunAgent, Judge.JudgeRun,
  PairwiseJudge.Judge, and OverfittingJudge.Analyze, forwarding to
  RetryHelper and LlmSession.SendAsync respectively.
- Pass CancellationToken to all ConcurrencyLimiter.RunAsync call sites so
  semaphore waits are cancellable.
- Re-throw OperationCanceledException at the top-level skill evaluation loop
  so Ctrl+C / CI cancellation propagates instead of being swallowed as a
  skill failure.

Reliability:
- Smart timeline truncation in Judge and PairwiseJudge: error events
  (session.error, runner.error) are always preserved during truncation
  regardless of position, with a summary marker showing event type breakdown.
- Add jitter (±25%) to RetryHelper exponential backoff to prevent thundering
  herd on rate-limit recovery.
- Add CancellationToken parameter to ConcurrencyLimiter.RunAsync<T>.
- Add ConcurrencyLimiter tests (6 tests) and RetryHelper jitter tests (2 tests).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback

- RetryHelper: re-clamp delay after jitter so it never exceeds the
  remaining budget or MaxSingleDelayMs.
- Judge/PairwiseJudge: handle edge case where error count alone exceeds
  MaxTimelineEvents by trimming errors with a summary instead of
  unconditionally appending all of them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add high-variance detection and refactor FailureKind to enum

- Add Statistics.CoefficientOfVariation() for measuring score consistency
- Flag scenarios with CV > 0.5 as high-variance in ScenarioComparison
- Show console warning and markdown footnote for high-variance scenarios
- Include high-variance scenario names in Comparator verdict reason
- Refactor FailureKind from magic strings to strongly-typed enum with
  JsonStringEnumMemberName attributes for backwards-compatible JSON output
- Add 6 new CV unit tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-31 13:58:28 +02:00
Mukund Raghav Sharma (Moko) c92faa7800 Add minimal-api-file-upload skill (#264) 
* Move minimal-api-file-upload skill to aspnetcore plugin with 3 eval scenarios

Per repo restructuring feedback, ASP.NET Core specific skills should
be under the aspnetcore plugin rather than the dotnet plugin.

* Increase eval timeouts to 180s to avoid timeout-related scoring penalties

* Generalized and addressed feedback

* Improve eval scenarios: remove noisy tests, add dual-limit and security review scenarios

* Add CODEOWNERS for aspnetcore plugin

* Fix eval: enrich prompts for skill activation, bump security review timeout to 240s

* Drop size-limit scenario (baseline already 5.0), keep 3 high-signal scenarios

* Reframe security review as file upload fix to improve skill activation

* Review

* Contribution guide followed

* Updated name

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-aspnet/skills/minimal-api-file-upload/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Address PR #264 review feedback from Brennan and Copilot

- Rename plugin from aspnetcore to dotnet-aspnet per repo conventions
- Fix streaming filename: remove user-controlled extension, use bare GUID
- Remove explicit sanitization code (Option B) per Brennan's feedback
- Add case-insensitive MIME type comparison
- Validate ContentType matches magic bytes (reject mismatches)
- Fix stream rewind + CopyToAsync: copy from already-opened stream
- Reword streaming section to accurately describe buffering vs chunks
- Update marketplace.json, README.md, CODEOWNERS for dotnet-aspnet rename

---------

Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-31 10:41:25 +02:00
Viktor Hofer 7747fcb427 Move skill-validator docs under src/ and include in packages (#476) 
* Move skill-validator docs under src/ and include in packages

Move README.md to src/ and InvestigatingResults.md + OverfittingDetection.md
to src/docs/ so they ship in the NuGet package and .tar.gz archives.

- Include docs/*.md in NuGet pack and publish output via CopyToPublishDirectory
- Add LICENSE to publish output (tar.gz only, not NuGet — uses PackageLicenseExpression)
- Replace enumerated docOnlyFiles lists with a regex pattern in CI workflows
- Deduplicate eval.yaml docs between CONTRIBUTING.md and the shipping README
- Add test fixture files walkthrough to README (moved from CONTRIBUTING.md)
- Add AGENTS.md guidance: src/ is the packaging boundary

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update link format in failure diagnosis message

* Update AGENTS.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-30 14:00:12 +00:00
Jan Krivanek 75f72efb0e Fix token usage collection (#471) 
* Fix token usage collection

* Ensure evaluate is run on schedule

* Fix PR title issue

* Update eng/dashboard/token-usage.js

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-30 13:03:58 +00:00
Amaury Levé b527b41573 Add exp-test-boilerplate-detection and exp-assertion-quality skills (#468) 
Two new experimental skills for dotnet-experimental plugin:

- exp-test-boilerplate-detection: Detects duplicate boilerplate patterns
  across .NET test suites and identifies refactoring opportunities.
  Categories: repeated construction, assertion patterns, copy-paste
  methods, duplicated setup/teardown, repeated infrastructure.

- exp-assertion-quality: Analyzes assertion variety and depth across
  .NET test suites. Produces metrics dashboard including assertion
  count, type spread, trivial %, assertion-free tests, and negative
  assertion coverage.

Both skills include eval.yaml with 3-4 scenarios each (positive,
negative, and non-activation) plus test fixtures.
 2026-03-30 13:01:03 +00:00
Jan Krivanek e6250320e8 Clean the agentic workflows report bugs (#474) 2026-03-30 12:45:24 +00:00
Copilot 447e7367c8 Move investigation prompt to workflow summary, rename Full Results link (#473) 
* Initial plan

* Move investigation prompt to workflow summary, remove InvestigatingResults.md callout, rename Full Results link

Agent-Logs-Url: https://github.com/dotnet/skills/sessions/b5b537b4-3d90-48ff-ab2d-3298660ccc97

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

* Revert Reporter.cs change per review feedback

Agent-Logs-Url: https://github.com/dotnet/skills/sessions/e5845ec8-06b3-4ffa-b282-fd63aa2a467c

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
 2026-03-30 14:33:46 +02:00
Copilot 6248ef2b4b Fix Action summary to show all columns and remove redundant "Full results" link (#470) 
* Initial plan

* Fix Action summary to show all columns and remove redundant Full results link

The comment-on-pr job was using the simplified PR comment table
(consolidated-comment.md) for the Action step summary. This change:

1. Writes the full summary-body.md (with all columns including
   Quality (Plugin) and Agents Invoked) to GITHUB_STEP_SUMMARY.
2. Removes the "Full results" link from the Action summary since
   it links to the same page.
3. Keeps the investigation prompt in both PR comment and Action summary.

Agent-Logs-Url: https://github.com/dotnet/skills/sessions/870c41f2-0e41-4e62-9a43-d1f7128bab06

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
 2026-03-30 12:55:53 +02:00
Dan Moseley 17376fd82d Exclude doc-only .md files from evaluation infra change detection (#466) 
* Exclude README.md from infra change detection in evaluation workflow

eng/skill-validator/README.md changes were triggering a full all-skills
evaluation. Add it to the doc-only exclusion list alongside
InvestigatingResults.md, and extend the exclusion to the discover job
which was missing it entirely.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-30 09:53:37 +00:00
Dan Moseley e1863cc526 Improve investigation guide: PR number in prompt, document reject_tools, baseline-already-good pattern (#451) 2026-03-27 07:09:29 +00:00
Amaury Levé a220f36256 Add exp-test-maintainability skill for test structure review (#418) 
New experimental skill that assesses test maintainability: duplication,
test size, data-driven patterns, display names, builder/helper extraction,
and shared setup.
 2026-03-26 14:28:43 +01:00
Dan Moseley 604c46c8c6 Fix eval investigation docs: download command, schema accuracy, example script (#449) 2026-03-25 15:24:33 -06:00
leslierichardson95 7286c31118 Add skills for C# MCP Server Development (#317) 
* Add C# MCP server skills (create, debug, test, publish)

Four new skills for the C# MCP server development lifecycle:
- mcp-csharp-create: Scaffolding with dotnet new mcpserver, tools/prompts/resources, transport config
- mcp-csharp-debug: MCP Inspector, VS Code integration, breakpoint debugging, logging
- mcp-csharp-test: Unit tests with ClientServerTestBase, integration with WebApplicationFactory, evals
- mcp-csharp-publish: NuGet packaging, Docker/Azure deployment, MCP Registry publishing

Each skill includes SKILL.md with progressive disclosure references/ and eval.yaml tests.

* Fix mcp-csharp-create eval: use regex assertions for combined attribute syntax

Replace scaffolding-heavy scenarios with implementation-focused ones that
test MCP-specific features (resources, prompts, logging). Fix assertion
patterns to match combined C# attribute syntax [McpServerTool, Description()]
instead of requiring standalone [McpServerTool]. Increase timeouts to 180s
to account for skill-reading overhead.

Validator result: passed=True, improvement=44.6% (threshold=10%)

* Add CODEOWNERS entries for MCP C# skills (create, debug, publish, test)

* Update plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-publish/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Remove evaluation step and guide from mcp-csharp-test skill

* Add CODEOWNERS entries for dotnet-maui skills

* Update CODEOWNERS to add @artl93 as co-owner for mcp-csharp skills

* Replace parent-directory file references with backtick skill names

Cross-skill references used ../sibling/SKILL.md paths which the
validator rejects. Replace with backtick-quoted skill names to match
the convention used by other skills in the repo.

* Add known domains for MCP server development skills

Add domains referenced by the mcp-csharp-* skills:
- github.com/modelcontextprotocol/registry
- github.com/open-telemetry/semantic-conventions
- npmjs.com, xunit.net, fluentassertions.com, nugettest.org

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update Docker commands to use placeholder for registry and image names

* moved mcp skills and their evals into dotnet-ai plugin

* cleaned up moved files to dotnet-ai

* cleaned up files

* Update .github/CODEOWNERS

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update nuget-packaging.md

* Update repository URL format in nuget-packaging.md

* Add github.com/ to known-domains to allow truncated placeholder URLs

The URL regex truncates at angle brackets (invalid in URIs), so
placeholder URLs like https://github.com/<yourusername>/<repo>
become https://github.com/ which didn't match any path-scoped entry.
Adding github.com/ (with trailing slash) covers only this case without
broadly allowing all github.com URLs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update Docker tag command to use placeholder for server name

* Update known domains and fix GitHub username placeholders in documentation

* Remove McpServerToolType from mcp-csharp-create eval

McpServerToolType is only needed with WithToolsFromAssembly(), which is
discouraged because it is not Native AOT compatible. Updated eval assertions
and rubrics to prefer WithTools<T>() instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve eval resilience for mcp-csharp-create and mcp-csharp-debug

- Increase mcp-csharp-create timeouts from 180s (or default 120s) to 360s
  for all 3 scenarios. These scenarios consistently time out because the
  model spends time on bash exploration/scaffolding before writing code.

- Add explicit timeout to scenario 3 which was relying on the 120s default.

- Broaden mcp-csharp-debug Inspector rubric to accept both the concise
  single CLI command style and the step-by-step UI configuration walkthrough,
  since both are correct approaches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update eng/known-domains.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Broaden debug 'failing tool' rubric to accept multiple debugging approaches

The rubric criterion 'Shows how to attach a debugger' was too narrow.
The skilled answer correctly focused on the #1 cause (stdout pollution)
but scored low because it didn't show a specific 'attach to process' flow.
Broadened to accept any valid debugging approach: attaching, Debugger.Launch(),
or launch.json configuration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix 4 eval failures in mcp-csharp-debug and mcp-csharp-test skills

mcp-csharp-debug:
- Trim SKILL.md verbosity (183->160 lines) and mcp-inspector.md (67->54 lines)
- Add Diagnosing Tool Errors section covering debugger, output panel, Inspector, common culprits
- Rebalance debugging narrative away from stderr-only focus
- Move HTTP logging config to ide-config.md reference

mcp-csharp-test:
- Add eval/evaluations keywords to frontmatter for activation
- Add HTTP tool invocation test pattern (tools/call via WebApplicationFactory)
- Trim test-patterns.md bloat (remove Test Categories, Coverage, Input Validation)
- Create references/evaluations.md with qa_pair format and read-only/deterministic guidance

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-25 14:17:57 -06:00
Copilot 97832189cf Simplify evaluation table in PR comments; keep full table in workflow summary (#448) 
* Initial plan

* Simplify evaluation workflow table: remove Quality (Plugin) and Agents Invoked columns from PR comment

- Add `simplified` parameter to GenerateMarkdownSummary in Reporter.cs
- When simplified=true, exclude Quality (Plugin) and Agents Invoked columns
- Update ConsolidateCommand with --summary-output for full table
- evaluation.yml: simplified table for PR comment, full table for step summary
- Update Full results link to indicate additional columns are available
- Add 3 tests for simplified table behavior

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/d2590b7a-023a-45a5-928b-9e658f40b58e

* Simplify evaluation table in PR comments; keep full table in workflow summary

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/d2590b7a-023a-45a5-928b-9e658f40b58e

* Delete .nuget/nuget.exe

* Move table simplification to workflow post-processing instead of skill-validator

Revert all C# changes (Reporter.cs, ConsolidateCommand.cs, tests). The table
simplification is now a post-processing step in evaluation.yml using a Python
script that strips "Quality (Plugin)" and "Agents Invoked" columns from the
markdown before posting as a PR comment. The full table goes to the workflow
step summary.

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/c8e0acc6-1128-4bf5-8170-382b278500e5

* Apply review suggestions: use heredoc, remove unused import, full step summary

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/be2dd88e-e30b-4c50-905c-1cff9cd21422

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
 2026-03-25 19:04:12 +01:00
Jeff Schwartz 55a7d80527 adding SIMD/TensorPrimitives to dotent-diag/analyzing-dotnet-performance (#330) 
* incorporated feedback from previous pr, iterated with 100+ scalar loop tests to come to this skill, moved from skill to reference in the performance skill

* Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Address review feedback on SIMD vectorization reference

- Check Span<T>/MemoryExtensions before TensorPrimitives (no extra dependency)
- Add all supported types (sbyte, ushort, uint, ulong, nint, nuint, char via ushort)
- TensorPrimitives: add constraint and applicable types columns (not just float/double)
- Add FusedMultiplyAdd, clarify AddMultiply vs MultiplyAdd distinction
- Prefer portable APIs over platform-specific intrinsics (allow when perf justifies)
- Fix dispatch pattern to use if/else if to avoid pessimizing small inputs
- Remove LINQ references

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* addressing more feedback

* move to the experimental plugin

* move the simd tests to exprimental

* add codeowners

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-25 10:51:28 -06:00
Amaury Levé 1d56634150 Create experimental plugin (#446) 2026-03-25 16:11:07 +00:00
Dan Moseley b39fe8541b Add evaluation troubleshooting guide for AI agents (#441) 
* Add evaluation troubleshooting guide and link from PR comments

Add InvestigatingResults.md with:
- How to find and download evaluation artifacts from workflow runs
- Full results.json schema documentation
- Common failure patterns (timeouts, empty output, variance, rubric
  issues, skill not activated) with diagnosis steps and fixes
- Example analysis script for AI agents to parse results

Link the doc from:
- Reporter.cs: show a link in the PR comment when any skill fails
- skill-validator README.md: Output section
- CONTRIBUTING.md: CI evaluation section

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update InvestigatingResults.md

* Address review feedback: fix hard-coded URLs and optional plugin run docs

- Reporter.cs: Use repo-relative link instead of hard-coded GitHub URL
- evaluation.yml: Use github.sha instead of hard-coded 'main' branch
- InvestigatingResults.md: Clarify skilledPlugin run is optional

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* THROWAWAY: add trailing newline to trigger eval workflow

This change is intentionally trivial and will be reverted before merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR #441 review feedback (round 2)

- Reporter.cs: Remove leading / from relative link
- evaluation.yml: Use jq on results.json instead of grepping for emoji
- evaluation.yml: Use needs.gate.outputs.head_sha for revision-pinned URL
- InvestigatingResults.md: Mark pluginImprovementScore/pluginBreakdown as optional
- InvestigatingResults.md: Fix skill-not-activated symptoms to use
  skillActivation fields instead of turnCount

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Use specific skillActivation field names and fix timestamp description

- Use skillActivationIsolated/skillActivationPlugin instead of generic skillActivation
- Clarify timestamp is when results were written, not run start

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix timestamp field description (was lost in prior commit)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Revert "THROWAWAY: add trailing newline to trigger eval workflow"

This reverts commit 83bd7f7601.

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-25 16:10:17 +00:00
Viktor Hofer 1762a78fed Add stale PRs agentic workflow (#435) 
* Add stale PRs agentic workflow

* Add fork-guard to close-stale-prs workflow

Skip scheduled runs on forked repositories to avoid consuming fork
owner minutes and failing due to missing secrets/context, matching
the pattern used by other scheduled workflows in this repo.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-25 13:00:10 +01:00
Jan Krivanek 19b6cadddf Move token usage data into separate branch (#431) 
* Move token usage data into separate branch

* Address PR review: add help doc, fix title injection, add token-usage fallback

* Add -SkipBenchmarkData switch, clean stale files on branch push
 2026-03-24 13:19:47 +01:00
Viktor Hofer c14e63ba6c Combine evaluation.yml and evaluation-run.yml into single workflow (#430) 
* Combine evaluation.yml and evaluation-run.yml into single workflow

Inline the three jobs from the reusable evaluation-run.yml workflow
(build-validator, evaluate, comment-on-pr) directly into evaluation.yml
and delete the now-unnecessary reusable workflow file.

Changes:
- Inline build-validator, evaluate, and comment-on-pr jobs
- Eliminate 8-secret forwarding block (direct secrets.* access)
- Replace inputs.* with direct needs.* references
- Add MODEL/JUDGE_MODEL workflow-level env vars
- Fix --verbose duplication in evaluate step
- Simplify infra-change detection regex
- Update dependency chains (report-status, publish-benchmark)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR feedback: fix consolidate subcommand, improve error reporting

- Fix consolidate invocation: use 'evaluate consolidate' subcommand
  (was incorrectly changed to just 'consolidate' during inlining)
- Add build-validator to report-status needs so build failures surface
  as a deterministic failure state instead of generic error
- Add discover to comment-on-pr needs and gate on has_entries so
  /evaluate on a no-op PR doesn't post a misleading failure comment

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix matrix fallback and header comment

- Use fromJson(... || '[]') fallback for matrix entries to prevent
  errors when discover is skipped on non-evaluation events
- Update header comment to accurately describe the is_fork mechanism

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-24 11:11:51 +01:00
Viktor Hofer b92dde1854 Refactor skill-validator to vertical slice architecture (#429) 
Reorganize the skill-validator project from layer-based grouping
(Commands/, Services/, Models/, Utilities/) to feature-based slices
(Check/, Evaluate/, Consolidate/, Shared/).

Key changes:
- Check/: CheckCommand, SkillProfiler, AgentProfiler, PluginValidator,
  ExternalDependencyChecker, ReferenceScanner
- Evaluate/: EvaluateCommand, RejudgeCommand (now a subcommand of
  evaluate), AgentRunner, Judge, PairwiseJudge, and other eval services
- Consolidate/: ConsolidateCommand
- Shared/: Models, SkillDiscovery, PluginParser (extracted from
  PluginValidator), Reporter, and utilities

Split PluginValidator: parsing helpers (ParsePluginJson,
TryGetSafeSubdirectory) moved to Shared/PluginParser.cs; validation
logic stays in Check/PluginValidator.cs.

Move RawFrontmatter and RawAgentFrontmatter types from EvalSchema into
SkillDiscovery since they are only consumed by discovery.

Wire RejudgeCommand as subcommand: skill-validator evaluate rejudge.

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-24 09:41:55 +00:00
Viktor Hofer c4c26175e2 Integrate reference scanner into skill-validator CheckCommand (#422) 
* Integrate reference scanner into skill-validator CheckCommand

- Port reference scanner logic from scan.ps1 to C# ReferenceScanner service
- Add --known-domains option to CheckCommand for reference scanning
- Move known-domains.txt and allowed-external-deps.txt to eng/
- Remove standalone scan.ps1 and reference-validation.yml workflow
- Reference scanning now runs as part of skill-check.yml via skill-validator
- Add 34 unit tests for ReferenceScanner (all 463 tests pass)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR feedback: repoRoot derivation, early validation, .agents scanning, SRI policy

- Derive repoRoot from --known-domains file path instead of cwd
- Fail early with clear error when known-domains file is missing
- Add .agents/**/*.md scanning to DiscoverFiles (regression fix)
- Fix SRI policy loophole: SRI suppresses SCRIPT-NO-SRI only, domain
  and HTTPS checks still apply
- Use actual known-domains file path in error messages instead of
  hardcoded eng/known-domains.txt
- Update doc comments to reflect full set of scanned paths
- Add tests for .agents discovery and SRI domain enforcement

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove repo-specific paths from ReferenceScanner

DiscoverFiles now only scans directories it is given -- no hardcoded
knowledge of .agents, agentic-workflows, eng, or README.md. The tool
is repo-agnostic; callers control what gets scanned via --plugin dirs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Drop repoRoot from ReferenceScanner API

ScanFile/ScanFiles no longer take a repoRoot parameter. Finding paths
use the file path as provided by the caller, keeping the scanner
completely decoupled from any repo layout assumptions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Prune stale domains and add json.schemastore.org to known-domains.txt

Removed 10 entries no longer referenced by any plugin content:
agentskills.io, visualstudio.com, mcr.microsoft.com, dotnet.github.io,
github.com/dotnet/docs, github.com/dotnet/dotnet-api-docs,
github.com/dotnet/skills, github.com/github/gh-aw,
github.com/modelcontextprotocol/csharp-sdk, developers.openai.com.

Added json.schemastore.org (referenced by dotnet-template-engine).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix case-insensitive regex, update docs, add tests

- Make UrlRegex and PipeToShellRegex case-insensitive to match the
  original PowerShell behavior (HTTP://, CURL, Wget all now detected)
- Update DiscoverFiles doc comment to remove stale 'individual files' claim
- Narrow CONTRIBUTING.md scope description to match actual plugin-only scanning
- Add tests for mixed/upper-case URL and pipe-to-shell detection

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove IPv6 loopback support, fix pipe-to-shell allowlist case sensitivity

- Remove Ipv6LoopbackRegex and IPv6 test data (not needed)
- Add case-insensitive + culture-invariant matching for pipe-to-shell allowlist
- Add mixed-case allowlist test

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-23 13:05:08 +00:00
Amaury Levé 6489fbee39 Add exp-test-anti-patterns skill for test quality review (#415) 
New experimental skill that detects anti-patterns and code smells in .NET
test suites. Covers assertion quality, test isolation, naming, flakiness
indicators, over-mocking, and structural problems across MSTest, xUnit,
NUnit, and TUnit.

Key features:
- 4-tier severity classification (Critical/High/Medium/Low)
- False-positive avoidance: calibrates severity honestly and recognizes
  well-written tests without inventing problems
- Positive observations alongside findings

Evaluation results (3 runs, claude-opus-4.6):
- Flakiness/coupling detection: 3.0 -> 5.0 (isolated), 4.0 (plugin)
- Duplicates/magic values: 3.0 -> 5.0 (isolated), 4.7 (plugin)
- Well-written test recognition: 2.0 -> 5.0 (both modes)
- Mixed severity detection: 5.0 baseline maintained
- Overfitting score: 0.06 across all scenarios
 2026-03-20 14:04:18 +00:00
YuliiaKovalova dcba4f3d8e Add dotnet-template-engine plugin (#341) 
* Add dotnet-template-engine plugin

Add template-engine skills for .NET project scaffolding powered by the
DotnetTemplateMCP MCP server (v1.1.0). Includes three skills and one agent:

Skills:
- template-discovery: find, inspect, and compare templates with intent
  resolution (70+ keyword mappings)
- template-instantiation: create projects with CPM adaptation, latest NuGet
  versions, parameter validation, and multi-template composition
- template-authoring: create templates from existing projects and validate
  template.json for authoring issues

Agent:
- template-engine: expert agent that routes to the appropriate skill based
  on user intent

Also adds CODEOWNERS entries, marketplace.json registration, eval tests,
and .mcp.json for DotnetTemplateMCP server configuration.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: mcpServers in plugin.json, expect_tools, path fix

- Move MCP server config into plugin.json under mcpServers so SkillValidator
  discovers the template_* tools during evaluation runs
- Add expect_tools constraints to all eval scenarios to verify actual MCP
  tool invocation instead of relying on rubric text alone
- Fix frontmatter path: template.config → .template.config (leading dot)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update DotnetTemplateMCP to v1.2.0

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Include 'dotnet new' in agent description for discoverability

Address review feedback from danmoseley: the frontmatter description
is how the caller decides whether to load the agent tokens, so it
needs to mention 'dotnet new' as a trigger keyword.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove MCP server configuration and references

- Delete .mcp.json MCP server config file
- Remove mcpServers section from plugin.json
- Remove MCP/DotnetTemplateMCP references from agent, skills, and marketplace descriptions
- Skills and eval files remain unchanged

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove ghost tool references and add CLI fallback guidance

- Remove references to 4 non-existent tools: template_compose,
  template_suggest_parameters, template_validate, solution_analyze
- Replace with real alternatives (template_inspect, manual review,
  sequential template_instantiate, direct file inspection)
- Add CLI fallback guidance so LLMs degrade gracefully when
  template tools are unavailable
- Fix authoring eval to not expect template_validate tool

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback: CODEOWNERS naming, claude marketplace, README

- Rename CODEOWNERS header to dotnet-template-engine for consistency
- Mirror plugin entry in .claude-plugin/marketplace.json
- Add plugin to README What's Included table

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix PR review comments: align eval rubric with expect_tools and sync marketplace.json

- Updated template-discovery eval.yaml rubric to match expect_tools
  (removed 'or template_search' since expect_tools enforces template_from_intent)
- Made .claude-plugin/marketplace.json byte-for-byte identical to
  .github/plugin/marketplace.json (removed extra trailing newline)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review comments: strengthen eval assertions and fix rubrics

- Add exit_success assertions to all eval scenarios
- Add file_exists and file_contains assertions for template-instantiation
- Add output_contains for shortName and datatype in template-authoring
- Fix rubrics to reference actual tools (skill + dotnet new CLI) instead
  of non-existent MCP tools (template_from_intent, template_inspect,
  template_instantiate) which caused the judge to penalize quality scores
- Update expect_tools from non-existent MCP tools to 'skill'
- Change plugin version from 1.2.0 to 0.1.0 for consistency with other plugins
- Remove extra blank line in README.md after plugin table

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Rewrite skills and agent to use dotnet new CLI instead of phantom MCP tools

The SKILL.md files and agent.md referenced MCP tools (template_from_intent,
template_inspect, template_instantiate, template_dry_run, etc.) that are
not available at runtime since no MCP server is declared in plugin.json.
This caused the agent to waste tokens attempting to find these tools before
falling back to dotnet new CLI commands.

Changes:
- template-discovery/SKILL.md: Replace MCP tool calls with dotnet new
  search, dotnet new list, dotnet new --help, and dotnet new --dry-run.
  Add intent-to-template keyword mapping table.
- template-instantiation/SKILL.md: Replace template_instantiate with
  dotnet new CLI commands. Add explicit CPM adaptation steps and
  multi-project composition with dotnet sln/dotnet add reference.
- template-authoring/SKILL.md: Replace template_create_from_existing
  with manual template.json creation guidance including example JSON.
  Replace template_dry_run/template_instantiate with CLI equivalents.
- template-engine.agent.md: Replace Available Tools table with CLI
  Commands Reference. Update all workflow steps to use CLI commands.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix file assertion paths: remove ./ prefix for glob matcher compatibility

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
 2026-03-18 22:51:20 +01:00
Jan Krivanek 30c19eae13 Add tokens usage dashboard and data collection (#385) 2026-03-18 20:04:46 +00:00
dependabot[bot] 8e27c14ea4 Bump actions/download-artifact in the github-actions-dependencies group (#396) 
Bumps the github-actions-dependencies group with 1 update: [actions/download-artifact](https://github.com/actions/download-artifact).


Updates `actions/download-artifact` from 8.0.0 to 8.0.1
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/70fc10c6e5e1ce46ad2ea6f2b72d43f7d47b13c3...3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-version: 8.0.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
 2026-03-18 09:58:46 +00:00
Viktor Hofer c5435dd3da Make evaluate a subcommand with tests-dir as required argument (#398) 
* Make evaluate a subcommand with tests-dir as required argument

Change EvaluateCommand from RootCommand to a regular Command('evaluate')
so it is a subcommand like check, consolidate, and rejudge. Convert
--tests-dir from an optional Option to a required positional Argument
placed after the skill paths.

Update .csproj RunArguments, evaluation-run workflow, README, and
CONTRIBUTING.md to reflect the new CLI syntax.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Default --parallel-skills/scenarios/runs to 3

Update the default from 1 to 3 for all three parallelism options and
remove the now-redundant overrides from RunArguments in the .csproj.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Replace RunArguments with launchSettings.json, publish in skill-check

Add Properties/launchSettings.json with default evaluate args for local
development. Remove RunArguments from .csproj. Update skill-check.yml to
publish the validator and invoke the binary directly instead of using
dotnet run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Keep --tests-dir as a required option, not positional

A variadic paths argument greedily consumes all positional values,
making a trailing positional tests-dir impossible to parse. Keep
--tests-dir as a named required option which matches the previous UX.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move testsDirOpt after resultsDirOpt, drop evaluate from CONTRIBUTING examples

Reorder testsDirOpt declaration and registration to follow resultsDirOpt.
Remove the evaluate subcommand from CONTRIBUTING.md dotnet run examples
since the launch profile already provides it.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-18 08:59:03 +00:00
Viktor Hofer bd965cbd55 Require /evaluate command for PR evaluation (#394) 
* Require /evaluate command for PR evaluation

Update evaluation.yml to require an /evaluate command to trigger evaluation,
matching the behavior in evaluation-fork-pr.yml. This change:

- Adds pr-status job that posts initial commit status on PR open/sync:
  - 'success' if no skills changed (PR not blocked)
  - 'pending' if skills changed (maintainer must post /evaluate)
- Adds gate job to validate /evaluate command (permissions, same-repo check)
- Replaces evaluation-status check-run with report-status using commit status API
- Scheduled runs continue to work unchanged (daily evaluation of all plugins)

The evaluation-status required check still works because:
- PRs without skill changes get immediate success status
- PRs with skill changes get pending status until /evaluate is posted

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Merge evaluation and evaluation-fork-pr into single workflow

Consolidate evaluation.yml and evaluation-fork-pr.yml into one unified
workflow that handles both same-repo and fork PRs via conditionals:

- pr-status: posts initial commit status for same-repo PRs (pull_request)
- fork-pr-status: posts initial commit status for fork PRs (pull_request_target)
- gate: unified /evaluate command validation with is_fork output
- discover: unified skill discovery with conditional worktree for forks
- report-status: single job for posting final commit status
- Scheduled runs and publish-benchmark unchanged

Also fixes:
- run-evaluation now uses always() so scheduled runs aren't skipped
- report-status scoped to issue_comment events only
- Removed stale evaluation-fork-pr.yml reference from infra detection regex
- Updated evaluation-run.yml comment

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback

- Fetch PR head and use worktree for both fork and same-repo PRs in discover
- Always pass checkout-ref (PR head SHA) to evaluation-run for /evaluate
- Fix runs count: use event_name == 'schedule' instead of github.ref

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Require /evaluate for infra changes too

Expand pr-status and fork-pr-status discovery to detect evaluation workflow
and skill-validator changes in addition to skill/test changes. Both now post
pending status requiring /evaluate when infra files are modified.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Build skill-validator from PR branch for non-fork PRs

Add is-fork input to evaluation-run.yml. When false (same-repo PRs), the
validator is built from checkout-ref so skill-validator changes are tested.
When true (fork PRs), the validator is built from the base branch to prevent
untrusted code from modifying tooling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review: fix security comment, add issues:write

- Update security model comment to reflect validator-ref behavior
- Add issues: write permission to gate and report-status jobs for
  reaction and comment API calls

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Add prominent note about issue_comment trigger limitation

Changes to this workflow file in a PR won't take effect until merged
because issue_comment always runs from the default branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Don't require /evaluate for workflow YAML changes

Workflow YAML changes can't be tested via /evaluate anyway (issue_comment
always runs from main). Only eng/skill-validator/ changes now require
/evaluate in the pr-status jobs. The discover job still detects workflow
changes to trigger all-plugin evaluation when /evaluate is run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
 2026-03-18 09:15:42 +01:00
Jeff Handley 646b5d93b0 Add Skill: Convert to Central Package Management (convert-to-cpm) (#75) 
* Add convert-to-cpm skill for NuGet Central Package Management conversion

Authoring process:
- Researched the Agent Skills standard at agentskills.io and reviewed repo conventions
- Studied the existing csharp-scripts skill and test structure as a reference implementation
- Studied Microsoft's CPM documentation at learn.microsoft.com
- Authored SKILL.md with a 9-step workflow covering scope determination through validation
- Created 5 test scenarios across simple, moderate, and advanced tiers
- Performed dry-run evaluation against all scenarios, identifying 15 findings
- Applied all 15 fixes (4 gaps, 6 ambiguities, 5 improvements) to SKILL.md
- Re-evaluated: 50 OK, 0 gaps, 0 ambiguities, 1 test fixture improvement
- Updated test fixture to resolve the remaining finding
- Final validation: 254 lines, 0 errors, 0 warnings

Skill implementation (skills/convert-to-cpm/SKILL.md):
- 9-step workflow: scope, CPM check, audit, create/update props, update projects, handle properties, restore/validate, cleanup, summary
- Handles single projects, solutions, and full repositories
- Audits packages with per-project detail table including version conflicts and security advisories
- Flags CVE/security vulnerabilities and recommends patched versions
- Supports MSBuild property-based versioning with inline vs keep decisions
- Covers conditional PackageReference items across target frameworks
- Discovers PackageReference in imported .props/.targets via Import chain scanning
- Provides cross-platform commands (Unix and Windows/PowerShell)
- Documents import order requirements for property references in Directory.Packages.props
- Mentions dotnet new packagesprops as preferred creation method (.NET 8+)
- Includes validation checklist and common pitfalls table

Test scenarios (tests/convert-to-cpm/):
- simple-single-project: 1 project, 3 packages, no conflicts
- simple-solution: 3 projects via .sln, 5 packages, PrivateAssets preservation
- moderate-version-conflicts: 4 projects, System.Text.Json version conflicts with CVE-2024-43485 advisory, AutoMapper major version conflict with VersionOverride
- moderate-msbuild-properties: 2 projects using MSBuild properties for versioning in Directory.Build.props, inline vs keep decisions, property cleanup verification
- advanced-multi-complexity: 5 projects combining version conflicts, MSBuild properties, conditional TFM PackageReference, shared Common.props imports, VersionOverride, and security advisory handling

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refactor SKILL.md with references, baseline build, and package comparison

Condensed SKILL.md using progressive disclosure:
- Reduced from 254 lines / ~3937 tokens to 120 lines / ~1778 tokens
- Extracted detailed procedural instructions into 5 focused reference files
- Main file retains workflow decision logic with links to references loaded on demand
- Follows agentskills.io progressive disclosure pattern: metadata -> instructions -> resources

Added baseline build verification (new Step 2):
- Requires clean build (dotnet clean && dotnet build) before any changes
- Captures baseline.binlog for structured build artifact
- Captures baseline-packages.json via dotnet list package --format json
- Blocks conversion if baseline build fails

Added post-conversion baseline comparison (Steps 8 and 10):
- Step 8 produces after-cpm.binlog and after-cpm-packages.json from clean build
- Step 10 compares baseline vs post-conversion package lists per project
- Changes table shows version bumps, VersionOverride entries, added/removed packages
- Unchanged table confirms version-neutral conversion for unmodified packages
- Binlog files produced as artifacts for user review in MSBuild Structured Log Viewer

Reference files created:
- references/audit-complexities.md: 7 complexity categories for package auditing
- references/baseline-comparison.md: dotnet list package comparison procedure, binlog production, artifact guidance, table formats for changes and unchanged packages
- references/directory-packages-props.md: File creation (dotnet new packagesprops), placement rules, conditional versions, VersionOverride patterns
- references/msbuild-property-handling.md: Property usage search, inline vs keep decisions, import order requirements, cleanup verification
- references/validation-and-errors.md: NuGet error codes (NU1008, NU1010, NU1507), clean build validation, multi-TFM guidance, common pitfalls

Workflow is now 10 steps (was 9):
1. Determine scope
2. Establish baseline build (new)
3. Check for existing CPM
4. Audit package references
5. Create or update Directory.Packages.props
6. Update project files
7. Handle MSBuild version properties
8. Restore and validate (now with binlog + package list capture)
9. Clean up obsolete properties
10. Summary and baseline comparison (expanded with diff tables)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Replace third-party packages with Microsoft equivalents, add baseline/comparison steps, add .gitignore for test solutions

Package replacements across all skill references and test scenarios:
- Serilog.AspNetCore -> OpenTelemetry.Extensions.Hosting 1.15.0
- Serilog (standalone) -> OpenTelemetry.Extensions.Hosting 1.15.0
- Swashbuckle.AspNetCore -> Microsoft.AspNetCore.OpenApi 8.0.24
- SerilogVersion MSBuild property -> OTelVersion
- DIVersion updated from 8.0.1 to 9.0.0 (DI/Abstractions alignment)
- Microsoft.AspNetCore.OpenApi uses 8.0.24 (not 9.x) for net8.0 compat

Test fixture improvements:
- Added Step 2 (baseline build with binlog) to all 5 with-skill.md files
- Added Step 10 (package comparison table + binlog artifact links) to all 5 with-skill.md files
- Expanded simple-solution without-skill.md with more realistic agent behavior
- Standardized security fix terminology: table markers cite CVEs, prose references CVE IDs
- Removed space-padded table formatting in favor of compact markdown with emojis

Test solution infrastructure:
- Added tests/convert-to-cpm/.gitignore to exclude src/ build artifacts
- Added generation prompt to test README for recreating buildable solutions
- All 5 test solutions verified to restore and build successfully

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review feedback on terminology, placement, and reference accuracy

- Rename Step 10 and validation checklist from 'binlog comparison' to 'package list comparison'
- Rewrite baseline-comparison.md: title to 'Baseline Comparison', lead with dotnet list package as primary comparison mechanism, position binlogs as supplementary artifacts for manual review/troubleshooting
- Add brief definition of binlogs before the artifacts section
- Fix Directory.Packages.props placement guidance: use first common ancestor of all .NET projects instead of assuming repository root (.git), since many repos nest source under src/ or similar
- Update SKILL.md scope input and Step 1 to reference directory-scoped conversion
- Standardize security fix terminology: table markers cite CVEs, prose uses 'security fix for CVE-...'

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address review findings: sort PackageVersion, add packages.config scenario, fix commands

Review findings addressed:
1. Fixed BlobsVersion grep step description in advanced scenario
2. Sorted PackageVersion entries alphabetically in all Directory.Packages.props outputs
3. Consolidated baseline-comparison.md to single-table format
4. Added solution file arguments and multi-line commands in test fixtures
5. Added Directory.Packages.props placement question for single-project scenario
6. Created simple-packages-config test scenario (packages.config rejection)
7. Use dotnet new packagesprops in simple-single-project workflow
8. Fixed More Info link text to match reference file title

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix test scenario accuracy: sln/slnx mix, package counts, ImplicitUsings

- Use mix of .sln and .slnx across scenarios (simple-solution and
  moderate-msbuild-properties use .sln; moderate-version-conflicts and
  advanced-multi-complexity use .slnx)
- Fix simple-solution package count from 5 to 6
- Fix advanced user response #2 to reference Azure.Storage.Blobs version
- Add ImplicitUsings to advanced Directory.Build.props in audit and cleanup
- Complete the advanced unchanged table with all expected package entries

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add skill evaluation section to test README

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Inventory.sln references and make evaluation opt-in

- Fix 4 occurrences of Inventory.sln to Inventory.slnx in
  moderate-version-conflicts/with-skill.md for consistency with the
  scenario README and test solution
- Update test README to only run skill evaluation when the user
  explicitly requests it

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Condense SKILL.md with progressive disclosure

- Remove duplicated bash code blocks from Steps 2 and 8, linking to
  baseline-comparison.md instead
- Merge Steps 7 (MSBuild property decisions) and 9 (cleanup) into a
  single step since both reference msbuild-property-handling.md
- Condense Step 10 summary to reference baseline-comparison.md for
  procedure details
- Reduces from 120 to 97 lines (10 steps to 9 steps)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Enterprise.sln references to Enterprise.slnx in with-skill.md

Address code review feedback: fix 4 occurrences of Enterprise.sln to
Enterprise.slnx in advanced-multi-complexity/with-skill.md for
consistency with the scenario README and test solution.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve convert-to-cpm skill: preserve whitespace, save summary report, fix links

- Instruct Step 6 to retain each file's existing indentation style and
  blank lines to keep diffs minimal
- Instruct Step 9 to save the full summary as convert-to-cpm.md alongside
  binlog and JSON artifacts, suggesting its contents for a PR description
- Strip en-us/ from learn.microsoft.com links in SKILL.md and
  validation-and-errors.md to make them culture-agnostic
- Update all with-skill.md test expectations to include the convert-to-cpm.md
  create step and artifact mention

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Indicate that the binlog viewer is for the user; not the tool/skill

* Move convert-to-cpm skill and tests to src/dotnet-msbuild layout

Move skill from skills/convert-to-cpm/ to src/dotnet-msbuild/skills/convert-to-cpm/
and tests from tests/convert-to-cpm/ to src/dotnet-msbuild/tests/convert-to-cpm/
to match the repository layout established on main.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Convert tests from with/without-skill format to eval.yaml

Replace manual test approach (README.md + with-skill.md + without-skill.md)
with automated eval.yaml scenarios containing assertions and rubric items.
Add committed test fixture files (.csproj, .sln, .slnx, .props) for each
scenario with copy_test_files setup, replacing the git-ignored generated
solutions.

6 scenarios converted:
- simple-single-project: basic single project CPM conversion
- simple-solution: multi-project solution conversion
- simple-packages-config: negative test (packages.config rejection)
- moderate-version-conflicts: version conflict resolution
- moderate-msbuild-properties: MSBuild property version handling
- advanced-multi-complexity: combined complexities (conflicts, properties,
  conditionals, shared props)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Consolidate eval.yaml files into single root eval.yaml

Move all 6 scenario definitions from individual subdirectory eval.yaml
files into a single src/dotnet-msbuild/tests/convert-to-cpm/eval.yaml
file, matching the skill-validator discovery convention that expects
{testsDir}/{skillname}/eval.yaml. Scenario prompts updated to reference
subdirectory-prefixed paths since copy_test_files now copies all fixture
subdirectories.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reorganize convert-to-cpm into plugins/tests folders

* Add post-conversion reporting rubrics to CPM eval scenarios

Add rubric items for baseline-vs-result comparison reporting, binlog
capture and guidance, version reconciliation, and risk/reassurance
messaging across all conversion scenarios. Increase complex scenario
timeout to 360s.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve post-conversion reporting, conflict resolution, and skill activation

SKILL.md:
- Surface binlog capture commands explicitly in Steps 2 and 8
- Expand Step 4 with detailed version conflict presentation and
  resolution workflow
- Restructure Step 9 into 5-section post-conversion report template:
  overview, conflict resolutions, baseline comparison, risk assessment,
  and artifacts/next steps with binlog education
- Broaden activation triggers in description and When to Use to match
  update/bump/upgrade/align/sync terminology from ecosystem PRs
- Add activation for out-of-sync/conflicting/mismatched version scenarios

audit-complexities.md:
- Expand version conflict section with per-conflict presentation format,
  4 resolution options with trade-offs, and risk-level guidance

eval.yaml:
- Pre-specify conflict resolution strategy in prompts to avoid blocking
  on unanswered user prompts during single-turn evaluation
- Add 3 new scenarios testing CPM recommendation when user asks to
  update/align packages without mentioning CPM
- Update rubrics to match directed resolution strategy
- Increase complex scenario timeout to 360s

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR feedback from kartheekp-ms

- Use GitHub markdown URL for CPM docs reference (SKILL.md)
- Use dotnet package list (renamed in .NET 10) with backward compat
  note for dotnet list package (SKILL.md, baseline-comparison.md)
- Use dotnet package list --format json for auditing (SKILL.md Step 4)
- Document Directory.Packages.props sub-folder support for test
  projects with different dependencies (directory-packages-props.md)
- Remove security advisory version bumping from CPM conversion scope;
  instead surface advisories as follow-up items in the post-conversion
  report (audit-complexities.md, SKILL.md Step 9 Section 5)
- Add Follow-up items section to report template covering security
  advisories, deprecated packages, version alignment opportunities,
  and test validation (SKILL.md Step 9)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve convert-to-cpm skill activation, artifact preservation, and eval assertions

- Broaden skill description to trigger on version drift/sync scenarios and
  MSBuild property contexts even when CPM is not explicitly mentioned
- Add explicit instructions to preserve binlog and JSON artifacts (not delete)
- Make convert-to-cpm.md report a mandatory deliverable, not optional
- Replace fragile output_contains assertions (ManagePackageVersionsCentrally,
  Directory.Packages.props) with file_contains checks on the actual generated
  files across 5 conversion scenarios

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Extend timeouts and add CODEOWNERS entry

* Move convert-to-cpm skill into a new dotnet-nuget plugin

* Fix erroneous plugin msbuild/nuget plugin split change. Add dotnet-nuget to marketplace.

* Add NuGet package management skills to README

* Address PR feedback: clarify CPM sub-folder semantics, remove security-fix example, require explicit conflict strategy

- directory-packages-props.md: Clarify that sub-folder Directory.Packages.props
  files are independent and replace (not extend) the parent; require explicit
  MSBuild Import to chain files
- baseline-comparison.md: Replace security-fix upgrade example with version
  conflict alignment example, consistent with audit guidance that security
  advisories are follow-up items
- SKILL.md: Remove default 'use highest version' conflict strategy; require
  explicit user choice or confirmation when conflicts are detected

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Extend eval timeouts for scenarios hitting time limits

- Convert single project: 180s -> 270s
- Recommend CPM (version conflicts): 180s -> 270s
- Recommend CPM (complex repo): 360s -> 540s
- Convert complex repository: 360s -> 540s

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add missing CODEOWNERS for dotnet-nuget plugin. Reorder convert-to-cpm evals and expand timeouts.

* Remove containing skill mentions from reference files; fix ordering

* Replace output_matches with deterministic file assertions in eval.yaml

Convert 6 non-deterministic output_matches assertions to 8 file-based
assertions (file_contains, file_not_contains) across conversion scenarios:

- Scenario 5: Verify PrivateAssets preserved in Tests.csproj
- Scenario 6: Verify OTelVersion/DIVersion removed from Directory.Build.props
- Scenario 7: Verify System.Text.Json and Azure.Identity in Directory.Packages.props
- Scenario 8: Verify System.Text.Json in Directory.Packages.props and
  BlobsVersion/HostingVersion removed from Directory.Build.props

Recommendation-only scenarios and conversational assertions (conflict
detection, conditionals, Common.props awareness) remain as output_matches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reorganize skill description with USE FOR / DO NOT USE FOR markers

Restructure the description to improve activation reliability by
adopting the USE FOR / DO NOT USE FOR pattern used by other skills
in the repo (thread-abort-migration, binlog-failure-analysis).

Key changes:
- Front-load activation cues before capability list
- Add explicit USE FOR / DO NOT USE FOR section markers
- Add .sln/.slnx file extensions for solution-path matching
- Move DO NOT USE exclusions into description (was only in body)
- Use YAML folded scalar for multi-line readability

All existing activation keywords preserved. Length reduced from
882 to 849 chars (max 1024).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix issues for external link references

* Increase timeouts

* Handle packages.config non-activation in convert-to-cpm skill

Set expect_activation: false on the packages.config eval scenario so
non-activation is reported as expected rather than flagged as a warning.

Add a packages.config guard in Step 1 of the skill workflow so that if
the skill is activated for a packages.config project, it stops early and
recommends migrating to PackageReference first.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove leading blank lines from .sln test files

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Use imperative 'Read' instead of 'See' for reference file links

Ensures the agent explicitly loads and reads reference files rather than
treating them as optional 'see also' links. This aligns with the repo
convention of explicitly instructing agents to load reference content.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Simplify dotnet-nuget CODEOWNERS

* Simplify packages.config guard

* Add minimal Program.cs entrypoints to Web/Worker test fixtures

Test fixture projects using Microsoft.NET.Sdk.Web and
Microsoft.NET.Sdk.Worker require a Program.cs entrypoint for dotnet build
to succeed. The skill workflow instructs the agent to run dotnet restore
and dotnet build for validation, so these fixtures must be buildable.

Adds minimal entrypoints (with explicit using directives since
ImplicitUsings is not enabled in these projects) to 6 projects:
- simple-solution/Web
- moderate-version-conflicts/Api, Worker
- moderate-msbuild-properties/Api
- advanced-multi-complexity/Api, Web

Also adds Microsoft.Extensions.Hosting package reference to the Worker
project, matching what the dotnet new worker template produces.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
 2026-03-18 08:41:57 +01:00
Copilot 0bb3606942 Fix release job skipped by transitive dependency chain from discover (#395) 
* Initial plan

* Fix release job being skipped due to tainted dependency chain from discover

The release job was always skipped for schedule and workflow_dispatch
events because the discover job (a transitive dependency via build) is
skipped for non-PR events, which taints the dependency chain. Without
always() in its if condition, GitHub skips release without evaluating
the expression.

Add always() so the if expression is always evaluated, and add an
explicit needs.build.result == 'success' guard so the release only
runs when the build actually passed.

Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
 2026-03-17 13:30:30 -07:00
Viktor Hofer 1a20e39a8d Refactor skill-validator: add check command, rename validate to evaluate (#388) 
* Refactor skill-validator: add check command, rename validate to evaluate

- Add new 'check' command for CI skill validation (frontmatter, structure)
- Rename ValidateCommand to EvaluateCommand to clarify its purpose
- Add skill-check.yml GitHub Actions workflow
- Refactor SkillDiscovery for improved skill detection
- Simplify OverfittingJudge and SkillProfiler
- Update tests to match refactored APIs

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix check subcommand: use EvaluateCommand as root command

System.CommandLine 2.0.4 requires the primary command to be the
RootCommand. Restores the upstream pattern where the evaluate/validate
command is the root and check/consolidate/rejudge are subcommands.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Pass all plugins in single check invocation

Build plugin args dynamically and pass them all at once instead of
looping per plugin.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review comments

- Remove 'evaluate' subcommand token from README, evaluation-run.yml,
  and RunArguments in csproj (evaluate is now the root command)
- Wrap ParsePluginJson in try/catch for JsonException with clean error
- Add CheckCommandTests with aggregate description limit coverage
  (under/at/over limit and multi-plugin cases)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-17 17:00:47 +01:00
Amaury Levé c55e4f222b Add MSTest migration skills (v1/v2→v3, v3→v4) (#377) 
* Add MSTest migration skills (v1/v2→v3, v3→v4)

New dotnet-test plugin with two step-by-step migration skills:

- migrate-mstest-v1v2-to-v3: covers assembly ref (v1) and NuGet (v2) projects,
  assertion overload changes, DataRow constructors, .testsettings replacement,
  timeout unification, and MSTest.Sdk adoption.
- migrate-mstest-v3-to-v4: covers Execute→ExecuteAsync, CallerInfo constructors,
  ClassCleanupBehavior removal, TestContext.Properties generic change, Assert API
  changes, ExpectedExceptionAttribute removal, dropped <net8.0 TFMs, and
  behavioral changes (DisableAppDomain, TestCase.Id, TreatDiscoveryWarningsAsErrors).

Includes eval.yaml test scenarios and fixtures for both skills, plugin.json,
and CODEOWNERS entries.

* Fixes

* Improve results

* Improve

* Improve description

* Copy fix for path

* Bump timeout
 2026-03-17 14:25:45 +01:00
Amaury Levé d5d2136157 Update code owners to use team alias (#382) 2026-03-17 08:13:48 +01:00
Amaury Levé 2a8388dd71 Add VSTest to MTP migration skill (#373) 
* Add VSTest to MTP migration skill

New migrate-vstest-to-mtp skill in the dotnet-test plugin covering:

- Framework-specific MTP runner enablement (MSTest, NUnit, xUnit.net)
- Directory.Build.props centralization of OutputType, runner properties
- MSTest.Sdk UseVSTest removal
- NUnit3TestAdapter 5.0.0+ upgrade requirement
- xunit v2 to xunit.v3 upgrade requirement
- dotnet test CLI argument translation (--logger to --report-trx, --collect
  to --coverage, --blame-crash to --crashdump, --blame-hang to --hangdump)
- xUnit.net --filter syntax migration to --filter-class/--filter-trait/
  --filter-query
- global.json native MTP mode (.NET 10+) vs TestingPlatformDotnetTestSupport
  VSTest-bridge mode (.NET 9 and earlier)
- Azure DevOps VSTest@3 to DotNetCoreCLI@2 pipeline migration
- MTP extension NuGet packages (TrxReport, CrashDump, HangDump, CodeCoverage)
- Exit code 8 behavioral difference (--ignore-exit-code 8,
  TESTINGPLATFORM_EXITCODE_IGNORE)

Includes 10 eval scenarios with tiered timeouts (240-360s), 5 fixture
directories, plugin.json, marketplace.json entries, CODEOWNERS, and
README.md updates.

* Fix codeowners

* Improve

* Improve further

* Improve prompt

* Improve further the description

* Address review comments
 2026-03-16 23:56:48 +01:00
Amaury Levé 45c66f87d4 Add prompt to evaluate skills description (#371) 2026-03-16 13:14:55 +01:00
Jan Krivanek 93ccead1e7 Adjust for timeout issues (#354) 2026-03-15 21:08:51 +00:00
Amaury Levé 1b8f2aa1e4 Address Copilot review comments on test agent/skills (#351) 
Follow-up from PR #343 where copilot added review after the PR was merged
 2026-03-15 06:45:40 +00:00
Dan Moseley 1ccd28b1ba Make publish-benchmark resilient to individual plugin eval failures (#365) 
* Make publish-benchmark resilient to individual plugin eval failures

When any single evaluate matrix leg fails (e.g. OOM kill), the entire
run-evaluation reusable workflow reports failure. Previously this caused
publish-benchmark to be skipped, blocking dashboard updates for ALL
plugins -- even the ones that succeeded.

Changes:
- publish-benchmark: use !cancelled() so it runs when some legs fail
- download-artifact: continue-on-error so missing artifacts don't abort
- The existing Generate Benchmark Data step already handles missing
  plugin results with Write-Warning and continue

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Make continue-on-error conditional on evaluation result

Only tolerate artifact download failures when run-evaluation didn't
fully succeed. When all legs passed, a download failure is a real error
that should fail the job.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-14 07:46:40 +01:00