skills/dotnet__skills
1
0
Fork 0
mirror of https://github.com/dotnet/skills.git synced 2026-09-20 09:49:54 +08:00
Code Issues Packages Projects Releases Wiki Activity
589 Commits 131 Branches 2 Tags
eval-exec-opus48
Commit Graph

8 Commits

Author SHA1 Message Date
Aditya Mandaleeka 52ba152ed6 Update to Vally 0.7.0 (#854) 2026-07-10 02:19:48 +00:00
Aditya Mandaleeka 9433a31e67 Initial port of evals to Vally (#615) 2026-05-06 10:11:45 -07:00
Milos Kotlar 038dd4f087 Add Apple crash symbolication skill (#201) 
* Add SKILL.md for iOS crash symbolication process

Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update plugins/dotnet/skills/ios-crash-symbolication/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add automation script, tests, and review fixes for ios-crash-symbolication

- Add Symbolicate-Crash.ps1 (664 lines): parses .ips JSON, searches
  local dSYMs (SDK packs, NuGet cache, user paths), verifies UUIDs via
  dwarfdump, batch-symbolicates with atos, identifies runtime version
- Update SKILL.md: add Automation Script and Runtime Version
  Identification sections, fix atos -o to point inside dSYM bundle,
  add dwarfdump and MAUI to frontmatter, wrap steps in Workflow
  heading for consistency with Android sibling, fix misleading
  'rebuild' guidance per review feedback (Rolf)
- Add eval.yaml with 7 test scenarios: Mono crash, CoreCLR crash,
  no .NET frames, NativeAOT, multi-library UUIDs, ASI field
  priority, and Android tombstone rejection
- Add 5 .ips test fixture files (two-part JSON format)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add iOS crash log and evaluation scenarios for symbolication tests

* Remove unused file

* Add CODEOWNERS entry for iOS crash symbolication

* Rename ios-crash-symbolication to apple-crash-symbolication

Address PR feedback to support all Apple platforms (tvOS, Mac Catalyst,
macOS) not just iOS:

- Add $appleRids array covering ios, tvos, maccatalyst, and osx RIDs
- Refactor Find-Dsym and Find-RuntimeVersion to search all platform packs
- Rewrite SKILL.md for orchestration focus and reduced token budget
- Extract domain knowledge to references/ips-crash-format.md
- Tune eval.yaml: outcome-based rubrics, broad assertions, overfit 0.26

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix script robustness: base address parsing and null-safe property access

- Simplify base address cast to [uint64] without manual hex prefix stripping
- Use PSObject.Properties check before accessing lastExceptionBacktrace

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add dSYM-path rubric items to CoreCLR and ASI scenarios

Add rubric items testing for skill-specific domain knowledge (SDK pack
paths, NuGet cache directories) that baseline agents cannot provide.
This creates the quality delta needed to pass the 10% improvement
threshold while keeping overfit at 0.12 (Low).

3-run validation: 30.1% improvement, 5/7 scenarios positive.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refactor code structure for improved readability and maintainability

* Fix crash symbolication script for real .ips format and improve eval scenarios

- Fix Symbolicate-Crash.ps1 strict-mode bugs with real .ips crash logs:
  safe property access for image name/path (sentinel entries), thread name
  (most threads unnamed), and single-element array unwrapping (.Count)
- Add SKILL.md efficiency guidance: resolve script path from skill directory
  (no find /), start with -ParseOnly, don't run broad filesystem searches
- Reframe eval scenarios for platform-independent evaluation (parse/analyze
  instead of requiring macOS-only atos/dwarfdump), tighten rubric to test
  skill-specific knowledge (NuGet package name, all .NET binary images)

Validation results (3 runs, claude-opus-4.6):
  Scenario 1 (parse frames):       3.3 → 4.0 (+0.7) ✅
  Scenario 2 (investigate crash):   3.3 → 4.0 (+0.7) ✅
  Scenario 3 (reject Android):      3.3 → 4.0 (+0.7) ✅

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Auto-fallback to parse-only output when atos is unavailable

On Linux (CI), atos and xcrun don't exist. Previously the script would
error out after completing all parsing, losing the results. Now it
detects the missing tool and falls back to ParseOnly output automatically,
ensuring the agent always gets structured parse data regardless of platform.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Move apple-crash-symbolication to dotnet-diag plugin

Relocate skill from plugins/dotnet to plugins/dotnet-diag and tests
from tests/dotnet to tests/dotnet-diag. Update CODEOWNERS accordingly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve apple-crash-symbolication: macOS symbols, bug fixes, training log

- Fix JSON case-conflict parsing (vmRegionInfo/vmregioninfo duplicate keys)
- Fix strict-mode safe access for optional asi field
- Expand Step 4 with macOS-specific symbol package guidance (.symbols NuGet)
- Add .dwarf to .dSYM conversion instructions
- Add src/coreclr/ to validation paths
- Soften stop signals to allow crash analysis and deeper investigation
- Add macOS Symbol Packages and JSON Parsing Gotchas to reference doc
- Create training log documenting session findings
- Add .github/skills/ project-local copy for CLI testing

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* apple-crash-symbolication: automated version extraction and symbol acquisition

Script improvements:
- Preserve full image paths from crash log (previously discarded by GetFileName)
- Add Get-RuntimeVersionFromPath: extracts .NET version from image paths
  (e.g., .../shared/Microsoft.NETCore.App/10.0.4/libcoreclr.dylib)
- Add Get-RidFromPath: infers RID from path or crash metadata (OS/CPU)
- Path-based version detection as fast primary method, UUID matching as fallback
- Emit copy-pasteable symbol acquisition commands when dSYMs are missing
- Show .NET version in ParseOnly library listing

SKILL.md updates:
- Step 2: document automated version detection and acquisition commands
- Step 4: script now prints ready-to-run download/conversion commands

Training log: record session 2 findings (5 issues, script + SKILL.md changes)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* apple-crash-symbolication: add symbol server anti-pattern

dotnet-symbol and msdl.microsoft.com do not serve macOS dSYM/DWARF
symbols — only Windows PDBs and Linux ELF debug info. NuGet packages
are the only public source. Added anti-pattern to SKILL.md (both
copies) and reference doc to prevent wasted tool calls.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* apple-crash-symbolication: fix symbol server guidance — dotnet-symbol works for macOS

dotnet-symbol --symbols <binary> successfully downloads .dwarf debug
symbols for macOS Mach-O binaries from msdl.microsoft.com. Previous
commit incorrectly claimed this didn't work. Replaced anti-pattern
with positive guidance in SKILL.md (both copies) and reference doc.
Added training log entry documenting the correction.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Promote dotnet-symbol as preferred macOS symbol acquisition method

- Script: macOS acquisition block now shows Option A (dotnet-symbol,
  preferred) and Option B (.symbols NuGet, fallback). Fixed .dwarf
  filename doubling bug in cp command.
- SKILL.md (both copies): Step 2 updated for dotnet-symbol preference,
  Step 4 reordered with dotnet-symbol as #2 and NuGet symbols as #3.
- Reference doc: Restructured macOS Symbol Packages section with
  Preferred/Fallback subsections and shared .dwarf→.dSYM conversion.
- Training log: Session 4 entry documenting the promotion.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add automated symbol server download for macOS crash symbolication

- Add Get-DebugSymbols function: downloads .dwarf files from Microsoft
  symbol server using Mach-O UUID (mirrors android tombstone approach)
- Add Convert-DwarfToDsym function: creates .dSYM bundle from .dwarf
  with UUID verification via dwarfdump
- New params: -SymbolCacheDir, -SymbolServerUrl, -SkipSymbolDownload
- Wire download+conversion into main flow after local dSYM search
- Refactor manual acquisition guidance as fallback-only
- Update both SKILL.md copies: frontmatter, Step 4, new flags
- Update ips-crash-format.md: automated download as primary method
- Add training log session 5

URL pattern: https://msdl.microsoft.com/download/symbols/_.dwarf/mach-uuid-sym-{UUID}/_.dwarf
Verified end-to-end: 391/391 .NET frames symbolicated with clean cache.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* apple-crash-symbolication: add triage order guidance to Step 3

Step 3 now instructs the agent to explain the faulting mechanism
(frames #0-#1) before examining cross-thread context. This addresses
a misdiagnosis where GC activity on neighboring threads was mistaken
for causation when the actual root cause (_sigtramp -> NULL signal
handler) was visible in the crashing thread's first two frames.

Training log updated with session retrospective and corrected the
original crash description from GC race to NULL signal handler.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove stale .github/skills/apple-crash-symbolication remnant

Skill lives in plugins/dotnet-diag/skills/apple-crash-symbolication/ since
the plugin restructuring. The old .github/skills/ copy was a leftover.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add libimobiledevice.org to known-domains.txt

The reference scanner CI check fails because the apple-crash-symbolication
skill references https://libimobiledevice.org/ which is not in the allowed
domains list. This domain hosts the libimobiledevice project, a community
library for communicating with iOS devices.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Delete eng/reference-scanner/known-domains.txt

* Apply suggestion from @Copilot

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Add libimobiledevice.org to known-domains allowlist

The apple-crash-symbolication SKILL.md references libimobiledevice.org
for the idevicecrashreport tool. Add the domain to the known-domains
file to fix the skill-check CI reference validation failure.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Restructure apple-crash-symbolication for analysis-first approach

The skill was getting ❌ verdicts on all eval scenarios because it
directed the LLM to 'run the script' instead of teaching crash analysis
domain knowledge. This mirrors the problem where quality decreased from
3.0 to 2.7 in the parse scenario.

Restructure to follow the Android sibling's proven pattern:
- Lead with parsing (.ips two-part JSON format, key fields)
- Teach .NET library identification (inline library table)
- Teach crash interpretation (asi, faulting thread, exception)
- Teach atos command construction (concrete examples)
- Teach dSYM search paths (ordered list with commands)
- Move automation script to optional section at the end
- Move crash log retrieval to a separate section (not Step 1)

Trim the reference doc to avoid duplication, keeping only macOS
symbol distribution differences and supported RID list.

Token budget: ~2,100 tokens (within 800-2,500 optimal range).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Refine prompts and assertions in eval.yaml for iOS crash symbolication scenarios

* Add format verification guard and fix overfitting in reject scenario

- SKILL.md: Add explicit format check at start of Step 1 to verify
  .ips JSON before proceeding; add 'Wrong file format' stop signal
- eval.yaml: Remove direct skill reference from scenario 3 prompt
  to reduce overfitting (was triggering skill activation in plugin mode)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix scenario 3: provide actual Android tombstone file for rejection test

- eval.yaml: Replace invalid 'extra_files' with correct 'files' syntax
  (extra_files was silently ignored, so crash_android.txt was never copied)
- eval.yaml: Remove copy_test_files for scenario 3 to avoid copying
  ios_crash.ips which tempts the agent in plugin mode
- Add tombstone_sample.txt to test directory (copy from android sibling)
- SKILL.md: Mention ndk-stack/addr2line in wrong-format stop signal

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Update tests/dotnet-diag/apple-crash-symbolication/tombstone_sample.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update eng/known-domains.txt

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Address review feedback: simulator RIDs, perf, safety, accuracy fixes

- Add simulator RIDs (iossimulator, tvossimulator) to script search list
- Build hashtable for O(1) image lookups in Get-ThreadFrames
- Remove -UseBasicParsing (unnecessary in pwsh 7+)
- Include UUID in Convert-DwarfToDsym cache path; sanitize library name
- Make version regex greedy to capture pre-release suffixes
- Improve format detection error message for non-.ips files
- Guard xcrun fallback with Get-Command check
- Add .dwarf-to-.dSYM conversion guidance for macOS manual fallback
- Escape .ips in eval.yaml regex assertion
- Fix UUID note in reference doc (normalize, not assume lowercase)
- Fix symbol server example to use crash-log image name

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address Copilot review round 2: case-sensitive replace, dedup, path fix, redact PII

- Use -creplace with guard for vmregioninfo duplicate key handling
- Use Sort-Object -Unique for proper library deduplication
- Fix macOS fallback to create .dSYM bundles under symbols-out/
- Redact device identifiers in test fixture .ips file

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Prevent dSYM cache poisoning: validate UUID on cache hit, remove on mismatch

- Convert-DwarfToDsym now verifies cached dSYM UUID before reusing
- On UUID mismatch during download, remove the bad cached bundle so
  subsequent runs can retry cleanly

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix cache validation: normalize UUID comparison, clean up dwarf + bundle on mismatch

- Use Format-Uuid to normalize dwarfdump output before comparing to
  already-normalized \ in Convert-DwarfToDsym cache check
- On UUID mismatch after download, remove the entire .dSYM bundle
  directory and the cached .dwarf file to prevent repeat failures

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Resolve-Frames null handling: use List[object] to preserve null entries

PowerShell array += \ silently drops the element, breaking
index alignment between results and input addresses. Switch to
List[object].Add() which correctly preserves null entries.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix dSYM bundle root traversal, qualify version-from-path note

- Walk up directory tree until *.dSYM is found instead of going up
  only 2 levels (which lands at Contents/Resources, not the bundle)
- SKILL.md: note that version-in-path only works on macOS shared
  framework installs; iOS paths don't embed the runtime version

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove incorrect .github/skills path reference from training log

The .github/skills/ directory doesn't exist in this repo. The file
was already listed under its actual plugins/dotnet-diag/ path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Detect simulator for RID inference in manual fallback guidance

Detect CoreSimulator in image paths to use iossimulator-/tvossimulator-
RIDs for simulator crashes, avoiding UUID mismatches from wrong runtime
packs. Also handle arm64e CPU type.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix Resolve-Frames return type and scope issue in simulator detection

- Return \.ToArray() instead of ,\ to avoid wrapping
  the list in a single-element array
- Use \.usedImages instead of undefined \ in RID
  inference block

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Clarify ParseOnly .NET Libraries section shows only frame-relevant images

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix YAML escape in assertion pattern (use single quotes for regex backslash)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Tighten stop signal to prevent Android symbolication spillover

The apple-crash-symbolication skill's stop signal for wrong file formats
was naming specific Android tools (ndk-stack, addr2line) and the
android-tombstone-symbolication skill. This caused models to learn about
Android symbolication from the Apple skill and then actually execute it,
resulting in 4-5x token bloat and completion regression on the Android
rejection eval scenario.

Remove tool suggestions from the stop signal — just say 'stop, don't
symbolicate.' The Android skill handles its own routing when loaded.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Simplify scenario 3 rubric to focus on positive skill knowledge

Replace 4 rubric items (including 2 'Did NOT' items) with 2 items
that test positive knowledge the skill provides. The 'Did NOT' items
gave credit to baseline responses that also don't attempt iOS workflow
(because they don't know about it), minimizing the quality delta.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix nuget.org domain reference: drop www. prefix to match known-domains.txt

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Merge upstream main and use api.nuget.org v3 endpoint

Main changed known-domains.txt to use path-scoped entries
(nuget.org/account/trustedpublishing instead of bare nuget.org).
Switch the symbols download URL to the v3 flatcontainer endpoint
on api.nuget.org which is in the allowlist.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix .gitignore merge conflict

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Steve Pfister <stpfiste@microsoft.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
 2026-04-10 10:54:46 -04:00
Jeff Schwartz 55a7d80527 adding SIMD/TensorPrimitives to dotent-diag/analyzing-dotnet-performance (#330) 
* incorporated feedback from previous pr, iterated with 100+ scalar loop tests to come to this skill, moved from skill to reference in the performance skill

* Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Address review feedback on SIMD vectorization reference

- Check Span<T>/MemoryExtensions before TensorPrimitives (no extra dependency)
- Add all supported types (sbyte, ushort, uint, ulong, nint, nuint, char via ushort)
- TensorPrimitives: add constraint and applicable types columns (not just float/double)
- Add FusedMultiplyAdd, clarify AddMultiply vs MultiplyAdd distinction
- Prefer portable APIs over platform-specific intrinsics (allow when perf justifies)
- Fix dispatch pattern to use if/else if to avoid pessimizing small inputs
- Remove LINQ references

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* addressing more feedback

* move to the experimental plugin

* move the simd tests to exprimental

* add codeowners

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-25 10:51:28 -06:00
Amaury Levé ab201ee691 Fix eval source file resolution to use eval directory (#374) 
* Fix eval source file resolution to use eval directory

The SetupWorkDir method resolved setup file 'source:' paths relative to the
skill directory (plugins/.../skills/<name>/), but eval.yaml files live under
tests/. The ../../../../ relative paths resolved correctly but were rejected
by the path traversal security check because the resolved path escaped the
skill directory boundary.

Fix by preferring the eval directory as the base for source resolution when
evalPath is available, falling back to skillPath for backward compatibility
with scenarios embedded in SKILL.md files.

Also update run-tests eval.yaml source paths from
'../../../../tests/dotnet-test/run-tests/fixtures/...' to 'fixtures/...'
(relative to the eval directory where the fixtures actually live).

This was causing all 11 source-based run-tests scenarios to get zero fixture
files, making the agent find an empty workspace and fail to evaluate.

* Address review: extract ResolveSourcePath helper, add warning, fix remaining eval.yaml paths

- Extract source-resolution and path-traversal check into internal
  ResolveSourcePath helper for testability
- Emit a clear warning when file.Source is set but no base directory
  is available (instead of silently continuing)
- Add unit tests covering: evalPath-based resolution, skillPath fallback,
  null paths warning, traversal rejection, and evalPath preference
- Fix path traversal patterns in analyzing-dotnet-performance and
  clr-activation-debugging eval.yaml files (same issue as run-tests)
 2026-03-16 16:24:45 +00:00
Elinor Fung ba56c936b3 Add binder tracing to dotnet-trace-collect skill (#250) 
* Add binder tracing to dotnet-trace-collect skill

* Update plugins/dotnet/skills/dotnet-trace-collect/SKILL.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* PR feedback

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Jeff Schwartz <jeffschw@microsoft.com>
 2026-03-11 09:31:33 -07:00
Dan Moseley 8e50fc8558 Add agent and plugin spec conformance validation (#279) 
* Add agent and plugin spec conformance validation

Extend the validation pipeline to check .agent.md files and plugin.json
files alongside skills. Previously only SKILL.md files were validated.

Agent validation (AgentProfiler):
- Frontmatter required (error)
- Name and description required (error)
- Name format: same rules as skills (lowercase alphanum + hyphens, 64 max)
- Name must match filename minus .agent.md suffix
- Description max 1024 chars
- Body max 500 lines

Plugin validation (PluginValidator):
- Name required, must match directory name
- Version and description required
- Description max 1024 chars
- Skills path required and must exist
- Agents path warned if specified but missing

Discovery (SkillDiscovery):
- DiscoverAgents() walks up from skill paths to plugin roots,
  scans agents/ directories for .agent.md files
- DiscoverPlugins() finds plugin.json from skill paths

Integration (ValidateCommand):
- Runs after skill discovery, before evaluation
- Errors block with exit code 1 (same as spec_conformance_failure)

All 4 existing agents and 2 plugins pass cleanly.
25 new tests (13 agent, 12 plugin).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Code review fixes: spec comments, ValidateNameFormat, kind-specific messages

- Refactor ValidateName into ValidateNameFormat (format-only) + ValidateName
  (format + directory match) so agent/plugin validation doesn't abuse the
  skill directory-match check by passing name twice.
- Warning messages now say 'Agent name' or 'Plugin name' instead of
  'Skill name' when called from AgentProfiler/PluginValidator.
- Add spec URL comments to AgentProfiler and PluginValidator referencing
  agentskills.io/specification, VS Code agent-plugins docs, and Claude Code
  plugins-reference.
- Add missing agent name edge-case tests: leading/trailing hyphen,
  consecutive hyphens, and message-prefix assertions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove unused agents path from plugins that have no agents

The dotnet, dotnet-data, and dotnet-upgrade plugins declared an agents
path but had no agents/ directory, producing validation warnings. Remove
the field; it can be added back when agents are actually created.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Promote agent/plugin name validation from warnings to errors

The agentskills.io spec uses 'Must' for all name constraints (length,
character set, no leading/trailing hyphens, no consecutive hyphens).
For a publishing validator these should block, not just warn. Agent
filename mismatch is also promoted since the IDE won't discover the
agent if the name doesn't match.

Skill name validation stays as warnings (lenient loading, not part
of this PR).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback

- Fix FrontmatterRegex: remove RegexOptions.Multiline so a horizontal
  rule (---) in the body is not mistaken for frontmatter. Detection and
  stripping now both anchor to start of string.
- Use filename as fallback identifier in AgentProfile when agent.Name
  is empty, so error output shows [agent:foo.agent.md] not [agent:].
- Add TryGetSafeSubdirectory to PluginValidator: rejects absolute paths
  and parent-directory traversal in skills/agents path fields. Used by
  both PluginValidator and SkillDiscovery.DiscoverAgents.
- Use DirectoryName as fallback for plugin result name when Name is
  empty, so error output shows [plugin:my-dir] not [plugin:].
- Add tests for path traversal rejection and name fallback behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Detect orphaned test directories with no matching plugin/skill

Add FindOrphanedTestDirectories to SkillDiscovery: checks that every
directory under tests/{plugin}/{skill}/ has a corresponding directory
under plugins/{plugin}/skills/{skill}/. Plugin-level orphans (tests/
directory with no matching plugins/ directory) are also flagged.

Integrated into ValidateCommand after plugin/agent validation so it
runs on every validation pass. Errors are blocking (exit 1).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR #279 review feedback

- Use case-sensitive (Ordinal) comparison for agent filename matching
  since agent names are required to be lowercase by spec.
- Replace OrdinalIgnoreCase path-under-root check in TryGetSafeSubdirectory
  with Path.GetRelativePath + '..' prefix check for correct behavior on
  case-sensitive filesystems.
- Let ParsePluginJson throw JsonException on malformed JSON instead of
  silently returning null. ValidateCommand catches it as a blocking error.
- Add test for malformed plugin.json.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Increase eval timeouts for dump-collect and dotnet-trace-collect to 120s

The with-skill runs for these skills were hitting the 60s scenario timeout
in CI, causing the agent to be killed before producing output. Other skills
in the same plugin already use 120s. The 30s decline/negative-test scenario
is left unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Fix fixture paths in clr-activation-debugging and analyzing-dotnet-performance evals

The source paths referenced tests/dotnet/ but the fixtures live under
tests/dotnet-diag/. This caused file-not-found exceptions at runtime,
which set hasRejections and caused unconditional exit 1 (bypassing
--verdict-warn-only).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Surface execution errors in validation summary

Previously, skills that threw exceptions (e.g. missing fixture files)
were silently excluded from the verdict count. The summary showed
'2/4 skills passed' with no indication that 2 other skills crashed
entirely. Now the summary shows '2/6 skills passed (2 rejected due
to execution errors)' and lists the specific error messages near
the summary rather than buried in the middle of the log.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Reduce parallelism for infra-change evaluation runs

When infrastructure changes trigger evaluation of all plugins
simultaneously, the high default parallelism (5x5x5) causes API
rate limits (429s) and scenario timeouts from contention. Reduce
to 2x3x3 for infra-change runs only; regular single-plugin PR
evaluations keep the default 5x5x5.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-08 15:40:06 +01:00
Dan Moseley 042a9d2b2c Reorganize plugins into domain-specific groupings (#274) 
* Reorganize plugins per agreed structure (#271)

Split the dotnet plugin into domain-specific plugins per discussion
in #271:

- dotnet: narrowed to common everyday C#/.NET coding tasks
- dotnet-diag: perf investigations, debugging, incident analysis
- dotnet-data: data access and Entity Framework
- dotnet-aspnet, dotnet-wpf, dotnet-winforms, dotnet-maui: scaffolded
  empty to convey intent for future framework-specific skills

Moved skills, tests, and CODEOWNERS entries accordingly.
No file content was changed; all skill/test moves are pure renames.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Improve dotnet-diag and dotnet-winforms descriptions

- dotnet-diag: add '.NET' qualifier to clarify scope
- dotnet-winforms: use official 'Windows Forms (WinForms)' name

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add .NET qualifier to dotnet-data description

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Remove empty plugins; list them in CONTRIBUTING.md as future options

Remove the four empty placeholder plugins (dotnet-aspnet, dotnet-wpf,
dotnet-winforms, dotnet-maui) that had only plugin.json and no skills.

Add a 'Plugin organization' section to CONTRIBUTING.md that describes the
domain-specific plugin groupings introduced in this PR and lists the four
removed plugin names as reserved candidates for future contributions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Create dotnet-upgrade plugin with migration/upgrade skills

Move migrate-nullable-references, thread-abort-migration, and
dotnet-aot-compat from dotnet to the new dotnet-upgrade plugin.

Update CODEOWNERS paths (same owners), CONTRIBUTING.md plugin table,
and fix the thread-abort-migration SKILL.md link in CONTRIBUTING.md.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-06 21:01:52 -08:00