PR #473 moved the failure investigation prompt out of the PR comment and
only into the workflow run summary. This made it harder for users to
discover — they had to click through to the run details to find it.
Move it back to the PR comment so the copy-paste prompt is visible
inline on the PR, while keeping it in the workflow summary as well.
* dotnet-msbuild: register AITools.BinlogMcp and use it from binlog-failure-analysis
Adds the AITools.BinlogMcp MCP server (pulled via dnx from the dotnet-eng public NuGet feed, prerelease, no version pin) to the dotnet-msbuild plugin and rewrites the binlog-failure-analysis SKILL.md to lead with the MCP tools (binlog_overview, binlog_errors, binlog_search, binlog_items, binlog_properties, binlog_imports, binlog_nuget). The text-log replay workflow is kept as a fallback for environments without the MCP.
* Improve skills
* Reflect PR comments
* Fix vally eval isolation: pin baseline to 0 skills, skilled to 1
vally evaluations were not isolating skills properly. In the failing run
https://github.com/dotnet/skills/actions/runs/26021981688, the dotnet-test
`code-testing-agent` eval reported "Found 77 skill(s)" for baseline and
"Found 22 skill(s)" for the skilled run instead of the expected 0 and 1.
Root cause: `vally eval` derives skills via `discoverSkills(skillRoot)`
where `skillRoot = opts.skillDir ?? cwd`:
- Baseline omitted `--skill-dir`, so vally walked up from the repo root
to `.vally.yaml` and loaded every skill under `plugins/` (77 total).
- The CI workflow's skilled run passed `--skill-dir` pointing at the
plugin-level `plugins/<plugin>/skills` (from
`matrix.entry.skills_path`). That directory has no `SKILL.md` at
its root, so vally's Step-4 child scan picked up every sibling skill in
the plugin (22 for dotnet-test).
Fix applied to both `eng/vally-adapter/run-vally-evals.sh` and
`.github/workflows/vally-evaluation.yml`:
- Baseline now passes `--skill-dir <empty mktemp dir>` so vally
discovers 0 skills.
- Skilled now uses `plugins/<plugin>/skills/<eval-name>` (the specific
skill directory, not the plugin-level one) so vally matches Step-1
(SKILL.md at root) and discovers exactly 1 skill.
- Evals with no matching `SKILL.md` (the `agent.*` orchestrator evals
that declare their own `environment.skills`) are explicitly skipped
with a warning -- baseline-vs-skilled isolation is not meaningful for
them, since they require multi-skill setup by design.
- `Find eval specs` now narrows to a single eval when
`workflow_dispatch` is invoked with a specific skill, so single-skill
dispatches do not accidentally run every eval in the plugin.
- Empty temp dirs are created per-eval and cleaned up after each run to
avoid any cross-run contention under parallel execution.
After the fix the CI log should report `Found 0 skill(s)` for baseline
and `Found 1 skill(s): <skill-name>` for skilled.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review: trap-based cleanup for EMPTY_SKILL_DIR; fail-loud on missing dispatch skill
- run-vally-evals.sh: register a RETURN trap immediately after creating
EMPTY_SKILL_DIR so it is removed on every exit path, including aborts
triggered by set -euo pipefail (e.g. if
ode adapt.mjs exits
non-zero inside the log-capture block). Removes the unconditional
rm at the end of the function which would not run on errors.
- vally-evaluation.yml: when workflow_dispatch supplies a skill input
but no ests/<plugin>/<skill>/eval.vally.yaml exists, emit an
explicit ::error:: that includes the missing candidate path and fail
the job, instead of silently falling through to the generic
`No eval.vally.yaml files found for <plugin>` warning.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
During multi-model skill-validator evals, judge calls intermittently
fail with:
`Failed to persist session events: Error: Failed to append to JSONL`
`file session-state\events.jsonl: Error: timeout while waiting for`
`mutex to become available`
The error wording (the word `mutex`) does not originate in our C# code
and is reported by the GitHub Copilot SDK over RPC, so the underlying
contention lives inside the SDK's session event-persistence layer.
AgentRunner already gives each session its own `configDir` and
LlmSession (judge) gives each call a unique temp root, so paths cannot
collide ACROSS sessions — this points to intra-session contention on a
single events.jsonl path.
Mitigation: serialize concurrent writes per resolved path inside our
handler with a `ConcurrentDictionary<string, SemaphoreSlim>` keyed by
the absolute path. Both `WriteFileAsync` and `AppendFileAsync` route
through a per-path lock helper. Reads remain unlocked.
Even though the conflict surfaces inside the SDK, removing one variable
(our handler issuing concurrent writes to the same file) is good defense
and makes the `ISessionFsHandler` thread-safe in its own right. The
dictionary is bounded by the number of unique files written within one
session (typically <10), so leak risk is negligible for the per-session
object lifetime.
Validation: build passes; full xUnit suite 545/545 tests pass.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* skill-validator: bump per-plugin aggregate description cap from 15K to 20K
The dotnet-test plugin (22 skills) is now at ~14K aggregate description
characters and bumping into the 15,000 cap on every meaningful description
change. This blocks legitimate plugin growth on what is, on inspection, an
informal repo guardrail rather than a documented Copilot constraint.
Origin of the 15K number
Traced via 'git log -S MaxAggregateDescriptionLength' to PR #238,
which references issue #222. The issue verbatim says:
`users may apply a limit to the total size of 'description' across`
`all skills in a plugin (...). 15K characters was mentioned, we`
`could choose smaller.`
So the value is informal — picked because it `was mentioned` and
explicitly leaving room to `choose smaller`.
Spec status
The agentskills.io specification (https://agentskills.io/specification)
documents per-skill description (1024), compatibility (500), and name
(64) limits. It does NOT define any aggregate per-plugin cap. The
`MaxDescriptionLength = 1024` constant in this same file already cites
the spec; the aggregate constant carried no citation comment.
What this PR does
- Bumps `MaxAggregateDescriptionLength` from 15,000 to 20,000. Picks a
round value that gives ~30% headroom over the largest current plugin
(dotnet-test ~14K) without making the cap meaningless.
- Adds a comment documenting the origin (informal, issue #222),
the absence of a spec source, and a TODO to validate this guardrail
against literature on skill-routing degradation and to run experiments
(skill-validator already has the infrastructure to A/B-measure
description noise vs activation accuracy).
What this PR does NOT do
- It does not remove the cap. The cost concern in #222 is real (every
description is loaded at agent startup for routing); we just don't yet
have evidence for the right ceiling.
- It does not change per-skill limits.
Validation
All 545 tests pass; the existing `CheckCommandAggregateDescriptionTests`
reference the constant symbolically and continue to pass with the new
value.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Potential fix for pull request finding
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Pin Nerdbank.MessagePack to 1.1.62 and add spec URL to cap comment
- Transitively pin Nerdbank.MessagePack from 1.0.2 (vulnerable via StreamJsonRpc 2.24.84 -> GitHub.Copilot.SDK) to 1.1.62, which contains the fix for the stackalloc DateTime vulnerability (GHSA-2cwq-pwfr-wcw3). Without this pin, restore fails with NU1903 since TreatWarningsAsErrors is true.
- Add the agentskills.io specification URL to the MaxAggregateDescriptionLength comment so future readers can verify the claim that the per-plugin aggregate cap is a local repo policy rather than a spec-mandated limit (addresses PR #655 review feedback).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
* Update .NET 10→11 migration skill for Preview 2 and 3
Add breaking changes from .NET 11 Preview 2 and Preview 3 to the
migrate-dotnet10-to-dotnet11 skill. The skill previously covered only
Preview 1.
Changes:
- Add 15 new breaking changes across all reference files
- Create new aspnetcore-dotnet10to11.md reference (OpenAPI v3, Blazor)
- Update SKILL.md workflow steps, assessment checklist, and patterns
- Add 4 new eval scenarios covering high-impact P2/P3 changes
- Add github.com/dotnet/aspnetcore to known-domains.txt
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback
- Replace prompt-echoable 'corrupt' in ZipArchive eval regex with 'checksum'
- Update EF Core reference note to say 'Preview 1 through Preview 3'
- Add missing Source link for Zstandard entry
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback: tighten eval regex, fix raw HTML
- Tighten OverscanCount grader regex to require 'OverscanCount' with
a number, instead of matching bare '3' anywhere
- Wrap Blazor component snippet in backticks so it renders as code
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
/-prefix paths bypassed the flag and hit the depth check unconditionally. Now treated symmetrically with .. traversal: allowed only with the flag, otherwise produce a clear absolute (repo-rooted) path error. Also classify // as a protocol-relative URL.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Introduce Measure-SkillCoverage.ps1 under eng/skill-coverage/ that
measures how much of a SKILL.md's teaching content is verified by
eval.yaml test scenarios — analogous to code coverage for skill files.
The script extracts testable 'coverage points' from four structural
sections of a skill file (Validation checklist, Common Pitfalls,
Workflow Steps, and code patterns from fenced blocks), then
cross-references them against eval.yaml assertions and rubric
criteria using regex matching and keyword overlap heuristics.
Features:
- Per-skill and aggregate (-All) analysis across all plugins
- Table output with colored pass/fail and per-category percentages
- JSON output for machine consumption (single object or array)
- Distinguishes deterministic assertions from rubric-only coverage
- MinCoverage threshold for CI gating
- Tracks source line numbers for all coverage point categories
* Add validator options
* Address PR review: validate options and add tests
- Reject --allow-repo-traversal with --plugin mode (plugins must be portable)
- Validate --max-declaration-lines and --max-agent-lines are positive integers
- Add unit tests for MaxDeclarationLines override in SkillProfiler
- Add unit tests for AllowRepoTraversal suppression and depth check preservation
- Add unit tests for MaxAgentLines override and fallback to MaxDeclarationLines
* Remove body line count checks and --max-declaration-lines/--max-agent-lines options
Line count checks are redundant with BPE token-based complexity classification,
which already provides better guidance. Keep --allow-repo-traversal only.
* Address review: remove unused CheckOptions from AgentProfiler, fix traversal test
- Remove unused CheckOptions parameter from AgentProfiler.AnalyzeAgent and
RunAgentsCheckCore since line checks were removed
- Fix AllowRepoTraversalSuppressesParentTraversalError test to use a non-deep
path (../SKILL.md) and assert no file-ref errors at all
* Add SKILL.md for iOS crash symbolication process
Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update plugins/dotnet/skills/ios-crash-symbolication/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add automation script, tests, and review fixes for ios-crash-symbolication
- Add Symbolicate-Crash.ps1 (664 lines): parses .ips JSON, searches
local dSYMs (SDK packs, NuGet cache, user paths), verifies UUIDs via
dwarfdump, batch-symbolicates with atos, identifies runtime version
- Update SKILL.md: add Automation Script and Runtime Version
Identification sections, fix atos -o to point inside dSYM bundle,
add dwarfdump and MAUI to frontmatter, wrap steps in Workflow
heading for consistency with Android sibling, fix misleading
'rebuild' guidance per review feedback (Rolf)
- Add eval.yaml with 7 test scenarios: Mono crash, CoreCLR crash,
no .NET frames, NativeAOT, multi-library UUIDs, ASI field
priority, and Android tombstone rejection
- Add 5 .ips test fixture files (two-part JSON format)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add iOS crash log and evaluation scenarios for symbolication tests
* Remove unused file
* Add CODEOWNERS entry for iOS crash symbolication
* Rename ios-crash-symbolication to apple-crash-symbolication
Address PR feedback to support all Apple platforms (tvOS, Mac Catalyst,
macOS) not just iOS:
- Add $appleRids array covering ios, tvos, maccatalyst, and osx RIDs
- Refactor Find-Dsym and Find-RuntimeVersion to search all platform packs
- Rewrite SKILL.md for orchestration focus and reduced token budget
- Extract domain knowledge to references/ips-crash-format.md
- Tune eval.yaml: outcome-based rubrics, broad assertions, overfit 0.26
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix script robustness: base address parsing and null-safe property access
- Simplify base address cast to [uint64] without manual hex prefix stripping
- Use PSObject.Properties check before accessing lastExceptionBacktrace
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add dSYM-path rubric items to CoreCLR and ASI scenarios
Add rubric items testing for skill-specific domain knowledge (SDK pack
paths, NuGet cache directories) that baseline agents cannot provide.
This creates the quality delta needed to pass the 10% improvement
threshold while keeping overfit at 0.12 (Low).
3-run validation: 30.1% improvement, 5/7 scenarios positive.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refactor code structure for improved readability and maintainability
* Fix crash symbolication script for real .ips format and improve eval scenarios
- Fix Symbolicate-Crash.ps1 strict-mode bugs with real .ips crash logs:
safe property access for image name/path (sentinel entries), thread name
(most threads unnamed), and single-element array unwrapping (.Count)
- Add SKILL.md efficiency guidance: resolve script path from skill directory
(no find /), start with -ParseOnly, don't run broad filesystem searches
- Reframe eval scenarios for platform-independent evaluation (parse/analyze
instead of requiring macOS-only atos/dwarfdump), tighten rubric to test
skill-specific knowledge (NuGet package name, all .NET binary images)
Validation results (3 runs, claude-opus-4.6):
Scenario 1 (parse frames): 3.3 → 4.0 (+0.7) ✅
Scenario 2 (investigate crash): 3.3 → 4.0 (+0.7) ✅
Scenario 3 (reject Android): 3.3 → 4.0 (+0.7) ✅
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Auto-fallback to parse-only output when atos is unavailable
On Linux (CI), atos and xcrun don't exist. Previously the script would
error out after completing all parsing, losing the results. Now it
detects the missing tool and falls back to ParseOnly output automatically,
ensuring the agent always gets structured parse data regardless of platform.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move apple-crash-symbolication to dotnet-diag plugin
Relocate skill from plugins/dotnet to plugins/dotnet-diag and tests
from tests/dotnet to tests/dotnet-diag. Update CODEOWNERS accordingly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve apple-crash-symbolication: macOS symbols, bug fixes, training log
- Fix JSON case-conflict parsing (vmRegionInfo/vmregioninfo duplicate keys)
- Fix strict-mode safe access for optional asi field
- Expand Step 4 with macOS-specific symbol package guidance (.symbols NuGet)
- Add .dwarf to .dSYM conversion instructions
- Add src/coreclr/ to validation paths
- Soften stop signals to allow crash analysis and deeper investigation
- Add macOS Symbol Packages and JSON Parsing Gotchas to reference doc
- Create training log documenting session findings
- Add .github/skills/ project-local copy for CLI testing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* apple-crash-symbolication: automated version extraction and symbol acquisition
Script improvements:
- Preserve full image paths from crash log (previously discarded by GetFileName)
- Add Get-RuntimeVersionFromPath: extracts .NET version from image paths
(e.g., .../shared/Microsoft.NETCore.App/10.0.4/libcoreclr.dylib)
- Add Get-RidFromPath: infers RID from path or crash metadata (OS/CPU)
- Path-based version detection as fast primary method, UUID matching as fallback
- Emit copy-pasteable symbol acquisition commands when dSYMs are missing
- Show .NET version in ParseOnly library listing
SKILL.md updates:
- Step 2: document automated version detection and acquisition commands
- Step 4: script now prints ready-to-run download/conversion commands
Training log: record session 2 findings (5 issues, script + SKILL.md changes)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* apple-crash-symbolication: add symbol server anti-pattern
dotnet-symbol and msdl.microsoft.com do not serve macOS dSYM/DWARF
symbols — only Windows PDBs and Linux ELF debug info. NuGet packages
are the only public source. Added anti-pattern to SKILL.md (both
copies) and reference doc to prevent wasted tool calls.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* apple-crash-symbolication: fix symbol server guidance — dotnet-symbol works for macOS
dotnet-symbol --symbols <binary> successfully downloads .dwarf debug
symbols for macOS Mach-O binaries from msdl.microsoft.com. Previous
commit incorrectly claimed this didn't work. Replaced anti-pattern
with positive guidance in SKILL.md (both copies) and reference doc.
Added training log entry documenting the correction.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Promote dotnet-symbol as preferred macOS symbol acquisition method
- Script: macOS acquisition block now shows Option A (dotnet-symbol,
preferred) and Option B (.symbols NuGet, fallback). Fixed .dwarf
filename doubling bug in cp command.
- SKILL.md (both copies): Step 2 updated for dotnet-symbol preference,
Step 4 reordered with dotnet-symbol as #2 and NuGet symbols as #3.
- Reference doc: Restructured macOS Symbol Packages section with
Preferred/Fallback subsections and shared .dwarf→.dSYM conversion.
- Training log: Session 4 entry documenting the promotion.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add automated symbol server download for macOS crash symbolication
- Add Get-DebugSymbols function: downloads .dwarf files from Microsoft
symbol server using Mach-O UUID (mirrors android tombstone approach)
- Add Convert-DwarfToDsym function: creates .dSYM bundle from .dwarf
with UUID verification via dwarfdump
- New params: -SymbolCacheDir, -SymbolServerUrl, -SkipSymbolDownload
- Wire download+conversion into main flow after local dSYM search
- Refactor manual acquisition guidance as fallback-only
- Update both SKILL.md copies: frontmatter, Step 4, new flags
- Update ips-crash-format.md: automated download as primary method
- Add training log session 5
URL pattern: https://msdl.microsoft.com/download/symbols/_.dwarf/mach-uuid-sym-{UUID}/_.dwarf
Verified end-to-end: 391/391 .NET frames symbolicated with clean cache.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* apple-crash-symbolication: add triage order guidance to Step 3
Step 3 now instructs the agent to explain the faulting mechanism
(frames #0-#1) before examining cross-thread context. This addresses
a misdiagnosis where GC activity on neighboring threads was mistaken
for causation when the actual root cause (_sigtramp -> NULL signal
handler) was visible in the crashing thread's first two frames.
Training log updated with session retrospective and corrected the
original crash description from GC race to NULL signal handler.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove stale .github/skills/apple-crash-symbolication remnant
Skill lives in plugins/dotnet-diag/skills/apple-crash-symbolication/ since
the plugin restructuring. The old .github/skills/ copy was a leftover.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add libimobiledevice.org to known-domains.txt
The reference scanner CI check fails because the apple-crash-symbolication
skill references https://libimobiledevice.org/ which is not in the allowed
domains list. This domain hosts the libimobiledevice project, a community
library for communicating with iOS devices.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Delete eng/reference-scanner/known-domains.txt
* Apply suggestion from @Copilot
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add libimobiledevice.org to known-domains allowlist
The apple-crash-symbolication SKILL.md references libimobiledevice.org
for the idevicecrashreport tool. Add the domain to the known-domains
file to fix the skill-check CI reference validation failure.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restructure apple-crash-symbolication for analysis-first approach
The skill was getting ❌ verdicts on all eval scenarios because it
directed the LLM to 'run the script' instead of teaching crash analysis
domain knowledge. This mirrors the problem where quality decreased from
3.0 to 2.7 in the parse scenario.
Restructure to follow the Android sibling's proven pattern:
- Lead with parsing (.ips two-part JSON format, key fields)
- Teach .NET library identification (inline library table)
- Teach crash interpretation (asi, faulting thread, exception)
- Teach atos command construction (concrete examples)
- Teach dSYM search paths (ordered list with commands)
- Move automation script to optional section at the end
- Move crash log retrieval to a separate section (not Step 1)
Trim the reference doc to avoid duplication, keeping only macOS
symbol distribution differences and supported RID list.
Token budget: ~2,100 tokens (within 800-2,500 optimal range).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Refine prompts and assertions in eval.yaml for iOS crash symbolication scenarios
* Add format verification guard and fix overfitting in reject scenario
- SKILL.md: Add explicit format check at start of Step 1 to verify
.ips JSON before proceeding; add 'Wrong file format' stop signal
- eval.yaml: Remove direct skill reference from scenario 3 prompt
to reduce overfitting (was triggering skill activation in plugin mode)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix scenario 3: provide actual Android tombstone file for rejection test
- eval.yaml: Replace invalid 'extra_files' with correct 'files' syntax
(extra_files was silently ignored, so crash_android.txt was never copied)
- eval.yaml: Remove copy_test_files for scenario 3 to avoid copying
ios_crash.ips which tempts the agent in plugin mode
- Add tombstone_sample.txt to test directory (copy from android sibling)
- SKILL.md: Mention ndk-stack/addr2line in wrong-format stop signal
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update tests/dotnet-diag/apple-crash-symbolication/tombstone_sample.txt
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update eng/known-domains.txt
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Address review feedback: simulator RIDs, perf, safety, accuracy fixes
- Add simulator RIDs (iossimulator, tvossimulator) to script search list
- Build hashtable for O(1) image lookups in Get-ThreadFrames
- Remove -UseBasicParsing (unnecessary in pwsh 7+)
- Include UUID in Convert-DwarfToDsym cache path; sanitize library name
- Make version regex greedy to capture pre-release suffixes
- Improve format detection error message for non-.ips files
- Guard xcrun fallback with Get-Command check
- Add .dwarf-to-.dSYM conversion guidance for macOS manual fallback
- Escape .ips in eval.yaml regex assertion
- Fix UUID note in reference doc (normalize, not assume lowercase)
- Fix symbol server example to use crash-log image name
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address Copilot review round 2: case-sensitive replace, dedup, path fix, redact PII
- Use -creplace with guard for vmregioninfo duplicate key handling
- Use Sort-Object -Unique for proper library deduplication
- Fix macOS fallback to create .dSYM bundles under symbols-out/
- Redact device identifiers in test fixture .ips file
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Prevent dSYM cache poisoning: validate UUID on cache hit, remove on mismatch
- Convert-DwarfToDsym now verifies cached dSYM UUID before reusing
- On UUID mismatch during download, remove the bad cached bundle so
subsequent runs can retry cleanly
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix cache validation: normalize UUID comparison, clean up dwarf + bundle on mismatch
- Use Format-Uuid to normalize dwarfdump output before comparing to
already-normalized \ in Convert-DwarfToDsym cache check
- On UUID mismatch after download, remove the entire .dSYM bundle
directory and the cached .dwarf file to prevent repeat failures
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Resolve-Frames null handling: use List[object] to preserve null entries
PowerShell array += \ silently drops the element, breaking
index alignment between results and input addresses. Switch to
List[object].Add() which correctly preserves null entries.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix dSYM bundle root traversal, qualify version-from-path note
- Walk up directory tree until *.dSYM is found instead of going up
only 2 levels (which lands at Contents/Resources, not the bundle)
- SKILL.md: note that version-in-path only works on macOS shared
framework installs; iOS paths don't embed the runtime version
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove incorrect .github/skills path reference from training log
The .github/skills/ directory doesn't exist in this repo. The file
was already listed under its actual plugins/dotnet-diag/ path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Detect simulator for RID inference in manual fallback guidance
Detect CoreSimulator in image paths to use iossimulator-/tvossimulator-
RIDs for simulator crashes, avoiding UUID mismatches from wrong runtime
packs. Also handle arm64e CPU type.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix Resolve-Frames return type and scope issue in simulator detection
- Return \.ToArray() instead of ,\ to avoid wrapping
the list in a single-element array
- Use \.usedImages instead of undefined \ in RID
inference block
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Clarify ParseOnly .NET Libraries section shows only frame-relevant images
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix YAML escape in assertion pattern (use single quotes for regex backslash)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Tighten stop signal to prevent Android symbolication spillover
The apple-crash-symbolication skill's stop signal for wrong file formats
was naming specific Android tools (ndk-stack, addr2line) and the
android-tombstone-symbolication skill. This caused models to learn about
Android symbolication from the Apple skill and then actually execute it,
resulting in 4-5x token bloat and completion regression on the Android
rejection eval scenario.
Remove tool suggestions from the stop signal — just say 'stop, don't
symbolicate.' The Android skill handles its own routing when loaded.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Simplify scenario 3 rubric to focus on positive skill knowledge
Replace 4 rubric items (including 2 'Did NOT' items) with 2 items
that test positive knowledge the skill provides. The 'Did NOT' items
gave credit to baseline responses that also don't attempt iOS workflow
(because they don't know about it), minimizing the quality delta.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix nuget.org domain reference: drop www. prefix to match known-domains.txt
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Merge upstream main and use api.nuget.org v3 endpoint
Main changed known-domains.txt to use path-scoped entries
(nuget.org/account/trustedpublishing instead of bare nuget.org).
Switch the symbols download URL to the v3 flatcontainer endpoint
on api.nuget.org which is in the allowlist.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix .gitignore merge conflict
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Steve Pfister <stpfiste@microsoft.com>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>
Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
* docs: document token count reporting in skill-validator
Add a 'Getting the token count for a skill' section to the README
explaining how to use 'check --verbose' to see BPE token counts and
complexity tiers.
Expand the Run metrics table in InvestigatingResults.md with the
inputTokens, outputTokens, cache, and judge token fields available
in results.json. Update the example analysis script to print
input/output tokens.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* docs: address review feedback on token count documentation
- Fix tier table: replace 'Optimal sweet spot' with 'Recommended tier (sweet spot: 800–2,500)' to align with earlier statement
- Fix tokenEstimate description to accurately reflect implementation behavior (cache fallback when inputTokens is zero)
Agent-Logs-Url: https://github.com/dotnet/skills/sessions/dd5ded07-a968-4574-bfe9-1602b4434f07
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
---------
Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Add exp-test-smell-detection skill and evaluation
- Language-agnostic core skill detecting 10 test smells (conditional logic,
mystery guest, sleepy test, assertion-free, eager test, magic numbers,
sensitive equality, exception handling, general fixture, ignored tests)
- .NET extension with framework-specific detection tables (MSTest/xUnit/NUnit/TUnit)
- Reference catalog with 19 smells from testsmells.org research
- Evaluation with 4 scenarios: smelly suite, clean suite, integration tests, non-activation
- CODEOWNERS entry for @dotnet/dotnet-testing
* Improve integration test fixture and non-activation timeout
- Add real smells to integration test fixture (Thread.Sleep, conditional
logic, assertion-free test) mixed with legitimate integration patterns
- Update rubric to test nuanced distinction between real smells and
acceptable integration test patterns
- Increase non-activation scenario timeout from 120s to 300s to prevent
timeout-related scoring penalties
* Fix test projects
* Add domains for testsmells
* Remove extra domains
* Skill-validator: perf, cancellation, and reliability improvements
Performance:
- Cache skill-validator-dist.tar.gz in CI (evaluation.yml and skill-check.yml)
to skip .NET SDK setup, build, and tar on cache hit. Evaluate and comment-on-pr
jobs restore from cache with artifact download fallback.
- Replace SemaphoreSlim with Lock in SessionDatabase for lighter-weight
synchronous locking.
Cancellation:
- Thread CancellationToken from System.CommandLine entry point through the
entire evaluate call chain: Run -> EvaluateTarget -> EvaluateSkill/EvaluateAgent
-> ExecuteScenario/ExecuteAgentScenario -> ExecuteRun/ExecuteAgentRun, plus
EvaluateSkillNoise -> ExecuteNoiseTest.
- Add CancellationToken parameter to AgentRunner.RunAgent, Judge.JudgeRun,
PairwiseJudge.Judge, and OverfittingJudge.Analyze, forwarding to
RetryHelper and LlmSession.SendAsync respectively.
- Pass CancellationToken to all ConcurrencyLimiter.RunAsync call sites so
semaphore waits are cancellable.
- Re-throw OperationCanceledException at the top-level skill evaluation loop
so Ctrl+C / CI cancellation propagates instead of being swallowed as a
skill failure.
Reliability:
- Smart timeline truncation in Judge and PairwiseJudge: error events
(session.error, runner.error) are always preserved during truncation
regardless of position, with a summary marker showing event type breakdown.
- Add jitter (±25%) to RetryHelper exponential backoff to prevent thundering
herd on rate-limit recovery.
- Add CancellationToken parameter to ConcurrencyLimiter.RunAsync<T>.
- Add ConcurrencyLimiter tests (6 tests) and RetryHelper jitter tests (2 tests).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR review feedback
- RetryHelper: re-clamp delay after jitter so it never exceeds the
remaining budget or MaxSingleDelayMs.
- Judge/PairwiseJudge: handle edge case where error count alone exceeds
MaxTimelineEvents by trimming errors with a summary instead of
unconditionally appending all of them.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add high-variance detection and refactor FailureKind to enum
- Add Statistics.CoefficientOfVariation() for measuring score consistency
- Flag scenarios with CV > 0.5 as high-variance in ScenarioComparison
- Show console warning and markdown footnote for high-variance scenarios
- Include high-variance scenario names in Comparator verdict reason
- Refactor FailureKind from magic strings to strongly-typed enum with
JsonStringEnumMemberName attributes for backwards-compatible JSON output
- Add 6 new CV unit tests
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move skill-validator docs under src/ and include in packages
Move README.md to src/ and InvestigatingResults.md + OverfittingDetection.md
to src/docs/ so they ship in the NuGet package and .tar.gz archives.
- Include docs/*.md in NuGet pack and publish output via CopyToPublishDirectory
- Add LICENSE to publish output (tar.gz only, not NuGet — uses PackageLicenseExpression)
- Replace enumerated docOnlyFiles lists with a regex pattern in CI workflows
- Deduplicate eval.yaml docs between CONTRIBUTING.md and the shipping README
- Add test fixture files walkthrough to README (moved from CONTRIBUTING.md)
- Add AGENTS.md guidance: src/ is the packaging boundary
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update link format in failure diagnosis message
* Update AGENTS.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add C# MCP server skills (create, debug, test, publish)
Four new skills for the C# MCP server development lifecycle:
- mcp-csharp-create: Scaffolding with dotnet new mcpserver, tools/prompts/resources, transport config
- mcp-csharp-debug: MCP Inspector, VS Code integration, breakpoint debugging, logging
- mcp-csharp-test: Unit tests with ClientServerTestBase, integration with WebApplicationFactory, evals
- mcp-csharp-publish: NuGet packaging, Docker/Azure deployment, MCP Registry publishing
Each skill includes SKILL.md with progressive disclosure references/ and eval.yaml tests.
* Fix mcp-csharp-create eval: use regex assertions for combined attribute syntax
Replace scaffolding-heavy scenarios with implementation-focused ones that
test MCP-specific features (resources, prompts, logging). Fix assertion
patterns to match combined C# attribute syntax [McpServerTool, Description()]
instead of requiring standalone [McpServerTool]. Increase timeouts to 180s
to account for skill-reading overhead.
Validator result: passed=True, improvement=44.6% (threshold=10%)
* Add CODEOWNERS entries for MCP C# skills (create, debug, publish, test)
* Update plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update plugins/dotnet/skills/mcp-csharp-publish/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Remove evaluation step and guide from mcp-csharp-test skill
* Add CODEOWNERS entries for dotnet-maui skills
* Update CODEOWNERS to add @artl93 as co-owner for mcp-csharp skills
* Replace parent-directory file references with backtick skill names
Cross-skill references used ../sibling/SKILL.md paths which the
validator rejects. Replace with backtick-quoted skill names to match
the convention used by other skills in the repo.
* Add known domains for MCP server development skills
Add domains referenced by the mcp-csharp-* skills:
- github.com/modelcontextprotocol/registry
- github.com/open-telemetry/semantic-conventions
- npmjs.com, xunit.net, fluentassertions.com, nugettest.org
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update Docker commands to use placeholder for registry and image names
* moved mcp skills and their evals into dotnet-ai plugin
* cleaned up moved files to dotnet-ai
* cleaned up files
* Update .github/CODEOWNERS
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Update nuget-packaging.md
* Update repository URL format in nuget-packaging.md
* Add github.com/ to known-domains to allow truncated placeholder URLs
The URL regex truncates at angle brackets (invalid in URIs), so
placeholder URLs like https://github.com/<yourusername>/<repo>
become https://github.com/ which didn't match any path-scoped entry.
Adding github.com/ (with trailing slash) covers only this case without
broadly allowing all github.com URLs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update Docker tag command to use placeholder for server name
* Update known domains and fix GitHub username placeholders in documentation
* Remove McpServerToolType from mcp-csharp-create eval
McpServerToolType is only needed with WithToolsFromAssembly(), which is
discouraged because it is not Native AOT compatible. Updated eval assertions
and rubrics to prefer WithTools<T>() instead.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve eval resilience for mcp-csharp-create and mcp-csharp-debug
- Increase mcp-csharp-create timeouts from 180s (or default 120s) to 360s
for all 3 scenarios. These scenarios consistently time out because the
model spends time on bash exploration/scaffolding before writing code.
- Add explicit timeout to scenario 3 which was relying on the 120s default.
- Broaden mcp-csharp-debug Inspector rubric to accept both the concise
single CLI command style and the step-by-step UI configuration walkthrough,
since both are correct approaches.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update eng/known-domains.txt
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Broaden debug 'failing tool' rubric to accept multiple debugging approaches
The rubric criterion 'Shows how to attach a debugger' was too narrow.
The skilled answer correctly focused on the #1 cause (stdout pollution)
but scored low because it didn't show a specific 'attach to process' flow.
Broadened to accept any valid debugging approach: attaching, Debugger.Launch(),
or launch.json configuration.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix 4 eval failures in mcp-csharp-debug and mcp-csharp-test skills
mcp-csharp-debug:
- Trim SKILL.md verbosity (183->160 lines) and mcp-inspector.md (67->54 lines)
- Add Diagnosing Tool Errors section covering debugger, output panel, Inspector, common culprits
- Rebalance debugging narrative away from stderr-only focus
- Move HTTP logging config to ide-config.md reference
mcp-csharp-test:
- Add eval/evaluations keywords to frontmatter for activation
- Add HTTP tool invocation test pattern (tools/call via WebApplicationFactory)
- Trim test-patterns.md bloat (remove Test Categories, Coverage, Input Validation)
- Create references/evaluations.md with qa_pair format and read-only/deterministic guidance
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Dan Moseley <danmose@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add evaluation troubleshooting guide and link from PR comments
Add InvestigatingResults.md with:
- How to find and download evaluation artifacts from workflow runs
- Full results.json schema documentation
- Common failure patterns (timeouts, empty output, variance, rubric
issues, skill not activated) with diagnosis steps and fixes
- Example analysis script for AI agents to parse results
Link the doc from:
- Reporter.cs: show a link in the PR comment when any skill fails
- skill-validator README.md: Output section
- CONTRIBUTING.md: CI evaluation section
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update InvestigatingResults.md
* Address review feedback: fix hard-coded URLs and optional plugin run docs
- Reporter.cs: Use repo-relative link instead of hard-coded GitHub URL
- evaluation.yml: Use github.sha instead of hard-coded 'main' branch
- InvestigatingResults.md: Clarify skilledPlugin run is optional
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* THROWAWAY: add trailing newline to trigger eval workflow
This change is intentionally trivial and will be reverted before merge.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #441 review feedback (round 2)
- Reporter.cs: Remove leading / from relative link
- evaluation.yml: Use jq on results.json instead of grepping for emoji
- evaluation.yml: Use needs.gate.outputs.head_sha for revision-pinned URL
- InvestigatingResults.md: Mark pluginImprovementScore/pluginBreakdown as optional
- InvestigatingResults.md: Fix skill-not-activated symptoms to use
skillActivation fields instead of turnCount
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use specific skillActivation field names and fix timestamp description
- Use skillActivationIsolated/skillActivationPlugin instead of generic skillActivation
- Clarify timestamp is when results were written, not run start
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix timestamp field description (was lost in prior commit)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert "THROWAWAY: add trailing newline to trigger eval workflow"
This reverts commit 83bd7f7601.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Move token usage data into separate branch
* Address PR review: add help doc, fix title injection, add token-usage fallback
* Add -SkipBenchmarkData switch, clean stale files on branch push
* Show subagents invocations
* Address PR feedback: fix XML doc and sanitize agent names in markdown
- Update ExtractSubagentActivation XML doc to accurately describe that
all subagent event types are counted (not just Started/Completed)
- Add markdown table cell sanitization for agent names in FormatSubagentCell
to prevent injection via pipe chars or newlines in plugin-controlled content
- Extract SanitizeMarkdownTableCell helper for reuse
Reorganize the skill-validator project from layer-based grouping
(Commands/, Services/, Models/, Utilities/) to feature-based slices
(Check/, Evaluate/, Consolidate/, Shared/).
Key changes:
- Check/: CheckCommand, SkillProfiler, AgentProfiler, PluginValidator,
ExternalDependencyChecker, ReferenceScanner
- Evaluate/: EvaluateCommand, RejudgeCommand (now a subcommand of
evaluate), AgentRunner, Judge, PairwiseJudge, and other eval services
- Consolidate/: ConsolidateCommand
- Shared/: Models, SkillDiscovery, PluginParser (extracted from
PluginValidator), Reporter, and utilities
Split PluginValidator: parsing helpers (ParsePluginJson,
TryGetSafeSubdirectory) moved to Shared/PluginParser.cs; validation
logic stays in Check/PluginValidator.cs.
Move RawFrontmatter and RawAgentFrontmatter types from EvalSchema into
SkillDiscovery since they are only consumed by discovery.
Wire RejudgeCommand as subcommand: skill-validator evaluate rejudge.
Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Integrate reference scanner into skill-validator CheckCommand
- Port reference scanner logic from scan.ps1 to C# ReferenceScanner service
- Add --known-domains option to CheckCommand for reference scanning
- Move known-domains.txt and allowed-external-deps.txt to eng/
- Remove standalone scan.ps1 and reference-validation.yml workflow
- Reference scanning now runs as part of skill-check.yml via skill-validator
- Add 34 unit tests for ReferenceScanner (all 463 tests pass)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR feedback: repoRoot derivation, early validation, .agents scanning, SRI policy
- Derive repoRoot from --known-domains file path instead of cwd
- Fail early with clear error when known-domains file is missing
- Add .agents/**/*.md scanning to DiscoverFiles (regression fix)
- Fix SRI policy loophole: SRI suppresses SCRIPT-NO-SRI only, domain
and HTTPS checks still apply
- Use actual known-domains file path in error messages instead of
hardcoded eng/known-domains.txt
- Update doc comments to reflect full set of scanned paths
- Add tests for .agents discovery and SRI domain enforcement
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove repo-specific paths from ReferenceScanner
DiscoverFiles now only scans directories it is given -- no hardcoded
knowledge of .agents, agentic-workflows, eng, or README.md. The tool
is repo-agnostic; callers control what gets scanned via --plugin dirs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Drop repoRoot from ReferenceScanner API
ScanFile/ScanFiles no longer take a repoRoot parameter. Finding paths
use the file path as provided by the caller, keeping the scanner
completely decoupled from any repo layout assumptions.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Prune stale domains and add json.schemastore.org to known-domains.txt
Removed 10 entries no longer referenced by any plugin content:
agentskills.io, visualstudio.com, mcr.microsoft.com, dotnet.github.io,
github.com/dotnet/docs, github.com/dotnet/dotnet-api-docs,
github.com/dotnet/skills, github.com/github/gh-aw,
github.com/modelcontextprotocol/csharp-sdk, developers.openai.com.
Added json.schemastore.org (referenced by dotnet-template-engine).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix case-insensitive regex, update docs, add tests
- Make UrlRegex and PipeToShellRegex case-insensitive to match the
original PowerShell behavior (HTTP://, CURL, Wget all now detected)
- Update DiscoverFiles doc comment to remove stale 'individual files' claim
- Narrow CONTRIBUTING.md scope description to match actual plugin-only scanning
- Add tests for mixed/upper-case URL and pipe-to-shell detection
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Remove IPv6 loopback support, fix pipe-to-shell allowlist case sensitivity
- Remove Ipv6LoopbackRegex and IPv6 test data (not needed)
- Add case-insensitive + culture-invariant matching for pipe-to-shell allowlist
- Add mixed-case allowlist test
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dependabot[bot]