mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
v1.0.0
175 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 github-actions[bot]
|
3aa1128ca9 |
chore: upgrade gh-aw runtime (#557)
* chore: upgrade gh-aw runtime * chore: recompile gh-aw workflows --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Jan Krivanek <jankrivanek@microsoft.com> |
||
|
Jan Krivanek
|
1dc3d36f9b |
Fix gh-aw upgrade workflow to work with GITHUB_TOKEN (#554)
GITHUB_TOKEN cannot push commits that modify files under .github/workflows/. This is a hard GitHub platform restriction (requires workflows scope, which is only available via PAT or GitHub App tokens). The workflow now: - Runs gh aw upgrade (without --create-pull-request) - Categorizes changes into pushable (outside .github/workflows/) and non-pushable (inside .github/workflows/) - Creates a PR with only the pushable changes (e.g. actions-lock.json) - Documents any workflow files that need manual recompilation in the PR body with instructions for running gh aw compile locally Tested successfully on https://github.com/JanKrivanek/skills2/pull/1 |
||
|
Jan Krivanek
|
ba3bcf122b | Fix: add workflows permission, remove temp file creation (#545) | ||
|
Amaury Levé
|
9792d18080 |
Add markdownlint analysis for local dev and CI (#534)
* Add markdownlint analysis for local dev and CI - Add .markdownlint.jsonc with rule configuration - Add .markdownlint-cli2.jsonc with ignore patterns - Add package.json with markdownlint-cli2 and lint scripts - Add GitHub Actions workflow running on PRs touching *.md files - Add VS Code extension recommendation for vscode-markdownlint * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review comments: use markdownlint-cli2-action, merge configs, lint only changed files - Replace manual Node.js + npm setup with DavidAnson/markdownlint-cli2-action (pinned to SHA) - Use tj-actions/changed-files to lint only PR-changed markdown files - Merge .markdownlint.jsonc into .markdownlint-cli2.jsonc (single config file) - Remove package.json and package-lock.json (no longer needed) --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
05e2868b00 | Reintroduce copilot sessions storing (#530) | ||
|
Jan Krivanek
|
e294085cf0 | Automate GH AW updates (#536) | ||
|
Amaury Levé
|
54cdb7ebca |
Add skill-coverage PR workflow with slash command support (#538)
* Add skill-coverage PR workflow with slash command support - Auto-triggers on PRs that change plugins/** or tests/** - Detects specific plugin/skill pairs from the PR diff - Runs Measure-SkillCoverage.ps1 and posts a coverage report as a PR comment - Supports /skill-coverage slash command on PR comments: /skill-coverage (runs on changed skills in the PR) /skill-coverage <plugin> (runs on all skills in the plugin) /skill-coverage <plugin> <skill> (runs on a specific skill) * Address Copilot review: security, permissions, and robustness fixes - Add permission check: require write access before running slash command - Tighten /skill-coverage matching to avoid false triggers on similar commands - Add issues: write permission (PR comments use Issues API) - Use refs/pull/<num>/head for checkout to support fork PRs - Remove unused \ variable - Fix multi-line JSON parsing with Out-String * Address PR #538 review: fork PR handling, reactions, and sourcing note - Skip PR comment posting for fork PRs (read-only token); coverage still runs in step summary - Add continue-on-error to reaction step to avoid 403 failures - Add header comment noting issue_comment trigger sources from default branch - Add continue-on-error to no-changes comment step * Split into analyze + comment jobs; fix pagination and dedup Security: - Split into two jobs: analyze (contents: read only) and comment (write only) - Analyze job checks out base branch for trusted tooling, PR head into a separate git worktree - untrusted PR code is never executed - Comment job only reads an uploaded artifact, no PR code checkout Fixes: - Use github.paginate() for listComments to handle PRs with >100 comments - Deduplicate targets: plugin-wide changes replace per-skill targets for that plugin instead of producing duplicates |
||
|
Jan Krivanek
|
74cf19f605 | Update gh-aw (#532) | ||
|
Jan Krivanek
|
9f2c53f68e |
Fix the DevOps health dashboard grooming (#529)
* Improve the groomer * Cleanup the daily health issue |
||
|
dependabot[bot]
|
7079f6c3ae |
Bump the github-actions-dependencies group with 3 updates (#526)
Bumps the github-actions-dependencies group with 3 updates: [actions/github-script](https://github.com/actions/github-script), [actions/cache](https://github.com/actions/cache) and [actions/upload-artifact](https://github.com/actions/upload-artifact). Updates `actions/github-script` from 8.0.0 to 9.0.0 - [Release notes](https://github.com/actions/github-script/releases) - [Commits](https://github.com/actions/github-script/compare/ed597411d8f924073f98dfc5c65a23a2325f34cd...3a2844b7e9c422d3c10d287c895573f7108da1b3) Updates `actions/cache` from 5.0.4 to 5.0.5 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/668228422ae6a00e4ad889ee87cd7109ec5666a7...27d5ce7f107fe9357f9df03efb73ab90386fccae) Updates `actions/upload-artifact` from 7.0.0 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/bbbca2ddaa5d8feaa63e36b76fdaad77386f024f...043fb46d1a93c77aae656e7c1c64a875d1fc6a0a) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 9.0.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/cache dependency-version: 5.0.5 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions-dependencies - dependency-name: actions/upload-artifact dependency-version: 7.0.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Jan Krivanek
|
70e2d3b333 |
Simplify the focus of devops health workflows (#521)
* Simplify the focus of devops health workflows * Updated gh-aw and recompiled * Fix scope wording to include resource usage per review feedback |
||
|
Devin Breshears
|
08ff14b009 |
Add Reviewer Teams (#323)
* Add Reviewer Teams * Add the org * Keep original msbuild owners alongside reviewer team Agent-Logs-Url: https://github.com/dotnet/skills/sessions/0f052a87-406b-4ec3-aa67-57c2d11c64b4 Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> * Update .github/CODEOWNERS Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: danmoseley <6385855+danmoseley@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Milos Kotlar
|
038dd4f087 |
Add Apple crash symbolication skill (#201)
* Add SKILL.md for iOS crash symbolication process Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/ios-crash-symbolication/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add automation script, tests, and review fixes for ios-crash-symbolication - Add Symbolicate-Crash.ps1 (664 lines): parses .ips JSON, searches local dSYMs (SDK packs, NuGet cache, user paths), verifies UUIDs via dwarfdump, batch-symbolicates with atos, identifies runtime version - Update SKILL.md: add Automation Script and Runtime Version Identification sections, fix atos -o to point inside dSYM bundle, add dwarfdump and MAUI to frontmatter, wrap steps in Workflow heading for consistency with Android sibling, fix misleading 'rebuild' guidance per review feedback (Rolf) - Add eval.yaml with 7 test scenarios: Mono crash, CoreCLR crash, no .NET frames, NativeAOT, multi-library UUIDs, ASI field priority, and Android tombstone rejection - Add 5 .ips test fixture files (two-part JSON format) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add iOS crash log and evaluation scenarios for symbolication tests * Remove unused file * Add CODEOWNERS entry for iOS crash symbolication * Rename ios-crash-symbolication to apple-crash-symbolication Address PR feedback to support all Apple platforms (tvOS, Mac Catalyst, macOS) not just iOS: - Add $appleRids array covering ios, tvos, maccatalyst, and osx RIDs - Refactor Find-Dsym and Find-RuntimeVersion to search all platform packs - Rewrite SKILL.md for orchestration focus and reduced token budget - Extract domain knowledge to references/ips-crash-format.md - Tune eval.yaml: outcome-based rubrics, broad assertions, overfit 0.26 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix script robustness: base address parsing and null-safe property access - Simplify base address cast to [uint64] without manual hex prefix stripping - Use PSObject.Properties check before accessing lastExceptionBacktrace Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dSYM-path rubric items to CoreCLR and ASI scenarios Add rubric items testing for skill-specific domain knowledge (SDK pack paths, NuGet cache directories) that baseline agents cannot provide. This creates the quality delta needed to pass the 10% improvement threshold while keeping overfit at 0.12 (Low). 3-run validation: 30.1% improvement, 5/7 scenarios positive. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refactor code structure for improved readability and maintainability * Fix crash symbolication script for real .ips format and improve eval scenarios - Fix Symbolicate-Crash.ps1 strict-mode bugs with real .ips crash logs: safe property access for image name/path (sentinel entries), thread name (most threads unnamed), and single-element array unwrapping (.Count) - Add SKILL.md efficiency guidance: resolve script path from skill directory (no find /), start with -ParseOnly, don't run broad filesystem searches - Reframe eval scenarios for platform-independent evaluation (parse/analyze instead of requiring macOS-only atos/dwarfdump), tighten rubric to test skill-specific knowledge (NuGet package name, all .NET binary images) Validation results (3 runs, claude-opus-4.6): Scenario 1 (parse frames): 3.3 → 4.0 (+0.7) ✅ Scenario 2 (investigate crash): 3.3 → 4.0 (+0.7) ✅ Scenario 3 (reject Android): 3.3 → 4.0 (+0.7) ✅ Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Auto-fallback to parse-only output when atos is unavailable On Linux (CI), atos and xcrun don't exist. Previously the script would error out after completing all parsing, losing the results. Now it detects the missing tool and falls back to ParseOnly output automatically, ensuring the agent always gets structured parse data regardless of platform. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move apple-crash-symbolication to dotnet-diag plugin Relocate skill from plugins/dotnet to plugins/dotnet-diag and tests from tests/dotnet to tests/dotnet-diag. Update CODEOWNERS accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve apple-crash-symbolication: macOS symbols, bug fixes, training log - Fix JSON case-conflict parsing (vmRegionInfo/vmregioninfo duplicate keys) - Fix strict-mode safe access for optional asi field - Expand Step 4 with macOS-specific symbol package guidance (.symbols NuGet) - Add .dwarf to .dSYM conversion instructions - Add src/coreclr/ to validation paths - Soften stop signals to allow crash analysis and deeper investigation - Add macOS Symbol Packages and JSON Parsing Gotchas to reference doc - Create training log documenting session findings - Add .github/skills/ project-local copy for CLI testing Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: automated version extraction and symbol acquisition Script improvements: - Preserve full image paths from crash log (previously discarded by GetFileName) - Add Get-RuntimeVersionFromPath: extracts .NET version from image paths (e.g., .../shared/Microsoft.NETCore.App/10.0.4/libcoreclr.dylib) - Add Get-RidFromPath: infers RID from path or crash metadata (OS/CPU) - Path-based version detection as fast primary method, UUID matching as fallback - Emit copy-pasteable symbol acquisition commands when dSYMs are missing - Show .NET version in ParseOnly library listing SKILL.md updates: - Step 2: document automated version detection and acquisition commands - Step 4: script now prints ready-to-run download/conversion commands Training log: record session 2 findings (5 issues, script + SKILL.md changes) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: add symbol server anti-pattern dotnet-symbol and msdl.microsoft.com do not serve macOS dSYM/DWARF symbols — only Windows PDBs and Linux ELF debug info. NuGet packages are the only public source. Added anti-pattern to SKILL.md (both copies) and reference doc to prevent wasted tool calls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: fix symbol server guidance — dotnet-symbol works for macOS dotnet-symbol --symbols <binary> successfully downloads .dwarf debug symbols for macOS Mach-O binaries from msdl.microsoft.com. Previous commit incorrectly claimed this didn't work. Replaced anti-pattern with positive guidance in SKILL.md (both copies) and reference doc. Added training log entry documenting the correction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Promote dotnet-symbol as preferred macOS symbol acquisition method - Script: macOS acquisition block now shows Option A (dotnet-symbol, preferred) and Option B (.symbols NuGet, fallback). Fixed .dwarf filename doubling bug in cp command. - SKILL.md (both copies): Step 2 updated for dotnet-symbol preference, Step 4 reordered with dotnet-symbol as #2 and NuGet symbols as #3. - Reference doc: Restructured macOS Symbol Packages section with Preferred/Fallback subsections and shared .dwarf→.dSYM conversion. - Training log: Session 4 entry documenting the promotion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add automated symbol server download for macOS crash symbolication - Add Get-DebugSymbols function: downloads .dwarf files from Microsoft symbol server using Mach-O UUID (mirrors android tombstone approach) - Add Convert-DwarfToDsym function: creates .dSYM bundle from .dwarf with UUID verification via dwarfdump - New params: -SymbolCacheDir, -SymbolServerUrl, -SkipSymbolDownload - Wire download+conversion into main flow after local dSYM search - Refactor manual acquisition guidance as fallback-only - Update both SKILL.md copies: frontmatter, Step 4, new flags - Update ips-crash-format.md: automated download as primary method - Add training log session 5 URL pattern: https://msdl.microsoft.com/download/symbols/_.dwarf/mach-uuid-sym-{UUID}/_.dwarf Verified end-to-end: 391/391 .NET frames symbolicated with clean cache. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * apple-crash-symbolication: add triage order guidance to Step 3 Step 3 now instructs the agent to explain the faulting mechanism (frames #0-#1) before examining cross-thread context. This addresses a misdiagnosis where GC activity on neighboring threads was mistaken for causation when the actual root cause (_sigtramp -> NULL signal handler) was visible in the crashing thread's first two frames. Training log updated with session retrospective and corrected the original crash description from GC race to NULL signal handler. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove stale .github/skills/apple-crash-symbolication remnant Skill lives in plugins/dotnet-diag/skills/apple-crash-symbolication/ since the plugin restructuring. The old .github/skills/ copy was a leftover. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add libimobiledevice.org to known-domains.txt The reference scanner CI check fails because the apple-crash-symbolication skill references https://libimobiledevice.org/ which is not in the allowed domains list. This domain hosts the libimobiledevice project, a community library for communicating with iOS devices. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Delete eng/reference-scanner/known-domains.txt * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add libimobiledevice.org to known-domains allowlist The apple-crash-symbolication SKILL.md references libimobiledevice.org for the idevicecrashreport tool. Add the domain to the known-domains file to fix the skill-check CI reference validation failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restructure apple-crash-symbolication for analysis-first approach The skill was getting ❌ verdicts on all eval scenarios because it directed the LLM to 'run the script' instead of teaching crash analysis domain knowledge. This mirrors the problem where quality decreased from 3.0 to 2.7 in the parse scenario. Restructure to follow the Android sibling's proven pattern: - Lead with parsing (.ips two-part JSON format, key fields) - Teach .NET library identification (inline library table) - Teach crash interpretation (asi, faulting thread, exception) - Teach atos command construction (concrete examples) - Teach dSYM search paths (ordered list with commands) - Move automation script to optional section at the end - Move crash log retrieval to a separate section (not Step 1) Trim the reference doc to avoid duplication, keeping only macOS symbol distribution differences and supported RID list. Token budget: ~2,100 tokens (within 800-2,500 optimal range). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refine prompts and assertions in eval.yaml for iOS crash symbolication scenarios * Add format verification guard and fix overfitting in reject scenario - SKILL.md: Add explicit format check at start of Step 1 to verify .ips JSON before proceeding; add 'Wrong file format' stop signal - eval.yaml: Remove direct skill reference from scenario 3 prompt to reduce overfitting (was triggering skill activation in plugin mode) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix scenario 3: provide actual Android tombstone file for rejection test - eval.yaml: Replace invalid 'extra_files' with correct 'files' syntax (extra_files was silently ignored, so crash_android.txt was never copied) - eval.yaml: Remove copy_test_files for scenario 3 to avoid copying ios_crash.ips which tempts the agent in plugin mode - Add tombstone_sample.txt to test directory (copy from android sibling) - SKILL.md: Mention ndk-stack/addr2line in wrong-format stop signal Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update tests/dotnet-diag/apple-crash-symbolication/tombstone_sample.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedback: simulator RIDs, perf, safety, accuracy fixes - Add simulator RIDs (iossimulator, tvossimulator) to script search list - Build hashtable for O(1) image lookups in Get-ThreadFrames - Remove -UseBasicParsing (unnecessary in pwsh 7+) - Include UUID in Convert-DwarfToDsym cache path; sanitize library name - Make version regex greedy to capture pre-release suffixes - Improve format detection error message for non-.ips files - Guard xcrun fallback with Get-Command check - Add .dwarf-to-.dSYM conversion guidance for macOS manual fallback - Escape .ips in eval.yaml regex assertion - Fix UUID note in reference doc (normalize, not assume lowercase) - Fix symbol server example to use crash-log image name Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review round 2: case-sensitive replace, dedup, path fix, redact PII - Use -creplace with guard for vmregioninfo duplicate key handling - Use Sort-Object -Unique for proper library deduplication - Fix macOS fallback to create .dSYM bundles under symbols-out/ - Redact device identifiers in test fixture .ips file Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Prevent dSYM cache poisoning: validate UUID on cache hit, remove on mismatch - Convert-DwarfToDsym now verifies cached dSYM UUID before reusing - On UUID mismatch during download, remove the bad cached bundle so subsequent runs can retry cleanly Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cache validation: normalize UUID comparison, clean up dwarf + bundle on mismatch - Use Format-Uuid to normalize dwarfdump output before comparing to already-normalized \ in Convert-DwarfToDsym cache check - On UUID mismatch after download, remove the entire .dSYM bundle directory and the cached .dwarf file to prevent repeat failures Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resolve-Frames null handling: use List[object] to preserve null entries PowerShell array += \ silently drops the element, breaking index alignment between results and input addresses. Switch to List[object].Add() which correctly preserves null entries. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix dSYM bundle root traversal, qualify version-from-path note - Walk up directory tree until *.dSYM is found instead of going up only 2 levels (which lands at Contents/Resources, not the bundle) - SKILL.md: note that version-in-path only works on macOS shared framework installs; iOS paths don't embed the runtime version Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove incorrect .github/skills path reference from training log The .github/skills/ directory doesn't exist in this repo. The file was already listed under its actual plugins/dotnet-diag/ path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Detect simulator for RID inference in manual fallback guidance Detect CoreSimulator in image paths to use iossimulator-/tvossimulator- RIDs for simulator crashes, avoiding UUID mismatches from wrong runtime packs. Also handle arm64e CPU type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix Resolve-Frames return type and scope issue in simulator detection - Return \.ToArray() instead of ,\ to avoid wrapping the list in a single-element array - Use \.usedImages instead of undefined \ in RID inference block Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify ParseOnly .NET Libraries section shows only frame-relevant images Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix YAML escape in assertion pattern (use single quotes for regex backslash) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten stop signal to prevent Android symbolication spillover The apple-crash-symbolication skill's stop signal for wrong file formats was naming specific Android tools (ndk-stack, addr2line) and the android-tombstone-symbolication skill. This caused models to learn about Android symbolication from the Apple skill and then actually execute it, resulting in 4-5x token bloat and completion regression on the Android rejection eval scenario. Remove tool suggestions from the stop signal — just say 'stop, don't symbolicate.' The Android skill handles its own routing when loaded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify scenario 3 rubric to focus on positive skill knowledge Replace 4 rubric items (including 2 'Did NOT' items) with 2 items that test positive knowledge the skill provides. The 'Did NOT' items gave credit to baseline responses that also don't attempt iOS workflow (because they don't know about it), minimizing the quality delta. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix nuget.org domain reference: drop www. prefix to match known-domains.txt Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Merge upstream main and use api.nuget.org v3 endpoint Main changed known-domains.txt to use path-scoped entries (nuget.org/account/trustedpublishing instead of bare nuget.org). Switch the symbols download URL to the v3 flatcontainer endpoint on api.nuget.org which is in the allowlist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix .gitignore merge conflict Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Steve Pfister <steveisok@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Steve Pfister <stpfiste@microsoft.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |
||
|
Jan Krivanek
|
a331d0994e | Fix the devops health workflows after gh-aw update (#511) | ||
|
Jan Krivanek
|
ba4ff9569c | Fix session visualisation filtering (#498) | ||
|
Amaury Levé
|
93dc33da86 |
Stabilize and unify some test skills (#501)
* Deduplicate test skill references and clarify skill boundaries - Move platform-detection.md and filter-syntax.md to plugins/dotnet-test/shared/, removing 3 identical copies of each from run-tests, mtp-hot-reload, and migrate-vstest-to-mtp reference directories. - Move dotnet.md from exp-test-smell-detection/extensions/ to shared/ as dotnet-test-frameworks.md in both dotnet-test and dotnet-experimental plugins. Update exp-assertion-quality, exp-test-boilerplate-detection, exp-test-tagging, and test-anti-patterns to reference the shared file instead of inlining framework detection tables. - Differentiate test-anti-patterns (quick pragmatic review) from exp-test-smell-detection (deep formal audit with academic taxonomy) by updating descriptions and cross-referencing each other in When Not to Use sections. - Update skill-validator to allow ../../shared/ file references while still blocking other parent-directory traversals. Add tests for the new rule. * Switch from shared/ directories to hidden reference skills Replace the plugin-level shared/ directories with non-invocable reference skills (user-invocable: false) that other skills reference by name. - Create platform-detection, filter-syntax, and dotnet-test-frameworks as hidden skills under plugins/dotnet-test/skills/. These contain the detection tables and syntax references previously duplicated across run-tests, mtp-hot-reload, and migrate-vstest-to-mtp. - Create exp-dotnet-test-frameworks as a hidden skill under plugins/dotnet-experimental/skills/ for the experimental test analysis skills (exp-test-smell-detection, exp-assertion-quality, etc.). - Update all consuming skills to reference these by skill name in backtick notation instead of file links. - Revert the skill-validator ../../shared/ exception — no longer needed since all references now use the standard skill name mechanism. * Merge exp-test-boilerplate-detection into exp-test-maintainability exp-test-maintainability was only 6 calibration rules with no workflow. exp-test-boilerplate-detection had the full 5-category detection workflow, examples, calibration, and validation. Merge the boilerplate content into exp-test-maintainability (the broader, more user-facing name) and add the two unique maintainability rules (DisplayName guidance, DataRow vs DynamicData preference) to Category 3. - Replace exp-test-maintainability SKILL.md with the merged content - Move test fixtures from exp-test-boilerplate-detection to exp-test-maintainability - Merge eval.yaml scenarios (4 total: 2 from each original skill) - Delete exp-test-boilerplate-detection skill and tests - Update all cross-references in exp-test-smell-detection, dotnet-test-frameworks, exp-dotnet-test-frameworks, and CODEOWNERS * Add cross-references to test-anti-patterns for deep mock and duplication analysis Point users to exp-mock-usage-analysis from the Over-mocking entry and to exp-test-maintainability from the Duplicate tests entry. * Add exp-dotnet-test-frameworks to CODEOWNERS * Improve run-tests SDK 10 MTP detection for blame-hang scenario Inline the critical SDK 10 detection signal (global.json test.runner) directly in Step 1 instead of deferring entirely to the platform-detection skill. This makes the distinction between SDK 10 (no -- separator) and SDK 8/9 (requires -- separator) more prominent. Add a quick detection summary table, strengthen the Common Pitfalls entry for SDK 10 with a blame-hang-timeout example, and keep the platform-detection skill reference for the full detection logic. * Improve skill activation keywords in descriptions - exp-test-maintainability: Add 'suggest a better test structure', 'consolidate similar test methods', 'convert copy-paste tests to data-driven parameterized tests' to match prompts like 'each new case needs a whole new method, suggest a better structure'. - test-anti-patterns: Add 'review tests', 'find test problems', 'check test quality', 'audit tests for common mistakes' to match review-style prompts that don't use the word 'anti-pattern'. - run-tests: Add 'hang timeout', 'blame-hang', 'blame-crash', 'TUnit' to match SDK 10 blame scenarios and TUnit filter scenarios that were intermittently not activating. |
||
|
dependabot[bot]
|
25e3b5fd0c |
Bump the github-actions-dependencies group with 2 updates (#503)
Bumps the github-actions-dependencies group with 2 updates: [actions/cache](https://github.com/actions/cache) and [actions/setup-node](https://github.com/actions/setup-node). Updates `actions/cache` from 4.2.3 to 5.0.4 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/v4.2.3...668228422ae6a00e4ad889ee87cd7109ec5666a7) Updates `actions/setup-node` from 4.4.0 to 6.3.0 - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/49933ea5288caeca8642d1e84afbd3f7d6820020...53b83947a5a98c8d113130e565377fae1a50d02f) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 5.0.4 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies - dependency-name: actions/setup-node dependency-version: 6.3.0 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Jan Krivanek
|
070f8b4648 |
Add pats rotation to agentic workflows (#500)
* Copy PATs rotation from dotnet/core * Recompile agentic workflows with gh-aw v0.67.1 Remove unsupported 'failure' safe-output (dropped in gh-aw v0.67.1). Regenerate lock files with the new compiler version. No functional changes. * Add Copilot PAT pool rotation to agentic workflows Use the select-copilot-pat action to randomly select from a pool of COPILOT_GITHUB_TOKEN / COPILOT_GITHUB_TOKEN_2..8 secrets, distributing Copilot API load across multiple PATs. This is a stop-gap until organization-level billing is available for Agentic Workflows. See .github/actions/select-copilot-pat/README.md. |
||
|
Mukund Raghav Sharma (Moko)
|
665c39e957 |
Add configuring-opentelemetry-dotnet skill (#268)
* Add configuring-opentelemetry-dotnet skill (+8.1% eval, near-miss) Teaches OpenTelemetry SDK configuration in ASP.NET Core: package selection, AddOpenTelemetry with tracing/metrics/logging, custom ActivitySource and Meter creation, OTLP exporter setup, and context propagation. Eval results: +8.1% improvement (threshold: 10%, near-miss) Includes eval.yaml with OTel setup scenario + negative test. * Sharpen eval.yaml: add custom-spans-invisible pain-point and name-matching rubric emphasis * Migrate configuring-opentelemetry-dotnet to plugins/ directory structure * Move OpenTelemetry skill to new dotnet-aspnet plugin, address review feedback - Move skill from plugins/dotnet-diag to new plugins/dotnet-aspnet plugin - Create dotnet-aspnet plugin.json - Fix SKILL.md frontmatter: remove code fence wrapper so YAML metadata parses - Fix undefined 'order' variable in ProcessOrderAsync example - Move SqlClient/Runtime instrumentation to clearly marked optional sections - Add missing using directives (System.Diagnostics, OpenTelemetry.Context.Propagation) - Handle null Activity.Current in context propagation snippet - Use IMeterFactory via DI instead of static Meter per official guidance - Add OpenTelemetry.Instrumentation.Http to eval rubric package list - Simplify eval prompt to be more natural/generalized - Update CODEOWNERS: add dotnet-aspnet plugin entries, remove old dotnet-diag entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Configure metrics OTLP exporter endpoint explicitly to match tracing config Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address round 2 review: logging endpoint, marketplace, RecordException using, ActivitySource in propagation, HttpClient pitfall - Configure logging OTLP exporter endpoint explicitly for consistency - Add dotnet-aspnet to both marketplace.json files - Add 'using OpenTelemetry.Trace' for RecordException extension method - Declare ActivitySource in context propagation snippet - Fix HttpClient instrumentation pitfall (works with new HttpClient() too) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify logging package source and add ActivitySource registration reminder Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add @dotnet/aspnet team to skill CODEOWNERS entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add dotnet-aspnet to README table, make propagation ActivitySource static Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add expect_activation:false, IMeterFactory using, clarify OTLP exporter scope Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add OtlpExporter using directive, clarify OTLP package covers logging Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove Prometheus from OTLP claim, add OTLP exporter assertion to eval Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify Jaeger is traces-only, document logging package provenance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval: increase timeout, reject tools, rewrite prompt as explanation-only - Scenario 1: rewrite prompt to ask for code explanation (not file modification), add reject_tools to prevent bash/edit overhead, increase timeout 120s->360s, update rubric wording - Scenario 2: add reject_tools to eliminate tool-overhead noise Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve OTel skill evals and address review feedback - Remove redundant per-skill CODEOWNERS lines (ViktorHofer feedback) - Add SetResourceBuilder + ParseStateValues to SKILL.md Step 3 (logging provider gap) - Rewrite eval scenarios: 2 targeted positive scenarios testing real skill gaps - Remove noisy negative activation scenario (high variance, -21.5% drag) - Eval results: +14.2% improvement (threshold: 10%), overfitting 0.10 (low) Scenario 1: 3.0/5 -> 5.0/5, Scenario 2: 3.8/5 -> 5.0/5 * Add propagate-trace-context scenario to OTel evals (+25.6%) Third scenario tests trace context propagation across message queues, where the base model scores 3.2/5 and skill-enhanced scores 5.0/5. Removed metrics and debug scenarios that baseline already aced (5.0/5) which only added token cost without measurable improvement. Final: 3 scenarios, +25.6% improvement, overfitting 0.09 (low) * Remove duplicate dotnet-aspnet CODEOWNERS entries (L69-70 shadowed by L86-87) * Use @dotnet/aspnet team alias in CODEOWNERS for wider reach * Addressed feedback --------- Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Jan Krivanek
|
d861bbf62e |
Add AGENTVIZ session replay integration (#494)
* Add AGENTVIZ session replay integration - Add workflow_dispatch trigger to evaluation.yml - Add --keep-sessions to skill-validator evaluate step - Add publish-session-data job (mirrors publish-token-data) - Add replay link to PR comments (comment-on-pr) - Add AGENTVIZ SPA build/deploy to deploy-dashboard job - Add setup-node step to deploy-dashboard - Add per-plugin Sessions Visualisation links to dashboard.js - Create build-replay-sessions.ps1 (manifest generation from sessions.db) - Create purge-replay-sessions.ps1 (7-day retention management) * Address code review feedback on PR #494 - Use [IO.Path]::PathSeparator instead of hardcoded ';' for cross-platform PATH - Compute cutoff date in UTC for correct retention comparisons - Precompute ID HashSet before merge loop to avoid O(n^2) - Pin actions/setup-node to commit SHA (49933ea5...#v4) - Pin AGENTVIZ clone to commit SHA with verification - Skip npm ci + build when deployed commit SHA matches pinned SHA * Remove hardcoded AGENTVIZ SHA; use cache + zero-clone checks - Resolve AGENTVIZ target SHA via git ls-remote (no clone) - Read deployed SHA via curl from raw.githubusercontent.com (no clone) - Cache build output keyed by commit SHA (skip npm ci+build on hit) - Only clone AGENTVIZ repo on cache miss when SPA needs rebuild * Address round 2 code review feedback on PR #494 - Compare scheduled dir dates at day granularity (dirDate.Date vs cutoffDate.Date) - Fix useTempDir null-safe path comparison via GetFullPath + null guard - Use UTC for dateTag in build-replay-sessions.ps1 (consistent with purge) - Fix cache/deploy path mismatch: deploy from /tmp/agentviz-dist (cache path) - Deterministic clone: full clone + git checkout TARGET_SHA (fail on mismatch) - URL-encode manifest param in PR comment link (jq @uri) - Derive manifest generated timestamp from newest session mtime to avoid churn |
||
|
David Ortinau
|
c4b6819dda |
Add 7 core .NET MAUI development skills to dotnet-maui plugin (#456)
* Add 7 core .NET MAUI development skills to dotnet-maui plugin Add maui-shell-navigation, maui-collectionview, maui-data-binding, maui-dependency-injection, maui-app-lifecycle, maui-safe-area, and maui-theming skills with eval.yaml tests and reference docs. Update plugin.json, marketplace.json (both copies), and CODEOWNERS. Resolves #455 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix cross-skill references to non-existent skills Replace 10 dangling references to skills not in this repo (maui-deep-linking, maui-platform-invoke, maui-unit-testing, maui-maps, maui-animations, maui-accessibility, maui-app-icons-splash, maui-bootstrap-theme, maui-localization) with plain-English guidance or references to skills that exist in this PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix code correctness bugs and remaining cross-references Address PR review feedback and skeptic review findings: Code correctness: - Fix BackgroundColor="Primary" (XAML crash) → use {StaticResource Primary} - Fix MergedDictionaries.Clear() (nukes all styles) → remove-then-add pattern - Fix HorizontalStackLayout Fill (no effect) → use Grid with ColumnDefinitions - Fix AncestorType with ViewModel type → use ContentPage (visual element) - Fix Frame usage → replace with Border (recommended control) - Remove redundant MultiBinding StringFormat="{}{0}" - Fix OnStart reference → use CreateWindow for theme restore - Soften lifecycle OnStart/OnSleep characterization (compat shims, not absent) - Qualify ResourceDictionary code-behind requirement (only for x:Class) Cross-references: - Remove all remaining body-text refs to non-existent skills - Replace with Microsoft Learn doc links or generic guidance Eval: - Relax SafeAreaRegions assertion to accept SafeAreaEdges or SafeAreaRegions - Relax theming rubric for ResourceDictionary code-behind Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Restore Clear() theme pattern with clarifying comment The Clear()+Add() pattern is intentional — theme dictionaries are designed to be the only merged dictionaries in this architecture. Added comment explaining the assumption and what to do if non-theme dictionaries exist. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix second-pass review findings: API accuracy and eval robustness Code correctness: - ScrollView SafeAreaEdges="All" → "Container" (ScrollView only honors Container and None; All silently fails for keyboard avoidance) - DI CreateWindow example: add comment noting AppShell must be registered - AddHttpClient: add comment noting Microsoft.Extensions.Http NuGet required - MultiBinding: remove redundant StringFormat from reference doc (matches SKILL.md) - RelativeSource Self table: Path=Width → Path=WidthRequest (Width is read-only -1 before layout; WidthRequest is the correct bindable property) - CollectionView StackLayout warning: clarify it refers to BindableLayout on StackLayout vs CollectionView ItemsLayout types - SafeAreaEdges: fix "attached property" → "property" (it's a regular property) - ios:NavigationPage example: add xmlns comment for ios namespace Eval assertions: - DI Scoped regex: accept "doesn't"/"lacks"/"without" not just "no" - CollectionView ViewCell: match negation in either direction around ViewCell - Lifecycle events: require 3+ distinct events instead of any single one Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix final PR review comments - safe-area eval: tighten Container assertion to require SafeAreaEdges context - DI: merge #else fallback into single compilable #if block - lifecycle: behaviour → behavior (American English consistency) - theming: macOS Catalyst minimum 10.14 → 10.15 (Catalina required) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
948a9a52ef |
Add exp-mock-usage-analysis skill and evaluation (#483)
* Add exp-mock-usage-analysis skill and evaluation New experimental skill that detects unused, redundant, or unnecessary mocks in .NET test suites. Covers Moq, NSubstitute, FakeItEasy, and manual test doubles. - SKILL.md with 6-step workflow: gather code, classify mocks, detect anti-patterns, incorporate runtime data, generate recommendations, report findings - eval.yaml with 10 scenarios: trivial type mocking, unused setups, redundant configs, stable utility mocks, well-placed mocks (no false positives), NSubstitute support, FakeItEasy support, setup sprawl, and 2 non-activation boundary tests - Fixture files for each scenario with production + test code - CODEOWNERS entry for @dotnet/dotnet-testing * Remove no-headroom scenarios, increase NSubstitute timeout Remove 4 scenarios where baseline already scores 5.0/5 (model handles these well without skill guidance): - Detect mocking of DTOs, records, and enums - Recognize well-placed mocks without inventing false positives - Decline when asked about non-mock test anti-patterns - Decline mock framework migration request Increase NSubstitute scenario timeout from 120s to 180s to avoid baseline timeouts impacting scoring. Remaining 6 scenarios all show positive skill impact (3→5, 3→4, etc). * Document areas the model handles well without the skill Add a subsection under 'When Not to Use' listing mock-related tasks where the base model already scores 5.0/5 without skill guidance: - Mocking trivial types (DTOs, records, enums) - Recognizing well-placed mocks (no false positives) * Fix md * Improve |
||
|
Amaury Levé
|
d5ca4c6c01 |
Add exp-test-smell-detection skill and evaluation (#491)
* Add exp-test-smell-detection skill and evaluation - Language-agnostic core skill detecting 10 test smells (conditional logic, mystery guest, sleepy test, assertion-free, eager test, magic numbers, sensitive equality, exception handling, general fixture, ignored tests) - .NET extension with framework-specific detection tables (MSTest/xUnit/NUnit/TUnit) - Reference catalog with 19 smells from testsmells.org research - Evaluation with 4 scenarios: smelly suite, clean suite, integration tests, non-activation - CODEOWNERS entry for @dotnet/dotnet-testing * Improve integration test fixture and non-activation timeout - Add real smells to integration test fixture (Thread.Sleep, conditional logic, assertion-free test) mixed with legitimate integration patterns - Update rubric to test nuanced distinction between real smells and acceptable integration test patterns - Increase non-activation scenario timeout from 120s to 300s to prevent timeout-related scoring penalties * Fix test projects * Add domains for testsmells * Remove extra domains |
||
|
Dan Moseley
|
165e6439fe |
Tighten known-domains allowlist (#496)
* Tighten known-domains allowlist Narrow broad domain entries to specific paths/subdomains actually used: - aka.ms -> aka.ms/dotnet-trace, aka.ms/perfcollect - nuget.org -> api.nuget.org, www.nuget.org/account/trustedpublishing - npmjs.com/package/@modelcontextprotocol -> pinned to inspector/v/0.21.1 Remove unused domains: fluentassertions.com, speedscope.app Update skill content to match: pin MCP Inspector link, remove FluentAssertions link, remove speedscope hyperlink. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * updated mcp skills codeowner * Refine scoped known-domain entries Remove the remaining nugettest.org publish example and drop the nugettest.org allowlist entry. Also normalize path-scoped entries to omit a hardcoded www prefix so both host forms match the existing ReferenceScanner logic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Leslie Richardson <lerich@microsoft.com> |
||
|
Amaury Levé
|
eb0f98eba6 |
Add exp-test-gap-analysis skill for pseudo-mutation test gap detection (#490)
New experimental skill that performs pseudo-mutation analysis on .NET production code to find gaps in existing test suites. Covers boundary, boolean/logic, return value, exception removal, arithmetic, and null-check mutation categories. Includes eval.yaml with 4 scenarios (boundary gaps, logic gaps, well-tested code recognition, non-activation) and fixture projects. |
||
|
dependabot[bot]
|
0417c8ec62 |
Bump actions/cache in the github-actions-dependencies group (#492)
Bumps the github-actions-dependencies group with 1 update: [actions/cache](https://github.com/actions/cache). Updates `actions/cache` from 4.2.3 to 5.0.4 - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](https://github.com/actions/cache/compare/5a3ec84eff668545956fd18022155c47e93e2684...668228422ae6a00e4ad889ee87cd7109ec5666a7) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 5.0.4 dependency-type: direct:production update-type: version-update:semver-major dependency-group: github-actions-dependencies ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Jan Krivanek
|
498755bc0d |
Make the infra eval gate faster (#489)
* Make the infra eval gate faster * Address PR feedback: handle infra+skill overlap, detect evaluation.yml changes - When both infra and skill changes exist in a PR, evaluate the changed skills normally instead of the random subset (fixes Copilot review). - Expand infra detection to include .github/workflows/evaluation.yml changes in all three detection sites (pr-status, fork-pr-status, discover). |
||
|
Viktor Hofer
|
167b7c3772 |
Fix evaluation cache key: compute explicitly instead of cache-primary-key (#488)
The pinned actions/cache SHA (5a3ec84, v4) does not expose the cache-primary-key output. This caused the evaluate job to fail with 'Input required and not supplied: key' because the build-validator job output was always empty. Fix: compute the cache key in a dedicated step and pass it as a job output, instead of relying on an output that doesn't exist at this action version. Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
3fff1f2573 |
Skill-validator: perf, cancellation, and reliability improvements (#482)
* Skill-validator: perf, cancellation, and reliability improvements Performance: - Cache skill-validator-dist.tar.gz in CI (evaluation.yml and skill-check.yml) to skip .NET SDK setup, build, and tar on cache hit. Evaluate and comment-on-pr jobs restore from cache with artifact download fallback. - Replace SemaphoreSlim with Lock in SessionDatabase for lighter-weight synchronous locking. Cancellation: - Thread CancellationToken from System.CommandLine entry point through the entire evaluate call chain: Run -> EvaluateTarget -> EvaluateSkill/EvaluateAgent -> ExecuteScenario/ExecuteAgentScenario -> ExecuteRun/ExecuteAgentRun, plus EvaluateSkillNoise -> ExecuteNoiseTest. - Add CancellationToken parameter to AgentRunner.RunAgent, Judge.JudgeRun, PairwiseJudge.Judge, and OverfittingJudge.Analyze, forwarding to RetryHelper and LlmSession.SendAsync respectively. - Pass CancellationToken to all ConcurrencyLimiter.RunAsync call sites so semaphore waits are cancellable. - Re-throw OperationCanceledException at the top-level skill evaluation loop so Ctrl+C / CI cancellation propagates instead of being swallowed as a skill failure. Reliability: - Smart timeline truncation in Judge and PairwiseJudge: error events (session.error, runner.error) are always preserved during truncation regardless of position, with a summary marker showing event type breakdown. - Add jitter (±25%) to RetryHelper exponential backoff to prevent thundering herd on rate-limit recovery. - Add CancellationToken parameter to ConcurrencyLimiter.RunAsync<T>. - Add ConcurrencyLimiter tests (6 tests) and RetryHelper jitter tests (2 tests). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback - RetryHelper: re-clamp delay after jitter so it never exceeds the remaining budget or MaxSingleDelayMs. - Judge/PairwiseJudge: handle edge case where error count alone exceeds MaxTimelineEvents by trimming errors with a summary instead of unconditionally appending all of them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add high-variance detection and refactor FailureKind to enum - Add Statistics.CoefficientOfVariation() for measuring score consistency - Flag scenarios with CV > 0.5 as high-variance in ScenarioComparison - Show console warning and markdown footnote for high-variance scenarios - Include high-variance scenario names in Comparator verdict reason - Refactor FailureKind from magic strings to strongly-typed enum with JsonStringEnumMemberName attributes for backwards-compatible JSON output - Add 6 new CV unit tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Mukund Raghav Sharma (Moko)
|
c92faa7800 |
Add minimal-api-file-upload skill (#264)
* Move minimal-api-file-upload skill to aspnetcore plugin with 3 eval scenarios Per repo restructuring feedback, ASP.NET Core specific skills should be under the aspnetcore plugin rather than the dotnet plugin. * Increase eval timeouts to 180s to avoid timeout-related scoring penalties * Generalized and addressed feedback * Improve eval scenarios: remove noisy tests, add dual-limit and security review scenarios * Add CODEOWNERS for aspnetcore plugin * Fix eval: enrich prompts for skill activation, bump security review timeout to 240s * Drop size-limit scenario (baseline already 5.0), keep 3 high-signal scenarios * Reframe security review as file upload fix to improve skill activation * Review * Contribution guide followed * Updated name * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-aspnet/skills/minimal-api-file-upload/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address PR #264 review feedback from Brennan and Copilot - Rename plugin from aspnetcore to dotnet-aspnet per repo conventions - Fix streaming filename: remove user-controlled extension, use bare GUID - Remove explicit sanitization code (Option B) per Brennan's feedback - Add case-insensitive MIME type comparison - Validate ContentType matches magic bytes (reject mismatches) - Fix stream rewind + CopyToAsync: copy from already-opened stream - Reword streaming section to accurately describe buffering vs chunks - Update marketplace.json, README.md, CODEOWNERS for dotnet-aspnet rename --------- Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
7747fcb427 |
Move skill-validator docs under src/ and include in packages (#476)
* Move skill-validator docs under src/ and include in packages Move README.md to src/ and InvestigatingResults.md + OverfittingDetection.md to src/docs/ so they ship in the NuGet package and .tar.gz archives. - Include docs/*.md in NuGet pack and publish output via CopyToPublishDirectory - Add LICENSE to publish output (tar.gz only, not NuGet — uses PackageLicenseExpression) - Replace enumerated docOnlyFiles lists with a regex pattern in CI workflows - Deduplicate eval.yaml docs between CONTRIBUTING.md and the shipping README - Add test fixture files walkthrough to README (moved from CONTRIBUTING.md) - Add AGENTS.md guidance: src/ is the packaging boundary Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update link format in failure diagnosis message * Update AGENTS.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
75f72efb0e |
Fix token usage collection (#471)
* Fix token usage collection * Ensure evaluate is run on schedule * Fix PR title issue * Update eng/dashboard/token-usage.js Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
b527b41573 |
Add exp-test-boilerplate-detection and exp-assertion-quality skills (#468)
Two new experimental skills for dotnet-experimental plugin: - exp-test-boilerplate-detection: Detects duplicate boilerplate patterns across .NET test suites and identifies refactoring opportunities. Categories: repeated construction, assertion patterns, copy-paste methods, duplicated setup/teardown, repeated infrastructure. - exp-assertion-quality: Analyzes assertion variety and depth across .NET test suites. Produces metrics dashboard including assertion count, type spread, trivial %, assertion-free tests, and negative assertion coverage. Both skills include eval.yaml with 3-4 scenarios each (positive, negative, and non-activation) plus test fixtures. |
||
|
Jan Krivanek
|
e6250320e8 | Clean the agentic workflows report bugs (#474) | ||
|
Copilot
|
447e7367c8 |
Move investigation prompt to workflow summary, rename Full Results link (#473)
* Initial plan * Move investigation prompt to workflow summary, remove InvestigatingResults.md callout, rename Full Results link Agent-Logs-Url: https://github.com/dotnet/skills/sessions/b5b537b4-3d90-48ff-ab2d-3298660ccc97 Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> * Revert Reporter.cs change per review feedback Agent-Logs-Url: https://github.com/dotnet/skills/sessions/e5845ec8-06b3-4ffa-b282-fd63aa2a467c Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Copilot
|
6248ef2b4b |
Fix Action summary to show all columns and remove redundant "Full results" link (#470)
* Initial plan * Fix Action summary to show all columns and remove redundant Full results link The comment-on-pr job was using the simplified PR comment table (consolidated-comment.md) for the Action step summary. This change: 1. Writes the full summary-body.md (with all columns including Quality (Plugin) and Agents Invoked) to GITHUB_STEP_SUMMARY. 2. Removes the "Full results" link from the Action summary since it links to the same page. 3. Keeps the investigation prompt in both PR comment and Action summary. Agent-Logs-Url: https://github.com/dotnet/skills/sessions/870c41f2-0e41-4e62-9a43-d1f7128bab06 Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Dan Moseley
|
17376fd82d |
Exclude doc-only .md files from evaluation infra change detection (#466)
* Exclude README.md from infra change detection in evaluation workflow eng/skill-validator/README.md changes were triggering a full all-skills evaluation. Add it to the doc-only exclusion list alongside InvestigatingResults.md, and extend the exclusion to the discover job which was missing it entirely. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <7412651+ViktorHofer@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
e1863cc526 | Improve investigation guide: PR number in prompt, document reject_tools, baseline-already-good pattern (#451) | ||
|
Amaury Levé
|
a220f36256 |
Add exp-test-maintainability skill for test structure review (#418)
New experimental skill that assesses test maintainability: duplication, test size, data-driven patterns, display names, builder/helper extraction, and shared setup. |
||
|
Dan Moseley
|
604c46c8c6 | Fix eval investigation docs: download command, schema accuracy, example script (#449) | ||
|
leslierichardson95
|
7286c31118 |
Add skills for C# MCP Server Development (#317)
* Add C# MCP server skills (create, debug, test, publish) Four new skills for the C# MCP server development lifecycle: - mcp-csharp-create: Scaffolding with dotnet new mcpserver, tools/prompts/resources, transport config - mcp-csharp-debug: MCP Inspector, VS Code integration, breakpoint debugging, logging - mcp-csharp-test: Unit tests with ClientServerTestBase, integration with WebApplicationFactory, evals - mcp-csharp-publish: NuGet packaging, Docker/Azure deployment, MCP Registry publishing Each skill includes SKILL.md with progressive disclosure references/ and eval.yaml tests. * Fix mcp-csharp-create eval: use regex assertions for combined attribute syntax Replace scaffolding-heavy scenarios with implementation-focused ones that test MCP-specific features (resources, prompts, logging). Fix assertion patterns to match combined C# attribute syntax [McpServerTool, Description()] instead of requiring standalone [McpServerTool]. Increase timeouts to 180s to account for skill-reading overhead. Validator result: passed=True, improvement=44.6% (threshold=10%) * Add CODEOWNERS entries for MCP C# skills (create, debug, publish, test) * Update plugins/dotnet/skills/mcp-csharp-test/references/test-patterns.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-publish/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-publish/references/nuget-packaging.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/mcp-csharp-debug/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Remove evaluation step and guide from mcp-csharp-test skill * Add CODEOWNERS entries for dotnet-maui skills * Update CODEOWNERS to add @artl93 as co-owner for mcp-csharp skills * Replace parent-directory file references with backtick skill names Cross-skill references used ../sibling/SKILL.md paths which the validator rejects. Replace with backtick-quoted skill names to match the convention used by other skills in the repo. * Add known domains for MCP server development skills Add domains referenced by the mcp-csharp-* skills: - github.com/modelcontextprotocol/registry - github.com/open-telemetry/semantic-conventions - npmjs.com, xunit.net, fluentassertions.com, nugettest.org Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Docker commands to use placeholder for registry and image names * moved mcp skills and their evals into dotnet-ai plugin * cleaned up moved files to dotnet-ai * cleaned up files * Update .github/CODEOWNERS Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update nuget-packaging.md * Update repository URL format in nuget-packaging.md * Add github.com/ to known-domains to allow truncated placeholder URLs The URL regex truncates at angle brackets (invalid in URIs), so placeholder URLs like https://github.com/<yourusername>/<repo> become https://github.com/ which didn't match any path-scoped entry. Adding github.com/ (with trailing slash) covers only this case without broadly allowing all github.com URLs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update Docker tag command to use placeholder for server name * Update known domains and fix GitHub username placeholders in documentation * Remove McpServerToolType from mcp-csharp-create eval McpServerToolType is only needed with WithToolsFromAssembly(), which is discouraged because it is not Native AOT compatible. Updated eval assertions and rubrics to prefer WithTools<T>() instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve eval resilience for mcp-csharp-create and mcp-csharp-debug - Increase mcp-csharp-create timeouts from 180s (or default 120s) to 360s for all 3 scenarios. These scenarios consistently time out because the model spends time on bash exploration/scaffolding before writing code. - Add explicit timeout to scenario 3 which was relying on the 120s default. - Broaden mcp-csharp-debug Inspector rubric to accept both the concise single CLI command style and the step-by-step UI configuration walkthrough, since both are correct approaches. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eng/known-domains.txt Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Broaden debug 'failing tool' rubric to accept multiple debugging approaches The rubric criterion 'Shows how to attach a debugger' was too narrow. The skilled answer correctly focused on the #1 cause (stdout pollution) but scored low because it didn't show a specific 'attach to process' flow. Broadened to accept any valid debugging approach: attaching, Debugger.Launch(), or launch.json configuration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix 4 eval failures in mcp-csharp-debug and mcp-csharp-test skills mcp-csharp-debug: - Trim SKILL.md verbosity (183->160 lines) and mcp-inspector.md (67->54 lines) - Add Diagnosing Tool Errors section covering debugger, output panel, Inspector, common culprits - Rebalance debugging narrative away from stderr-only focus - Move HTTP logging config to ide-config.md reference mcp-csharp-test: - Add eval/evaluations keywords to frontmatter for activation - Add HTTP tool invocation test pattern (tools/call via WebApplicationFactory) - Trim test-patterns.md bloat (remove Test Categories, Coverage, Input Validation) - Create references/evaluations.md with qa_pair format and read-only/deterministic guidance Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Copilot
|
97832189cf |
Simplify evaluation table in PR comments; keep full table in workflow summary (#448)
* Initial plan * Simplify evaluation workflow table: remove Quality (Plugin) and Agents Invoked columns from PR comment - Add `simplified` parameter to GenerateMarkdownSummary in Reporter.cs - When simplified=true, exclude Quality (Plugin) and Agents Invoked columns - Update ConsolidateCommand with --summary-output for full table - evaluation.yml: simplified table for PR comment, full table for step summary - Update Full results link to indicate additional columns are available - Add 3 tests for simplified table behavior Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/skills/sessions/d2590b7a-023a-45a5-928b-9e658f40b58e * Simplify evaluation table in PR comments; keep full table in workflow summary Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/skills/sessions/d2590b7a-023a-45a5-928b-9e658f40b58e * Delete .nuget/nuget.exe * Move table simplification to workflow post-processing instead of skill-validator Revert all C# changes (Reporter.cs, ConsolidateCommand.cs, tests). The table simplification is now a post-processing step in evaluation.yml using a Python script that strips "Quality (Plugin)" and "Agents Invoked" columns from the markdown before posting as a PR comment. The full table goes to the workflow step summary. Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/skills/sessions/c8e0acc6-1128-4bf5-8170-382b278500e5 * Apply review suggestions: use heredoc, remove unused import, full step summary Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> Agent-Logs-Url: https://github.com/dotnet/skills/sessions/be2dd88e-e30b-4c50-905c-1cff9cd21422 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Jeff Schwartz
|
55a7d80527 |
adding SIMD/TensorPrimitives to dotent-diag/analyzing-dotnet-performance (#330)
* incorporated feedback from previous pr, iterated with 100+ scalar loop tests to come to this skill, moved from skill to reference in the performance skill * Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests/dotnet-diag/analyzing-dotnet-performance/eval.yaml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update plugins/dotnet-diag/skills/analyzing-dotnet-performance/references/simd-vectorization.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Address review feedback on SIMD vectorization reference - Check Span<T>/MemoryExtensions before TensorPrimitives (no extra dependency) - Add all supported types (sbyte, ushort, uint, ulong, nint, nuint, char via ushort) - TensorPrimitives: add constraint and applicable types columns (not just float/double) - Add FusedMultiplyAdd, clarify AddMultiply vs MultiplyAdd distinction - Prefer portable APIs over platform-specific intrinsics (allow when perf justifies) - Fix dispatch pattern to use if/else if to avoid pessimizing small inputs - Remove LINQ references Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * addressing more feedback * move to the experimental plugin * move the simd tests to exprimental * add codeowners --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
1d56634150 | Create experimental plugin (#446) | ||
|
Dan Moseley
|
b39fe8541b |
Add evaluation troubleshooting guide for AI agents (#441)
* Add evaluation troubleshooting guide and link from PR comments
Add InvestigatingResults.md with:
- How to find and download evaluation artifacts from workflow runs
- Full results.json schema documentation
- Common failure patterns (timeouts, empty output, variance, rubric
issues, skill not activated) with diagnosis steps and fixes
- Example analysis script for AI agents to parse results
Link the doc from:
- Reporter.cs: show a link in the PR comment when any skill fails
- skill-validator README.md: Output section
- CONTRIBUTING.md: CI evaluation section
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update InvestigatingResults.md
* Address review feedback: fix hard-coded URLs and optional plugin run docs
- Reporter.cs: Use repo-relative link instead of hard-coded GitHub URL
- evaluation.yml: Use github.sha instead of hard-coded 'main' branch
- InvestigatingResults.md: Clarify skilledPlugin run is optional
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* THROWAWAY: add trailing newline to trigger eval workflow
This change is intentionally trivial and will be reverted before merge.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address PR #441 review feedback (round 2)
- Reporter.cs: Remove leading / from relative link
- evaluation.yml: Use jq on results.json instead of grepping for emoji
- evaluation.yml: Use needs.gate.outputs.head_sha for revision-pinned URL
- InvestigatingResults.md: Mark pluginImprovementScore/pluginBreakdown as optional
- InvestigatingResults.md: Fix skill-not-activated symptoms to use
skillActivation fields instead of turnCount
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use specific skillActivation field names and fix timestamp description
- Use skillActivationIsolated/skillActivationPlugin instead of generic skillActivation
- Clarify timestamp is when results were written, not run start
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix timestamp field description (was lost in prior commit)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert "THROWAWAY: add trailing newline to trigger eval workflow"
This reverts commit
|
||
|
Viktor Hofer
|
1762a78fed |
Add stale PRs agentic workflow (#435)
* Add stale PRs agentic workflow * Add fork-guard to close-stale-prs workflow Skip scheduled runs on forked repositories to avoid consuming fork owner minutes and failing due to missing secrets/context, matching the pattern used by other scheduled workflows in this repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
19b6cadddf |
Move token usage data into separate branch (#431)
* Move token usage data into separate branch * Address PR review: add help doc, fix title injection, add token-usage fallback * Add -SkipBenchmarkData switch, clean stale files on branch push |
||
|
Viktor Hofer
|
c14e63ba6c |
Combine evaluation.yml and evaluation-run.yml into single workflow (#430)
* Combine evaluation.yml and evaluation-run.yml into single workflow Inline the three jobs from the reusable evaluation-run.yml workflow (build-validator, evaluate, comment-on-pr) directly into evaluation.yml and delete the now-unnecessary reusable workflow file. Changes: - Inline build-validator, evaluate, and comment-on-pr jobs - Eliminate 8-secret forwarding block (direct secrets.* access) - Replace inputs.* with direct needs.* references - Add MODEL/JUDGE_MODEL workflow-level env vars - Fix --verbose duplication in evaluate step - Simplify infra-change detection regex - Update dependency chains (report-status, publish-benchmark) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback: fix consolidate subcommand, improve error reporting - Fix consolidate invocation: use 'evaluate consolidate' subcommand (was incorrectly changed to just 'consolidate' during inlining) - Add build-validator to report-status needs so build failures surface as a deterministic failure state instead of generic error - Add discover to comment-on-pr needs and gate on has_entries so /evaluate on a no-op PR doesn't post a misleading failure comment Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix matrix fallback and header comment - Use fromJson(... || '[]') fallback for matrix entries to prevent errors when discover is skipped on non-evaluation events - Update header comment to accurately describe the is_fork mechanism Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
b92dde1854 |
Refactor skill-validator to vertical slice architecture (#429)
Reorganize the skill-validator project from layer-based grouping (Commands/, Services/, Models/, Utilities/) to feature-based slices (Check/, Evaluate/, Consolidate/, Shared/). Key changes: - Check/: CheckCommand, SkillProfiler, AgentProfiler, PluginValidator, ExternalDependencyChecker, ReferenceScanner - Evaluate/: EvaluateCommand, RejudgeCommand (now a subcommand of evaluate), AgentRunner, Judge, PairwiseJudge, and other eval services - Consolidate/: ConsolidateCommand - Shared/: Models, SkillDiscovery, PluginParser (extracted from PluginValidator), Reporter, and utilities Split PluginValidator: parsing helpers (ParsePluginJson, TryGetSafeSubdirectory) moved to Shared/PluginParser.cs; validation logic stays in Check/PluginValidator.cs. Move RawFrontmatter and RawAgentFrontmatter types from EvalSchema into SkillDiscovery since they are only consumed by discovery. Wire RejudgeCommand as subcommand: skill-validator evaluate rejudge. Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
c4c26175e2 |
Integrate reference scanner into skill-validator CheckCommand (#422)
* Integrate reference scanner into skill-validator CheckCommand - Port reference scanner logic from scan.ps1 to C# ReferenceScanner service - Add --known-domains option to CheckCommand for reference scanning - Move known-domains.txt and allowed-external-deps.txt to eng/ - Remove standalone scan.ps1 and reference-validation.yml workflow - Reference scanning now runs as part of skill-check.yml via skill-validator - Add 34 unit tests for ReferenceScanner (all 463 tests pass) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback: repoRoot derivation, early validation, .agents scanning, SRI policy - Derive repoRoot from --known-domains file path instead of cwd - Fail early with clear error when known-domains file is missing - Add .agents/**/*.md scanning to DiscoverFiles (regression fix) - Fix SRI policy loophole: SRI suppresses SCRIPT-NO-SRI only, domain and HTTPS checks still apply - Use actual known-domains file path in error messages instead of hardcoded eng/known-domains.txt - Update doc comments to reflect full set of scanned paths - Add tests for .agents discovery and SRI domain enforcement Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove repo-specific paths from ReferenceScanner DiscoverFiles now only scans directories it is given -- no hardcoded knowledge of .agents, agentic-workflows, eng, or README.md. The tool is repo-agnostic; callers control what gets scanned via --plugin dirs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Drop repoRoot from ReferenceScanner API ScanFile/ScanFiles no longer take a repoRoot parameter. Finding paths use the file path as provided by the caller, keeping the scanner completely decoupled from any repo layout assumptions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Prune stale domains and add json.schemastore.org to known-domains.txt Removed 10 entries no longer referenced by any plugin content: agentskills.io, visualstudio.com, mcr.microsoft.com, dotnet.github.io, github.com/dotnet/docs, github.com/dotnet/dotnet-api-docs, github.com/dotnet/skills, github.com/github/gh-aw, github.com/modelcontextprotocol/csharp-sdk, developers.openai.com. Added json.schemastore.org (referenced by dotnet-template-engine). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix case-insensitive regex, update docs, add tests - Make UrlRegex and PipeToShellRegex case-insensitive to match the original PowerShell behavior (HTTP://, CURL, Wget all now detected) - Update DiscoverFiles doc comment to remove stale 'individual files' claim - Narrow CONTRIBUTING.md scope description to match actual plugin-only scanning - Add tests for mixed/upper-case URL and pipe-to-shell detection Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove IPv6 loopback support, fix pipe-to-shell allowlist case sensitivity - Remove Ipv6LoopbackRegex and IPv6 test data (not needed) - Add case-insensitive + culture-invariant matching for pipe-to-shell allowlist - Add mixed-case allowlist test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |