mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-opus48
589 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Manish Jayaswal
|
9cf10eebd2 |
Update CONTRIBUTING.md with new guidelines (#319)
* Update CONTRIBUTING.md with new guidelines Added clarification on contribution expectations and skill proposals. * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
Amaury Levé
|
45c66f87d4 | Add prompt to evaluate skills description (#371) | ||
|
Amaury Levé
|
8cb692082a | Improve run-tests skill description (#372) | ||
|
Jan Krivanek
|
93ccead1e7 | Adjust for timeout issues (#354) | ||
|
Amaury Levé
|
1b8f2aa1e4 |
Address Copilot review comments on test agent/skills (#351)
Follow-up from PR #343 where copilot added review after the PR was merged |
||
|
Dan Moseley
|
a0f77070f5 |
Reduce false positives in reference scanner (#360)
- Skip HTTP-not-HTTPS inside fenced code blocks (code examples use http:// listen addresses). EXTERNAL-DOMAIN checks still run inside fences since agents see raw text, not rendered markdown. - CommonMark-compliant fence tracking (type, length, closing-only whitespace, max 3-space indent). - Skip placeholder/template URLs by checking the host portion only (not the full URL) to prevent bypass via path components. - Case-insensitive placeholder matching. - Generalize path-scoped domain matching to all domains (not just github.com), enabling entries like npmjs.com/package/@org. - Treat wildcard listen addresses (http://+:port, http://*:port) as local URLs. - Add xunit.net to known-domains.txt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
1ccd28b1ba |
Make publish-benchmark resilient to individual plugin eval failures (#365)
* Make publish-benchmark resilient to individual plugin eval failures When any single evaluate matrix leg fails (e.g. OOM kill), the entire run-evaluation reusable workflow reports failure. Previously this caused publish-benchmark to be skipped, blocking dashboard updates for ALL plugins -- even the ones that succeeded. Changes: - publish-benchmark: use !cancelled() so it runs when some legs fail - download-artifact: continue-on-error so missing artifacts don't abort - The existing Generate Benchmark Data step already handles missing plugin results with Write-Warning and continue Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make continue-on-error conditional on evaluation result Only tolerate artifact download failures when run-evaluation didn't fully succeed. When all legs passed, a download failure is a real error that should fail the job. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
492ab1d174 |
Reduce scheduled evaluation from every 3h to daily (#362)
* Reduce scheduled evaluation from every 3h to daily, skip if no changes The evaluation schedule was consuming ~/5 days in Copilot premium requests from scheduled runs alone (~/month pace). This change: 1. Reduces the cron schedule from every 3 hours (8 runs/day) to once daily at 08:00 UTC. This alone saves ~87% of scheduled evaluation cost. 2. Adds a check-for-changes guard that skips the scheduled run entirely if there have been no new commits to main since the last successful scheduled evaluation. This avoids wasting tokens re-evaluating identical code. The PR trigger and all PR evaluation behavior is unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: harden skip-if-no-changes guard - Add actions: read permission so the workflow run lookup succeeds - Use status=completed with jq conclusion filter instead of status=success - Make all gh api calls non-fatal (fallback to has_changes=true on error) - Make compare API call best-effort for logging only Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Scope actions:read to discover job, increase per_page to 100 - Move actions:read from workflow-level to discover job permissions for least-privilege (only the schedule guard needs it) - Increase per_page from 10 to 100 so the skip guard finds the last successful run even after long failure streaks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Key skip guard off most recent completed run, not just successful Only skip when the most recent completed scheduled run was successful AND at the same SHA. If it failed/cancelled at the current SHA, proceed to retry. This prevents the guard from skipping indefinitely after a transient failure. Also simplifies to per_page=1 since we only inspect the latest run. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix jq filter to yield empty when no prior runs exist Use // empty to guard against null when workflow_runs array is empty, preventing 'null null' from leaking into LAST_SHA/LAST_CONCLUSION. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bypass skip guard on manual reruns (run_attempt > 1) When a maintainer clicks 'Re-run jobs' on a scheduled run, the event is still schedule with the same SHA. Without this check, the guard would skip the rerun if the previous attempt succeeded. Now reruns always proceed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
74c4a1c7ac |
Fix PowerShell parsing of three-dot diff range in reference scanner (#359)
When pwsh parses git diff --name-only "origin/main"...HEAD, the partially-quoted argument is tokenized incorrectly, causing git to receive malformed arguments and print its usage message. Build the range string in a variable first so it is passed to git as a single token. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
06d2921a5b |
Add workflow to warn on fork PRs touching infrastructure (#356)
Adds a pull_request_target workflow that posts an advisory comment when a fork PR is opened that modifies eng/ or .github/ files, suggesting the author recreate the PR from an upstream branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
b966ac4dec |
Add structural external dependency checker (#355)
* Add structural external dependency checker for skills, agents, and plugins Add ExternalDependencyChecker to the skill-validator that produces advisory warnings for structural dependencies that need human review: - Script files in skill scripts/ directories (.ps1, .sh, .py, .bat, .cmd, .bash) - INVOKES patterns referencing external scripts in skill descriptions - #tool: references to non-built-in tools in skill/agent content - Non-built-in tools declared in agent frontmatter tools array - MCP server declarations in plugin.json files URL scanning is intentionally excluded — it is handled by the reference scanner (eng/reference-scanner/scan.ps1) with its known-domains.txt allowlist. Also adds Tools property to AgentInfo record and RawAgentFrontmatter to support parsing the tools array from agent YAML frontmatter. Based on dotnet/skills#337 by ManishJayaswal, trimmed to structural checks only (URL scanning superseded by #336). Co-authored-by: ManishJayaswal <ManishJayaswal@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add allowlist support for external dependency checker Introduce an allowlist file (eng/skill-validator/allowed-external-deps.txt) so known external dependencies can be pre-approved. Each finding generates a deterministic key (type:name:detail) that is checked against the list. Allowed items are silently suppressed; unrecognized entries produce warnings. The allowlist is pre-seeded with the 5 findings currently on main so the checker runs clean out of the box. Over time, entries can be removed as issues are resolved, and checks can be promoted to hard errors. Adds 7 tests covering allowlist loading and filtering. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: naming, dedup, test robustness - Rename 'errors' variables to 'findings' in all Check* methods to reflect advisory (warning) semantics, not hard failures. - Rename test methods from FlagsError to FlagsWarning, NoError to NoWarning for consistency. - De-duplicate #tool: reference warnings so repeated mentions of the same tool in a skill/agent produce only one finding. - Use GUID-based temp path instead of hardcoded '/nonexistent/path.txt' in LoadAllowList_MissingFile test. - Add 2 dedup tests (skill and agent) bringing total to 379. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix comments: tool refs scan full content, not just body Update comments in CheckSkill and CheckAgent to accurately reflect that #tool: references are scanned in the full markdown content (including frontmatter), not just the body. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: ManishJayaswal <ManishJayaswal@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
5a199da6f6 | Fix the compilation issues with devops agentic workflows (#353) | ||
|
Dan Moseley
|
3f510a8c55 |
Fix minor review feedback on migrate-dotnet9-to-dotnet10 (#180) (#335)
* Fix post-merge review feedback on #180 - sdk-msbuild: Fix incorrect 'use single quotes' guidance for #r/#load directives — the correct fix is to remove the quotes entirely - serialization: Split dual Animal class declaration into separate before/after code blocks so each is valid C# Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-upgrade/skills/migrate-dotnet9-to-dotnet10/references/serialization-networking-dotnet9to10.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
6666c9e5e8 |
[Infra] Add validation with whole plugin loaded (#316)
* Add plugin-aware skill validation with activation detection * Address Copilot review comments (round 4) - DESIGN doc: mark Targeted Skill Activation TODO as implemented - ValidateCommand: fix misleading --plugin-dir comment - Comparator: set PluginScore to null when no plugin runs exist - ComparatorTests: copy missing fields in rehydrated ScenarioComparison - AgentRunner: fix GetPluginClient docstring, route RunAgentCore through it - AgentRunner: stage isolated skill to temp dir for true single-skill isolation - dashboard.js: fix triple chart comment to match actual series names - RunnerTests: update assertions for new isolated skill staging behavior * Remove implementation doc |
||
|
Dan Moseley
|
144b129995 |
Fix reference scanner silently passing when git diff fails (#348)
The PR-mode code path used --depth=1 when fetching the base branch, which prevented git from finding the merge-base for the three-dot diff. The git diff failure was silently swallowed because its stderr output was piped through Where-Object and filtered out, causing the scanner to report success without scanning anything. Fix: - Remove --depth=1 from git fetch (checkout already uses fetch-depth: 0) - Check LASTEXITCODE after both git fetch and git diff - Separate git diff from the Where-Object filter so exit code is captured Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Amaury Levé
|
7afa81f831 | Run tests supports platforms and frameworks (#343) | ||
|
Abhitej John
|
bb974f7360 |
Security Update (#331)
* Security Update Identified and remediated 15 vulnerabilities across the skill-validator runtime, dashboard, and CI/CD pipelines. Changes: - Runtime (AgentRunner.cs): path traversal prevention, deny-by-default permissions, env scrubbing, MCP server hardening - Regex (AssertionEvaluator.cs): ReDoS timeout - Logging (SkillDiscovery.cs): exception logging - Dashboard (dashboard.html, dashboard.js): SRI hash, XSS escaping - CI/CD (5 workflow files): SHA pinning, persist-credentials: false - Tests (RunnerTests.cs): 45+ new security test cases MCP Server Defense-in-Depth (5 layers): 1. Command allowlist: only dotnet, node, npx, python, python3, uvx 2. Path rejection: no /, \, or .. in command names 3. Args validation: per-runtime dangerous flag blocklist 4. Env sanitization: strips PATH, LD_PRELOAD, NODE_OPTIONS, DOTNET_STARTUP_HOOKS, etc. 5. Cwd dropped: custom working directories not passed to SDK Subprocess Isolation: - Sensitive env vars + prefix patterns scrubbed from child processes - 120-second timeout with Kill(entireProcessTree: true) - Deny-by-default permission model Supply Chain: - All GitHub Action references pinned to full commit SHAs - All checkout steps set persist-credentials: false - Chart.js CDN pinned to v4.4.7 with SRI integrity hash * Update .github/workflows/skill-validator.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update eng/skill-validator/src/Services/AgentRunner.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update eng/skill-validator/src/Services/AgentRunner.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update tests to use fullCommandText field name Update ExtractsCommandProperty and PrefersPathOverCommand tests to use the fullCommandText property instead of command, matching the updated permission request schema in AgentRunner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add 'dnx' to allowed MCP commands * Address PR review feedback: security hardening fixes - CheckPermission: normalize paths with TrimEndingDirectorySeparator to prevent double-separator false denials when dirs have trailing slash - SanitizeMcpArgs: reject prefix forms (-econsole.log) and --flag=value variants that bypass exact-match blocklist checks - Setup file path traversal: normalize canonicalWorkDir and canonicalSkillPath to avoid double-separator in StartsWith checks - IsAllowedMcpCommand: use GetFileName instead of GetFileNameWithoutExtension to block dotnet.bat/node.cmd; only strip .exe on Windows Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * prune * fix * Fix concurrency setting * Add binlog generation and upload to skill-validator workflow Produce binary logs for build, test, and pack steps using /bl: flag. Upload binlogs as artifacts (with if: always()) for build diagnostics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve CheckPermission diagnostics and fix AOT build error Log allowed directories alongside denied paths to help diagnose permission mismatches. Replace JsonSerializer.Serialize with manual string building to fix IL3050 AOT compilation error. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Extract fileName from permission requests Some permission requests use 'fileName' instead of 'path'. Handle this property to avoid false denials for file-level operations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * nownar nu1903 temporarily * Log config dir in verbose mode alongside work dir Include skilled/baseline label for consistency with the work dir log. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add skilled/baseline label to permission denial logs Helps identify which run type triggered the denial when debugging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix empty runLabel brackets and add labels to judge permission checks Conditionally format the label suffix to avoid empty () when no runLabel is supplied. Pass 'judge' and 'pairwise-judge' labels from the respective judge call sites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Allow pairwise judge to read from skilled work directory The pairwise judge compares baseline and skilled runs in one session, but previously only allowed reads from the baseline work directory. Add additionalAllowedDirs parameter to CheckPermission and pass the skilled work directory through PairwiseJudgeOptions.SkilledWorkDir. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix tests to match allow-by-default for unrecognized permission requests CheckPermission was changed to allow requests with no extractable path/command (allow-by-default), but four tests still expected the old deny-by-default behavior. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove NoWarn * Pin discover job checkout action to full commit SHA Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> |
||
|
Dan Moseley
|
64464e87e3 |
Add external reference scanner for skill/agent content (#336)
* Add external reference scanner for skill/agent content - PowerShell scanner (eng/reference-scanner/scan.ps1) checks for: - http:// URLs (blocked except localhost and XML namespace URIs) - External <script> tags without SRI integrity attribute - Pipe-to-shell patterns (except dotnet install scripts) - URLs referencing domains not in .github/known-domains.txt - CI workflow runs on PRs and pushes touching skill/agent paths - CONTRIBUTING.md updated with external references policy - README.md: fixed http:// to https:// for agentskills.io Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move known-domains.txt to eng/reference-scanner/, fix comments, add ollama.com - Moved from .github/ to eng/reference-scanner/ (co-located with scanner) - Updated header comments: errors not informational notices, removed misleading org-scoping language - Added ollama.com (pre-existing reference in dotnet-ai skill) - Updated all references in scan.ps1, CI workflow, CONTRIBUTING.md Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback - Add ? and # as valid URL terminators in path-scoped domain matching - Strip # from host extraction in Test-KnownDomain and Test-HttpNotHttps - Make script/SRI regex patterns case-insensitive - Emit error finding on file read failure instead of silent skip - Fix help comment (two levels up, not three) - Unify CONTRIBUTING.md error language (all checks are CI-blocking) - Add eng/reference-scanner/** to workflow path triggers Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add cdn.jsdelivr.net to known-domains (used by dashboard after #331 adds SRI) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove cdn.jsdelivr.net from known-domains jsdelivr serves arbitrary npm packages -- keeping it off the allowlist means any use requires both SRI and reviewer approval (two gates). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Skip domain check for scripts with SRI, fix single-line file handling Scripts with integrity attributes are secured by hash -- the domain is irrelevant. This means cdn.jsdelivr.net does not need to be in known-domains.txt as long as SRI is present. Also wrap Get-Content in @() to handle single-line files correctly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address round-2 review feedback - Multi-line <script> tag detection: join file content so SRI attributes spanning lines are correctly detected - Safe relative path: use GetRelativePath() with fallback instead of bare Substring that throws for out-of-repo files - Remove dead notice plumbing: all findings are errors now Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Scan only changed files on PRs, full scan on push to main PRs scan only changed .md/.html files so pre-existing issues on main don't block unrelated PRs. Push to main runs the full -All scan. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
676523ca42 | Bump dependency (#345) | ||
|
Viktor Hofer
|
3a9e611f32 |
Convert .claude-plugin/marketplace.json from symlink to regular file (#339)
The symlink mode (120000) caused issues on Windows with git operations like checkout and reset. Convert to a regular file (100644) with identical content. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jason Malinowski
|
59241eabcf | Add the C# LSP configuration to the 'dotnet' plugin (#298) | ||
|
Jeff Schwartz
|
551461a926 |
dotnet-ai-ml (#177)
* moved from private to public fork * add code owner * accomodate feedback * update eval to make more natural language * address eval feedback * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * Update plugins/dotnet-ai/skills/technology-selection/SKILL.md Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> * update supporting documentation of adding a new plugin * update claude-plugin --------- Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Luis Quintanilla <46974588+luisquintanilla@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
3923c078b9 |
Add migrate-dotnet9-to-dotnet10 skill (#180)
* Add migrate-dotnet9-to-dotnet10 skill Adds a new migration skill that guides upgrading .NET 9 projects to .NET 10, covering TFM updates, NuGet package upgrades, and source-breaking changes across the full .NET 10 surface area. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback: fix eval/reference consistency and CODEOWNERS - Fix ExecuteUpdateAsync rubric: use 'regular lambda/delegate instead of expression tree' instead of incorrect 'Func instead of Expression' - Fix config null binding eval prompt: .NET 9 bound null as empty string, not preserving constructor default - Clarify .NET 10 null binding overwrites constructor default in reference - Add @ViktorHofer as CODEOWNER for migrate-dotnet9-to-dotnet10 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Suffix reference filenames with -dotnet9to10 to avoid collisions Rename all reference files under migrate-dotnet9-to-dotnet10/references/ from e.g. efcore.md to efcore-dotnet9to10.md so they don't collide with reference files from other version migration skills. Update all references in SKILL.md to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address BrennanConroy review: ForwardedHeaders KnownIpNetworks example and IApiEndpointMetadata guidance - Show KnownIpNetworks property with UseForwardedHeaders context in example - Add IApiEndpointMetadata explanation for cookie login redirect behavior - Link to aspnetcore#62883 for details on influencing the behavior Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Shorten skill description to fit 1024-char limit Reduce description from 1,370 to 1,017 characters by trimming redundant phrasing while preserving all key search terms. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update CODEOWNERS --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
db33912b98 |
Add migrate-dotnet10-to-dotnet11 skill (#181)
* Add migrate-dotnet10-to-dotnet11 skill (Preview 1 coverage) Adds a new migration skill that guides upgrading .NET 10 projects to .NET 11, covering TFM updates, NuGet package upgrades, and breaking changes across the .NET 11 Preview 1 surface area. Sources: - https://learn.microsoft.com/en-us/dotnet/core/compatibility/11 - https://learn.microsoft.com/en-us/ef/core/what-is-new/ef-core-11.0/breaking-changes - https://learn.microsoft.com/en-us/dotnet/csharp/whats-new/breaking-changes/compiler%20breaking%20changes%20-%20dotnet%2011 - https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotnet-11/overview Skill coverage: - C# 15 compiler: Span safe-context, ref readonly InAttribute, nameof(this.), with() parsing, dynamic &&/|| with interfaces - Core libraries: DeflateStream/GZipStream empty payloads, MemoryStream capacity, TAR checksum validation, ZipArchive.CreateAsync eager loading, Environment.TickCount consistency - EF Core 11: Cosmos DB sync I/O fully removed - Cryptography: DSA removed from macOS - Runtime/JIT: x86-64-v2 minimum, Arm64 LSE requirement on Windows - SDK: Mono launch target removed for .NET Framework apps 6 reference files loaded based on project type + 7 eval scenarios. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add SYSLIB0063 NamedPipeClientStream breaking change to 10→11 skill Add documentation for the NamedPipeClientStream constructor obsoletion (SYSLIB0063) which was merged in dotnet/runtime#120328 for .NET 11 but not yet documented on learn.microsoft.com. The isConnected parameter overload is obsoleted, and projects with TreatWarningsAsErrors will fail to build. Changes: - core-libraries.md: Add Obsoleted APIs section with SYSLIB0063 - SKILL.md: Add to description, Step 1 assessment, Step 3 patterns Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tune skill description and eval prompts for activation - Add explicit migration framing to skill description to differentiate from csharp-scripts skill (SDK was routing to wrong skill) - Add 'Use the migrate-dotnet10-to-dotnet11 skill' to eval prompts, matching real-world usage where users explicitly invoke migration skills Eval results (3 runs each, explicit skill invocation): Haiku: 6/7 activated, best +1.3 (compression/TAR scenario) Sonnet: 7/7 activated, best +0.3 (EF Core, TFM update scenarios) Skill activates reliably now. Quality gains are modest because the skill covers only Preview 1 breaking changes (training data already includes MS Learn docs). Expect larger gains as content grows with later previews. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/migrate-dotnet10-to-dotnet11/references/csharp-compiler.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Split nameof(this.) example into before/after code blocks Separate the breaking and fixed code into distinct blocks to avoid ambiguity about which line to use vs remove. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback: fix code examples and global.json guidance - SKILL.md: Show global.json as a diff edit preserving existing keys (rollForward, other settings) instead of replacing the entire file - csharp-compiler.md: Fix ref readonly lambda example to use a static field instead of a captured local (which won't compile in a lambda) - csharp-compiler.md: Declare missing local variable x in ref readonly local function example - eval.yaml: Fix ref readonly lambda snippet to use a class with a static field, matching the reference file fix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR feedback: add second CODEOWNER, add preview version to EF Core - Add @ViktorHofer as second owner for migrate-dotnet10-to-dotnet11 and migrate-nullable-references paths per CONTRIBUTING.md requirements - Add Preview 1 annotation to EF Core reference file per reviewer feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Suffix reference filenames with -dotnet10to11 to avoid collisions Rename e.g. efcore.md to efcore-dotnet10to11.md so names won't collide with reference files from other version migration skills. Update all paths in SKILL.md to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove explicit skill name from eval prompts to reduce overfitting The eval prompts were explicitly saying 'Use the migrate-dotnet10-to-dotnet11 skill' which biases baseline runs — the agent wastes time searching for a nonexistent skill, guaranteeing worse performance. Rephrased all 7 prompts to describe the migration task naturally without naming the skill. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: fix with/when label, add exit_success assertions - Rename 'with()' to 'when' in switch-expression-arm parsing label (SKILL.md and csharp-compiler reference) to match the actual breaking change - Add exit_success assertion to all 7 eval scenarios to match repo convention Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move Japanese Calendar breaking change to core-libraries reference The globalization change was in the runtime-jit reference, which is only conditionally loaded for older-hardware deployments. Since this affects all projects regardless of hardware, move it to core-libraries (always loaded) so it is not missed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify workflow note: allow local repo inspection, forbid only external sources The previous wording ("Do not search the filesystem") was too broad — the agent needs to read .csproj, solution, and source files to determine which breaking changes apply. Reword to explicitly allow local repository inspection while forbidding web/external sources for breaking change info. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Shorten skill description to fit 1,024-char limit (762 chars) Remove granular breaking change names (covered in body/references), drop LOADS REFERENCES (implementation detail), remove unsupported ASP.NET Core 11 claim (no reference docs shipped), and tighten DO NOT USE FOR section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update CODEOWNERS to use @dotnet/compat team instead of individual Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move 'when' parsing change from Step 4 to Step 3 The when switch-expression-arm parsing change is a source/compilation breaking change, not a runtime behavioral change. It belongs in Step 3 (source-breaking changes) alongside other C# 15 compiler changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
e9370fe663 |
Add migrate-dotnet8-to-dotnet9 skill (#182)
* Add migrate-dotnet8-to-dotnet9 skill with references and eval scenarios Mirrors the structure and approach of the existing migrate-dotnet9-to-dotnet10 skill. Covers all .NET 8→9 breaking changes organized by technology area: - C# 13 compiler (InlineArray records, iterator safe context, collection expressions) - Core libraries (params span overloads, BinaryFormatter removal, BigInteger, TimeSpan) - SDK/MSBuild (Terminal Logger, version requirements, .NET 7/Standard warnings) - ASP.NET Core (HostBuilder validation, forwarded headers, middleware constructors) - EF Core (pending migrations, explicit transactions, Cosmos DB changes) - Cryptography (X509Certificate2 SYSLIB0057, SafeEvpPKeyHandle) - Serialization/Networking (BinaryFormatter, JsonDocument null, HttpClientFactory) - WinForms/WPF (security analyzers, StatusStrip, PictureBox, XmlNamespaceMaps) - Containers/Interop (zlib removal, CET support) - Deployment/Runtime (env var precedence, FP-to-int saturation, MonoVM deprecated) 12 eval scenarios covering all technology areas with assertions and rubrics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * BinaryFormatter: stop and ask user for replacement strategy BinaryFormatter migration is a major architectural decision affecting data format, backward compatibility, and stored data migration. The skill should present options and let the user decide rather than picking a serializer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Trim SKILL.md to delegate BinaryFormatter detail to reference Keep SKILL.md as concise index; full guidance lives in serialization-networking.md reference. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Trim SKILL.md: delegate fix details to references, remove BinaryFormatter duplicate - Remove BinaryFormatter duplicate from Step 4 (already covered in Step 3) - Delegate inline fix details for EF Core, FP saturation, HttpClientFactory to references - Collapse niche SYSLIB0055/0056 into one-liner pointing to reference - Shorten params span and String.Trim entries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve eval scenarios: setup files, task-oriented prompts, expect_activation - Add setup files (csproj, .cs, Dockerfile, etc.) to 10/12 scenarios so agent works with real code instead of pure knowledge questions - Make prompts task-oriented ('Migrate this app to .NET 9') instead of knowledge-oriented ('What breaks?') - Set expect_activation: false on all scenarios — this is a pure knowledge/ reference skill with no MCP tools, so activation via SkillInvokedEvent is unreliable - Increase timeouts to 420s to avoid penalizing skilled runs that do more work Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refocus eval scenarios on Opus blind spots Replace easy scenarios (BinaryFormatter/X509) where Opus always ties with harder ones targeting areas where the skill adds measurable value: - Empty env var behavior on Unix (null vs empty string) - ZipArchiveEntry UTF-8 bit flag mechanism - FromKeyedServices no-fallback behavior - Cosmos DB id format change (discriminator no longer embedded) - Env var precedence reversal with concrete GC/ThreadPool impact Keep scenarios where skill consistently helps: Cosmos DB id format, container zlib removal, CI pipeline net7.0 warnings, WinForms PictureBox. Increase timeouts to 420s to avoid penalizing skilled runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add forward-chaining guidance to migrate-dotnet8-to-dotnet9 skill Tell the agent about the migrate-dotnet9-to-dotnet10 skill so it knows to continue the chain after completing the 8-to-9 migration. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix EF Core eval scenarios to start with 8.0.0 package versions The eval scenarios claim to test a .NET 8 to .NET 9 migration but the starting csproj files already reference EF Core 9.0.0 packages. This reduces test signal since the package upgrade step is already done. Change all EF Core package versions to 8.0.0 in scenarios 4, 5, and 10. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix InlineArray eval to actually exceed 1 MiB size limit The rubric expects the model to warn about the InlineArray 1 MiB runtime size limit, but the example was only 8 KiB (InlineArray(1024) with byte element). Increase to 1,048,577 bytes so the breaking change is concretely applicable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix non-compiling ConfigurePrimaryHttpMessageHandler in eval The eval used a non-existent overload of ConfigurePrimaryHttpMessageHandler with a (handler, _) => {} lambda that doesn't return a handler. Replace with ConfigureHttpMessageHandlerBuilder which correctly casts PrimaryHandler to HttpClientHandler -- the pattern that actually breaks when the default switches to SocketsHttpHandler in .NET 9. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix non-compiling HttpClientFactory example in reference doc The breaking change example used a non-existent overload of ConfigurePrimaryHttpMessageHandler. Replace with ConfigureHttpMessageHandlerBuilder which is the actual API that casts PrimaryHandler and would break when the default changes to SocketsHttpHandler. Also fix Option 2 to use the same correct API. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unsafe KnownProxies.Clear() guidance from aspnet-core reference Clearing KnownProxies and KnownNetworks accepts forwarded headers from any source, enabling attackers to spoof X-Forwarded-For/Proto/Host headers. Replace with a warning against this pattern and steer toward registering only trusted proxy addresses. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove unsafe RedactLoggedHeaders guidance from reference doc Disabling header redaction globally with RedactLoggedHeaders(_ => false) logs Authorization tokens, cookies, and other credentials in cleartext. Replace with guidance to whitelist only specific non-sensitive headers and add an explicit warning against disabling redaction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove insecure BinaryFormatter re-enablement option from guidance Replace the unsupported NuGet package option with a clear security warning that BinaryFormatter deserialization enables remote code execution and should not be re-enabled. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add security warning against suppressing WFO1000 analyzer WFO1000 guards against insecure deserialization of WinForms control properties. Replace the suppression option with a warning explaining the security risk and recommend the attribute-based fix instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add security warning to CET opt-out guidance Warn that disabling CET removes hardware-enforced control-flow integrity (protection against ROP/JOP exploits) and should only be done when a specific native library is confirmed incompatible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Label EF Core warning suppressions as temporary workarounds Add context that suppressing PendingModelChangesWarning risks silent schema drift and that MigrationsUserTransactionWarning suppression has transaction safety implications. Both are temporary workarounds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Prioritize behavioral changes and add global.json rollForward guidance Mark behavioral changes that can cause runtime exceptions with a warning indicator and move them to the top of the list so agents address them first. Add rollForward policy guidance to the global.json update step to prevent SDK resolution issues. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Split BinaryFormatter code snippet into separate serialize/deserialize blocks The combined snippet used obj before declaration, which was misleading. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix SYSLIB heading to include full range SYSLIB0054-SYSLIB0057 Previously listed only SYSLIB0054 and SYSLIB0057, omitting 0055 and 0056. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix eval rubric to match setup code: Migrate/MigrateAsync Setup code uses synchronous Migrate(), not MigrateAsync(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use distinct variable names in X509CertificateLoader examples Duplicate 'var cert' declarations would cause compilation errors if pasted as-is. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove misleading 'backported from .NET 8' from ForwardedHeaders section The claim is confusing in an 8-to-9 migration guide context. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Clarify GetXmlNamespaceMaps InvalidCastException explanation Previous wording was logically inconsistent about the direction of the type change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet/skills/migrate-dotnet8-to-dotnet9/references/efcore.md Co-authored-by: Andriy Svyryd <AndriySvyryd@users.noreply.github.com> * Fix ReadOnlySpan<char> String.Trim example and eval scenario The code example incorrectly used ReadOnlySpan<char> as a static field, which is impossible (ref struct). The eval scenario used a local ReadOnlySpan<char> variable which actually compiles fine on .NET 9 GA, so it didn't demonstrate the breaking change. Updated both to correctly show the RC2-to-GA overload removal issue. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add -dotnet8to9 suffix to reference filenames to avoid cross-skill collisions Other version migration skills (e.g., 9-to-10) have similarly named reference files. Adding the suffix disambiguates them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Shorten skill description to fit 1024-char limit Remove LOADS REFERENCES section (implementation detail already covered by the reference loading table in the body) and tighten wording. All USE FOR activation keywords and DO NOT USE FOR rejection keywords preserved. Description is now 1006 characters. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review feedback - CODEOWNERS: Replace @ViktorHofer with @jeffschwMSFT for migrate-dotnet8-to-dotnet9 - SKILL.md: Shorten description from 1,400 to 671 chars (max 1,024) - sdk-msbuild-dotnet8to9.md: Clarify VS 17.11 SDK targeting phrasing - core-libraries-dotnet8to9.md: Fix cross-reference to serialization-networking-dotnet8to9.md - serialization-networking-dotnet8to9.md: Replace 'whitelist' with 'allowlist' - winforms-wpf-dotnet8to9.md: Rename heading to cover both Get/SetXmlNamespaceMaps - eval.yaml: Use ReadOnlySpan<char> to properly demonstrate String.Trim RC2 breakage Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * CODEOWNERS: Use solo @danmoseley (consistent with migrate-nullable-references) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update CODEOWNERS to use @dotnet/compat team instead of individual Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: fix factual errors in references and eval - Fix URL with %20 spaces to use hyphens (csharp-compiler ref) - Fix DOTNET_GCConcurrent -> DOTNET_gcConcurrent casing (eval.yaml) - Remove invalid [StructLayout] combined with [InlineArray] (eval.yaml) - Move JSON comment outside code fence (deployment-runtime ref) - Remove confusing preview-era sentence (winforms-wpf ref) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert URL change: %20 encoding is correct for this MS Learn page The original URL with percent-encoded spaces is the actual working URL; the hyphenated version returns 404. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve EF Core reference doc to help failing eval scenarios - Promote HasIndex from Low to Medium Impact with crash detail and removal guidance (Scenario 5 missed this causing quality regression) - Expand id format change section with data migration implications and HasRootDiscriminatorInJsonId mitigation (Scenario 5 rubric) - Add concrete DateTime.UtcNow in HasData before/after example (Scenario 4 missed connecting HasData to pending changes exception) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add explicit HasData search action for pending model changes The agent reads the reference doc but doesn't connect DateTime.UtcNow in HasData to the pending changes exception. Make the SKILL.md workflow step prescriptive: search for non-deterministic values and replace them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Andriy Svyryd <AndriySvyryd@users.noreply.github.com> |
||
|
Korolev Dmitry
|
a2be8cce61 |
feat: noise-test evaluation option (#310)
* use BPE tokenizer * noise-test impl * address metrics calculation * address copilot comments * more separation & refactor * cleanup safely * nit |
||
|
Elinor Fung
|
ba56c936b3 |
Add binder tracing to dotnet-trace-collect skill (#250)
* Add binder tracing to dotnet-trace-collect skill * Update plugins/dotnet/skills/dotnet-trace-collect/SKILL.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * PR feedback --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Dan Moseley <danmose@microsoft.com> Co-authored-by: Jeff Schwartz <jeffschw@microsoft.com> |
||
|
Jan Krivanek
|
4a4306f563 |
Disable scheduled workflows on forks (#318)
* Disable schedules on forks * Reflect PR comments |
||
|
Viktor Hofer
|
ea6f594295 |
Enable "evaluation-status" as a required status check (#320)
* Enable evaluation as a required status check - Remove path filters from evaluation.yml pull_request trigger so the workflow runs on every PR - Add statuses: write permission and a report-status job that posts a commit status with context 'evaluation' for non-fork PRs - Change evaluation-fork-pr.yml commit status context from 'evaluation (fork PRs)' to 'evaluation' so both workflows share the same status context - The 'evaluation' status can now be configured as a required check in the GitHub ruleset Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: handle discover failure, least-privilege perms - Remove workflow-level statuses: write (job already declares it) - Broaden report-status condition to handle discover failure/cancellation instead of leaving PRs stuck with no status - Exclude fork PRs (discover skipped) to avoid conflicting with evaluation-fork-pr.yml Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Handle fork PRs in evaluation.yml for required status check - Remove fork guard from discover so it runs for all PRs - Add is_fork output to detect fork PRs - Skip run-evaluation for fork PRs (no access to secrets) - Post 'pending' status for fork PRs with skill changes, prompting the author to use /evaluate - Post 'success' for fork PRs with no skill changes - This ensures every PR gets an 'evaluation' commit status, making it viable as a required check without fork PRs getting stuck Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add statuses: write at workflow level (required for job-level grant) Job-level permissions cannot exceed the workflow-level permissions. The report-status job needs statuses: write but was getting read-only because the workflow level didn't include it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Use pull_request_target for fork PR status posting Fork PRs need an 'evaluation' commit status but pull_request triggers have restricted GITHUB_TOKEN permissions for forks. Use pull_request_target in evaluation-fork-pr.yml instead, which runs in the base repo context with full permissions. The new fork-status job: - Triggers on pull_request_target for fork PRs only - Checks if the PR changes any skills (git diff only, no code execution) - Posts 'pending' status if skills changed (needs /evaluate) - Posts 'success' if no skills to evaluate evaluation.yml reverts to the original fork guard on discover (skips fork PRs entirely). report-status excludes skipped discover results so it only posts for non-fork PRs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: fix header comment, use shallow checkout - Correct security model comment: pull_request_target runs from the base branch, not necessarily the default branch - Use default shallow checkout (depth 1) instead of fetch-depth: 0 - Diff against HEAD instead of base SHA since checkout is the base Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Simplify evaluation.yml: use plain check-run job instead of commit status API Replace report-status job (which called gh api to post commit statuses) with a plain evaluation job whose check run can be required directly. Removes the need for statuses: write permission in this workflow. Fork PRs are handled by evaluation-fork-pr.yml which still uses commit statuses (necessary because issue_comment check runs are not associated with the PR head SHA). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix fork-status: pin checkout to base SHA, use merge-base for diff - Pin actions/checkout to the PR base SHA to avoid checking out the merge ref which includes untrusted fork content - Use explicit base/head SHAs with merge-base for reliable diff, consistent with the non-fork workflow's discovery logic Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Rename evaluation job and context to evaluation-status Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: least-privilege permissions, full clone for merge-base - Add permissions: {} to evaluation-status job (evaluation.yml) - Use fetch-depth: 0 for fork-status checkout so merge-base works - Remove --depth=1 from PR head fetch - Keep statuses: write at workflow level (needed by reusable workflow) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix header comment: evaluation → evaluation-status Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
0f72bde887 | Update STJ in samples to avoid security vulnerability reports (#313) | ||
|
dependabot[bot]
|
f8d6e3170f |
Bump the microsoft-and-system group with 2 updates (#324)
Bumps Microsoft.Extensions.FileSystemGlobbing from 10.0.3 to 10.0.4 Bumps System.CommandLine from 2.0.3 to 2.0.4 --- updated-dependencies: - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: Microsoft.Extensions.FileSystemGlobbing dependency-version: 10.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system - dependency-name: System.CommandLine dependency-version: 2.0.4 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: microsoft-and-system ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com> |
||
|
dependabot[bot]
|
b482146c0a |
Bump the all-other-nuget group with 1 update (#325)
* Bump the all-other-nuget group with 1 update Bumps GitHub.Copilot.SDK from 0.1.30 to 0.1.32 --- updated-dependencies: - dependency-name: GitHub.Copilot.SDK dependency-version: 0.1.32 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-other-nuget - dependency-name: GitHub.Copilot.SDK dependency-version: 0.1.32 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: all-other-nuget ... Signed-off-by: dependabot[bot] <support@github.com> * React to PermissionRequestResultKind breaking change in GitHub.Copilot.SDK 0.1.32 (#326) * Initial plan * Fix PermissionRequestResultKind breaking change in SDK 0.1.32 Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> |
||
|
Viktor Hofer
|
7be01578bd | Rename status job to skill-validator-status | ||
|
Viktor Hofer
|
8fecbba151 |
Rename status job to skill-validator-status (#322)
* Rename status job to skill-validator-status * Update .github/workflows/skill-validator.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
7bd004ee9f |
Enable skill-validator as a required status check (#321)
* Enable skill-validator as a required status check - Remove path filters from pull_request trigger - Add discover job to skip the expensive 5-platform build matrix when no skill-validator files changed - Add report-status job that posts commit status 'skill-validator': - No relevant changes → success - Build passed → success - Build failed → failure - Schedule and workflow_dispatch triggers unchanged (discover is skipped, build runs unconditionally) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Handle fork PRs with pull_request_target for status posting Fork PRs have restricted GITHUB_TOKEN and can't post commit statuses via pull_request trigger. Add pull_request_target trigger with a fork-status job that posts success for fork PRs (they should never change skill-validator). - Add pull_request_target trigger - Gate build to skip for pull_request_target events - Gate report-status to non-fork PRs only - Add fork-status job for fork PRs (always posts success) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Enable skill-validator as a required check - Remove path filters from pull_request trigger so the workflow runs on every PR - Add discover job to skip the expensive 5-platform build matrix when no skill-validator files changed - Add status job that can be required in the GitHub ruleset: succeeds when no relevant changes or when the build passes, fails otherwise - No commit status API calls or pull_request_target needed — just a plain job whose check run is required Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
c735d84f9d |
Improve CODEOWNERS validation: require sufficient owners and enable as required check (#314)
* Improve CODEOWNERS validation workflow - Remove path filters from pull_request trigger so it can be a required check - Add owner sufficiency validation: each skill/test folder must have either 2+ individual owners or 1+ team in CODEOWNERS - Update issue creation to report both missing entries and insufficient owners - Update failure step to trigger on either validation failure Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update CODEOWNERS to add additional code owners Added @agocke and @hoyosjs as code owners for migrate-nullable-references and dotnet-trace-collect skills. * Update CODEOWNERS for dotnet-trace-collect --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Korolev Dmitry
|
484d5d2c1f |
tokenization: use BPE instead of chars\4 approximation (#309)
* use BPE tokenizer * address PR comments 1 * elaborate on model * nit |
||
|
Viktor Hofer
|
140fb601b0 |
Replace marketplace.json symlink with file copy (#312)
* Replace marketplace.json symlink with file copy On Windows without developer mode, git checks out symlinks as plain text files containing the target path. This caused Claude Code to fail parsing .claude-plugin/marketplace.json since it contained the string '../.github/plugin/marketplace.json' instead of valid JSON. Replace the symlink with an actual copy of the file and update CONTRIBUTING.md to instruct contributors to update both locations. Fixes #300 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
70ea8cf225 |
Include error details in evaluation failure PR comments (#297)
* Include error details in evaluation failure PR comments When evaluation fails due to validation errors (e.g., description too long), the PR comment previously only said 'Evaluation failed. View workflow run', requiring a click-through to discover the actual error. Two fixes: - Reporter.cs: GenerateMarkdownSummary() now renders an Errors section for verdicts with FailureKind set, so the marker comment shows what went wrong. - evaluation-fork-pr.yml: The report-status job now downloads result artifacts and extracts error reasons from results.json, including them directly in the failure comment posted to the PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * TEST: inflate build-perf-diagnostics description to trigger validation error Temporarily makes the description 1221 chars (limit is 1024) to verify that the new error reporting shows the validation failure in the PR comment. This commit should be reverted after verifying the error display. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Tighten error filter to pre-eval failures and add tests Address review feedback: - Restrict the Errors section in markdown summary to verdicts with no scenarios (pre-eval failures like spec_conformance_failure), avoiding duplication with threshold/regression failures already shown in the table. - Add three tests: errors section appears for pre-eval failures, is omitted when all verdicts pass, and is omitted for failures that have scenarios. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix error details in evaluation-run.yml post-completion-comment The failure comment was posted by evaluation-run.yml's post-completion-comment job (shared by both same-repo and fork PR workflows), not just by the fork PR workflow. Apply the same artifact download + error extraction to this job so error details appear for same-repo PRs too. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Promote eval prompt skill-name mention from warning to error When an eval scenario prompt explicitly references the skill name, it biases baseline runs and forces activation instead of testing organic discovery. Move the check from warnings to errors so it blocks evaluation via the existing profile.Errors fail-fast path. No existing evals on main have this pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Sanitize @ in error output to prevent mention injection Neutralize @ characters with a zero-width space in all three locations where error text from PR-controlled content is rendered into PR comments: Reporter.cs, evaluation-run.yml, and evaluation-fork-pr.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Distinguish errors vs warnings in evaluation failure comments Pre-eval structural errors (spec conformance, missing eval) get a red X icon, while eval verdict failures (below threshold, completion regression) get a warning icon. This makes it clear which failures are blocking issues vs expected score variance. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Only show pre-eval errors in failure comment, link to results table The failure comment no longer repeats verdict warnings (below threshold, completion regression) that are already visible in the results table. It only shows pre-eval structural errors (spec conformance failures) and links to the results table for full details. Three cases: - Pre-eval errors: show them + link to results table - No errors but failed (verdict): just link to results table - No results table: fallback with workflow run link Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove duplicate failure comment from fork workflow The reusable evaluation-run.yml already posts failure comments when pr-number is provided. The fork workflow was duplicating this by posting its own failure comment, resulting in two comments per failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Revert "TEST: inflate build-perf-diagnostics description to trigger validation error" This reverts commit |
||
|
Jan Krivanek
|
b93d6c7d15 | Prevent rep health description duplication (#306) | ||
|
Viktor Hofer
|
b6bc33c926 |
Purge dashboard entries older than 14 days (#304)
* Purge dashboard entries older than 14 days When updating plugin JSON files, filter out Quality and Efficiency entries whose date is older than 14 days. If all entries are purged, delete the JSON file so it is excluded from the components.json manifest and no longer appears in the dashboard. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move stale file cleanup to workflow level Address review feedback: the delete-if-empty branch in the script was unreachable because fresh entries are always appended before purging. Instead, add a workflow step that scans all existing data files after per-plugin generation and removes any whose entries are entirely older than 14 days. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Move stale file purge logic into the script Add a -PurgeStaleFiles/-DataDir parameter set so the workflow calls the script instead of duplicating the retention logic inline. The retention window is defined once in the script. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Make retention days configurable via workflow env var Add -RetentionDays parameter to the script (mandatory for -PurgeStaleFiles, optional for generate mode) and define DASHBOARD_RETENTION_DAYS env var at the workflow level so the value is controlled in one place. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review feedback: fix comment and add entries guard - Fix workflow comment to accurately describe the purge behavior - Guard against missing 'entries' key in purge mode to avoid silently skipping malformed plugin JSON files Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update eng/dashboard/generate-benchmark-data.ps1 Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update .github/workflows/evaluation.yml Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
Dan Moseley
|
abdd4d03d4 |
Improve evaluation results table readability and add verdict explanations (#299)
* Add verdict footnotes to evaluation results table
The results table showed a Verdict column (pass/fail) based on a weighted
composite score, but only displayed the raw quality delta. When the
composite disagreed with quality (e.g., quality improved but token/time
overhead made the composite negative), the verdict was unexplained.
Add a Notes column with footnote references [1], [2] etc. that link to
explanations below the table. Footnotes are only added when the verdict
disagrees with the quality delta direction, keeping the table clean for
straightforward cases.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* TEST: inflate csharp-scripts description to exercise verdict footnotes
Expands the skill description to trigger token overhead in evaluation,
which should produce a scenario where the verdict footnote feature
activates. **Revert before merge.**
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Address review feedback on BuildVerdictFootnote
- Return null for neutral verdicts (ImprovementScore == 0) and unknown
quality delta (null), avoiding misleading footnotes for the yellow
neutral icon or when the table shows delta as dash.
- Align quality direction threshold with the F1 rounding used in the
delta column so footnotes only fire when the displayed value disagrees
with the verdict.
- Use DefaultWeights.Values instead of hard-coded weight literals to
avoid drift if weights change.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix roundedDelta==0 footnote bug and improve test coverage
- Return null from BuildVerdictFootnote when rounded quality delta is zero,
preventing misleading 'offset quality gain' text when the table shows 0.0
- Make test ImprovementScore values consistent with breakdown * DefaultWeights
- Add test for 'quality dropped but composite positive' branch
- Add test for roundedDelta==0 edge case
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Improve results table: collapse quality columns and clarify footnotes
- Merge Baseline, With Skill, and delta columns into a single Quality
column showing 'base -> skill' with the better score bolded and an
arrow indicator, reducing table width for narrow viewports
- Reword footnotes from cryptic 'composite -X% -- factors offset
quality gain' to readable 'Quality improved but weighted score is
-X% due to: factors'
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Show raw metric values in verdict footnotes
Footnotes now include the actual baseline and skill values for each
contributing metric, e.g.:
tokens (1000 -> 8000), tool calls (5 -> 15), time (2.1s -> 8.5s)
This lets readers immediately see the magnitude of each factor without
digging into workflow logs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use emoji arrows, align footnote gating with Quality column
- Replace tiny unicode arrows with emoji (📈/📉) in Quality column
- Use raw quality delta for footnote direction instead of rounded delta,
matching FormatQualityCell which compares unrounded scores
- Rename test to MarkdownTable_NoFootnoteWhenQualityUnchanged with
truly equal scores instead of near-zero delta
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Make footnote references bidirectional hyperlinks
Table [1] links jump to the footnote explanation at the bottom,
and footnote [1] links jump back to the table row.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Restore dash assertion in ShowsDashWhenNoOverfitting test
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix flaky ReturnsEmptyForNonSkillDirectory test
Use a fresh empty temp directory instead of Path.GetTempPath() which
can contain stray SKILL.md files left by other tests or processes.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Use <strong> instead of ** for bold inside HTML anchor tags
GitHub markdown does not render ** inside raw HTML elements.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Merge Notes into Verdict column and add footnote for quality-unchanged cases
- Remove Notes column (7 -> 6 columns) to reduce table width; footnote
reference now appears next to the verdict icon
- Add footnote when quality is unchanged but verdict is negative, with
text: 'Quality unchanged but weighted score is -X% due to: ...'
- Update tests accordingly
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Revert "TEST: inflate csharp-scripts description to exercise verdict footnotes"
This reverts commit
|
||
|
Gerald Versluis
|
ebe71e2f06 |
Add dotnet-maui plugin with dotnet-maui-doctor skill (#199)
* Add dotnet-maui-doctor skill and refs Add a new .agents/skills/dotnet-maui-doctor skill bundle. Includes SKILL.md describing an autonomous workflow to detect and remediate .NET MAUI environment issues and platform-specific reference docs for installation commands, platform requirements, troubleshooting, Microsoft OpenJDK guidance, and a WorkloadDependencies discovery guide that pulls authoritative versions from NuGet. Documents validation/remediation steps for .NET SDK, workloads, JDK, Android SDK, Xcode, and Windows SDK (macOS/Windows/Linux) and includes a temporary recommendation to prefer Microsoft OpenJDK (JDK 21) until manifests are updated. * MAUI doesn't quite yet support dotnet run * Add dotnet-maui plugin and dotnet-maui-doctor skill Introduce the dotnet-maui plugin and move the dotnet-maui-doctor skill under plugins/ layout. Update marketplace, CODEOWNERS, and README accordingly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Add comprehensive eval scenarios for dotnet-maui-doctor 10 scenarios covering: macOS/Linux/Windows setup, JDK vendor enforcement, workload update/repair guardrails, dynamic version discovery, JAVA_HOME misconception, Android SDK diagnosis, out-of-scope decline, and stale workload remediation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Refine eval scenarios: remove weak scenarios, fix overfitting Remove 'Resist hardcoding' (penalizes skill — baseline gives more useful direct answer) and 'Decline runtime bug' (skill makes agent refuse to help). Rework Android SDK scenario to test CI-oriented package discovery where baseline lacks knowledge. Replace vocabulary-level NuGet/API assertions with outcome-focused rubric items across all scenarios. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Remove hardcoded JDK version workarounds in favor of dynamic discovery Replace temporary JDK 21 workaround notes with guidance to use the version from WorkloadDependencies.json. Replace hardcoded JDK paths in troubleshooting.md with {VERSION} placeholders. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Improve eval scenarios: fix JAVA_HOME timeout, macOS tie, reduce overfitting - JAVA_HOME: rewrite prompt to request conceptual answer (prevents tool timeout) - macOS Xcode: add installation source guidance, remove equal-penalty assertion - Remove irrelevant workload update/repair assertions from 3 off-topic scenarios - Keep workload assertions in 5 scenarios where workload management is the topic Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reframe JAVA_HOME scenario as prevent-harmful-action pattern Rewrite prompt from conceptual yes/no question (which the agent answered from wrong general knowledge without activating the skill) to a scenario where the user is about to apply incorrect Stack Overflow advice. This mirrors the successful workload guardrail pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Reduce overfitting: rephrase assertions and rubric to be outcome-focused - Replace literal '--version' assertions with broader 'version|pin' pattern - Rephrase Xcode rubric from skill-specific source to version control risk - Rephrase version discovery rubric from 'NuGet API' to outcome-focused - Rephrase workload install rubric from '--version flag' to 'pinned version' Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Inline critical JAVA_HOME auto-detection guidance into SKILL.md The JAVA_HOME eval scenario scored poorly because the agent couldn't read the reference file (references/microsoft-openjdk.md) and fell back on general knowledge, which incorrectly says JAVA_HOME must be set. Changes: - Task 5: Add inline JAVA_HOME guidance with decision table showing that JAVA_HOME is NOT required and MAUI auto-detects JDK installations - Common Pitfalls: Add 'Unnecessary JAVA_HOME' entry with unset commands This ensures the agent knows JAVA_HOME is unnecessary even when reference files can't be loaded. File stays at 210 lines (limit: 500). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Inline JAVA_HOME auto-detection knowledge into SKILL.md Add JAVA_HOME guidance directly to Task 5, Task 9, and Common Pitfalls so the agent knows JAVA_HOME is not required even when reference files can't be read. Based on Redth's original microsoft-openjdk.md reference. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address code review feedback from Copilot - Fix cmd.exe syntax in installation-commands.md (use PowerShell $env:) - Replace hardcoded simulator device/runtime with placeholders - Add System.IO.Compression.FileSystem assembly load for PS 5.1 compat - Clarify workload reinstall example is macOS-specific (omit ios on Linux) - Replace hardcoded android-35 with $API_LEVEL placeholder Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address jonathanpeppers review feedback - Add Apple Developer login/2FA note for Xcode downloads - Fix Windows SDK detection: extract KitsRoot10 and version subkeys instead of dumping full registry (was outputting noise) - Replace manual NuGet API approach with dotnet workload search version CLI command for discovering workload versions - Reframe JAVA_HOME guidance: report as anomaly rather than prescribing unset/set commands (per jonathanpeppers recommendation) - Use $env:ProgramFiles instead of hardcoded C:\Program Files paths in PowerShell commands - Update SKILL.md to match: softer JAVA_HOME guidance, improved Windows SDK detection command Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Redth review feedback - Remove Windows SDK detection commands — installed automatically via MAUI workload/VS Installer (per Redth recommendation) - Use dotnet workload search version --format json --take 1 for workload version discovery instead of NuGet search API (per both Redth and jonathanpeppers) - Update workload-dependencies-discovery.md Step 2 to match - Expand Android SDK detection with known filesystem paths instead of relying solely on environment variables Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align eval rubric with reviewer feedback to reduce overfitting Replace technique-specific rubric items with outcome-focused criteria: - Replace PowerShell API query check with dotnet CLI discovery check - Replace manifest extraction check with workload-version-based discovery - Replace prescriptive version discovery with CLI/docs verification - Convert workload update/repair regex assertions to rubric criteria - Remove Invoke-RestMethod assertion (no longer using API queries) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Align JAVA_HOME guidance with reviewer consensus, add Xcode download expectations - Consistently use ⚠️ anomaly approach for non-Microsoft JAVA_HOME across microsoft-openjdk.md and troubleshooting.md (matching SKILL.md) - Remove contradictory Note in microsoft-openjdk.md that conflicted with table - Reframe unset commands as user's choice, not automatic action - Add Xcode download size (~12GB) and time expectations per Redth's feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update plugins/dotnet-maui/skills/dotnet-maui-doctor/references/workload-dependencies-discovery.md Co-authored-by: Jonathan Peppers <jonathan.peppers@gmail.com> * Update JDK example to version 21 per jonathanpeppers feedback Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: redth <jondick@gmail.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Jonathan Peppers <jonathan.peppers@gmail.com> |
||
|
Jan Krivanek
|
d709e0b55a | Add checking for orphaned skills/plugins (#291) | ||
|
Jan Krivanek
|
a9335d4116 |
Workaround gh-aw concurrency group issue (#290)
* Workaround gh-aw concurrency group issue * Clarify concurrency bug comment per code review Rephrase comment to make clear the bug causes mutual cancellation, not a 2-run limit. The 2-run cap is the workaround, not the bug. Co-authored-by: Copilot <copilot@github.com> --------- Co-authored-by: Copilot <copilot@github.com> |
||
|
Viktor Hofer
|
a3f424de82 | Add missing plugins to marketplace.json (#287) | ||
|
Viktor Hofer
|
1fa63e60e6 |
Move completion comment into shared evaluation-run workflow (#286)
Move the 'Post completion comment' step from evaluation-fork-pr.yml into evaluation-run.yml as a dedicated post-completion-comment job, so both evaluation.yml and evaluation-fork-pr.yml share the same logic. The fork-pr workflow retains a slim step that only handles the skipped case (no entries / discover failed) since evaluation-run.yml isn't invoked then. Also extract the '<!-- skill-validator-results -->' marker into a workflow- level PR_COMMENT_MARKER env var to eliminate duplication. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
7884803830 |
Fix infra change detection to use merge-base instead of base tip (#285)
The evaluation workflow was using a two-point diff (base_sha..head_sha) to detect changed files, which includes files changed on main that the PR branch hasn't merged yet. This caused false 'Infrastructure changes detected' triggers when evaluation workflow files were modified on main after a PR branch diverged. Switch to computing the merge-base first, so only changes actually introduced by the PR are considered. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
dcdad225a9 |
Fix 429 rate limiting in evaluation pipeline (#284)
* Fix 429 rate limiting in evaluation pipeline - P0: Add retry with exponential backoff to ListModelsAsync() in ValidateCommand using the existing RetryHelper (3 retries, 2s base delay, 60s budget) - P1: Reduce scheduled-run parallelism to match infra-change settings (2/3/3) to avoid overloading the API when all 5 plugins run in parallel - P2: Assign Copilot tokens deterministically by matrix job index instead of randomly, eliminating ~96% token collision probability - P3: Add retry with backoff to AgentRunner.RunAgent() for mid-eval 429s (2 retries, 5s base delay, scenario timeout + 60s budget) Fixes dotnet/skills#168 * Delete docs/fix-429-rate-limiting.md |
||
|
Viktor Hofer
|
1ae3b61830 |
Add Skill-Validator & Evaluation workflow guidelines (#282)
* Add Skill-Validator & Evaluation workflow guidelines Added guidelines for the Skill-Validator and evaluation workflow, emphasizing security measures for changes made from non-forked branches. * Update CONTRIBUTING.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update CONTRIBUTING.md --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |