mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Jan Krivanek
14ee3c90f9
* Shard plugin evaluation via executionShard tag in eval.yaml MCP-heavy plugins like dotnet-msbuild reliably exhaust the GitHub-hosted runner host after ~87 min of cumulative MCP traffic (Copilot CLI plus dotnet dnx Microsoft.AITools.BinlogMcp child processes), which has produced no dashboard data for dotnet-msbuild scheduled runs since 2026-05-20. Cutting per-runner concurrency alone was insufficient: at the default 18-way concurrency the runner OOMs in 17-30 min; at 3-way it survives longer but still hits a host-level termination at ~87 min. Two mitigations, both required: 1. Sharding (this change). The discover step reads an optional top-level 'executionShard:' string from each tests/<plugin>/<skill>/eval.yaml and groups skills by tag, emitting one matrix entry per shard so each shard runs on a fresh runner. Untagged skills fall into a 'default' bucket. Plugins where all skills resolve to a single bucket emit one entry (unchanged behavior). The validator already calls IgnoreUnmatchedProperties so no Models.cs change is needed. Shard names follow '<plugin>--shard-<tag>' so the existing 'skill-validator-results-<plugin>--*' artifact glob keeps working. 2. Per-MCP-plugin concurrency caps (already in place). When plugin.json declares mcpServers, parallelism is forced to 1x3x1 with --judge-timeout 600 to avoid acute OOM within a shard. Also adds a workflow_dispatch 'plugin' input for targeted re-runs of a single plugin, and tags 10 dotnet-msbuild eval.yamls (heavy/medium); the remaining 8 fall into 'default'. Empirically the three shards complete in parallel in ~30-40 min each, well inside the 180-min job timeout, and successfully upload dashboard data. * Address PR review: lenient shard regex and validate dispatch plugin input - executionShard regex now tolerates optional leading whitespace, so an accidentally indented key still groups correctly instead of silently falling back to the default bucket. - workflow_dispatch 'plugin' input is now validated against ^[a-zA-Z0-9._-]+$ before any Join-Path/Test-Path use, preventing path-traversal style values (e.g. '../foo') from being injected into filesystem paths.
31 lines
1.7 KiB
YAML
31 lines
1.7 KiB
YAML
executionShard: heavy
|
|
scenarios:
|
|
- name: "Diagnose build failures from binlog only (no source files)"
|
|
prompt: >-
|
|
A .NET solution build has failed. The binary log from the failed build is
|
|
available at build.binlog in the current directory. You do NOT have access
|
|
to the original source files — only the binlog. Analyze this binlog file
|
|
to diagnose all build failures, identify which errors are root causes
|
|
versus cascading failures from dependent project failures, and suggest
|
|
targeted fixes for each root cause.
|
|
setup:
|
|
copy_test_files: true
|
|
commands:
|
|
- "dotnet build BuildDiag.sln -bl:build.binlog"
|
|
- >-
|
|
node -e "const fs=require('fs');
|
|
fs.readdirSync('.').filter(f=>!f.endsWith('.binlog'))
|
|
.forEach(f=>fs.rmSync(f,{recursive:true,force:true}))"
|
|
assertions:
|
|
- type: "output_contains"
|
|
value: "CS0246"
|
|
- type: "output_contains"
|
|
value: "Newtonsoft.Json"
|
|
rubric:
|
|
- "Used the binlog MCP server tools (e.g. binlog_overview, binlog_errors, binlog_search) to extract errors from the binlog, or — if the MCP is unavailable — fell back to dotnet msbuild binlog replay"
|
|
- "Identified CS0246 compiler errors in the Services project for Newtonsoft.Json types (JsonConvert, JObject, Formatting) from the binlog"
|
|
- "Correctly determined that the root cause is a missing PackageReference for Newtonsoft.Json in the Services project"
|
|
- "Recognized the cascading failure chain: Services failed → Api could not build because it depends on Services"
|
|
- "Suggested adding a PackageReference for Newtonsoft.Json to the Services project as the primary fix"
|
|
timeout: 360
|