* Delete msbuild-server skill
msbuild-server is now on by default since .NET 11 Preview 7. We don't want to recommend turning it on, automatically in other scenarios as critical fixes could be missing.
* Remove MSBuild Server recommendations
Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Pilot repository overlays for test skills
Add revisioned, repo-specific overlay bindings for selected dotnet-test skills and eval coverage for valid, scoped, and stale bindings.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Allow attributed overlay references
Add the JeremyKuhne source repositories to the reviewed external reference allowlist.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Address overlay pilot review feedback
Strengthen the MSTest eval against duplicate/default policy output and make the suite fixture parameter drive its test filter.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Clarify invalid overlay fallback
Make invalid overlays fall back to portable guidance and accept natural revision mismatch explanations in the stale-binding eval.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix overlay precedence wording
Restore the separator between repository and portable precedence rules.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Use a literal overlay project path check
Prevent the scaffold eval grader from interpreting dots in the expected project path as regex wildcards.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Enforce command-only overlay evals
Reject shell, edit, and create tools in both run-tests overlay prompts that prohibit execution.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
* Fix Codex discovery of the dotnet-msbuild binlog MCP server
.codex-plugin/plugin.json declared "mcpServers": "./.mcp.json", but the file
was packaged at .codex-plugin/.mcp.json. Codex resolves manifest resource
paths against the plugin root, so it looked for
plugins/dotnet-msbuild/.mcp.json and never found the binlog server.
Declare the server inline in .codex-plugin/plugin.json, matching the root
plugin.json and .claude-plugin/plugin.json, and drop the unreachable file.
Add a packaging regression check to skill-validator: every companion manifest
must declare the same MCP servers as the root plugin.json, and a manifest
referencing an external .mcp.json must resolve it from the plugin root the way
hosts do. skill-check.yml already runs `skill-validator check` over plugins/*
on every PR, so this now blocks in CI. A test also loads the shipped
dotnet-msbuild manifests and asserts binlog is present in each.
Fixes#1069
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Report non-object JSON roots as validation errors
JsonElement.TryGetProperty throws InvalidOperationException when the root
value is not an object, so a manifest or referenced .mcp.json that is valid
JSON but not an object (null, array, string) crashed skill-validator instead
of producing a validation error.
Check the root kind while reading and surface it as a structured error.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Describe the observed absence of counted pass data without assuming that a skill has no boolean graders.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5b4ee7ec-7244-4668-a151-5abee0ef2c4a
Keep one in-flight initialization request, retry failed index loads, and clarify why the derived index is excluded from plugin tabs.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5b4ee7ec-7244-4668-a151-5abee0ef2c4a
When the baseline metric is 0, reduction() is null and the Tokens/Time delta
cell rendered a blank percent label. Show an explicit n/a instead, matching the
rollup cells, so the delta is never an ambiguous blank.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
Generate a retained skill-value.json index from per-plugin histories so the default dashboard tab loads only the compact value data instead of every full benchmark file. Keep derived data out of plugin tabs and harden purge enumeration.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 5b4ee7ec-7244-4668-a151-5abee0ef2c4a
- Drill-down paired-observation count now uses min(baseline.n, treatment.n),
matching the table's paired-n, so inconsistent or legacy entries cannot
overstate the paired sample size.
- signedPct helper renders an exact-or-rounded zero reduction as "0%" with no
sign, so "no change" no longer reads as a "+0%" regression (deltaCell and the
rollup percent both use it).
- When skill-value.js is present but there are no evaluation plugins, fall back
to the Token Usage tab instead of leaving the page on the Skill Value panel's
permanent "Loading...".
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
- Load skill-value.js (and token-usage.js) before dashboard.js so
window.initSkillValue exists when the default Skill Value tab renders on
first load; previously the fallback activated the first plugin tab instead.
- Aggregate key now includes the plugin, so two plugins that share a skill
name no longer blend histories (the view is grouped Plugin -> Skill -> Model).
- valueSentence distinguishes "not-passed rate still gathering data" (pass
checks present but below the gate) from "No pass/fail data" (no boolean
graders), and describes the unit as counted trials rather than scenarios.
- Document that only the primary judge (vally-results-*) feeds benchmark data;
the dual-judge re-score (vally-crossjudge-*) feeds only judge-comparison.json,
so Skill Value and Quality/Efficiency both reflect the primary judge.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
Adds a Plugin -> Skill -> Model "Skill Value" tab that surfaces, per skill,
the token and time a skill saves and how often scenarios pass without it.
Reuses the existing publish-eval-data job by emitting an entries.SkillValue
block from generate-benchmark-data.ps1 (no new job, no adapter change).
Fixes in this change:
- Quality/Efficiency stay one entry per executor model per run. The scheduled
benchmark merge now groups by (model, judge) for Skill Value, but passes
-SkillValueOnly for the second+ judge of a model so dual-judge runs no longer
duplicate same-model Quality/Efficiency points.
- Pass telemetry is labelled honestly. adapt.mjs notes the per-arm pass booleans
may include LLM-grader results, so the field is hasPassData (not
hasDeterministic), the column reads "Not-passed", and the claim reads "do not
pass their checks" with a tooltip caveat rather than an objective failure gate.
- Single-model groups show that model's real deltas; multi-model rollups read
"N model/judge result(s) show measured value" instead of an ambiguous 0/m.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e23e5efe-713f-4898-a158-8970f70fdcaf
* Improve dense dashboard charts
Hide oversized quality-chart legends when their rendered series exceed the compact threshold, and direct viewers to the existing model filter for the color key.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f626eeb3-fc9f-4e9f-9a88-4dbde7594557
* Add accessible model markers to dashboard charts
Pair model colors with stable shape and rotation markers in quality charts and the Models filter. Reserve issue-marker shapes and keep vanilla points visible when series overlap.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f626eeb3-fc9f-4e9f-9a88-4dbde7594557
---------
Copilot-Session: f626eeb3-fc9f-4e9f-9a88-4dbde7594557
* Fix graph-inflated plugin versions
Use first-parent release checkpoints and net plugin content instead of raw NBGV DAG height. Add regression coverage for stale merges, concurrent predictions, base resets, reverts, and pending main changes.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fd5e638c-30e6-41ac-8be9-f035bb4f44a1
* Remove obsolete NBGV tooling
The release checkpoint algorithm uses Git directly, so the local NBGV tool and its private restore configuration are no longer required.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fd5e638c-30e6-41ac-8be9-f035bb4f44a1
* Fail closed on plugin diff errors
Treat git diff failures as version computation errors instead of silently reporting no content change.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fd5e638c-30e6-41ac-8be9-f035bb4f44a1
* Keep existing NBGV tooling configuration
Restore the local tool manifest and locked NuGet configuration because removing them is not required for the version-height correction.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fd5e638c-30e6-41ac-8be9-f035bb4f44a1
* Restore tool manifest byte-for-byte
Preserve the existing file exactly, including its original end-of-file form, so it is absent from the final PR diff.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fd5e638c-30e6-41ac-8be9-f035bb4f44a1
* Validate plugin release checkpoints
Reject arbitrary manifest patch edits after the new checkpoint-validation boundary while preserving legacy NBGV history and deliberate major or minor base resets.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a6779e5c-658b-40ae-8123-18d8432f74c6
* Align release checkpoint terminology
Rename effective-content helpers and update contributor and weekly-sync wording to match the checkpoint-based version model.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: a6779e5c-658b-40ae-8123-18d8432f74c6
---------
Copilot-Session: fd5e638c-30e6-41ac-8be9-f035bb4f44a1
Copilot-Session: a6779e5c-658b-40ae-8123-18d8432f74c6
Fail closed on missing or duplicate comparison slots, reject duplicate stimulus names before execution, and clarify how the practical floor changes power at its boundary.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1c6716a2-172b-461a-adb5-127b1ba7e96b
Collapse repeated runs to one majority-direction vote per stimulus, preserve run-level reliability evidence, and align authoring checks, reports, and guidance with that inference unit.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 1c6716a2-172b-461a-adb5-127b1ba7e96b
Abhitej John