mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
Abhitej John
0723ac2b44
Run every skill against a 3-model matrix (Opus 4.8, GPT 5.5, Sonnet 4.6) with a cross-family judge that is never the executing model (default judge = latest Opus). Adds eval-models.json as the single source of truth, resolve-judge.mjs to derive model/judge pairs, preflight-models.mjs and a ListModels command for availability checks (without exposing the full model roster), and publish-vally-data.mjs for carrying run history to skills-data. - eng/eval-models.json: model matrix + judge map (Sonnet 4.6) - eng/resolve-judge.mjs: judge/matrix/required/selftest resolver - eng/preflight-models.mjs + ListModelsCommand.cs: privacy-safe preflight - vally-evaluation.yml: multi-model fan-out; redirect verbose trajectories to per-eval vally.log + tail-on-failure; 8-token selection pool parity - .NET CLI defaults + README bumped claude-opus-4.6 -> claude-opus-4.8 - vally-adapter: adapt.mjs/run-vally-evals.sh default Sonnet 4.6 - dashboard + InvestigatingResults.md updates Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
26 lines
1.4 KiB
JSON
26 lines
1.4 KiB
JSON
{
|
|
"$comment": "Single source of truth for the models the skill-evaluation infra runs against and how a judge is chosen. Consumed by resolve-judge.mjs, which the evaluation and vally-evaluation workflows and eng/vally-adapter/run-vally-evals.sh call. Bump the IDs here (verified against the CI Copilot token's ListModels output) to roll models forward everywhere at once.",
|
|
|
|
"latest": {
|
|
"opus": "claude-opus-4.8",
|
|
"sonnet": "claude-sonnet-4.6",
|
|
"gpt": "gpt-5.5"
|
|
},
|
|
|
|
"$comment_gating": "Agent model used for PR (gating) skill-validator runs. Kept to a single model for stable, comparable-to-history gating. Judge is still resolved via the judge rules below so the judge is never the same model as the agent.",
|
|
"gatingModel": "claude-opus-4.8",
|
|
|
|
"$comment_matrix": "Agent models used for non-gating (nightly/scheduled) skill-validator runs and for all vally runs. Each is judged by the model resolved from the rules below.",
|
|
"matrix": [
|
|
"claude-opus-4.8",
|
|
"gpt-5.5",
|
|
"claude-sonnet-4.6"
|
|
],
|
|
|
|
"$comment_judge": "Judge selection: the judge defaults to defaultJudge (latest Opus). When the agent model IS the defaultJudge (or otherwise appears in judgeOverrides), the override is used so a model never judges itself. Every resolved judge is asserted to differ from the agent model.",
|
|
"defaultJudge": "claude-opus-4.8",
|
|
"judgeOverrides": {
|
|
"claude-opus-4.8": "gpt-5.5"
|
|
}
|
|
}
|