mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
YuliiaKovalova
76a5ec8cb3
* Decouple agent runs from judging in skill-validator Allow baseline and treatment agent runs to execute in one parallel pool, with judging deferred to a final step, removing the serial precompute-baseline -> evaluate barrier. evaluate --no-judge: runs the requested agent arms and persists sessions.db + metrics, but skips all judging and requires no baseline file (exits 0 on successful runs). Forces session persistence; mutually exclusive with --baseline-out/--baseline-from; requires --results-dir. Only the agent model is validated (judge model recorded as metadata); warns when zero runs persist. rejudge --baseline-dir: cross-directory judging that pairs each treatment run with its baseline run by the BaselineStore key (prompt SHA + target SHA, persisted in a new sessions.db baseline_key column), runs the same pairwise/independent judges and pass/fail gates evaluate uses, and writes verdicts/reports. Baseline and treatment must share --model (rejected if either side mixes models); effective judge model resolves to --judge-model, else the treatment DB's, else the baseline DB's, rejecting a persisted mismatch. The inline per-run judging is extracted into a shared JudgeRunGroup helper reused by both the same-dir and cross-dir paths (no duplicated judging logic). Adds tests for baseline_key round-trip + legacy migration, ComputeScenarioKey, cross-dir pairing, and model/judge-model compatibility. Updates README and InvestigatingResults docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Hoist ComputeScenarioKey to scenario level to avoid per-run fixture hashing Compute the baseline key once per scenario instead of once per run, so multi-run evaluations (--runs N) don't re-hash fixture files N times. The key is threaded into ExecuteRun/ExecuteAgentRun as a parameter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Memoize scenario-key fixture hashing for the duration of an evaluation Route the persisted baseline_key through an evaluation-scoped BaselineStore cache (ComputeScenarioKeyCached) instead of the static ComputeScenarioKey, so a scenario's setup fixtures are hashed at most once across every run, arm, and scenario of an evaluation. When a baseline store already exists (--baseline-from /--baseline-out) its warm input-hash cache is reused; otherwise a dedicated, side-effect-free cache (BaselineStore.ForKeyCache) is threaded through. Addresses the PR review comment about ComputeScenarioKey re-hashing large fixture directories. Adds a test asserting the cached variant equals the canonical static computation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump session schema version to 3 for the baseline_key column The baseline_key column was added without bumping schema_info.version, so external tools keying off the version couldn't detect the new column. Introduce a single SchemaVersion constant (= 3), stamp it after migrations run, and drop the drift-prone version literal from the raw CREATE SQL so SetSchemaInfo is the sole source of truth. Update the schema-info and legacy-upgrade tests to expect version 3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fail fast on judging-dependent options under --no-judge; fix README prefix Under --no-judge the noise test and overfitting fix cannot run (they require judging), so reject --noise-skills-dir and --overfitting-fix up front instead of silently ignoring them, mirroring the existing --baseline-out/--baseline-from guard. Add EvaluateCommandTests covering these early rejections (they short- circuit before any model/network call). Also prefix the decoupled-runs README example with 'skill-validator' for consistency with the other CLI invocations in the doc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>