* Decouple agent runs from judging in skill-validator
Allow baseline and treatment agent runs to execute in one parallel pool, with
judging deferred to a final step, removing the serial precompute-baseline ->
evaluate barrier.
evaluate --no-judge: runs the requested agent arms and persists sessions.db +
metrics, but skips all judging and requires no baseline file (exits 0 on
successful runs). Forces session persistence; mutually exclusive with
--baseline-out/--baseline-from; requires --results-dir. Only the agent model is
validated (judge model recorded as metadata); warns when zero runs persist.
rejudge --baseline-dir: cross-directory judging that pairs each treatment run
with its baseline run by the BaselineStore key (prompt SHA + target SHA,
persisted in a new sessions.db baseline_key column), runs the same
pairwise/independent judges and pass/fail gates evaluate uses, and writes
verdicts/reports. Baseline and treatment must share --model (rejected if either
side mixes models); effective judge model resolves to --judge-model, else the
treatment DB's, else the baseline DB's, rejecting a persisted mismatch. The
inline per-run judging is extracted into a shared JudgeRunGroup helper reused by
both the same-dir and cross-dir paths (no duplicated judging logic).
Adds tests for baseline_key round-trip + legacy migration, ComputeScenarioKey,
cross-dir pairing, and model/judge-model compatibility. Updates README and
InvestigatingResults docs.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Hoist ComputeScenarioKey to scenario level to avoid per-run fixture hashing
Compute the baseline key once per scenario instead of once per run, so
multi-run evaluations (--runs N) don't re-hash fixture files N times.
The key is threaded into ExecuteRun/ExecuteAgentRun as a parameter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Memoize scenario-key fixture hashing for the duration of an evaluation
Route the persisted baseline_key through an evaluation-scoped BaselineStore
cache (ComputeScenarioKeyCached) instead of the static ComputeScenarioKey, so
a scenario's setup fixtures are hashed at most once across every run, arm, and
scenario of an evaluation. When a baseline store already exists (--baseline-from
/--baseline-out) its warm input-hash cache is reused; otherwise a dedicated,
side-effect-free cache (BaselineStore.ForKeyCache) is threaded through.
Addresses the PR review comment about ComputeScenarioKey re-hashing large
fixture directories. Adds a test asserting the cached variant equals the
canonical static computation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Bump session schema version to 3 for the baseline_key column
The baseline_key column was added without bumping schema_info.version, so
external tools keying off the version couldn't detect the new column. Introduce
a single SchemaVersion constant (= 3), stamp it after migrations run, and drop
the drift-prone version literal from the raw CREATE SQL so SetSchemaInfo is the
sole source of truth. Update the schema-info and legacy-upgrade tests to expect
version 3.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fail fast on judging-dependent options under --no-judge; fix README prefix
Under --no-judge the noise test and overfitting fix cannot run (they require
judging), so reject --noise-skills-dir and --overfitting-fix up front instead
of silently ignoring them, mirroring the existing --baseline-out/--baseline-from
guard. Add EvaluateCommandTests covering these early rejections (they short-
circuit before any model/network call).
Also prefix the decoupled-runs README example with 'skill-validator' for
consistency with the other CLI invocations in the doc.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reorganize the skill-validator project from layer-based grouping
(Commands/, Services/, Models/, Utilities/) to feature-based slices
(Check/, Evaluate/, Consolidate/, Shared/).
Key changes:
- Check/: CheckCommand, SkillProfiler, AgentProfiler, PluginValidator,
ExternalDependencyChecker, ReferenceScanner
- Evaluate/: EvaluateCommand, RejudgeCommand (now a subcommand of
evaluate), AgentRunner, Judge, PairwiseJudge, and other eval services
- Consolidate/: ConsolidateCommand
- Shared/: Models, SkillDiscovery, PluginParser (extracted from
PluginValidator), Reporter, and utilities
Split PluginValidator: parsing helpers (ParsePluginJson,
TryGetSafeSubdirectory) moved to Shared/PluginParser.cs; validation
logic stays in Check/PluginValidator.cs.
Move RawFrontmatter and RawAgentFrontmatter types from EvalSchema into
SkillDiscovery since they are only consumed by discovery.
Wire RejudgeCommand as subcommand: skill-validator evaluate rejudge.
Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
YuliiaKovalova