mirror of
https://github.com/dotnet/skills.git
synced 2026-09-20 09:49:54 +08:00
eval-exec-sonnet46
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 YuliiaKovalova
|
76a5ec8cb3 |
Decouple agent runs from judging in skill-validator (--no-judge + cross-dir rejudge) (#769)
* Decouple agent runs from judging in skill-validator Allow baseline and treatment agent runs to execute in one parallel pool, with judging deferred to a final step, removing the serial precompute-baseline -> evaluate barrier. evaluate --no-judge: runs the requested agent arms and persists sessions.db + metrics, but skips all judging and requires no baseline file (exits 0 on successful runs). Forces session persistence; mutually exclusive with --baseline-out/--baseline-from; requires --results-dir. Only the agent model is validated (judge model recorded as metadata); warns when zero runs persist. rejudge --baseline-dir: cross-directory judging that pairs each treatment run with its baseline run by the BaselineStore key (prompt SHA + target SHA, persisted in a new sessions.db baseline_key column), runs the same pairwise/independent judges and pass/fail gates evaluate uses, and writes verdicts/reports. Baseline and treatment must share --model (rejected if either side mixes models); effective judge model resolves to --judge-model, else the treatment DB's, else the baseline DB's, rejecting a persisted mismatch. The inline per-run judging is extracted into a shared JudgeRunGroup helper reused by both the same-dir and cross-dir paths (no duplicated judging logic). Adds tests for baseline_key round-trip + legacy migration, ComputeScenarioKey, cross-dir pairing, and model/judge-model compatibility. Updates README and InvestigatingResults docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Hoist ComputeScenarioKey to scenario level to avoid per-run fixture hashing Compute the baseline key once per scenario instead of once per run, so multi-run evaluations (--runs N) don't re-hash fixture files N times. The key is threaded into ExecuteRun/ExecuteAgentRun as a parameter. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Memoize scenario-key fixture hashing for the duration of an evaluation Route the persisted baseline_key through an evaluation-scoped BaselineStore cache (ComputeScenarioKeyCached) instead of the static ComputeScenarioKey, so a scenario's setup fixtures are hashed at most once across every run, arm, and scenario of an evaluation. When a baseline store already exists (--baseline-from /--baseline-out) its warm input-hash cache is reused; otherwise a dedicated, side-effect-free cache (BaselineStore.ForKeyCache) is threaded through. Addresses the PR review comment about ComputeScenarioKey re-hashing large fixture directories. Adds a test asserting the cached variant equals the canonical static computation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bump session schema version to 3 for the baseline_key column The baseline_key column was added without bumping schema_info.version, so external tools keying off the version couldn't detect the new column. Introduce a single SchemaVersion constant (= 3), stamp it after migrations run, and drop the drift-prone version literal from the raw CREATE SQL so SetSchemaInfo is the sole source of truth. Update the schema-info and legacy-upgrade tests to expect version 3. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fail fast on judging-dependent options under --no-judge; fix README prefix Under --no-judge the noise test and overfitting fix cannot run (they require judging), so reject --noise-skills-dir and --overfitting-fix up front instead of silently ignoring them, mirroring the existing --baseline-out/--baseline-from guard. Add EvaluateCommandTests covering these early rejections (they short- circuit before any model/network call). Also prefix the decoupled-runs README example with 'skill-validator' for consistency with the other CLI invocations in the doc. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
YuliiaKovalova
|
bcaa91882c |
Add reusable shared baseline to skill-validator evaluate (--baseline-out / --baseline-from) (#754)
* Add shared/precomputed baseline reuse to skill-validator evaluate (#751) Add --baseline-out and --baseline-from options to the evaluate command so the no-skill/no-agent baseline arm can be computed once and reused as a shared control group across multiple skill/agent evaluations. This eliminates redundant baseline runs and removes baseline run-to-run variance from cross-config comparisons. - New BaselineStore + BaselineFile/BaselineScenarioEntry models, keyed per scenario on SHA-256(prompt) with a header recording version, model, validator version and runs. Load validates version + model and fails fast on mismatch or missing scenarios. - Register the new serializable types in SkillValidatorJsonContext (AOT source-gen). - Wire two mutually-exclusive CLI options into ValidatorConfig; thread an optional BaselineStore through both execution paths. - On reuse, skip the baseline agent run, its assertions/constraints/ task-completion/judging, and attribute no extra pairwise tokens to the baseline; report the scenario with the baseline-reused session phase and a reused status. In write mode, record each scenario's averaged baseline and persist it after the run. - Add unit tests for BaselineStore and document the feature in README and InvestigatingResults. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Bind reused baseline to fixture identity (targetSha), not just prompt Align baseline reuse with the (prompt, model, targetSha) identity contract from the upstream eval-harness design: previously the per-scenario reuse key was the prompt SHA + model only, so two scenarios that share an identical prompt but feed the agent different input artifacts (e.g. a different build.binlog) would collide and silently reuse the wrong baseline. - Add BaselineScenarioEntry.TargetSha: a SHA-256 over the scenario's materialized inputs — files auto-copied via copy_test_files, explicit setup files (inline content or copied sources), and the setup command recipe. The reuse key is now (promptSha, targetSha); both must match. Bump the on-disk schema to version 2. - Memoize target hashing per process via a cheap, file-I/O-free setup signature to avoid re-hashing large fixtures across the N runs. - Thread the originating eval.yaml path into Record/TryGetBaseline/ FindMissingScenarios so inputs can be fingerprinted. - Tests: target SHA is stable and content-sensitive; same-prompt/different-fixture scenarios do not reuse each other's baseline and are surfaced by FindMissingScenarios. Update README and InvestigatingResults. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Harden baseline reuse identity after review Address rubber-duck review findings on the shared-baseline feature: - Fix Sha256Hex 32-byte bug: a 32-byte input was treated as an already-computed digest and not hashed. Split into Sha256Hex (always hash) + HexDigest (encode existing digest). - Broaden reuse identity: the cached baseline RunResult depends on the judge model and on per-scenario evaluation criteria (rubric, assertions, expect/reject tools, turn/token/timeout limits). Add JudgeModel to the baseline header (validated on load) and fold the criteria into the per-scenario targetSha so changing them invalidates reuse instead of silently serving a stale result. - Mirror AgentRunner.SetupWorkDir exactly when hashing copied fixtures: exclude only the top-level eval.yaml (nested eval.yaml files are copied, so they must be hashed). - Make the target-SHA cache instance-scoped (memoizing only the expensive fixture-input hashing) so it can't serve stale hashes or leak across evaluations/tests; hash inline file Content in the cache key instead of embedding it. - Deterministic Save ordering (Name, PromptSha, TargetSha); guard Load against null Scenarios; enrich FindMissingScenarios output with the eval path. - Document that setup commands are fingerprinted by recipe, so reuse assumes they are deterministic/hermetic. - Tests + docs updated; add judge-model-mismatch and criteria-identity tests (562 pass). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address PR review comments on baseline reuse - Mirror AgentRunner.SetupWorkDir exactly when hashing copied fixtures: enumerate only the files actually copied (top-level siblings except eval.yaml, recursing into directories) and skip reparse points and out-of-root junctions, instead of blindly hashing every file under the eval directory. This keeps the fixture identity restricted to the intentionally-copied set so stray output/log files can't poison reuse. - Stream baseline JSON to/from disk (File.OpenRead/File.Create with JsonSerializer) so large baselines never materialize as one giant in-memory string. - Enrich the fail-fast 'missing scenario' output with the eval path and short prompt/target SHA prefixes so it is actionable when scenario names collide across eval files. - Add a test locking in recursive (nested-directory) fixture hashing. 563 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address second round of PR review comments - Skip top-level reparse points in EnumerateCopiedFixtures (not just nested ones) so a top-level symlink/junction can't cause hashing of data outside the eval directory; code now matches the docstring. - Record uses first-writer-wins (TryAdd) instead of overwriting, so a scenario identity recorded by multiple parallel targets yields a deterministic --baseline-out regardless of completion order. - Persist the baseline judge result to the session DB even when the baseline is reused, so the registered 'baseline-reused' session record is complete for downstream investigation tooling (pairwise was already saved; the judge result was incorrectly gated on a fresh run). - Add first-writer-wins test (564 pass). Note: BaselineStore stays internal — the test project already has InternalsVisibleTo, so it compiles. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Pin MessagePack to 2.5.301 to fix NU1903 vulnerability The transitive MessagePack 2.5.198 (via GitHub.Copilot.SDK -> StreamJsonRpc) has a high-severity vulnerability (GHSA-hv8m-jj95-wg3x) that fails the build under TreatWarningsAsErrors. Pin a direct reference to the patched 2.5.301. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address review: bare-filename fixture hashing, clone reused baseline, consistent token attribution - ComputeInputsSha: normalize evalPath via Path.GetFullPath so a bare filename still hashes sibling fixtures (avoids TargetSha collisions / unsafe reuse). - RunMetrics.Clone(): per-run copy with fresh collections; reuse paths now clone the cached baseline so concurrent evaluations never share a mutable instance. - Pairwise judge tokens attributed to both compared runs in every mode (the baseline clone makes this safe), keeping token deltas comparable across --baseline-from modes. - Reword Record first-writer-wins doc to describe the within-run stabilization guarantee rather than order-independence. - Add tests for bare-filename fixture hashing and clone isolation (566 pass). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Aaron Powell
|
3134b816de |
Add JSON output for skill-validator check (#601)
* Add JSON output for skill-validator check Fixes #600 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix empty check discovery results Address PR feedback by failing when explicit skill or agent paths discover nothing, including the combined check path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address Copilot review comments - Replace AddPlainError with AddGeneralError (remove duplicate method) - Key pluginSkills by DirectoryPath instead of display name to avoid mismatch when plugin.Name differs from directory name - Use OS-aware path comparison in IsPathWithin (Ordinal on Unix, OrdinalIgnoreCase on Windows) - Pre-build name lookup dictionaries in CreateJsonOutput to eliminate O(n*m) FirstOrDefault scans for external dependency attachment - Extract CheckJsonSerializerContext scoped to check JSON output so UseStringEnumConverter does not affect unrelated JSON payloads Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address latest Copilot PR feedback - Attach external dependency warnings by stable path identifiers (skill SKILL.md path, agent path, plugin directory path) - Use OS-aware comparer for plugin directory path matching - Restore console warning formatting via SkillProfiler.FormatProfileWarnings - Move CheckJsonSerializerContext into SkillValidator.Check namespace - Precompute reference-attachment targets to avoid repeated full-path normalization and container recalculation per finding - Add regression test covering duplicate skill names with external dependency warnings in JSON output Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Fix flaky duplicate-skill JSON test path - Run duplicate-name external dependency assertion through plugin mode, which is where external dependency checks are executed - Add plugin fixture helper for JSON output tests Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Address latest Copilot follow-up feedback - Reintroduce PluginValidationResult as obsolete compatibility type while keeping PluginCheckResult as the primary model - Avoid unnecessary profile-line formatting when verbose output is off - Replace string discriminators for external dependency kind with enum - Centralize JSON warning kind literals as constants Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> |
||
|
Jan Krivanek
|
6ebb5616b2 | Support checking skills and agents at single invocation (#533) | ||
|
Viktor Hofer
|
3b16e4e4c8 |
docs: document token count reporting in skill-validator (#497)
* docs: document token count reporting in skill-validator Add a 'Getting the token count for a skill' section to the README explaining how to use 'check --verbose' to see BPE token counts and complexity tiers. Expand the Run metrics table in InvestigatingResults.md with the inputTokens, outputTokens, cache, and judge token fields available in results.json. Update the example analysis script to print input/output tokens. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: address review feedback on token count documentation - Fix tier table: replace 'Optimal sweet spot' with 'Recommended tier (sweet spot: 800–2,500)' to align with earlier statement - Fix tokenEstimate description to accurately reflect implementation behavior (cache fallback when inputTokens is zero) Agent-Logs-Url: https://github.com/dotnet/skills/sessions/dd5ded07-a968-4574-bfe9-1602b4434f07 Co-authored-by: ViktorHofer <7412651+ViktorHofer@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> |
||
|
Viktor Hofer
|
7747fcb427 |
Move skill-validator docs under src/ and include in packages (#476)
* Move skill-validator docs under src/ and include in packages Move README.md to src/ and InvestigatingResults.md + OverfittingDetection.md to src/docs/ so they ship in the NuGet package and .tar.gz archives. - Include docs/*.md in NuGet pack and publish output via CopyToPublishDirectory - Add LICENSE to publish output (tar.gz only, not NuGet — uses PackageLicenseExpression) - Replace enumerated docOnlyFiles lists with a regex pattern in CI workflows - Deduplicate eval.yaml docs between CONTRIBUTING.md and the shipping README - Add test fixture files walkthrough to README (moved from CONTRIBUTING.md) - Add AGENTS.md guidance: src/ is the packaging boundary Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Update link format in failure diagnosis message * Update AGENTS.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |