Viktor Hofer 3fff1f2573 Skill-validator: perf, cancellation, and reliability improvements (#482) ...

* Skill-validator: perf, cancellation, and reliability improvements

Performance:
- Cache skill-validator-dist.tar.gz in CI (evaluation.yml and skill-check.yml)
  to skip .NET SDK setup, build, and tar on cache hit. Evaluate and comment-on-pr
  jobs restore from cache with artifact download fallback.
- Replace SemaphoreSlim with Lock in SessionDatabase for lighter-weight
  synchronous locking.

Cancellation:
- Thread CancellationToken from System.CommandLine entry point through the
  entire evaluate call chain: Run -> EvaluateTarget -> EvaluateSkill/EvaluateAgent
  -> ExecuteScenario/ExecuteAgentScenario -> ExecuteRun/ExecuteAgentRun, plus
  EvaluateSkillNoise -> ExecuteNoiseTest.
- Add CancellationToken parameter to AgentRunner.RunAgent, Judge.JudgeRun,
  PairwiseJudge.Judge, and OverfittingJudge.Analyze, forwarding to
  RetryHelper and LlmSession.SendAsync respectively.
- Pass CancellationToken to all ConcurrencyLimiter.RunAsync call sites so
  semaphore waits are cancellable.
- Re-throw OperationCanceledException at the top-level skill evaluation loop
  so Ctrl+C / CI cancellation propagates instead of being swallowed as a
  skill failure.

Reliability:
- Smart timeline truncation in Judge and PairwiseJudge: error events
  (session.error, runner.error) are always preserved during truncation
  regardless of position, with a summary marker showing event type breakdown.
- Add jitter (±25%) to RetryHelper exponential backoff to prevent thundering
  herd on rate-limit recovery.
- Add CancellationToken parameter to ConcurrencyLimiter.RunAsync<T>.
- Add ConcurrencyLimiter tests (6 tests) and RetryHelper jitter tests (2 tests).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Address PR review feedback

- RetryHelper: re-clamp delay after jitter so it never exceeds the
  remaining budget or MaxSingleDelayMs.
- Judge/PairwiseJudge: handle edge case where error count alone exceeds
  MaxTimelineEvents by trimming errors with a summary instead of
  unconditionally appending all of them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

* Add high-variance detection and refactor FailureKind to enum

- Add Statistics.CoefficientOfVariation() for measuring score consistency
- Flag scenarios with CV > 0.5 as high-variance in ScenarioComparison
- Show console warning and markdown footnote for high-variance scenarios
- Include high-variance scenario names in Comparator verdict reason
- Refactor FailureKind from magic strings to strongly-typed enum with
  JsonStringEnumMemberName attributes for backwards-compatible JSON output
- Add 6 new CV unit tests

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Viktor Hofer <vihofer@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

2026-03-31 13:58:28 +02:00

43 KiB

Raw Permalink Blame History

View Raw