rUv aee3b99910 feat(harness): anti-overfitting — frozen public human eval + per-gen red/blue deltas + clean-room replay (ADR-176) (#2580) ...

Addresses the biggest failure mode: the loop could overfit the self-supervised
proxy while human relevance is merely PRESERVED, not improved. Fix makes that
visible + falsifiable:

1. FROZEN PUBLIC HUMAN EVAL SET — .claude/eval/human-relevance-frozen-v1.json
   (10 ADR-081 labelled queries), loaded via harness-frozen-eval.ts which PINS
   the canonical content hash (sha256:6096e48e…) and THROWS on drift. Single
   source of truth for the red/blue anchor; supersede with a new versioned file,
   never edit. The daemon runtime now loads the anchor from it (was an inline
   duplicate).

2. PER-GENERATION HUMAN-RELEVANCE DELTA — every receipt records
   deltas.humanRelevance (candidate − baseline nDCG@3 on the frozen set) +
   humanEvalHash. flywheelStatus exposes cumulativeBenchmarkDelta (proxy) vs
   cumulativeHumanRelevanceDelta (human); the status script flags
   'proxy up but human flat → OVERFITTING'. The overfitting mode is now
   observable in the lineage instead of hidden.

3. CLEAN-ROOM REPLAY ACCEPTANCE TEST — scripts/replay-generation.mjs: from a
   clean install, replay one PROMOTED generation from its receipt alone —
   recompute every hash bit-identically + re-run accept/v1+sig to reproduce
   promoted=true, with fetch trapped (offline). Wired into v3-ci alongside the
   proof guard (which now also verifies the frozen-eval pinned hash). Verified:
   real-generation-1 replays with identical hashes, offline.

Additive + backwards-compatible (new optional receipt fields; old committed
bundles still replay). 76 harness+config tests green; both CI guards pass.


Claude-Session: https://claude.ai/code/session_01S7GYqnVUVxBfZ5W8znqry3

2026-07-05 12:56:22 -04:00

2.9 KiB

Raw Permalink Blame History

View Raw