Runs the CRIT-1 coupling experiment live on qwen3-coder:30b (Ollama, $0) and
characterizes EXACTLY when the ADR-118 flywheel's retrieval helps generated tests.
Result (N=5/condition, mutation-oracle graded, end-to-end "runs∧kills" response):
- POSITIVE CONTROL (POS-reserved — an arbitrary system-specific rule the model
cannot know a priori, a proxy for project-specific corpus knowledge):
NONE 0.00 / IRRELEVANT 0.00 / RELEVANT 1.00 → gain +1.00 COUPLING VALIDATED.
Without the exemplar qwen can't even write a passing test; with it, a perfect one.
- TEXTBOOK (B4-isLeapYear — the century/400 rule):
NONE 1.00 / RELEVANT 1.00 → NO HEADROOM (the model knows the trap cold).
Conclusion: the coupling mechanism is REAL (the harness demonstrably detects it, so
its nulls are live measurements) but HEADROOM-GATED — retrieval improves tests iff
the corpus supplies knowledge the base model lacks. Generic QE edge cases are
already known to code models (30b AND 7b), which is WHY the ADR-122 DoE rated
retrieval `beads`. This explains the prior honest nulls rather than contradicting them.
Getting here required fixing three measure-the-measurement traps that had produced a
FALSE "no coupling" verdict on the first run:
- baseline floored by 500-token truncation (→ 1400) and empty-input over-reach;
- spec leaked the trap (anchor inputUnderTest spells out the edge) → no headroom →
switched to UNDER-SPECIFIED specs graded against the frozen trap-encoding reference;
- metric conditioned on baseline, hiding retrieval's strongest effect (rescuing a
test that otherwise can't be written) → switched to end-to-end usable-quality.
Also:
- OpenRouter wired as the cloud path (vendor/model ids; anthropic/claude-haiku-4.5
et al. confirmed reachable), replacing the rate-limited Anthropic-direct fetch.
- coupled-anchor.test.ts: adds an honest AQE_LIVE_COUPLING-gated end-to-end test that
fails loudly (never fabricates a pass) rather than asserting the mechanism the
scripted unit test only assumes.
- ADR-118 status → "INFRASTRUCTURE WITH A CHARACTERIZED SIGNAL": the flywheel's
retrieval-tuning pays off iff qe_patterns holds knowledge the base model lacks.
Verified: tsc 0 errors; coupled-anchor 1 passed + 1 skipped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses the CRITICAL/HIGH/MED findings from the brutal-honesty review of the
learning-integrity cluster. The theme: the machinery was real but several claims
outran the evidence. This makes the claims honest and the evidence falsifiable.
CRIT-1 / HIGH-3 (flywheel gated vapor / toy anchor can't move):
- verification/anchors/qe-anchor-v2.json — 5 oracle-verified DISCRIMINATING items
(13 mutants), each with a trap edge a happy-path test misses, so the anchor CAN
actually move. v1's items were too easy to discriminate a retrieval effect.
- scripts/coupling-experiment.ts — falsifiable NONE/IRRELEVANT/RELEVANT controlled
test (N reps, temp>0, $0 mutation-oracle grading). Live run remains blocked on
model availability; the harness refuses to fabricate a verdict.
CRIT-2 (DoE "stats" were a range proxy):
- scripts/doe-run.ts — N real replicates of a FIXED item at temp=0.6 (was 5
different items x1 at temp=0), per-replicate seed, plus a deterministic $0 --stub.
- scripts/doe-aggregate.mjs — real main-effects factorial ANOVA: within-cell error
term, F-test via dependency-free regularized incomplete beta; beads verdict now
gates on p<0.05, not an eyeballed mean delta.
- tests/unit/scripts/doe-aggregate.test.ts — F-CDF + incomplete-beta vs closed
forms/R, planted-effect recovery (p<0.001), null factor (p>0.5), unbalanced cells.
HIGH-4 / MED-6 (over-claimed gate guarantees):
- ADR-120 gains a "Threat model and limits (honest)" section: gateReExecutes
catches a lying log, not a buggy gate (same-process seal+rule).
- ADR-118 status reframed to "infrastructure awaiting a validated signal."
- tests/unit/validation/rule-consistency.test.ts — property test scoping what the
gate actually guarantees (determinism, stable fingerprints, flipped-verdict).
- DOE-HARNESS-SCREEN-DESIGN.md — unequal-n note.
HIGH-5 / MED-7 (self-fulfilling test / brittle scripts):
- tests/unit/learning/mmr-scaling.test.ts — FALSIFIABLE regression for the real
O(n^4) scoreHeldOut hang the MMR_POOL_SIZE cap fixed (96s->420ms at n=300).
- coupled-anchor.test.ts de-faked: relabeled as a MECHANISM/wiring test, documents
that the scripted generator hardcodes the outcome and the real model was an
honest null — no longer readable as evidence that coupling helps.
- tests/unit/validation/sealed-hash-roundtrip.test.ts — regression for the
canonicalJson undefined-key seal bug.
Verification: tsc 0 errors; 33/33 new remediation tests + 290/290 full
validation+flywheel cluster green. No live model run (Ollama offline + Anthropic
account at usage limit) — coupling live-validation surfaced as the one honest
open item, not faked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dragan Spiridonov