FINDING: the cheapest model that recovers macro structure under the climb
is claude-haiku-4-5 -- it converges in 5 rounds given --max-rounds 8 (was
capped-at-1-residual at cap 4; one round short, not a hard ceiling). On the
GPT side only the frontier (gpt-5.5) converges; gpt-5.4-mini oscillates
(trajectories 4->1->5->2 at cap 4 and 3->3->2->1->6->2->1->5 at cap 8) and
never reaches clean. "Cheap tiers partially recover" was model-dependent,
not a property of cheap: read the two families separately.
Codex CLI as the GPT-side generator (OpenRouter still 402-blocked):
evals/model_generate.py gains a `codex` kind. codex exec is an agent CLI
whose stdout is an interleaved transcript, so call_codex extracts ONLY the
final message via -o/--output-last-message (verified clean by hand), runs
read-only/ephemeral, and wraps the call in its own process group with a
hard timeout that SIGKILLs the group -- the known silent-hang failure mode
becomes an honest failed round. CLIMB-07 gates the extraction and the hang
kill offline against a fake codex binary (evals/fixtures/climb/fake_codex.py).
All four live runs (gpt-5.5 cap 4 converged 4->1->1->0; gpt-5.4-mini caps 4
and 8 capped; haiku-4-5 cap 8 converged 1->1->1->1->0) preserved every
source fact every round, on the same SKILL-MACRO-01 fixture and settings as
the recorded matrix. pipeline.md's climb section gains the follow-up table
and the cheapest-viable doctrine; the tiering row and PRODUCT.md bullet
updated to the model-dependent reading.
Gates: adversarial 469 pass / 1 documented xfail (CLIMB-07 new), schema,
gates-doc, taboo parity, build --check, strict-leakage all green; 0 scanner
hits (hard or soft) on both edited docs. Behavioral tune: 33/34 tasks ran;
the one failure was a transient API server error mid-response on
SKILL-MACRO-01/with_skill (runner infra, not a model answer), which aborted
the non-blocking judge step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
THESIS: the recorded parity matrix showed single-pass macro-structure
self-checking failed on every model tested -- told in prose to drop a
conclusion coda / connective scaffold / outline echo, each shipped it
anyway. But structure_scan + silhouette_scan SEE those failures precisely,
so the failure is correctable even where it is not self-checkable.
PROOF: evals/run_structure_climb.py closes the loop -- generate -> scan both
-> turn each finding into a TARGETED directive that names WHERE and WHAT
(the model can't see the tell, so the directive locates it) -> feed the
directives + current draft back -> regenerate, capped at N rounds, with a
validate_preservation guard so climbing can never eat a fact. Honest
terminal states (converged / capped / preservation_violation) carry
distinct exit codes.
Core IP is the directive builder: a deterministic map from every macro flag
both scanners can emit to a location-naming instruction (coda -> names the
final paragraph; connective openers -> names the paragraph numbers;
callback_content -> names the recapped opening vocabulary). CLIMB-06 gates
1:1 flag->directive coverage.
Eval-first (red then green): six CLIMB-* rows drive the loop offline through
a mock generator (evals/fixtures/climb/) -- converge (monotonic consumption,
exit 0), capped-still-dirty (honest, nonzero), single-pass control (dirty
without the loop), preservation abort (names the dropped fact), directive
location-naming, and full directive coverage. Registered a blocking
mock-only gate (structure-climb) and synced evals/CHECKS.md.
LIVE (2026-07-07, same SKILL-MACRO-01 fixture as Task C, source-anchored
preservation): single pass ships macro tells on both Claude tiers; the climb
recovers macro on the frontier where single pass failed (sonnet-4-5
converged 2->1->0, facts preserved) -- macro is now machine-CORRECTED, not
merely machine-gated. On the cheap tier the recovery is partial: haiku-4-5
cleared silhouette immediately but oscillated on surface structure and
capped at one residual flag; honest numbers recorded. The OpenRouter
spectrum (gpt-5.5, glm-5.2) is pending on account billing (HTTP 402);
exact reproduction commands recorded in references/pipeline.md.
Docs: pipeline.md tiering macro row + measured-conclusions bullet updated to
machine-corrected; new "Macro structure under the climb" subsection with the
loop, directive builder, and before/after evidence; PRODUCT.md macro bullet;
one-line SKILL.md pointer in the validation flow.
Battery green: adversarial 468 pass / 1 documented xfail, schema, gates-doc,
coverage, kata, taboo parity, build --check, strict-leakage; 0 hard scanner
hits on edited docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
Plan 013: the Growth promise in docs/PRODUCT.md relied on someone
remembering to run wiki_sync, re-run the parity bench, and funnel misses
through contribute. This lands the three missing pieces:
- references/refresh.md: the agent-runnable refresh procedure — staleness
check, wiki-sync flow, parity-bench flow with the eval-gating rule,
every-miss-becomes-a-row routing through contribute, and result-recording
conventions (pipeline.md tables, TUNE-RESULTS.md style).
- scripts/refresh_status.py: stdlib, network-free staleness reporter.
JSON with wiki_sync / parity_bench / newest_pattern_row, each carrying
last/days/stale. Sources: .wiki_sync_state.json (last_timestamp, mtime
fallback), the "Live matrix recorded **YYYY-MM-DD**" line in
references/pipeline.md, and git log -1 --format=%cs on
evals/adversarial-evals.json (argv list, no shell). Thresholds (90/180/60
days) are commented guidance; always exits 0 — reporter, not gate.
- docs/DECISIONS.md: durable home for decided tradeoffs, each entry with
Decision / Why / Revisit-when / Source. Seeds the PRODUCT.md decisions
(agent-invoked only, English-only, no packaging, no rights checks,
removal-dominant axis) plus the commit-message-only ones: WP8 fixture
word-floor/length-balance queue (05f2363), voice impostor/background
calibration queue (41b60d1), protects-grain + build_report deferrals
(d6f6321), and FP-06 as the single intentional xfail (efd7e0e,
evals/CRITIQUE.md).
- One-line pointers to references/refresh.md from references/maintenance.md
and docs/PRODUCT.md (Growth).
All four touched/new docs pass banned_phrase_scan and
structure_scan --genre docs. Suite: 439 PASS / 1 XFAIL (FP-06) / 0 FAIL.
New decisions go in DECISIONS.md at decision time; the reporter is the
building block if the operator ever wants scheduled refresh outside this
repo.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K6CYksdLbXbTAxcAQjvHz5
- docs/PRODUCT.md: the grill-session decisions as durable doctrine; passes all
three scanners (and dogfooding it surfaced the carve-out below)
- references/pipeline.md: Model Parity results table populated from the live
2026-07-06 benches; tiering guidance updated to measured conclusions
- Bare 'research indicates/shows/suggests' joins vague_attribution
(FN-31/FP-84/REC-49, eval-first) — caught live when a model paraphrased
around 'studies show' in the replacement bench
- silhouette_scan --genre docs suppresses callback_content only: doctrine/spec
register reprises opening themes by convention (SIL-9 pins the carve-out;
essay recap codas still flag by default)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clayton Kim