Files
daymade 80115b243c fix(read-claude-code-history): prune injected Codex records, dedupe boilerplate chunks, replace the fixed embed sleep (#527)
Two staged changes to daymade-claude-code/read-claude-code-history's hybrid
index (scripts/history_index.py), plus a round of independent review and
fixes on top.

P0: Codex's injected preamble blocks (<goal_context>, <subagent_notification>,
<skill>, plus the existing preamble tags) are now skipped at extraction time
and swept from already-stored records during update_index (anchored with
substr(fts_text,1,n)=prefix, not LIKE, so `_` can't wildcard-match a real
record). These blocks measured 27.6M of a 60.7M-token backlog. Schema v3 adds
chunks.text_hash (content sha1), backfilled in batches of 5,000 with
per-batch commits. A new boilerplate policy keeps one usable copy of any
chunk duplicated 3+ times (68% of the chunk backlog, 94% cross-session) and
reverses the demotion if the copy count later drops back below the
threshold.

P1: host-pressure pause is now bounded (EMBED_PAUSE_CEILING_SECONDS=600,
stop_reason=host_pressure, exits 0) instead of pausing forever while holding
the writer lock; token/ETA progress uses a windowed rate; embed_stop_reason
covers running/complete/max_seconds/host_pressure/memory_boundary/
warmup_failed and is written before work starts; vectors_drop_deferred is
renamed non_embeddable_chunks; the cross-process writer lock retries for up
to 900s instead of failing an entire nightly run on a one-second overlap.

Round-1 review (three fresh-context lenses: data-safety, nightly-compat,
embed-loop; data-safety found 0 issues) found and this change fixes 3 more
minors: index --json leaked a progress line to stdout while building a
fresh/--rebuild database (now stderr); embed's deferred vector deletes
(orphaned and boilerplate-demoted) were not counted in either return path
(now orphan_vectors_dropped/demoted_vectors_dropped, verified against real
sqlite-vec rowcount); the reference doc filed host_pressure alongside the
stop reasons that raise before writing their marker, when it actually exits
0 (doc now routes alerting through embed_stop_reason and a non-decreasing
remaining, not $?). All three were reproduced as failing before the fix and
independently re-confirmed on this exact head.

Tests: 221 in this suite (194 after P0 alone, 218 after P1 and its own
review-fix round). 17/17 registered CI suites, check_marketplace.py (57
plugins), check_shell_syntax.sh (51 scripts) and check_version_progression.py
all pass. daymade-claude-code bumped 3.25.0 -> 3.26.0.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-12 21:52:30 +08:00
..