mirror of
https://github.com/mksglu/context-mode.git
synced 2026-09-19 03:27:16 +08:00
Mert Köseoğlu
f1878a703b
* fix(stats): lazy schema migration in aggregator — recover legacy DB signal (#683 follow-up) Symptom: post-v1.0.147 upgrade, ctx_stats Section 1 displays "Without context-mode 158 KB / With context-mode 158 KB / 0% kept out" — a degenerate identity bar instead of the historical savings ratio. Root cause: three intertwined bugs. BUG A — Schema migration only runs in SessionDB constructor. db.ts L754-776 contains an ALTER TABLE migration for the post-v1.0.130 columns (project_dir, attribution_source, attribution_confidence, bytes_avoided, bytes_returned). It only runs when SessionDB is instantiated for a SPECIFIC active DB file. Historical session DBs that never get opened through SessionDB (because the SQLite session moved on) keep their pre-migration schema indefinitely. On my disk: 197 total DBs, only 28 fully migrated, 38 partially, 131 still legacy. BUG C — Aggregator catch-all swallows missing-column errors. analytics.ts getRealBytesStats() loops over every session DB file under the storage root and runs a combined SUM query referencing the new columns. When the column doesn't exist on a legacy DB, the prepare() throws "no such column", the surrounding catch at the end of the loop body skips the ENTIRE DB — not just the bytes_avoided field — so even the LENGTH(data) signal is lost. 131 of 197 DBs contributed zero to all columns. BUG B — Display formula identity-collapse when bytesAvoided == 0. analytics.ts L1991-1993: convBytesWithout = measuredAvoided + measuredReturned + measuredEvent; convBytesWith = max(1, measuredReturned + measuredEvent); With measuredAvoided == 0 (which Bug A + C effectively guaranteed for every conversation that spanned the v1.0.130 upgrade boundary), both sides equal identically → 0% displayed. Fix: extract the schema migration from the SessionDB ctor into a shared top-level helper, call it from the analytics aggregator before each per-DB readonly open. Self-healing: every getRealBytesStats invocation backfills any legacy DBs it scans, so the user's next ctx_stats call silently fixes their historical data — no manual migration command needed. Empirically verified on my machine: 197 DBs → 196 migrated + 1 leftover (probably a lock holder) after one stats call. Display went from 158/158/0% to 168/158/6% — the 6% is honest for this conversation (heavy editing, few routing redirects). Pre-Bug-A baseline showed 98% because the lifetime aggregator was reading historical data that no longer falls into the same per-conversation rollup. ADR-0001 compatible: no EXCLUSIVE pragma, no acquireDbLock — the helper opens writable, runs idempotent PRAGMA-guarded ALTER, closes. SQLite busy_timeout + WAL semantics from SQLiteBase already provide the multi-writer concurrency contract per ADR-0001. Shared `applyMissingSessionEventsColumns(db)` helper now used by both SessionDB ctor AND the new `ensureSessionEventsSchema(dbPath, ctor)` wrapper. Single column list at the top of db.ts — no drift between ctor migration and aggregator backfill. All 449 server tests pass (including the 3 previously-failing ctx_doctor / ctx_index storage e2e tests that turn out to have been casualties of the same schema-skip path — they pass now too). * test(stats): regression coverage for schema-migration recovery (#683 follow-up) Three behavioural tests in tests/session/real-bytes-stats.test.ts pin the v1.0.148 hotfix through the public getRealBytesStats API: 1. **Recovery** — A pre-v1.0.130 legacy-schema session DB (no bytes_avoided / bytes_returned / project_dir columns) with two events on disk MUST contribute its LENGTH(data) signal. Pre-fix: prepare() threw "no such column", catch skipped the WHOLE DB, eventDataBytes == 0 even though LENGTH(data) > 0. Post-fix: signal recovered (assertion bounds 40 < n < 500 — guards against the identity-collapse failure mode without pinning fragile exact byte counts). 2. **In-place migration** — Calling getRealBytesStats against a legacy DB MUST add all five post-v1.0.130 columns (project_dir, attribution_source, attribution_confidence, bytes_avoided, bytes_returned) to the on-disk schema. Asserted via PRAGMA table_xinfo before/after. 3. **Idempotency** — Second aggregator call against an already-migrated DB MUST NOT throw and MUST NOT add new columns. Pins the PRAGMA-guarded early-return contract in applyMissingSessionEventsColumns. RED-GREEN proven: with the ensureSessionEventsSchema call temporarily commented out in analytics.ts, all 3 tests fail; re-enabling makes them pass. Confirms the tests exercise the actual regression path, not just shape. Test helpers (createLegacySessionDb, readSessionEventsColumns) bypass SessionDB's ctor migration via raw better-sqlite3 SQL so the seed DB genuinely reproduces the broken-on-upgrade state observed in the wild. Per CONTRIBUTING line 282, folded into the existing real-bytes-stats.test.ts — no new test file. 14/14 tests in real-bytes-stats.test.ts pass (the new 3 + existing 11). * fix(stats): Bug E+F — META-scoped per-conversation aggregation (#683 follow-up) The v1.0.148 schema-migration fix (903ddd2) unblocked the SUM query on legacy DBs but uncovered a second-order under-attribution bug. Empirical evidence from the reporter's machine: Display Section 1: Without 168 KB / With 158 KB / 6% kept out Real signal: Without 5.1 MB / With 2.2 MB / 56% kept out → 49 percentage points of attribution loss Root cause is two intertwined bugs nested inside the per-conversation aggregator path. BUG E — aggregator scopes by single session_id. A Claude Code conversation routinely spans 80+ session_ids: resume cycles, /compact rebirths, and most importantly the PID sub-process sessions spawned by ctx_execute / ctx_fetch_and_index for each sandboxed run. The renderer at server.ts:3454 was passing only the top-level main session_id, so every sandbox burst's bytes_avoided got dropped. BUG F — sandbox-burst PID-session EVENTS write project_dir=''. A naive project_dir filter on session_events would still miss them. On the reporter's machine the 11 PID-* sessions active in the last 24h carried 475.7 KB of bytes_avoided in events whose event-level project_dir was empty string — even though their META row had the parent cwd. Event-level filtering would lose them. Fix: 1. getRealBytesStats accepts a new `projectDir` option (mutually exclusive with `sessionId`). When set, the SQL uses a META subquery — `WHERE session_id IN (SELECT session_id FROM session_meta WHERE project_dir = ?)` — to pick sibling sessions by META, then sums ALL events for those sessions regardless of the events' own project_dir. Solves Bug E (broader session scope) AND Bug F (META-not-event scoping catches PID bursts whose events write empty project_dir) in one query change. 2. ctx_stats renderer (server.ts) looks up project_dir from the active session's META row, then calls getRealBytesStats with projectDir instead of sessionId. Best-effort: falls back to the old sessionId scope if the META lookup fails for any reason (corrupt DB, missing META row), so the bar never disappears. ADR-0001 compatible: pure read path, no writes, no locks. The META subquery runs against the already-open readonly handle. Test coverage (real-bytes-stats.test.ts): - getRealBytesStats({ projectDir }) sums bytes from EVERY session whose META project_dir matches, including PID-burst sessions whose EVENTS have empty project_dir but whose META has the parent cwd. Excludes sessions from other projects entirely. - The fixture mirrors the real-world Bug F shape: session B has META.project_dir=target but events.project_dir=''. Pre-fix: excluded entirely. Post-fix: bytes attributed correctly. 15/15 real-bytes-stats tests pass. 449/449 server tests pass. * fix(stats): Bug G — Section 1 strict-compression formula (#683 follow-up) Empirical evidence converged the 7-agent EM ops audit on the strict- compression formula for the per-conversation Section 1 bar. Replaces the v1.0.134 SLICE B incidental fix (commitce62275) that folded eventDataBytes into both sides of the Without/With ratio. ROOT CAUSE: eventDataBytes is hook payload metadata written to SessionDB (496 duplicate CLAUDE.md copies, hundreds of tool_response records, etc.). It NEVER enters the model context window — it is analytics infrastructure. After PR #685's Bug A+C+D+E+F fixes let real bytesAvoided flow into the formula, SLICE B crushed the display from the literal compression ratio (~95% on real conversations) down to 56%. THE FORMULA (src/session/analytics.ts ~L2031-2074): if (bytesAvoided + bytesReturned == 0) { // honest empty-state hint, no degenerate bar } else { Without = bytesAvoided + bytesReturned With = max(1, bytesReturned) pct = (1 - With / Without) * 100 } EMPIRICAL (reporter's machine, project_dir scope): bytesAvoided = 2,898 KB | bytesReturned = 140 KB | eventDataBytes = 2,136 KB (excluded) Without = 3,038 KB | With = 140 KB | pct = 95.4% | mult = 22x runtime (was 0% / 6% / 56% across the cascade) Lifetime totals UNCHANGED — they use a different aggregation that includes eventDataBytes correctly. TESTS (RED→GREEN proven): - tests/analytics/format-report.test.ts: SLICE B describe superseded by "Bug G — strict-compression formula" (empty-state, mixed 60%, only-avoided 100%) — 3 GREEN. - tests/session/real-bytes-stats.test.ts: 15 pass (no regression). - tests/core/server.test.ts: 525 pass / 3 pre-existing PR #617 failures unchanged. ADR-0001 compliance: pure read-side formula, no schema, no locks. ADR-0004 (new) documents the formula + SLICE B archaeology + empirical reporter data + consequence (display jumps 56% → 95%). release-notes-v1.0.148.md walks the entire 7-bug cascade.