Dragan Spiridonov 874ae39c6a feat(learning,bridge): dream_insights retention + bridge cursor guard (#488 Phase 3) ...

Two small defensive changes that close the remaining cleanup-and-race
gaps from #488. No new ADRs needed — both are additive within
already-documented subsystem boundaries.

C.2 — dream_insights retention pruner
=====================================

The dream_insights schema has no expires_at column and no foreign-key
cascade fires on regular cycle delete (cycles aren't routinely purged
either). Without pruning the table grows linearly with hook activity —
V0.03 evidence in #488 documents 120+ rows in 30h, most with
applied=0. Over months `getPendingInsights` queries get slower and
vector embeddings (if attached) grow unbounded.

src/learning/dream/dream-insights-pruner.ts (NEW)
  - pruneStaleDreamInsights(db, { retentionDays = 30 }):
      DELETE FROM dream_insights
        WHERE applied = 0
          AND created_at < datetime('now', '-N days')
  - Applied insights stay forever (audit trail).
  - Recently-created unapplied insights stay regardless of applied
    state — they might still be picked up by a future applyInsight.
  - Guards against the table not existing (fresh init before the
    dream schema is applied). Idempotent.

src/workers/workers/learning-consolidation.ts
  - new Step 8 in runContinuousLearningLoop after the mineExperiences
    sweep: prune insights, emit a finding if pruned > 0, surface
    `dreamInsightsPruned` in ConsolidationResult and domainMetrics.
  - Failures isolated per-try-block (worker continues with the rest
    of its lifecycle work).

C.3 — bridge cursor monotonic guard
===================================

CapturedExperienceBridge.drain wrote `this.cursor` to the kv unconditionally.
Cross-process race: two daemons (possible before #488 B.1 fixed the
pidfile, still possible if user manually spawns extras) drain
overlapping rows, then the "slower" process overwrites the kv with its
older value. Next drain re-publishes already-drained rows; downstream
plugins without event-ID dedup see duplicates.

src/bridge/captured-experience-bridge.ts
  - Before persisting the cursor, read back what's stored and write
    Math.max(this.cursor, persisted). The cursor never regresses.
  - Read-then-max-write is a TOCTOU but it collapses the regression
    window to milliseconds — sufficient for the duplicate-daemon edge
    case. Stronger guarantees (SQL UPDATE WHERE, advisory lock) would
    bypass the MemoryBackend abstraction without a real-world benefit
    post-B.1.
  - Cursor persistence is now best-effort (catch and warn). On failure
    the next bridge boot re-reads from kv and resumes — a missed write
    just means we re-publish a small batch, idempotent for ID-dedup'd
    consumers.

Tests
=====

tests/unit/learning/dream/dream-insights-pruner.test.ts (NEW)
  - 8 tests: deletes stale unapplied, keeps applied regardless of age,
    keeps recent unapplied, honors custom retention, no-op on empty
    table, no-op when table doesn't exist, default 30-day retention,
    idempotent re-run

tests/unit/bridge/captured-experience-bridge.test.ts (extended)
  - 3 new tests in describe('cursor monotonic guard'):
      - does not regress when another writer is ahead
      - advances normally when this bridge is ahead
      - sensible fallback when no persisted value exists

594/594 tests pass across CLI, workers, bridge, learning, init/phases,
architecture, and kernel suites; build clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

2026-05-15 14:39:41 +00:00

..

captured-experience-bridge.test.ts

feat(learning,bridge): dream_insights retention + bridge cursor guard (#488 Phase 3)

2026-05-15 14:39:41 +00:00