Both containers up after the chmod fix; synthetic checks green on
three-body and maturin; outage proof passed (servedBy: three-body with
flagg-local skipped). Keep-alive flake fixed: clients send
connection:close, shim declares HTTP/1.1 for the next image build.
Growth alert true positives, root cause 32e6e13a: lost ack after
server commit lets a fresh run_id bypass run-keyed prefix-resync.
Fix step (source_identity+from_offset keying) launched; dedupe and
legacy outbox drain chartered behind it.
Detection unchanged: incident store + fatal OTEL record every finding.
The replay itself (post-outage capture catch-up / Typesense recovery
backfill) still needs its cursor fixed — that work belongs to the
typesense-recovery effort; this stops the pager noise.
Scheduling was never broken. Room-keyed idempotency (24h window) swallowed
every same-day re-drive (07-13: 30 NO-RUN; 07-16 probes NO-RUN 46min after
the room's run) while the flapping GQL API hid the successful 07-16 batch.
Fix: idempotency dropped from voice-call-judge (re-judging is a feature).
Proof: event 01KY0EJ88046NDQZBM10321MZH -> run 01KY0EJ8CAARKPA8N651SCSWM9
COMPLETED -> analysis row 1784574252721 via callTelemetry:sessionDetail.
Retro + A/B unblocked.
Bounce-minting lands on whatever was in flight, so the five-prefix fence
misses fresh mints (9 items across 5 arbitrary functions post-purge).
Authorized by Joel 2026-07-20 for window-scoped runs; ceiling unchanged.
sendHardAlert threw when the terminal delivery receipt timed out, AFTER
the platform send succeeded. Callers (capture-growth, startup-budget,
critical-db) retried the whole notify path, re-sending the same Telegram
alert on every event/retry — the 2026-07-20 spam incident. Receipt
failure now returns {sent, receiptConfirmed:false} and emits a warn
otel event; only a failed send throws.
NOTE: takes effect on next system-bus worker publish. Do not publish
from a dirty tree — other sessions have uncommitted system-bus work.
Bounces mint envID-less items (595→597 across one bounce), so an exact
pre-pinned count is stale by apply time. Joel authorized window-scoped
re-scan 2026-07-20: same five function prefixes + missing-envID criteria,
hard ceiling 650, manifest regenerated at scan. Default exact-count mode
unchanged.
Five pi workers investigated every post-reboot workspace. Verdicts:
15 shipped, 11 blocked-on-Joel, 6 active, 4 interesting with
uncommitted work at risk, 1 deliberately abandoned. Report with
inline continuation prompts at
.brain/projects/herdr-reboot-triage/herdr-reboot-triage-report.svx
Dispatcher gate fix, wake reconciler, and both chain revives proven with
live receipts. Purge re-blocked on the minter fix (e40dd493) deploying
and a fresh stable count.
recover() restarted BOTH Inngest and the host worker for every actionable
reason and advanced the cooldown only when both kicks succeeded — a
failing worker kick meant every 60s pass re-killed healthy Inngest (54
forced replacements after the 2026-07-19 reboot; boot-timeline asset in
.brain/projects/reboot-hardening). Now: only inngest_unavailable touches
the event server; worker_unavailable/index_lag_exceeded kick the worker
alone; the cooldown stamps before success evaluation because kickstart -k
is destructive the moment it runs. Bounces also mint envID-less poisoned
queue items (595→597 across one bounce), so stopping storm restarts
chokes the poison source. Tests updated to the new doctrine + an
inngest-scoping case.
pane/schedule.reconcile: cron */5 + startup event; quarantines malformed
registry entries, re-emits pane.schedule.due (existing scheduleId,
late/recovered flags) past a 10-min grace, hdel only after emit succeeds
(separate memoized steps, replay-safe). onFailure on pane/schedule is
observability-only and never deletes the pending entry — that entry is
what recovery needs (the de16dde1 reboot case). 6 tests.
Part of .brain/projects/reboot-hardening; verified by adversarial review
in the reboot-automation-fixes workflow.
recover() only advances the cooldown when both Inngest and worker kicks
succeed, so a failing worker kick re-kills Inngest every 60s pass; plus
index-lag wrongly restarts the event server. New step charters the fix.
Chartered from the 2026-07-19 reboot fallout diagnosis. Finish line and
purge/exec authorizations confirmed by Joel 2026-07-20. Two steps in
flight via the reboot-automation-fixes workflow; notify-wait step closed
(stale binary, rebuilt 07-19).
The capture replay verifier read the retired runs_dev collection, which
schema-drifted (missing source_identity -> HTTP 400) and false-conflicted
on fresh accepts. Query the canonical session index directly instead:
- exactRun/capturedSiblings are bun:sqlite queries against
~/.joelclaw/search/sessions.db (--sessions-db / SESSION_INDEX_PATH)
- resolve run-store-relative jsonl_path against RUN_STORE_PATH
(~/.joelclaw/runs-dev), same convention as joelclaw sessions
- open read-write with PRAGMA query_only=ON: WAL sidecars vanish between
writer sessions and a readonly open then fails with SQLITE_CANTOPEN
- tests seed a fixture sessions.db instead of faking a Typesense server
Six-hour rebuild through the hardened builder with stored-incident
failure alerts (no error-text keying), 8h build and 24h high-water
budgets, and a freshness contract behaviorally locked to the port's
rule by fixtures on both sides. Refuted twice by adversarial
verification (text-keyed IDs, split contract), reworked, re-verified
CONFIRMED behaviorally; the remaining rule duplication is chartered as
unify-freshness-rule, blocked on the replication worker's territory.
Joel-approved cutover (2026-07-20): sessions.db is the only
full-transcript chunk index. The index-chunks Typesense import is
removed, ensureCollections no longer recreates run_chunks_dev, and the
stale web polling hint now points at joelclaw sessions search. Typesense
keeps runs_dev metadata for provenance and health.
joelclaw recall and knowledge search open ~/.joelclaw/search/critical.db
first (29,906 docs: observations, Brain pages, vault, knowledge,
archived memory) with per-source freshness and clean fallback to
Typesense on any SQLite failure — proven through the compiled binary.
Builder holds an exclusive lock, refuses degraded or collapsed sources
(caught five malformed live pages), and swaps atomically. Observer refs
are honest source-labels (resolvableInRunsDev: false).
Refuted once by adversarial verification (fallback crash, unsafe
builder, fake backlinks); reworked and re-verified CONFIRMED on every
count. Receipts: critical-port-verification.svx
Typesense recovered (~04:00), so real parity replaces the raw-truth
waiver. Wiring step owns WAL flip, append consumer under real retries,
grep-proven reader inventory and switch, parity vs healthy
run_chunks_dev, rollback plan, and the review packet. Cutover review
now blocks on it.
search/capture-prefix-growth-alert: source-scoped incidents (one alert
per incident, explicit clear + quiet-period reset), bounded Redis range
ledger. search/typesense-startup-budget: tracks typesense:runs_dev via
a real projection query; outage clock clears only on query success;
strict budget parsing. /api/runs/health fails truthfully (503) on
stale, empty, outage, and monitor-error states.
Refuted once by adversarial verification (per-Run-pair storms,
process-only budget clock, ok:true on monitor error); reworked and
re-verified CONFIRMED on all five counts. Receipts:
.brain/projects/typesense-reboot-recovery/alerts-verification.svx
Fallback via Effect.catchAll with compiled-binary failure tests,
exclusive builder lock, 80% count-collapse publication gate (refused a
real rebuild over five malformed live pages), per-source freshness,
observerRunId label replacing the false runBacklink claim.
Verifier confirmed the July 19 signature fires and patched paths stay
quiet, but found per-Run-pair dedup storms on continued incidents, the
budget clock clears on process health while runs_dev is still 503, and
/api/runs/health hides monitor failure. Conditions 5-7 added; worker
re-engaged.
Verifier confirmed the happy path but found the fallback crashes on
missing/corrupt critical.db, the builder can replace a healthy db with
an empty one, concurrent builds clobber, aggregate freshness masks
stale data, and runBacklink does not resolve into runs-dev. Conditions
5-8 added; worker re-engaged.
FTS5 sessions.db from the manifest keep-set: 134,013 Runs, 447,609
chunks, chunking via packages/memory/src/chunking.ts directly.
Adversarially verified: independent re-hash of all skipped bodies,
independent chunk parse, integrity clean. Cutover preconditions
recorded on the review step (WAL flip, reader/append wiring, parity).
Joel Hooks