evaluateCriticalDbFreshness + CRITICAL_DB_REQUIRED_SOURCES exported
from packages/memory; the port and the maintenance alert import it and
their duplicated copies are gone. Both fixture suites pass byte-
unchanged — behavior identical by construction. pnpm-lock entry for the
new cli->memory workspace dep rides with the pending message-event lock
update.
Closes the ack-loss inflation class (root cause 32e6e13a): the server
resyncs any post matching an existing (source_identity, from_offset)
regardless of run_id; clients write pending entries on ack-ambiguous
paths; the session-index consumer rejects ANY non-exact-start range
intersection loudly. QA-verified with the live failure chain replayed
plus retry/concurrency/isolation variants; the one NOT PROVEN claim
(different-start overlap) fixed with the verifier's fixture as a
regression test.
Capture clients (pi, claude, codex) send byte-accurate from_offset,
to_offset, jsonl_sha256, and deterministic source_identity. Stale
cursors coalesce into one pending outbox segment under one Run ID.
capture-outbox-replay.ts is the only replay implementation; the audit
script fails closed on legacy flags and the scheduled nas-backup caller
is migrated. Raw Run writes reject divergent redelivery; a larger
proven-prefix body returns accepted_prefix without overwrite.
Verified adversarially (multi-byte offsets, repeated stale cursors,
fixture-only replay): 20 bun tests + 2 vitest + tsc clean. Receipts in
.brain/projects/typesense-reboot-recovery/patch-verification.svx.
Replaces the dev-bearer table with PDS-backed App Password authentication
per ADR-0243 Rule 20, scoped to single-user (Joel only) for now.
packages/system-bus/src/lib/pds.ts
+ pdsCreateAppPassword — calls com.atproto.server.createAppPassword on
the caller's session; returns { name, password, did, handle }
+ pdsRevokeAppPassword — revokes by name
+ pdsValidateAppPassword — one-shot validation via createSession; used
at register time only, NOT per-request (hash lookup is cheaper +
safer, and doesn't create server-side session state at PDS)
packages/memory/src/schemas/machines.ts — NEW Typesense schema for
machines_dev collection: id, user_id, did, handle, machine_name,
app_password_name, app_password_sha256, created_at, last_seen_at,
revoked_at
apps/web/lib/memory-auth.ts — NEW auth middleware
authenticateMemoryRequest(req) → MemoryIdentity | null
Hot path: sha256(bearer) → Typesense machines_dev lookup →
{ user_id, machine_id, did, source: "app-password" }
No PDS roundtrip per request. App Password validity is established
at register time; the hash is the identity key thereafter.
Dev-bearer fallback retained during transition (MEMORY_DEV_BEARER_TOKENS
env var); effectively disappears when that var is empty in prod.
Phase 3.5 TODO: users_dev collection (DID→user_id map is hardcoded for
now).
apps/web/app/api/runs/*/route.ts — all 6 handlers now call
authenticateMemoryRequest instead of the inline DEV_BEARER_TOKENS
table + local authenticate() helper. Rule 4 privacy filtering is
unchanged (still keyed on auth.user_id from the middleware).
scripts/joelclaw-machine-register.ts — NEW provisioning CLI
Flow: load cached PDS session from ~/.joelclaw/pds-session.json →
call com.atproto.server.createAppPassword → sha256(plaintext) →
upsert machines_dev row → back up old auth.json → write new
auth.json (0600) with the plaintext bearer. Ready to register Panda
once the infra is back.
Verification status (pending Typesense reachability):
[x] pds.ts compiles; curl-tested createAppPassword directly against
PDS (returned valid password)
[x] Bun fetch works against PDS directly (ruled out fetch-lib bugs)
[x] All route handlers typecheck + biome clean
[x] Auth middleware compiles
[ ] End-to-end: register panda → POST /api/runs with new bearer →
Run indexes → search returns it — BLOCKED on Colima SSH tunnel
which is flapping (ssh 192.168.64.2:22 timing out; matches the
2026-04-17 healer-suicide-loop memory note)
When infra comes back:
TYPESENSE_API_KEY=<...> joelclaw-machine-register --name panda --user joel
then POST /api/runs with the new bearer to confirm.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Phase 1 is production-directed; the spike collection (run_chunks_spike)
has served its purpose and is decoupled from the active code path. The
45 spike chunks can be dropped per ADR-0243's verification checkbox
once we're ready for the final cutover.
E2E smoke now passes:
POST /api/runs (169KB fixture) → 82 chunks indexed → hybrid search
returns 5/11 hits in 202ms (192ms embed + 10ms Typesense).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the central ingest path for agent Run capture per ADR-0243 build order
steps 5 and 6:
packages/memory
- src/schemas/runs.ts — Typesense `runs_dev` collection schema covering
the full Run row from types.ts
- src/nas.ts — blob writer honoring Rule 11 (user-partitioned paths);
env var MEMORY_RUN_STORE defaults to ~/.joelclaw/runs-dev/ for local
development, set to /nas/memory/runs in production
packages/system-bus
- new memory/run.captured + memory/run.indexed event contracts
- new functions/memory/run-captured.ts — receives event, loads jsonl
from NAS (or inline payload), format-detects, chunks via per-turn
chunker, embeds each chunk at ingest-realtime priority through
@joelclaw/inference-router (so queries preempt), writes chunks to
run_chunks_dev and Run row to runs_dev, emits OTEL + fanout indexed
event; concurrency:4, retries:3; ensures collections exist idempotently
- @joelclaw/memory added to dependencies
apps/web
- new app/api/runs/route.ts — POST handler; dev bearer token auth
(hardcoded allowlist, PDS flow lands Phase 3), writes jsonl to NAS
via @joelclaw/memory writeRunBlob, fires memory/run.captured via
HTTP to local Inngest, returns 202 HATEOAS envelope with run_id +
_links + next_actions
- @joelclaw/inference-router + @joelclaw/memory added to dependencies
Still v1 / dev scope:
- Dev bearer token, not PDS App Password (Phase 3)
- _dev collection suffix, will alias to run_chunks_current after cutover
- No search endpoint yet (next slice)
- No capture hooks yet (Phase 4)
- No entity enrichment yet (Phase 6)
Typecheck clean, biome clean, 12 chunking tests still green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Validates ADR-0243 Rule 9a with a decisive contention test. Priority-lane
embed scheduling measured at 338ms query latency with 98 ingest-bulk embeds
queued — a ~25x improvement over the 8-10s observed in the spike without
priority discipline.
@joelclaw/inference-router
- new src/embeddings.ts — PriorityEmbedClient with single-writer queue
honoring query > ingest-realtime > ingest-bulk; embed() returns queued_ms,
compute_ms, total_ms for observability
- new __tests__/embeddings-priority.test.ts — drives 100 ingest-bulk embeds
then submits one query-priority embed mid-stream; asserts query < 3s
(observed 338ms); skips gracefully when Ollama is unreachable
- vitest added as devDependency; tests pass under vitest 4
@joelclaw/memory
- new __tests__/chunking.test.ts — 12 tests locking in the tool_result
role detection fix that landed during the spike, format detection
(claude-code vs pi), meta-entry filtering, thinking-content handling,
oversized-turn sub-chunk splitting, and malformed-JSON tolerance
packages/system-bus
- memory/embed.requested + memory/embed.completed event contracts added
to Events map
- new functions/memory/embed.ts — Inngest durable wrapper, concurrency:1
(matches Ollama internal serialization), priority.run expression mapping
the three lanes to numeric scheduling scores, OTEL emission + fanout
completed event; calls @joelclaw/inference-router under the hood
Query path bypasses Inngest and calls the router directly (fast, no
durability needed). Only ingest paths fire memory/embed.requested. Both
converge on the same priority queue — that's where the contention is
resolved.
ADR-0243 status: accepted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces the raw-Run capture architecture (ADR-0243 in the Vault):
- CONTEXT.md at repo root — 13 terms + 21 architectural rules grilled out via the
domain-model skill (central ingestion, private-by-default Share Grants,
PDS+App Password identity, NAS-authoritative storage with Typesense as
rebuildable index, qwen3-embedding:8b @ 768-dim Matryoshka, agent-first API).
- packages/memory/ — types, per-turn chunker (claude-code + pi format detection,
tool-result role fix), Ollama embedding client with concurrency pool, Typesense
run_chunks schema, barrel exports. Mirrors @joelclaw/telemetry hexagonal pattern.
- scripts/memory-spike/ — end-to-end validation against real data:
* ingest.ts ingests a single jsonl into run_chunks_spike
* bulk-ingest.ts walks ~/.claude/projects/ and ~/.pi/agent/sessions/ with
sha256 dedup via ~/.joelclaw/memory-spike-ingested.jsonl
* search.ts supports hybrid / semantic / keyword modes
* README.md documents observations from the validation run (~708 chunks from
a 1247-line claude-code session at 1.2 ch/s sequential / 0.9 ch/s observed
under bulk concurrency; ~420ms end-to-end semantic queries; retrieval
quality validated on real Runs).
Typecheck clean, biome clean. No production infrastructure affected; spike
writes to isolated run_chunks_spike Typesense collection on the joelclaw
cluster (cleanly deletable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shitratgit[bot]