Adds @<machine> after the runtime badge in tree output so cross-Machine
activity is obvious at a glance. Stable per-Machine color via content
hash (bright colors 91-96 range) — Panda consistently appears in one
hue, dark-wizard in another, without a lookup table.
Also rsync'd updated script to dark-wizard's ~/.joelclaw/bin/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Recover pdf-brain + docs-api after a rebuild left Typesense with partial
old data. Build a resumable chunked importer so the 3GB docs_chunks_v2
JSONL (inline ollama vectors) can be replayed without paying for embeddings
and without getting wedged by server-side stalls.
Two root causes fixed in k8s/typesense.yaml so this doesn't repeat:
- "Too many open files" at ~146k HNSW docs looked identical to a raft
ERROR state. Fix: wrap the server command with ulimit -n 1048576.
- Talos container has a 4 GiB hard cap regardless of Colima VM size,
which pegged the whole cluster once otel_events passed ~350k docs.
Raised Typesense pod limit from 4 Gi -> 8 Gi and left the live
docker-update runbook in skills/k8s/SKILL.md.
All 19 source collections restored, post-rebuild backup replayed on top
with emplace, and public /api/docs/search, joelclaw knowledge search,
and joelclaw otel search all verified.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the dev-bearer table with PDS-backed App Password authentication
per ADR-0243 Rule 20, scoped to single-user (Joel only) for now.
packages/system-bus/src/lib/pds.ts
+ pdsCreateAppPassword — calls com.atproto.server.createAppPassword on
the caller's session; returns { name, password, did, handle }
+ pdsRevokeAppPassword — revokes by name
+ pdsValidateAppPassword — one-shot validation via createSession; used
at register time only, NOT per-request (hash lookup is cheaper +
safer, and doesn't create server-side session state at PDS)
packages/memory/src/schemas/machines.ts — NEW Typesense schema for
machines_dev collection: id, user_id, did, handle, machine_name,
app_password_name, app_password_sha256, created_at, last_seen_at,
revoked_at
apps/web/lib/memory-auth.ts — NEW auth middleware
authenticateMemoryRequest(req) → MemoryIdentity | null
Hot path: sha256(bearer) → Typesense machines_dev lookup →
{ user_id, machine_id, did, source: "app-password" }
No PDS roundtrip per request. App Password validity is established
at register time; the hash is the identity key thereafter.
Dev-bearer fallback retained during transition (MEMORY_DEV_BEARER_TOKENS
env var); effectively disappears when that var is empty in prod.
Phase 3.5 TODO: users_dev collection (DID→user_id map is hardcoded for
now).
apps/web/app/api/runs/*/route.ts — all 6 handlers now call
authenticateMemoryRequest instead of the inline DEV_BEARER_TOKENS
table + local authenticate() helper. Rule 4 privacy filtering is
unchanged (still keyed on auth.user_id from the middleware).
scripts/joelclaw-machine-register.ts — NEW provisioning CLI
Flow: load cached PDS session from ~/.joelclaw/pds-session.json →
call com.atproto.server.createAppPassword → sha256(plaintext) →
upsert machines_dev row → back up old auth.json → write new
auth.json (0600) with the plaintext bearer. Ready to register Panda
once the infra is back.
Verification status (pending Typesense reachability):
[x] pds.ts compiles; curl-tested createAppPassword directly against
PDS (returned valid password)
[x] Bun fetch works against PDS directly (ruled out fetch-lib bugs)
[x] All route handlers typecheck + biome clean
[x] Auth middleware compiles
[ ] End-to-end: register panda → POST /api/runs with new bearer →
Run indexes → search returns it — BLOCKED on Colima SSH tunnel
which is flapping (ssh 192.168.64.2:22 timing out; matches the
2026-04-17 healer-suicide-loop memory note)
When infra comes back:
TYPESENSE_API_KEY=<...> joelclaw-machine-register --name panda --user joel
then POST /api/runs with the new bearer to confirm.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Inspired by pi's /tree (badlogic/pi-mono). Pi's tree is per-turn within
one session; this one is per-Run across all sessions and all runtimes.
Every Run carries parent_run_id + root_run_id (Rule 3), so the forest
exists in the schema — we just didn't have a view.
apps/web/app/api/runs/forest/route.ts
GET /api/runs/forest
?since=<epoch_ms> default: 7d window
?runtime=pi,claude-code comma-separated filter
?root_run_id=<id> zoom into a subtree
?limit=<n> cap 1-250 (Typesense per_page max)
Returns a flat list of Runs sorted by started_at + metadata for
client-side tree assembly. Privacy-filtered by readable_by (Rule 4).
scripts/joelclaw-runs-tree.ts → ~/.bun/bin/joelclaw-runs-tree
Terminal renderer with ANSI-colored runtime badges, box-drawing tree
connectors, relative timestamps, turn counts, tag chips, intent
clipped to 80 chars, clean sort order (started_at asc per bucket).
Orphan-root handling: a Run whose parent_run_id isn't in the result
set (trimmed by limit/since/runtime filters) becomes a visible root
rather than being dropped. Deterministic output.
Verified live:
$ joelclaw-runs-tree --since=6h --limit=100
3 Runs (3 roots) since 2026-04-19T20:09
├─ claude-code 22m ago 2t #phase1-v3
│ third attempt with typesense key in env
├─ claude-code 21m ago 1t #smoke-e2e
│ test
└─ claude-code 18m ago 82t #smoke-e2e
Explore the project at /Users/joel/Code/joelhooks/joelclaw…
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
You work on sessions for days. SessionEnd never fires in that workflow,
so capture would never happen. Pivoting to Stop (per Rule 3 — Run = one
invocation), with per-session byte-offset state so each Stop only ships
the delta.
~/.joelclaw/session-state.json tracks { last_byte_offset, last_run_id,
turn_count, last_captured_at } keyed by Claude Code session_id. Each
capture uses the prior last_run_id as parent_run_id, so a long session
becomes a chain of per-turn Runs with tree linkage intact.
Safety: ALWAYS exits 0 (never blocks Claude continuation). Failures
outbox the delta and leave state unadvanced so the next Stop retries
the same range.
Verified manually:
- 3.4MB / 694-turn backfill of this live session → 1 POST, state
advanced to current EOF
- 15 seconds later, second invocation captured a 504-byte delta
(the conversation that happened in between) with parent_run_id
pointing back at the first
settings.json flipped from SessionEnd → Stop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the feedback loop: Claude Code sessions now flow automatically
into the memory archive, and Joel can search from any terminal without
curl.
scripts/joelclaw-capture-session.ts — Claude Code SessionEnd hook target.
Reads the hook context JSON from stdin (session_id + transcript_path),
loads bearer from ~/.joelclaw/auth.json, POSTs the transcript to
/api/runs. Outboxes to ~/.joelclaw/outbox/ on network failure
(documented Rule 8 path). Exits 0 for success/skip, 2 for network-
recovered, 1 for hard failures.
scripts/joelclaw-runs-search.ts — terminal-native hybrid search.
Reads auth from ~/.joelclaw/auth.json, POSTs /api/runs/search with
the same bearer, formats hits with ANSI color + relative timestamps.
Supports --mode, --limit, --tag, --runtime, --json.
Installation (this commit only sets up the repo-canonical scripts;
symlinks and settings.json hook are set up manually):
ln -sf ~/Code/joelhooks/joelclaw/scripts/joelclaw-capture-session.ts \\
~/.bun/bin/joelclaw-capture-session
ln -sf ~/Code/joelhooks/joelclaw/scripts/joelclaw-runs-search.ts \\
~/.bun/bin/joelclaw-runs-search
And ~/.claude/settings.json gets a SessionEnd hook pointing at the
capture script.
Proper Phase 4 graduation (pi extensions, codex hooks, Outbox drain
timer, incremental capture) is still ahead — this slice is the
ambient-capture minimum that turns the system on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Completes the Phase 1 ingest + retrieval loop per ADR-0243 build order step 7.
apps/web
- new app/api/runs/search/route.ts — hybrid Typesense search with the
full agent-first API surface (Rule 12 + 13):
* Auto-applied privacy filters (user_id + readable_by) from the
bearer token — NEVER from the request body (Rule 4)
* Tag filters default to AND semantics
* Modes: hybrid (default), semantic, keyword
* Embeds the query at priority="query" so it preempts any pending
ingest work in the Ollama queue (Rule 9a)
* HATEOAS envelope with _links, next_actions, and timing breakdown
(query_embed_ms, query_queued_ms, typesense_ms, total_ms)
scripts/memory-spike
- new smoke-e2e.ts — full pipeline validation:
1. POST /api/runs with a real claude-code jsonl fixture
2. Poll Typesense for chunks matching the new run_id
3. POST /api/runs/search filtered to the smoke-e2e tag
4. Assert the ingested Run is findable
Run against a live dev stack (Next.js on :3000, worker deployed,
Typesense + Ollama + Inngest healthy).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces the raw-Run capture architecture (ADR-0243 in the Vault):
- CONTEXT.md at repo root — 13 terms + 21 architectural rules grilled out via the
domain-model skill (central ingestion, private-by-default Share Grants,
PDS+App Password identity, NAS-authoritative storage with Typesense as
rebuildable index, qwen3-embedding:8b @ 768-dim Matryoshka, agent-first API).
- packages/memory/ — types, per-turn chunker (claude-code + pi format detection,
tool-result role fix), Ollama embedding client with concurrency pool, Typesense
run_chunks schema, barrel exports. Mirrors @joelclaw/telemetry hexagonal pattern.
- scripts/memory-spike/ — end-to-end validation against real data:
* ingest.ts ingests a single jsonl into run_chunks_spike
* bulk-ingest.ts walks ~/.claude/projects/ and ~/.pi/agent/sessions/ with
sha256 dedup via ~/.joelclaw/memory-spike-ingested.jsonl
* search.ts supports hybrid / semantic / keyword modes
* README.md documents observations from the validation run (~708 chunks from
a 1247-line claude-code session at 1.2 ch/s sequential / 0.9 ch/s observed
under bulk concurrency; ~420ms end-to-end semantic queries; retrieval
quality validated on real Runs).
Typecheck clean, biome clean. No production infrastructure affected; spike
writes to isolated run_chunks_spike Typesense collection on the joelclaw
cluster (cleanly deletable).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Create a standalone Bun script that paginates Slack conversations.history and conversations.replies for C0211NSK3TP, caches users.info display-name lookups, converts Slack mrkdwn to markdown, writes monthly vault files plus index.md under ~/Vault/Areas/ai-hero/runbook/slack, and keep the script compiling with bunx tsc --noEmit --strict.
- posts.ts: strip leading frontmatter/metadata before rendering
- content-review.ts: normalize content on fetch/persist to prevent
future writes from reintroducing raw frontmatter
- seed-articles.ts: defensive strip on seed path
Fixes frontmatter metadata rendering as visible article body text.
Pi's custom SYSTEM.md branch doesn't inject identity files separately.
Append them directly so they're in the system prompt.
SYSTEM-base.md is the base prompt without identity files.
scripts/build-system-prompt.sh rebuilds SYSTEM.md for any role.
Joel Hooks