Commit Graph

337 Commits

Author SHA1 Message Date
Joel Hooks cafdf2b45e feat(memory): Phase 3 — PDS App Password auth (code-complete; verify pending infra)
Replaces the dev-bearer table with PDS-backed App Password authentication
per ADR-0243 Rule 20, scoped to single-user (Joel only) for now.

packages/system-bus/src/lib/pds.ts
  + pdsCreateAppPassword — calls com.atproto.server.createAppPassword on
    the caller's session; returns { name, password, did, handle }
  + pdsRevokeAppPassword — revokes by name
  + pdsValidateAppPassword — one-shot validation via createSession; used
    at register time only, NOT per-request (hash lookup is cheaper +
    safer, and doesn't create server-side session state at PDS)

packages/memory/src/schemas/machines.ts — NEW Typesense schema for
  machines_dev collection: id, user_id, did, handle, machine_name,
  app_password_name, app_password_sha256, created_at, last_seen_at,
  revoked_at

apps/web/lib/memory-auth.ts — NEW auth middleware
  authenticateMemoryRequest(req) → MemoryIdentity | null
  Hot path: sha256(bearer) → Typesense machines_dev lookup →
    { user_id, machine_id, did, source: "app-password" }
  No PDS roundtrip per request. App Password validity is established
  at register time; the hash is the identity key thereafter.
  Dev-bearer fallback retained during transition (MEMORY_DEV_BEARER_TOKENS
  env var); effectively disappears when that var is empty in prod.
  Phase 3.5 TODO: users_dev collection (DID→user_id map is hardcoded for
  now).

apps/web/app/api/runs/*/route.ts — all 6 handlers now call
  authenticateMemoryRequest instead of the inline DEV_BEARER_TOKENS
  table + local authenticate() helper. Rule 4 privacy filtering is
  unchanged (still keyed on auth.user_id from the middleware).

scripts/joelclaw-machine-register.ts — NEW provisioning CLI
  Flow: load cached PDS session from ~/.joelclaw/pds-session.json →
  call com.atproto.server.createAppPassword → sha256(plaintext) →
  upsert machines_dev row → back up old auth.json → write new
  auth.json (0600) with the plaintext bearer. Ready to register Panda
  once the infra is back.

Verification status (pending Typesense reachability):
  [x] pds.ts compiles; curl-tested createAppPassword directly against
      PDS (returned valid password)
  [x] Bun fetch works against PDS directly (ruled out fetch-lib bugs)
  [x] All route handlers typecheck + biome clean
  [x] Auth middleware compiles
  [ ] End-to-end: register panda → POST /api/runs with new bearer →
      Run indexes → search returns it — BLOCKED on Colima SSH tunnel
      which is flapping (ssh 192.168.64.2:22 timing out; matches the
      2026-04-17 healer-suicide-loop memory note)

When infra comes back:
  TYPESENSE_API_KEY=<...> joelclaw-machine-register --name panda --user joel
  then POST /api/runs with the new bearer to confirm.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 20:12:04 -07:00
Joel Hooks 90e283ccdd feat(memory): GET /api/runs/forest + joelclaw-runs-tree — global pi-tree
Inspired by pi's /tree (badlogic/pi-mono). Pi's tree is per-turn within
one session; this one is per-Run across all sessions and all runtimes.
Every Run carries parent_run_id + root_run_id (Rule 3), so the forest
exists in the schema — we just didn't have a view.

apps/web/app/api/runs/forest/route.ts
  GET /api/runs/forest
    ?since=<epoch_ms>           default: 7d window
    ?runtime=pi,claude-code     comma-separated filter
    ?root_run_id=<id>           zoom into a subtree
    ?limit=<n>                  cap 1-250 (Typesense per_page max)

  Returns a flat list of Runs sorted by started_at + metadata for
  client-side tree assembly. Privacy-filtered by readable_by (Rule 4).

scripts/joelclaw-runs-tree.ts → ~/.bun/bin/joelclaw-runs-tree
  Terminal renderer with ANSI-colored runtime badges, box-drawing tree
  connectors, relative timestamps, turn counts, tag chips, intent
  clipped to 80 chars, clean sort order (started_at asc per bucket).

Orphan-root handling: a Run whose parent_run_id isn't in the result
set (trimmed by limit/since/runtime filters) becomes a visible root
rather than being dropped. Deterministic output.

Verified live:
  $ joelclaw-runs-tree --since=6h --limit=100
  3 Runs (3 roots) since 2026-04-19T20:09
  ├─ claude-code 22m ago 2t #phase1-v3
  │     third attempt with typesense key in env
  ├─ claude-code 21m ago 1t #smoke-e2e
  │     test
  └─ claude-code 18m ago 82t #smoke-e2e
        Explore the project at /Users/joel/Code/joelhooks/joelclaw…

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 19:09:44 -07:00
Joel Hooks 47188aebd2 feat(memory): Phase 1 slice — Run traversal endpoints
Adds the three GET traversal endpoints from ADR-0243 Rule 13 (D shape):

apps/web/app/api/runs/[id]/route.ts
  - GET: fetch the Run metadata row from runs_dev
  - Enforces Rule 4 at read time by checking readable_by.includes(caller)
  - 404 (not 403) on unauthorized — leaks zero information about existence

apps/web/app/api/runs/[id]/jsonl/route.ts
  - GET: stream the full jsonl transcript from NAS
  - Verifies readable_by on the Run row before streaming
  - 410 Gone if the blob is missing on disk (distinct from 404 "no Run")
  - Returns application/x-ndjson with Content-Length; zero buffering

apps/web/app/api/runs/[id]/descendants/route.ts
  - GET: walk a Run's subtree via root_run_id filter
  - Returns root + all descendants ordered by started_at ascending
  - Lets agents reconstruct a workload-rig DAG or a gateway session tree
    in one call

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 18:29:24 -07:00
Joel Hooks e269155fbe feat(memory): Phase 1 slice — POST /api/runs/search + E2E smoke script
Completes the Phase 1 ingest + retrieval loop per ADR-0243 build order step 7.

apps/web
  - new app/api/runs/search/route.ts — hybrid Typesense search with the
    full agent-first API surface (Rule 12 + 13):
      * Auto-applied privacy filters (user_id + readable_by) from the
        bearer token — NEVER from the request body (Rule 4)
      * Tag filters default to AND semantics
      * Modes: hybrid (default), semantic, keyword
      * Embeds the query at priority="query" so it preempts any pending
        ingest work in the Ollama queue (Rule 9a)
      * HATEOAS envelope with _links, next_actions, and timing breakdown
        (query_embed_ms, query_queued_ms, typesense_ms, total_ms)

scripts/memory-spike
  - new smoke-e2e.ts — full pipeline validation:
      1. POST /api/runs with a real claude-code jsonl fixture
      2. Poll Typesense for chunks matching the new run_id
      3. POST /api/runs/search filtered to the smoke-e2e tag
      4. Assert the ingested Run is findable
    Run against a live dev stack (Next.js on :3000, worker deployed,
    Typesense + Ollama + Inngest healthy).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 18:23:57 -07:00
Joel Hooks a02ece9702 feat(memory): Phase 1 slice — memory/run.captured + POST /api/runs
Adds the central ingest path for agent Run capture per ADR-0243 build order
steps 5 and 6:

packages/memory
  - src/schemas/runs.ts — Typesense `runs_dev` collection schema covering
    the full Run row from types.ts
  - src/nas.ts — blob writer honoring Rule 11 (user-partitioned paths);
    env var MEMORY_RUN_STORE defaults to ~/.joelclaw/runs-dev/ for local
    development, set to /nas/memory/runs in production

packages/system-bus
  - new memory/run.captured + memory/run.indexed event contracts
  - new functions/memory/run-captured.ts — receives event, loads jsonl
    from NAS (or inline payload), format-detects, chunks via per-turn
    chunker, embeds each chunk at ingest-realtime priority through
    @joelclaw/inference-router (so queries preempt), writes chunks to
    run_chunks_dev and Run row to runs_dev, emits OTEL + fanout indexed
    event; concurrency:4, retries:3; ensures collections exist idempotently
  - @joelclaw/memory added to dependencies

apps/web
  - new app/api/runs/route.ts — POST handler; dev bearer token auth
    (hardcoded allowlist, PDS flow lands Phase 3), writes jsonl to NAS
    via @joelclaw/memory writeRunBlob, fires memory/run.captured via
    HTTP to local Inngest, returns 202 HATEOAS envelope with run_id +
    _links + next_actions
  - @joelclaw/inference-router + @joelclaw/memory added to dependencies

Still v1 / dev scope:
  - Dev bearer token, not PDS App Password (Phase 3)
  - _dev collection suffix, will alias to run_chunks_current after cutover
  - No search endpoint yet (next slice)
  - No capture hooks yet (Phase 4)
  - No entity enrichment yet (Phase 6)

Typecheck clean, biome clean, 12 chunking tests still green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-19 18:21:35 -07:00
Joel Hooks cc600aaf32 fix(docs-api): remove embedding from query_by — v2 has no auto-embed field 2026-03-18 22:27:47 -07:00
Joel Hooks c84d808d10 fix(docs-api): graceful degradation + unhandled rejection handler
fetchDocSummaries wrapped in try/catch — search still works if Typesense is slow.
handleStatus wrapped in try/catch — returns zeros if Typesense unreachable.
Global unhandledRejection handler prevents silent Bun crashes.
2026-03-18 21:05:54 -07:00
Joel Hooks 78ebf41148 fix: disable vector_query for v2 collection (raw float[], no auto-embed)
docs-api and CLI both tried vector_query with empty [] which only works with
Typesense auto-embed. v2 uses pre-computed ollama vectors. Text search on
retrieval_text still works. TODO: embed query via ollama for vector search.
2026-03-17 22:27:57 -07:00
Joel Hooks 4a6700d506 Upgrade docs-api for ADR-0234 by switching to a configurable chunks collection, adding NAS artifact endpoints, and enriching agentic HATEOAS search context 2026-03-17 22:08:24 -07:00
joelclaw-agent 8ef1329760 feat: LRU caching for docs, chunks, TOC (ADR-0228 Phase 5) — autonomous agent 2026-03-17 14:26:22 +00:00
joelclaw-agent 2b837b4dbf feat: TTL cache for doc lookups (ADR-0228 Phase 5) — autonomous 2026-03-17 14:22:31 +00:00
joelclaw-agent 3d8a46a083 feat: add LRU caching for doc/chunk/TOC lookups (ADR-0228 Phase 5) 2026-03-17 14:20:04 +00:00
Joel Hooks 2e119b4beb Deepen ADR-0228 phase 4 taxonomy with 11 SKOS sub-concepts and sync docs-api 2026-03-16 21:04:25 -07:00
Joel Hooks 6e563302c4 Restore ADR-0228 docs-api agentic search and Phase 3 browsing routes 2026-03-16 19:41:00 -07:00
Joel Hooks 089a3da562 Implement ADR-0228 docs-api metadata search plus TOC and chunk browsing endpoints 2026-03-16 19:35:47 -07:00
Joel Hooks 72d36ab05f Implement ADR-0228 Phase 2 docs-api search filters, concept facets, expansion, and assembly in apps/docs-api 2026-03-16 19:34:11 -07:00
Joel Hooks 2caaaad91b Implement ADR-0228 Phase 1 taxonomy graph endpoints in apps/docs-api and update the README 2026-03-16 19:12:06 -07:00
Joel Hooks f11d7e9084 chore: auto-commit from o11y triage 2026-03-10 09:15:39 -07:00
Joel Hooks aae6ab0c8c Degrade apps/web Convex static generation when env is missing, seed placeholder slug params for Cache Components, and no-op client Convex providers during prerender 2026-03-10 09:15:07 -07:00
Joel Hooks 16cd6db06f Implement ADR-0224 so discovery capture stores site and visibility metadata, content-sync removes ineligible joelclaw discoveries, and the operator-facing discovery flow returns the final created link 2026-03-10 09:02:50 -07:00
Joel Hooks 26c7614233 Sanitize Convex env reads and keep dynamic slug helper routes out of static build collection so Vercel production builds pass 2026-03-09 13:36:49 -07:00
Joel Hooks 83770d22a8 Revalidate markdown twins and sitemap.md whenever post content changes so /slug.md cannot stay stale after content review 2026-03-09 12:52:12 -07:00
Joel Hooks bd191e83ad fix(web): handle missing CONVEX_URL in network page for Vercel builds 2026-03-09 07:09:06 -07:00
Joel Hooks 5d56dac230 Guard apps/web search top-hit access, route convex-content through canonical readers, and pass resourceId into the registry review FAB so apps/web/tsconfig compiles clean again 2026-03-08 20:49:22 -07:00
Joel Hooks 53207f60ca Add short ADR route aliases, resolve them to canonical Convex slugs, and redirect legacy full-slug ADR URLs 2026-03-07 21:55:25 -08:00
Joel Hooks 843b03655e Wire the existing joel-writing-style skill, add an ADR-0217 architecture banner asset, and document the site writing path so an architecture spec draft can ship cleanly 2026-03-07 21:17:04 -08:00
Joel Hooks 069bb7cb8d cleanup: remove temporary .convex/deployment file 2026-03-07 14:21:07 -08:00
Joel Hooks 3f49d388b7 force build: add comment to trigger static path regeneration 2026-03-07 14:13:43 -08:00
Joel Hooks d1f688d358 migrate AI job scheduling article from discovery to tutorial with redirect
- Created tutorial:ai-job-scheduling-macos-launchd
- Removed discovery:ai-job-scheduling-mac-local-first-video
- Added permanent redirect from /cool/ai-job-scheduling-mac-local-first-video to /ai-job-scheduling-macos-launchd
2026-03-07 14:09:43 -08:00
Joel Hooks 16ba67e7d3 Strip leading markdown H1 from post content in the article page renderer so published posts do not render a duplicate title below the page header 2026-03-07 13:08:53 -08:00
Joel Hooks 8a62b9b3ad Update /api/pi-mono and docs/web.md to replace the planned placeholder with real install steps for the public contributing-to-pi-mono skill and extension repo 2026-03-07 12:48:28 -08:00
Joel Hooks a9466b058e Implement pi-mono Restate corpus sync, expose public joelclaw.com discovery/search, and replace the bogus async /output hint with real workflow inspection 2026-03-07 11:48:41 -08:00
Joel Hooks 19e092c8ec web: render CLAWMAIL marker as deterministic head script 2026-03-03 13:53:08 -08:00
Joel Hooks 22ae72fab5 Move CLAWMAIL marker to head and add ASCII header 2026-03-03 13:47:16 -08:00
Joel Hooks 0e83fe6f31 web: move clawmail source comment to html root 2026-03-03 13:42:20 -08:00
Joel Hooks 919fd104ae web: place CLAWMAIL marker in head with strict content-type guidance 2026-03-03 13:37:54 -08:00
Joel Hooks e76ca4bca6 chore(web): use fully qualified CLAWMAIL endpoint URLs 2026-03-03 13:36:33 -08:00
Joel Hooks 34a006f437 refine clawmail html source marker instructions 2026-03-03 13:33:15 -08:00
Joel Hooks 27d6b5d120 web: move CLAWMAIL marker to top html and add content-type checks 2026-03-03 13:32:16 -08:00
Joel Hooks ca0ce29784 web: add CLAWMAIL source comment for HTML pages 2026-03-03 13:16:36 -08:00
Joel Hooks 9ce664c877 fix: getPost() supports all content types, not just article
Sequentially tries article/tutorial/essay/note resourceId prefixes.
Removes article-only type guard. Uses matched resourceId in diagnostics.
2026-03-02 10:52:41 -08:00
Joel Hooks 0e1a4ba629 fix: query all content types in getAllPosts/getPostSlugs
getAllPosts() and getPostSlugs() were hardcoded to type='article'.
Now queries article, tutorial, essay, note via Promise.all and merges.
2026-03-02 10:46:16 -08:00
Joel Hooks e20d5a96a7 refactor(web): centralize copy prompt template 2026-03-01 16:34:38 -08:00
Joel Hooks 6b57097808 fix(web): soften copy-for-claw prompt text 2026-03-01 16:23:08 -08:00
Joel Hooks ab7732d0ba fix: remove stale 'markdown' ref in useCallback deps — broke prerender for 3hrs 2026-03-01 15:50:40 -08:00
Joel Hooks 39900845b4 chore: bump next 16.1.5 → 16.1.6 — fix workUnitAsyncStorage prerender bug 2026-03-01 15:12:37 -08:00
Joel Hooks fafdfda76f refactor: copy-for-your-claw now copies link to .md route with brief context, not full markdown 2026-03-01 15:07:26 -08:00
Joel Hooks f367e7996d style: right-justify copy-as-prompt button 2026-03-01 12:52:07 -08:00
Joel Hooks 01dd9267bf design: clean up article header — draft badge, inline copy button with tags, claw pink containment breach 2026-03-01 12:51:48 -08:00
Joel Hooks 47826b2a19 feat: copy-as-prompt with claw logo, grab animation, and 'snatched' confirmation 2026-03-01 12:47:33 -08:00