Commit Graph

1973 Commits

Author SHA1 Message Date
Joel Hooks ee80f1c8e7 feat: Talon infrastructure watchdog daemon (ADR-0159)
444KB zero-dep Rust binary, 2366 lines across 7 modules:
- worker.rs: bun process supervisor (port cleanup, signal forwarding, crash recovery)
- probes.rs: 10 health probes (Colima, Docker, Talos, k8s, Redis, Inngest, Typesense, worker)
- state.rs: persistent state machine (healthy → degraded → failed → investigating → critical → SOS)
- escalation.rs: 4-tier recovery (heal script → cloud agent → Ollama local → iMessage SOS)
- config.rs: hand-rolled TOML parser, zero deps
- log.rs: structured JSON logging with rotation
- main.rs: CLI (--check, --status, --worker-only, --dry-run)

All 10 probes passing. Binary at ~/.local/bin/talon.
Launchd plist ready but not yet loaded (worker still managed by existing plist).
2026-02-27 13:09:51 -08:00
Joel Hooks 2ab6eac694 chore: ignore .DS_Store in skills/ 2026-02-27 12:59:41 -08:00
Joel Hooks 62991921aa restore: ADR-0159 + 0158 superseded status (vault sync had deleted) 2026-02-27 12:57:19 -08:00
Joel Hooks f5d8903a96 sync: vault content (2 files)
adrs (2):
  updated: 0158-worker-supervisor-binary.md
  deleted: 0159-talon-k8s-watchdog-daemon.md
2026-02-27 11:01:55 -08:00
Joel Hooks a70de2ceb3 chore: gitignore Rust build artifacts 2026-02-27 10:46:34 -08:00
Joel Hooks 5886437570 feat: Rust worker-supervisor binary (ADR-0158)
525KB binary, zero external dependencies. Replaces start.sh as launchd target.
- Pre-start port cleanup (kills orphans holding 3111)
- Signal forwarding (SIGTERM/SIGINT/SIGHUP → bun child)
- Crash recovery with exponential backoff (1s→30s max)
- Health checks every 30s, auto-restart after 3 consecutive failures
- Inngest function registry PUT sync after startup
- Secret loading via `secrets` CLI
- TOML config at ~/.config/worker-supervisor.toml

Tested: bun crash → auto-restart in 1s, SIGTERM → clean forwarding,
launchd restart → no orphans, no EADDRINUSE.
2026-02-27 10:46:27 -08:00
Joel Hooks a1da0bb978 adr: propose ADR-0159 Talon infrastructure watchdog daemon
Compiled Rust binary that:
- Supervises system-bus worker process (absorbs ADR-0158)
- Monitors full k8s stack (Colima, Docker, Talos, pods, services)
- 4-tier escalation: heal script → cloud agent → local agent (Ollama) → iMessage SOS
- Compiles its own PATH (eliminates launchd env brittleness class)
- State machine prevents spam, cooldowns between escalations

Supersedes ADR-0158 (worker-supervisor-binary) — one binary to watch everything.

Also: installed Ollama + qwen3:8b (5.2GB) for Tier 3 local fallback.
2026-02-27 10:40:09 -08:00
Joel Hooks 57c48c550a sync: vault content (2 files)
adrs (2):
  new: 0158-automated-skill-gardening.md
  new: 0158-worker-supervisor-binary.md
2026-02-27 10:34:05 -08:00
Joel Hooks 36e25d24d6 fix: start.sh signal forwarding + pre-start port cleanup
Root cause: bun backgrounded in bash, launchd SIGTERM kills shell not bun.
Belt-and-suspenders fix until Rust supervisor (ADR-0158) replaces this.
2026-02-27 10:32:55 -08:00
Joel Hooks bc19202cdc fix: add PATH to Colima launchd plist and heal script
Root cause: Colima internally shells to limactl (/opt/homebrew/bin/limactl)
but launchd's default PATH is /usr/bin:/bin:/usr/sbin:/sbin — no homebrew.
Both com.joel.colima and k8s-reboot-heal.sh failed silently for 6 days
(since Feb 21), leaving the entire k8s cluster unrecoverable from any
restart or crash without manual intervention.

Fixes:
- com.joel.colima.plist: add EnvironmentVariables with PATH
- k8s-reboot-heal.sh: export PATH at script top (belt-and-suspenders)
- k8s-reboot-heal.sh: add kubectl uncordon (node was SchedulingDisabled)
- k8s skill: document PATH requirement as resilience rule #6 and #7
- operations.md: document PATH requirement, uncordon step, sticky
  shutdown state after unclean Talos container stop
2026-02-27 10:31:17 -08:00
Joel Hooks d2ef42b928 feat: automated skill gardening (ADR-0158)
New Inngest function: skill-garden
- Daily cron (6am PT) + on-demand via skill-garden/check event
- Structural checks: broken symlinks, non-canonical REAL DIRs,
  missing frontmatter, orphan skills
- Pattern-based staleness: greps for k3d, k3s, qdrant, old paths
- Monthly LLM deep review (1st of month): compares skill content
  against current AGENTS.md via pi inference
- Gateway notification when issues found, zero noise on clean days

New skill: skill-review
- Codifies the review process: canonical contract, structural
  checks, fix procedures, stale pattern registry
- Documents the automated garden triggers and output

Updated skills:
- joelclaw: full rewrite v2.0.0 — Talos/k8s v1.35.0, 110+ functions,
  all CLI commands (subscribe, gateway, otel, recall, discover, etc.),
  current pod list, ADR-0155/0156 references
- sync-system-bus: v2.0.0 — publish script as primary method,
  ADR-0156 restart safety, current gotcha table
2026-02-27 10:25:21 -08:00
Joel Hooks 83bbf5d3cc docs: comprehensive AGENTS.md + .claude/.pi/.agents config files
Rewrote AGENTS.md from scratch with:
- Full architecture diagram (Talos/k8s/Colima, accurate pod list)
- Complete monorepo layout with all 17 packages
- Tech stack table
- Hexagonal architecture rules with import examples
- CLI command reference (including new subscribe command)
- System bus deploy workflow
- Gateway + observability docs
- Skills contract (51 skills, repo canonical)
- Key ADR table
- Git workflow + logging rules
- Model policy

Added tool-specific config:
- .claude/CLAUDE.md — Claude Code project instructions
- .claude/settings.json — permission allowlist
- .agents/AGENTS.md — symlink to root AGENTS.md
- .pi/AGENTS.md — symlink to root AGENTS.md

Updated .gitignore to track only thin config files in dot dirs,
not skill dumps or bulk content.
2026-02-27 10:09:09 -08:00
Joel Hooks d3c43f0092 fix: use timeouts.finish (not start) for story pipeline
timeouts.start = queued-before-execution timeout
timeouts.finish = total execution time timeout
Was using wrong one, causing 5m default to kill codex runs.
2026-02-27 10:07:19 -08:00
Joel Hooks a462816704 fix: recover inngest-* skills as real dirs (were dead symlinks to .agents/) 2026-02-27 10:06:09 -08:00
Joel Hooks 2cd4e9cc6f chore: purge dot directory graveyard, canonicalize skills in repo
Removed from tracking:
- .agents/ (200+ skills, stale copies and axiom/marketing dumps)
- .pi/skills/ (208 symlinks to .agents/)
- .claude/skills/ (204 symlinks to .agents/)
- .cortex/skills/ (6 symlinks to .agents/)
- .bun-tmp/, .tmp/ (bun hash files, should never be tracked)

Physically removed 29 dead agent tool directories:
.adal .agent .augment .cline .codebuddy .commandcode .continue
.crush .cursor .factory .goose .iflow .junie .kilocode .kiro
.kode .mcpjam .mux .neovate .openhands .pochi .qoder .qwen
.roo .trae .vibe .windsurf .zencoder .bun .bun-home

Canonicalized 9 skills from ~/.pi/agent/skills/ (REAL DIRs) into
repo skills/:
  agent-loop, egghead-slack, github-bot, gogcli, joelclaw,
  loop-diagnosis, loop-nanny, ui-animation, video-ingest

Fixed 6 inngest-* symlinks in ~/.agents/ and ~/.pi/agent/ that
pointed to repo .agents/ — now point to repo skills/.

Moved AGENTS.md from .agents/AGENTS.md to repo root.

Fixed 2 dead symlinks in ~/.pi/agent/skills/:
  graphite-skill, add-skills (plural)

Updated .gitignore to block all agent tool dot dirs.

Skills contract: repo skills/ is canonical. Home dirs symlink in.
2026-02-27 10:05:12 -08:00
Joel Hooks 748c124506 fix: set 30m Inngest timeout + 15m execSync timeout for story pipeline
Default 5m step timeout was killing codex implement stages that take 10-15 min.
2026-02-27 10:01:09 -08:00
Joel Hooks 97b6505fc9 chore: mark CFP-2 done 2026-02-27 09:55:25 -08:00
Joel Hooks d3653de1fd fix: add REPOSITORY absolute path to all codex stage prompts
All three stages (implement, prove, judge) now include the full
absolute repo path in the prompt so codex knows where it's working.
2026-02-27 09:52:43 -08:00
Joel Hooks bc3c458c77 fix: getHeadSha fallback to monorepo root + debug logging for cwd resolution 2026-02-27 09:51:17 -08:00
Joel Hooks 923381353e feat(CFP-2): Read articles from Convex instead of filesystem
Codex implement step completed — tsc clean, build passes.
Committed manually because codex didn't commit (prompt strengthened).
Also: stronger commit requirement in story-pipeline implement prompt.
2026-02-27 09:49:47 -08:00
Joel Hooks 1f7b1748ee fix(story-pipeline): default cwd to monorepo root, not $HOME 2026-02-27 09:39:30 -08:00
Joel Hooks a00d5c99bd fix: remove stale prd.json from packages/system-bus (was masking root prd.json) 2026-02-27 09:37:53 -08:00
Joel Hooks 11102a69f8 sync: vault content (1 files)
adrs (1):
  updated: 0127-feed-subscriptions-and-resource-monitoring.md
2026-02-27 09:37:43 -08:00
Joel Hooks 318430e7c0 docs: update monitor skill — CLI is live, remove caveat 2026-02-27 09:36:43 -08:00
Joel Hooks ab2bac2340 fix(prd): context null→{}, mark CFP-1 done 2026-02-27 09:36:40 -08:00
Joel Hooks 6619f44124 feat(cli): add joelclaw subscribe command group (ADR-0127)
Implements the CLI surface for feed subscription management:
- subscribe list: show all monitored subscriptions from Redis
- subscribe add <url>: add with auto-type-detection, --name, --type, --interval, --filter
- subscribe remove <id>: remove by ID with not-found error handling
- subscribe check [--id]: trigger Inngest check for one or all subscriptions
- subscribe summary: health overview with stale detection and type breakdown

All commands return HATEOAS JSON envelopes per cli-design skill.
Duplicate detection on add with fix suggestion.
Lazy Redis connection (same pattern as gateway commands).
Inngest events for check: subscription/check.requested, subscription/check-feeds.requested
2026-02-27 09:36:15 -08:00
Joel Hooks 4bf0a79785 feat(skills): add monitor skill, update discovery with monitoring recommendations
- New monitor skill: manage feed subscriptions (ADR-0127)
  - Joel's interest profile for relevance scoring
  - CLI commands and Inngest fallback patterns
  - Subscription type detection heuristics
  - Notification and /cool publishing flow
- Discovery skill: add step 2 'Assess for Monitoring'
  - Recommend monitoring for monitorable + relevant resources via MCQ
  - Skip for one-shot content (articles, tweets)
  - Brief mention for monitorable but low-relevance items
2026-02-27 09:32:07 -08:00
Joel Hooks 8863160e60 sync: vault content (1 files)
discoveries (1):
  new: Constrained Generation as the Missing Layer in Agent-Driven UI.md
2026-02-27 09:28:49 -08:00
Joel Hooks fe7b41248b fix(prd): CFP-2 must land with CFP-3 cache boundaries — PPR constraint 2026-02-27 09:21:59 -08:00
Joel Hooks 1077942d50 revert(CFP-2): restore filesystem article reads — async Convex breaks PPR prerender
CFP-2 made getPost/getAllPosts async for Convex reads, but Next.js 16
with PPR rejects uncached async data access outside <Suspense>.
ALL article pages failed prerender with 'Uncached data was accessed
outside of <Suspense>'.

CFP-2 and CFP-3 need to land together — Convex reads require
'use cache' boundaries first.
2026-02-27 09:21:43 -08:00
Joel Hooks 561b636f52 fix(story-pipeline): retries: 2 for transient SDK failures (ADR-0156)
retries: 0 made worker restarts fatal — Inngest couldn't retry
the step on the new worker. With retries: 2, the 1s restart window
is covered by Inngest's built-in retry backoff.

ADR-0156 rewritten: blue/green port swap > drain-then-restart.
Inngest is step-level stateless — we don't need to wait for runs,
just for the current in-flight step.
2026-02-27 09:19:32 -08:00
Joel Hooks 3634895fd2 sync: vault content (1 files)
adrs (1):
  new: 0156-graceful-worker-restart.md
2026-02-27 09:18:55 -08:00
Joel Hooks d5680566af feat(CFP-1): seed articles into Convex — 19 articles seeded
scripts/seed-articles.ts created by codex pipeline, manually verified.
All 19 MDX articles upserted to Convex contentResources.
2026-02-27 09:10:54 -08:00
Joel Hooks 8194d1a819 fix(story-pipeline): OpenAI structured output requires all props in required
- Add sha, next_actions, fix_commit_sha to required arrays
- Use type: ['string', 'null'] for nullable fields
- OpenAI rejects schemas where properties exist but aren't required
2026-02-27 09:03:06 -08:00
Joel Hooks 0471118469 fix(story-pipeline): Zod runtime validation for PRD + story normalization
- Replace manual interfaces with Zod schemas (PrdSchema, StorySchema)
- Runtime parse with safeParse — fails fast with actionable error messages
- StorySchema.transform normalizes acceptance/acceptance_criteria
- Lists available story IDs on not-found error
- Add zod dep to system-bus
2026-02-27 09:01:05 -08:00
Joel Hooks 8477f29b14 fix(story-pipeline): guard prd.context optional access 2026-02-27 08:58:11 -08:00
Joel Hooks 0a2587553c feat(story-pipeline): structured output schemas, self-healing, gateway signals
- Output contracts per stage (implement/prove/judge) with JSON schemas
- codexExecWithHealing: JSON extraction + repair codex call on parse failure
- Gateway progress notifications at each stage transition
- Configurable codex timeout via PRD context
- Rewrote prd.json as content feedback pipeline (CFP-1 through CFP-7)
- Fix TS error: spread GatewaySignalMeta for Record<string,unknown> compat
2026-02-27 08:56:01 -08:00
Joel Hooks 013f86b25a feat(ADR-0155): 3-stage story pipeline + content migration PRD
- story-pipeline.ts: implement → prove → judge per story
- Event type agent/story.start registered in client
- PRD with 6 stories: seed, read, cache, feedback, inngest, revision
- Concurrency limit 1 (one story at a time)
- Max 3 attempts per story before blocking
2026-02-27 08:28:38 -08:00
Joel Hooks ba4afc1d3e sync: vault content (1 files)
adrs (1):
  new: 0155-three-stage-story-pipeline.md
2026-02-27 08:27:06 -08:00
Joel Hooks b4a525f9d1 sync: vault content (1 files)
adrs (1):
  updated: 0154-content-migration-mdx-to-convex.md
2026-02-27 08:04:31 -08:00
Joel Hooks f13c4a4525 sync: vault content (1 files)
adrs (1):
  new: 0154-content-migration-mdx-to-convex.md
2026-02-27 07:57:46 -08:00
Joel Hooks 03a95719bb sync: vault content (3 files)
adrs (3):
  updated: 0106-adr-review-pipeline.md
  updated: 0084-unified-content-resource-schema.md
  updated: 0039-self-host-convex-for-joelclaw.md
2026-02-27 07:52:31 -08:00
Joel Hooks c82474bf47 OG images from post frontmatter: image field → openGraph + twitter cards
- Added image? to PostMeta
- [slug]/page.tsx wires image into og:image and twitter:image
- MineClaw post with hero image + OG
2026-02-27 07:28:36 -08:00
Joel Hooks a6de1e0c7c fix(memory): add embedding to query_by for vector search in nightly maintenance
Typesense auto-embedding requires the embedding field in query_by
for vector_query with empty array [] to work. Without it:
'Vector query could not find any embedded fields.'

Other callers (observe.ts, context-prefetch.ts) already had this
correct. Only nightly-maintenance was broken.
2026-02-27 07:24:48 -08:00
Joel Hooks cec1b4b7ba sync: vault content (1 files)
adrs (1):
  updated: 0088-nas-backed-storage-tiering.md
2026-02-27 07:09:18 -08:00
Joel Hooks b58b0d402f feat(health): add NFS mount probes to core health slice
Probes stat /Volumes/nas-nvme and /Volumes/three-body with 5s
timeout. Stale or missing mounts surface as degraded in gateway
alerts before backup jobs fail silently.

Ref: ADR-0088 addendum (2026-02-27)
2026-02-27 07:08:49 -08:00
Joel Hooks acd0dcd1a3 move agent search to /api/search, remove /api/agent prefix
- /api/search now serves HATEOAS envelope for agents AND web UI
- Search dialog reads from .result.hits (envelope-aware)
- Updated /api discovery and layout comment
- Removed /api/agent/ directory
2026-02-26 18:48:50 -08:00
Joel Hooks 828bd65b1e api discovery: cacheLife max — only changes on deploy 2026-02-26 18:45:12 -08:00
Joel Hooks 43596db249 fix: separate cached data from NextResponse in /api route
use cache only works on serializable return values, not NextResponse objects
2026-02-26 18:44:09 -08:00
Joel Hooks bad43bf1b4 root /api discovery: cached HATEOAS map of all APIs, content, and agent tips 2026-02-26 18:42:41 -08:00