joelhooks__joelclaw/docs/cli.md

joelclaw CLI

Canonical operator interface for joelclaw.

Contract

• JSON envelope output (ok, command, result, next_actions)
• Deterministic error codes via respondError
• HATEOAS navigation in every command response
• Heavy dependencies loaded lazily when possible
• Capability adapter registry with typed command contracts (packages/cli/src/capabilities/; otel/recall/deploy/log/secrets/notify/mail/subscribe/heal adapter implementations live in @joelclaw/sdk)

SDK surface (@joelclaw/sdk)

@joelclaw/sdk is the programmatic wrapper for software integrations that need CLI parity without rebuilding adapters.

Current contract:

• transport modes:
  • subprocess — shell to joelclaw
  • inprocess — run SDK capability adapters directly (no shell) for supported capabilities (deploy, heal, log, mail, notify, otel, recall, secrets, subscribe)
  • hybrid (default) — inprocess first, subprocess fallback
• parses canonical JSON envelopes (ok, command, result, next_actions)
• provides typed convenience methods for common routes:
  • status()
  • deployWorker(options)
  • logWrite({ action, tool, detail, reason? })
  • notifySend({ message, channel?, priority?, context?, type?, source?, telegramOnly? })
  • secretsStatus/lease/revoke/audit/env
  • otelList/search/stats/emit
  • recall / recallRaw
  • vaultRead/search/ls/tree + vaultAdrList/collisions/audit/rank
• exposes structured errors:
  • JoelclawProcessError (spawn/exit/parse failures)
  • JoelclawEnvelopeError (ok:false envelopes via runOrThrow)
  • JoelclawCapabilityError (inprocess capability failures)

Example:

import { createJoelclawClient } from "@joelclaw/sdk";

const client = createJoelclawClient({
  timeoutMs: 15_000,
  transport: "inprocess",
});
const otel = await client.otelSearch("gateway", { hours: 1, limit: 20 });

Health endpoint fallback (ADR-0182)

CLI health probes for Inngest and worker resolve endpoints in this order:

1. localhost
2. discovered Colima VM IP (JOELCLAW_COLIMA_VM_IP, fallback 192.168.64.2)
3. k8s service DNS (*.joelclaw.svc.cluster.local)

Probe detail strings include the selected endpoint class (localhost|vm|svc_dns) and skipped-candidate counts.

Inngest status truth model (ADR-0187 + ADR-0159)

joelclaw inngest status [--heal] [--wait-ms 1500]

Semantics:

• reports both raw and normalized checks:
  • checks_raw: direct probe outcomes
  • checks: normalized truth surface used for ok
• worker truth is route-aware:
  • if direct worker endpoint probe fails but deployment/system-bus-worker is ready, worker is reported as healthy with summary.route = "k8s-only"
• k8s truth uses core workload readiness (not every pod phase):
  • statefulset/inngest, statefulset/redis, statefulset/typesense, deployment/system-bus-worker
• the lightweight joelclaw status pod summary now ignores terminal Succeeded / Completed pods from old ReplicaSets so reboot rollouts don't stay falsely red after replacement pods are healthy.
• includes Talon watchdog health (http://127.0.0.1:9999/health) plus launchd state to guide escalation.

--heal performs targeted remediation before re-check:

1. enforce worker single-source binding
2. restart/register worker when worker checks fail
3. kickstart Talon + run talon --check when server/k8s checks fail
4. re-collect status and return before/after snapshots

Inngest Connect WebSocket auth probe

joelclaw inngest connect-auth [--start-only] [--timeout-ms 8000] [--instance-id joelclaw-cli-probe] [--url http://localhost:8288]

Semantics:

• uses INNGEST_SIGNING_KEY as Bearer auth against /v0/connect/start.
• decodes the protobuf StartResponse (connection_id, gateway_endpoint, gateway_group, session/sync token presence).
• full mode runs the websocket handshake on the returned gateway endpoint:
  1. connects with subprotocol v0.connect.inngest.com
  2. expects GATEWAY_HELLO
  3. sends WORKER_CONNECT with session_token + sync_token
  4. expects GATEWAY_CONNECTION_READY
• output never includes raw tokens (metadata only: length/format/JWT timestamps).
• --start-only validates HTTP auth path without websocket handshake.

Command roots

• joelclaw status
  • --agent-dispatch-canary runs the deterministic non-LLM timeout canary and folds its terminal truth into the status envelope
• joelclaw summary [--hours N] [--format json|text]
• joelclaw runs
• joelclaw run
• joelclaw agent
• joelclaw content
• joelclaw gateway
• joelclaw loop
• joelclaw docs
• joelclaw vault
• joelclaw skills
• joelclaw mail
• joelclaw secrets
• joelclaw log
• joelclaw notify
• joelclaw otel
• joelclaw o11y
• joelclaw recall
• joelclaw memory
• joelclaw subscribe
• joelclaw webhook
• joelclaw inngest
• joelclaw restate
• joelclaw pds
• joelclaw knowledge
• joelclaw capabilities
• joelclaw queue
• joelclaw workload
• joelclaw jobs

PDS command tree

joelclaw pds
├── health
├── describe
├── collections
├── records <collection> [--limit <n>]
├── write <collection> --data <json>
├── delete <collection> <rkey>
└── session [--refresh]

Semantics:

• targets the rebuilt host-published PDS at http://localhost:9627
• caches sessions at ~/.joelclaw/pds-session.json
• resolves the login identifier from pds_joel_did by calling describeRepo and using the repo handle before createSession, because the rebuilt PDS accepted the handle while rejecting raw DID login
• returns token metadata in session output without dumping raw JWTs
• write auto-adds $type and createdAt

Docs command tree (ADR-0234 docs pipeline v2)

joelclaw docs
├── add <path> [--title <title>] [--tags a,b,c] [--category <storage-category>]
├── search <query> [--limit <n>] [--category <category>] [--concept <concept-id>] [--chunk-type <section|snippet>] [--doc <doc-id>] [--semantic]
├── context <chunk-id> [--mode snippet-window|parent-section|section-neighborhood] [--before <n>] [--after <n>] [--neighbors <n>]
├── list [--category <category>] [--limit <n>]
├── show <doc-id>
├── markdown <doc-id>
├── summary <doc-id>
├── status
├── reconcile [--manifest <path>] [--sample <n>]
├── enrich <doc-id>
├── reindex [--doc <doc-id>]
├── reindex-v2 <path> [--title <title>] [--skip-existing]
└── batch-reindex [--from-collection] [--skip-existing]

Semantics:

• active chunk search collection is switchable via DOCS_CHUNKS_COLLECTION and now defaults to docs_chunks_v2
• markdown + summary read durable artifacts from DOCS_ARTIFACTS_DIR (default /Volumes/three-body/docs-artifacts)
• docs status reports both docs_chunks (v1) and docs_chunks_v2 plus artifact directory availability
• agentic expansion flow is docs search → docs context → docs markdown / docs summary

Skills command tree

joelclaw skills
├── ensure <skill>
│   [--source-root <repo>]
│   [--consumer all|agents|pi|claude]
└── audit [--deep] [--wait-ms <wait-ms>] [--poll-ms <poll-ms>]

joelclaw skills ensure semantics:

• canonical local-repo install/maintenance surface for skills that already live in a repo skills/ directory
• resolves skills/<name>/SKILL.md from --source-root, cwd/ancestor repos, or the joelclaw repo fallback
• creates missing consumer symlinks and repairs wrong symlinks in ~/.agents/skills/, ~/.pi/agent/skills/, and ~/.claude/skills/
• fails loudly if a consumer target exists as a real file/dir instead of a symlink
• returns the installed skill path so agents can read it immediately
• for external third-party skill packages, the CLI points at the upstream installer: npx skills add -y -g <source>

Workload command tree

joelclaw workload
├── plan "<intent>"
│   [--preset docs-truth|research-compare|refactor-handoff]
│   [--kind auto|repo.patch|repo.refactor|repo.docs|repo.review|research.spike|runtime.proof|cross-repo.integration]
│   [--shape auto|serial|parallel|chained]
│   [--autonomy inline|supervised|afk|blocked]
│   [--proof none|dry-run|canary|soak|full]
│   [--risk reversible-only,host-okay]
│   [--artifacts patch,verification,summary]
│   [--acceptance "criterion one|criterion two"]
│   [--repo /abs/path/or/owner/repo]
│   [--paths docs/workloads.md,docs/cli.md]
│   [--paths-from status|head|recent:<n>]
│   [--stages-from /abs/path/to/stages.json]
│   [--write-plan ~/.joelclaw/workloads/]
│   [--requested-by Joel]
├── dispatch <plan-artifact>
│   [--stage <stage-id>]
│   [--to <to>]
│   [--from <from>]
│   [--send-mail]
│   [--write-dispatch ~/.joelclaw/workloads/]
├── run <plan-artifact>
│   [--stage <stage-id>]
│   [--tool pi|codex|claude]
│   [--execution-mode auto|host|sandbox]
│   [--sandbox-backend local|k8s]
│   [--sandbox-mode minimal|full]
│   [--skip-dep-check]
│   [--repo-url <repo-url>]
│   [--dry-run]
└── sandboxes
    ├── list [--state active|completed|failed|cancelled] [--mode minimal|full] [--expired] [--limit <n>]
    ├── cleanup [--request-id <id> | --sandbox-id <id> | --expired | --all-terminal] [--dry-run] [--force]
    └── janitor [--dry-run]

joelclaw workload plan semantics:

• planner surface for ADR-0217 Phase 4.3
• returns a canonical request + plan envelope using docs/workloads.md
• also returns guidance with:
  • recommendedExecution (execute-inline-now, tighten-scope-first, dispatch-after-health-check, etc.)
  • operatorSummary so the CLI says what to do next instead of shrugging
  • adrCoverage to show which ADRs likely govern the slice already; on fresh repo-local ADR clusters it remains best-effort guidance and may still need human reconciliation
  • recommendedSkills with install/read readiness, including joelclaw skills ensure <name> for local repo skills and npx skills add -y -g <source> for external skills
  • executionExamples for serial / parallel / chained coding workloads, including setup + execution few-shot patterns
  • executionLoop so the agent gets the honest plan → approve → execute/watch → summarize posture after the operator says yes
• infers kind, shape, mode, and backend when the caller leaves them open
• supports reusable planner presets for common docs/research/refactor shapes
• preserves Acceptance: clauses embedded in the prompt when --acceptance is omitted
• prefers implementation intent over docs follow-through, so mixed intents like refactor ... then update docs or extend ... then update README stay implementation-shaped
• validates known risk and artifacts values and emits warnings for unknown ones instead of silently inventing vocabulary
• mentioning sandboxes as the topic of a comparison does not force sandbox mode by itself; isolation has to be explicit or implied by AFK autonomy
• deploy-allowed is inferred only from explicit release/deploy intent; nouns like published skills do not count as deploy requests
• proof=canary|soak no longer forces supervised repo work onto durable / restate by itself
• --paths-from can seed file scope from local git activity, and --write-plan writes the full envelope to a reusable JSON artifact
• --stages-from <file> loads an explicit JSON stage DAG, validates dependencies/cycles, carries per-stage acceptance into plan verification, and adds DAG metadata to the plan result
• when --shape auto is still in effect, an explicit stage DAG now decides whether the plan is serial, parallel, or chained
• chained repo.patch/refactor work can decompose a Goal: section into explicit milestones and add a reflection/update stage when the prompt asks for it
• defaults --repo to the current working directory and infers branch / baseSha when that target is a local git repo; if the cwd is not a git repo, it warns and points the caller at --repo
• does not execute code or mutate repos

joelclaw workload dispatch semantics:

• reads a saved plan artifact from joelclaw workload plan --write-plan ...
• turns it into a stage-specific dispatch/handoff contract with canonical handoff data plus a clawmail-ready subject/body
• also returns dispatch guidance so the CLI can say whether the right move is to execute the stage now, keep the slice inline, or pause for a health check/recipient clarification
• dispatch guidance also carries executionLoop so the receiving agent knows the approval, progress-reporting, and closeout posture instead of inventing workflow theatre
• defaults to the first stage, but --stage can target a later stage explicitly
• preserves scoped file boundaries through selectedStage.reservedPaths / handoff.reservedPaths
• carries forward ADR coverage + recommended skill setup/readiness for the receiving agent
• --write-dispatch writes the dispatch contract as a reusable JSON artifact
• --send-mail --to <to> --from <from> sends that contract through joelclaw mail
• does not execute code or mutate repos

joelclaw workload run semantics:

• reads a saved plan artifact and normalizes it into the canonical queue-backed runtime request
• emits the queue family workload/requested, which the registry maps to system/agent.requested
• current durable runtime path is Redis queue → Restate dagOrchestrator → dagWorker
• dagWorker handlers currently cover shell, infer, and microvm
• defaults to --tool pi, with codex|claude as explicit opt-ins
• supports --sandbox-backend local|k8s plus --sandbox-mode minimal|full when sandbox execution is the point
• explicit-stage plans now gate stage execution on dependency inbox truth; use --skip-dep-check only for deliberate manual recovery or replay
• supports --dry-run for request inspection before queue admission
• returns queue admission details once the request is enqueued
• if queue admission fails before the runtime request is accepted, workload run now writes a terminal inbox snapshot for that requestId immediately instead of leaving operators with no truth artifact to inspect
• status|explain|cancel remain planned, not shipped
• joelclaw workload sandboxes is now the operator surface for ADR-0221 local sandbox state:
  • list reconciles the registry against per-sandbox sandbox.json metadata before reporting retention + filesystem truth, so operator output stops lying about terminal state after older partial writeback failures
  • cleanup is the bounded manual deletion path with --dry-run and --force, and it performs the same reconciliation before deciding whether a sandbox is still active
  • janitor is the dedicated expired-sandbox cleanup path instead of waiting for startup-time opportunistic pruning, and it also reconciles registry drift before computing candidates
  • scheduled janitoring now lives in the repo-managed launchd service com.joel.local-sandbox-janitor, which runs joelclaw workload sandboxes janitor at load and every 30 minutes

Restate command tree

joelclaw restate
├── status [--namespace <namespace>] [--admin-url <url>]
├── deployments [--admin-url <url>] [--cli-bin <bin>]
├── smoke [--script <path>]
├── enrich "<name>" [--github <user>] [--twitter <user>] [--depth quick|full] [--sync]
├── pi-mono-sync [--repo <owner/repo>] [--full-backfill] [--max-pages <n>] [--sync]
└── cron
    ├── status [--namespace <namespace>] [--service-name <service>] [--base-url <url>]
    ├── list [--namespace <namespace>] [--service-name <service>] [--base-url <url>]
    ├── enable-health [--schedule "0 7 * * * *"] [--run-now] [--restate-url <url>]
    └── delete <job>

joelclaw restate smoke semantics:

• resolves the smoke script path in this order:
  1. exact absolute path
  2. relative to current working directory
  3. relative to JOELCLAW_ROOT
  4. relative to ~/Code/joelhooks/joelclaw
• runs scripts/restate/test-workflow.sh by default.
• default smoke validates deployGate end-to-end.
• DAG smoke is available via script override:
  • joelclaw restate smoke --script scripts/restate/test-dag-workflow.sh

joelclaw restate pi-mono-sync semantics:

• triggers the pi-mono-sync Restate DAG pipeline, which runs the host-side direct task runner scripts/restate/run-tier1-task.ts --task pi-mono-artifacts-sync.
• syncs repo docs, issues, issue comments, pull requests, pull-request review comments, commits, and releases into the Typesense collection pi_mono_artifacts.
• writes two materialized documents into the same collection:
  • maintainer profile (kind=maintainer_profile, currently for badlogic)
  • sync checkpoint (kind=sync_state) so later runs can stay incremental unless --full-backfill is set.
• default repo is badlogic/pi-mono.
• --sync waits for the DAG result; async mode returns a workflow ID and lets Restate finish in the background.
• the new collection is queryable through joelclaw search --collection pi_mono_artifacts.

joelclaw restate cron semantics:

• manages Dkron scheduler jobs for Restate pipelines.
• default access path is a short-lived CLI-managed kubectl port-forward to svc/dkron-svc.
• pass --base-url only when you already have a direct Dkron API endpoint.
• enable-health seeds the health proof job: restate-health-check.
• sync-tier1 upserts the full ADR-0216 tier-1 set:
  • restate-health-check
  • restate-skill-garden
  • restate-typesense-full-sync
  • restate-daily-digest
  • restate-subscription-check-feeds
• list includes migratedFrom, successCount, errorCount, lastSuccess, and lastError so the soak is visible from the CLI without spelunking Dkron by hand.
• the jobs use Dkron's shell executor plus wget; it appends epoch seconds to each workflow ID prefix so every scheduled run is a fresh Restate workflow.
• Dkron cron expressions are six-field by default (sec min hour dom month dow), so hourly-at-minute-7 is 0 7 * * * *, not 7 * * * *.

Discover command

joelclaw discover <url> [-c <context>] [--site <site>] [--visibility <visibility>]

Semantics:

• --site choices: joelclaw, wizardshit, shared
• --visibility choices: public, private, archived, migration-only
• sensible defaults when omitted: site=joelclaw, visibility=public
• default path still emits discovery/noted directly to Inngest.
• when QUEUE_PILOTS=discovery, joelclaw discover now posts raw event intent to the worker admission endpoint (POST /internal/queue/enqueue) instead of writing Redis directly:
  • the worker owns queue admission, static registry lookup, and bounded triage mode resolution
  • returns queue metadata (streamId, eventId, priority) instead of Inngest run ids
  • includes triageMode + triage metadata whenever the family is enabled for shadow or enforce
  • relies on the Restate queue drainer to forward the event onward
• this keeps discovery pilot clients thin while the server remains the only queue policy surface.
• joelclaw discover is the thin fire-and-forget shortcut. If you need the final link for the created piece in the same turn, use the canonical follow path instead:

joelclaw send discovery/noted --data '{"url":"<url>","context":"<optional>","site":"joelclaw","visibility":"public"}' --follow

The terminal result from discovery-capture now includes finalLink. send --follow treats transient Inngest run lookup misses (for example immediate sql: no rows in result set after event send) as polling gaps, not fatal send failures: it keeps the event-sent result, emits a warning if final state cannot be confirmed, and points operators to joelclaw run / joelclaw runs for inspection.

Subscribe check queue pilot

joelclaw subscribe check [--id <id>]

Semantics:

• scoped checks (--id <id>) still emit subscription/check.requested directly to Inngest.
• all-subscription checks keep the legacy Inngest path by default.
• when QUEUE_PILOTS=subscriptions, joelclaw subscribe check without --id posts subscription/check-feeds.requested to the worker admission endpoint instead of writing Redis directly:
  • returns queue metadata (streamId, eventId, priority)
  • includes triageMode + triage metadata whenever the family is enabled for shadow or enforce
  • points next actions at joelclaw queue inspect / joelclaw queue depth
  • relies on the Restate queue drainer to forward the actual subscription/check-feeds.requested event name onward

Jobs command tree (ADR-0217 runtime monitor)

joelclaw jobs
└── status [--hours <n>] [--count <n>] [--namespace <namespace>] [--restate-admin-url <url>] [--dkron-service-name <service>] [--dkron-base-url <url>]

Semantics:

• jobs status is the first operator glance for real workloads during the ADR-0217 transition. It aggregates the queue/Redis substrate, Restate runtime, Dkron scheduler, and still-live Inngest surfaces into one JSON snapshot.
• overall.status is a bounded truth surface (healthy|degraded|down) derived from those four components, not a raw dump of every underlying health probe.
• queue section:
  • reads the canonical Redis queue directly through @joelclaw/queue
  • reports depth, priority buckets, oldest age, and active deterministic pauses
• Restate section:
  • mirrors joelclaw restate status in-place so the operator can see statefulset readiness + admin health without command hopping
• Dkron section:
  • mirrors joelclaw restate cron status in-place and includes the count of Restate-tagged scheduler jobs when the API is reachable
• Inngest section:
  • stays visible during migration, but its status is scoped to the transitional job path that still matters here: server/worker health plus recent run outcomes
  • broad informational checks (for example unrelated k8s pod drift) stay in the payload without poisoning the top-level job monitor status
• next actions point directly at the management surfaces that actually move workload state: queue control status, queue observe, queue resume, restate status, restate cron status, and runs

Pi async jobs monitor (runtime_jobs_monitor)

The loaded pi extension at packages/pi-extensions/inngest-monitor/index.ts now does two jobs:

1. tracks followed Inngest runs (inngest_send, inngest_runs)
2. exposes runtime_jobs_monitor for the ADR-0217 runtime substrate

runtime_jobs_monitor semantics:

• action=start|status|stop (default start)
• on start, it polls joelclaw jobs status in the background, paints a persistent widget, emits OTEL on severity changes and meaningful workload-state changes, and sends hidden follow-up messages for async report-back
• on status, it returns the latest runtime snapshot (overall status, queue depth, active pause count, Restate/Dkron/Inngest state)
• on stop, it stops the poller and sends a final follow-up summary
• widget posture is intentionally operator-first: current runtime state on top, active followed runs underneath

This is the canonical async monitoring path when you want a pi session to keep an eye on real workloads while you do other things.

Queue command tree (ADR-0217 Phase 1)

joelclaw queue
├── emit <event> [-d <json>] [-p P0|P1|P2|P3]
├── depth
├── stats [--hours <n>] [--limit <n>]
├── observe [--hours <n>] [--limit <n>] [--since <iso|ms>]
├── pause <family> [--ttl <duration>] [--reason <text>]
├── resume <family> [--reason <text>]
├── control
│   └── status [--hours <n>] [--limit <n>] [--since <iso|ms>]
├── list [--limit <n>]
└── inspect <stream-id>

Semantics:

• all queue subcommands return a clean JSON envelope; read-oriented subcommands close their Redis client before exit and emit stays a thin worker client instead of writing Redis directly
• queue state/control commands resolve Redis from the canonical joelclaw CLI config (~/.config/system-bus.env → REDIS_URL) before considering ambient shell env so the installed operator surface stays pointed at the same localhost queue as the worker and Restate drainer
• emit posts queue admission intent to the worker endpoint (POST /internal/queue/enqueue).
  • accepts event name (e.g., discovery/noted, content/updated)
  • accepts optional JSON payload via -d
  • optional priority override via -p is normalized client-side, but static registry routing and bounded triage stay server-side
  • the worker generates the canonical QueueEventEnvelope, adds trace metadata, resolves shadow/enforce mode for the event family, evaluates bounded triage, and persists the queue record
  • returns the Redis stream ID, priority, and any triageMode / triage metadata from admission
• depth reports queue depth, priority distribution (P0/P1/P2/P3 counts), oldest/newest message timestamps
• stats summarizes recent Restate queue-drainer behavior plus Phase 2 triage behavior from OTEL over a lookback window.
  • dispatch section reports sampled/found dispatch events, live queue depth, started/completed/failed counts, success rate, queue wait-time percentiles (p50/p95), dispatch-duration percentiles, promotion count, top event families, and recent failures
  • triage section reports attempts, completed/failed/fallback counts, fallback counts by reason, disagreement count, applied-change count, suggested-not-applied count, route mismatches, latency percentiles, per-family rollups, and recent mismatch/fallback samples
  • uses metadata.waitTimeMs from queue.dispatch.started as the Story 5 queue-to-dispatch latency signal
  • uses queue.triage.* OTEL metadata as the Story 3 source of truth for queue-admission disagreements and fallback behavior
  • --since <iso|ms> overrides the lower bound so operators can anchor soak evidence to a known clean point (for example the supervised queue.drainer.started after a rollout) instead of mixing fresh traffic with a dirty pre-fix window
  • keeps the operator in CLI-land; no raw Redis keys or manual OTEL spelunking required for the first sanity pass
• jobs status is the first unified runtime view; drop to queue subcommands when the aggregated surface says the queue itself needs attention.
• observe is now the Phase 3 Story 2-4 dry-run Sonnet operator surface.
  • builds a canonical live snapshot from current queue depth + queued messages + recent drainer OTEL + recent triage OTEL + gateway sleep/muted-channel state + active deterministic pauses
  • runs the bounded Sonnet observer in dry-run mode only and returns the current snapshot plus the current decision
  • long Sonnet summaries are trimmed instead of turning otherwise-useful observations into schema-only fallbacks
  • if all queued work is intentionally held behind fresh active manual pauses and no recent failures suggest downstream trouble, it short-circuits to a deterministic noop instead of wasting a 60s Sonnet call on an obvious hold state
  • history summarizes recent queue.observe.* OTEL for the same window so operators can compare the latest dry-run against raw history without spelunking Typesense by hand
  • control now reflects the shipped deterministic queue-control plane: active manual pauses, queue.control.applied|expired|rejected counts, and recent control events come from the same Redis + OTEL truth the drainer uses
  • --since <iso|ms> anchors the related OTEL history window the same way queue stats does
• pause applies a deterministic manual family pause with bounded TTL and emits queue.control.applied telemetry
• resume clears a deterministic family pause and emits either queue.control.applied or queue.control.rejected when the family was not paused
• control status is the dedicated deterministic control-plane operator surface.
  • reports active pauses (family, reason, TTL, applied/expiry timestamps, actor)
  • summarizes queue.control.applied|expired|rejected OTEL for the same window
  • is the first CLI answer to “what queue controls are active right now?” before any automatic Sonnet mutation ships
• list lists recent messages in priority order (highest priority first), does not ack/remove
• inspect loads a message by Redis stream ID and returns full payload + metadata
  • if the message is already acked/expired, it now returns a structured QUEUE_MESSAGE_MISSING error envelope with queue-state next actions instead of crashing the CLI

Queue configuration:

• Stream key: joelclaw:queue:events
• Priority index: joelclaw:queue:priority
• Consumer group: joelclaw:queue:cli
• Phase 1 pilot events: discovery/noted, discovery/captured, content/updated, subscription/check-feeds.requested, github/workflow_run.completed

OTEL telemetry under queue.* namespace:

• CLI queue commands forward queue package telemetry to OTEL with source=cli and component=queue
• queue.enqueue — message enqueued
• queue.lease — message leased for processing (includes wait time, priority, promotion metadata)
• queue.ack — message acknowledged
• queue.replay — unacked messages loaded for replay

Daily summary command

joelclaw summary [--hours 24] [--format json|text]

Semantics:

• aggregates recent git activity (joelclaw + Vault), Inngest run rollups, k8s pod health, OTEL stats, slog deploy/config events, and ADR churn into one response envelope.
• default output is JSON summary payload; --format text adds a compact text rendition under result.text for downstream chat/mobile surfaces.

Memory command group

joelclaw memory write "<observation text>" [--type observation|lesson|pattern|failed_target] [--source cli]
joelclaw memory search "<query>"

Semantics:

• memory write sends memory/observation.submitted to Inngest with { text, type, source, ts }.
• memory search is an alias surface for recall so read/write memory workflows live in one command group.

Knowledge turn-write command

joelclaw knowledge note \
  --source gateway \
  --agent gateway-daemon \
  --session <session-id> \
  --turn <turn-number> \
  --summary "<what changed>"

Use --skip-reason routine-heartbeat|duplicate-signal|no-new-information when a turn is eligible but has no durable signal to capture.

Gateway known issues / muted channels

joelclaw gateway known-issues
joelclaw gateway mute <channel> [--reason "<why muted>"]
joelclaw gateway unmute <channel>

Semantics:

• stores muted channel IDs at Redis key gateway:health:muted-channels (JSON array).
• stores optional mute reasons at Redis key gateway:health:mute-reasons (JSON object).
• muted channels remain in probe telemetry but are excluded from gateway.channels.degraded alerts.

Gateway process-layer diagnostics (joelclaw gateway diagnose) now inspect exact launchd state for com.joel.gateway and report disabled services explicitly. joelclaw gateway restart now re-enables the launch agent before bootstrap/kickstart to avoid restart failures when launchd has the service disabled.

Use joelclaw gateway enable for direct launch-agent recovery (enable + bootstrap + kickstart) without manual launchctl usage.

Gateway status/diagnose now separate daemon availability from Redis bridge health:

• joelclaw gateway status prefers daemon /health when available, so Redis loss no longer makes the gateway look fully dead.
• status now returns mode (normal or redis_degraded), degradedCapabilities, sessionPressure (context %, compaction age, session age, next action), and guardrails (current turn tool budget state + pending deploy verifications).
• joelclaw gateway diagnose treats redis_degraded as a degraded runtime, not a process failure, skips Redis-dependent E2E checks in that mode, and surfaces runtime guardrail findings when a checkpoint/deploy verification is active.

Gateway behavior control plane (ADR-0211)

joelclaw gateway behavior add --type keep|more|less|stop|start --text "..."
joelclaw gateway behavior list
joelclaw gateway behavior promote --id <candidate-id>
joelclaw gateway behavior remove --id <directive-id>
joelclaw gateway behavior apply
joelclaw gateway behavior stats

Semantics:

• Single write authority is CLI. Extensions must call these commands; no direct Redis/Typesense writes.
• Active runtime contract lives in Redis key joelclaw:gateway:behavior:contract.
• Directive/candidate history lives in Typesense collection gateway_behavior_history.
• add normalizes directives, enforces conflict + dedupe + cap rules, and updates Redis + Typesense.
• promote moves a pending daily-review candidate into the active contract (manual gate; no auto-activation).
• apply re-runs governance over the active contract (dedupe, conflict cleanup, cap enforcement) and expires stale candidates.
• stats reports contract hash/version, candidate lifecycle counts, and governance settings.

Run listing semantics

joelclaw runs [--status RUNNING|FAILED|COMPLETED|QUEUED|CANCELLED]

Semantics:

• applies backend status filtering and a local status guard so mixed-status payloads cannot leak through.
• the runs list query uses a longer timeout budget than generic Inngest GraphQL calls because self-hosted runs(...) can take ~50s even for small filtered windows; timeout scales upward with requested count, and timeout aborts get one bounded retry with a larger budget before the CLI gives up.
• for suspicious RUNNING rows (endedAt present while running, or long-running health checks), runs performs bounded detail reconciliation (max 5 lookups) to detect stale SDK-unreachable ghosts.
• runs with stale indicators include staleSignal with likely, confidence, and machine-readable reasons.
• response includes staleSignals summary (detected, likely, detailChecked) and suggests joelclaw inngest sweep-stale-runs when likely ghosts are present.
• count in response reflects post-filter rows (what the operator actually sees).

Run inspection + cancellation

joelclaw run <run-id> [--cancel] [--wait-ms 3000]

Semantics:

• default mode returns run detail, trigger event, trace, and step errors.
• run detail and joelclaw event <event-id> now use a much larger timeout/retry budget than generic GraphQL calls because self-hosted detail lookups can take more than a minute under load even while the runtime itself is healthy.
• --cancel issues Inngest GraphQL cancelRun for active runs, then polls status up to --wait-ms.
• deterministic error envelopes:
  • RUN_CANCEL_FAILED when cancelRun mutation fails.
  • RUN_CANCEL_TIMEOUT when run remains RUNNING|QUEUED after the wait window.
  • RUN_STALE_SDK_UNREACHABLE when a run appears RUNNING but trace errors show Unable to reach SDK URL and cancellation has no live execution to target.
• terminal runs are never re-cancelled; response includes cancellation.skipped = "already_terminal".

Capability adapter config precedence (ADR-0169 phase 0)

Resolution order is deterministic:

1. CLI flags (e.g. --adapter)
2. Environment variables
3. Project config (.joelclaw/config.toml)
4. User config (~/.joelclaw/config.toml)
5. Built-in defaults

Current env keys:

• JOELCLAW_CAPABILITY_<CAPABILITY>_ADAPTER
• JOELCLAW_CAPABILITY_<CAPABILITY>_ENABLED

Capability-backed command roots (ADR-0169 through phase 4)

joelclaw secrets status
joelclaw secrets lease <name> --ttl 15m
joelclaw secrets revoke <lease-id>
joelclaw secrets revoke --all
joelclaw secrets audit --tail 50
joelclaw secrets env --dry-run [--ttl 1h] [--force]

joelclaw log write --action <action> --tool <tool> --detail <detail> [--reason <reason>] [--session <session>] [--system <system>]

joelclaw notify send "<message>" [--priority low|normal|high|urgent] [--channel gateway|main|all] [--context '{"k":"v"}']

joelclaw heal {list|run}

joelclaw mail {status|register|send|inbox|read|reserve|renew|release|locks|search}

joelclaw otel {list|search|stats|emit}

joelclaw o11y {session|system}

joelclaw recall <query> [--limit N] [--min-score F] [--raw] [--include-hold] [--include-discard] [--budget auto|lean|balanced|deep] [--category <id|alias>]

joelclaw subscribe {list|add|remove|check|summary}

Semantics:

• log writes structured system entries (slog backend).
• log write now accepts explicit --session / --system provenance flags and also falls back to SLOG_SESSION_ID / SLOG_SYSTEM_ID env vars before handing off to the slog backend.
• logs reads/analyzes runtime logs.
• notify is the canonical operator alert command; gateway push remains transport/debug.
• deploy, heal, log, notify, secrets, mail, otel, recall, and subscribe keep their existing UX/envelopes while executing through capability registry adapters (scripted-deploy, runbook-heal, slog-cli, gateway-redis, agent-secrets-cli, mcp-agent-mail, typesense-otel, typesense-recall, redis-subscriptions).
• typesense-otel, typesense-recall, scripted-deploy, runbook-heal, slog-cli, agent-secrets-cli, gateway-redis, mcp-agent-mail, and redis-subscriptions adapter logic is canonical in @joelclaw/sdk (packages/sdk/src/capabilities/adapters/*); CLI adapter files are thin re-exports.
• otel emit accepts stdin JSON payloads (or convenience args/positional action), normalizes defaults (id, timestamp, level=info, success=true), and forwards to the worker ingest endpoint (/observability/emit). Provenance defaults are now role-aware: explicit event/session fields win, then SLOG_SESSION_ID / SLOG_SYSTEM_ID, then session handle envs when present, then role/hostname fallbacks (gateway or JOELCLAW_ROLE, system hostname stripped to panda).
• otel list and otel search accept exact --session / --system filters, mapped to sessionId / systemId in Typesense.
• knowledge search now auto-heals the common post-rebuild failure mode where system_knowledge is missing: on a 404 Collection not found response it creates the collection, reindexes ADRs + skills once, retries the query, and reports the repair in-band.
• o11y session / o11y system run a unified multi-search across otel_events and system_log, merge both timelines by timestamp, and tag each hit with its source collection.
• Software surfaces should route OTEL through this command contract (or shared CLI ingest helper), not ad-hoc raw HTTP calls.
• mail search auto-falls back to /mail/api/unified-inbox filtering when MCP search_messages returns transient DB/tool errors, so steering signals remain usable.
• mail reserve now sends explicit lease TTL (--ttl-seconds, default 900) and enforces a minimum of 60s.
• mail renew extends active file reservations without releasing/reacquiring (--extend-seconds, default 900, optional --paths).
• mail locks now prefers the local git-mailbox file_reservations/ artifact store when available because /mail/api/locks can under-report advisory file reservations while still reporting mailbox internals like archive/commit locks. Responses expose source and fallback_reason when artifact fallback was required.
• subscribe check emits Inngest request events for scoped checks and for all-subscription checks when the queue pilot is off; response.ids are event/request IDs (inspect via joelclaw event <event-id>), not run IDs unless explicitly returned as runIds.
• when QUEUE_PILOTS=subscriptions, unscoped subscribe check returns queue metadata instead of Inngest response ids because the request first lands in Redis and is forwarded by the Restate drainer.
• recall rewrite telemetry now exposes rewrite.strategy (disabled|skipped|haiku|openai|fallback) and rewrite.reason so low-ROI rewrite skips/fallbacks are queryable; short/literal/direct-id queries skip LLM rewrite by design.

Webhook command tree (ADR-0185)

joelclaw webhook
├── subscribe <provider> <event>
│   [--repo <owner/repo>] [--workflow <name>] [--branch <name>] [--conclusion <status>]
│   [--session <session-id>] [--ttl <duration>] [--stream] [--timeout <seconds>] [--replay <count>]
├── unsubscribe <subscription-id>
├── list [--provider <provider>] [--event <event>] [--session <session-id>]
└── stream <subscription-id> [--timeout <seconds>] [--replay <count>]

Semantics:

• Subscriptions are Redis-backed and session-scoped (joelclaw:webhook:*).
• subscribe --stream starts an NDJSON stream immediately after creation.
• stream emits ADR-0058 NDJSON (start, log, event, terminal result|error).
• Default session target is gateway for central gateway role, otherwise pid-<ppid>.
• TTL defaults to 24h and is enforced at match time.

Skills command tree (ADR-0179)

joelclaw skills
└── audit [--deep] [--wait-ms <wait-ms>] [--poll-ms <poll-ms>]

joelclaw skills audit purpose

• triggers the skill-garden/check event on-demand
• waits for the corresponding run and returns the findings report in-envelope
• supports --deep for LLM staleness checks

Agent command tree (ADR-0180 phases 2-4)

joelclaw agent
├── list
├── show <name>
├── run <name> <task> [--cwd <cwd>] [--timeout <seconds>]
├── chain <steps> --task <task> [--cwd <cwd>] [--fail-fast]
└── watch <id> [--timeout <seconds>]

Semantics:

• run emits agent/task.run for single roster agent execution and returns taskId plus eventIds from the Inngest send response.
• run next_actions are truthful: use joelclaw event <event-id> when an event ID exists (or joelclaw events ... fallback), and never assume taskId is a run ID.
• chain emits agent/chain.run with comma-separated sequential steps and + parallel groups (e.g. scout,planner+reviewer,coder).
• watch streams NDJSON progress for a task (at-...) or chain (ac-...) by subscribing to joelclaw:notify:gateway, replaying joelclaw:events:gateway, and falling back to Inngest polling.
• watch default timeout is 300 seconds for tasks and 900 seconds for chains; terminal events always include next_actions on completion, timeout, or interrupt.
• Runtime-proof recipe (ADR-0180):
  1. joelclaw agent list (expect builtin coder/designer/ops/story-executor)
  2. joelclaw agent run coder "reply with OK" --timeout 20
  3. joelclaw event <event-id> (expect Agent Task Run status COMPLETED with output payload)
• If Unknown agent roster entry: coder appears, treat it as worker-runtime drift: deploy latest system-bus-worker, restart the host worker, then rerun the three-step proof.

Vault command tree

joelclaw vault
├── read <ref>
├── search <query> [--semantic] [--limit <limit>]
├── ls [section]
├── tree
└── adr
    ├── list [--status <status>] [--limit <limit>]
    ├── collisions
    ├── audit
    ├── locate <query> [--limit <limit>]
    ├── refs <query>
    ├── prompt <text>
    └── rank [--band <band>] [--unscored] [--all]

joelclaw vault adr purpose

• list — inventory ADR metadata with optional status filter
• collisions — detect duplicate ADR numeric prefixes
• audit — full ADR hygiene check:
  • missing/non-canonical status values
  • number collisions
  • missing superseded-by targets
  • broken or ambiguous wiki links inside ADR bodies (skips custom directive tags like [[tts:text]])
  • README index alignment against ADR files
• locate — section/file lookup across the ADR corpus using ADR numbers, slugs, titles, and heading names
• refs — backlink discovery for a resolved ADR file or section id
• prompt — expand [[ADR refs]] in prompt text into canonical ids and append an <adr-context> block for agents
• rank — score + rank ADRs by NRC+novelty rubric for daily prioritization:
  • default scope: open ADRs (accepted + proposed)
  • --all includes shipped/superseded/deprecated/rejected ADRs
  • --band <band> filters ranked rows (do-now|do-next|de-risk|park; alias next → do-next)
  • --unscored returns ADRs missing priority-score
  • required axes: priority-need, priority-readiness, priority-confidence
  • novelty facet: priority-novelty (or alias priority-interest), defaults to neutral 3 when missing
  • score formula: clamp(round(20*(0.5*Need + 0.3*Readiness + 0.2*Confidence)) + round((Novelty-3)*5), 0, 100)
  • bands: do-now (80-100), do-next (60-79), de-risk (40-59), park (0-39)
  • emits CLI OTEL via component=vault-cli:
    • vault.adr.rank.started
    • vault.adr.rank.completed
    • vault.adr.rank.failed

Canonical statuses:

• proposed
• accepted
• shipped
• superseded
• deprecated
• rejected

Content command tree (ADR-0168)

joelclaw content
├── seed
├── verify
└── prune [--apply]

Semantics:

• seed — full Vault ADR sync to Convex for canonical ADR filenames only (NNNN-*.md).
• verify — strict ADR drift check against canonical ADR files (fails healthy state on both missing and extra ADR records in Convex).
• prune — dry-run report of Convex ADR extras (status: dry_run).
• prune --apply — removes ADR extras from Convex (status: pruned) and should be followed by joelclaw content verify.

Inngest source guard (ADR-0089)

joelclaw inngest source [--repair]

Semantics:

• Verifies launchd binding for com.joel.system-bus-worker against the canonical infra/launchd/com.joel.system-bus-worker.plist values (program + working directory).
• --repair copies canonical plist into ~/Library/LaunchAgents, performs launchctl bootout, then bootstrap with retry for transient Bootstrap failed: 5 launchd races.
• Use before joelclaw inngest restart-worker when host runtime/source drift is suspected.

Inngest stale-run sweep (ADR-0194)

joelclaw inngest sweep-stale-runs
joelclaw inngest sweep-stale-runs --apply

Semantics:

• preview-first by default (--apply required for mutation).
• scope defaults to stale health checks older than 30 minutes:
  • check/o11y-triage
  • check/system-health
• runtime target defaults:
  • namespace joelclaw
  • pod inngest-0
  • sqlite path /data/main.db
• apply mode safety gates:
  • refuses if age threshold is too young (<5m)
  • refuses when candidate count exceeds --max-apply-candidates
  • refuses when function_runs rows are missing (cannot insert terminal history safely)
  • always creates and verifies point-in-time backup before transaction:
    • /data/main.db.pre-sweep-<UTC-stamp>.sqlite
• terminalization contract (single transaction):
  1. insert missing history.type = FunctionCancelled
  2. insert missing function_finishes
  3. set trace_runs.status = 500 + terminal ended_at

This command exists for cases where Inngest API cancellation returns not found for stale RUNNING ghosts after SDK reachability failures.

Status command

joelclaw status [--agent-dispatch-canary]

Semantics:

• default joelclaw status remains the fast base worker/server health surface
• default output now also includes latestAgentDispatchCanary when a persisted deterministic canary snapshot exists, so operators can see the last proof result without spelunking runs or inbox files
• --agent-dispatch-canary runs scripts/verify-agent-dispatch-timeout.ts and folds the deterministic non-LLM system/agent-dispatch timeout proof into the returned envelope
• when requested, the command only reports healthy if both the base health probes and the canary pass
• this is the canonical on-demand proof surface for the live outer-timeout closeout path; it exists so operators do not have to run the verifier script manually

Build and verify

bunx tsc --noEmit
pnpm biome check packages/ apps/
bun test packages/cli/src/commands/*.test.ts
bun build packages/cli/src/cli.ts --compile --outfile ~/.bun/bin/joelclaw
joelclaw status
joelclaw vault
joelclaw vault adr audit
joelclaw vault adr locate "gateway guardrails"
joelclaw vault adr refs 0189-gateway-guardrails
joelclaw vault adr prompt "compare [[0189-gateway-guardrails]] with [[0218-gateway-availability-lifecycle-qol-improvements]]"

Add a command

1. Create command module in packages/cli/src/commands/.
2. Return envelopes with respond/respondError only.
3. Include useful next_actions with param hints.
4. Wire command in packages/cli/src/cli.ts.
5. Add/extend tests in packages/cli/src/commands/*.test.ts.
6. Update this file when command tree or contracts change.