mirror of
https://github.com/joelhooks/joelclaw.git
synced 2026-09-19 01:24:04 +08:00
951 lines
56 KiB
Markdown
951 lines
56 KiB
Markdown
# joelclaw CLI
|
|
|
|
Canonical operator interface for joelclaw.
|
|
|
|
## Contract
|
|
|
|
- JSON envelope output (`ok`, `command`, `result`, `next_actions`)
|
|
- Deterministic error codes via `respondError`
|
|
- HATEOAS navigation in every command response
|
|
- Heavy dependencies and privileged service credentials loaded lazily when possible; commands that do not need Inngest must not crash when `INNGEST_EVENT_KEY` is absent on thin Machines
|
|
- Capability adapter registry with typed command contracts (`packages/cli/src/capabilities/`; otel/recall/deploy/log/secrets/notify/mail/subscribe/heal adapter implementations live in `@joelclaw/sdk`)
|
|
|
|
## SDK surface (`@joelclaw/sdk`)
|
|
|
|
`@joelclaw/sdk` is the programmatic wrapper for software integrations that need CLI parity without rebuilding adapters.
|
|
|
|
Current contract:
|
|
|
|
- transport modes:
|
|
- `subprocess` — shell to `joelclaw`
|
|
- `inprocess` — run SDK capability adapters directly (no shell) for supported capabilities (`deploy`, `heal`, `log`, `mail`, `notify`, `otel`, `recall`, `secrets`, `subscribe`)
|
|
- `hybrid` (default) — inprocess first, subprocess fallback
|
|
- parses canonical JSON envelopes (`ok`, `command`, `result`, `next_actions`)
|
|
- provides typed convenience methods for common routes:
|
|
- `status()`
|
|
- `deployWorker(options)`
|
|
- `logWrite({ action, tool, detail, reason? })`
|
|
- `notifySend({ message, channel?, priority?, context?, type?, source?, telegramOnly? })` — producer-facts surface; every field is evidence for the gateway agent, never routing instruction
|
|
- `secretsStatus/lease/revoke/audit/env`
|
|
- `otelList/search/stats/emit`
|
|
- `recall` / `recallRaw`
|
|
- `vaultRead/search/ls/tree` + `vaultAdrList/collisions/audit/rank`
|
|
- exposes structured errors:
|
|
- `JoelclawProcessError` (spawn/exit/parse failures)
|
|
- `JoelclawEnvelopeError` (`ok:false` envelopes via `runOrThrow`)
|
|
- `JoelclawCapabilityError` (inprocess capability failures)
|
|
|
|
Example:
|
|
|
|
```ts
|
|
import { createJoelclawClient } from "@joelclaw/sdk";
|
|
|
|
const client = createJoelclawClient({
|
|
timeoutMs: 15_000,
|
|
transport: "inprocess",
|
|
});
|
|
const otel = await client.otelSearch("gateway", { hours: 1, limit: 20 });
|
|
```
|
|
|
|
## Health endpoint fallback (ADR-0182)
|
|
|
|
CLI health probes for Inngest and worker resolve endpoints in this order:
|
|
|
|
1. `localhost`
|
|
2. discovered Colima VM IP (`JOELCLAW_COLIMA_VM_IP`, fallback `192.168.64.2`)
|
|
3. k8s service DNS (`*.joelclaw.svc.cluster.local`)
|
|
|
|
Probe detail strings include the selected endpoint class (`localhost|vm|svc_dns`) and skipped-candidate counts.
|
|
|
|
## Inngest status truth model (ADR-0187 + ADR-0159)
|
|
|
|
```bash
|
|
joelclaw inngest status [--heal] [--wait-ms 1500]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- reports both raw and normalized checks:
|
|
- `checks_raw`: direct probe outcomes
|
|
- `checks`: normalized truth surface used for `ok`
|
|
- worker truth is route-aware:
|
|
- if direct worker endpoint probe fails but `deployment/system-bus-worker` is ready, worker is reported as healthy with `summary.route = "k8s-only"`
|
|
- k8s truth uses core workload readiness (not every pod phase):
|
|
- `statefulset/inngest`, `statefulset/redis`, `statefulset/typesense`, `deployment/system-bus-worker`
|
|
- the lightweight `joelclaw status` pod summary now ignores terminal `Succeeded` / `Completed` pods from old ReplicaSets so reboot rollouts don't stay falsely red after replacement pods are healthy.
|
|
- includes Talon watchdog health (`http://127.0.0.1:9999/health`) plus launchd state to guide escalation.
|
|
|
|
`--heal` performs targeted remediation before re-check:
|
|
|
|
1. enforce worker single-source binding
|
|
2. restart/register worker when worker checks fail
|
|
3. kickstart Talon + run `talon --check` when server/k8s checks fail
|
|
4. re-collect status and return before/after snapshots
|
|
|
|
## Brain and observation recall
|
|
|
|
`joelclaw recall <query>` queries the disposable `observations` and `brain_graph_nodes` Typesense projections. It preserves the existing recall result envelope, including `hits[].observation`, so voice and other callers do not need a coordinated cutover. Brain `.svx` files remain canonical; Typesense is rebuildable.
|
|
|
|
The legacy `memory_observations` health/e2e probes are retired with that collection. Use `joelclaw observations index` to rebuild observation pages and the Brain graph index owner to rebuild `brain_graph_nodes`.
|
|
|
|
## Inngest Connect WebSocket auth probe
|
|
|
|
```bash
|
|
joelclaw inngest connect-auth [--start-only] [--timeout-ms 8000] [--instance-id joelclaw-cli-probe] [--url http://localhost:8288]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- uses `INNGEST_SIGNING_KEY` as Bearer auth against `/v0/connect/start`.
|
|
- decodes the protobuf `StartResponse` (`connection_id`, `gateway_endpoint`, `gateway_group`, session/sync token presence).
|
|
- full mode runs the websocket handshake on the returned gateway endpoint:
|
|
1. connects with subprotocol `v0.connect.inngest.com`
|
|
2. expects `GATEWAY_HELLO`
|
|
3. sends `WORKER_CONNECT` with `session_token` + `sync_token`
|
|
4. expects `GATEWAY_CONNECTION_READY`
|
|
- output never includes raw tokens (metadata only: length/format/JWT timestamps).
|
|
- `--start-only` validates HTTP auth path without websocket handshake.
|
|
|
|
## Command roots
|
|
|
|
- `joelclaw status`
|
|
- `--agent-dispatch-canary` runs the deterministic non-LLM timeout canary and folds its terminal truth into the status envelope
|
|
- `joelclaw summary [--hours N] [--format json|text]`
|
|
- `joelclaw runs`
|
|
- `joelclaw run`
|
|
- `joelclaw agent`
|
|
- `joelclaw content`
|
|
- `joelclaw gateway`
|
|
- `joelclaw loop`
|
|
- `joelclaw docs`
|
|
- `joelclaw vault`
|
|
- `joelclaw skills`
|
|
- `joelclaw mail`
|
|
- `joelclaw secrets`
|
|
- `joelclaw log`
|
|
- `joelclaw notify`
|
|
- `joelclaw otel`
|
|
- `joelclaw o11y`
|
|
- `joelclaw recall`
|
|
- `joelclaw memory`
|
|
- `joelclaw subscribe`
|
|
- `joelclaw webhook`
|
|
- `joelclaw inngest`
|
|
- `joelclaw restate`
|
|
- `joelclaw pds`
|
|
- `joelclaw knowledge`
|
|
- `joelclaw capabilities`
|
|
- `joelclaw queue`
|
|
- `joelclaw workload`
|
|
- `joelclaw jobs`
|
|
- `joelclaw fleet`
|
|
|
|
## Fleet command tree
|
|
|
|
```bash
|
|
joelclaw fleet
|
|
├── status [--host <alias>] [--config <private-manifest>]
|
|
└── diff [--host <alias>] [--config <private-manifest>]
|
|
```
|
|
|
|
Both commands are read-only and return the standard JSON envelope. `status` gathers declared-host facts; `diff` classifies declared expectations as `in_sync`, `expected_difference`, `drift`, or `unavailable`. The manifest is local private configuration: never commit or print its contents. Fleet plan/apply commands do not exist yet; use a reviewed, host-scoped change path only after explicit approval.
|
|
|
|
## PDS command tree
|
|
|
|
```bash
|
|
joelclaw pds
|
|
├── health
|
|
├── describe
|
|
├── collections
|
|
├── records <collection> [--limit <n>]
|
|
├── write <collection> --data <json>
|
|
├── delete <collection> <rkey>
|
|
└── session [--refresh]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- targets the rebuilt host-published PDS at `http://localhost:9627`
|
|
- caches sessions at `~/.joelclaw/pds-session.json`
|
|
- resolves the login identifier from `pds_joel_did` by calling `describeRepo` and using the repo handle before `createSession`, because the rebuilt PDS accepted the handle while rejecting raw DID login
|
|
- returns token metadata in `session` output without dumping raw JWTs
|
|
- `write` auto-adds `$type` and `createdAt`
|
|
|
|
## Docs command tree (ADR-0234 docs pipeline v2)
|
|
|
|
```bash
|
|
joelclaw docs
|
|
├── add <path> [--title <title>] [--tags a,b,c] [--category <storage-category>]
|
|
├── search <query> [--limit <n>] [--category <category>] [--concept <concept-id>] [--chunk-type <section|snippet>] [--doc <doc-id>] [--semantic]
|
|
├── context <chunk-id> [--mode snippet-window|parent-section|section-neighborhood] [--before <n>] [--after <n>] [--neighbors <n>]
|
|
├── list [--category <category>] [--limit <n>]
|
|
├── show <doc-id>
|
|
├── markdown <doc-id>
|
|
├── summary <doc-id>
|
|
├── status
|
|
├── reconcile [--manifest <path>] [--sample <n>]
|
|
├── enrich <doc-id>
|
|
├── reindex [--doc <doc-id>]
|
|
├── reindex-v2 <path> [--title <title>] [--skip-existing]
|
|
└── batch-reindex [--from-collection] [--skip-existing]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- active chunk search collection is switchable via `DOCS_CHUNKS_COLLECTION` and now defaults to `docs_chunks_v2`
|
|
- markdown + summary read durable artifacts from `DOCS_ARTIFACTS_DIR` (default `/Volumes/three-body/docs-artifacts`)
|
|
- `docs status` reports both `docs_chunks` (v1) and `docs_chunks_v2` plus artifact directory availability
|
|
- agentic expansion flow is `docs search` → `docs context` → `docs markdown` / `docs summary`
|
|
|
|
## Skills command tree
|
|
|
|
```bash
|
|
joelclaw skills
|
|
├── ensure <skill>
|
|
│ [--source-root <repo>]
|
|
│ [--consumer all|agents|pi|claude]
|
|
└── audit [--deep] [--wait-ms <wait-ms>] [--poll-ms <poll-ms>]
|
|
```
|
|
|
|
`joelclaw skills ensure` semantics:
|
|
|
|
- compatibility-shim maintenance surface for skills that already live in a repo `skills/` directory
|
|
- resolves `skills/<name>/SKILL.md` from `--source-root`, cwd/ancestor repos, or the joelclaw repo fallback
|
|
- creates missing flat per-skill symlinks and repairs wrong symlinks in `~/.agents/skills/`, `~/.pi/agent/skills/`, and `~/.claude/skills/`
|
|
- expects those consumer roots to be real directories; whole-root symlinks are a layering error
|
|
- fails loudly if a consumer target exists as a real file/dir instead of a symlink
|
|
- returns the installed skill path so agents can `read` it immediately
|
|
- for external third-party skill packages, the CLI points at the upstream installer: `npx skills add -y -g <source>`
|
|
|
|
## Workload command tree
|
|
|
|
```bash
|
|
joelclaw workload
|
|
├── plan "<intent>"
|
|
│ [--preset docs-truth|research-compare|refactor-handoff]
|
|
│ [--kind auto|repo.patch|repo.refactor|repo.docs|repo.review|research.spike|runtime.proof|cross-repo.integration]
|
|
│ [--shape auto|serial|parallel|chained]
|
|
│ [--autonomy inline|supervised|afk|blocked]
|
|
│ [--proof none|dry-run|canary|soak|full]
|
|
│ [--risk reversible-only,host-okay]
|
|
│ [--artifacts patch,verification,summary]
|
|
│ [--acceptance "criterion one|criterion two"]
|
|
│ [--repo /abs/path/or/owner/repo]
|
|
│ [--paths docs/workloads.md,docs/cli.md]
|
|
│ [--paths-from status|head|recent:<n>]
|
|
│ [--stages-from /abs/path/to/stages.json]
|
|
│ [--write-plan ~/.joelclaw/workloads/]
|
|
│ [--requested-by Joel]
|
|
├── dispatch <plan-artifact>
|
|
│ [--stage <stage-id>]
|
|
│ [--to <to>]
|
|
│ [--from <from>]
|
|
│ [--send-mail]
|
|
│ [--write-dispatch ~/.joelclaw/workloads/]
|
|
├── run <plan-artifact>
|
|
│ [--stage <stage-id>]
|
|
│ [--tool pi|codex|claude]
|
|
│ [--execution-mode auto|host|sandbox]
|
|
│ [--sandbox-backend local|k8s]
|
|
│ [--sandbox-mode minimal|full]
|
|
│ [--skip-dep-check]
|
|
│ [--repo-url <repo-url>]
|
|
│ [--dry-run]
|
|
└── sandboxes
|
|
├── list [--state active|completed|failed|cancelled] [--mode minimal|full] [--expired] [--limit <n>]
|
|
├── cleanup [--request-id <id> | --sandbox-id <id> | --expired | --all-terminal] [--dry-run] [--force]
|
|
└── janitor [--dry-run]
|
|
```
|
|
|
|
`joelclaw workload plan` semantics:
|
|
|
|
- planner surface for ADR-0217 Phase 4.3
|
|
- returns a canonical `request` + `plan` envelope using `docs/workloads.md`
|
|
- also returns `guidance` with:
|
|
- `recommendedExecution` (`execute-inline-now`, `tighten-scope-first`, `dispatch-after-health-check`, etc.)
|
|
- `operatorSummary` so the CLI says what to do next instead of shrugging
|
|
- `adrCoverage` to show which ADRs likely govern the slice already; on fresh repo-local ADR clusters it remains best-effort guidance and may still need human reconciliation
|
|
- `recommendedSkills` with install/read readiness, including `joelclaw skills ensure <name>` for local repo skills and `npx skills add -y -g <source>` for external skills
|
|
- `executionExamples` for serial / parallel / chained coding workloads, including setup + execution few-shot patterns
|
|
- `executionLoop` so the agent gets the honest plan → approve → execute/watch → summarize posture after the operator says yes
|
|
- infers `kind`, `shape`, `mode`, and `backend` when the caller leaves them open
|
|
- supports reusable planner presets for common docs/research/refactor shapes
|
|
- preserves `Acceptance:` clauses embedded in the prompt when `--acceptance` is omitted
|
|
- prefers implementation intent over docs follow-through, so mixed intents like `refactor ... then update docs` or `extend ... then update README` stay implementation-shaped
|
|
- validates known `risk` and `artifacts` values and emits warnings for unknown ones instead of silently inventing vocabulary
|
|
- mentioning sandboxes as the topic of a comparison does **not** force sandbox mode by itself; isolation has to be explicit or implied by AFK autonomy
|
|
- `deploy-allowed` is inferred only from explicit release/deploy intent; nouns like `published skills` do not count as deploy requests
|
|
- `proof=canary|soak` no longer forces supervised repo work onto `durable` / `restate` by itself
|
|
- `--paths-from` can seed file scope from local git activity, and `--write-plan` writes the full envelope to a reusable JSON artifact
|
|
- `--stages-from <file>` loads an explicit JSON stage DAG, validates dependencies/cycles, carries per-stage acceptance into plan verification, and adds DAG metadata to the plan result
|
|
- when `--shape auto` is still in effect, an explicit stage DAG now decides whether the plan is `serial`, `parallel`, or `chained`
|
|
- chained repo.patch/refactor work can decompose a `Goal:` section into explicit milestones and add a reflection/update stage when the prompt asks for it
|
|
- defaults `--repo` to the current working directory and infers `branch` / `baseSha` when that target is a local git repo; if the cwd is not a git repo, it warns and points the caller at `--repo`
|
|
- does **not** execute code or mutate repos
|
|
|
|
`joelclaw workload dispatch` semantics:
|
|
|
|
- reads a saved plan artifact from `joelclaw workload plan --write-plan ...`
|
|
- turns it into a stage-specific dispatch/handoff contract with canonical `handoff` data plus a clawmail-ready subject/body
|
|
- also returns dispatch `guidance` so the CLI can say whether the right move is to execute the stage now, keep the slice inline, or pause for a health check/recipient clarification
|
|
- dispatch guidance also carries `executionLoop` so the receiving agent knows the approval, progress-reporting, and closeout posture instead of inventing workflow theatre
|
|
- defaults to the first stage, but `--stage` can target a later stage explicitly
|
|
- preserves scoped file boundaries through `selectedStage.reservedPaths` / `handoff.reservedPaths`
|
|
- carries forward ADR coverage + recommended skill setup/readiness for the receiving agent
|
|
- `--write-dispatch` writes the dispatch contract as a reusable JSON artifact
|
|
- `--send-mail --to <to> --from <from>` sends that contract through `joelclaw mail`
|
|
- does **not** execute code or mutate repos
|
|
|
|
`joelclaw workload run` semantics:
|
|
|
|
- reads a saved plan artifact and normalizes it into the canonical queue-backed runtime request
|
|
- emits the queue family `workload/requested`, which the registry maps to `system/agent.requested`
|
|
- current durable runtime path is `Redis queue → Restate dagOrchestrator → dagWorker`
|
|
- `dagWorker` handlers currently cover `shell`, `infer`, and `microvm`
|
|
- defaults to `--tool pi`, with `codex|claude` as explicit opt-ins
|
|
- supports `--sandbox-backend local|k8s` plus `--sandbox-mode minimal|full` when sandbox execution is the point
|
|
- explicit-stage plans now gate stage execution on dependency inbox truth; use `--skip-dep-check` only for deliberate manual recovery or replay
|
|
- supports `--dry-run` for request inspection before queue admission
|
|
- returns queue admission details once the request is enqueued
|
|
- if queue admission fails before the runtime request is accepted, `workload run` now writes a terminal inbox snapshot for that `requestId` immediately instead of leaving operators with no truth artifact to inspect
|
|
- `status|explain|cancel` remain planned, not shipped
|
|
- `joelclaw workload sandboxes` is now the operator surface for ADR-0221 local sandbox state:
|
|
- `list` reconciles the registry against per-sandbox `sandbox.json` metadata before reporting retention + filesystem truth, so operator output stops lying about terminal state after older partial writeback failures
|
|
- `cleanup` is the bounded manual deletion path with `--dry-run` and `--force`, and it performs the same reconciliation before deciding whether a sandbox is still active
|
|
- `janitor` is the dedicated expired-sandbox cleanup path instead of waiting for startup-time opportunistic pruning, and it also reconciles registry drift before computing candidates
|
|
- scheduled janitoring now lives in the repo-managed launchd service `com.joel.local-sandbox-janitor`, which runs `joelclaw workload sandboxes janitor` at load and every 30 minutes
|
|
|
|
## Restate command tree
|
|
|
|
```bash
|
|
joelclaw restate
|
|
├── status [--namespace <namespace>] [--admin-url <url>]
|
|
├── deployments [--admin-url <url>] [--cli-bin <bin>]
|
|
├── smoke [--script <path>]
|
|
├── enrich "<name>" [--github <user>] [--twitter <user>] [--depth quick|full] [--sync]
|
|
├── pi-mono-sync [--repo <owner/repo>] [--full-backfill] [--max-pages <n>] [--sync]
|
|
└── cron
|
|
├── status [--namespace <namespace>] [--service-name <service>] [--base-url <url>]
|
|
├── list [--namespace <namespace>] [--service-name <service>] [--base-url <url>]
|
|
├── enable-health [--schedule "0 7 * * * *"] [--run-now] [--restate-url <url>]
|
|
└── delete <job>
|
|
```
|
|
|
|
`joelclaw restate smoke` semantics:
|
|
|
|
- resolves the smoke script path in this order:
|
|
1. exact absolute path
|
|
2. relative to current working directory
|
|
3. relative to `JOELCLAW_ROOT`
|
|
4. relative to `~/Code/joelhooks/joelclaw`
|
|
- runs `scripts/restate/test-workflow.sh` by default.
|
|
- default smoke validates `deployGate` end-to-end.
|
|
- DAG smoke is available via script override:
|
|
- `joelclaw restate smoke --script scripts/restate/test-dag-workflow.sh`
|
|
|
|
`joelclaw restate pi-mono-sync` semantics:
|
|
|
|
- triggers the `pi-mono-sync` Restate DAG pipeline, which runs the host-side direct task runner `scripts/restate/run-tier1-task.ts --task pi-mono-artifacts-sync`.
|
|
- syncs repo docs, issues, issue comments, pull requests, pull-request review comments, commits, and releases into the Typesense collection `pi_mono_artifacts`.
|
|
- writes two materialized documents into the same collection:
|
|
- maintainer profile (`kind=maintainer_profile`, currently for `badlogic`)
|
|
- sync checkpoint (`kind=sync_state`) so later runs can stay incremental unless `--full-backfill` is set.
|
|
- default repo is `badlogic/pi-mono`.
|
|
- `--sync` waits for the DAG result; async mode returns a workflow ID and lets Restate finish in the background.
|
|
- the new collection is queryable through `joelclaw search --collection pi_mono_artifacts`.
|
|
|
|
`joelclaw restate cron` semantics:
|
|
|
|
- manages Dkron scheduler jobs for Restate pipelines.
|
|
- default access path is a **short-lived CLI-managed `kubectl port-forward`** to `svc/dkron-svc`.
|
|
- pass `--base-url` only when you already have a direct Dkron API endpoint.
|
|
- `enable-health` seeds the health proof job: `restate-health-check`.
|
|
- `sync-tier1` upserts the full ADR-0216 tier-1 set:
|
|
- `restate-health-check`
|
|
- `restate-skill-garden`
|
|
- `restate-typesense-full-sync`
|
|
- `restate-daily-digest`
|
|
- `restate-subscription-check-feeds`
|
|
- `list` includes `migratedFrom`, `successCount`, `errorCount`, `lastSuccess`, and `lastError` so the soak is visible from the CLI without spelunking Dkron by hand.
|
|
- the jobs use Dkron's shell executor plus `wget`; it appends epoch seconds to each workflow ID prefix so every scheduled run is a fresh Restate workflow.
|
|
- Dkron cron expressions are **six-field** by default (`sec min hour dom month dow`), so hourly-at-minute-7 is `0 7 * * * *`, not `7 * * * *`.
|
|
|
|
## Discover command
|
|
|
|
```bash
|
|
joelclaw discover <url> [-c <context>] [--site <site>] [--visibility <visibility>]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- `--site` choices: `joelclaw`, `wizardshit`, `shared`
|
|
- `--visibility` choices: `public`, `private`, `archived`, `migration-only`
|
|
- sensible defaults when omitted: `site=joelclaw`, `visibility=public`
|
|
- default path still emits `discovery/noted` directly to Inngest.
|
|
- when `QUEUE_PILOTS=discovery`, `joelclaw discover` now posts raw event intent to the worker admission endpoint (`POST /internal/queue/enqueue`) instead of writing Redis directly:
|
|
- the worker owns queue admission, static registry lookup, and bounded triage mode resolution
|
|
- returns queue metadata (`streamId`, `eventId`, `priority`) instead of Inngest run ids
|
|
- includes `triageMode` + `triage` metadata whenever the family is enabled for shadow or enforce
|
|
- relies on the Restate queue drainer to forward the event onward
|
|
- this keeps discovery pilot clients thin while the server remains the only queue policy surface.
|
|
- `joelclaw discover` is the thin fire-and-forget shortcut. If you need the final link for the created piece in the same turn, use the canonical follow path instead:
|
|
|
|
```bash
|
|
joelclaw send discovery/noted --data '{"url":"<url>","context":"<optional>","site":"joelclaw","visibility":"public"}' --follow
|
|
```
|
|
|
|
The terminal result from `discovery-capture` now includes `finalLink`. `send --follow` treats transient Inngest run lookup misses (for example immediate `sql: no rows in result set` after event send) as polling gaps, not fatal send failures: it keeps the event-sent result, emits a warning if final state cannot be confirmed, and points operators to `joelclaw run` / `joelclaw runs` for inspection.
|
|
|
|
## Subscribe check queue pilot
|
|
|
|
```bash
|
|
joelclaw subscribe check [--id <id>]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- scoped checks (`--id <id>`) still emit `subscription/check.requested` directly to Inngest.
|
|
- all-subscription checks keep the legacy Inngest path by default.
|
|
- when `QUEUE_PILOTS=subscriptions`, `joelclaw subscribe check` without `--id` posts `subscription/check-feeds.requested` to the worker admission endpoint instead of writing Redis directly:
|
|
- returns queue metadata (`streamId`, `eventId`, `priority`)
|
|
- includes `triageMode` + `triage` metadata whenever the family is enabled for shadow or enforce
|
|
- points next actions at `joelclaw queue inspect` / `joelclaw queue depth`
|
|
- relies on the Restate queue drainer to forward the **actual** `subscription/check-feeds.requested` event name onward
|
|
|
|
## Jobs command tree (ADR-0217 runtime monitor)
|
|
|
|
```bash
|
|
joelclaw jobs
|
|
└── status [--hours <n>] [--count <n>] [--namespace <namespace>] [--restate-admin-url <url>] [--dkron-service-name <service>] [--dkron-base-url <url>]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- `jobs status` is the **first operator glance** for real workloads during the ADR-0217 transition. It aggregates the queue/Redis substrate, Restate runtime, Dkron scheduler, and still-live Inngest surfaces into one JSON snapshot.
|
|
- `overall.status` is a bounded truth surface (`healthy|degraded|down`) derived from those four components, not a raw dump of every underlying health probe.
|
|
- queue section:
|
|
- reads the canonical Redis queue directly through `@joelclaw/queue`
|
|
- reports depth, priority buckets, oldest age, and active deterministic pauses
|
|
- Restate section:
|
|
- mirrors `joelclaw restate status` in-place so the operator can see statefulset readiness + admin health without command hopping
|
|
- Dkron section:
|
|
- mirrors `joelclaw restate cron status` in-place and includes the count of Restate-tagged scheduler jobs when the API is reachable
|
|
- Inngest section:
|
|
- stays visible during migration, but its status is scoped to the transitional job path that still matters here: server/worker health plus recent run outcomes
|
|
- broad informational checks (for example unrelated k8s pod drift) stay in the payload without poisoning the top-level job monitor status
|
|
- next actions point directly at the management surfaces that actually move workload state: `queue control status`, `queue observe`, `queue resume`, `restate status`, `restate cron status`, and `runs`
|
|
|
|
## Pi async jobs monitor (`runtime_jobs_monitor`)
|
|
|
|
The loaded pi extension at `packages/pi-extensions/inngest-monitor/index.ts` now does two jobs:
|
|
|
|
1. tracks followed Inngest runs (`inngest_send`, `inngest_runs`)
|
|
2. exposes `runtime_jobs_monitor` for the ADR-0217 runtime substrate
|
|
|
|
`runtime_jobs_monitor` semantics:
|
|
|
|
- `action=start|status|stop` (default `start`)
|
|
- on `start`, it polls `joelclaw jobs status` in the background, paints a persistent widget, emits OTEL on severity changes and meaningful workload-state changes, and sends hidden follow-up messages for async report-back
|
|
- on `status`, it returns the latest runtime snapshot (overall status, queue depth, active pause count, Restate/Dkron/Inngest state)
|
|
- on `stop`, it stops the poller and sends a final follow-up summary
|
|
- widget posture is intentionally operator-first: current runtime state on top, active followed runs underneath
|
|
|
|
This is the canonical async monitoring path when you want a pi session to keep an eye on real workloads while you do other things.
|
|
|
|
## Queue command tree (ADR-0217 Phase 1)
|
|
|
|
```bash
|
|
joelclaw queue
|
|
├── emit <event> [-d <json>] [-p P0|P1|P2|P3]
|
|
├── depth
|
|
├── stats [--hours <n>] [--limit <n>]
|
|
├── observe [--hours <n>] [--limit <n>] [--since <iso|ms>]
|
|
├── pause <family> [--ttl <duration>] [--reason <text>]
|
|
├── resume <family> [--reason <text>]
|
|
├── control
|
|
│ └── status [--hours <n>] [--limit <n>] [--since <iso|ms>]
|
|
├── list [--limit <n>]
|
|
└── inspect <stream-id>
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- all queue subcommands return a clean JSON envelope; read-oriented subcommands close their Redis client before exit and `emit` stays a thin worker client instead of writing Redis directly
|
|
- queue state/control commands resolve Redis from the canonical joelclaw CLI config (`~/.config/system-bus.env` → `REDIS_URL`) before considering ambient shell env so the installed operator surface stays pointed at the same localhost queue as the worker and Restate drainer
|
|
- `emit` posts queue admission intent to the worker endpoint (`POST /internal/queue/enqueue`).
|
|
- accepts event name (e.g., `discovery/noted`, `content/updated`)
|
|
- accepts optional JSON payload via `-d`
|
|
- optional priority override via `-p` is normalized client-side, but static registry routing and bounded triage stay server-side
|
|
- the worker generates the canonical `QueueEventEnvelope`, adds trace metadata, resolves shadow/enforce mode for the event family, evaluates bounded triage, and persists the queue record
|
|
- returns the Redis stream ID, priority, and any `triageMode` / `triage` metadata from admission
|
|
- `depth` reports queue depth, priority distribution (P0/P1/P2/P3 counts), oldest/newest message timestamps
|
|
- `stats` summarizes recent Restate queue-drainer behavior plus Phase 2 triage behavior from OTEL over a lookback window.
|
|
- dispatch section reports sampled/found dispatch events, live queue depth, started/completed/failed counts, success rate, queue wait-time percentiles (`p50`/`p95`), dispatch-duration percentiles, promotion count, top event families, and recent failures
|
|
- triage section reports attempts, completed/failed/fallback counts, fallback counts by reason, disagreement count, applied-change count, suggested-not-applied count, route mismatches, latency percentiles, per-family rollups, and recent mismatch/fallback samples
|
|
- uses `metadata.waitTimeMs` from `queue.dispatch.started` as the Story 5 queue-to-dispatch latency signal
|
|
- uses `queue.triage.*` OTEL metadata as the Story 3 source of truth for queue-admission disagreements and fallback behavior
|
|
- `--since <iso|ms>` overrides the lower bound so operators can anchor soak evidence to a known clean point (for example the supervised `queue.drainer.started` after a rollout) instead of mixing fresh traffic with a dirty pre-fix window
|
|
- keeps the operator in CLI-land; no raw Redis keys or manual OTEL spelunking required for the first sanity pass
|
|
- `jobs status` is the first unified runtime view; drop to `queue` subcommands when the aggregated surface says the queue itself needs attention.
|
|
- `observe` is now the Phase 3 Story 2-4 dry-run Sonnet operator surface.
|
|
- builds a canonical live snapshot from current queue depth + queued messages + recent drainer OTEL + recent triage OTEL + gateway sleep/muted-channel state + active deterministic pauses
|
|
- runs the bounded Sonnet observer in `dry-run` mode only and returns the current `snapshot` plus the current `decision`
|
|
- long Sonnet summaries are trimmed instead of turning otherwise-useful observations into schema-only fallbacks
|
|
- if all queued work is intentionally held behind fresh active **manual** pauses and no recent failures suggest downstream trouble, it short-circuits to a deterministic `noop` instead of wasting a 60s Sonnet call on an obvious hold state
|
|
- `history` summarizes recent `queue.observe.*` OTEL for the same window so operators can compare the latest dry-run against raw history without spelunking Typesense by hand
|
|
- `control` now reflects the shipped deterministic queue-control plane: active manual pauses, `queue.control.applied|expired|rejected` counts, and recent control events come from the same Redis + OTEL truth the drainer uses
|
|
- `--since <iso|ms>` anchors the related OTEL history window the same way `queue stats` does
|
|
- `pause` applies a deterministic manual family pause with bounded TTL and emits `queue.control.applied` telemetry
|
|
- `resume` clears a deterministic family pause and emits either `queue.control.applied` or `queue.control.rejected` when the family was not paused
|
|
- `control status` is the dedicated deterministic control-plane operator surface.
|
|
- reports active pauses (family, reason, TTL, applied/expiry timestamps, actor)
|
|
- summarizes `queue.control.applied|expired|rejected` OTEL for the same window
|
|
- is the first CLI answer to “what queue controls are active right now?” before any automatic Sonnet mutation ships
|
|
- `list` lists recent messages in priority order (highest priority first), does not ack/remove
|
|
- `inspect` loads a message by Redis stream ID and returns full payload + metadata
|
|
- if the message is already acked/expired, it now returns a structured `QUEUE_MESSAGE_MISSING` error envelope with queue-state next actions instead of crashing the CLI
|
|
|
|
Queue configuration:
|
|
|
|
- Stream key: `joelclaw:queue:events`
|
|
- Priority index: `joelclaw:queue:priority`
|
|
- Consumer group: `joelclaw:queue:cli`
|
|
- Phase 1 pilot events: `discovery/noted`, `discovery/captured`, `content/updated`, `subscription/check-feeds.requested`, `github/workflow_run.completed`
|
|
|
|
OTEL telemetry under `queue.*` namespace:
|
|
|
|
- CLI queue commands forward queue package telemetry to OTEL with `source=cli` and `component=queue`
|
|
- `queue.enqueue` — message enqueued
|
|
- `queue.lease` — message leased for processing (includes wait time, priority, promotion metadata)
|
|
- `queue.ack` — message acknowledged
|
|
- `queue.replay` — unacked messages loaded for replay
|
|
|
|
## Daily summary command
|
|
|
|
```bash
|
|
joelclaw summary [--hours 24] [--format json|text]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- aggregates recent git activity (joelclaw + Vault), Inngest run rollups, k8s pod health, OTEL stats, slog deploy/config events, and ADR churn into one response envelope.
|
|
- default output is JSON summary payload; `--format text` adds a compact text rendition under `result.text` for downstream chat/mobile surfaces.
|
|
|
|
## Memory command group
|
|
|
|
```bash
|
|
joelclaw memory review --since 48h
|
|
joelclaw memory search "<query>"
|
|
joelclaw memory write "<legacy input>" # typed retirement response; no write occurs
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- `memory review` returns recent accepted-session, Git, Brain, OTEL, and flowing evidence in separate lanes.
|
|
- `memory search` is an alias for composed recall.
|
|
- `memory write` is a fail-closed compatibility pointer. Agent work enters through accepted Runs; durable curation belongs in Brain `.svx` pages.
|
|
|
|
## Knowledge turn-write command
|
|
|
|
```bash
|
|
joelclaw knowledge note \
|
|
--source gateway \
|
|
--agent gateway-daemon \
|
|
--session <session-id> \
|
|
--turn <turn-number> \
|
|
--summary "<what changed>"
|
|
```
|
|
|
|
Use `--skip-reason routine-heartbeat|duplicate-signal|no-new-information` when a turn is eligible but has no durable signal to capture.
|
|
|
|
## Gateway channel control
|
|
|
|
```bash
|
|
joelclaw gateway channel list
|
|
joelclaw gateway channel status [channel]
|
|
joelclaw gateway channel disable <channel> [--restart] [--force]
|
|
joelclaw gateway channel enable <channel> [--restart]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- edits `~/.joelclaw/scripts/gateway-start.sh` for the known runtime channels: `telegram`, `discord`, `imessage`, `slack`.
|
|
- writes a `/tmp/joelclaw/gateway-start.sh.*` backup before changing the script.
|
|
- never prints secret values; output reports variable names plus source classes (`secret`, `derived`, `literal`, `blank`, `missing`).
|
|
- `--restart` applies the change immediately via `joelclaw gateway restart`; otherwise next actions tell the agent to restart.
|
|
- disabling `telegram` requires `--force` because it is a primary operator channel.
|
|
|
|
## Gateway known issues / muted channels
|
|
|
|
```bash
|
|
joelclaw gateway known-issues
|
|
joelclaw gateway mute <channel> [--reason "<why muted>"]
|
|
joelclaw gateway unmute <channel>
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- stores muted channel IDs at Redis key `gateway:health:muted-channels` (JSON array).
|
|
- stores optional mute reasons at Redis key `gateway:health:mute-reasons` (JSON object).
|
|
- muted channels remain in probe telemetry but are excluded from `gateway.channels.degraded` alerts.
|
|
|
|
Gateway process-layer diagnostics (`joelclaw gateway diagnose`) now inspect exact launchd state for `com.joel.gateway` and report disabled services explicitly. `joelclaw gateway restart` now re-enables the launch agent before bootstrap/kickstart to avoid restart failures when launchd has the service disabled.
|
|
|
|
Use `joelclaw gateway enable` for direct launch-agent recovery (enable + bootstrap + kickstart) without manual `launchctl` usage.
|
|
|
|
Gateway status/diagnose now separate **daemon availability** from **Redis bridge health**:
|
|
|
|
- `joelclaw gateway status` prefers daemon `/health` when available, so Redis loss no longer makes the gateway look fully dead.
|
|
- status now returns `mode` (`normal` or `redis_degraded`), `degradedCapabilities`, `sessionPressure` (context %, compaction age, session age, next action), and `guardrails` (current turn tool budget state + pending deploy verifications).
|
|
- `joelclaw gateway diagnose` treats `redis_degraded` as a degraded runtime, not a process failure, skips Redis-dependent E2E checks in that mode, and surfaces runtime guardrail findings when a checkpoint/deploy verification is active.
|
|
|
|
## Gateway behavior control plane (ADR-0211)
|
|
|
|
```bash
|
|
joelclaw gateway behavior add --type keep|more|less|stop|start --text "..."
|
|
joelclaw gateway behavior list
|
|
joelclaw gateway behavior promote --id <candidate-id>
|
|
joelclaw gateway behavior remove --id <directive-id>
|
|
joelclaw gateway behavior apply
|
|
joelclaw gateway behavior stats
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- **Single write authority** is CLI. Extensions must call these commands; no direct Redis/Typesense writes.
|
|
- Active runtime contract lives in Redis key `joelclaw:gateway:behavior:contract`.
|
|
- Directive/candidate history lives in Typesense collection `gateway_behavior_history`.
|
|
- `add` normalizes directives, enforces conflict + dedupe + cap rules, and updates Redis + Typesense.
|
|
- `promote` moves a pending daily-review candidate into the active contract (manual gate; no auto-activation).
|
|
- `apply` re-runs governance over the active contract (dedupe, conflict cleanup, cap enforcement) and expires stale candidates.
|
|
- `stats` reports contract hash/version, candidate lifecycle counts, and governance settings.
|
|
|
|
## Run listing semantics
|
|
|
|
```bash
|
|
joelclaw runs [--status RUNNING|FAILED|COMPLETED|QUEUED|CANCELLED]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- applies backend status filtering and a local status guard so mixed-status payloads cannot leak through.
|
|
- the runs list query uses a longer timeout budget than generic Inngest GraphQL calls because self-hosted `runs(...)` can take ~50s even for small filtered windows; timeout scales upward with requested `count`, and timeout aborts get one bounded retry with a larger budget before the CLI gives up.
|
|
- for suspicious `RUNNING` rows (endedAt present while running, or long-running health checks), runs performs bounded detail reconciliation (max 5 lookups) to detect stale SDK-unreachable ghosts.
|
|
- runs with stale indicators include `staleSignal` with `likely`, `confidence`, and machine-readable `reasons`.
|
|
- response includes `staleSignals` summary (`detected`, `likely`, `detailChecked`) and suggests `joelclaw inngest sweep-stale-runs` when likely ghosts are present.
|
|
- `count` in response reflects post-filter rows (what the operator actually sees).
|
|
|
|
## Run inspection + cancellation
|
|
|
|
```bash
|
|
joelclaw run <run-id> [--cancel] [--wait-ms 3000]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- default mode returns run detail, trigger event, trace, and step errors.
|
|
- run detail and `joelclaw event <event-id>` now use a much larger timeout/retry budget than generic GraphQL calls because self-hosted detail lookups can take more than a minute under load even while the runtime itself is healthy.
|
|
- `--cancel` issues Inngest GraphQL `cancelRun` for active runs, then polls status up to `--wait-ms`.
|
|
- deterministic error envelopes:
|
|
- `RUN_CANCEL_FAILED` when `cancelRun` mutation fails.
|
|
- `RUN_CANCEL_TIMEOUT` when run remains `RUNNING|QUEUED` after the wait window.
|
|
- `RUN_STALE_SDK_UNREACHABLE` when a run appears `RUNNING` but trace errors show `Unable to reach SDK URL` and cancellation has no live execution to target.
|
|
- terminal runs are never re-cancelled; response includes `cancellation.skipped = "already_terminal"`.
|
|
|
|
## Capability adapter config precedence (ADR-0169 phase 0)
|
|
|
|
Resolution order is deterministic:
|
|
|
|
1. CLI flags (e.g. `--adapter`)
|
|
2. Environment variables
|
|
3. Project config (`.joelclaw/config.toml`)
|
|
4. User config (`~/.joelclaw/config.toml`)
|
|
5. Built-in defaults
|
|
|
|
Current env keys:
|
|
|
|
- `JOELCLAW_CAPABILITY_<CAPABILITY>_ADAPTER`
|
|
- `JOELCLAW_CAPABILITY_<CAPABILITY>_ENABLED`
|
|
|
|
## Capability-backed command roots (ADR-0169 through phase 4)
|
|
|
|
```bash
|
|
joelclaw secrets status
|
|
joelclaw secrets lease <name> --ttl 15m
|
|
joelclaw secrets revoke <lease-id>
|
|
joelclaw secrets revoke --all
|
|
joelclaw secrets audit --tail 50
|
|
joelclaw secrets env --dry-run [--ttl 1h] [--force]
|
|
|
|
joelclaw log write --action <action> --tool <tool> --detail <detail> [--reason <reason>] [--session <session>] [--system <system>]
|
|
|
|
joelclaw notify send "<message>" [--kind memory|alert|digest|ask|receipt] [--priority low|normal|high|urgent|critical] [--channel gateway|main|all] [--context '{"k":"v"}']
|
|
joelclaw notify wait <event-id> --source <source> [--timeout 15s]
|
|
joelclaw messages trace <flowId>
|
|
|
|
joelclaw heal {list|run}
|
|
|
|
joelclaw mail {status|register|send|inbox|read|reserve|renew|release|locks|search}
|
|
|
|
joelclaw otel {list|search|stats|emit}
|
|
|
|
joelclaw o11y {session|system}
|
|
|
|
joelclaw recall <query> [--limit N] [--min-score F] [--raw] [--include-hold] [--include-discard] [--budget auto|lean|balanced|deep] [--category <id|alias>]
|
|
|
|
joelclaw sessions search <query> [--source typesense|local|ssh|both] [--runtime pi|codex|claude-code|all] [--extract] [--machine dark-wizard] [--ssh-target joel@dark-wizard] [--limit N]
|
|
joelclaw session search <query> ... # singular alias
|
|
joelclaw sessions extract <session-id-or-path> --query <topic> [--format json|markdown]
|
|
joelclaw sessions chunks <query> [--source typesense|local|both] [--limit N] [--context-before N] [--context-after N]
|
|
joelclaw sessions inspect <session-id-or-path> --around <regex> [--before N] [--after N]
|
|
joelclaw sessions signals [--kind friction|preference|decision|praise|mode-mismatch|any] [--source local|ssh|both] [--machine dark-wizard] [--since 14d] [--limit N] [--sample N] [--review-out path.jsonl] [--evaluate] [--format json|ndjson]
|
|
joelclaw sessions friction [--source local|ssh|both] [--machine dark-wizard] [--since 14d] [--limit N]
|
|
|
|
joelclaw satellite health [--notify] [--central-ssh joel@panda] [--priority high]
|
|
joelclaw satellite repair-request [--central-ssh joel@panda] [--priority high]
|
|
|
|
joelclaw subscribe {list|add|remove|check|summary}
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- `log` writes structured system entries (slog backend).
|
|
- `log write` now accepts explicit `--session` / `--system` provenance flags and also falls back to `SLOG_SESSION_ID` / `SLOG_SYSTEM_ID` env vars before handing off to the slog backend.
|
|
- `logs` reads/analyzes runtime logs.
|
|
- `notify` is the canonical producer command for messages to Joel; `gateway push` remains transport/debug. Plain text is a complete payload. Producers do not need to add or change flags for the Agent Comms Gateway.
|
|
- Message text, `--kind`, `--priority`, `--channel`, `--telegram-only`, and structured evidence are evidence for the gateway agent. They never instruct routing, urgency, timing, formatting, batching, suppression, or platform choice. Old flags remain accepted without deprecation nagging.
|
|
- The current CLI accepts structured JSON evidence through `--context`. The producer-facts contract reserves `--data <json>` and repeated `--ref`, but `joelclaw notify send` does not implement those flags yet. Add run IDs, receipts, links, or available actions only when they help the agent make or verify its decision.
|
|
- The gateway agent records one `gateway.decision.recorded` receipt for each consumed external event before advancing its stream cursor. Recorded `deliver` and `aggregate/close-deliver` decisions are executed mechanically by the slim transport. The current decision executor delivers only to Telegram.
|
|
- `messages trace <flowId>` reads the stream and delivery lifecycle. Correlate replies, reactions, button taps, decisions, and platform receipts by `flowId`.
|
|
- `deploy`, `heal`, `log`, `notify`, `secrets`, `mail`, `otel`, `recall`, and `subscribe` keep their existing UX/envelopes while executing through capability registry adapters (`scripted-deploy`, `runbook-heal`, `slog-cli`, `gateway-redis`, `agent-secrets-cli`, `mcp-agent-mail`, `typesense-otel`, `typesense-recall`, `redis-subscriptions`).
|
|
- `typesense-otel`, `typesense-recall`, `scripted-deploy`, `runbook-heal`, `slog-cli`, `agent-secrets-cli`, `gateway-redis`, `mcp-agent-mail`, and `redis-subscriptions` adapter logic is canonical in `@joelclaw/sdk` (`packages/sdk/src/capabilities/adapters/*`); CLI adapter files are thin re-exports.
|
|
- `otel emit` accepts stdin JSON payloads (or convenience args/positional action), normalizes defaults (`id`, `timestamp`, `level=info`, `success=true`), and forwards to the worker ingest endpoint (`/observability/emit`). Provenance defaults are now role-aware: explicit event/session fields win, then `SLOG_SESSION_ID` / `SLOG_SYSTEM_ID`, then session handle envs when present, then role/hostname fallbacks (`gateway` or `JOELCLAW_ROLE`, system hostname stripped to `panda`).
|
|
- `otel list` and `otel search` accept exact `--session` / `--system` filters, mapped to `sessionId` / `systemId` in Typesense.
|
|
- `knowledge search` now auto-heals the common post-rebuild failure mode where `system_knowledge` is missing: on a `404 Collection not found` response it creates the collection, reindexes ADRs + skills once, retries the query, and reports the repair in-band.
|
|
- `o11y session` / `o11y system` run a unified multi-search across `otel_events` and `system_log`, merge both timelines by `timestamp`, and tag each hit with its source collection.
|
|
- `sessions search` is explicit raw-history drill-down. It reads native transcript adapters and the current local SQLite projection when available; `--source local` never depends on retired Typesense Run collections. Use `--runtime pi|codex|claude-code|all` and verify the returned native path/runtime before claiming capture success. Remote index pointers are compatibility evidence, not a memory lane.
|
|
- `sessions search --extract` runs bounded deterministic extraction for top raw hits so recovery prompts can include decisions, commands, files, receipts, verification, blockers, next actions, and exact transcript line pointers without dumping the full JSONL. Each emitted hit carries its own `extraction` when available; `.result.extractions` remains as a backward-compatible convenience list. Large session envelopes are written through direct stdout, not Effect Console, so `| jq` remains safe above 64KB.
|
|
- `sessions extract` reads one raw local transcript by path or session-id substring, redacts likely secrets, and returns JSON extraction fields. `--format markdown` still uses the standard JSON envelope and places rendered markdown at `.result.markdown`; raw markdown stdout would violate the CLI JSON contract.
|
|
- `sessions chunks` exposes matching Typesense chunks and/or raw local neighboring transcript context. Use top-level `.result.chunks`/`.result.hits`; source-specific mirrors remain under `.result.local.chunks` and `.result.typesense.chunks`.
|
|
- Source metadata uses `rawReturned` for pre-dedupe raw source count and `emittedHits` / `emittedChunks` for top-level emitted counts.
|
|
- `joelclaw session ...` is a singular alias for `joelclaw sessions ...`.
|
|
- `sessions signals` mines role-aware high-signal user turns from raw local/remote Pi transcripts. It is ADR-0247-backed and starts deterministic: user turns are the primary signal source; assistant/tool turns are bounded evidence context. `--kind friction` detects correction/frustration/output-quality signals, including Joel's strong-emphasis profanity where it is attached to critique. `sessions friction` is an alias for `sessions signals --kind friction`. `--sample N` balances candidates across signal categories; `--review-out path.jsonl` writes a tiny golden-set review file with stable `hitId`, turnKind, prediction, improvement route, evidence, and empty review fields. `--evaluate` adds a small routing summary by improvement surface/kind/turnKind/confidence/review priority. V1 excludes task payloads, source material, and handoffs by default so friction analysis focuses on operator intent, review feedback, and approvals. `--kind mode-mismatch` finds cases where agents picked the wrong execution shape: inline vs background, visual vs response, durable Inngest vs ad-hoc script, or feedback-blocking vs async. `--format ndjson` streams `meta`, `hit`, and `summary` rows for deep scans without building one huge envelope.
|
|
- `satellite health` is the thin-Machine self-check surface. It verifies local prerequisites like `python3`, `jq`, `joelclaw`, raw Pi session storage, and local session search JSON. With `--notify`, degraded health is relayed to Central via SSH (`--central-ssh`, default `joel@panda`) using `joelclaw notify send --type satellite/repair.requested`; this lets a satellite ask the gateway for repair without local Redis/secrets.
|
|
- `satellite repair-request` always sends the Central gateway a repair request with the local probe payload.
|
|
- New satellite Machines should use `scripts/setup-satellite-rig.sh` plus `docs/runbooks/satellite-rig-setup.md`. The bootstrap keeps satellites thin: repo + CLI + skills + raw session search + Central repair relay. It writes `~/.local/bin/joelclaw` as a wrapper that sources `~/.config/system-bus.env`, so Typesense and PDF Brain docs-api credentials work over non-interactive SSH without installing Panda's agent-secrets stack. Do not install Panda's k8s/Inngest/Redis/gateway stack on a satellite by default.
|
|
- Software surfaces should route OTEL through this command contract (or shared CLI ingest helper), not ad-hoc raw HTTP calls.
|
|
- `mail search` auto-falls back to `/mail/api/unified-inbox` filtering when MCP `search_messages` returns transient DB/tool errors, so steering signals remain usable.
|
|
- `mail reserve` now sends explicit lease TTL (`--ttl-seconds`, default `900`) and enforces a minimum of 60s.
|
|
- `mail renew` extends active file reservations without releasing/reacquiring (`--extend-seconds`, default `900`, optional `--paths`).
|
|
- `mail locks` now prefers the local git-mailbox `file_reservations/` artifact store when available because `/mail/api/locks` can under-report advisory file reservations while still reporting mailbox internals like archive/commit locks. Responses expose `source` and `fallback_reason` when artifact fallback was required.
|
|
- `subscribe check` emits Inngest request events for scoped checks and for all-subscription checks when the queue pilot is off; `response.ids` are event/request IDs (inspect via `joelclaw event <event-id>`), not run IDs unless explicitly returned as `runIds`.
|
|
- when `QUEUE_PILOTS=subscriptions`, unscoped `subscribe check` returns queue metadata instead of Inngest response ids because the request first lands in Redis and is forwarded by the Restate drainer.
|
|
- `recall` rewrite telemetry now exposes `rewrite.strategy` (`disabled|skipped|haiku|openai|fallback`) and `rewrite.reason` so low-ROI rewrite skips/fallbacks are queryable; short/literal/direct-id queries skip LLM rewrite by design. The default rewrite model is `openai-codex/gpt-5.6-sol`; set `JOELCLAW_RECALL_REWRITE_MODEL` only to an approved model.
|
|
|
|
## Webhook command tree (ADR-0185)
|
|
|
|
```bash
|
|
joelclaw webhook
|
|
├── subscribe <provider> <event>
|
|
│ [--repo <owner/repo>] [--workflow <name>] [--branch <name>] [--conclusion <status>]
|
|
│ [--session <session-id>] [--ttl <duration>] [--stream] [--timeout <seconds>] [--replay <count>]
|
|
├── unsubscribe <subscription-id>
|
|
├── list [--provider <provider>] [--event <event>] [--session <session-id>]
|
|
└── stream <subscription-id> [--timeout <seconds>] [--replay <count>]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- Subscriptions are Redis-backed and session-scoped (`joelclaw:webhook:*`).
|
|
- `subscribe --stream` starts an NDJSON stream immediately after creation.
|
|
- `stream` emits ADR-0058 NDJSON (`start`, `log`, `event`, terminal `result|error`).
|
|
- Default session target is `gateway` for central gateway role, otherwise `pid-<ppid>`.
|
|
- TTL defaults to `24h` and is enforced at match time.
|
|
|
|
## Skills command tree (ADR-0179)
|
|
|
|
```bash
|
|
joelclaw skills
|
|
└── audit [--deep] [--wait-ms <wait-ms>] [--poll-ms <poll-ms>]
|
|
```
|
|
|
|
### `joelclaw skills audit` purpose
|
|
|
|
- triggers the `skill-garden/check` event on-demand
|
|
- waits for the corresponding run and returns the findings report in-envelope
|
|
- supports `--deep` for LLM staleness checks
|
|
|
|
## Agent command tree (ADR-0180 phases 2-4)
|
|
|
|
```bash
|
|
joelclaw agent
|
|
├── list
|
|
├── show <name>
|
|
├── run <name> <task> [--cwd <cwd>] [--timeout <seconds>]
|
|
├── chain <steps> --task <task> [--cwd <cwd>] [--fail-fast]
|
|
└── watch <id> [--timeout <seconds>]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- `run` emits `agent/task.run` for single roster agent execution and returns `taskId` plus `eventIds` from the Inngest send response.
|
|
- `run` `next_actions` are truthful: use `joelclaw event <event-id>` when an event ID exists (or `joelclaw events ...` fallback), and never assume `taskId` is a run ID.
|
|
- `chain` emits `agent/chain.run` with comma-separated sequential steps and `+` parallel groups (e.g. `scout,planner+reviewer,coder`).
|
|
- `watch` streams NDJSON progress for a task (`at-...`) or chain (`ac-...`) by subscribing to `joelclaw:notify:gateway`, replaying `joelclaw:events:gateway`, and falling back to Inngest polling.
|
|
- `watch` default timeout is 300 seconds for tasks and 900 seconds for chains; terminal events always include `next_actions` on completion, timeout, or interrupt.
|
|
- Runtime-proof recipe (ADR-0180):
|
|
1. `joelclaw agent list` (expect builtin `coder/designer/ops/story-executor`)
|
|
2. `joelclaw agent run coder "reply with OK" --timeout 20`
|
|
3. `joelclaw event <event-id>` (expect `Agent Task Run` status `COMPLETED` with output payload)
|
|
- If `Unknown agent roster entry: coder` appears, treat it as worker-runtime drift: deploy latest `system-bus-worker`, restart the host worker, then rerun the three-step proof.
|
|
|
|
## Vault command tree
|
|
|
|
```bash
|
|
joelclaw vault
|
|
├── read <ref>
|
|
├── search <query> [--semantic] [--limit <limit>]
|
|
├── ls [section]
|
|
├── tree
|
|
└── adr
|
|
├── list [--status <status>] [--limit <limit>]
|
|
├── collisions
|
|
├── audit
|
|
├── locate <query> [--limit <limit>]
|
|
├── refs <query>
|
|
├── prompt <text>
|
|
└── rank [--band <band>] [--unscored] [--all]
|
|
```
|
|
|
|
### `joelclaw vault adr` purpose
|
|
|
|
- `list` — inventory ADR metadata with optional status filter
|
|
- `collisions` — detect duplicate ADR numeric prefixes
|
|
- `audit` — full ADR hygiene check:
|
|
- missing/non-canonical status values
|
|
- number collisions
|
|
- missing `superseded-by` targets
|
|
- broken or ambiguous wiki links inside ADR bodies (skips custom directive tags like `[[tts:text]]`)
|
|
- README index alignment against ADR files
|
|
- `locate` — section/file lookup across the ADR corpus using ADR numbers, slugs, titles, and heading names
|
|
- `refs` — backlink discovery for a resolved ADR file or section id
|
|
- `prompt` — expand `[[ADR refs]]` in prompt text into canonical ids and append an `<adr-context>` block for agents
|
|
- `rank` — score + rank ADRs by NRC+novelty rubric for daily prioritization:
|
|
- default scope: open ADRs (`accepted` + `proposed`)
|
|
- `--all` includes shipped/superseded/deprecated/rejected ADRs
|
|
- `--band <band>` filters ranked rows (`do-now|do-next|de-risk|park`; alias `next` → `do-next`)
|
|
- `--unscored` returns ADRs missing `priority-score`
|
|
- required axes: `priority-need`, `priority-readiness`, `priority-confidence`
|
|
- novelty facet: `priority-novelty` (or alias `priority-interest`), defaults to neutral `3` when missing
|
|
- score formula: `clamp(round(20*(0.5*Need + 0.3*Readiness + 0.2*Confidence)) + round((Novelty-3)*5), 0, 100)`
|
|
- bands: `do-now` (80-100), `do-next` (60-79), `de-risk` (40-59), `park` (0-39)
|
|
- emits CLI OTEL via `component=vault-cli`:
|
|
- `vault.adr.rank.started`
|
|
- `vault.adr.rank.completed`
|
|
- `vault.adr.rank.failed`
|
|
|
|
Canonical statuses:
|
|
|
|
- `proposed`
|
|
- `accepted`
|
|
- `shipped`
|
|
- `superseded`
|
|
- `deprecated`
|
|
- `rejected`
|
|
|
|
## Content command tree (ADR-0168)
|
|
|
|
```bash
|
|
joelclaw content
|
|
├── seed
|
|
├── verify
|
|
└── prune [--apply]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- `seed` — full Vault ADR sync to Convex for canonical ADR filenames only (`NNNN-*.md`).
|
|
- `verify` — strict ADR drift check against canonical ADR files (fails healthy state on both missing and extra ADR records in Convex).
|
|
- `prune` — dry-run report of Convex ADR extras (`status: dry_run`).
|
|
- `prune --apply` — removes ADR extras from Convex (`status: pruned`) and should be followed by `joelclaw content verify`.
|
|
|
|
## Inngest source guard (ADR-0089)
|
|
|
|
```bash
|
|
joelclaw inngest source [--repair]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- Verifies launchd binding for `com.joel.system-bus-worker` against the canonical `infra/launchd/com.joel.system-bus-worker.plist` values (program + working directory).
|
|
- `--repair` copies canonical plist into `~/Library/LaunchAgents`, performs `launchctl bootout`, then `bootstrap` with retry for transient `Bootstrap failed: 5` launchd races.
|
|
- Use before `joelclaw inngest restart-worker` when host runtime/source drift is suspected.
|
|
|
|
## Inngest stale-run sweep (ADR-0194)
|
|
|
|
```bash
|
|
joelclaw inngest sweep-stale-runs
|
|
joelclaw inngest sweep-stale-runs --apply
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- preview-first by default (`--apply` required for mutation).
|
|
- scope defaults to stale health checks older than 30 minutes:
|
|
- `check/o11y-triage`
|
|
- `check/system-health`
|
|
- runtime target defaults:
|
|
- namespace `joelclaw`
|
|
- pod `inngest-0`
|
|
- sqlite path `/data/main.db`
|
|
- apply mode safety gates:
|
|
- refuses if age threshold is too young (`<5m`)
|
|
- refuses when candidate count exceeds `--max-apply-candidates`
|
|
- refuses when `function_runs` rows are missing (cannot insert terminal history safely)
|
|
- always creates and verifies point-in-time backup before transaction:
|
|
- `/data/main.db.pre-sweep-<UTC-stamp>.sqlite`
|
|
- terminalization contract (single transaction):
|
|
1. insert missing `history.type = FunctionCancelled`
|
|
2. insert missing `function_finishes`
|
|
3. set `trace_runs.status = 500` + terminal `ended_at`
|
|
|
|
This command exists for cases where Inngest API cancellation returns `not found` for stale RUNNING ghosts after SDK reachability failures.
|
|
|
|
## Status command
|
|
|
|
```bash
|
|
joelclaw status [--agent-dispatch-canary]
|
|
```
|
|
|
|
Semantics:
|
|
|
|
- default `joelclaw status` remains the fast base worker/server health surface
|
|
- default output now also includes `latestAgentDispatchCanary` when a persisted deterministic canary snapshot exists, so operators can see the last proof result without spelunking runs or inbox files
|
|
- `--agent-dispatch-canary` runs `scripts/verify-agent-dispatch-timeout.ts` and folds the deterministic non-LLM `system/agent-dispatch` timeout proof into the returned envelope
|
|
- when requested, the command only reports healthy if both the base health probes and the canary pass
|
|
- this is the canonical on-demand proof surface for the live outer-timeout closeout path; it exists so operators do not have to run the verifier script manually
|
|
|
|
## Build and verify
|
|
|
|
```bash
|
|
bunx tsc --noEmit
|
|
pnpm biome check packages/ apps/
|
|
bun test packages/cli/src/commands/*.test.ts
|
|
bun build packages/cli/src/cli.ts --compile --outfile ~/.bun/bin/joelclaw
|
|
joelclaw status
|
|
joelclaw vault
|
|
joelclaw vault adr audit
|
|
joelclaw vault adr locate "gateway guardrails"
|
|
joelclaw vault adr refs 0189-gateway-guardrails
|
|
joelclaw vault adr prompt "compare [[0189-gateway-guardrails]] with [[0218-gateway-availability-lifecycle-qol-improvements]]"
|
|
```
|
|
|
|
## Add a command
|
|
|
|
1. Create command module in `packages/cli/src/commands/`.
|
|
2. Return envelopes with `respond`/`respondError` only.
|
|
3. Include useful `next_actions` with param hints.
|
|
4. Wire command in `packages/cli/src/cli.ts`.
|
|
5. Add/extend tests in `packages/cli/src/commands/*.test.ts`.
|
|
6. Update this file when command tree or contracts change.
|