The F-B lockout (live-verified in class1 run #2): provision rotated
HELMOR_COMPANION_TOKEN via 'secret put' BEFORE deploy/verify, and the desktop
only receives the token on the terminal 'deployed' line — so any mid-run
failure stranded the Worker on a NEW token the desktop never saw
(desktop/Worker/container three-way mismatch, existing teams only).
Fix (architect-approved failure matrix in class4-design.md par.2): reuse-or-
generate. On a RE-provision the frontend passes the desktop's stored admin
token (getTeamAdminToken) -> deploy_team_cloud forwards it to the script as
HELMOR_EXISTING_COMPANION_TOKEN (env only, never logged) -> provision reuses
it, so 'secret put' writes the value the desktop ALREADY holds. After that
point every failure row has desktop == Worker; a pre-existing F-B mismatch
even self-heals (the put forces the Worker back to the desktop's value).
The step ORDER is deliberately unchanged: rotate-then-put stays for FRESH
teams (nothing held -> nothing to lock out; 'wrangler secret put' before the
first deploy also auto-creates the draft Worker, a load-bearing behavior both
rollouts exercised). F-C's 401 propagation grace and destroy-warm keep their
semantics — on the reuse path the propagation window physically disappears
(same value), on the fresh path they still do their verified job. Forced
rotation remains reachable via Leave team -> Create team (clears the token).
resolveAdminToken extracted pure (cloud/scripts/admin-token.ts) with unit
tests; the frontend wiring is pinned by team-create-flow tests (red-verified
against the unwired call).
Cold start previously minted EVERY member's ForgeIdentity DO and wrote
the whole team's gh/glab tokens into the shared container — one induced
'cat' away from a full-team credential leak. Per the user product
ruling the creator-identity model stays (everyone's pushes reuse the
creator's forge identity); only the storage narrows:
- teams.forge_identity_member_id: the forge 'creator', bound
FIRST-authorizer-wins on PUT /team/forge-identity (mirrors
cloud_identity_member_id's single-row upsert, but COALESCE keeps an
existing binding; a non-creator racing to authorize first is the same
accepted TOFU edge as cloud identity). Added via the P1-4c idempotent
D1 ALTER runner (0002) + schema.sql for fresh DBs.
- collectMemberForgeCreds injects tokens ONLY for the creator entry
(flagged creator:true); every other member is login-only (kept for
per-member commit authorship). Local-docker launcher mirrors the same
narrowing.
- member_creds.rs (CONTAINER RUST — needs the next team-image bump):
token lookups fall back to the creator's tokens when the acting
member has none, and when no acting member is bound at all (the
detached auto-push thread). Old injected files without the creator
flag still parse (no fallback). Authorship lookups (member_for) do
NOT fall back.
NOTE: P1-8a (scrubbing the token out of .git/config) depends on this
creator fallback for push auth — rolling back this commit requires
rolling back the P1-8a commit with it.
bun 1.3.2 running wrangler 4.100's deploy / deployments list swallows the
output (banner only, exit 0) → provision's Worker-URL parsing dies on the
dev default path (repo .ts under PATH bun); d1 list --json fails the same
way intermittently. The scripts now self-decide wrangler's runtime
(wrangler-runtime.ts, pure + unit-tested): HELMOR_WRANGLER_BIN override
unchanged → under bun prefer the HELMOR_WRANGLER_NODE hint (vendored node,
injected by the Rust spawn points when it exists) → a PATH node → loud
warning + bun as the last resort. Under node (release .mjs) nothing
changes. The .ts orchestration layer stays on bun — the vendored node
can't execute .ts.
provision rotates HELMOR_COMPANION_TOKEN right before verifyLive, but CF
secret propagation lags ~3-10s (measured live) — the old code hard-failed
on the first 401. The health poll now retries 401/403 within a 60s grace
window (the value the acceptance run verified green), then fails with the
named auth error; permanent errors and the 180s total deadline keep their
semantics. destroy-warm-container gets the same bounded, still-best-effort
401 retry. Loop extracted to verify-live.ts (injected probe/clock/sleep)
and unit-tested red-first in cloud/test/verify-live.test.ts.
schema.sql claimed "the orchestrator runs the ALTER once at deploy" — no such
orchestrator existed. Any D1 predating a column (long-lived dev DBs;
re-provision onto an already-set-up account, where d1 create and the schema
apply both no-op) kept missing cloud_identity_member_id and team.ts's
INSERT/SELECT hit "no such column" → 500, and the next additive column would
have hit the same wall with no runner.
- d1-migrations.ts: additive-migration list + classifyMigrationError —
SQLite's "duplicate column name" (fresh DB / re-run) is already-applied,
everything else is fatal. Comment documents the upgrade path: if migrations
ever stop being additive/duplicate-classifiable, move to a versioned
schema_migrations table instead of stretching this list.
- provision-team.ts executes the list right after the schema apply on every
provision / re-provision.
- schema.sql's lying migration comment rewritten to the real mechanism.
Validated against a real D1 (scratch DB, deleted after): old-shape teams
table → migration exit 0 and the column SELECTs; re-run → wrangler-wrapped
"duplicate column name … SQLITE_ERROR [code: 7500]" (captured verbatim into
the test) → classified already-applied.
A fresh Cloudflare account never had BROKER_ENC_KEY (it was only ever set by
hand on the dev account), so the first agent authorize threw on
atob(undefined) inside the identity DOs and every fresh team 500'd its way to
an un-finishable create flow.
- provision-team.ts: after the companion-token put, probe `wrangler secret
list` and decide via pure logic in the new broker-key.ts. The four branches
(unit-tested): key present → skip; absent → put (32 random bytes, standard
base64, never logged); list failed with the Worker-not-found error (fresh
account, verbatim shape captured from wrangler 4.100.0) → put; ANY other
list failure → the whole provision fails. ⚠️ Writing only-when-absent is a
hard safety constraint: rotating an existing key bricks every stored
identity. Black-box check against the live dev account (key present since
the manual setup) decides "skip"; the live worker-not-found error decides
"put".
- codex/claude/forge identity DOs: importKey now names the real problem
("BROKER_ENC_KEY is not configured … re-run team setup") instead of an
opaque atob crash wrapped as a blind 500 — covers Workers provisioned
before this fix. Guard covered by the codex + claude twin suites; the forge
copy is verbatim-identical.
- wrangler.toml: BROKER_ENC_KEY added to the documented secrets list (it was
missing there too), with the do-not-rotate warning.
A release install could never create a team — three independent dead ends:
cloud/scripts wasn't bundled, \`bun\` was spawned by bare name (Finder PATH has
none), and wrangler was never locatable (the "Rust passes HELMOR_WRANGLER_BIN"
comment was a lie). The error then pointed at the Advanced-setup panel R5-A
deleted — an infinite loop with no exit.
- stage-vendor: new team-cloud lane (release-gated like cursor-worker) stages
vendor/team-cloud/ = prebuilt provision-team.mjs + team-containers.mjs
(bun build --target=node), wrangler.toml + schema.sql + Worker src, and a
PRUNED wrangler node_modules (keep-the-JS-drop-the-payloads: workerd binary
×2 + sharp/@img removed; 189MB → 67MB, every provision verb validated green
— evidence in .agent-contexts/team-cloud-round6/class1-findings.md F1).
Integrity asserts fail the build if any provisioning file is missing, the
prune regressed, a wrong-arch esbuild slipped in, or the pruned tree stops
loading (wrangler --version smoke).
- commands/team_cloud: new tooling module resolves script + runtime — dev:
repo .ts under PATH bun (unchanged); release: staged .mjs under the vendored
Node (zero PATH dependency). Truthful errors, no more Advanced-setup ghost.
- provision-team.ts/team-containers.ts: wrangler self-resolved next to the
script and run under process.execPath (the .bin shim's \`env node\` shebang
can't work from Finder); wrangler cwd moved to a writable temp dir (release
payload lives in read-only Resources); schema path made absolute; login
detection fixed to parse whoami OUTPUT (wrangler 4.100 exits 0 when logged
out, so fresh machines never got the browser login).
- create-flow: error phase gains Copy details next to Try again.
- cloud/scripts now typechecked (tsc -p scripts, node types only) via cloud's
typecheck chain; bun-only local-docker-team.ts stays excluded by design.
Every idle-sleep destroyed Claude/Codex thread state: the serve process
HOME is pinned to /root (VERIFIED auth semantics), so claude-code wrote
threads to /root/.claude/projects and codex to /root/.codex — both
OUTSIDE the backup dir (/home/helmor, the only allowed backup root
covering them is /home). Result (R2-F4, and retroactively R2-F3 — same
root cause, formally overturned): after every wake, resuming any
pre-sleep thread found nothing, the provider emitted an empty end, and
the turn failed with 'empty response' forever (the #402 guard keeps the
resume id, so retries never recover).
Fix = relocation, not backup surgery:
- startProcess env: CLAUDE_CONFIG_DIR=/home/helmor/.claude and
CODEX_HOME=/home/helmor/.codex. Both are first-class CLI vars and the
existing seeding paths already honor them (codex auth.json write uses
$CODEX_HOME; the Claude onboarding seed writes
$CLAUDE_CONFIG_DIR/.claude.json). HOME stays /root — auth env
semantics untouched.
- start-serve.sh exports the same defaults so the boot path matches.
- backup excludes gain glob entries for the heavy non-thread noise the
relocated homes carry (telemetry/downloads/paste-cache/snapshots/
file-history, codex worktrees + logs) to guard the 15s budget.
Existing threads under /root on the live container are lost at rollout
— equivalent to the status quo (they were lost on every sleep anyway).
Ships as :0.1.6.
local-docker never injected HELMOR_SYNC_URL, so the in-container TeamSync stayed
inert — team-mode history (GET /team/messages) never populated locally and the
Stage B write-through path (which WP3 changes) couldn't be exercised. Inject
HELMOR_SYNC_URL=http://host.docker.internal:<proxyPort> (mirroring what the real
Worker's ensureServe does), bind the proxy on 0.0.0.0 so the container can reach
it, and classify the shared companion token as admin on the local gateway (the
real Worker treats companion==admin; the split local tokens otherwise 401 the
PUT /team/sync). Enables full local-docker e2e of the room-chat → D1 mirror.
A re-provision rotates HELMOR_COMPANION_TOKEN, but a container still WARM from a
prior run caches the OLD token (injected at its startProcess) and 401s every
proxied rpc — so Reset→re-provision fails verify's model check and recovery
needed a manual `wrangler containers delete` + redeploy. `/admin/restart-sandbox`
can't fix it: it uses stop() (SIGTERM), which `helmor serve` ignores (the VM
stays up with the old token — see onActivityExpired). destroy() (SIGKILL) scales
it to zero.
- index.ts: add `/admin/destroy-sandbox` (exported `handleAdminDestroySandbox`)
— ADMIN-token only (companion token AND no derived member id), calls
sandbox.destroy(). Handled before ensureServe so it never cold-starts itself.
- provision-team.ts: after deploy, before verify, best-effort POST the endpoint
to destroy any warm container. A non-2xx / network error is logged and
swallowed — verify stays the backstop that catches a token mismatch.
Tests: 3 cases for the route (admin bearer → destroy + 200; missing/wrong token
→ 401 + no destroy; member bearer with X-Helmor-Member-Id → 401 + no destroy).
cloud/ only — no frontend, no Rust.
Setup went "all green" on a backend that could not run a turn (S3), and a
permanent container-start failure surfaced as a ~20s hang + raw "Load failed"
copy, or a 180s "stuck connecting" (S1/S2). Root-cause work also found the
in-app provision path shipped a STALE container image, diverging the container
from the Worker it runs under.
- provision: stop hard-coding the container image tag. wrangler.toml is now the
single source of truth (kept in lockstep with publish-team-image.yml); only
HELMOR_TEAM_IMAGE overrides it. Removes the Worker↔container version skew.
- provision VERIFY is now REAL end-to-end: Worker auth → container actually
starts (/v1/health through ensureServe) → model catalog returns. A failing
stage exits non-zero with a stage-named error; the create-flow surfaces it.
- Worker ensureServe classifies the Sandbox SDK's PERMANENT start error (a 500
with context.phase "startup") and FAST-FAILS with a structured `permanent`
503 (+ phase) instead of burning the ~15s restore/mint + 180s readiness poll.
- team-readiness degrades FAST on a permanent error (no 180s cold-start wait);
composer maps raw "Load failed" / permanent errors to friendly copy.
- create-flow Finish is gated on ≥1 authorized agent identity (a team with none
can't run a turn).
- listTeamContainers throws on an unparseable wrangler payload instead of
silently returning [] (which read as a reassuring "No containers found").
Tests: cloud ensureServe permanent/transient classification; team-readiness
permanent degrade; describeCloudError mapping; Finish-gate. No pipeline/schema/
storage-shape changes (frontend + Worker only).
The in-app Create flow failed at Deploy ('Cloud setup ended without a result
(exit Some(1))'). Root cause: the generated config lived in an OS temp dir, but
wrangler resolves `main` relative to the CONFIG file — so `main = src/index.ts`
404'd ('entry-point file not found'). Fix: rewrite `main` to an absolute path.
Also replace the `d1 create --update-config` step (which silently skips the
binding when the DB already exists, leaving no DB binding) with create + resolve
the real id via `d1 list --json` + inject the [[d1_databases]] block — works for
fresh and already-set-up accounts. Verified: provision-team.ts now exits 0 and
emits the deployed result end-to-end.
Investigating a stuck team-cloud connection found the deployed Worker had NO
container (wrangler containers list = []), so requests had nothing to route to.
Redeploying surfaced the real cause: CF Containers reject :latest ('Latest tags
are not allowed'). Pin the reference to :0.1.0 (re-tagged from the published
:latest) in wrangler.toml; the provision engine default + publish CI now use the
pinned tag too (CI still pushes :latest as a human pointer). Redeployed the
worker + container from the public image — /v1/health is 200, container active.
cloud/scripts/team-containers.ts (list --json / delete <id> via wrangler) + two
thin Rust commands (list_team_containers / delete_team_container, reusing the
deploy command's cloud-script runner) + a 'Remote containers' row in the
Developer panel's Team cloud group (List -> shows them -> Delete each). Added to
LOCAL_ONLY (ipc.ts + companion test, in sync). 'stop' isn't a wrangler container
primitive (they idle-sleep); list + delete are the meaningful ops.
All green: cargo check/clippy, companion LOCAL_ONLY test, cloud + frontend tsc,
biome.
Add publish-team-image.yml — builds the headless serve image natively on amd64
and pushes it to docker.io/caspianzhao/helmor-team-sandbox:latest (public), so
any operator's Cloudflare account can reference it with no per-account docker
build. Uses the DOCKERHUB_TOKEN repo secret. Point the provision engine's
default HELMOR_TEAM_IMAGE at the same ref.
cloud/scripts/provision-team.ts is the deploy ENGINE behind the in-app 'Create
a team' flow: drives wrangler to stand up a fresh team backend on the operator's
OWN Cloudflare account — reuse 'wrangler login' (OAuth), generate a per-account
config from wrangler.toml (public image swap + fresh D1 via --update-config),
create R2 + apply schema + put a generated admin token secret, deploy, parse the
workers.dev URL. Emits step progress as JSON on stdout (the Rust 'deploy_team_
cloud' layer forwards it to the setup card); needs-upgrade is surfaced from the
deploy error.
Deliberately a standalone TS script (runnable + verifiable directly via bun) so
the CF-interaction logic isn't blind Rust. The 3 spots that need a real paid CF
account to confirm are flagged inline with TODO(verify). Image reference is
parameterised (HELMOR_TEAM_IMAGE) pending the public-registry publish.
The compiled sidecar has no node_modules, so it needs HELMOR_CLAUDE_CODE_BIN_PATH
to resolve the claude-code binary. Without this env var, Claude hangs silently.
Route git/gh/glab in the shared team sandbox to the acting member's own
credentials (trust-model attribution within the shared container).
- ForgeIdentity DO + PUT/GET /team/forge-identity broker, encrypted at
rest (reuses BROKER_ENC_KEY); desktop authorize_cloud_forge_identity
(LOCAL_ONLY) auto-syncs gh token + glab config on team connect.
- forge layer selects the acting member's GH_TOKEN/GITLAB_TOKEN via an
ambient member id propagated across the async->blocking boundary in
run_blocking; falls back to the repo-bound account off-team.
- Credential injection: local-docker dir-mount with mtime hot-reload, and
cloud cold-start writeFile of all members' creds outside the backup tree.
- Per-member private clone (gitCheckout URL rewrite) and post-turn
auto-commit authorship (<id>+<login>\@users.noreply.github.com).
- start-serve.sh: clear stale Xvfb socket on container restart; add glab
to PATH + wire git credential helpers.
- Dockerfile.local: Add BuildKit cache mounts for bun package cache and vendor
archives (gh/glab/Claude/Codex), plus streaming output for visibility.
- local-docker-team.ts: Fix container recreation on image changes, build by
default with cache awareness, auto-prune superseded images, seed fixed dev
member token for zero-entry Team mode toggle, use known Claude token from
registry for container injection, and support setup-only mode.
- proxy.ts: Recreate container after successful Claude (re)authorize so the new
token is injected as environment.
- registry.ts: Store raw Claude OAuth tokens locally (not exposed over HTTP) for
container injection; seed fixed invite tokens in local dev.
- companion/server.rs: Route UI mutations over shared /v1/stream SSE (with
keepalive) instead of one-shot /rpc-stream, preventing silent event loss and
proxy idle-close issues in team mode.
- ipc.ts & ipc.companion.test.ts: Wire subscribe_ui_mutations onto the shared
stream, add test coverage.
- team.tsx & team-mode.ts: Provide dev-build defaults (fixed URL + token) so
toggling Team mode requires no manual entry; add getDevTeamDefault().
- package.json: Add `dev:team` script to run local Docker team setup.
- scripts/dev-team.ts: New script to orchestrate dev:team workflow.
Cloud Claude agents now authenticate via the user's Claude subscription (never an API key), mirroring the Codex broker. A per-team ClaudeIdentity Durable Object holds the `claude setup-token` OAuth token (sk-ant-oat01..., a ~1-year inference-only token) encrypted at rest (AES-GCM via the shared BROKER_ENC_KEY); the Worker injects it as the CLAUDE_CODE_OAUTH_TOKEN env var on serve cold-start (no file write; the token is consumed verbatim, no auth_mode field to mis-case).
Desktop authorize_cloud_claude_identity captures the token by running `claude setup-token` in a PTY (it is a raw-mode Ink TTY app, so Stdio::piped fails), ANSI-stripping + loose sk-ant- matching the output (longest match, wide PTY so it never wraps), then PUTs it to /team/claude-identity. Held in Zeroizing; never logged, returned to the frontend, or persisted in plaintext. Timeout path kills the process tree before wait().
start-serve.sh unsets ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN (they outrank the OAuth token non-interactively) + seeds ~/.claude.json hasCompletedOnboarding; HOME pinned so the seed dir matches the SDK child. Settings -> Cloud Run Identity gains a Claude section (status + Authorize button). Verified doc-first against claude-code 2.1.173.
Also corrects the codex auth.json test assertion to camelCase chatgptAuthTokens (regression guard for the earlier casing fix).
Persist the container's Helmor data dir to R2 so chat/session history survives
a sandbox sleep (CF wipes the disk on sleep). Per finished agent turn the serve
process folds the SQLite WAL into helmor.db (synchronous wal_checkpoint(TRUNCATE),
env-gated on HELMOR_CLOUD_AUTOPUSH); after the turn stream closes the Worker
snapshots /home/helmor to R2 in localBucket mode (excluding the git worktrees,
which are restored via autopush) and stores the DirectoryBackup handle in D1
teams.backup_handle. On cold start ensureServe restores from that handle before
launching serve. Relocate the data dir to /home/helmor (an allowed backup root)
via HELMOR_DATA_DIR in both start-serve.sh and boot.sh so serve and boot share
one DB. Zero pipeline-snapshot drift; all team UI/proxy paths unchanged.
Verified end-to-end in the live container BEFORE wiring (temp /__diag probe):
vendored codex read a dropped ~/.codex/auth.json, `codex login status` =
'Logged in using ChatGPT', container egressed, real turn completed (gpt-5.5,
HELLO_FROM_CLOUD). Subscription path works headless — no API key.
- index.ts: pass CODEX_AUTH_JSON through startProcess env to the container.
- start-serve.sh: write it to $CODEX_HOME/auth.json (0600) before serve starts,
so Helmor's sidecar-spawned codex authenticates as the user's subscription.
Phase-1 will swap the whole-credential passthrough for a control-plane token
broker (mint per-turn access_token via auth.openai.com/oauth/token public
client; inject a ChatgptAuthTokens auth.json), keeping the rotating refresh
token in the durable control plane, never the ephemeral container.
First real CF cold start 503'd (companion didn't bind within the 60s poll).
- wrangler.toml: instance_type standard(½ vCPU) -> standard-3 (2 vCPU/8GiB);
WebKitGTK + Xvfb init needs real CPU. Also clears the standard->standard-1
rename warning.
- index.ts: health poll 60s -> 120s for a heavy first-boot init.
- start-serve.sh: skip if a serve host is already running (the Worker calls
startProcess on every not-ready request; a 2nd serve fought for the port).
Single-user cloud workspace: the desktop app can point its backend at a
remote Helmor companion running headless inside a Cloudflare Sandbox,
with zero UI changes (a transport-only mode switch).
- PR1 bootstrap::init_core(): extract the deterministic startup core
(data dir, logging, DB schema, pools) shared by GUI + serve.
- PR2 helmor serve: build_app(mode) shares one Tauri builder; a
windowless Wry host wires the functional trio + companion bound to
0.0.0.0 with an injected capability token. Adds a serve smoke test.
- PR3 vendor Linux: vendor-platform.ts gains a linux target; the serve
image stages only codex + gh.
- PR4 Dockerfile: 4-stage image FROM cloudflare/sandbox with webkit/gtk
+ xvfb; start-serve.sh runs an independent Xvfb daemon. ENTRYPOINT is
left to the CF base.
- PR5 cloud/: stateless thin Worker proxy (ensureServe -> startProcess
-> containerFetch / wsConnect), @cloudflare/sandbox 0.12.1.
- PR6 workflow closure: boot.sh reuses CLI verbs; cloud_autopush commits
+ pushes after each agent turn (env-gated; desktop is a hard no-op).
- PR7 desktop entry: team-mode.ts + ipc.ts runtime transport; a Settings
Team panel and a sidebar Local<->Team switch.
Desktop behavior is unchanged: every path is either a pure refactor or
env-gated. The full Rust and frontend test suites stay green.
Caspian Zhao