Lever A of the gray-cloud switch fix: get_workspace was the ONLY switch-time
input still proxied to the live container. For worktree-mode (container)
workspaces its result is 100% DB-derived, so mirror it and answer from the
control plane — switching never depends on the container being awake.
- D1: new workspace_details table (opaque payload, write-path self-heal
CREATE — the git_snapshots precedent, no migration step needed).
- PUT /team/sync: new workspaceDetails rows (exact WorkspaceDetail
serialization); GET /team/workspace-detail serves them member-readable,
D1-only. Local-dev registry mirrors the semantics in-memory.
- Container (team_sync.rs): push one workspace's detail on WorkspaceChanged /
SessionListChanged / WorkspaceChangeRequestChanged, all workspaces on
WorkspaceListChanged, plus a full push in the startup backfill. Detail-
mirrored events defer their realtime broadcast until the mirror lands
(same P1-6b contract as the session mirror).
- Desktop: loadWorkspaceDetail branches to the mirror in team mode (the
loadWorkspaceSessions pattern); a null mirror falls through to /rpc.
The Codex DO's PutResult.changed is account hygiene only (account_id
switched), but putCloudIdentity reused it as the restart trigger — so a
same-account re-authorization (the common brick-recovery path) never
restarted the warm container, which kept serving with the old
startProcess-env auth (401 token_invalidated) until natural sleep.
PutResult grows an independent `stored` signal ("a token was actually
written by this call") and the restart condition becomes
stored || changed. Claude path unchanged (its changed already covers
re-auth); restartSandboxForReauth still guards the not-serving case.
The class3+4 rollout autopsy proved the container's older squashfs-tools
drops the ENTIRE parent tree for a nested exclude pattern: the
'.codex/.tmp' entry in BACKUP_EXCLUDES was silently discarding the whole
.codex session-thread tree from BOTH backup paths (idle AND post-turn),
so Codex resume came back empty after every sleep — the R2-F4a failure
mode, on the Codex half this time. The R3-D autopsy had only asserted
.claude + archive size, never .codex, so this hid since R3-D.
Fix, per the architect-reviewed design (p13b-prune-design.md):
- BACKUP_EXCLUDES is top-level names only; a new test pins 'no / in any
pattern' (red-verified against the nested entry).
- The R3-D OOM guard moves container-side: CodexTmpPruneGuard prunes
$CODEX_HOME/.tmp with FINALLY semantics when the stream worker exits —
success, persist failure, abort, or panic (red-verified against a
no-op body; panic path pinned via catch_unwind). A local guard drops
before the closure's captured event Channel, so the prune also lands
before the EOF that triggers the Worker's post-stream backup.
- Accepted residual window (architect): R3-A's idle expiry ignores
in-flight requests, so a LONG turn can be snapshotted before its
prune — rare, non-silent (warnIfBackupOversized is the tripwire),
self-healing on the next turn. Documented at both sites.
Touches container Rust -> requires the team image bump to :0.1.12 at
rollout (bump commit follows).
Two sub-fixes, both Worker-only (no container Rust, no image bump):
Sub-problem 1 — forge (re-)authorize never re-injected: injectForgeMembers
ran ONLY at cold start, so PUT /team/forge-identity left a warm container's
members file stale until a manual destroy (live-reproduced in the class2
rollout as 'could not read Username' on a private clone). The in-container
member_creds loader already hot-reloads the file via mtime, so the fix is a
guarded live rewrite: putForgeIdentity now fires reinjectForgeMembersIfServing
(healthOk-gated, best-effort, no restart). Forge collection/injection moved to
cloud/src/forge-creds.ts so both injection sites share one implementation
without a team.ts <-> index.ts import cycle.
Sub-problem 2 — cloud-identity re-authorize used stop(): SIGTERM that
helmor serve ignores, so the VM stayed up caching the old startProcess-env
credential and nothing ever re-minted (the docstring's 'the NEXT request
cold-starts' was a lying comment; rewritten to the real mechanism). Replaced
with a guarded backup-then-DESTROY: a new Sandbox DO method
backupThenDestroyForReauth snapshots /home/helmor first (same budget/handle
semantics as the idle path — NEVER a bare destroy, which would roll back to
the last idle snapshot), then SIGKILLs so the next request truly cold-starts.
healthOk-gated: a scaled-to-zero container is skipped entirely (no serve to
kill, no real disk to snapshot). /admin/restart-sandbox switched to the same
path; /admin/destroy-sandbox intentionally unchanged (deliberate reset).
Supporting moves: containerFetchThroughPort/withTimeout/healthOk extracted to
cloud/src/container-net.ts so team.ts can probe liveness without importing
index.ts. Identity PUT side-effects are injectable store deps so the wiring
is pinned by tests (forge-reinject.test.ts, reauth-restart.test.ts —
red-verified against the neutered fixes).
A team idle past the backup TTL (3 days) has its archive deleted, but the D1
backup_handle lived forever — every wake then failed restore and R3-D's
loud-503 (correct for OOM) wedged the team permanently on data that was
already gone at expiry.
ensureServe now runs a DETERMINISTIC pre-restore check, backupGone(): it
reads the SDK's own R2 metadata (backups/<id>/meta.json) and applies the
exact createdAt+ttl+60s-buffer arithmetic restoreBackup itself uses. Proven
gone (meta deleted or past TTL) → clear the stale D1 handle + cold-start
empty like a brand-new team (info log, no degrade). Everything else is
untouched: restore FAILURES are never interpreted as expiry — OOM, timeout,
corruption, and R2 pre-check hiccups all still hit the loud typed 503, so a
dead DO can't hide behind a silent empty start (R3-D regression-guarded in
tests). Deliberately NOT classifying on error.name: the SDK's typed
BackupExpiredError doesn't reliably survive the DO RPC boundary.
Face: Worker JS only (wrangler deploy) — no container image impact.
The idle (pre-sleep) backup carried its own inline exclude list whose comment
claimed it was 'IDENTICAL to backupAndStore's' — it had silently drifted:
R3-D added .codex/.tmp (Codex's 76MB+ plugin-marketplace cache) to
BACKUP_EXCLUDES but not to the inline copy. Any Codex team that slept via the
idle path re-inflated its archive (~48MB) and re-opened the R3-D restore-OOM
wake wedge.
The idle path now builds its options via idleBackupOptions(), which
references the shared BACKUP_EXCLUDES constant (a test pins the reference by
identity, so a copy can't drift again), and gains the warnIfBackupOversized
tripwire the post-turn path already had. Comment rewritten to describe the
real mechanism; the idle-archive autopsy (.claude/.codex present,
.codex/.tmp absent) stays a rollout acceptance gate.
Face: Worker JS only (wrangler deploy) — no container image impact.
The team clone rides a minted token embedded in the https URL (the SDK
gitCheckout has no auth option), and git persisted that token verbatim
into the shared container's .git/config remote line — plaintext,
'cat'-able by any member. After the R3-E refs fetch (which still uses
the token'd origin), the Worker now rewrites origin back to the clean
URL; a scrub failure is FATAL (rm -rf + 502) so a token can never
survive on disk. Desktop-side remote_url redaction (R2-F7) is untouched.
Container Rust (SAME team-image bump as P1-2a): the two ls-remote call
sites that silently relied on the embedded-token URL — publish-status
remote_branch_exists and default-branch resolve_head_from_ls_remote —
now inject the same per-invocation argv auth (http.extraheader) as
push/fetch. Full sweep of git network verbs found no other
unauthenticated prod site (repos.rs clone_repository_from_url is
desktop-only; the Worker intercepts it in team mode).
DEPENDS ON P1-2a's creator-token fallback for push/fetch auth after the
scrub — roll both back together. Worker tests written red-first (scrub
ordering, fatal-scrub, public-clone no-op).
Cold start previously minted EVERY member's ForgeIdentity DO and wrote
the whole team's gh/glab tokens into the shared container — one induced
'cat' away from a full-team credential leak. Per the user product
ruling the creator-identity model stays (everyone's pushes reuse the
creator's forge identity); only the storage narrows:
- teams.forge_identity_member_id: the forge 'creator', bound
FIRST-authorizer-wins on PUT /team/forge-identity (mirrors
cloud_identity_member_id's single-row upsert, but COALESCE keeps an
existing binding; a non-creator racing to authorize first is the same
accepted TOFU edge as cloud identity). Added via the P1-4c idempotent
D1 ALTER runner (0002) + schema.sql for fresh DBs.
- collectMemberForgeCreds injects tokens ONLY for the creator entry
(flagged creator:true); every other member is login-only (kept for
per-member commit authorship). Local-docker launcher mirrors the same
narrowing.
- member_creds.rs (CONTAINER RUST — needs the next team-image bump):
token lookups fall back to the creator's tokens when the acting
member has none, and when no acting member is bound at all (the
detached auto-push thread). Old injected files without the creator
flag still parse (no fallback). Authorship lookups (member_for) do
NOT fall back.
NOTE: P1-8a (scrubbing the token out of .git/config) depends on this
creator fallback for push auth — rolling back this commit requires
rolling back the P1-8a commit with it.
The team clone endpoint accepted any cloneDirectory passing a bare
startsWith("/workspace") check and interpolated it into sandbox.exec
shell strings: "/workspaceevil" (prefix bypass), "/workspace/../.."
(traversal) and quote/metacharacter payloads (command injection) all
went through. sanitizeCloneRoot now whitelists /workspace(/<segment>)*
with [A-Za-z0-9._-] segments (no "."/"..") and rejects everything else
with a 400; all exec interpolations are additionally shq()-quoted.
Also corrects the stale "targetDir is sanitized" comment that this
change makes true. Litmus tests written red-first.
A fresh Cloudflare account never had BROKER_ENC_KEY (it was only ever set by
hand on the dev account), so the first agent authorize threw on
atob(undefined) inside the identity DOs and every fresh team 500'd its way to
an un-finishable create flow.
- provision-team.ts: after the companion-token put, probe `wrangler secret
list` and decide via pure logic in the new broker-key.ts. The four branches
(unit-tested): key present → skip; absent → put (32 random bytes, standard
base64, never logged); list failed with the Worker-not-found error (fresh
account, verbatim shape captured from wrangler 4.100.0) → put; ANY other
list failure → the whole provision fails. ⚠️ Writing only-when-absent is a
hard safety constraint: rotating an existing key bricks every stored
identity. Black-box check against the live dev account (key present since
the manual setup) decides "skip"; the live worker-not-found error decides
"put".
- codex/claude/forge identity DOs: importKey now names the real problem
("BROKER_ENC_KEY is not configured … re-run team setup") instead of an
opaque atob crash wrapped as a blind 500 — covers Workers provisioned
before this fix. Guard covered by the codex + claude twin suites; the forge
copy is verbatim-identical.
- wrangler.toml: BROKER_ENC_KEY added to the documented secrets list (it was
missing there too), with the do-not-rotate warning.
Repo clones and worktrees are derived state outside the backup boundary;
they evaporate across container generations while helmor.db survives. This
adds the rebuild path the DB rows were always missing:
- ensure_repo_source: re-clone repos.root_path from the clean remote_url
(tmp+rename atomic, full refspec fetch, per-repo lock + double-check).
- ensure_workspace_materialized: rebuild the worktree tracking origin/<branch>;
a never-pushed branch rebuilds from origin/<default> with an honest
workspace_rebuilt system notice persisted into the session.
- Hooks: workspace prepare/finalize (create path) and the agent-turn entry
(transparent to the user); PASSIVE reads never rebuild.
- get_workspace now returns the expected rootPath plus materialized:false
instead of nulling it when the directory is missing.
- Credentials ride only in the child process argv (http.extraheader from
member_creds); the rebuilt origin URL stays userinfo-free (regression-
tested), and push/fetch inject the same process-level auth.
- New typed errors RepoSourceUnavailable / WorkspaceRematerializeFailed.
- Container image bump :0.1.8.
Forensics (live, via curl replay against the container): the create
handlers never hang — prepare answers in <300ms and finalize fails FAST
with a named error. The actual break was one layer up and one down:
- handleTeamClone's SDK checkout materializes the working tree but no
remote-tracking refs, so every team build-workspace finalize died with
'Base branch is missing in source repo: origin/<default>'. Clone now
runs a full-refspec git fetch after checkout (typed 502 + dir cleanup
on failure). team+build now passes end-to-end for the first time:
clone -> prepare -> finalize returns setup_pending with a worktree
materialized under the writable /home/helmor tree (live-verified).
- plain /rpc proxy hops get a hard 60s deadline -> typed 504
ContainerRpcTimeout (WP2 error contract extended from the stream entry
(R3-B) to the plain-RPC face) via proxyPlainRpcWithDeadline.
- the start composer's create call gets a 75s client-side backstop
timeout, and EVERY create failure now restores the consumed draft via
composerRestore — a submit can no longer vanish silently.
The sleep backup ballooned 700x (70KB -> 48MB): F4a moved the provider
dirs under the backed-up tree and Codex's plugin-marketplace clone cache
(.codex/.tmp, 76MB unpacked) was not excluded. Restoring that archive
exceeded the Sandbox DO's isolate memory limit, the DO reset mid-wake,
and every subsequent wake looped through restore-OOM -> 503 while the
catalog probe kept readiness on 'Connected'.
Autopsy-confirmed root fix + three hardening layers:
- exclude .codex/.tmp from createBackup (archive back to KB-scale;
live-verified 48MB -> 8KB) and document the F4a provider-dir hazard.
- restore failures now throw typed ContainerRestoreError -> structured
503 ContainerRestoreFailed (no more silent empty-DB cold starts against
a reset DO). Missing handle still cold-starts empty as designed.
- readiness wake-health counter: 3 consecutive WAKE-request failures
degrade readiness and HOLD the probe's catalog-200 'ready' verdict
until a wake succeeds (user retry forgives). Fixes the 'Connected over
a dead container' blind spot.
- backup size budget alarm (50MB warn) as the excludes-regression
tripwire.
Service restored via runbook: bad handle cleared in D1, companion token
rotated, sandbox destroyed, fresh cold start verified (health 200,
container create/list round-trip OK). Data cost as ruled (F-5 boundary):
container DB starts fresh; D1 mirror history intact.
Every agent-stream close snapshotted /home/helmor unconditionally —
failed requests paid 48MB×3s backups for streams that did nothing.
backupAfterStream now scans the drained NDJSON: keepalive blanks and
{"kind":"error"} terminal events (the R3-B(3) error surface —
persisted:false, nothing written) don't qualify; any other event does.
A mid-stream abort still backs up conservatively. Tests lock the
architect-mandated interlock: a failed turn emits the error event AND
skips the backup.
Read path: ORDER BY datetime(created_at) normalizes legacy space-format
vs RFC 3339 rows, so mixed stock sorts chronologically before the
container migration propagates (id ASC keeps R2-D tie determinism).
Write path: the upsert now follows the container's created_at on update
(COALESCE guards null), so the container-side normalization self-heals
stale mirror rows on the next sync — no manual D1 backfill needed.
Live tail caught renew caller=<constructor blockConcurrencyWhile> after
passive requests: the containers base re-arms the full idle window every
time a DO instance is (re)created, so passive polling that wakes an
evicted DO kept re-arming a running container through eviction cycles.
Skip exactly that one constructor renew per instance — wake traffic renews
explicitly and onStart re-arms on real container boot. A fresh DO instance
over a running container now starts already-expired (sleeps early, the
safe polarity) unless wake traffic arms it.
The wake-intent marker is a custom header, so marked requests preflight;
without it in Access-Control-Allow-Headers the browser blocked every WAKE
POST from the desktop webview (live-verified: room chat rolled back with
'Can't reach the team cloud').
Live tail showed [idle] renew caller=decrementInflight on every passive
request release: the containers SDK base renews the idle timer whenever
inflight hits 0, re-arming the countdown from each request's completion —
a fourth renew site the gate missed. Passive releases now decrement the
counter manually (F-4 accounting identical) without touching the timer;
wake releases keep the base semantics. F-4 suite pinned to the wake path;
wake-intent suite now mimics the base renew-on-zero to lock the fix.
Gate 1 (route): only requests carrying X-Helmor-Wake-Intent may ensureServe.
Unmarked traffic is proxied as-is — an awake container answers normally, a
sleeping (or mid-cold-start) one gets a structured 503
{code: "ContainerAsleep"} instead of a wake. Path exemptions, both
intentional wake-capable: /v1/health (diagnostic endpoint provision/WP6
verify depend on; no frontend poller) and the model-catalog RPC on a D1
cache miss (WP5's one legitimate non-@agent wake).
Gate 2 (fetchCompanionPort): only marked requests renew the idle timer — at
request start and through the streaming progress tap. A watch stream's 30s
keepalives now feed the corpse watchdog only, so watching is FREE and the
container sleeps on schedule even with observers attached (supersedes
R2-A's cost-neutral compromise). Inflight accounting (F-4) unchanged.
New wake-intent.test.ts covers both gates, the asleep shape contract, and
the previously-untested progressTap renew throttling.
Archive autopsy during the F4a acceptance: the idle-sleep backup (with
the seven nested 'dir/sub' exclude globs added in F4a) contained NO
.claude tree at all, while the post-turn backup taken minutes later on
the same disk state — same dir, same five top-level excludes, no nested
patterns — carried the provider threads correctly. mksquashfs's exclude
handling of slash patterns swallowed the parent dot-dir, silently
defeating the provider-state relocation: every wake restored a
threadless snapshot and same-thread resumes kept failing with 'empty
response'.
Fix: the idle backup's excludes are now IDENTICAL to backupAndStore's
five top-level names (contract documented inline). Thread-bearing
archives measure ~75KB — nowhere near the 15s budget; if provider noise
ever grows, prune container-side rather than re-risking nested exclude
semantics. Worker-only change (the backup runs in the DO): no new image
needed.
Every idle-sleep destroyed Claude/Codex thread state: the serve process
HOME is pinned to /root (VERIFIED auth semantics), so claude-code wrote
threads to /root/.claude/projects and codex to /root/.codex — both
OUTSIDE the backup dir (/home/helmor, the only allowed backup root
covering them is /home). Result (R2-F4, and retroactively R2-F3 — same
root cause, formally overturned): after every wake, resuming any
pre-sleep thread found nothing, the provider emitted an empty end, and
the turn failed with 'empty response' forever (the #402 guard keeps the
resume id, so retries never recover).
Fix = relocation, not backup surgery:
- startProcess env: CLAUDE_CONFIG_DIR=/home/helmor/.claude and
CODEX_HOME=/home/helmor/.codex. Both are first-class CLI vars and the
existing seeding paths already honor them (codex auth.json write uses
$CODEX_HOME; the Claude onboarding seed writes
$CLAUDE_CONFIG_DIR/.claude.json). HOME stays /root — auth env
semantics untouched.
- start-serve.sh exports the same defaults so the boot path matches.
- backup excludes gain glob entries for the heavy non-thread noise the
relocated homes carry (telemetry/downloads/paste-cache/snapshots/
file-history, codex worktrees + logs) to guard the 15s budget.
Existing threads under /root on the live container are lost at rollout
— equivalent to the status quo (they were lost on every sleep anyway).
Ships as :0.1.6.
Polling disposition (team mode only; local untouched):
- forge action status / change request / workspace forge: intervals off —
refreshed on focus and on the container's turn-driven pushes; external-
only transitions (CI finishing elsewhere) wait for the next attention
(accepted trade-off per plan).
- rate limits: interval off; focus/popover-driven.
- local-llm family joins LOCAL_ONLY_INVOKES: the local LLM runtime is
THIS Mac's endpoint — its panel was polling the cloud container every
2s by pure routing mistake. Full command set aligned against api.ts.
Presence (ruling, correction C): typing presence publishes DIRECTLY to
the TeamHub DO (POST /team/event) instead of /rpc/report_presence —
typing must not wake the container. The Worker now accepts member
tokens on that route under hard constraints, locked by tests: only the
whitelisted event kind (roomPresenceChanged); memberId + ts are STAMPED
from the token identity (a forged body id is overwritten); any other
ui-mutation kind → 403; unknown token → 401; companion relay unchanged.
Warm-up: composer FOCUS no longer wakes the sandbox — the pre-warm fires
on the first TYPED INPUT (send intent), riding the HasContentPlugin
onEditing signal, same 60s throttle.
The git-status + changes polls (10s each) were the biggest reason an
attended team session pinned the container awake. The container's git
watcher already publishes WorkspaceGitStateChanged/WorkspaceFilesChanged
(and the desktop bridge already invalidates on them via the Stage C
relay), so the only missing piece was the data path:
- team_sync: on those events, compute a git snapshot (the EXACT structs
the /rpc commands return — zero shape invention) and PUT it on the
existing /team/sync write-through, throttled to one per 2s per
workspace. Chat / non-operational workspaces skip (no worktree).
- cloud: git_snapshots D1 table (write-path self-heal CREATE, the
model_catalog precedent), stored opaque + versioned inside the blob;
GET /team/git-status (member-or-admin) reads it. Local-dev proxy
registry mirrors the store. Contract test covers write/read/miss/401.
- frontend: loadWorkspaceGitActionStatus / listWorkspaceChanges get an
isTeamModeActive branch reading the D1 mirror (quiet zero-status
before the first push); both queries stop polling in team mode
(refetchInterval false) — refresh rides the existing
workspaceGitStateChanged invalidation, which now refetches from D1.
Local mode is untouched. Container Rust ships in :0.1.5.
R1 (blank first paint after Local<->Team switch): the remote transport
was explicitly downgraded to a bare QueryClientProvider, so every
switch rebuilt the cache from zero. The asymmetry existed because the
persister's cache commands would have been routed to the Worker.
- Team transport now persists into a per-backend bucket:
helmor-query-cache--team-<sha256(url \n token)>. Token is part of the
key on purpose (security over convenience): a different member on the
same URL gets a different bucket — cached workspace lists never leak
across identities; a token rotation abandons the old bucket, which
the pruning below reclaims.
- read/write/delete_query_cache join LOCAL_ONLY_INVOKES: the cache is
desktop disk state even while the transport points at the Worker
(routing decision — the companion dispatch does implement them, see
the note in companion_dispatch_coverage.rs).
- Aggressive bucket retention (by ruling): only the current team bucket
+ the local bucket survive; every other indexed bucket is DELETED at
team-persister init (leaving a team must not keep its list cache on
disk — privacy). Locked by an adjudication-anchor test asserting the
old-token bucket is actually deleted when the new token initializes.
- clearTeamConfig deletes the leaving team's bucket immediately.
- Persist whitelist: workspaceSessions joins PERSIST_META (list-shaped,
instant tab paint); message threads stay memory/DB-only (WP3 owns
that truth). Pure-browser companion (no Tauri runtime) keeps the bare
provider.
Rider fixes per plan:
- R2: cloud team.ts session listing now ORDER BY created_at ASC, id ASC
(matches local models/sessions.rs) — new sessions append at the end.
- R4 Gate2: the standalone 'Add repository' button is gone; the
'Connect a repository' fragment is the CTA (underlined, clickable),
opening the same add-repo dropdown as the sidebar via the existing
open-add-repository shell event. en/zh-CN strings split accordingly.
Root cause (live-verified): the SDK re-arms the idle timer on every alarm tick
while inflightRequests > 0, and a SIGKILLed desktop leaves companion pipes that
never settle — inflight pinned (observed 4 for 7h+), container never slept.
Three layers:
- fetchCompanionPort pipes bodies through IdentityTransformStream with a
finally-guaranteed decrement (SDK containerFetch pattern) + a 120s
no-progress watchdog that force-settles dead pipes.
- isActivityExpired (per-instance override) keys ONLY on sleepAfterMs; live
streams renew it by actually moving bytes (throttled 60s renewal in the
progress tap). Dead pipes stop renewing and the 5m timer runs out.
- [idle] renewal diagnostics + a backup-before-sleep log line (F-5 boundary:
idle path backs up to R2 before destroy; /admin/destroy-sandbox does not,
by design).
Acceptance: app killed 08:24:04 → container active:0 at 08:28:09 (4m05s,
within SANDBOX_IDLE_TIMEOUT + buffer); idle backup handle present in D1.
+4 unit tests pin the inflight accounting on every body-termination path.
The composer's model list (list_agent_model_sections) previously proxied
through ensureServe, so opening the app against a sleeping container burned a
cold start (up to 180s) just to render the model picker — and the WP1
readiness probe (/v1/health through the same proxy) woke the container too.
Cache the catalog (metadata only, never credentials) in a new D1
`model_catalog` row and answer the RPC BEFORE ensureServe:
- Worker intercept: authed (member/admin, mirrors /admin/*) cache hit returns
the verbatim payload with `x-helmor-model-catalog: cached` and ZERO
container interaction; a miss falls through to the normal ensureServe+proxy
path (the one legitimate non-@agent wake: brand-new/reset teams —
provision's WP6 verify normally seeds the cache at setup).
- Invalidation, no TTL: write-through on every LIVE pass of the RPC, plus
ensureServe now reports `coldStarted` and every wake (including
/admin/warm-up) refreshes the cache in the background — an image update is
picked up on the first real wake after it.
- Self-healing schema: the write path batches the CREATE TABLE, so a pre-WP5
D1 that never re-ran schema.sql works without a manual migration; a missing
table reads as a plain cache miss.
- local-dev proxy mirrors the intercept with an in-memory registry cache so
the no-wake behavior is e2e-verifiable on local-docker.
No container/Rust changes — no image publish needed for prod rollout.
local-docker never injected HELMOR_SYNC_URL, so the in-container TeamSync stayed
inert — team-mode history (GET /team/messages) never populated locally and the
Stage B write-through path (which WP3 changes) couldn't be exercised. Inject
HELMOR_SYNC_URL=http://host.docker.internal:<proxyPort> (mirroring what the real
Worker's ensureServe does), bind the proxy on 0.0.0.0 so the container can reach
it, and classify the shared companion token as admin on the local gateway (the
real Worker treats companion==admin; the split local tokens otherwise 401 the
PUT /team/sync). Enables full local-docker e2e of the room-chat → D1 mirror.
A re-provision rotates HELMOR_COMPANION_TOKEN, but a container still WARM from a
prior run caches the OLD token (injected at its startProcess) and 401s every
proxied rpc — so Reset→re-provision fails verify's model check and recovery
needed a manual `wrangler containers delete` + redeploy. `/admin/restart-sandbox`
can't fix it: it uses stop() (SIGTERM), which `helmor serve` ignores (the VM
stays up with the old token — see onActivityExpired). destroy() (SIGKILL) scales
it to zero.
- index.ts: add `/admin/destroy-sandbox` (exported `handleAdminDestroySandbox`)
— ADMIN-token only (companion token AND no derived member id), calls
sandbox.destroy(). Handled before ensureServe so it never cold-starts itself.
- provision-team.ts: after deploy, before verify, best-effort POST the endpoint
to destroy any warm container. A non-2xx / network error is logged and
swallowed — verify stays the backstop that catches a token mismatch.
Tests: 3 cases for the route (admin bearer → destroy + 200; missing/wrong token
→ 401 + no destroy; member bearer with X-Helmor-Member-Id → 401 + no destroy).
cloud/ only — no frontend, no Rust.
Setup went "all green" on a backend that could not run a turn (S3), and a
permanent container-start failure surfaced as a ~20s hang + raw "Load failed"
copy, or a 180s "stuck connecting" (S1/S2). Root-cause work also found the
in-app provision path shipped a STALE container image, diverging the container
from the Worker it runs under.
- provision: stop hard-coding the container image tag. wrangler.toml is now the
single source of truth (kept in lockstep with publish-team-image.yml); only
HELMOR_TEAM_IMAGE overrides it. Removes the Worker↔container version skew.
- provision VERIFY is now REAL end-to-end: Worker auth → container actually
starts (/v1/health through ensureServe) → model catalog returns. A failing
stage exits non-zero with a stage-named error; the create-flow surfaces it.
- Worker ensureServe classifies the Sandbox SDK's PERMANENT start error (a 500
with context.phase "startup") and FAST-FAILS with a structured `permanent`
503 (+ phase) instead of burning the ~15s restore/mint + 180s readiness poll.
- team-readiness degrades FAST on a permanent error (no 180s cold-start wait);
composer maps raw "Load failed" / permanent errors to friendly copy.
- create-flow Finish is gated on ≥1 authorized agent identity (a team with none
can't run a turn).
- listTeamContainers throws on an unparseable wrangler payload instead of
silently returning [] (which read as a reassuring "No containers found").
Tests: cloud ensureServe permanent/transient classification; team-readiness
permanent degrade; describeCloudError mapping; Finish-gate. No pipeline/schema/
storage-shape changes (frontend + Worker only).
onActivityExpired → destroy() (SIGKILL) so the idle VM actually stops (helmor serve ignores SIGTERM); onStart arms the idle timer on rollout. backupBeforeSleep bounded by a 15s budget so a hung backup cannot block the stop. ensureServe runs restore + identity-mint concurrently and fires injectForgeMembers off the critical path — trims steady-state cold-start ~7.5s → ~6s.
With Stage C the container sleeps when idle; pre-warm it on composer focus so a
following @agent run is hot.
- cloud: POST /admin/warm-up (member/admin auth) fire-and-forgets ensureServe via
ctx.waitUntil + returns 202 immediately (never blocks the caller).
- team-api: warmUpSandbox(cfg) — best-effort POST, swallows errors.
- composer: onFocusCapture pre-warms in team mode only, throttled to once/60s
(no focus-storm spam). Inert outside team mode.
Verified: cloud tsc, frontend typecheck, biome, 53 composer tests.
Adds the hibernating WebSocket relay so the realtime event stream can live OFF
the compute container — the sandbox sleeps while members stay connected
(WebSocket Hibernation accrues no billable Duration while idle).
- cloud/src/team-hub.ts: TeamHub Durable Object (raw Hibernation API — chosen
over partyserver to fit the existing custom multi-DO Worker; rationale in
.agent-contexts/team-cloud-rearch/stage-c-design.md). Pure relay: no secrets,
no D1. `/broadcast` (binding-only reach) + WS accept with member-id
attachment + ping/pong auto-response (keepalive without waking the DO).
- cloud/src/index.ts: export TeamHub; Env.TEAM_HUB; GET /v1/ws (bearer rides the
WS subprotocol, validated via classifyBearer, forwarded to the hub) + POST
/team/event (companion-token only -> broadcast). Both run BEFORE the team
registry + ensureServe, so they never wake the container.
- cloud/wrangler.toml: TEAM_HUB binding + migration v5 (new_sqlite_classes).
Additive — the container-SSE path is untouched until the desktop cutover (3/3).
cloud tsc + 47 tests green.
Next: container pushes ui-mutations to /team/event (2/3); desktop WS + unpin (3/3).
The container now sends an authoritative session-id set per workspace on
every SessionListChanged (PUT /team/sync `replaceWorkspaceSessions`); the
Worker deletes any D1 session (+ its messages) for that workspace not in
the set. An empty set prunes the workspace's last deleted session. This
catches deletes across a sandbox restart — which a delta on the existing
`deletedSessionIds` can't, since the container keeps no persisted known-set.
- cloud: replaceWorkspaceSessions on TeamSyncInput + prune in syncTeamData
(one D1 batch) + the in-memory local-dev registry; +1 contract test.
- container: SyncPayload.replace_workspace_sessions; sync_workspace_sessions
always POSTs the full id set (even empty); +1 serialization test.
Verified: cloud 47 tests + tsc; cargo clippy -D warnings + team_sync units.
Next: wrangler deploy (Worker B.1+B.2a+B.4) + rebuild container image
(B.2b+B.4 binary) -> e2e browse-asleep.
The container write-through (next: the Rust side) needs the Worker's public
origin to POST session/message mirror updates to PUT {origin}/team/sync. Thread
the request origin into ensureServe(options.syncUrl) and inject it as the
HELMOR_SYNC_URL env on the serve process. Absent on local-dev (the proxy mirrors
directly), so the write-through stays inert there.
Data-plane foundation (additive; dead until the desktop is wired in B.3):
- schema.sql: new `sessions` + `messages` mirror tables (browse-essential session
columns; append-only message rows w/ verbatim content JSON). Mirrors the
sandbox's SQLite so history is readable with the container asleep.
- team-gateway/core.ts: TeamSyncInput types + 3 routes — PUT /team/sync (container
write-through, companion-gated), GET /team/sessions, GET /team/messages
(member-or-admin reads).
- team.ts: D1-backed store impls — idempotent id-keyed upserts, append-only
message inserts (ON CONFLICT DO NOTHING), cascade deletes, batched.
- local-dev registry + proxy: matching in-memory mirror so `bun run dev:team`
serves /team/sessions|messages the same way.
- contract test: round-trip + auth (member can't write the mirror; unauth can't read).
Worker typecheck + 46 cloud tests green. Schema is applied to the live D1 and the
Worker redeployed in B.3, when the desktop starts reading these routes.
Three-plane re-architecture, Stage A (UX + pragmatic cost; no new infra):
- Connecting overlay see-through: bg-background/40 backdrop-blur-md → /20 backdrop-blur-sm.
- use-companion-idle-suspend: also drop the team /v1/stream when the window is
VISIBLE but idle (10m, no turn in flight), not just when hidden — the real lever
against an open-but-unused Helmor pinning the sandbox awake all day. Any
activity / re-focus resumes; never suspends mid-turn. + unit test.
- wrangler.toml: SANDBOX_IDLE_TIMEOUT 15m → 5m so the container stops promptly
once the SSE drops.
- cloud/index.ts: mint Codex + Claude tokens CONCURRENTLY (independent identity
DOs) — shaves up to ~15s off every cold start.
Interim (removed in Stage C): resuming after an idle suspend cold-starts the
sandbox. The "instant Local↔Team switch without the overlay" lands with Stage D's
connection-state split (data-ready vs compute-warming).
Route git/gh/glab in the shared team sandbox to the acting member's own
credentials (trust-model attribution within the shared container).
- ForgeIdentity DO + PUT/GET /team/forge-identity broker, encrypted at
rest (reuses BROKER_ENC_KEY); desktop authorize_cloud_forge_identity
(LOCAL_ONLY) auto-syncs gh token + glab config on team connect.
- forge layer selects the acting member's GH_TOKEN/GITLAB_TOKEN via an
ambient member id propagated across the async->blocking boundary in
run_blocking; falls back to the repo-bound account off-team.
- Credential injection: local-docker dir-mount with mtime hot-reload, and
cloud cold-start writeFile of all members' creds outside the backup tree.
- Per-member private clone (gitCheckout URL rewrite) and post-turn
auto-commit authorship (<id>+<login>\@users.noreply.github.com).
- start-serve.sh: clear stale Xvfb socket on container restart; add glab
to PATH + wire git credential helpers.
- Dockerfile.local: Add BuildKit cache mounts for bun package cache and vendor
archives (gh/glab/Claude/Codex), plus streaming output for visibility.
- local-docker-team.ts: Fix container recreation on image changes, build by
default with cache awareness, auto-prune superseded images, seed fixed dev
member token for zero-entry Team mode toggle, use known Claude token from
registry for container injection, and support setup-only mode.
- proxy.ts: Recreate container after successful Claude (re)authorize so the new
token is injected as environment.
- registry.ts: Store raw Claude OAuth tokens locally (not exposed over HTTP) for
container injection; seed fixed invite tokens in local dev.
- companion/server.rs: Route UI mutations over shared /v1/stream SSE (with
keepalive) instead of one-shot /rpc-stream, preventing silent event loss and
proxy idle-close issues in team mode.
- ipc.ts & ipc.companion.test.ts: Wire subscribe_ui_mutations onto the shared
stream, add test coverage.
- team.tsx & team-mode.ts: Provide dev-build defaults (fixed URL + token) so
toggling Team mode requires no manual entry; add getDevTeamDefault().
- package.json: Add `dev:team` script to run local Docker team setup.
- scripts/dev-team.ts: New script to orchestrate dev:team workflow.
Preview A — make the single-team product usable from the app: TeamAdminPanel (Settings -> Team) to create a team + mint/copy an invite link (admin-token-gated, mirrors the cloud-identity panel); populate the D1 workspaces mirror on team clone so the sidebar shared-workspaces list fills and rows open the workspace; composer cloud-error recovery CTA (auth/billing failures surface a Re-authorize / View Team settings action routing to Settings -> Team). All team UI gated on isTeamModeActive() so single-user is byte-identical.
Merge hygiene — narrow the cloud-sandbox image CI workflow to workflow_dispatch-only (drop the feature-branch push trigger before merging to main); add a changeset for the Codex + Claude subscription brokers.
IS_SANDBOX=1: claude-code refuses --dangerously-skip-permissions as root (binary guard getuid()===0 && IS_SANDBOX!=="1" && !CLAUDE_CODE_BUBBLEWRAP). The cloud Agent SDK runs claude-code as root with bypassPermissions, so every claude cloud turn exited with code 1. Inject IS_SANDBOX=1 into the serve startProcess env — the CF container is an isolated, ephemeral sandbox. Live-verified: a cloud Opus turn now succeeds.
auto-restart: a changed cloud identity (PUT /team/claude-identity or /team/codex-identity) now stops the sandbox so the next request cold-starts and injects the new credential. start-serve.sh is idempotent (bails if a serve is already running), so a serve started BEFORE authorization never picked up the new token — previously this required a manual restart.
Cloud Claude agents now authenticate via the user's Claude subscription (never an API key), mirroring the Codex broker. A per-team ClaudeIdentity Durable Object holds the `claude setup-token` OAuth token (sk-ant-oat01..., a ~1-year inference-only token) encrypted at rest (AES-GCM via the shared BROKER_ENC_KEY); the Worker injects it as the CLAUDE_CODE_OAUTH_TOKEN env var on serve cold-start (no file write; the token is consumed verbatim, no auth_mode field to mis-case).
Desktop authorize_cloud_claude_identity captures the token by running `claude setup-token` in a PTY (it is a raw-mode Ink TTY app, so Stdio::piped fails), ANSI-stripping + loose sk-ant- matching the output (longest match, wide PTY so it never wraps), then PUTs it to /team/claude-identity. Held in Zeroizing; never logged, returned to the frontend, or persisted in plaintext. Timeout path kills the process tree before wait().
start-serve.sh unsets ANTHROPIC_API_KEY/ANTHROPIC_AUTH_TOKEN (they outrank the OAuth token non-interactively) + seeds ~/.claude.json hasCompletedOnboarding; HOME pinned so the seed dir matches the SDK child. Settings -> Cloud Run Identity gains a Claude section (status + Authorize button). Verified doc-first against claude-code 2.1.173.
Also corrects the codex auth.json test assertion to camelCase chatgptAuthTokens (regression guard for the earlier casing fix).
codex 0.139's AuthMode enum renames this variant to camelCase "chatgptAuthTokens". The broker wrote lowercase "chatgptauthtokens", so the whole auth.json failed serde parsing -> auth=None -> codex fell back to api.openai.com with no bearer (401 "Missing bearer"). Corrected to camelCase. Verified live: cloud Codex turns now authenticate via the subscription broker and respond (incl. web-search tools).
Also add /admin/restart-sandbox (member/admin-gated, calls sandbox.stop()) so the next request cold-starts and re-mints the auth.json -- applies a re-authorization or broker change without waiting for an idle sleep.
- Instant team/local switch with no full-page reload; in-app "Connecting" banner while a cold backend wakes (IPC transport hot-swap + generation guard).
- Persistent team-mode add-repo (BUG-3): the Worker clones via the Sandbox SDK gitCheckout into the persistent /workspace, then registers via the existing add_repository_from_local_path RPC.
- Team-mode add-repo UI (BUG-2): hide local-only Open project/Browse; default the clone path to /workspace.
- Route authorize_cloud_codex_identity locally even in team mode so the broker login runs on the desktop (BUG-4).
- Stage claude-code on linux for the cloud image (BUG-1).
- Worker CORS for the desktop webview; exclude cloud/ from the frontend vitest run; biome-ignore .agent-contexts.
Persist the container's Helmor data dir to R2 so chat/session history survives
a sandbox sleep (CF wipes the disk on sleep). Per finished agent turn the serve
process folds the SQLite WAL into helmor.db (synchronous wal_checkpoint(TRUNCATE),
env-gated on HELMOR_CLOUD_AUTOPUSH); after the turn stream closes the Worker
snapshots /home/helmor to R2 in localBucket mode (excluding the git worktrees,
which are restored via autopush) and stores the DirectoryBackup handle in D1
teams.backup_handle. On cold start ensureServe restores from that handle before
launching serve. Relocate the data dir to /home/helmor (an allowed backup root)
via HELMOR_DATA_DIR in both start-serve.sh and boot.sh so serve and boot share
one DB. Zero pipeline-snapshot drift; all team UI/proxy paths unchanged.
- BLOCKER: an unauthenticated request (no Authorization) no longer gets the
shared companion token injected — the Worker only swaps in the shared token
for an authenticated caller (shared token or valid invite), so the companion
still gates /rpc (401) while /v1/health stays public.
- Strip any client-supplied X-Helmor-Member-Id at the very top of fetch()
(before proxyToSandbox) so the never-client-asserted invariant holds on every
path, including the SDK preview path — not just the derived proxy hop.
- acceptInvite: fail-closed on an unparseable expires_at, and reject re-accept
of an already-claimed invite with a different id (409) to block seat takeover.
Re-verified on real CF: no-auth->401, health->200, garbage->401, shared->200,
accepted-invite->200, re-accept-different-id->409.
Caspian Zhao