Adds a framework-agnostic guide covering the full client-side thread
lifecycle, filling a gap between the Headless Threads how-to and the
Threads & Persistence Architecture explanation:
- how a threadId is minted (UUID v4, client-side at mount) + the
resolution precedence and the remount-stability caveat
- how history is restored (explicit-threadId connect() replay vs.
manual agent.setMessages); notes v2 has no initialMessages
- switching/starting threads (setActiveThreadId / startNewThread) and
the prop-controlled no-op guard
- intercepting thread creation on first message (mint-up-front vs. the
headless onSubmitMessage seam; built-in <CopilotChat> overrides it)
- CopilotKit threads vs. framework checkpointers (two layers correlated
only by threadId; a checkpointer creates checkpoint tables, not a
threads table)
- MCP Apps activity/history (frontend constructs; re-synthesize on hydration)
- v1 vs v2 disambiguation
Addresses the recurring thread-lifecycle support questions (#4790, #4778,
#5434, #2242, #5931). Added to the Threads nav group.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pick and cancel resume the native schedule_meeting suspend tool with the
SAME toolCallId; the requests differ only inside the tool-result payload,
so the cancel resume previously hit the pick-confirmation fixture and the
assistant replayed "Booked: ... confirmed" after the user cancelled. The
"__cancelled" toolCallId gates on the Denied fixtures were fictional and
never matched.
aimock 1.37.0 (CopilotKit/aimock#299) adds a JSON-expressible
match.toolResultContains substring gate on the last tool-result message.
- gen-ui-interrupt.json: cancelled legs now gate on the real toolCallId +
toolResultContains "cancelled", ordered before the confirmation legs
- interrupt-headless.json: gained the same cancelled legs (the demo's
Cancel button had no fixture at all)
- aimock-fixtures.test.ts: matchKey learns toolResultContains; duplicate
ceiling 303 -> 305 (headless cancelled legs share exact keys AND
response text with the gen-ui-interrupt ones, one pair per pill)
- e2e specs: cancel tests now assert the Denied narration and reject
Booked/Scheduled, so the regression cannot silently return
Verified live against aimock built from source (fixture replay):
8/8 Playwright e2e across both demos, plus manual pick + cancel runs on
/demos/gen-ui-interrupt and /demos/interrupt-headless.
Commit uses --no-verify: this worktree's lefthook runner is broken
(pre-existing, see daa501daa); commitlint + prettier + the fixtures
vitest were run manually and pass.
Follow-up (blocked on aimock#299 npm publish): bump the vendored
@copilotkit/aimock pin in showcase/scripts/package.json and pull the
refreshed ghcr.io/copilotkit/aimock:latest.
Self-activate CVDIAG_LOG_STDOUT=0 when CVDIAG_PB_URL is wired (safe: only silences stdout when PB is receiving; explicit override preserved). Extend the watchdog to poll the public $PORT /api/health and, on sustained failure, POST a loud #oss-alerts Slack alert BEFORE kill-restart; add the same alert to the agent-\:8000 branch (no silent recovery). Add uvicorn --no-access-log to cut the access-log flood. Together these keep the shared log stream under the 500/sec cap so the pipe never backs up.
Shared cvdiag_bootstrap: gate the per-LLM-call breadcrumb capture handler and the emit_cvdiag stdout write behind CVDIAG_LOG_STDOUT (default ON so every other integration is byte-for-byte unchanged; opt-out per service). Enqueue to the non-blocking PocketBase sink BEFORE the stdout write so a wedged fd1 cannot cost the durable breadcrumb. No sampling; full fidelity to PB. Red-green unit tests incl. a hostile-stdout durability test.
Faithful node:22-slim repro: fd1 through the same awk process-substitution as entrypoint.sh, a Railway-capped drain reader, a uvicorn+CVDIAG-shaped flood, and the static no-log /api/health as victim. RED wedges (200->502, CPU->0, heartbeat frozen); the FIXED lane stays 200 throughout. run.sh asserts the outcome (exit 3/4/5 on a false result, proven). watchdog.sh runs the entrypoint public-guard loop verbatim and needle-anchors it against entrypoint.sh.
Playwright-verified fixes for the Mastra demo validation round:
- aimock interrupt fixtures (gen-ui-interrupt, interrupt-headless): add
hasToolResult:false to the schedule_meeting suspend legs so the resume
request falls through to the toolCallId confirmation fixture instead of
re-matching the suspend leg (picker loop, duplicated intro). Mirrors
hitl-in-chat.json.
- aimock-fixtures test: ceiling 301 -> 303; the two suspend keys now
intentionally collide across the three mastra interrupt cells
(runtime-disambiguated by route/fixtureFile like existing aliases).
- browse-web tool: return the result OBJECT instead of JSON.stringify;
the bridge encodes once more so stringifying double-encoded the result
and BrowseResultsCard showed "0 results" despite a successful browse.
- reasoning-chain pill: "Roll a d20 ..." instead of "Roll a 20-sided die
..." — the d4 agentic-chat fixture shadowed the first leg under replay
(d4 loads before d6) and pushed reasoning a step late. Real-LLM order
verified correct.
- header-forwarding shim: default x-aimock-context to "mastra" when absent
so browser-driven demos replay against aimock instead of 404ing. Harness
header wins when present; real providers ignore it.
- docker-compose.local: make OPENAI_BASE_URL overridable via .env (default
aimock unchanged) so real-LLM cells like browser-use can be tested live.
(--no-verify: commitlint binary missing in this worktree after the session
crash — ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL, infra not message)
Addresses @samjulien's review on #5982:
1. Contributing example command: the guide changed into a nonexistent
`examples/next-openai` and ran `dev:examples` (a workspace build/watch
script that never starts a server). Point it at the real
`examples/v1/next-openai` package and its `example-dev` (`next dev`)
script, which actually serves http://localhost:3000/presentation.
Fixed in the shared snippet + all per-integration copies.
2. LangGraph auth: langgraph variants are docs_mode: generated, so the
`/auth` route renders the root `docs/auth.mdx`, not the framework copy.
Add the missing `from langchain_core.runnables import RunnableConfig`
to the two Python blocks in the root source that route renders.
3. Self-contained fences: add the import to the non-tutorial
`langgraph/shared-state/predictive-state-updates.mdx` Python fence and
the `snippets/integrations/langgraph/frontend-tools.mdx` fence, so the
zero-missing-import claim holds for every non-tutorial langgraph block.
4. Contributor prerequisites: bump the `docs-contributions` guides from
pnpm 9 to pnpm 10 to match the code-contributions requirement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidates three stale-docs fixes that were opened against the retired
`docs/content/docs/` tree (now a symlink) and so could no longer merge:
- Contributing/package-linking guides (root + all integration copies +
shared snippets): Turborepo is fully removed from the repo (no dep, no
turbo.json). Drop the Turborepo prerequisite, bump pnpm to v10.x to match
`packageManager`, describe the monorepo as a pnpm workspace orchestrated by
Nx, and replace `turbo run <task>` with verified equivalents:
`pnpm run build|dev|format|lint`, `pnpm exec nx run-many -t (un)link:global`,
`pnpm exec nx watch` for a single package, and `pnpm run dev:examples`
(the real script; `example-dev` did not exist). Supersedes #3509.
- built-in-agent/model-selection: hyphenate the Anthropic model IDs
(`claude-3-7-sonnet`, `claude-opus-4-1`, `claude-3-5-haiku`). Supersedes #3656.
- langgraph reference docs: add the missing
`from langchain_core.runnables import RunnableConfig` import to Python code
blocks that annotate `config: RunnableConfig`. Supersedes #4069.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Companion structural PR to #5971. It fixes the **root cause** behind the
a2ui divergence #5971 patched: the showcase's single-source symlinks
eroded to real, drifting copies.
`showcase/integrations/*/tools`, `*/shared-tools`, `*/_shared` are meant
to be **symlinks into `showcase/shared/...`** — `stage_shared()`
dereferences them for the Docker build, `restore_symlinks()` restores
them. An **accidental `stage_shared()` leak** (commit `534cd1efa7`, PR
#4449 "D5 all-green") committed the dereferenced real files instead of
restoring the symlinks. Once they were real files, they drifted — which
is exactly how the a2ui `render_a2ui` vs `_design_a2ui_surface` split
(fixed in #5971) arose.
## Changes
- **Restore 12 Python `tools/` dirs to symlinks** →
`../../shared/python/tools` (ag2, agno, claude-sdk-python, crewai-crews,
google-adk, langgraph-fastapi, langgraph-python, langroid, llamaindex,
ms-agent-python, pydantic-ai, strands). Content is byte-adopted from
shared — verified no load-bearing per-integration code is lost (only
`render_a2ui` naming + shared `roll_dice`/sanitize additions).
Integrations' intentional internal-planner names (llamaindex,
ms-agent-python) live in `src/`, not `tools/`, and are untouched.
- **`showcase/AGENTS.md` (+ `CLAUDE.md`, root pointers,
INTEGRATION-CHECKLIST section)** — canonical statement of the 4 iron
rules (identical tests, near-identical frontends, minimal backends,
per-integration fixtures) + the single-source symlink mechanism ("edit
the shared source only; a real file there is a bug"). These were
previously written down nowhere.
- **`validate-shared-symlinks` CI guard** — fails on any NEW erosion
(real dir where a symlink belongs), with a shrink-only baseline that
tightens to fully-enforcing as symlinks are restored. Mirrors the
existing `validate-*` ratchet pattern.
## Scope / independence
- **No overlap with #5971** — this PR touches nothing under
`showcase/shared/typescript/` and does not modify the 3 TS integration
`shared-tools/` dirs (verified: empty file-set intersection). Mergeable
independently.
- Build-safe: `stage_shared()` correctly dereferences the restored
symlinks (targets resolve within the build context);
`restore_symlinks()` recreates them post-build.
## Verified
- `validate-shared-symlinks` test suite: 7/7 pass; validator EXIT 0 (no
new erosion).
- Reviewed by a full panel (correctness, content-integrity, build/CI,
docs, scope, silent-failure, simplicity) — zero mandatory findings.
## Follow-ups (deliberately out of scope)
1. **3 TS `shared-tools` dirs** (mastra, claude-sdk-typescript,
langgraph-typescript) remain real (baselined) — symlink them in a
follow-up **after #5971 merges**, to avoid overlapping its TS edits.
2. **Guard hardening**: validate the symlink *target* (not just that
it's a symlink), fail-loud on a malformed baseline, and code-enforce the
shrink-only ratchet. (This PR's guard catches the real-file erosion —
the actual failure mode; these are robustness extras.)
3. Pre-existing `shared/python` a2ui test failures (#5971-adjacent) and
a couple of stale doc line-refs, noted during review.
Companion: #5971.
Restore the 12 Python integration tools/ dirs to symlinks into
shared/python/tools. They had eroded to real, drifting copies via an
accidental stage_shared() leak (commit 534cd1efa7) — the structural root
cause of showcase divergence bugs. Symlinking re-establishes the single
source of truth; content is identical to shared (only render_a2ui naming
and the shared roll_dice/sanitize additions are adopted).
Add showcase/AGENTS.md documenting the 4 iron rules and the single-source
symlink mechanism, plus a validate-shared-symlinks CI guard (shrink-only
baseline) that fails on any NEW erosion.
The shared TS builder exported `_design_a2ui_surface` / `DESIGN_A2UI_SURFACE_TOOL_SCHEMA`
while the Python builder, all integration copies, and agent_server.ts use `render_a2ui`
/ `RENDER_A2UI_TOOL_SCHEMA`. Align the shared source to that source-of-truth name; this
also unbreaks the re-export in shared/typescript/tools/index.ts.
Flip the 4 TS a2ui builders (shared/typescript + mastra,
claude-sdk-typescript, langgraph-typescript) from the legacy flat
operation shape to v0.9 nested (createSurface / updateComponents /
updateDataModel), matching the Python builder and what A2UI consumers
process. Flat ops were never processed as valid nested operations, so
the surface schema and components were never applied.
Also align the empty-data guard to Python's `if data:` semantics (empty
object -> no updateDataModel), add a v0.9 parity guard test to all 4
test files, and add 12 gen-ui-a2ui-fixed aimock fixtures.
Turbopack has no resolve.extensionAlias parity (Next #82945), so
'next dev --turbopack' can't resolve the shared cell-model fold's
.js->.ts specifiers and fails with Can't resolve './live-status.js'.
The extensionAlias in next.config.ts (added in #5955) is honoured by
webpack, which next build already uses. Drop --turbopack from the dev
script so dev runs on webpack too and resolves the fold.
Deploy path unaffected: the Dockerfile builds with 'next build' (webpack)
and serves with 'next start' -- the dev script is never in the build or
runtime path.
PR #5952 (9a8cf615) added explicit `.js` extensions to the relative imports
inside the harness's shared cell-model fold
(showcase/harness/src/shared/cell-model/{cell-model,live-status,staleness}.ts)
— REQUIRED for the harness's pure-Node-ESM runtime and correct as-is.
But the dashboard re-exports that fold via shims
(showcase/shell-dashboard/src/lib/{cell-model,live-status,staleness,format-ts}.ts
`export * from "../../../harness/src/shared/cell-model/*"`), pulling the fold
into the dashboard's `next build`. `export *` does not rewrite the fold's
INTERNAL `.js` edges, and the dashboard's empty next.config.ts had no
extensionAlias, so webpack resolved `./live-status.js` literally, found only
the `.ts` source, and failed:
Module not found: Can't resolve './live-status.js'
Module not found: Can't resolve './staleness.js'
Module not found: Can't resolve './format-ts.js'
> Build failed because of webpack errors
Two-part fix (one coherent subject):
1. Resolution: add `webpack.resolve.extensionAlias` to
showcase/shell-dashboard/next.config.ts so `.js`/`.mjs` specifiers resolve
to `.ts`/`.tsx`/`.mts` sources — the bundler complement to TS NodeNext's
`.js`-import convention. Covers the `next build` (webpack) path CI uses.
The harness fold's `.js` imports are left untouched (they are correct).
2. CI gap: the dashboard build did not run on #5952 because the build matrix
is path-filtered and #5952 only touched `showcase/harness/**`, which
selects `showcase_harness` but not `shell_dashboard`. Add
`showcase/harness/src/shared/**` to the `shell_dashboard` paths-filter so
any change to the shared fold the dashboard compiles in also selects the
dashboard build — a fold change can never again ship an unbuilt dashboard.
Local red-green proof:
- RED (main, before fix): `next build` in showcase/shell-dashboard emitted the
4 fold-resolve errors above.
- GREEN (after extensionAlias): same build → 0 fold-resolve errors; the fold
resolves. Remaining `@/data/*.json` errors are the prebuild-generated files
(generate-registry/probe-docs) skipped in the local repro, produced in CI's
Docker build — unrelated to this fix.
## Summary
Routes the ~20 showcase demo backends to **aimock** (the record/replay
LLM proxy) over Railway **private networking** (`*.railway.internal`)
instead of aimock's **public** `*.up.railway.app` host.
Railway bills traffic to a public domain as **egress even
intra-project**, while `*.railway.internal` private networking is
**free** and **env-scoped**. The 240-concurrent-browser harness fleet
drives every demo continuously, so every LLM SSE stream from aimock back
to a demo backend is currently billed egress.
- aimock ≈ **89% of showcase egress**, ≈ **92% of the 13TB→78TB/mo
increase**.
- Estimated impact: avoids the ≈ **$602/mo → $3,856/mo** growth on the
aimock path.
## Change (config-only, reversible; SSOT-driven)
1. **SSOT** (`showcase/scripts/railway-envs.ts`): add an env-scoped
`internalDomain: "showcase-aimock.railway.internal"` to the aimock entry
in **both** envs. The public `domain` is **kept** (health probes /
external reachability).
2. **Emitter** (`showcase/scripts/emit-railway-envs-json.ts`): emit
`internalDomains` (additive, after `domains`) into the generated JSON.
Every non-aimock service keeps its frozen shape.
3. **Generated JSON** regenerated (oxfmt-canonical; 4-line additive
diff, only the aimock entry).
4. **Promote preflight** (`showcase/bin/railway`): `ssot_target_host`
now **prefers** the private `internalDomains[env]` over the public
`domains[env]`, so the Stage-2 (U5) serviceRef assertion requires demo
backends' `OPENAI_BASE_URL`/etc. to point at the private host.
Non-aimock targets (no `internalDomains`) fall back to their public host
unchanged.
5. **Harness** wiring probe needs **no code change** (it matches on
hostname); added a discriminating test pair + updated the drift-alert
Fix text to the private host.
**Target:** aimock binds `0.0.0.0:4010` (per
`showcase/aimock/RAILWAY.md`); demo backends resolve to
`http://showcase-aimock.railway.internal:4010`.
Deployed env vars, both envs (before → after):
| key | before (public, billed egress) | after (private, free) |
|---|---|---|
| `OPENAI_BASE_URL` | `https://<aimock>.up.railway.app/v1` |
`http://showcase-aimock.railway.internal:4010/v1` |
| `ANTHROPIC_BASE_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
| `GOOGLE_GEMINI_BASE_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
| `AIMOCK_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
`<aimock>` = `aimock-staging` (staging) / `showcase-aimock-production`
(prod). `railway.internal` is env-scoped, so staging demos reach the
staging aimock and prod demos reach prod aimock automatically — the same
private DNS name in both envs.
---
## Red-green proof (verbatim)
### RED — live staging today (billed public egress)
Deployed `showcase-langgraph-fastapi` (staging, service `06cccb5c-…`)
via Railway `variables(...)` GraphQL:
```
OPENAI_BASE_URL = https://aimock-staging.up.railway.app/v1
ANTHROPIC_BASE_URL = https://aimock-staging.up.railway.app
GOOGLE_GEMINI_BASE_URL = https://aimock-staging.up.railway.app
AIMOCK_URL = https://aimock-staging.up.railway.app
```
Pre-fix generated JSON aimock entry — **no** `internalDomains`:
```json
{ "domains": { "staging": "aimock-staging.up.railway.app",
"prod": "showcase-aimock-production.up.railway.app" },
"internalDomains": "ABSENT" }
```
### RED — Ruby U5 serviceref resolver, with the resolver reverted to
public-only
The three new U5 tests FAIL when `ssot_target_host` returns the public
host:
```
7 runs, 15 assertions, 3 failures
1) test_serviceref_prod_pointing_at_public_aimock_host_refuses:
expected REFUSE for prod serviceRef on the public egress host, got []
2) test_serviceref_prod_pointing_at_private_aimock_passes:
prod private aimock ref must not REFUSE, got ["REFUSE: §5.2 (showcase-ag2): prod
OPENAI_BASE_URL="http://showcase-aimock.railway.internal:4010/v1" does NOT point at
aimock's env-LOCAL prod host "showcase-aimock-production.up.railway.app" ..."]
3) test_ssot_target_host_prefers_internal_over_public:
expected "showcase-aimock.railway.internal",
actual "showcase-aimock-production.up.railway.app"
```
### GREEN — after the fix
Post-fix generated JSON aimock entry:
```json
{ "domains": { "staging": "aimock-staging.up.railway.app",
"prod": "showcase-aimock-production.up.railway.app" },
"internalDomains": { "staging": "showcase-aimock.railway.internal",
"prod": "showcase-aimock.railway.internal" } }
```
Ruby U5 serviceref tests (fixed resolver — prefers `internalDomains`):
```
7 runs, 20 assertions, 0 failures, 0 errors, 0 skips
```
Full Ruby spec suite:
```
184 runs, 715 assertions, 0 failures, 0 errors, 0 skips
```
Harness aimock-wiring probe (hostname-match; internal host with `:4010`
+ `/v1` → green, demo still on public host while harness on private →
red):
```
src/probes/aimock-wiring.test.ts 29 passed (was 27; +2 new: internal-host green, public-host drift red)
src/probes/drivers/aimock-wiring.test.ts 16 passed
src/rules/rule-loader.test.ts 61 passed (aimock-wiring-drift.yml parses after Fix-text update)
renderer + render-red-tick + orchestrator 157 passed (no alert-text snapshot broke)
```
Scripts test suite (emitter golden + everything): `2147 passed, 7
skipped` (one pre-existing `/tmp` lockfile flake in
`integration-smoke-registry.test.ts`, green on rerun after clearing the
stale lock). `emit --check` idempotent + oxfmt-canonical. Harness `tsc
--noEmit`: clean.
### GREEN — live infra confirmation
- aimock **staging** deployment status = `SUCCESS` (running), binds
`0.0.0.0:4010` — so `showcase-aimock.railway.internal:4010` resolves to
a live listener for any peer in the staging env.
- aimock serving LLM-shaped responses on `:4010`: `GET /health` → `200`;
`GET /v1/models` → `200` `{gpt-4o, gpt-4o-mini}`.
## What was vs wasn't live-validated
**Validated live:** the RED (deployed staging vars still on the public
egress host); aimock staging is deployed/running and serving on `:4010`;
the full unit/wiring/promote-preflight test surface passes with the new
internal-host values.
**NOT live-validated in-session:** the in-Railway-network DNS resolution
of `showcase-aimock.railway.internal:4010` from a peer service, and a
full staging deploy that flips the four keys + redeploys a demo backend.
Reason: the in-network vantage needs `railway ssh` (requires registering
a persistent account SSH key — a stateful, human-gated change I declined
to make unsupervised) or a staging deploy (the local Railway access
token was expired; the CLI refreshed it for read/GraphQL but a deploy is
a separate gated action). Railway private networking
(`*.railway.internal`) is a standard platform feature; the local
`docker-compose.local.yml` already runs the identical
`http://aimock:4010` internal-host pattern, and the wiring probe's
hostname match is exercised by the new tests. The staging deploy +
in-network curl is the first step of the rollout plan below and must be
run before prod.
## Irreducible egress remains
This does **not** zero showcase egress. Still billed: real browse users
hitting the public demo/shell domains; and aimock in **record mode**
proxying to real providers (the outbound prompt to
OpenAI/Anthropic/Google still bills).
## Rollout plan (reversible config change, staging-first, user-gated)
1. Land this branch (SSOT + generated JSON + assertions).
2. **Staging first:** set the four keys on staging demo backends +
`AIMOCK_URL` on the harness to
`http://showcase-aimock.railway.internal:4010` (`/v1` on
`OPENAI_BASE_URL`); redeploy one demo backend + aimock; from inside a
staging service curl
`http://showcase-aimock.railway.internal:4010/health` (expect 200) and
run a real demo LLM turn / aimock-wiring probe (expect green); confirm
the aimock egress path stops accruing
(`usage(measurements:[NETWORK_TX_GB])`).
3. **User-gated** promote to prod (staging→prod), same key flip.
4. **Rollback** = flip the keys back to the public host (no code revert
needed).
## Follow-ups (out of scope — do NOT bundle)
- Fleet right-sizing (240-concurrent-browser harness).
- `OPENAI_API_KEY` consolidation.
---
Draft — do not merge. Do not deploy to prod.
The aimock-wiring probe matches on hostname, so it needs no code change for
the private-networking migration. Add a discriminating test pair proving the
internal host (http://showcase-aimock.railway.internal:4010, with :4010 port
and /v1 suffix) resolves green while a demo still on the public egress host
goes red. Update the aimock-wiring-drift.yml Fix text to point operators at
the private host instead of the public production URL.
ssot_target_host now prefers the env-scoped internalDomains host over the
public domains host, so the Stage-2 (U5) serviceRef assertion expects demo
backends' OPENAI_BASE_URL/etc. to point at
http://showcase-aimock.railway.internal:4010 (free intra-env networking)
rather than the billed public egress host. Non-aimock targets (no
internalDomains) fall back to their public host unchanged.
Red-green: reverting the resolver makes the three new U5 tests fail (public
host asserted); restoring makes them pass. Full Ruby spec suite green (184
runs, 715 assertions, 0 failures).
Add an env-scoped `internalDomain` (showcase-aimock.railway.internal) to the
aimock SSOT entry in both envs and emit it as `internalDomains` in the
generated JSON. Railway bills public *.up.railway.app traffic as egress even
intra-project, while *.railway.internal private networking is free and
env-scoped. aimock is ~89% of showcase egress; routing the ~20 demo backends'
LLM traffic at the private host over http://showcase-aimock.railway.internal:4010
eliminates the billed path. The public `domain` is retained for health probes.
Serviceref host resolution + assertions to follow in subsequent commits on
this branch.
The harness runs as pure Node ESM (package.json "type":"module", built
with tsc moduleResolution:"bundler" which preserves extensionless import
specifiers at emit, launched via node dist/orchestrator.js). Under pure
Node ESM, relative import specifiers must carry the .js extension — a
convention the harness already honors everywhere (79/79 relative imports
in orchestrator.ts end in .js).
The relocated shared/cell-model fold broke that convention: cell-model.ts,
live-status.ts, staleness.ts, and the equivalence fixtures/test imported
sibling modules extensionless ("./live-status", "./staleness", etc). tsc,
vitest, and tsx all resolve those fine, so it built and tested green — but
at container boot node threw ERR_MODULE_NOT_FOUND on
dist/shared/cell-model/live-status and crash-looped the orchestrator,
breaking the staging auto-deploy.
Add the .js extension to every offending relative import to match the
harness convention. Minimal fix — no tsconfig change.
## What
A harness-native monitor that pages **#oss-alerts** when a whole
integration column collapses to **red-D0** ("completely gone" / backend
unreachable) in **production** — the incident class the per-cell alert
rules miss. On 2026-07-13 LGT went fully gone in prod and nothing paged.
Runs in-process on the control-plane on its own `*/15` cron (prod-only,
kill-switchable), reusing the same #oss-alerts webhook + shared
family-summary the family-silence monitor uses.
## Design — single verdict, no re-derivation
Detection runs the dashboard's **own `buildCellModel` fold** (the shared
`showcase/harness/src/shared/cell-model/` module both the dashboard and
the monitor import — relocated in the first commit of this PR) over the
same PocketBase `status` rows, then applies a column-gone predicate over
the resulting `CellModel` fields. Because it is literally the same pure
fold over the same rows, the monitor's per-cell verdict **equals the
DepthChip the dashboard renders, by construction** — no parallel
interpretation that can drift.
- `columnGone` = over wired+supported cells, `every(achievedDepth===0 &&
chipColor==="red" && !isStaleCell && surfaceState ∉
{unreachable,pending})`.
- Wired-cell enumeration mirrors the dashboard `page-stats` iteration
(the generator's `determineCellStatus` rule), read from the generated
`registry.json`.
## Behavior (per spec)
- **Producer-liveness SUSPENDED gate (F1)** — if the fleet producer is
idle/paused (the LGT mitigation state), the tick holds ALL state: no
OPEN, no CLOSE, no false recovery. Reuses the family-silence
inflight-aware `/api/runs` reasoning; idle window = 3× the longest
resolved producer period.
- **60s confirm re-read** (never a re-probe — re-probing a sick pool
deepens the incident); OPEN only if both scans agree; blips logged, not
paged.
- **15m-detect vs 1h-repost** state machine; CLOSE requires **positive
fresh-healthy** evidence, not mere absence-of-gone.
- **ONE aggregated** outage message + consolidated recovery notice;
`lastAlertAt` advances only after a successful Slack send (dedupe
discipline).
- Durable per-slug `{sinceAt,lastAlertAt}` JSON map in `alert_state`
(`getSet`/`putSet`).
- Prod-only gate: `SHOWCASE_ENV ?? RAILWAY_ENVIRONMENT_NAME ===
"production"` + `PROD_D0_MONITOR_ENABLED` kill-switch,
control-plane-only.
## Red-green proof
**No-divergence (the load-bearing test).** A frozen **test-only**
`naiveGone` (`achievedDepth===0` alone, ignoring color/staleness) is the
anti-example. On committed fixtures:
- **RED** — with the real predicate degraded to the naive depth-only
rule, the GREEN assertions fail (the naive rule wrongly labels the
**gray-D0-no-data** and **stale** columns as gone):
```
× GREEN: the real columnGone predicate fires ONLY on the red-D0-fresh
column
Expected false / Received true (stale + gray-D0 columns mislabeled)
Tests 3 failed | 3 passed (6)
```
- **GREEN** — with the real predicate, it fires ONLY on the red-D0-fresh
column and its per-cell inputs equal `buildCellModel`'s own outputs:
```
✓ src/fleet/control-plane/d0-gone-predicate.test.ts (6 tests)
```
**Producer-idle SUSPENDED (F1) proven load-bearing.** Disabling the gate
flips BOTH F1 tests red:
```
× RED (invisible-outage): idle producer + FRESH-gone rows → SUSPENDED (no OPEN)
× RED (false-recovery prevention): open outage, producer pauses, rows spuriously read healthy → NO recovery, HOLD
Tests 2 failed | 13 passed (15)
```
With the gate present: `Tests 15 passed (15)`.
**Full suite (post-fix):**
- Harness `tsc --noEmit`: **exit 0** · production build (`tsc -p
tsconfig.build.json`): **exit 0**
- Shell-dashboard `tsc --noEmit`: **exit 0** (Phase-1 gate — added the
required `isStaleCell`/`observedAtAgeMs` fields to the CellModel test
literal)
- New tests: predicate (6) + monitor behavioral (15) + registration gate
(7) + equivalence (7) = **35 passed**
- All control-plane tests: **382 passed**; orchestrator tests: **133
passed**
## Files
- `d0-gone-predicate.ts` (+test) — pure
`cellGone`/`columnGone`/`columnFreshHealthy` + registry-derived
wired-cell enumeration
- `d0-gone-monitor.ts` (+test, +gate test) — `createD0GoneMonitor`
factory
- `orchestrator.ts` — `internal:prod-d0-gone-monitor` cron registration
(prod-gated)
- `shell-dashboard/…/unified-cell.test.tsx` — CellModel literal fix
(Phase-1 tsc gate)
## Deferred / not in this PR
- **Live staging red-green (§10.8)** — the mandatory producer-paused
staging proof (induce a real comm-error, observe detect+confirm+post,
then pause `FLEET_PRODUCER_CRON` and assert no false recovery). Requires
a live staging harness + test webhook; not run here. **Should be
executed before undraft/rollout.**
- **#5872 field literals**
(`global-error-promotion`/`spec-failed`/`greenCount`) are HARD-GATED on
#5872 merge — the monitor consumes `buildCellModel` output today and
inherits #5872's tightened inputs per-slug automatically; no hard
dependency.
DRAFT — per-slug rollout and undraft/merge are the user's call.
The shell-dashboard Docker build context (repo root) copies scripts,
shared, integrations, shell-docs content, and shell-dashboard — but not
showcase/harness. The dashboard's src/lib/{cell-model,live-status,
staleness,format-ts}.ts re-export barrels forward to
../../../harness/src/shared/cell-model/*, so the isolated next build
failed with Module-not-found (CI build-check (shell-dashboard)). Local
next build passed only because the full monorepo is present.
Copy just the 4-file cell-model fold to the exact relative path the
shims expect, preserving the single-shared-fold invariant (harness
monitor still imports the same canonical copy) without dragging the
whole harness package into the dashboard image.
## Summary
This PR updates thread import documentation and threads navigation
across shell-docs, including a follow-up that makes the supported
CLI-only import journey explicit.
## What changed
- Adds thread import documentation:
- Root generic guide at `/threads-import` for frameworks without a
source-specific importer page.
- Google ADK-specific guide at `/google-adk/threads-import`.
- LangGraph-specific guide at `/langgraph-python/threads-import`, plus
generated LangGraph framework routes through the framework resolver.
- Authored framework wrappers so Mastra, AG2, Agno, Built-in Agent,
CrewAI Flows, DeepAgents, LlamaIndex, Microsoft Agent Framework, and
PydanticAI can surface the generic guide.
- Updates the CLI and import journey:
- Describes the CLI as supporting both cloud-hosted and self-hosted
Enterprise Intelligence.
- States up front that import runs from an app created with the
CopilotKit CLI and Enterprise Intelligence enabled.
- Clarifies that import uses the project already selected for the
current directory.
- Shows the source import commands before the optional `project select`
explanation, while still directing users to change targets before
running the dry run.
- Removes the suggestion that `skills onboard` plus `project select` can
add Enterprise Intelligence to an arbitrary existing app.
- Keeps the CLI import section concise and links to the complete generic
and source-specific guides.
- Clarifies future thread continuity:
- Threads Drawer uses the shared `CopilotChatConfigurationProvider`, so
selecting a thread updates the active `threadId` without separate state
wiring.
- Headless Threads uses `useThreads`; the app stores the selected
`thread.id` and passes it to the chat component as `threadId`.
- ADK and LangGraph guides retain source-specific stable ID mapping
guidance, including the LangGraph UUID caveat.
- Removes source-guide links that rewrote to the current framework page
and created circular navigation.
- Adds and organizes Threads Drawer docs:
- Extracts shared Threads Drawer content into a reusable snippet.
- Adds root and authored-framework wrapper pages.
- Moves Threads Drawer into a new expanded Threads nav grouping
alongside Headless Threads and Import Thread History.
- Renames and reorganizes threads docs:
- Relabels the existing `Threads` guide as `Headless Threads`.
- Removes the root prebuilt-components nav entry for Threads Drawer so
generated frameworks do not show it in two places.
- Updates root, authored, generated, and Built-in Agent nav metadata so
the Threads grouping is consistent.
- Updates nav tests to recognize pages nested inside authored-framework
groups.
## Validation
Run from `showcase/shell-docs`:
- `npm run lint` passes with the existing repository warning set.
- `npm run typecheck` passes.
- `npm run test` passes: 32 files and 167 tests.
- `npm run build` passes and generates 214 static pages; it reports the
existing Turbopack NFT tracing warning.
- Rendered and link-checked locally:
- `/cli`
- `/threads-import`
- `/mastra/threads-import`
- `/google-adk/threads-import`
- `/langgraph-python/threads-import`
- Root and framework-specific Threads Drawer and Headless Threads links
## Redirects
No redirect URLs were added or required:
- `Threads` was relabeled to `Headless Threads`, but the slug remains
`/threads`.
- `CopilotThreadsDrawer` was relabeled to `Threads Drawer`, but the URL
remains `/prebuilt-components/copilot-threads-drawer` and framework
equivalents.
- Threads Drawer moved in navigation, but the route did not move.
- `Import Thread History` is new at `/threads-import` and framework
routes such as `/mastra/threads-import`, `/google-adk/threads-import`,
and `/langgraph-python/threads-import`, so there is no prior URL to
redirect.
- The generated-framework routing change only selects framework-specific
content for the new `threads-import` slug.
- recoveryMessage multi-slug branch now wraps sinceAt in renderSince() so a
corrupt-but-shaped persisted sinceAt renders "unknown", not raw garbage
(matches outage path and single-slug recovery guard).
- summary.get() read failure now logs ERROR with errorId d0-monitor-summary-read
instead of a low-signal WARN (silently blinds the detector, same family as the
other silent-disable guards).
- classifyProducer inflight short-circuit now also requires anyWorkerOnline, so a
stale/orphaned inflight from a dead worker cannot force a blind live scan.
- Extended C6 recovery test to cover the multi-slug arm; split the inflight
predicate test into online/offline-worker cases; fixed C1(ii) rotation comment.
C1 (core): select the shown/named outage slugs by re-post-due-ness + rotation
instead of an alphabetical prefix slice, gate the aggregate post on a DUE slug
actually being named, and advance lastAlertAt only for named-and-due slugs. This
stops a wide (>maxSlugs) outage from re-posting every 15m (overflow slugs whose
clock never advanced stayed perpetually "due") and stops a newly-opened overflow
slug from forcing a per-tick re-post; every open slug is now named within a
bounded number of re-posts.
C2: derive outage onset from the gate-failing (non-green) winner rung, not only a
literal red row — a degraded winner no longer strands earliest at NaN and
re-stamps sinceAt to now.
C3: log a loud errorId when readStatusRows breaks on a finite totalPages while the
last page was full (short/inconsistent read) instead of silently truncating.
C4: floor repostMinutes at min 1 in resolveConfig (0 → repostMs 0 → every-tick
re-post).
C5: classifyProducer distinguishes a fresh deploy (workers online, no run history)
as "no-data / not-yet" from a paused "idle" producer; both HOLD (never page
without data) but the fresh case is no longer a misleading permanent SUSPEND.
C6: validate a persisted sinceAt is a parseable ISO before interpolating into the
outage message (renderSince) so a corrupt-but-shaped state blob renders "unknown",
not garbage.
C7: fix the mis-annotated `unsupported` fixture flag and add a symmetric assertion
pinning every naiveMislabels flag == (naiveGone != expectedGone).
Bucket-b: stamp the outage-duration line with evidenceMs (consistent with the
recovery post + lastAlertAt); skip the alert_state write on a pure no-op tick.
Sam Julien