The declarative-gen-ui sales-dashboard hero pill on ms-agent-python was
missing two aimock fixtures for its `generate_a2ui` two-stage flow:
1. the inner `_design_a2ui_surface` secondary-LLM fixture, and
2. the outer-agent narration fixture keyed by
`toolCallId: call_d6_decl_dash_outer_mspy_001`.
Without the narration fixture, after `generate_a2ui` returned there was no
matching outer turn to close the tool call, so the run looped re-emitting
`generate_a2ui` and eventually tripped the AG-UI protocol guard:
`Cannot send 'RUN_FINISHED' while tool calls are still active:
call_d6_decl_dash_outer_mspy_001` (surfaced as an error banner, turn 1
never completes, runsFinished=0).
Added both fixtures, mirroring the canonical langgraph-python
gen-ui-declarative.json sales-dashboard entries (inner keyed by
`toolName: _design_a2ui_surface`, narration keyed by the outer toolCallId).
After the fix the outer tool call closes, RUN_FINISHED fires cleanly
(runsFinished=1, no error banner), and the narration renders.
The previous probe relied on a fragile cross-pill-difference heuristic:
it fingerprinted the rendered step text and used inter-pill differences
as a settling signal, accepting any render with >=2 rows at the deadline.
That heuristic false-greened a real pill-3 stale-card regression, because
a stale render (pill 3 still showing pill 1's content) still satisfied
">=2 rows present" and the dedup was only a wait mechanism, never a hard
gate.
This rewrites the assertion to check EXPECTED CONTENT per pill. Each pill
carries a small set of low-brittleness content markers derived from its
step titles in the d5 fixture (product-launch -> launch/marketing,
team-offsite -> venue/agenda, competitor-research -> competitor/weakness).
The assertion polls the swap window until the card shows >=2 NON-EMPTY
step rows whose joined text contains ALL of that pill's markers; otherwise
it hard-fails. Marker matching is partial and case-insensitive so it stays
robust to live-LLM (--direct) nondeterminism while still proving the RIGHT
pill's content rendered.
Verified with a dynamic-fake red-green plus three retained false-green
guards: identical-across-pills canned steps, stale non-adjacent content
(pill 3 showing pill 1), and empty/whitespace-only rows all turn RED;
distinct-per-pill content passes.
NOTE: this EXPOSES a genuine pill-3 stale-state regression on
agno/langroid/crewai-crews. Those cells are legitimately RED under the
corrected probe until that backend/frontend bug is fixed (separate
follow-up). The langgraph-python reference passes.
The aimock_wiring:global probe went red on the residual-6 live services
(harness-workers + 5 starters). Root cause is EXCLUDE naming drift after
the egress/private-networking migration: EXCLUDE_SERVICES keyed starters
as showcase-starter-<framework> (matching only starter-<framework>), but
live Railway names are bare starter-<framework>[-lang] (e.g.
starter-strands-python, starter-langgraph-js) — no match, so they fell
through to being checked, landed in unwired, and kept the probe red.
harness-workers had no exclude entry at all.
Fix: exclude the whole starter-* family by prefix in isExcluded (starters
are contributor scaffolds, categorically not wired through aimock; safe
because no showcase-* backend name starts with starter-, so it never
over-excludes a real backend), and add bare harness-workers to the infra
exclude set. Superseded showcase-starter-* literals removed; inert
showcase-shell-* legacy literals retained to keep the diff minimal.
The 20 showcase-* LLM backends were already re-wired via Railway
AIMOCK_URL; this is a naming/exclusion fix only (no starter is repointed).
## What
The `claude-sdk-python` showcase agent `:8000` wedges under D6/LLM load:
two **synchronous** `anthropic.Anthropic().messages.create()` calls run
directly on the uvicorn asyncio event loop, freezing it for the full LLM
round-trip so `/health` stops answering. The watchdog counts 3
consecutive failures (~90s) and kill-restarts the container, dropping
active sessions. This is the pre-existing root cause behind the
#oss-alerts restart noise that #5987's alerting surfaced (it was never a
#5987 regression).
## Root cause (sync-in-async)
All in `integrations/claude-sdk-python/`:
- **`src/agents/agent.py`** — `_execute_tool`'s `generate_a2ui` branch
builds a sync `anthropic.Anthropic()` and calls
`client.messages.create()` synchronously. `_execute_tool` is a sync
callback invoked on the loop from **two** async callers: `run_agent`'s
agentic loop, and the Claude-Agent-SDK MCP tool handler in
`claude_agent_sdk_adapter.py`.
- **`src/agents/a2ui_dynamic.py`** — `_generate_a2ui`, same sync
pattern, invoked on the loop from the `run_a2ui_dynamic_agent`
generator.
## Fix — approach (a): `await asyncio.to_thread(...)` at the call sites
Chosen over approach (b) (`AsyncAnthropic` + `async def`) because it is
the **lowest blast radius**: the sync functions and the shared
`ExecuteTool` callback type stay unchanged, and wrapping at the call
sites fixes the **whole** tool-dispatch path uniformly (any current or
future sync tool in the dispatcher), not just `generate_a2ui`. Approach
(b) would only fix `generate_a2ui` unless `_execute_tool` were made
fully async — which ripples into the `ExecuteTool` type and both call
sites anyway.
Every occurrence fixed (file:line):
- `src/agents/agent.py:~1355` — `run_agent` call site → `await
asyncio.to_thread(_execute_tool, ...)`
- `src/agents/claude_agent_sdk_adapter.py:~152` — MCP `sdk_tool_handler`
→ `await asyncio.to_thread(execute_tool, ...)`
- `src/agents/a2ui_dynamic.py:~287` — secondary call site → `await
asyncio.to_thread(_generate_a2ui, ...)`
Whole-integration grep for the pattern (`anthropic.Anthropic(`, sync
`.messages.create`, `OpenAI(`, `time.sleep`, blocking I/O): only these
two sites existed. Every other agent in the integration already uses
`AsyncAnthropic`.
## Showcase parity verdict
`a2ui_dynamic.py` is **per-integration**, not shared. Each framework
(langgraph-python, llamaindex, ms-agent-python, pydantic-ai, ag2,
strands, agno, …) ships its own copy that "mirrors" langgraph-python but
uses that framework's own client. Only claude-sdk-python's copy used the
sync `anthropic.Anthropic()` pattern, so **blast radius is
claude-sdk-python only** — no other integration has this wedge. The
`tools` symlink (→ `showcase/shared/python/tools`) was **not** touched
(iron-rule: edit shared source only, never symlink copies; here no
shared change was needed).
## entrypoint.sh alert scoping
- `:8000` agent watchdog branch: **removed** the Slack POST, **kept**
the `kill -9 $AGENT_PID` self-heal. It now self-heals silently (root
cause fixed).
- Public `$PORT` `/api/health` branch: Slack POST to
`$SLACK_WEBHOOK_OSS_ALERTS` **intact** — the public Next.js wedge still
pages.
## Red → green proof
Faithful harness at `showcase/tests/repro/async-wedge/`: a real
`anthropic.Anthropic` sync client (real httpx transport) pointed via
`ANTHROPIC_BASE_URL`/`base_url` at a controllable slow local
Anthropic-compatible endpoint (`slow_anthropic.py`, `SLOW_SECONDS`
latency, serves both `messages.create` JSON and `messages.stream` SSE).
5× concurrent `POST /generate`; poll `/health` 1/s for 10s; assert &
exit non-zero on a false result.
**Minimal replica (`server.py`) — the load-bearing construct:**
```
REPLICA RED (FIXED=0, sync client on loop): ASSERT_SUMMARY is_fixed=0 ok=6 wedge=4 → PASS RED
REPLICA GREEN (FIXED=1, asyncio.to_thread): ASSERT_SUMMARY is_fixed=1 ok=10 wedge=0 → PASS GREEN
```
**Real production code (`prod_server.py`):**
```
PROD GENERATOR GREEN (real run_a2ui_dynamic_agent, fix in source):
ASSERT_SUMMARY expect=green ok=10 wedge=0 → PASS GREEN
(secondary sync _generate_a2ui fired 5x via to_thread; /health stayed 200)
MUTATION GUARD RED (real _generate_a2ui, sync-on-loop):
ASSERT_SUMMARY expect=red ok=5 wedge=5 → PASS RED (proves the harness fails on the bug)
PROD DIRECT GREEN (real _generate_a2ui via to_thread):
ASSERT_SUMMARY expect=green ok=10 wedge=0 → PASS GREEN
```
The mutation guard exercises the **real** production `_generate_a2ui`
sync-on-loop and confirms it wedges (`wedge=5`), while the `to_thread`
path stays fast-200 (`wedge=0`) — the harness is not vacuously green.
## Regression check
`pytest tests/python/` → **6 passed**. `ruff format --check` clean on
all 3 edited files (no new lint; the pre-existing origin/main ruff
findings are untouched, out of scope). `bash -n` clean on entrypoint.sh
and all repro scripts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
## CR round 2 — repro hardening
Addressed the one mandatory CR finding (M1) plus two optional items on
the `showcase/tests/repro/async-wedge/` harness. **Repro-harness only —
no production source touched** (`src/agents/a2ui_dynamic.py` unchanged).
**M1 (mandatory) — MODE=generator false-green closed.** The GREEN lane
previously asserted only `WEDGE==0`. If the mock's SSE were mis-parsed,
the `generate_a2ui` tool_use dropped, or the generator early-exited,
`_generate_a2ui` (the bug site) would never run, the loop would never
park, and `WEDGE==0` would pass trivially — proving nothing. Fix:
`prod_server.py` now wraps the real `a2ui_dynamic._generate_a2ui` with a
counter (`tool_dispatch_fired`), exposed via `/stats` and the
`/generate` response. `run_prod.sh` reads it after the poll window and
the GREEN lane now **fails (exit 6)** unless `tool_dispatch_fired >= 1`.
The counter tracks genuine execution regardless of call shape
(`asyncio.to_thread` offload or sync-on-loop).
**O1** — `slow_anthropic.py` now uses `await
asyncio.sleep(SLOW_SECONDS)` instead of blocking `time.sleep`, so the
mock's own loop stays free under concurrency.
**O2** — `run.sh` adds a 0.5s gap after firing load, before the first
`/health` poll, so poll 1 isn't wasted on a pre-block fast-200.
Also fixed a latent hang: `run_prod.sh` ended with a bare `wait` that
blocked forever on the long-lived uvicorn server jobs, so the
summary/assertion never printed. It now tracks and reaps only the
load-curl PIDs (bash-3.2 / `set -u` safe).
### Red-green proof (harness assertion)
RED (false-green injected via `REPRO_DROP_TOOL_USE=1` — tool_use
dropped, generator drains but never dispatches; env-only, no source
mutation):
```
ASSERT_SUMMARY expect=green ok=10 wedge=0 tool_dispatch_fired=0
FAIL GREEN: 0 wedges but tool_dispatch_fired=0 — _generate_a2ui never ran; WEDGE==0 is a false green (bug site never exercised)
EXIT=6
```
Old assertion (`WEDGE==0` only) would have PASSED here; hardened
assertion correctly FAILS.
GREEN (normal operation — tool dispatch fires):
```
ASSERT_SUMMARY expect=green ok=10 wedge=0 tool_dispatch_fired=9
PASS GREEN: 0 wedges AND tool_dispatch_fired=9 (>=1) — real production code exercised the bug site and kept /health fast-200 under load
EXIT=0
```
### No regression — deterministic MODE=direct mutation guard
```
MODE=direct EXPECT=red -> ASSERT_SUMMARY expect=red ok=8 wedge=2 tool_dispatch_fired=3 -> PASS RED (exit 0)
MODE=direct EXPECT=green -> ASSERT_SUMMARY expect=green ok=10 wedge=0 tool_dispatch_fired=3 -> PASS GREEN (exit 0)
```
---
## Fold-in: ag2 + llamaindex (fleet-wide sweep)
A CR flagged the same sync-LLM-on-the-event-loop wedge in two more
integrations,
and a fleet-wide sweep of every `integrations/*/src/**` Python file for
a sync
LLM `.create()` running **directly inside an `async def`** on the
uvicorn loop
found a THIRD previously-missed site. All three are now fixed with the
same
lowest-blast-radius approach (extract the blocking round-trip into a
sync
`_generate_a2ui`, and `await asyncio.to_thread(...)` from the async
wrapper):
- `integrations/ag2/src/agents/beautiful_chat.py` — `async def
generate_a2ui` (sync `openai.OpenAI().chat.completions.create`)
- `integrations/llamaindex/src/agents/a2ui_dynamic.py` — `async def
generate_a2ui` (sync `OpenAI().chat.completions.create`)
- `integrations/llamaindex/src/agents/agent.py` — `async def
generate_a2ui` (sync `OpenAI().chat.completions.create`) — **found by
the sweep, not in the original report**
### Fleet-sweep result (what was NOT touched, and why)
- **ag2 `agent.py` + `a2ui_dynamic.py`**: already non-blocking (`await
_async_openai_client.chat.completions.create`, i.e. `AsyncOpenAI`).
Confirmed, not re-touched.
- **agno, ms-agent-python, pydantic-ai, strands, crewai** sync `.create`
sites: all inside plain `def` framework tools (`@tool`-style), which the
frameworks dispatch in their own worker/executor context — never
bare-`await`ed on the loop. No wedge.
- **`.ts` voice routes / `agent_server.ts`**: TypeScript/Node, not the
uvicorn asyncio loop.
- **`llama_index.llms.openai.OpenAI(model=…)`** objects:
framework-managed async LLMs, distinct from the raw `openai` SDK client
at the wedge sites.
So the wedge exists at exactly these three `async def generate_a2ui`
sites (plus the two claude-sdk-python sites fixed above). All three
edited files are per-integration REAL files (not symlinks to
`showcase/shared/`) — verified per the iron rule.
`ag2`/`llamaindex` `entrypoint.sh` were intentionally **not** touched —
the Slack-alert scoping was specific to the claude-sdk-python incident.
### Red → green proof (real production code, deterministic)
New OpenAI-SDK repro harness (dev-only, under
`showcase/tests/repro/async-wedge/`, sibling of the anthropic one): a
real `openai` SDK client (real httpx transport) pointed via
`OPENAI_BASE_URL` at a controllable slow local OpenAI-compatible
endpoint (`slow_openai.py`). `run_prod_openai.sh` drives the **real
production `_generate_a2ui`** for each `TARGET`, 5× concurrent `POST
/generate`, poll `/health` 1/s ×10, with the `tool_dispatch_fired >= 1`
anti-false-green guard.
```
ag2-beautiful-chat RED: ASSERT_SUMMARY expect=red ok=5 wedge=5 tool_dispatch_fired=5 -> PASS RED
ag2-beautiful-chat GREEN: ASSERT_SUMMARY expect=green ok=10 wedge=0 tool_dispatch_fired=5 -> PASS GREEN
llamaindex-agent RED: ASSERT_SUMMARY expect=red ok=5 wedge=5 tool_dispatch_fired=5 -> PASS RED
llamaindex-agent GREEN: ASSERT_SUMMARY expect=green ok=10 wedge=0 tool_dispatch_fired=5 -> PASS GREEN
llamaindex-a2ui RED: ASSERT_SUMMARY expect=red ok=5 wedge=5 tool_dispatch_fired=5 -> PASS RED
llamaindex-a2ui GREEN: ASSERT_SUMMARY expect=green ok=10 wedge=0 tool_dispatch_fired=5 -> PASS GREEN
```
Source-level cross-check driving the **real on-disk `async
generate_a2ui` wrapper** directly: 5 concurrent 3s calls complete in
~3.1s (parallel offload threads, not serialized ~15s) while a bare
asyncio heartbeat keeps ticking (64 ticks through the load window) —
loop stays free. Negative control (ag2 wrapper temporarily reverted to
sync-on-loop): 15.11s serialized, heartbeat starved to 4 ticks — wedge
reproduced; source restored.
Existing claude-sdk-python lanes re-run green (no regression).
A fleet-wide sweep for sync LLM .create() calls running directly inside an
async def on the uvicorn event loop found three more wedge sites (same class
as the claude-sdk-python fix in this PR):
- integrations/ag2/src/agents/beautiful_chat.py
- integrations/llamaindex/src/agents/a2ui_dynamic.py
- integrations/llamaindex/src/agents/agent.py (missed by the original report)
Each extracts the blocking secondary-LLM round-trip into a sync _generate_a2ui
helper and offloads it via await asyncio.to_thread(...) from the async
generate_a2ui wrapper (lowest blast radius; sync body unchanged). ag2's other
agents already use AsyncOpenAI; all other sync .create sites are inside plain
def framework tools dispatched off-loop by their frameworks, so they do not
wedge. entrypoint.sh alert-scoping left untouched (claude-sdk-python-specific).
Adds a dev-only OpenAI-SDK repro harness (slow_openai.py, prod_server_openai.py,
run_prod_openai.sh) that drives the REAL production _generate_a2ui via a slow
local OpenAI-compatible endpoint, with a tool_dispatch_fired>=1 anti-false-green
guard. RED (sync-on-loop) -> GREEN (to_thread) verified for all three sites.
## Summary
- turn `/threads` into a product-oriented overview that explains why
developers use CopilotKit Threads and routes them by job to Drawer,
Headless, import, architecture, and deployment docs
- label the overview as `Overview` in the Threads navigation while
retaining `Threads` as the page title
- move the existing custom UI implementation guide to
`/headless-threads` across root, generated, authored, and Built-in
framework surfaces
- present the architecture as a product-to-system story: what users
experience, the UI/runtime/agent pieces in the app, and Enterprise
Intelligence as the cloud-hosted or self-hosted Threads platform
- provide responsive desktop and mobile diagram assets in light and dark
modes, showing durable history, replay to live, realtime sync,
lifecycle, and locking
- move `Threads & Persistence Architecture` into the Threads navigation
group across all framework modes
- replace the standalone migration CTA with contextual prose that leads
naturally to `Import Thread History`
- replace ambiguous linked-card layouts with a comparison table,
explicit action links, and a conventional next-steps list
- migrate implementation-intent links to `/headless-threads` while
keeping product-level links on `/threads`
- correct the ADK Vertex importer project-variable reference
- add regression coverage for route availability, nav labels/order,
page-title separation, shared architecture placement, and
framework-aware link rewriting
## Authoring surfaces
- **Shared/root:** `src/content/docs/{threads,headless-threads}.mdx`,
shared overview and headless snippets, root `meta.json`, and responsive
light/dark diagram assets
- **Authored frameworks:** integration wrappers and navigation metadata;
shared navigation logic inserts the architecture page into each Threads
group
- **Generated frameworks:** shared root routes, snippets, and root
navigation; generated data files are not hand-edited
- **Built-in Agent:** authored wrapper plus the same shared navigation
injection
- **Cross-links/reference:** Drawer, import, CLI, architecture,
tutorials, and `useThreads` reference pages
## Routing and redirects
No redirect is added for the old `/threads` implementation URL because
`/threads` is intentionally reused by the new overview. Existing
external links to `/threads` now land on the product overview, and
internal links that specifically mean the custom `useThreads`
implementation guide have moved to `/headless-threads`. Framework-aware
link rewriting scopes both routes normally.
## Base
This PR targets `main` after #5915 merged. Its diff contains only the
Threads overview follow-up commits.
## Validation
- `npm run lint` (passes with existing repository warnings only)
- `npm run test` (32 files, 170 tests)
- `npm run typecheck`
- `npm run build` (214 static pages generated; existing Turbopack
tracing warning only)
- `git diff --check`
- SVG XML validation for both architecture assets
- live browser checks on root, Mastra, LangGraph Python, and Built-in
Agent routes
- light/dark diagram rendering and narrow/desktop layout passes
The claude-sdk-python agent :8000 wedges under D6/LLM load: two synchronous
anthropic.Anthropic().messages.create() calls run directly on the uvicorn
asyncio event loop, freezing it for the full LLM round-trip so /health stops
responding. The watchdog counts 3 consecutive failures (~90s) and kill-restarts
the container, dropping active sessions.
Root cause (sync-in-async), all in integrations/claude-sdk-python/:
- src/agents/agent.py: _execute_tool's generate_a2ui branch builds a sync
anthropic.Anthropic() and calls messages.create() synchronously; invoked on
the loop from run_agent's agentic loop AND from the Claude-Agent-SDK MCP tool
handler in claude_agent_sdk_adapter.py.
- src/agents/a2ui_dynamic.py: _generate_a2ui, same sync pattern, invoked on the
loop from the run_a2ui_dynamic_agent generator.
Fix: wrap every async call site in `await asyncio.to_thread(...)` (lowest blast
radius — the sync functions and the shared ExecuteTool callback type are
unchanged, and the whole tool-dispatch path is fixed uniformly, not just
generate_a2ui):
- agent.py run_agent call site
- claude_agent_sdk_adapter.py MCP tool handler
- a2ui_dynamic.py secondary call site
Blast radius: claude-sdk-python only. a2ui_dynamic.py is per-integration (each
framework has its own copy); every other claude-sdk-python agent already uses
AsyncAnthropic. The `tools` symlink to shared/python was not touched.
entrypoint.sh: drop the Slack alert from the :8000 agent watchdog branch (keep
the kill-restart — it self-heals silently now that the root cause is fixed);
keep the LOUD #oss-alerts page on the public $PORT /api/health branch.
Adds showcase/tests/repro/async-wedge/ — a faithful RED/GREEN harness driving
the real anthropic sync client against a controllable slow endpoint, plus a
mutation guard on the real _generate_a2ui.
## What happened
The `claude-sdk-python` showcase column went fully red on **staging**
(37 cells, `BE ✗` cascading) while prod and the TypeScript sibling
stayed green on the same image. It wasn't 37 bugs — it was one wedged
replica.
**Root cause:** a D6 fan-out pushed the container's combined log volume
past Railway's ~500 logs/sec drain cap. `entrypoint.sh` pipes both
processes' stdout through an `awk` process-substitution, so Next.js's
`fd1` is a **synchronous pipe**; when Railway stopped draining, the pipe
filled and Next.js's next `console.log` blocked in `write(2)`,
**freezing the event loop**. Even the static `/api/health` went 502,
CPU→0, memory flat, process alive — so `restartPolicyType: ON_FAILURE`
never fired and `numReplicas: 1` meant one wedge reds the whole column.
Load-triggered, not a code/env diff.
Full analysis (root cause + mitigation trade-offs): Notion → *Showcase
stdout-backpressure wedge* under Plans / Proposals.
## The fix (two coordinated MUSTs)
Showcase's promise is preserved throughout — real D6 traffic, live page,
and full CVDIAG diagnostics all intact. Nothing is sampled or dropped.
- **MUST-1 — take the flood off stdout, losslessly.** Route CVDIAG's
per-LLM-call breadcrumb off stdout behind a new `CVDIAG_LOG_STDOUT` gate
(**default ON**, so every other integration is byte-for-byte unchanged);
the non-blocking PocketBase sink still receives every envelope at full
fidelity — which is the path `cvdiag classify` and the dashboard already
read from. `emit_cvdiag` now enqueues to PB **before** the stdout write
so a wedged fd1 can't cost the durable breadcrumb. `entrypoint.sh`
self-activates `CVDIAG_LOG_STDOUT=0` only when `CVDIAG_PB_URL` is wired
(safe: never silences stdout when PB isn't receiving). Plus uvicorn
`--no-access-log` to drop the access-log noise.
- **MUST-2 — detect and recover the hang, loudly.** Extend the watchdog
to poll the public `$PORT /api/health` and, on sustained failure, POST a
`#oss-alerts` Slack alert **before** kill-restarting — and add the same
alert to the agent-`:8000` branch. No silent recovery: every wedge
pages.
## Red → green proof
A docker `node:22-slim` repro (`showcase/tests/repro/stdout-wedge/`)
reproduces the real topology (same `awk` pipe, Railway-capped drain
reader, uvicorn+CVDIAG-shaped flood, static no-log `/api/health`
victim):
- **RED** (unmitigated): `/api/health` 200 → 502/timeout the instant the
flood crosses the cap; heartbeat frozen; CPU parked at 0. `200=6 /
WEDGE=11`.
- **GREEN** (flood cut below cap): health stays fast-200 across the
whole window; heartbeat advances. `200=16 / WEDGE=0`.
- `run.sh` asserts the outcome and **exits non-zero on a false result**
(a deliberately-forced false-GREEN exits 5, was exit 0 pre-fix).
- MUST-2: the actual `entrypoint.sh` watchdog loop, run verbatim against
a genuinely wedged port, detects → POSTs the captured alert → `kill -9`s
the real Next.js PID. `watchdog.sh` needle-anchors the public
probe/kill/alert against `entrypoint.sh` (mutating the probe fails the
test).
- Unit: `test_cvdiag_log_stdout_gate.py` — 5/5 incl. a hostile-stdout
durability test (RED: enqueue starved; GREEN: enqueue preserved). Full
`_shared` suite 19 passed / 2 skipped.
## Review
Tier-3 cr-loop (shared source + deploy config + kill path): 7-agent
review → adversarial per-finding verification → 5 mandatory fixes (all
red-green'd, isolated worktrees) → 7-agent confirmation round converged
to **zero mandatory findings** → bucket-(c) promotion audit
`PROMOTE_TO_A: 0`.
## ⚠️ Not merge-ready yet — draft on purpose
- [ ] **Staging branch-deploy validation** — local can't exercise the
one runtime unknown: that the container actually reaches
`showcase-pocketbase.railway.internal:8090` and lands a `cvdiag_events`
row. Must confirm on a staging deploy of this branch before merge.
- [ ] **Railway env wiring (out-of-band, human-gated):** set on
`showcase-claude-sdk-python` (staging + prod) —
`CVDIAG_BACKEND_EMITTER=1`,
`CVDIAG_PB_URL=http://showcase-pocketbase.railway.internal:8090`,
`CVDIAG_WRITER_KEY` (op:// in DevOps `showcase`),
`SLACK_WEBHOOK_OSS_ALERTS`. Without these MUST-1/MUST-2 stay inert
(safe: default is current behavior).
- [ ] Green CI.
## Follow-up (non-blocking, fail-safe)
Repro-harness polish (none false-GREEN-capable): GREEN heartbeat
assertion is timing-fragile under a raised start-delay (false-RED only);
`run.sh` local lane doesn't forward all tunables; `watchdog.sh`
webhook-assert race + helper-death flake; reader has no close handler on
the local lane. Pre-existing (bucket c): `OPENAI_API_KEY` warning is
mislabeled; Next.js has no readiness gate; `_SETUP_DONE` degrade-latch.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Self-activate CVDIAG_LOG_STDOUT=0 when CVDIAG_PB_URL is wired (safe: only silences stdout when PB is receiving; explicit override preserved). Extend the watchdog to poll the public $PORT /api/health and, on sustained failure, POST a loud #oss-alerts Slack alert BEFORE kill-restart; add the same alert to the agent-\:8000 branch (no silent recovery). Add uvicorn --no-access-log to cut the access-log flood. Together these keep the shared log stream under the 500/sec cap so the pipe never backs up.
Shared cvdiag_bootstrap: gate the per-LLM-call breadcrumb capture handler and the emit_cvdiag stdout write behind CVDIAG_LOG_STDOUT (default ON so every other integration is byte-for-byte unchanged; opt-out per service). Enqueue to the non-blocking PocketBase sink BEFORE the stdout write so a wedged fd1 cannot cost the durable breadcrumb. No sampling; full fidelity to PB. Red-green unit tests incl. a hostile-stdout durability test.
Faithful node:22-slim repro: fd1 through the same awk process-substitution as entrypoint.sh, a Railway-capped drain reader, a uvicorn+CVDIAG-shaped flood, and the static no-log /api/health as victim. RED wedges (200->502, CPU->0, heartbeat frozen); the FIXED lane stays 200 throughout. run.sh asserts the outcome (exit 3/4/5 on a false result, proven). watchdog.sh runs the entrypoint public-guard loop verbatim and needle-anchors it against entrypoint.sh.
Addresses @samjulien's review on #5982:
1. Contributing example command: the guide changed into a nonexistent
`examples/next-openai` and ran `dev:examples` (a workspace build/watch
script that never starts a server). Point it at the real
`examples/v1/next-openai` package and its `example-dev` (`next dev`)
script, which actually serves http://localhost:3000/presentation.
Fixed in the shared snippet + all per-integration copies.
2. LangGraph auth: langgraph variants are docs_mode: generated, so the
`/auth` route renders the root `docs/auth.mdx`, not the framework copy.
Add the missing `from langchain_core.runnables import RunnableConfig`
to the two Python blocks in the root source that route renders.
3. Self-contained fences: add the import to the non-tutorial
`langgraph/shared-state/predictive-state-updates.mdx` Python fence and
the `snippets/integrations/langgraph/frontend-tools.mdx` fence, so the
zero-missing-import claim holds for every non-tutorial langgraph block.
4. Contributor prerequisites: bump the `docs-contributions` guides from
pnpm 9 to pnpm 10 to match the code-contributions requirement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Consolidates three stale-docs fixes that were opened against the retired
`docs/content/docs/` tree (now a symlink) and so could no longer merge:
- Contributing/package-linking guides (root + all integration copies +
shared snippets): Turborepo is fully removed from the repo (no dep, no
turbo.json). Drop the Turborepo prerequisite, bump pnpm to v10.x to match
`packageManager`, describe the monorepo as a pnpm workspace orchestrated by
Nx, and replace `turbo run <task>` with verified equivalents:
`pnpm run build|dev|format|lint`, `pnpm exec nx run-many -t (un)link:global`,
`pnpm exec nx watch` for a single package, and `pnpm run dev:examples`
(the real script; `example-dev` did not exist). Supersedes #3509.
- built-in-agent/model-selection: hyphenate the Anthropic model IDs
(`claude-3-7-sonnet`, `claude-opus-4-1`, `claude-3-5-haiku`). Supersedes #3656.
- langgraph reference docs: add the missing
`from langchain_core.runnables import RunnableConfig` import to Python code
blocks that annotate `config: RunnableConfig`. Supersedes #4069.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Companion structural PR to #5971. It fixes the **root cause** behind the
a2ui divergence #5971 patched: the showcase's single-source symlinks
eroded to real, drifting copies.
`showcase/integrations/*/tools`, `*/shared-tools`, `*/_shared` are meant
to be **symlinks into `showcase/shared/...`** — `stage_shared()`
dereferences them for the Docker build, `restore_symlinks()` restores
them. An **accidental `stage_shared()` leak** (commit `534cd1efa7`, PR
#4449 "D5 all-green") committed the dereferenced real files instead of
restoring the symlinks. Once they were real files, they drifted — which
is exactly how the a2ui `render_a2ui` vs `_design_a2ui_surface` split
(fixed in #5971) arose.
## Changes
- **Restore 12 Python `tools/` dirs to symlinks** →
`../../shared/python/tools` (ag2, agno, claude-sdk-python, crewai-crews,
google-adk, langgraph-fastapi, langgraph-python, langroid, llamaindex,
ms-agent-python, pydantic-ai, strands). Content is byte-adopted from
shared — verified no load-bearing per-integration code is lost (only
`render_a2ui` naming + shared `roll_dice`/sanitize additions).
Integrations' intentional internal-planner names (llamaindex,
ms-agent-python) live in `src/`, not `tools/`, and are untouched.
- **`showcase/AGENTS.md` (+ `CLAUDE.md`, root pointers,
INTEGRATION-CHECKLIST section)** — canonical statement of the 4 iron
rules (identical tests, near-identical frontends, minimal backends,
per-integration fixtures) + the single-source symlink mechanism ("edit
the shared source only; a real file there is a bug"). These were
previously written down nowhere.
- **`validate-shared-symlinks` CI guard** — fails on any NEW erosion
(real dir where a symlink belongs), with a shrink-only baseline that
tightens to fully-enforcing as symlinks are restored. Mirrors the
existing `validate-*` ratchet pattern.
## Scope / independence
- **No overlap with #5971** — this PR touches nothing under
`showcase/shared/typescript/` and does not modify the 3 TS integration
`shared-tools/` dirs (verified: empty file-set intersection). Mergeable
independently.
- Build-safe: `stage_shared()` correctly dereferences the restored
symlinks (targets resolve within the build context);
`restore_symlinks()` recreates them post-build.
## Verified
- `validate-shared-symlinks` test suite: 7/7 pass; validator EXIT 0 (no
new erosion).
- Reviewed by a full panel (correctness, content-integrity, build/CI,
docs, scope, silent-failure, simplicity) — zero mandatory findings.
## Follow-ups (deliberately out of scope)
1. **3 TS `shared-tools` dirs** (mastra, claude-sdk-typescript,
langgraph-typescript) remain real (baselined) — symlink them in a
follow-up **after #5971 merges**, to avoid overlapping its TS edits.
2. **Guard hardening**: validate the symlink *target* (not just that
it's a symlink), fail-loud on a malformed baseline, and code-enforce the
shrink-only ratchet. (This PR's guard catches the real-file erosion —
the actual failure mode; these are robustness extras.)
3. Pre-existing `shared/python` a2ui test failures (#5971-adjacent) and
a couple of stale doc line-refs, noted during review.
Companion: #5971.
Restore the 12 Python integration tools/ dirs to symlinks into
shared/python/tools. They had eroded to real, drifting copies via an
accidental stage_shared() leak (commit 534cd1efa7) — the structural root
cause of showcase divergence bugs. Symlinking re-establishes the single
source of truth; content is identical to shared (only render_a2ui naming
and the shared roll_dice/sanitize additions are adopted).
Add showcase/AGENTS.md documenting the 4 iron rules and the single-source
symlink mechanism, plus a validate-shared-symlinks CI guard (shrink-only
baseline) that fails on any NEW erosion.
The shared TS builder exported `_design_a2ui_surface` / `DESIGN_A2UI_SURFACE_TOOL_SCHEMA`
while the Python builder, all integration copies, and agent_server.ts use `render_a2ui`
/ `RENDER_A2UI_TOOL_SCHEMA`. Align the shared source to that source-of-truth name; this
also unbreaks the re-export in shared/typescript/tools/index.ts.
Flip the 4 TS a2ui builders (shared/typescript + mastra,
claude-sdk-typescript, langgraph-typescript) from the legacy flat
operation shape to v0.9 nested (createSurface / updateComponents /
updateDataModel), matching the Python builder and what A2UI consumers
process. Flat ops were never processed as valid nested operations, so
the surface schema and components were never applied.
Also align the empty-data guard to Python's `if data:` semantics (empty
object -> no updateDataModel), add a v0.9 parity guard test to all 4
test files, and add 12 gen-ui-a2ui-fixed aimock fixtures.
Turbopack has no resolve.extensionAlias parity (Next #82945), so
'next dev --turbopack' can't resolve the shared cell-model fold's
.js->.ts specifiers and fails with Can't resolve './live-status.js'.
The extensionAlias in next.config.ts (added in #5955) is honoured by
webpack, which next build already uses. Drop --turbopack from the dev
script so dev runs on webpack too and resolves the fold.
Deploy path unaffected: the Dockerfile builds with 'next build' (webpack)
and serves with 'next start' -- the dev script is never in the build or
runtime path.
PR #5952 (9a8cf615) added explicit `.js` extensions to the relative imports
inside the harness's shared cell-model fold
(showcase/harness/src/shared/cell-model/{cell-model,live-status,staleness}.ts)
— REQUIRED for the harness's pure-Node-ESM runtime and correct as-is.
But the dashboard re-exports that fold via shims
(showcase/shell-dashboard/src/lib/{cell-model,live-status,staleness,format-ts}.ts
`export * from "../../../harness/src/shared/cell-model/*"`), pulling the fold
into the dashboard's `next build`. `export *` does not rewrite the fold's
INTERNAL `.js` edges, and the dashboard's empty next.config.ts had no
extensionAlias, so webpack resolved `./live-status.js` literally, found only
the `.ts` source, and failed:
Module not found: Can't resolve './live-status.js'
Module not found: Can't resolve './staleness.js'
Module not found: Can't resolve './format-ts.js'
> Build failed because of webpack errors
Two-part fix (one coherent subject):
1. Resolution: add `webpack.resolve.extensionAlias` to
showcase/shell-dashboard/next.config.ts so `.js`/`.mjs` specifiers resolve
to `.ts`/`.tsx`/`.mts` sources — the bundler complement to TS NodeNext's
`.js`-import convention. Covers the `next build` (webpack) path CI uses.
The harness fold's `.js` imports are left untouched (they are correct).
2. CI gap: the dashboard build did not run on #5952 because the build matrix
is path-filtered and #5952 only touched `showcase/harness/**`, which
selects `showcase_harness` but not `shell_dashboard`. Add
`showcase/harness/src/shared/**` to the `shell_dashboard` paths-filter so
any change to the shared fold the dashboard compiles in also selects the
dashboard build — a fold change can never again ship an unbuilt dashboard.
Local red-green proof:
- RED (main, before fix): `next build` in showcase/shell-dashboard emitted the
4 fold-resolve errors above.
- GREEN (after extensionAlias): same build → 0 fold-resolve errors; the fold
resolves. Remaining `@/data/*.json` errors are the prebuild-generated files
(generate-registry/probe-docs) skipped in the local repro, produced in CI's
Docker build — unrelated to this fix.
## Summary
Routes the ~20 showcase demo backends to **aimock** (the record/replay
LLM proxy) over Railway **private networking** (`*.railway.internal`)
instead of aimock's **public** `*.up.railway.app` host.
Railway bills traffic to a public domain as **egress even
intra-project**, while `*.railway.internal` private networking is
**free** and **env-scoped**. The 240-concurrent-browser harness fleet
drives every demo continuously, so every LLM SSE stream from aimock back
to a demo backend is currently billed egress.
- aimock ≈ **89% of showcase egress**, ≈ **92% of the 13TB→78TB/mo
increase**.
- Estimated impact: avoids the ≈ **$602/mo → $3,856/mo** growth on the
aimock path.
## Change (config-only, reversible; SSOT-driven)
1. **SSOT** (`showcase/scripts/railway-envs.ts`): add an env-scoped
`internalDomain: "showcase-aimock.railway.internal"` to the aimock entry
in **both** envs. The public `domain` is **kept** (health probes /
external reachability).
2. **Emitter** (`showcase/scripts/emit-railway-envs-json.ts`): emit
`internalDomains` (additive, after `domains`) into the generated JSON.
Every non-aimock service keeps its frozen shape.
3. **Generated JSON** regenerated (oxfmt-canonical; 4-line additive
diff, only the aimock entry).
4. **Promote preflight** (`showcase/bin/railway`): `ssot_target_host`
now **prefers** the private `internalDomains[env]` over the public
`domains[env]`, so the Stage-2 (U5) serviceRef assertion requires demo
backends' `OPENAI_BASE_URL`/etc. to point at the private host.
Non-aimock targets (no `internalDomains`) fall back to their public host
unchanged.
5. **Harness** wiring probe needs **no code change** (it matches on
hostname); added a discriminating test pair + updated the drift-alert
Fix text to the private host.
**Target:** aimock binds `0.0.0.0:4010` (per
`showcase/aimock/RAILWAY.md`); demo backends resolve to
`http://showcase-aimock.railway.internal:4010`.
Deployed env vars, both envs (before → after):
| key | before (public, billed egress) | after (private, free) |
|---|---|---|
| `OPENAI_BASE_URL` | `https://<aimock>.up.railway.app/v1` |
`http://showcase-aimock.railway.internal:4010/v1` |
| `ANTHROPIC_BASE_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
| `GOOGLE_GEMINI_BASE_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
| `AIMOCK_URL` | `https://<aimock>.up.railway.app` |
`http://showcase-aimock.railway.internal:4010` |
`<aimock>` = `aimock-staging` (staging) / `showcase-aimock-production`
(prod). `railway.internal` is env-scoped, so staging demos reach the
staging aimock and prod demos reach prod aimock automatically — the same
private DNS name in both envs.
---
## Red-green proof (verbatim)
### RED — live staging today (billed public egress)
Deployed `showcase-langgraph-fastapi` (staging, service `06cccb5c-…`)
via Railway `variables(...)` GraphQL:
```
OPENAI_BASE_URL = https://aimock-staging.up.railway.app/v1
ANTHROPIC_BASE_URL = https://aimock-staging.up.railway.app
GOOGLE_GEMINI_BASE_URL = https://aimock-staging.up.railway.app
AIMOCK_URL = https://aimock-staging.up.railway.app
```
Pre-fix generated JSON aimock entry — **no** `internalDomains`:
```json
{ "domains": { "staging": "aimock-staging.up.railway.app",
"prod": "showcase-aimock-production.up.railway.app" },
"internalDomains": "ABSENT" }
```
### RED — Ruby U5 serviceref resolver, with the resolver reverted to
public-only
The three new U5 tests FAIL when `ssot_target_host` returns the public
host:
```
7 runs, 15 assertions, 3 failures
1) test_serviceref_prod_pointing_at_public_aimock_host_refuses:
expected REFUSE for prod serviceRef on the public egress host, got []
2) test_serviceref_prod_pointing_at_private_aimock_passes:
prod private aimock ref must not REFUSE, got ["REFUSE: §5.2 (showcase-ag2): prod
OPENAI_BASE_URL="http://showcase-aimock.railway.internal:4010/v1" does NOT point at
aimock's env-LOCAL prod host "showcase-aimock-production.up.railway.app" ..."]
3) test_ssot_target_host_prefers_internal_over_public:
expected "showcase-aimock.railway.internal",
actual "showcase-aimock-production.up.railway.app"
```
### GREEN — after the fix
Post-fix generated JSON aimock entry:
```json
{ "domains": { "staging": "aimock-staging.up.railway.app",
"prod": "showcase-aimock-production.up.railway.app" },
"internalDomains": { "staging": "showcase-aimock.railway.internal",
"prod": "showcase-aimock.railway.internal" } }
```
Ruby U5 serviceref tests (fixed resolver — prefers `internalDomains`):
```
7 runs, 20 assertions, 0 failures, 0 errors, 0 skips
```
Full Ruby spec suite:
```
184 runs, 715 assertions, 0 failures, 0 errors, 0 skips
```
Harness aimock-wiring probe (hostname-match; internal host with `:4010`
+ `/v1` → green, demo still on public host while harness on private →
red):
```
src/probes/aimock-wiring.test.ts 29 passed (was 27; +2 new: internal-host green, public-host drift red)
src/probes/drivers/aimock-wiring.test.ts 16 passed
src/rules/rule-loader.test.ts 61 passed (aimock-wiring-drift.yml parses after Fix-text update)
renderer + render-red-tick + orchestrator 157 passed (no alert-text snapshot broke)
```
Scripts test suite (emitter golden + everything): `2147 passed, 7
skipped` (one pre-existing `/tmp` lockfile flake in
`integration-smoke-registry.test.ts`, green on rerun after clearing the
stale lock). `emit --check` idempotent + oxfmt-canonical. Harness `tsc
--noEmit`: clean.
### GREEN — live infra confirmation
- aimock **staging** deployment status = `SUCCESS` (running), binds
`0.0.0.0:4010` — so `showcase-aimock.railway.internal:4010` resolves to
a live listener for any peer in the staging env.
- aimock serving LLM-shaped responses on `:4010`: `GET /health` → `200`;
`GET /v1/models` → `200` `{gpt-4o, gpt-4o-mini}`.
## What was vs wasn't live-validated
**Validated live:** the RED (deployed staging vars still on the public
egress host); aimock staging is deployed/running and serving on `:4010`;
the full unit/wiring/promote-preflight test surface passes with the new
internal-host values.
**NOT live-validated in-session:** the in-Railway-network DNS resolution
of `showcase-aimock.railway.internal:4010` from a peer service, and a
full staging deploy that flips the four keys + redeploys a demo backend.
Reason: the in-network vantage needs `railway ssh` (requires registering
a persistent account SSH key — a stateful, human-gated change I declined
to make unsupervised) or a staging deploy (the local Railway access
token was expired; the CLI refreshed it for read/GraphQL but a deploy is
a separate gated action). Railway private networking
(`*.railway.internal`) is a standard platform feature; the local
`docker-compose.local.yml` already runs the identical
`http://aimock:4010` internal-host pattern, and the wiring probe's
hostname match is exercised by the new tests. The staging deploy +
in-network curl is the first step of the rollout plan below and must be
run before prod.
## Irreducible egress remains
This does **not** zero showcase egress. Still billed: real browse users
hitting the public demo/shell domains; and aimock in **record mode**
proxying to real providers (the outbound prompt to
OpenAI/Anthropic/Google still bills).
## Rollout plan (reversible config change, staging-first, user-gated)
1. Land this branch (SSOT + generated JSON + assertions).
2. **Staging first:** set the four keys on staging demo backends +
`AIMOCK_URL` on the harness to
`http://showcase-aimock.railway.internal:4010` (`/v1` on
`OPENAI_BASE_URL`); redeploy one demo backend + aimock; from inside a
staging service curl
`http://showcase-aimock.railway.internal:4010/health` (expect 200) and
run a real demo LLM turn / aimock-wiring probe (expect green); confirm
the aimock egress path stops accruing
(`usage(measurements:[NETWORK_TX_GB])`).
3. **User-gated** promote to prod (staging→prod), same key flip.
4. **Rollback** = flip the keys back to the public host (no code revert
needed).
## Follow-ups (out of scope — do NOT bundle)
- Fleet right-sizing (240-concurrent-browser harness).
- `OPENAI_API_KEY` consolidation.
---
Draft — do not merge. Do not deploy to prod.
The aimock-wiring probe matches on hostname, so it needs no code change for
the private-networking migration. Add a discriminating test pair proving the
internal host (http://showcase-aimock.railway.internal:4010, with :4010 port
and /v1 suffix) resolves green while a demo still on the public egress host
goes red. Update the aimock-wiring-drift.yml Fix text to point operators at
the private host instead of the public production URL.
ssot_target_host now prefers the env-scoped internalDomains host over the
public domains host, so the Stage-2 (U5) serviceRef assertion expects demo
backends' OPENAI_BASE_URL/etc. to point at
http://showcase-aimock.railway.internal:4010 (free intra-env networking)
rather than the billed public egress host. Non-aimock targets (no
internalDomains) fall back to their public host unchanged.
Red-green: reverting the resolver makes the three new U5 tests fail (public
host asserted); restoring makes them pass. Full Ruby spec suite green (184
runs, 715 assertions, 0 failures).
Add an env-scoped `internalDomain` (showcase-aimock.railway.internal) to the
aimock SSOT entry in both envs and emit it as `internalDomains` in the
generated JSON. Railway bills public *.up.railway.app traffic as egress even
intra-project, while *.railway.internal private networking is free and
env-scoped. aimock is ~89% of showcase egress; routing the ~20 demo backends'
LLM traffic at the private host over http://showcase-aimock.railway.internal:4010
eliminates the billed path. The public `domain` is retained for health probes.
Serviceref host resolution + assertions to follow in subsequent commits on
this branch.
The harness runs as pure Node ESM (package.json "type":"module", built
with tsc moduleResolution:"bundler" which preserves extensionless import
specifiers at emit, launched via node dist/orchestrator.js). Under pure
Node ESM, relative import specifiers must carry the .js extension — a
convention the harness already honors everywhere (79/79 relative imports
in orchestrator.ts end in .js).
The relocated shared/cell-model fold broke that convention: cell-model.ts,
live-status.ts, staleness.ts, and the equivalence fixtures/test imported
sibling modules extensionless ("./live-status", "./staleness", etc). tsc,
vitest, and tsx all resolve those fine, so it built and tested green — but
at container boot node threw ERR_MODULE_NOT_FOUND on
dist/shared/cell-model/live-status and crash-looped the orchestrator,
breaking the staging auto-deploy.
Add the .js extension to every offending relative import to match the
harness convention. Minimal fix — no tsconfig change.
## What
A harness-native monitor that pages **#oss-alerts** when a whole
integration column collapses to **red-D0** ("completely gone" / backend
unreachable) in **production** — the incident class the per-cell alert
rules miss. On 2026-07-13 LGT went fully gone in prod and nothing paged.
Runs in-process on the control-plane on its own `*/15` cron (prod-only,
kill-switchable), reusing the same #oss-alerts webhook + shared
family-summary the family-silence monitor uses.
## Design — single verdict, no re-derivation
Detection runs the dashboard's **own `buildCellModel` fold** (the shared
`showcase/harness/src/shared/cell-model/` module both the dashboard and
the monitor import — relocated in the first commit of this PR) over the
same PocketBase `status` rows, then applies a column-gone predicate over
the resulting `CellModel` fields. Because it is literally the same pure
fold over the same rows, the monitor's per-cell verdict **equals the
DepthChip the dashboard renders, by construction** — no parallel
interpretation that can drift.
- `columnGone` = over wired+supported cells, `every(achievedDepth===0 &&
chipColor==="red" && !isStaleCell && surfaceState ∉
{unreachable,pending})`.
- Wired-cell enumeration mirrors the dashboard `page-stats` iteration
(the generator's `determineCellStatus` rule), read from the generated
`registry.json`.
## Behavior (per spec)
- **Producer-liveness SUSPENDED gate (F1)** — if the fleet producer is
idle/paused (the LGT mitigation state), the tick holds ALL state: no
OPEN, no CLOSE, no false recovery. Reuses the family-silence
inflight-aware `/api/runs` reasoning; idle window = 3× the longest
resolved producer period.
- **60s confirm re-read** (never a re-probe — re-probing a sick pool
deepens the incident); OPEN only if both scans agree; blips logged, not
paged.
- **15m-detect vs 1h-repost** state machine; CLOSE requires **positive
fresh-healthy** evidence, not mere absence-of-gone.
- **ONE aggregated** outage message + consolidated recovery notice;
`lastAlertAt` advances only after a successful Slack send (dedupe
discipline).
- Durable per-slug `{sinceAt,lastAlertAt}` JSON map in `alert_state`
(`getSet`/`putSet`).
- Prod-only gate: `SHOWCASE_ENV ?? RAILWAY_ENVIRONMENT_NAME ===
"production"` + `PROD_D0_MONITOR_ENABLED` kill-switch,
control-plane-only.
## Red-green proof
**No-divergence (the load-bearing test).** A frozen **test-only**
`naiveGone` (`achievedDepth===0` alone, ignoring color/staleness) is the
anti-example. On committed fixtures:
- **RED** — with the real predicate degraded to the naive depth-only
rule, the GREEN assertions fail (the naive rule wrongly labels the
**gray-D0-no-data** and **stale** columns as gone):
```
× GREEN: the real columnGone predicate fires ONLY on the red-D0-fresh
column
Expected false / Received true (stale + gray-D0 columns mislabeled)
Tests 3 failed | 3 passed (6)
```
- **GREEN** — with the real predicate, it fires ONLY on the red-D0-fresh
column and its per-cell inputs equal `buildCellModel`'s own outputs:
```
✓ src/fleet/control-plane/d0-gone-predicate.test.ts (6 tests)
```
**Producer-idle SUSPENDED (F1) proven load-bearing.** Disabling the gate
flips BOTH F1 tests red:
```
× RED (invisible-outage): idle producer + FRESH-gone rows → SUSPENDED (no OPEN)
× RED (false-recovery prevention): open outage, producer pauses, rows spuriously read healthy → NO recovery, HOLD
Tests 2 failed | 13 passed (15)
```
With the gate present: `Tests 15 passed (15)`.
**Full suite (post-fix):**
- Harness `tsc --noEmit`: **exit 0** · production build (`tsc -p
tsconfig.build.json`): **exit 0**
- Shell-dashboard `tsc --noEmit`: **exit 0** (Phase-1 gate — added the
required `isStaleCell`/`observedAtAgeMs` fields to the CellModel test
literal)
- New tests: predicate (6) + monitor behavioral (15) + registration gate
(7) + equivalence (7) = **35 passed**
- All control-plane tests: **382 passed**; orchestrator tests: **133
passed**
## Files
- `d0-gone-predicate.ts` (+test) — pure
`cellGone`/`columnGone`/`columnFreshHealthy` + registry-derived
wired-cell enumeration
- `d0-gone-monitor.ts` (+test, +gate test) — `createD0GoneMonitor`
factory
- `orchestrator.ts` — `internal:prod-d0-gone-monitor` cron registration
(prod-gated)
- `shell-dashboard/…/unified-cell.test.tsx` — CellModel literal fix
(Phase-1 tsc gate)
## Deferred / not in this PR
- **Live staging red-green (§10.8)** — the mandatory producer-paused
staging proof (induce a real comm-error, observe detect+confirm+post,
then pause `FLEET_PRODUCER_CRON` and assert no false recovery). Requires
a live staging harness + test webhook; not run here. **Should be
executed before undraft/rollout.**
- **#5872 field literals**
(`global-error-promotion`/`spec-failed`/`greenCount`) are HARD-GATED on
#5872 merge — the monitor consumes `buildCellModel` output today and
inherits #5872's tightened inputs per-slug automatically; no hard
dependency.
DRAFT — per-slug rollout and undraft/merge are the user's call.
Jordan Ritter