## What
A harness-native monitor that pages **#oss-alerts** when a whole
integration column collapses to **red-D0** ("completely gone" / backend
unreachable) in **production** — the incident class the per-cell alert
rules miss. On 2026-07-13 LGT went fully gone in prod and nothing paged.
Runs in-process on the control-plane on its own `*/15` cron (prod-only,
kill-switchable), reusing the same #oss-alerts webhook + shared
family-summary the family-silence monitor uses.
## Design — single verdict, no re-derivation
Detection runs the dashboard's **own `buildCellModel` fold** (the shared
`showcase/harness/src/shared/cell-model/` module both the dashboard and
the monitor import — relocated in the first commit of this PR) over the
same PocketBase `status` rows, then applies a column-gone predicate over
the resulting `CellModel` fields. Because it is literally the same pure
fold over the same rows, the monitor's per-cell verdict **equals the
DepthChip the dashboard renders, by construction** — no parallel
interpretation that can drift.
- `columnGone` = over wired+supported cells, `every(achievedDepth===0 &&
chipColor==="red" && !isStaleCell && surfaceState ∉
{unreachable,pending})`.
- Wired-cell enumeration mirrors the dashboard `page-stats` iteration
(the generator's `determineCellStatus` rule), read from the generated
`registry.json`.
## Behavior (per spec)
- **Producer-liveness SUSPENDED gate (F1)** — if the fleet producer is
idle/paused (the LGT mitigation state), the tick holds ALL state: no
OPEN, no CLOSE, no false recovery. Reuses the family-silence
inflight-aware `/api/runs` reasoning; idle window = 3× the longest
resolved producer period.
- **60s confirm re-read** (never a re-probe — re-probing a sick pool
deepens the incident); OPEN only if both scans agree; blips logged, not
paged.
- **15m-detect vs 1h-repost** state machine; CLOSE requires **positive
fresh-healthy** evidence, not mere absence-of-gone.
- **ONE aggregated** outage message + consolidated recovery notice;
`lastAlertAt` advances only after a successful Slack send (dedupe
discipline).
- Durable per-slug `{sinceAt,lastAlertAt}` JSON map in `alert_state`
(`getSet`/`putSet`).
- Prod-only gate: `SHOWCASE_ENV ?? RAILWAY_ENVIRONMENT_NAME ===
"production"` + `PROD_D0_MONITOR_ENABLED` kill-switch,
control-plane-only.
## Red-green proof
**No-divergence (the load-bearing test).** A frozen **test-only**
`naiveGone` (`achievedDepth===0` alone, ignoring color/staleness) is the
anti-example. On committed fixtures:
- **RED** — with the real predicate degraded to the naive depth-only
rule, the GREEN assertions fail (the naive rule wrongly labels the
**gray-D0-no-data** and **stale** columns as gone):
```
× GREEN: the real columnGone predicate fires ONLY on the red-D0-fresh
column
Expected false / Received true (stale + gray-D0 columns mislabeled)
Tests 3 failed | 3 passed (6)
```
- **GREEN** — with the real predicate, it fires ONLY on the red-D0-fresh
column and its per-cell inputs equal `buildCellModel`'s own outputs:
```
✓ src/fleet/control-plane/d0-gone-predicate.test.ts (6 tests)
```
**Producer-idle SUSPENDED (F1) proven load-bearing.** Disabling the gate
flips BOTH F1 tests red:
```
× RED (invisible-outage): idle producer + FRESH-gone rows → SUSPENDED (no OPEN)
× RED (false-recovery prevention): open outage, producer pauses, rows spuriously read healthy → NO recovery, HOLD
Tests 2 failed | 13 passed (15)
```
With the gate present: `Tests 15 passed (15)`.
**Full suite (post-fix):**
- Harness `tsc --noEmit`: **exit 0** · production build (`tsc -p
tsconfig.build.json`): **exit 0**
- Shell-dashboard `tsc --noEmit`: **exit 0** (Phase-1 gate — added the
required `isStaleCell`/`observedAtAgeMs` fields to the CellModel test
literal)
- New tests: predicate (6) + monitor behavioral (15) + registration gate
(7) + equivalence (7) = **35 passed**
- All control-plane tests: **382 passed**; orchestrator tests: **133
passed**
## Files
- `d0-gone-predicate.ts` (+test) — pure
`cellGone`/`columnGone`/`columnFreshHealthy` + registry-derived
wired-cell enumeration
- `d0-gone-monitor.ts` (+test, +gate test) — `createD0GoneMonitor`
factory
- `orchestrator.ts` — `internal:prod-d0-gone-monitor` cron registration
(prod-gated)
- `shell-dashboard/…/unified-cell.test.tsx` — CellModel literal fix
(Phase-1 tsc gate)
## Deferred / not in this PR
- **Live staging red-green (§10.8)** — the mandatory producer-paused
staging proof (induce a real comm-error, observe detect+confirm+post,
then pause `FLEET_PRODUCER_CRON` and assert no false recovery). Requires
a live staging harness + test webhook; not run here. **Should be
executed before undraft/rollout.**
- **#5872 field literals**
(`global-error-promotion`/`spec-failed`/`greenCount`) are HARD-GATED on
#5872 merge — the monitor consumes `buildCellModel` output today and
inherits #5872's tightened inputs per-slug automatically; no hard
dependency.
DRAFT — per-slug rollout and undraft/merge are the user's call.
The shell-dashboard Docker build context (repo root) copies scripts,
shared, integrations, shell-docs content, and shell-dashboard — but not
showcase/harness. The dashboard's src/lib/{cell-model,live-status,
staleness,format-ts}.ts re-export barrels forward to
../../../harness/src/shared/cell-model/*, so the isolated next build
failed with Module-not-found (CI build-check (shell-dashboard)). Local
next build passed only because the full monorepo is present.
Copy just the 4-file cell-model fold to the exact relative path the
shims expect, preserving the single-shared-fold invariant (harness
monitor still imports the same canonical copy) without dragging the
whole harness package into the dashboard image.
## Summary
This PR updates thread import documentation and threads navigation
across shell-docs, including a follow-up that makes the supported
CLI-only import journey explicit.
## What changed
- Adds thread import documentation:
- Root generic guide at `/threads-import` for frameworks without a
source-specific importer page.
- Google ADK-specific guide at `/google-adk/threads-import`.
- LangGraph-specific guide at `/langgraph-python/threads-import`, plus
generated LangGraph framework routes through the framework resolver.
- Authored framework wrappers so Mastra, AG2, Agno, Built-in Agent,
CrewAI Flows, DeepAgents, LlamaIndex, Microsoft Agent Framework, and
PydanticAI can surface the generic guide.
- Updates the CLI and import journey:
- Describes the CLI as supporting both cloud-hosted and self-hosted
Enterprise Intelligence.
- States up front that import runs from an app created with the
CopilotKit CLI and Enterprise Intelligence enabled.
- Clarifies that import uses the project already selected for the
current directory.
- Shows the source import commands before the optional `project select`
explanation, while still directing users to change targets before
running the dry run.
- Removes the suggestion that `skills onboard` plus `project select` can
add Enterprise Intelligence to an arbitrary existing app.
- Keeps the CLI import section concise and links to the complete generic
and source-specific guides.
- Clarifies future thread continuity:
- Threads Drawer uses the shared `CopilotChatConfigurationProvider`, so
selecting a thread updates the active `threadId` without separate state
wiring.
- Headless Threads uses `useThreads`; the app stores the selected
`thread.id` and passes it to the chat component as `threadId`.
- ADK and LangGraph guides retain source-specific stable ID mapping
guidance, including the LangGraph UUID caveat.
- Removes source-guide links that rewrote to the current framework page
and created circular navigation.
- Adds and organizes Threads Drawer docs:
- Extracts shared Threads Drawer content into a reusable snippet.
- Adds root and authored-framework wrapper pages.
- Moves Threads Drawer into a new expanded Threads nav grouping
alongside Headless Threads and Import Thread History.
- Renames and reorganizes threads docs:
- Relabels the existing `Threads` guide as `Headless Threads`.
- Removes the root prebuilt-components nav entry for Threads Drawer so
generated frameworks do not show it in two places.
- Updates root, authored, generated, and Built-in Agent nav metadata so
the Threads grouping is consistent.
- Updates nav tests to recognize pages nested inside authored-framework
groups.
## Validation
Run from `showcase/shell-docs`:
- `npm run lint` passes with the existing repository warning set.
- `npm run typecheck` passes.
- `npm run test` passes: 32 files and 167 tests.
- `npm run build` passes and generates 214 static pages; it reports the
existing Turbopack NFT tracing warning.
- Rendered and link-checked locally:
- `/cli`
- `/threads-import`
- `/mastra/threads-import`
- `/google-adk/threads-import`
- `/langgraph-python/threads-import`
- Root and framework-specific Threads Drawer and Headless Threads links
## Redirects
No redirect URLs were added or required:
- `Threads` was relabeled to `Headless Threads`, but the slug remains
`/threads`.
- `CopilotThreadsDrawer` was relabeled to `Threads Drawer`, but the URL
remains `/prebuilt-components/copilot-threads-drawer` and framework
equivalents.
- Threads Drawer moved in navigation, but the route did not move.
- `Import Thread History` is new at `/threads-import` and framework
routes such as `/mastra/threads-import`, `/google-adk/threads-import`,
and `/langgraph-python/threads-import`, so there is no prior URL to
redirect.
- The generated-framework routing change only selects framework-specific
content for the new `threads-import` slug.
- recoveryMessage multi-slug branch now wraps sinceAt in renderSince() so a
corrupt-but-shaped persisted sinceAt renders "unknown", not raw garbage
(matches outage path and single-slug recovery guard).
- summary.get() read failure now logs ERROR with errorId d0-monitor-summary-read
instead of a low-signal WARN (silently blinds the detector, same family as the
other silent-disable guards).
- classifyProducer inflight short-circuit now also requires anyWorkerOnline, so a
stale/orphaned inflight from a dead worker cannot force a blind live scan.
- Extended C6 recovery test to cover the multi-slug arm; split the inflight
predicate test into online/offline-worker cases; fixed C1(ii) rotation comment.
C1 (core): select the shown/named outage slugs by re-post-due-ness + rotation
instead of an alphabetical prefix slice, gate the aggregate post on a DUE slug
actually being named, and advance lastAlertAt only for named-and-due slugs. This
stops a wide (>maxSlugs) outage from re-posting every 15m (overflow slugs whose
clock never advanced stayed perpetually "due") and stops a newly-opened overflow
slug from forcing a per-tick re-post; every open slug is now named within a
bounded number of re-posts.
C2: derive outage onset from the gate-failing (non-green) winner rung, not only a
literal red row — a degraded winner no longer strands earliest at NaN and
re-stamps sinceAt to now.
C3: log a loud errorId when readStatusRows breaks on a finite totalPages while the
last page was full (short/inconsistent read) instead of silently truncating.
C4: floor repostMinutes at min 1 in resolveConfig (0 → repostMs 0 → every-tick
re-post).
C5: classifyProducer distinguishes a fresh deploy (workers online, no run history)
as "no-data / not-yet" from a paused "idle" producer; both HOLD (never page
without data) but the fresh case is no longer a misleading permanent SUSPEND.
C6: validate a persisted sinceAt is a parseable ISO before interpolating into the
outage message (renderSince) so a corrupt-but-shaped state blob renders "unknown",
not garbage.
C7: fix the mis-annotated `unsupported` fixture flag and add a symmetric assertion
pinning every naiveMislabels flag == (naiveGone != expectedGone).
Bucket-b: stamp the outage-duration line with evidenceMs (consistent with the
recovery post + lastAlertAt); skip the alert_state write on a pure no-op tick.
Central structural lever: derive every monitor state from a single per-cell
classifier (classifyCell → gone | healthy | unknown). Treat UNKNOWN (gray /
no-data / stale / amber / comm-error) as UNKNOWN everywhere — never "gone",
never positive-healthy.
- B-F1: recovery/CLOSE requires POSITIVE green cells (cellHealthy: chipColor
green, achievedDepth>=3, fresh), not the mere absence of red. A gone column
decaying to no-data no longer auto-recovers. Fixed the healthyRows test
fixture to emit a genuine green D5/D6 ladder.
- B-onset: derive sinceAt from the folded verdict's contributing ladder rows,
not a raw row.state==="red" re-scan (removed keyBelongsToSlug).
- B-A5gap: guard self-heal/loud-log on "no slug has any wired cell", not
map.size (every integration slug is keyed even with zero wired cells).
- B-env: normalize the prod gate (trim+lowercase, empty-as-unset) so an empty
SHOWCASE_ENV no longer shadows a prod Railway env and a mis-cased/padded
value no longer silently disables the monitor. Added resolveMonitorEnv +
shouldRegister; orchestrator + gate test both use the real predicate.
- B-flap: an already-open outage keeps its hourly re-post even when a later
confirm scan is inconclusive (confirm gates only OPEN and CLOSE).
- B-cadence: advance lastAlertAt only for slugs actually named in the message
(respect maxSlugsInMessage); overflow slugs keep their clock.
- Cheap: no-wired-cells logs once per tick; MAX_SLUGS floors at 1; lastAlertAt
stamps evidenceMs; resolveConfig negative/NaN/empty coverage.
Bucket-(a) fixes, each with a local red-green test:
- A1: replace the substring `:${slug}` onset match with an anchored
exact slug-segment match (`keyBelongsToSlug`) so a prefix-colliding
sibling (`strands` vs `strands-typescript`) no longer mis-attributes
the earlier sibling's red onset. RED: strands' sinceAt was pulled to
the strands-typescript onset; GREEN: each slug gets its own onset.
- A2: recovery/CLOSE is now SYMMETRIC with OPEN — a recovery requires a
second agreeing fresh-healthy read (confirm scan). RED: a single
transient healthy read fired a false "recovered"; GREEN: held until
two reads agree.
- A3: guard an empty/degenerate schedule set — longestPeriodMs 0/NaN
would make idleWindowMs 0 → isProducerLive permanently false → the
monitor SUSPENDS forever and never pages. Falls back to a 45m default
window (DEFAULT_IDLE_WINDOW_MS) and logs at error.
- A4: bound readStatusRows — guard NaN/undefined totalPages (a `page >=
NaN` break never trips) and add a hard MAX_STATUS_PAGES cap so a full
page + bad totalPages cannot infinite-loop/OOM. RED: OOM; GREEN:
terminates at the cap.
- A5: a registry-load failure logs at error with a stable errorId (not a
silent warn-once permanent no-op), and the monitor accepts a loader
thunk so it re-reads registry.json each tick while the wired-cell set
is empty — a transiently-missing file self-heals without a redeploy.
- A6 (verified, no code change): createSlackWebhookTarget already throws
on every non-2xx (4xx/5xx/429/3xx/network-exhausted); added a test
asserting the monitor does NOT delete recovery state when the post
throws.
Bucket-(b): stamp the recovery message with the confirm-scan instant
(evidenceMs) not tick-start; wrap the scheduler tick handler in a
catch+errorId; log the prod env-gate skip at warn with a reason; add a
clarifying comment that the aggregate lastAlertAt reset is intentional
one-message-one-clock cadence; simplify the three dashboard barrel-shim
comments (drop the rot-prone enumerated symbol lists).
Add a harness-native monitor that pages #oss-alerts when a whole
integration column collapses to red-D0 ("completely gone" / backend
unreachable) in production — the incident class the per-cell alert rules
miss (LGT went fully gone on 2026-07-13 and nothing paged).
Detection runs the dashboard's OWN buildCellModel fold (the shared
cell-model module both the dashboard and the monitor import) over the
same PocketBase status rows and applies a column-gone predicate over the
resulting CellModel fields, so the monitor's verdict equals the DepthChip
the dashboard renders by construction — no parallel re-derivation.
- d0-gone-predicate.ts: pure cellGone/columnGone/columnFreshHealthy over
buildCellModel outputs + registry-derived wired-cell enumeration
(mirrors the dashboard page-stats iteration / determineCellStatus rule).
- d0-gone-monitor.ts: createD0GoneMonitor factory — producer-liveness
SUSPENDED gate (reuses the family-silence inflight-aware /api/runs
reasoning, 3x-longest-period idle window), 60s confirm re-read (never a
re-probe), 15m-detect vs 1h-repost state machine, positive-fresh-healthy
CLOSE gate, ONE aggregated outage / consolidated recovery Slack message,
durable per-slug JSON map in alert_state (getSet/putSet).
- orchestrator.ts: register internal:prod-d0-gone-monitor @ */15, gated on
SHOWCASE_ENV ?? RAILWAY_ENVIRONMENT_NAME === production + kill-switch,
control-plane-only (inside runControlPlane), reusing the oss_alerts
webhook target + shared memoized family summary.
- unified-cell.test.tsx: add the required isStaleCell/observedAtAgeMs
fields to the CellModel test literal (Phase-1 dashboard tsc gate).
Red-green: a frozen test-only naiveGone (achievedDepth===0 alone)
mislabels gray-D0-no-data and stale columns as gone on committed
fixtures (RED); the real predicate fires only on red-D0-fresh and matches
buildCellModel's own outputs (GREEN). Producer-idle SUSPENDED proven
load-bearing (disabling the gate flips both F1 tests red). Plus
confirm-scan blip-rejection, hourly dedup, recovery-clear, failure modes,
and the prod-only/kill-switch registration gate.
Move the pure cell-classification fold cluster (cell-model, live-status,
staleness, format-ts) out of showcase/shell-dashboard/src/lib/ into
showcase/harness/src/shared/cell-model/ so BOTH the dashboard and a new
harness monitor import ONE copy with zero duplication and no behavior change.
The harness builds via tsc -p tsconfig.build.json with rootDir:"src" and
cannot import outside its own src/, so the harness is the correct library
home. The dashboard consumes the cluster via relative path across the package
boundary (established precedent, e.g. d5-cadence-banner.redgreen.test.ts).
- git mv the four files into harness shared/cell-model/; their intra-cluster
relative imports stay valid (they move together, no external coupling).
- Replace the four original shell-dashboard paths with thin export-* barrels
so all ~51 existing dashboard import sites resolve unchanged.
- Repoint commError-contract-drift.test.ts's source-text drift parse at the
new canonical harness location (the barrels carry no derivation body).
- Add cell-model.equivalence.test.ts + committed fixtures + a pre-move
baseline JSON (generated from the original git-HEAD code) proving the move
is byte-identical across a red-D0, gray no-data, stale, mixed, all-green,
and unsupported column.
Round-3 CR fixes for the LGT persistence-disable preload.
HIGH-1: after installing the fs-write patches, import the node:fs/promises
namespace and assert each patched member is identity-equal to the installed
function; throw (fail boot) naming any mismatched member. Catches the
load-order case where fs/promises was linked before the reassignment and the
namespace snapshotted the original fn (silent bypass -> disk-growth recurrence).
HIGH-2: make the real-package behavioral test non-skippable under
LGT_REQUIRE_BEHAVIORAL=1 (missing runtime fails, not skips), and wire the
python-unit-tests job to set up Node, npm install the agent deps, and run the
langgraph-typescript pytest with that flag so a green check proves interception.
LOW: tolerant writer-shape guard regex (quote/whitespace/alias agnostic; still
trips on a named-import switch); read-only open/openSync reject ENOENT for
suppressed paths (write-intent still no-ops); mkdir recursive returns the
topmost-created dir per the real fs contract.
Adds a HIGH-1 guard-fires regression test.
Merging main into the branch pushed the aimock exact-duplicate count to 301
(main added cross-demo fixture aliases of the runtime-disambiguated-by-fixtureFile
kind, e.g. ag2 headless-complete/gen-ui-headless-complete). Verified the +4 are
NOT in the mastra context — the Partner Refresh's new fixtures introduce zero new
exact dupes. Ratchet the ceiling to match; 828/828 aimock-fixtures tests pass.
Round-2 CR fixes for the LGT file-persistence disable. Empirically verified
against the real @langchain/langgraph-api@1.1.17: the FileSystemPersistence
writer uses namespace fs access (fs.writeFile via import * as fs), so the
CJS .promises patch IS observed — but the round-1 patch covered only
writeFile+mkdir with an unanchored substring, leaving atomic write-then-rename
/ appendFile / *Sync / stream surfaces as silent bypasses.
- Patch every fs write surface (promises + sync + open/openSync/createWriteStream)
for the persist dir, so no future writer shape can silently grow .langgraph_api.
- Anchor path matching to the .langgraph_api path segment (not a substring);
normalise string/Buffer/URL forms — no more over-match data loss.
- Add a version + writer-shape guard that fails loudly at boot if the package
is upgraded or switches to named imports (prevents a silent recurrence).
- Honour mkdir {recursive:true} return contract; accept 1/true/yes/on env
conventions and log on both enabled and not-enabled branches.
- Tests: assert the SHIPPED reaper (_reap_watchdog_children/$BASHPID walk) not
a stale comment-only trap; replace the mock-shape behavioral reaper test with
one that drives the real entrypoint helpers; fix a flaky single-shot poll with
a bounded retry loop; refresh the stale truncate-era docstring; add a
real-package behavioral test proving disk stays empty and in-memory round-trip
still returns an assistant response.
The langgraph-typescript backend's @langchain/langgraph-api FileSystemPersistence
serialises all accumulated thread/run/checkpoint state to .langgraph_api on a
3-second timer. Under the D6 probe fan-out (36 parallel probes) the dir filled
past the 200MB size-watchdog threshold in ~90s, the watchdog killed the agent,
and on rapid restart the D6 cron refilled and re-tripped it until Railway
crash-loop backoff stopped restarting the container (2026-07-13 outage, staging
and prod).
Mirror PR #5825's langgraph-python fix, which exported
LANGGRAPH_DISABLE_FILE_PERSISTENCE=true so the python inmem runtime skips its
flush-to-disk loop. The TS package has no such switch and its persistence
writers are unexported module singletons behind an exports-map wall, so ship a
node --import preload (src/agent/disable-file-persistence.mjs) that, gated on the
same env var, no-ops node:fs/promises writeFile/mkdir for .langgraph_api paths
while leaving in-memory state (the real runtime state) intact. Wire it into
npm start and export the env var in entrypoint.sh.
Behavior preserved: runs still execute and thread state reads back from the
in-memory checkpointer within the container lifetime; only disk persistence is
removed, so the size-watchdog has nothing to fill and never trips under load.
## Problem
Claude Agent SDK docs could render malformed or missing extracted
snippets across the generated Python and TypeScript integration docs.
The generative UI pages had stale duplicate regions and generic setup
leakage, while the custom look-and-feel reasoning and slots pages
referenced demo cells or regions that did not exist for every Claude
integration.
## Why
The docs pipeline treated accidental duplicate region names across files
as intentional multi-file regions, and several authored docs pages
drifted from the actual generated showcase demo IDs/regions. That left
some pages visually correct at a glance but broken when users opened
specific extracted code snippets.
## Fix
- Move the shared `bar-chart-renderer` regions to the complete
`useComponent` call and delete stale duplicate snippet files.
- Add an explicit duplicate-region guard and verifier coverage for
accidental cross-file region collisions.
- Add line-emphasis support for extracted `<Snippet>` blocks and setup
`<DemoCode>` output.
- Scope generative UI feature pages away from generic `agent-setup`
boilerplate.
- Repair the shared reasoning-messages docs to use the generated
`reasoning-default` and `reasoning-custom` demo cells.
- Add the missing Claude Python chat-slots teaching snippet regions and
keep both Claude slot snippets self-contained.
- Broad-audit both Claude integration docs locally, then targeted-audit
the repaired reasoning/slots pages in light and dark mode.
- Rewrite interrupt-flow.mdx to explicitly state Mastra lacks native interrupt support
- Add clear warning callout and comparison table showing tool-based as the working approach
- Update index.mdx to mark interrupt-based as 'Not Supported' and tool-based as 'Supported'
- Provide concrete migration path with useHumanInTheLoop examples
- Fixes FAC-64: prevents users from following broken interrupt examples
The aggregate e2e-smoke:<slug> red signal omitted errorDesc on the normal
return path, so an abort/timeout/send-budget-exhausted red that runLevel
RETURNS (not throws) showed on the PRIMARY dashboard tick as an unclassified
content-shaped red — only the side chat:/tools: rows kept the classifier.
Thread the failing level's errorDesc (L3 precedence, L4 fallback) onto the
aggregate so the primary tick matches the side row and the launcher-phase
abort path. Does not change red/green — only carries the classifier.
Also reorder the aborted-and-empty short-circuit ABOVE the alternate-content
/ raw-byte evaluate reads: an aborted run's page is tearing down, so those
reads were swallowed against a dead page and emitted an ambiguous empty
histogram. Non-aborted runs still perform the alternate-content salvage.
A mid-poll abort — the external ctx.abortSignal firing, or the driver's
own hard-timeout landing during the first-token poll — makes runAttempt
return empty WITHOUT throwing. The retry loop breaks and control falls to
the clean-exit path, where the level was misclassified as a generic
content red ("empty assistant response", probe.exit outcome "err", no
errorDesc). That masqueraded a teardown/abort/timeout as a CONTENT
failure on the dashboard + CVDIAG.
Add an aborted-AND-empty guard before the content-red gate that
short-circuits to the same abort classification the other paths use
(errorDesc "abort", probe.exit outcome "timeout"). Discriminator is
abortSignal.aborted, not emptiness alone: a genuinely-completed-empty
turn (not aborted) stays the content-red "empty assistant response".
Harmonize the three readTurnState() consumers in the d4 chat-roundtrip probe
through one guarded safeReadTurnState() wrapper so a mid-poll readTurnState()
throw is handled consistently everywhere: it means "no reliable signal" ->
degraded widen + observable telemetry, never a silent false-red (the prior
readDegraded swallow) nor a spurious level-error (the prior unguarded
readBaseline/readTurnComplete escape). A genuinely-empty degraded turn still
reds at the ceiling.
Folds completing the PR's own items:
- item-1 first-send cap: guard the in-send press against SEND_PRESS_MIN_BUDGET_MS
so a near-hang type can't floor press to ~1ms and produce a generic
level-error; classify distinctly as send-budget-exhausted. Only press is
guarded (type opens the envelope), so a legitimately-small pageTimeoutMs still
issues a healthy first send.
- item-3 null-header: suppress the finally-block fallback probe.message.send once
a real-header boundary already fired, so a retry whose winning resend lands no
POST no longer emits a second null-header boundary (mis-attributed
edge_interference_signal).
Also add "abort" to the errorDesc JSDoc enumeration (zero-risk).
Red-green covered for all three behavioral items against the real
runLevel/readTurnComplete path with a faithful fake.
Tests for the #5882 bucket-(b) follow-up hardening:
- Budget-exhaustion retry guard (red-green): a near-exhausted-budget retry no
longer attempts a doomed ~1ms-floored resend (type invoked exactly once).
- Degraded-path floor (red-green): a degraded page (sseAttachFailed, no
completion signal) with a late-but-present token resolves GREEN instead of a
base-floor false-red; a genuinely-empty degraded run still reds.
- Retry telemetry re-attribution (focused test): a retry-rescued GREEN turn
records the winning attempt's edge headers on probe.message.send.
- Coverage: FIFO-cap CVDIAG_MAX_OUTSTANDING_STARTS_PER_URL eviction backstop;
DEBUG-auto-disarm fail-closed (disarmed => no raw-byte capture);
alternate-content / raw-byte block SKIPPED on the container-success path.
- Fake fix: makeLateTokenBrowser now mints per-page state so L3 and L4 each run
an independent stall+retry cycle (was a shared-page singleton that leaked
sendCount from L3 into L4, so the L4 retry path was never genuinely exercised).
Follow-up to #5882 (bucket-(b) CR items). Three behavioral/telemetry fixes
plus one documentation clarification, all in d4-chat-roundtrip.ts:
- Budget-exhaustion retry guard: skip a non-completion retry resend when the
remaining wall-clock budget is below RETRY_MIN_BUDGET_MS (750ms). A late
resend previously floored its type/press action timeout to ~1ms, throwing a
page-fault-shaped error that mis-classified the stall as a generic red — a
spurious-red flap source. The stall now reds on its own terms.
- Degraded-path floor: when the SSE interceptor silently no-ops
(sseAttachFailed), no completion signal ever arrives, so the poll could only
fall into the never-observed branch and pin the deadline to the base floor —
reintroducing the slow-first-token false-red #5882 targets. Consult
sseAttachFailed to WIDEN the never-observed wait to the per-attempt ceiling so
a late-but-present token on a degraded page is still captured.
- Retry edge-header re-attribution: on a retry-rescued turn, re-arm the
message-POST edge-header capture (messageSendEdge / lastMessagePostResp /
emitMessageSend latch) so probe.message.send / edge_interference_signal / the
DEBUG raw-byte sample reflect the WINNING attempt, not the stalled first one.
- Document why lastStoppedAtMs is retained on the d4 TurnState (write-only in
d4; part of the shared attachSseInterceptor global shape the d6 run-signal
snapshot also mirrors) so it does not read as dead code.
## Root cause
D4's L4 flap was a **stalled turn, not a client render race**: aimock
served content but the page never rendered it (the real 20:16:52Z
failure). The turn stream never delivered a first token to the DOM
within the fixed `textPollTimeoutMs`, so the poll read the container
empty → spurious "empty assistant response" red.
The **prior fix (`ceae0c2c9`) was inert in production**. It keyed the
turn-complete decision off the `onSseEvent` Node-side seam, which the
real launchers never wire (Playwright has no per-SSE-event signal).
`sseObserved` therefore stayed `false` on the real path and the whole
extension collapsed to the pre-fix base budget floor. Its red-green used
a fake page that invoked the seam synthetically, so the deadness was
never caught.
## The fix (3 parts, mirrors `d6-all-pills.ts`)
1. **Wire the real completion signal.** `wirePlaywrightPage.goto` now
calls `attachSseInterceptor(page)` before navigation (injectable for
tests), seeding the page-side `__hk_runsFinished` /
`__hk_copilotRunning` turn-lifecycle globals at `document_start`. A new
`readTurnState()` `E2ePage` seam reads them via `page.evaluate`; the
first-token poll keys off **that** — the same transport-level
`RUN_FINISHED` + DOM run-stop edge d6 trusts — not the dead `onSseEvent`
seam.
2. **Fast-fail genuinely-empty turns.** With a real turn-complete edge,
a turn that completes with empty assistant text reds in ~`completion +
FIRST_TOKEN_GRACE_MS` (bounded by `FIRST_TOKEN_FAST_FAIL_MS` ≈ 15s)
instead of burning the flat 60s. A completed-empty turn still reds (no
masking) and is never retried.
3. **Retry-on-non-completion.** A turn **observed in-flight** that never
signals completion within budget (stalled/dropped stream) retries once
before red. **Never-observed** (dead/no-turn) runs stop at the base
floor with no retry. Total wall-clock is bounded by `pageTimeoutMs`
(per-attempt budget split), so the retry can't blow past the ceiling.
## CR findings resolved
- `abortSignal.aborted` checked **inside** the poll loop (was only at
level entry).
- Body-scrape fallback keeps `fromAssistantContainer=false` **and** no
longer clears `cvdiagResponseEmpty` — so a fallback-salvaged red can no
longer emit `terminal_outcome=ok`.
- Red/green never gated on `cvdiag` (telemetry-only); the `onSseEvent`
seam is documented telemetry-only.
- No-turn fallback budget capped by the per-attempt ceiling (bounded by
`pageTimeoutMs`).
- Fallback `tail.length > 20` floor and `split("\n")[0]` truncation
**removed** (they false-red'd short/multiline valid answers).
- Corrected stale "not started" / dead-seam comments.
## Real-surface red-green (NO fake page)
Real chromium + real `attachSseInterceptor` + real D4 driver against a
local fixture serving an SSE `/api/copilotkit` with an injected
first-token/stream delay (aimock-style). `sseObserved-on-real-path =
(runsFinished>0 || attrPresent)` is read from the actual page-side
globals.
**RED — pre-fix (interceptor UNWIRED), late-token (token@3s):**
```
MODE: unwired scenario: late-token
L3(chat).state: red summary: "empty assistant response"
elapsed_ms: 1444
sseObserved-on-real-path: false <-- dead seam proven
```
**GREEN — post-fix (interceptor WIRED), same late-token:**
```
MODE: wired scenario: late-token
L3(chat).state: green summary: ""
elapsed_ms: 6333
LAST readTurnState on REAL path: {"runsFinished":0,"attrPresent":true,"sawRunningTrue":true,"runningNow":true}
sseObserved-on-real-path: true <-- interceptor fires on real path
```
**completed-empty (WIRED) — still RED, fast-fail:**
```
L3(chat).state: red summary: "empty assistant response"
elapsed_ms: 5340 (~2.5s/level, not the 60s ceiling)
LAST readTurnState on REAL path: {"runsFinished":1,"attrPresent":true,"sawRunningTrue":true,"runningNow":false}
sseObserved-on-real-path: true
```
**recoverable-stall (WIRED) — attempt 1 never completes → retry →
GREEN:**
```
L3(chat).state: green summary: ""
elapsed_ms: 13423 (attempt 1 polls to per-attempt ceiling observed=true/complete=false, then resends; attempt 2 renders)
sseObserved-on-real-path: true
```
## Checks
- `tsc --noEmit`: clean
- `tsc -p tsconfig.build.json` (build): clean
- vitest `d4-chat-roundtrip.test.ts`: 56/56 pass (suite rewritten to
exercise the real `readTurnState` path + retry/fast-fail;
late-token→green, completed-empty→red, recoverable-stall→green via
retry, permanent-stall→red)
- oxlint: 0 errors (pre-existing warnings only)
Kept as a **draft**; do not merge.
---
## Follow-up (b572697ab): attempt-scoped completion — the dominant a1
false-red
CR flagged that the poll's `sseDone = runsFinished >= 1` read the
page-GLOBAL monotonic counter. A **prior run on the page**
(auto-greeting / initial-mount run) leaves `runsFinished >= 1` and
`sawRunningTrue` already latched **before the user's turn starts**, so
the poll thought THIS turn had already completed, saw the still-empty
container, and spuriously **fast-failed RED**. It only passed the old
tests because the fake reset per send.
### Fixes in this commit
1. **Completion is now ATTEMPT-SCOPED.** A per-attempt BASELINE
(`runsFinished` + `runStartCount`) is captured at send time; the turn is
complete only on a **new edge past the baseline** (`runsFinished >
baseline`, or a new `runStartCount` DOM run-start), never the
page-global `>= 1`. A fresh baseline is taken before each retry resend,
so a stale prior edge can't defeat the retry.
`TurnState`/`readTurnState` now surface `runStartCount` +
`lastStoppedAtMs` (the sse-interceptor already latches them), and the
grace window is stamped from the **real finished edge** rather than the
poll's local clock (fixes the ~500ms-short grace).
2. **Retry resend bounded by remaining budget.** The retry `sendTurn()`
type/press timeouts are capped to `hardCeiling` (were flat
`pageTimeoutMs`, which let a stalled resend push poll-phase wall-clock
~2x past the ceiling).
3. **Attach-fault is observable.** A failed `attachSseInterceptor` now
emits an `onAttachFault` marker + sets `TurnState.sseAttachFailed`, so a
silent regression to the inert base-floor path is detectable.
Folded (cheap): `probe.message.send` fallback moved into `finally`
(fires on nav/send throw); external `ctx.abortSignal` aborts labeled
`"abort"` (not `driver-error`) in the aggregate; corrected
fast-fail-floor / hardCeiling / poll-deadline doc comments.
### Real-surface RED→GREEN (real Chromium + real `attachSseInterceptor`
+ local SSE fixture)
The fixture fires a PRIOR run on mount (bumps the page-global counters),
then the user turn renders a first token at 2800ms — PAST the 2s grace
window. Driven through the REAL `createE2eSmokeDriver` +
`wirePlaywrightPage` (L3/chat, the every-service level).
**PRE-FIX (`sseDone = runsFinished >= 1`):**
```
a1-priorrun L3.state: red "empty assistant response" elapsed 2141ms <-- FALSE-RED (the bug)
recoverable L3.state: red "empty assistant response" elapsed 2171ms <-- retry defeated by stale prior edge
wired-late L3.state: green elapsed 3136ms (no prior run → no false completion)
completed-empty L3.state: red "empty assistant response" elapsed 2156ms
```
**POST-FIX (attempt-scoped baseline):**
```
a1-priorrun L3.state: green elapsed 3285ms <-- fixed: prior run no longer false-reds
recoverable L3.state: green elapsed 5192ms <-- retry rescues (fresh baseline per resend)
wired-late L3.state: green elapsed 3150ms
completed-empty L3.state: red "empty assistant response" elapsed 2707ms <-- fast-fail (~completion+grace, not the 9s ceiling)
sseObserved-on-real-path: true (all) sseAttachFailed: false
```
### Checks
- `tsc --noEmit`: clean · `tsc -p tsconfig.build.json` (build): clean
- vitest `d4-chat-roundtrip.test.ts`: **63/63** (added: a1-regression
L3+L4, completed-empty-one-send, L3 grace/fast-fail/retry, attach-fault
telemetry)
- oxlint: **0 errors** (4 pre-existing warnings only)
Still a **draft**; do not merge.
---
## Follow-up (a): grace-window timestamp — Node-stamp completion instant
Four reviewers flagged the same line in `runAttempt`:
```ts
completeAtMs = stoppedAtMs > 0 ? stoppedAtMs : Date.now();
```
where `stoppedAtMs` came from the page-side
`readTurnState().lastStoppedAtMs`. Two defects in one line:
1. **Stale on SSE-only completion.** When the turn is detected complete
via `sseDone = runsFinished > baseline` but no fresh DOM stop-edge fires
for THIS turn, `lastStoppedAtMs` still holds a **stale prior-run** value
→ `graceEnd = staleStop + FIRST_TOKEN_GRACE_MS` lands in the past → the
~2s grace window collapses to the base floor → a late-but-present first
token **false-REDs**.
2. **Cross-clock skew.** `lastStoppedAtMs` is stamped on the
**browser-page** `Date.now()`, but `graceEnd`/deadline math runs on the
**Node** clock → page↔Node skew mis-sizes the window.
### Fix (single-point, simplifying)
Stamp `completeAtMs = Date.now()` (**Node**) at the first poll iteration
that observes THIS turn complete; `readTurnComplete` no longer threads
`stoppedAtMs`. Both clocks are now consistent; the stamp lags the true
finished edge by at most one 500ms poll interval, well inside
`FIRST_TOKEN_GRACE_MS`. Preserved: completed-empty still fast-reds;
base-floor and `hardCeiling` caps intact; no masking. `lastStoppedAtMs`
is retained on `TurnState` (still consumed by conversation-runner /
sse-interceptor / d6).
### Tightenings (net-negative source LOC: +54/−58 = **−4**)
- Extracted the attempt-0 baseline IIFE onto the existing `readBaseline`
helper (DRY; removes a drift hazard on the completion-scoping path).
- Fixed stale comments: fallback `emitMessageSend()` now runs in the
`finally`; eviction comment (rides the `onResponse` wiring, not "always
active"); `lastStoppedAtMs` doc.
### Red-green (mandatory)
**Unit — stale-`lastStoppedAtMs` grace collapse** (`SSE-only completion
with a STALE lastStoppedAtMs …`): prior finished run +
`sseOnlyStaleStop` + prior stop aged 5s + token 700ms after completion
(inside a healthy grace window).
```
RED (pre-fix stamping: completeAtMs = staleStop): expected 'red' to be 'green' → RED
GREEN (Node-stamp fix): 64/64 pass → GREEN
```
**Real-surface no-regression** (real chromium + `wirePlaywrightPage` →
real `attachSseInterceptor` + local SSE fixture,
`repro_d4_a1_realsurface.mts`):
```
a1-priorrun aggregate green · L3 green elapsed 3292ms (SSE-only-late, lastStoppedAtMs stale at completion)
completed-empty aggregate red · L3 "empty assistant response" elapsed 2618ms (fast-fail, not the 9s ceiling)
recoverable aggregate green · L3 green elapsed 5161ms
wired-late aggregate green · L3 green elapsed 3131ms
sseObserved-on-real-path: true (all) sseAttachFailed: false
```
### Checks (this commit)
- `tsc --noEmit`: clean · build (`tsc -p tsconfig.build.json`): clean
- vitest `d4-chat-roundtrip.test.ts`: **64/64**
- oxlint: **0 errors** (4 pre-existing warnings only)
- commit: `7bb47e64d`
Still a **draft**; do not merge.
---
## Follow-up (commit `5047f7ba9`): make the SSE-only-stale grace guard
actually bite
Three reviewers noted that the `sseOnlyStaleStop` guard above did not,
in fact, guard the Node-stamp fix. Root cause in the fake
(`makeLateTokenBrowser.readTurnState()`): the driver takes the
per-attempt baseline via `readTurnState()` **before** the first send,
and at that point `lastSendAtMs===0` made `elapsed` (~epoch ms) exceed
`completeAt`, spuriously flipping `complete=true` at the **baseline**
read. That inflated the baseline `runsFinished` to `prior+1`, so the
current turn's real finished edge never rose **past** the baseline and
the driver's `sseDone = runsFinished > baseline.runsFinished` could
never fire. The SSE-only-stale-grace path was therefore never entered —
the test passed only because the token rendered directly (path (a)), so
it did **not** guard the Node-stamp fix.
**Test-only fix:** gate the fake's `complete` on `started` (a turn has
actually been sent). The pre-send baseline read is now a **true**
baseline (`runsFinished = prior + priorSendsDone`, no current-turn
finish); the finished edge is a genuine THIS-turn transition the driver
observes via `sseDone`; and the SSE-only completion path with a stale
`lastStoppedAtMs` is genuinely exercised.
### RED-on-revert proof (the guard bites)
Temporarily reverted the production stamp back to the pre-fix form
(`completeAtMs = stoppedAtMs > 0 ? stoppedAtMs : Date.now()`,
re-threading `stoppedAtMs` through `readTurnComplete`), ran the reworked
test, then restored the Node-stamp:
```
RED (pre-fix revert: completeAtMs = stoppedAtMs):
× SSE-only completion with a STALE lastStoppedAtMs … → GREEN
AssertionError: expected 'red' to be 'green'
Test Files 1 failed (1)
GREEN (Node-stamp restored: completeAtMs = Date.now()):
✓ SSE-only completion with a STALE lastStoppedAtMs … → GREEN
Test Files 1 passed (1)
```
The production revert was **not committed** — final state has the
correct Node-stamp; the production diff in this commit is
**comment-only** (no logic change).
### Comment corrections (this commit)
- Completed-empty deadline comment: previously claimed "base floor
always respected"; corrected to state that
`Math.min(Math.max(baseBudgetEnd, graceEnd), fastFailEnd,
attemptCeiling)` intentionally clamps **below** the floor (fast-fail — a
completed-empty turn must red fast, not burn the base budget).
- `FIRST_TOKEN_FAST_FAIL_MS` doc: now spells out the full `Math.min`
term rather than only the `Math.max(base, grace)` cap.
### Checks (this commit)
- `tsc --noEmit`: clean · build (`tsc -p tsconfig.build.json`): clean
- vitest `d4-chat-roundtrip.test.ts`: **64/64**
- oxlint: **0 errors** (4 pre-existing warnings only)
- diff: driver (comments) + test file only; no `repro_*`/`baseline`
staged
- commit: `5047f7ba9`
Still a **draft**; do not merge.
The sseOnlyStaleStop guard in makeLateTokenBrowser was ineffective: the
driver reads the per-attempt baseline via readTurnState() BEFORE the
first send, and at that point lastSendAtMs===0 made elapsed (~epoch ms)
exceed completeAt, spuriously flipping complete=true at the baseline
read. That inflated the baseline runsFinished to prior+1, so the current
turn's real finished edge never rose PAST the baseline and the driver's
sseDone = runsFinished > baseline.runsFinished could never fire. The
SSE-only-stale-grace path was therefore never entered — the test passed
only because the token rendered directly, so it did NOT guard the
Node-stamp fix.
Gate the fake's complete on started (a turn has been sent) so the
pre-send baseline read is a TRUE baseline (runsFinished = prior +
priorSendsDone). The finished edge is now a genuine THIS-turn transition
the driver observes via sseDone, and the SSE-only completion path with a
stale lastStoppedAtMs is genuinely exercised.
Proof the guard now bites (temporary production revert, not committed):
- pre-fix stamp (completeAtMs = stoppedAtMs > 0 ? stoppedAtMs : now):
test FAILS, expected 'red' to be 'green' (grace collapses).
- restored Node-stamp (completeAtMs = Date.now()): test PASSES.
Also corrects two production comments (3 reviewers flagged): the
completed-empty deadline comment claimed the base floor is always
respected, but Math.min(..., fastFailEnd, attemptCeiling) intentionally
clamps below the floor (fast-fail); and the FIRST_TOKEN_FAST_FAIL_MS doc
now spells out the full Math.min term. Comment-only, no logic change.
On an SSE-only completion (turn detected complete via runsFinished>baseline
with no fresh DOM stop-edge for THIS turn), readTurnState().lastStoppedAtMs
still held a stale prior-run value, and it is stamped on the browser-page
clock while graceEnd/deadline math runs on the Node clock. Feeding it into
Node-clock arithmetic pushed graceEnd into the past and collapsed the
FIRST_TOKEN_GRACE_MS window to the base floor, false-REDing a late-but-present
first token.
Stamp completeAtMs from Date.now() (Node) at the first poll that observes
THIS turn complete; readTurnComplete no longer threads stoppedAtMs. Also DRY
the attempt-0 baseline onto the existing readBaseline helper and fix stale
comments (fallback emit is in finally; eviction rides the onResponse wiring;
lastStoppedAtMs doc). completed-empty still fast-reds; base floor and
hardCeiling caps preserved.
Adds a red-green unit test modelling an SSE-only completion with a stale
lastStoppedAtMs (grace collapses pre-fix, honored post-fix).
Sam Julien