The aimock-wiring probe matches on hostname, so it needs no code change for
the private-networking migration. Add a discriminating test pair proving the
internal host (http://showcase-aimock.railway.internal:4010, with :4010 port
and /v1 suffix) resolves green while a demo still on the public egress host
goes red. Update the aimock-wiring-drift.yml Fix text to point operators at
the private host instead of the public production URL.
ssot_target_host now prefers the env-scoped internalDomains host over the
public domains host, so the Stage-2 (U5) serviceRef assertion expects demo
backends' OPENAI_BASE_URL/etc. to point at
http://showcase-aimock.railway.internal:4010 (free intra-env networking)
rather than the billed public egress host. Non-aimock targets (no
internalDomains) fall back to their public host unchanged.
Red-green: reverting the resolver makes the three new U5 tests fail (public
host asserted); restoring makes them pass. Full Ruby spec suite green (184
runs, 715 assertions, 0 failures).
Add an env-scoped `internalDomain` (showcase-aimock.railway.internal) to the
aimock SSOT entry in both envs and emit it as `internalDomains` in the
generated JSON. Railway bills public *.up.railway.app traffic as egress even
intra-project, while *.railway.internal private networking is free and
env-scoped. aimock is ~89% of showcase egress; routing the ~20 demo backends'
LLM traffic at the private host over http://showcase-aimock.railway.internal:4010
eliminates the billed path. The public `domain` is retained for health probes.
Serviceref host resolution + assertions to follow in subsequent commits on
this branch.
The boot-smoke step only treated ERR_MODULE_NOT_FOUND as failure, so other
module-resolution regressions (ERR_UNSUPPORTED_DIR_IMPORT,
ERR_PACKAGE_PATH_NOT_EXPORTED, ERR_UNKNOWN_FILE_EXTENSION,
ERR_INVALID_MODULE_SPECIFIER) were swallowed as BOOT_OK/exit 0 — the very
class of bug this gate exists to catch could slip through. It also had no
process.exit(0) on the success/expected-error paths and no bounded timeout,
so a future open handle at import time could hang node -e to the job's
25-minute ceiling.
- Broaden the failure condition to a MODULE_RESOLUTION_CODES set (any of the
five codes => BOOT_FAIL, exit 1). Non-module-resolution runtime errors
(e.g. the HARNESS_ROLE env guard, no such code) stay BOOT_OK/exit 0.
- Add explicit process.exit(0) on both the success and expected-error paths.
- Wrap the node invocation in `timeout 120s` (non-zero on timeout => step
fails) and add step-level timeout-minutes: 5.
Red-green proof (extracted guard logic vs synthetic modules): current logic
passes ERR_UNSUPPORTED_DIR_IMPORT / ERR_PACKAGE_PATH_NOT_EXPORTED at exit 0
(RED gap); hardened logic fails all five codes at exit 1, keeps benign
runtime error at exit 0, and against the real built dist/orchestrator.js
reports BOOT_OK and exits promptly (779ms, no hang).
CI missed the extensionless-import regression because tsc (bundler
resolution), vitest, and tsx all resolve extensionless relative
specifiers fine — no existing step ever ran the real node dist module
graph, which is what the container actually does at boot.
Add a boot-smoke to the Validate Showcase job (already gated on
showcase/harness/**): after building the harness dist, load
dist/orchestrator.js via a node import() and fail hard on
ERR_MODULE_NOT_FOUND. A later runtime error from missing env/PocketBase
is expected and passes — only a module-resolution failure reddens the
build. Verified red-green: the guard exits 1 on the pre-fix
extensionless imports and 0 once the .js extensions are added.
The harness runs as pure Node ESM (package.json "type":"module", built
with tsc moduleResolution:"bundler" which preserves extensionless import
specifiers at emit, launched via node dist/orchestrator.js). Under pure
Node ESM, relative import specifiers must carry the .js extension — a
convention the harness already honors everywhere (79/79 relative imports
in orchestrator.ts end in .js).
The relocated shared/cell-model fold broke that convention: cell-model.ts,
live-status.ts, staleness.ts, and the equivalence fixtures/test imported
sibling modules extensionless ("./live-status", "./staleness", etc). tsc,
vitest, and tsx all resolve those fine, so it built and tested green — but
at container boot node threw ERR_MODULE_NOT_FOUND on
dist/shared/cell-model/live-status and crash-looped the orchestrator,
breaking the staging auto-deploy.
Add the .js extension to every offending relative import to match the
harness convention. Minimal fix — no tsconfig change.
## Problem
Docs-only showcase runs build successfully, but result aggregation fails
with zero `build-result-*` directories. That prevents the Railway
staging redeploy and posts a misleading build-failure alert.
## Why
After `actions/download-artifact` was upgraded from v4 to v8, a pattern
matching exactly one artifact is extracted directly into the configured
directory as `result.json`. The aggregator only accepted the
multi-artifact layout with nested `build-result-*/result.json` files.
## Fix
- Read the root-level `result.json` when no nested artifact directories
exist.
- Keep the existing fail-loud behavior when neither layout is present.
- Add a regression test for the single-artifact layout.
## What
A harness-native monitor that pages **#oss-alerts** when a whole
integration column collapses to **red-D0** ("completely gone" / backend
unreachable) in **production** — the incident class the per-cell alert
rules miss. On 2026-07-13 LGT went fully gone in prod and nothing paged.
Runs in-process on the control-plane on its own `*/15` cron (prod-only,
kill-switchable), reusing the same #oss-alerts webhook + shared
family-summary the family-silence monitor uses.
## Design — single verdict, no re-derivation
Detection runs the dashboard's **own `buildCellModel` fold** (the shared
`showcase/harness/src/shared/cell-model/` module both the dashboard and
the monitor import — relocated in the first commit of this PR) over the
same PocketBase `status` rows, then applies a column-gone predicate over
the resulting `CellModel` fields. Because it is literally the same pure
fold over the same rows, the monitor's per-cell verdict **equals the
DepthChip the dashboard renders, by construction** — no parallel
interpretation that can drift.
- `columnGone` = over wired+supported cells, `every(achievedDepth===0 &&
chipColor==="red" && !isStaleCell && surfaceState ∉
{unreachable,pending})`.
- Wired-cell enumeration mirrors the dashboard `page-stats` iteration
(the generator's `determineCellStatus` rule), read from the generated
`registry.json`.
## Behavior (per spec)
- **Producer-liveness SUSPENDED gate (F1)** — if the fleet producer is
idle/paused (the LGT mitigation state), the tick holds ALL state: no
OPEN, no CLOSE, no false recovery. Reuses the family-silence
inflight-aware `/api/runs` reasoning; idle window = 3× the longest
resolved producer period.
- **60s confirm re-read** (never a re-probe — re-probing a sick pool
deepens the incident); OPEN only if both scans agree; blips logged, not
paged.
- **15m-detect vs 1h-repost** state machine; CLOSE requires **positive
fresh-healthy** evidence, not mere absence-of-gone.
- **ONE aggregated** outage message + consolidated recovery notice;
`lastAlertAt` advances only after a successful Slack send (dedupe
discipline).
- Durable per-slug `{sinceAt,lastAlertAt}` JSON map in `alert_state`
(`getSet`/`putSet`).
- Prod-only gate: `SHOWCASE_ENV ?? RAILWAY_ENVIRONMENT_NAME ===
"production"` + `PROD_D0_MONITOR_ENABLED` kill-switch,
control-plane-only.
## Red-green proof
**No-divergence (the load-bearing test).** A frozen **test-only**
`naiveGone` (`achievedDepth===0` alone, ignoring color/staleness) is the
anti-example. On committed fixtures:
- **RED** — with the real predicate degraded to the naive depth-only
rule, the GREEN assertions fail (the naive rule wrongly labels the
**gray-D0-no-data** and **stale** columns as gone):
```
× GREEN: the real columnGone predicate fires ONLY on the red-D0-fresh
column
Expected false / Received true (stale + gray-D0 columns mislabeled)
Tests 3 failed | 3 passed (6)
```
- **GREEN** — with the real predicate, it fires ONLY on the red-D0-fresh
column and its per-cell inputs equal `buildCellModel`'s own outputs:
```
✓ src/fleet/control-plane/d0-gone-predicate.test.ts (6 tests)
```
**Producer-idle SUSPENDED (F1) proven load-bearing.** Disabling the gate
flips BOTH F1 tests red:
```
× RED (invisible-outage): idle producer + FRESH-gone rows → SUSPENDED (no OPEN)
× RED (false-recovery prevention): open outage, producer pauses, rows spuriously read healthy → NO recovery, HOLD
Tests 2 failed | 13 passed (15)
```
With the gate present: `Tests 15 passed (15)`.
**Full suite (post-fix):**
- Harness `tsc --noEmit`: **exit 0** · production build (`tsc -p
tsconfig.build.json`): **exit 0**
- Shell-dashboard `tsc --noEmit`: **exit 0** (Phase-1 gate — added the
required `isStaleCell`/`observedAtAgeMs` fields to the CellModel test
literal)
- New tests: predicate (6) + monitor behavioral (15) + registration gate
(7) + equivalence (7) = **35 passed**
- All control-plane tests: **382 passed**; orchestrator tests: **133
passed**
## Files
- `d0-gone-predicate.ts` (+test) — pure
`cellGone`/`columnGone`/`columnFreshHealthy` + registry-derived
wired-cell enumeration
- `d0-gone-monitor.ts` (+test, +gate test) — `createD0GoneMonitor`
factory
- `orchestrator.ts` — `internal:prod-d0-gone-monitor` cron registration
(prod-gated)
- `shell-dashboard/…/unified-cell.test.tsx` — CellModel literal fix
(Phase-1 tsc gate)
## Deferred / not in this PR
- **Live staging red-green (§10.8)** — the mandatory producer-paused
staging proof (induce a real comm-error, observe detect+confirm+post,
then pause `FLEET_PRODUCER_CRON` and assert no false recovery). Requires
a live staging harness + test webhook; not run here. **Should be
executed before undraft/rollout.**
- **#5872 field literals**
(`global-error-promotion`/`spec-failed`/`greenCount`) are HARD-GATED on
#5872 merge — the monitor consumes `buildCellModel` output today and
inherits #5872's tightened inputs per-slug automatically; no hard
dependency.
DRAFT — per-slug rollout and undraft/merge are the user's call.
The shell-dashboard Docker build context (repo root) copies scripts,
shared, integrations, shell-docs content, and shell-dashboard — but not
showcase/harness. The dashboard's src/lib/{cell-model,live-status,
staleness,format-ts}.ts re-export barrels forward to
../../../harness/src/shared/cell-model/*, so the isolated next build
failed with Module-not-found (CI build-check (shell-dashboard)). Local
next build passed only because the full monorepo is present.
Copy just the 4-file cell-model fold to the exact relative path the
shims expect, preserving the single-shared-fold invariant (harness
monitor still imports the same canonical copy) without dragging the
whole harness package into the dashboard image.
## Summary
This PR updates thread import documentation and threads navigation
across shell-docs, including a follow-up that makes the supported
CLI-only import journey explicit.
## What changed
- Adds thread import documentation:
- Root generic guide at `/threads-import` for frameworks without a
source-specific importer page.
- Google ADK-specific guide at `/google-adk/threads-import`.
- LangGraph-specific guide at `/langgraph-python/threads-import`, plus
generated LangGraph framework routes through the framework resolver.
- Authored framework wrappers so Mastra, AG2, Agno, Built-in Agent,
CrewAI Flows, DeepAgents, LlamaIndex, Microsoft Agent Framework, and
PydanticAI can surface the generic guide.
- Updates the CLI and import journey:
- Describes the CLI as supporting both cloud-hosted and self-hosted
Enterprise Intelligence.
- States up front that import runs from an app created with the
CopilotKit CLI and Enterprise Intelligence enabled.
- Clarifies that import uses the project already selected for the
current directory.
- Shows the source import commands before the optional `project select`
explanation, while still directing users to change targets before
running the dry run.
- Removes the suggestion that `skills onboard` plus `project select` can
add Enterprise Intelligence to an arbitrary existing app.
- Keeps the CLI import section concise and links to the complete generic
and source-specific guides.
- Clarifies future thread continuity:
- Threads Drawer uses the shared `CopilotChatConfigurationProvider`, so
selecting a thread updates the active `threadId` without separate state
wiring.
- Headless Threads uses `useThreads`; the app stores the selected
`thread.id` and passes it to the chat component as `threadId`.
- ADK and LangGraph guides retain source-specific stable ID mapping
guidance, including the LangGraph UUID caveat.
- Removes source-guide links that rewrote to the current framework page
and created circular navigation.
- Adds and organizes Threads Drawer docs:
- Extracts shared Threads Drawer content into a reusable snippet.
- Adds root and authored-framework wrapper pages.
- Moves Threads Drawer into a new expanded Threads nav grouping
alongside Headless Threads and Import Thread History.
- Renames and reorganizes threads docs:
- Relabels the existing `Threads` guide as `Headless Threads`.
- Removes the root prebuilt-components nav entry for Threads Drawer so
generated frameworks do not show it in two places.
- Updates root, authored, generated, and Built-in Agent nav metadata so
the Threads grouping is consistent.
- Updates nav tests to recognize pages nested inside authored-framework
groups.
## Validation
Run from `showcase/shell-docs`:
- `npm run lint` passes with the existing repository warning set.
- `npm run typecheck` passes.
- `npm run test` passes: 32 files and 167 tests.
- `npm run build` passes and generates 214 static pages; it reports the
existing Turbopack NFT tracing warning.
- Rendered and link-checked locally:
- `/cli`
- `/threads-import`
- `/mastra/threads-import`
- `/google-adk/threads-import`
- `/langgraph-python/threads-import`
- Root and framework-specific Threads Drawer and Headless Threads links
## Redirects
No redirect URLs were added or required:
- `Threads` was relabeled to `Headless Threads`, but the slug remains
`/threads`.
- `CopilotThreadsDrawer` was relabeled to `Threads Drawer`, but the URL
remains `/prebuilt-components/copilot-threads-drawer` and framework
equivalents.
- Threads Drawer moved in navigation, but the route did not move.
- `Import Thread History` is new at `/threads-import` and framework
routes such as `/mastra/threads-import`, `/google-adk/threads-import`,
and `/langgraph-python/threads-import`, so there is no prior URL to
redirect.
- The generated-framework routing change only selects framework-specific
content for the new `threads-import` slug.
- recoveryMessage multi-slug branch now wraps sinceAt in renderSince() so a
corrupt-but-shaped persisted sinceAt renders "unknown", not raw garbage
(matches outage path and single-slug recovery guard).
- summary.get() read failure now logs ERROR with errorId d0-monitor-summary-read
instead of a low-signal WARN (silently blinds the detector, same family as the
other silent-disable guards).
- classifyProducer inflight short-circuit now also requires anyWorkerOnline, so a
stale/orphaned inflight from a dead worker cannot force a blind live scan.
- Extended C6 recovery test to cover the multi-slug arm; split the inflight
predicate test into online/offline-worker cases; fixed C1(ii) rotation comment.
Fixes#5775.
## Problem
After #5099 scoped the streamdown markdown/table styles under
`[data-copilotkit] [data-streamdown="…"]`, the **table action controls**
(copy / download) are still unstyled for hosts that import
`@copilotkit/react-core/v2/styles.css` but don't also ship streamdown's
raw Tailwind utilities. streamdown renders the controls row, per-button
wrappers, trigger buttons, dropdown popovers and menu items with
unprefixed utilities (`flex`, `items-center`, `justify-end`, `gap-1`,
`cursor-pointer`, `p-1`, …) and **no stable `data-streamdown`
attribute**, so CopilotKit's packaged CSS didn't cover them — the
controls rendered as vertically stacked plain icons instead of a
right-aligned row.
## Fix
Add scoped fallback selectors under `[data-copilotkit]
[data-streamdown="table-wrapper"]`, targeting the controls chrome
**structurally** (since it has no `data-streamdown` hook):
- controls row → `> div:first-child:not(:last-child)` (flex,
right-aligned, gap)
- per-button wrapper → `… > div` (relative, positions the popover)
- trigger buttons → `… > div > button` (matches the code-block
copy/download button styling)
- dropdown popover → `… > div > div`
- popover menu items → `… > div > div > button`
The controls row is `table-wrapper`'s first child **only when controls
are enabled**; `:not(:last-child)` leaves a control-less table (whose
single child is the scroll container, already styled by #5099)
untouched.
## Verification
- **Compiles.** Built `globals.css` through the Tailwind v4 CLI — every
`@apply` resolves (e.g. `bg-background` → `var(--background)`,
`shadow-lg` → the shadow vars, `min-w-[120px]` → `min-width:120px`) and
all five rules emit with correct values.
- **Selectors match the real DOM.** A new DOM test
(`streamdown-table-controls.test.tsx`) renders a real `<Streamdown>`
table and asserts the controls row is `table-wrapper`'s first-non-only
child, carries no `data-streamdown` attribute, and contains the trigger
buttons under `> div > button` — i.e. the scoped selectors target real
elements. This also guards against streamdown markup drift.
- **Selector presence** guarded by `streamdown-styles.test.ts`
(whitespace-robust).
- Full `styles/__tests__` suite green; `oxlint`/`oxfmt` clean.
## Note (out of scope, discovered while fixing)
At runtime in streamdown `1.6.11` the `<table>` element is stamped
`data-streamdown="table-wrapper"` (not `"table"`): `MarkdownTable`
passes `data-streamdown="table-wrapper"` as a prop that leaks through
`...rest` onto the `<table>`, overriding the intended `"table"`. So
#5099's `[data-streamdown="table"]` selector currently matches nothing,
and the table also matches the `table-wrapper` rules. The controls fix
here is unaffected (its child-combinator selectors don't match the
table's `thead`/`tbody` children), but the `[data-streamdown="table"]`
selector is worth a separate follow-up / upstream report.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Fixes#5819.
## Problem
An MCP app `ui/message` follow-up is queued (via the module-level MCP
request queue) until the agent is idle. The queued closure ran
`copilotkit.runAgent({ agent })` against the **shared registry agent** —
reading `agent.threadId`/`agent.messages` at *execution* time, not
enqueue time.
CopilotKit keeps a single shared agent per id and mutates its
`threadId`/`messages` on a thread switch (`CopilotChat.tsx` sets
`agent.threadId = resolvedThreadId` and `setMessages(...)`). So if the
host switches threads while a follow-up is queued (agent busy), the
follow-up — and its streamed events — execute against and render into
the **now-foreground thread**. That's the leak in #5819.
### Why this is a regression
This class of bug was previously prevented by the per-thread clone
routing in the activity renderers (#3630); commit `762370a4e5`
**deliberately** reverted that ("remove per-thread agent cloning,
restore single registry agent per id"), re-opening CPK-7155/#2957.
Re-introducing implicit clones would undo that decision.
## Fix — skip stale follow-ups
Capture the thread the follow-up is enqueued for, and at execution
compare it against the shared agent's current thread
(`ɵrunMcpFollowUp`):
- **Same thread** (common case) → run on the shared agent, unchanged
(still streams into the visible chat).
- **Thread changed** → the shared agent has moved on and can no longer
run the follow-up in its originating thread's context, so **drop it**
(with a warning) rather than leak it into the current thread. The MCP
app already received its `ui/message` ack at enqueue time; only the
optional agent turn on an abandoned thread is skipped.
### Why not re-home the run to the original thread?
An earlier revision re-homed the stale run onto an isolated
`registerProxiedAgent` proxy pinned to the origin thread. Review (thanks
@tylerslaton) surfaced two P1s that are inherent to *running* a stale
follow-up, both confirmed:
1. **Tools/context leak.** `buildFrontendTools`/`getContextForAgent`
include *unscoped* entries (the `useFrontendTool`/`useAgentContext`
default) for any agent id, read live — so the background thread-A run
received thread-B's current context and could execute global frontend
handlers against B.
2. **Lost continuity.** With multiple queued follow-ups, a fresh
per-item proxy seeded from an enqueue-time snapshot meant run 2 never
saw run 1's result (the runtime replaces the clone with
`input.messages`/`input.state`, so persistence didn't recover it).
Both dissolve if the stale follow-up simply doesn't run. Skip-stale is
also far smaller and needs no proxy lifecycle, so it doesn't fight the
post-revert "single registry agent" architecture.
## Testing
- **TDD.** The e2e regression test reproduced #5819 on the pre-fix code
(`run` invoked once with the switched-to `threadId`) and passes after.
- **Unit** (`mcp-followup.test.ts`) — `ɵrunMcpFollowUp` runs on the
shared agent when the thread matches (incl. the `undefined`→`"default"`
case) and drops (no run, warns) when it changed.
- **e2e** (`MCPAppsUiMessage.e2e.test.tsx`) — "does not run a queued
ui/message follow-up against a thread the host switched to (#5819)".
- 30/30 across the MCP + follow-up suites; existing ui/message tests
unchanged. `tsc` 0 errors, `oxlint` 0 errors (my files), `oxfmt` clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
C1 (core): select the shown/named outage slugs by re-post-due-ness + rotation
instead of an alphabetical prefix slice, gate the aggregate post on a DUE slug
actually being named, and advance lastAlertAt only for named-and-due slugs. This
stops a wide (>maxSlugs) outage from re-posting every 15m (overflow slugs whose
clock never advanced stayed perpetually "due") and stops a newly-opened overflow
slug from forcing a per-tick re-post; every open slug is now named within a
bounded number of re-posts.
C2: derive outage onset from the gate-failing (non-green) winner rung, not only a
literal red row — a degraded winner no longer strands earliest at NaN and
re-stamps sinceAt to now.
C3: log a loud errorId when readStatusRows breaks on a finite totalPages while the
last page was full (short/inconsistent read) instead of silently truncating.
C4: floor repostMinutes at min 1 in resolveConfig (0 → repostMs 0 → every-tick
re-post).
C5: classifyProducer distinguishes a fresh deploy (workers online, no run history)
as "no-data / not-yet" from a paused "idle" producer; both HOLD (never page
without data) but the fresh case is no longer a misleading permanent SUSPEND.
C6: validate a persisted sinceAt is a parseable ISO before interpolating into the
outage message (renderSince) so a corrupt-but-shaped state blob renders "unknown",
not garbage.
C7: fix the mis-annotated `unsupported` fixture flag and add a symmetric assertion
pinning every naiveMislabels flag == (naiveGone != expectedGone).
Bucket-b: stamp the outage-duration line with evidenceMs (consistent with the
recovery post + lastAlertAt); skip the alert_state write on a pure no-op tick.
Addresses two P1s on #5940: the re-home path (a) injected unscoped frontend
tools/readable context into the background run — `buildFrontendTools`/
`getContextForAgent` include entries with no agentId, so the thread-A run
received thread-B's live context and could execute global frontend handlers
against B — and (b) lost continuity across multiple queued follow-ups (fresh
per-item proxy + enqueue-time snapshot, so run 2 never saw run 1's result).
Both stem from *running* the stale follow-up. Switch to skip-stale: when the
shared agent's threadId no longer matches the thread the follow-up was enqueued
for, drop it (with a warning) rather than run it against the now-foreground
thread. This removes the proxy/registerProxiedAgent machinery entirely and
resolves both P1s by construction. The MCP app still gets its ui/message ack at
enqueue time; only the optional agent turn on an abandoned thread is skipped.
Removes the re-home unit/integration tests; the e2e regression test (no
cross-thread run after a switch) and simplified unit tests cover the behavior.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drives the real CopilotKitCore + RunHandler + ProxiedCopilotRuntimeAgent via
registerProxiedAgent against a mocked transport (mirrors
proxied-runtime-transport.test.ts). Asserts the re-homed run reaches the runtime
addressed to the ORIGINAL threadId (not the foreground one), carries the captured
message, runs on an isolated instance whose events never reach the shared agent,
and unregisters the transient proxy after. Closes the delegate/replay-lifecycle
gap the mocked-host unit tests couldn't reach — in CI, no live runtime needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Queued MCP app ui/message follow-up work executed against whatever thread the
shared registry agent pointed at when the queue drained. If the host switched
threads while the follow-up was queued (agent busy), the run — and its streamed
events — leaked into the now-foreground thread.
Capture the thread context at enqueue and route the follow-up through
ɵrunMcpFollowUp: same thread runs live on the shared agent (unchanged); a
changed thread re-homes the run onto an isolated registerProxiedAgent sibling
pinned to the original thread (own event stream, persists + reconciles on
return); a changed thread on a non-runtime agent drops the follow-up rather
than leaking it.
Regression from 762370a4e5 (revert of per-thread activity-renderer clone
routing, #3630); uses the sanctioned registerProxiedAgent primitive (#4629)
instead of reintroducing implicit clones.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After #5099 the table wrapper/cells are styled via [data-streamdown] selectors,
but the copy/download controls row, button wrappers, trigger buttons, dropdown
popovers and menu items render with raw Tailwind utilities and no stable
data-streamdown attribute — so hosts that import @copilotkit/react-core/v2
styles without shipping streamdown's own utilities saw them unstyled (icons
stacked vertically instead of a right-aligned row).
Add scoped fallback selectors under [data-copilotkit] [data-streamdown=
"table-wrapper"], targeting the controls chrome structurally. The controls row
is the first child ONLY when controls are enabled, so :not(:last-child) leaves a
control-less table (single child = the scroll container) untouched.
Verified against streamdown 1.6.11's actual rendered DOM (a new DOM test guards
that the structure the selectors assume still holds) and by compiling the CSS.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Central structural lever: derive every monitor state from a single per-cell
classifier (classifyCell → gone | healthy | unknown). Treat UNKNOWN (gray /
no-data / stale / amber / comm-error) as UNKNOWN everywhere — never "gone",
never positive-healthy.
- B-F1: recovery/CLOSE requires POSITIVE green cells (cellHealthy: chipColor
green, achievedDepth>=3, fresh), not the mere absence of red. A gone column
decaying to no-data no longer auto-recovers. Fixed the healthyRows test
fixture to emit a genuine green D5/D6 ladder.
- B-onset: derive sinceAt from the folded verdict's contributing ladder rows,
not a raw row.state==="red" re-scan (removed keyBelongsToSlug).
- B-A5gap: guard self-heal/loud-log on "no slug has any wired cell", not
map.size (every integration slug is keyed even with zero wired cells).
- B-env: normalize the prod gate (trim+lowercase, empty-as-unset) so an empty
SHOWCASE_ENV no longer shadows a prod Railway env and a mis-cased/padded
value no longer silently disables the monitor. Added resolveMonitorEnv +
shouldRegister; orchestrator + gate test both use the real predicate.
- B-flap: an already-open outage keeps its hourly re-post even when a later
confirm scan is inconclusive (confirm gates only OPEN and CLOSE).
- B-cadence: advance lastAlertAt only for slugs actually named in the message
(respect maxSlugsInMessage); overflow slugs keep their clock.
- Cheap: no-wired-cells logs once per tick; MAX_SLUGS floors at 1; lastAlertAt
stamps evidenceMs; resolveConfig negative/NaN/empty coverage.
Bucket-(a) fixes, each with a local red-green test:
- A1: replace the substring `:${slug}` onset match with an anchored
exact slug-segment match (`keyBelongsToSlug`) so a prefix-colliding
sibling (`strands` vs `strands-typescript`) no longer mis-attributes
the earlier sibling's red onset. RED: strands' sinceAt was pulled to
the strands-typescript onset; GREEN: each slug gets its own onset.
- A2: recovery/CLOSE is now SYMMETRIC with OPEN — a recovery requires a
second agreeing fresh-healthy read (confirm scan). RED: a single
transient healthy read fired a false "recovered"; GREEN: held until
two reads agree.
- A3: guard an empty/degenerate schedule set — longestPeriodMs 0/NaN
would make idleWindowMs 0 → isProducerLive permanently false → the
monitor SUSPENDS forever and never pages. Falls back to a 45m default
window (DEFAULT_IDLE_WINDOW_MS) and logs at error.
- A4: bound readStatusRows — guard NaN/undefined totalPages (a `page >=
NaN` break never trips) and add a hard MAX_STATUS_PAGES cap so a full
page + bad totalPages cannot infinite-loop/OOM. RED: OOM; GREEN:
terminates at the cap.
- A5: a registry-load failure logs at error with a stable errorId (not a
silent warn-once permanent no-op), and the monitor accepts a loader
thunk so it re-reads registry.json each tick while the wired-cell set
is empty — a transiently-missing file self-heals without a redeploy.
- A6 (verified, no code change): createSlackWebhookTarget already throws
on every non-2xx (4xx/5xx/429/3xx/network-exhausted); added a test
asserting the monitor does NOT delete recovery state when the post
throws.
Bucket-(b): stamp the recovery message with the confirm-scan instant
(evidenceMs) not tick-start; wrap the scheduler tick handler in a
catch+errorId; log the prod env-gate skip at warn with a reason; add a
clarifying comment that the aggregate lastAlertAt reset is intentional
one-message-one-clock cadence; simplify the three dashboard barrel-shim
comments (drop the rot-prone enumerated symbol lists).
Fixes#5801.
## Problem
`CopilotKitProvider` constructs the core **during render** (the `useRef`
init pattern), and the `CopilotKitCore` constructor synchronously starts
the `/info` network request. React can start-and-discard renders
(concurrent rendering / Suspense / StrictMode), and for a *mounting*
fiber `copilotkitRef.current` is `null` again on each discarded attempt
— so a brand-new core is constructed, and fires its own `/info`, per
discarded render. The reporter observed **70–80 `/info` requests on a
single page load** (production build, self-hosted runtime with real
latency).
Root causes (all confirmed on current `main`):
1. Core constructed during render — `CopilotKitProvider.tsx`.
2. Constructor kicks off network I/O — ctor → `setRuntimeUrl` →
`updateRuntimeConnection()` → `fetch(/info)`.
3. `updateRuntimeConnection()` had no in-flight guard.
The dominant multiplier is **cross-instance** (N discarded renders → N
cores → N fetches), so an in-flight guard alone can't fix it — the fetch
must not fire during construction.
## Fix
Separate **construction** (pure) from **connection** (network I/O), as
the issue recommends:
- **core** — new `deferInitialConnection` option: the constructor
records the runtime config (so `copilotkit.runtimeUrl` stays available
*synchronously* to hooks like `use-threads` / the CopilotChat context —
no first-render regression) but does **not** start the `/info` fetch. A
new `connect()` method starts the single connection; it's idempotent
(no-op unless status is `Disconnected`, which `updateRuntimeConnection`
flips to `Connecting` synchronously), so a StrictMode double-invoked
mount effect collapses to one request. `updateRuntimeConnection` also
gains an in-flight guard keyed by `url + requested transport` so
concurrent same-target calls de-dupe (a change to a different target
supersedes).
- **react-core** — the provider constructs with `deferInitialConnection:
true` and calls `copilotkit.connect()` from its existing commit-phase
mount effect. Renders discarded before commit never reach the effect, so
they never fetch. The subscribe effect runs before `connect()`, so the
status-change event is caught (no reliance on the catch-up read for the
common path).
**Backward compatible:** without `deferInitialConnection`, the
constructor still connects — Vue/Angular/vanilla consumers are
unaffected.
## Testing
- **TDD.** New core tests fail on the pre-fix code (constructor fetches)
and pass after.
- **core** (`core-defer-runtime-connection.test.ts`, 7 tests) — the
authoritative proof of the mechanism: deferred construct fires **zero**
`/info`; `runtimeUrl` still exposed synchronously; `connect()` → exactly
one `/info`; `connect()` idempotent across repeated calls; **N orphaned
cores that never `connect()` → 0 fetches, only the committed one
fetches** (directly models the discarded-render bug); in-flight de-dupe;
backward-compat.
- **react-core** (`CopilotKitProvider.deferWiring.test.tsx`) — asserts
the provider constructs with `deferInitialConnection: true` and drives
`connect()` from an effect; **this is the provider-level test that
distinguishes the fix from the bug** (it fails if the deferral wiring is
dropped — verified by reverting).
- **react-core** (`CopilotKitProvider.deferConnection.test.tsx`, 2
tests) — normal-mount regression guards: exactly one `/info` on mount,
and one under `React.StrictMode`.
*Scope note (found via adversarial review):* the raw dozens-of-`/info`
symptom only manifests with React *discarding* in-progress renders and
constructing multiple orphaned cores — Testing Library can't reproduce
that (a single committed mount yields one `/info` either way, and the
ctor's fetch is several microtasks deep so ordering can't distinguish
it). Hence the multi-instance proof lives in the core test above, and
the provider is guarded by the wiring assertion.
- Full suites green: **core 576/576**, **react-core provider 141/141**
(updated one fake-core mock to implement `connect()`). `tsc` 0 errors
(core + react-core); `oxlint` 0 errors; `oxfmt` clean.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Adversarial review found the original provider tests passed even with the fix
removed: a single committed mount yields exactly one /info whether the ctor or
an effect fires it, and the ctor's fetch is several microtasks deep so ordering
can't distinguish it — only the multi-instance (discarded-render) case differs,
which Testing Library can't reproduce.
- Add CopilotKitProvider.deferWiring.test.tsx (mocked core): asserts the provider
constructs with `deferInitialConnection: true` and calls `connect()` from an
effect. This FAILS if the deferral wiring is dropped (verified).
- Keep the two real-core tests as normal-mount regression guards (one /info on
mount; idempotent under StrictMode) and document that the multi-instance proof
lives in core-defer-runtime-connection.test.ts.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CopilotKitProvider constructs the core during React's render phase, and React
can start-and-discard renders (concurrent rendering / Suspense / StrictMode).
Because the constructor fired the `/info` request synchronously, every discarded
-and-recreated core issued its own request — a single page load was observed
firing 70-80 `/info` requests instead of one.
Separate construction (pure) from connection (network I/O):
- core: `deferInitialConnection` lets the constructor record the runtime config
(so `runtimeUrl` stays available synchronously to hooks) WITHOUT starting the
`/info` fetch. `connect()` starts the single connection and is idempotent
(bails unless status is Disconnected), so a double-invoked mount effect
collapses to one request. `updateRuntimeConnection` also gains an in-flight
guard keyed by url+transport so concurrent same-target calls de-dupe.
- react-core: the provider constructs with `deferInitialConnection: true` and
calls `copilotkit.connect()` from its commit-phase mount effect — so renders
discarded before commit never fetch.
Backward compatible: without `deferInitialConnection` the constructor still
connects (Vue/Angular/vanilla unaffected).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a harness-native monitor that pages #oss-alerts when a whole
integration column collapses to red-D0 ("completely gone" / backend
unreachable) in production — the incident class the per-cell alert rules
miss (LGT went fully gone on 2026-07-13 and nothing paged).
Detection runs the dashboard's OWN buildCellModel fold (the shared
cell-model module both the dashboard and the monitor import) over the
same PocketBase status rows and applies a column-gone predicate over the
resulting CellModel fields, so the monitor's verdict equals the DepthChip
the dashboard renders by construction — no parallel re-derivation.
- d0-gone-predicate.ts: pure cellGone/columnGone/columnFreshHealthy over
buildCellModel outputs + registry-derived wired-cell enumeration
(mirrors the dashboard page-stats iteration / determineCellStatus rule).
- d0-gone-monitor.ts: createD0GoneMonitor factory — producer-liveness
SUSPENDED gate (reuses the family-silence inflight-aware /api/runs
reasoning, 3x-longest-period idle window), 60s confirm re-read (never a
re-probe), 15m-detect vs 1h-repost state machine, positive-fresh-healthy
CLOSE gate, ONE aggregated outage / consolidated recovery Slack message,
durable per-slug JSON map in alert_state (getSet/putSet).
- orchestrator.ts: register internal:prod-d0-gone-monitor @ */15, gated on
SHOWCASE_ENV ?? RAILWAY_ENVIRONMENT_NAME === production + kill-switch,
control-plane-only (inside runControlPlane), reusing the oss_alerts
webhook target + shared memoized family summary.
- unified-cell.test.tsx: add the required isStaleCell/observedAtAgeMs
fields to the CellModel test literal (Phase-1 dashboard tsc gate).
Red-green: a frozen test-only naiveGone (achievedDepth===0 alone)
mislabels gray-D0-no-data and stale columns as gone on committed
fixtures (RED); the real predicate fires only on red-D0-fresh and matches
buildCellModel's own outputs (GREEN). Producer-idle SUSPENDED proven
load-bearing (disabling the gate flips both F1 tests red). Plus
confirm-scan blip-rejection, hourly dedup, recovery-clear, failure modes,
and the prod-only/kill-switch registration gate.
Move the pure cell-classification fold cluster (cell-model, live-status,
staleness, format-ts) out of showcase/shell-dashboard/src/lib/ into
showcase/harness/src/shared/cell-model/ so BOTH the dashboard and a new
harness monitor import ONE copy with zero duplication and no behavior change.
The harness builds via tsc -p tsconfig.build.json with rootDir:"src" and
cannot import outside its own src/, so the harness is the correct library
home. The dashboard consumes the cluster via relative path across the package
boundary (established precedent, e.g. d5-cadence-banner.redgreen.test.ts).
- git mv the four files into harness shared/cell-model/; their intra-cluster
relative imports stay valid (they move together, no external coupling).
- Replace the four original shell-dashboard paths with thin export-* barrels
so all ~51 existing dashboard import sites resolve unchanged.
- Repoint commError-contract-drift.test.ts's source-text drift parse at the
new canonical harness location (the barrels carry no derivation body).
- Add cell-model.equivalence.test.ts + committed fixtures + a pre-move
baseline JSON (generated from the original git-HEAD code) proving the move
is byte-identical across a red-D0, gray no-data, stale, mixed, all-green,
and unsupported column.
## Problem
The package and docs routes were renamed from Bots to Channels, but the
docs still presented the old “Bots” and “Bot SDK” product name in
navigation and integration guides. The Channels reference page was
titled only “Channels” instead of “Channels SDK.”
## Why
The stale naming makes the Channels SDK links unclear in launch
communications and creates an inconsistent journey between the package
names, Slack and Teams quickstarts, and API reference.
## Fix
- Title the Channels guide and `/reference/channels` page “Channels
SDK.”
- Update the reference chooser and Slack, Teams, and WhatsApp guides to
use the Channels SDK name.
- Remove the stale wrench icon from the Slack and Teams early-access
sidebar link, with a regression test for the navigation node.
- Preserve the existing `/channels`, `/reference/channels`, `/slack`,
and `/teams` routes and all existing link targets.
- Verify permanent redirects from `/bots` and `/bots/*` to `/channels`,
and from `/reference/bot` and `/reference/bot/*` to
`/reference/channels`, with regression coverage for both exact and
nested routes.
Verification:
- `npm run lint`
- `npm run typecheck`
- `npm test`
- `npm run build`
- Local production smoke checks returned HTTP 200 for `/channels`,
`/reference/channels`, `/slack`, and `/teams`; rendered `/slack`
contains no wrench markup.
## What & why
Permanent fix for the recurring langgraph-typescript (LGT) Railway
outage — most recently **2026-07-13**, when both staging and prod went
dead at D0.
**Root cause.** The LGT backend uses `@langchain/langgraph-api@1.1.17`,
whose `FileSystemPersistence` (`dist/storage/persist.mjs`) serialises
**all** accumulated thread/run/checkpoint state to
`/app/src/agent/.langgraph_api` on a 3-second timer. Under the D6 probe
fan-out (36 parallel probes in ~55s) the dir filled to **201MB in
~90s**, tripping the 200MB size-watchdog in `entrypoint.sh`:
```
[watchdog:size] Size threshold exceeded (201MB >= 200MB) — killing agent PID 5 ... exit code 137
```
Each restart refilled and re-tripped; after enough rapid kill→restart
cycles Railway crash-loop backoff stopped restarting the container → the
whole LGT column dead on **both** staging and prod.
## How this mirrors #5825
PR #5825 fixed the **python** side (commit `b4adfc6296`) by exporting
`LANGGRAPH_DISABLE_FILE_PERSISTENCE=true`, which python's
`langgraph_runtime_inmem` reads at import time to skip its
pickle-flush-to-disk loop. **The TS side (commit `ef103f5f58`) never
disabled persistence** — it only added the boot-purge + size-gated
watchdog, which is exactly the mechanism that failed on 2026-07-13 (it
kills on oversize, but the dir refills faster than backoff tolerates).
This PR does for TS what #5825 did for python: **stop the unbounded disk
writes.** The mechanism differs because `@langchain/langgraph-api` has
**no** `LANGGRAPH_DISABLE_FILE_PERSISTENCE` switch, its persistence
writers are unexported module singletons, and the package `exports` map
blocks a deep import of `persist.mjs` (`ERR_PACKAGE_PATH_NOT_EXPORTED`,
verified). The only disk-write surface is `fs.writeFile`/`fs.mkdir` from
`node:fs/promises`, so:
- **`src/agent/disable-file-persistence.mjs`** (new) — a `node --import`
preload that, gated on `LANGGRAPH_DISABLE_FILE_PERSISTENCE=true`, no-ops
`node:fs/promises` `writeFile`/`mkdir` for `.langgraph_api` paths only.
Uses the CJS handle `require("node:fs").promises` because an ESM
namespace object is sealed (`fsPromises.writeFile = …` throws —
verified); the package's `import * as fs` named exports are backed by
that same object, so the patch is observed by the package. In-memory
state (the actual runtime state) is untouched.
- **`src/agent/package.json`** — `start` now `node --import
./disable-file-persistence.mjs --import tsx liveness.mjs` (patch
installs before any langgraph code touches the fs).
- **`entrypoint.sh`** — `export LANGGRAPH_DISABLE_FILE_PERSISTENCE=true`
before launching the agent (same placement/intent as #5825's python
entrypoint).
Behavior preserved: runs still execute, thread state reads back from the
in-memory checkpointer within a container's lifetime (D4/D5/D6
round-trips unaffected). Only disk persistence is removed — bounded by
process memory, discarded on restart, matching python.
## LOCAL RED-GREEN PROOF (real surface)
Exercised the **real** `@langchain/langgraph-api@1.1.17` `startServer`
with a real compiled `StateGraph`, drove real runs via
`@langchain/langgraph-sdk`, then ran the **real** `entrypoint.sh
--check-size-once` watchdog seam against the produced dir.
### RED (pre-fix) — unbounded growth trips the real watchdog
40 real runs, no preload → `.langgraph_api` = **43,320 KB**
(checkpointer 32MB + ops 12MB). Real `entrypoint.sh` watchdog against it
(threshold 30MB):
```
[watchdog:size] Persistence dir size: 43MB (threshold: 30MB)
[watchdog:size] Size threshold exceeded (43MB >= 30MB) — killing agent PID 78845 (and its npm→node tree) to trigger container restart and boot-purge
watchdog exit code (1 = killed): 1
dummy agent was KILLED by watchdog (RED reproduced)
```
### GREEN (post-fix) — no growth, watchdog never fires, round-trip still
works
Same 40 real runs, with preload + env var → `.langgraph_api` = **0 KB
(no dir)**. Real watchdog, same threshold:
```
[persistence] LANGGRAPH_DISABLE_FILE_PERSISTENCE=true — FileSystemPersistence disk flush disabled (in-memory only)
===RESULT=== runs=40 persistDirSizeKB=0
[watchdog:size] Persistence dir /tmp/lgt-persist-proof/.langgraph_api does not exist — skipping size check
watchdog exit code (0 = no kill): 0
dummy agent STILL ALIVE — watchdog did NOT fire (GREEN)
```
Behavior-preserved round-trip (with fix):
```
ROUND-TRIP: messages returned = 2 | assistant content length = 200000
ROUND-TRIP OK: true
persist dir after round-trip: 0 (no dir)
```
## Tests
`tests/python/test_entrypoint_watchdog.py` gains
`TestFilePersistenceDisabled` (behavioral `bash -x` export-on-boot-path
check + source/wiring guards). Full suite: **22 passed** (18
pre-existing + 4 new). No watchdog-logic regressions.
## Deferred (out of scope)
The size-watchdog restart contract still has no restart cooldown /
crash-loop guard. With this fix the dir never grows so the watchdog
should never trip under probe load, but a restart cooldown would harden
against any future writer. Left out deliberately — root cause (the
unbounded write) is eliminated here.
## ⚠ Before merge
The entrypoint env-var + preload are source-verified and locally
red-green-proven against the real package, but the full container has
not been run on Railway from this branch. Recommend a branch deploy
(boots, serves 200, `.langgraph_api` stays empty under a probe wave)
before merge. Kept as a **draft** pending that validation + maintainer
approval.
---
## Round-2 CR fixes (7 reviewers + pre-audit)
Seven reviewers plus a pre-audit flagged that the round-1
`disable-file-persistence.mjs` was **surface-coupled and unanchored**,
and that the entrypoint watchdog tests asserted a stale mechanism. All
mandatory findings resolved and re-proven against the **real** package.
### Empirical truth about the writer (verified on installed
`@langchain/langgraph-api@1.1.17`)
`FileSystemPersistence` (`dist/storage/persist.mjs`) uses `import * as
fs from "node:fs/promises"` and calls `fs.writeFile(...)` /
`fs.mkdir(...)` via **namespace property access** — which DOES read
through to the CJS `.promises` patch. So the round-1 mechanism was *not*
a silent no-op on this version (the pre-audit's named-import fear does
not apply here). But the patch covered **only `writeFile` + `mkdir`**
with an **unanchored substring**, leaving atomic write-then-rename,
`appendFile`, the `*Sync` variants, `open`+handle, and
`createWriteStream` as **silent bypasses** — the exact way the outage
recurs if the pinned package's writer shape changes on upgrade. There is
**no** env/config knob to disable or redirect persistence (the dir is
hardcoded `path.resolve(cwd, ".langgraph_api", name)`), and
`persist.mjs` is not deep-importable — so a hardened monkeypatch is the
only faithful option.
### What changed
- **Patch every fs write surface** for the persist dir —
`promises.{writeFile,appendFile,mkdir,rename,cp,open}`,
`fs.{writeFileSync,appendFileSync,mkdirSync,renameSync,openSync,createWriteStream}`
— so no future writer shape silently grows `.langgraph_api`.
- **Anchor path matching to the `.langgraph_api` path segment** (not a
bare substring); normalise string/Buffer/URL forms. Fixes over-match
data loss (`/tmp/x.langgraph_api.log`, `.langgraph_api_backup` now write
normally).
- **Version + writer-shape guard that fails loudly at boot** if
`@langchain/langgraph-api` is upgraded off `1.1.17` or switches to named
imports — a silent recurrence on upgrade becomes an obvious boot crash
with an actionable message.
- **mkdir `{recursive:true}` return contract honoured**; **env gate
accepts `1`/`true`/`TRUE`/`yes`/`on`** and **logs on both branches**
(enabled AND "not enabled — persistence ACTIVE").
- **Observability**: one-time "intercepted first persist-dir write via
<surface>" log.
- **Tests**: assert the SHIPPED reaper (`trap _reap_watchdog_children
EXIT` + `$BASHPID` PPID-walk), not a comment-only stale trap string;
replace the mock-of-old-shape behavioral reaper test with one that
drives the **real** extracted `entrypoint.sh` helpers; fix a flaky
single-shot `poll()` with a bounded retry loop; refresh the stale
truncate-era docstring; **add a real-package behavioral test** that
drives the real `FileSystemPersistence` and asserts `.langgraph_api`
stays 0 bytes while an in-memory round-trip still returns an assistant
response.
### Round-2 RED-GREEN — real package, bypass surface
RED — round-1 fix (from committed HEAD), flag ON, an atomic
write-then-rename (`writeFileSync`+`rename`) into `.langgraph_api` still
lands on disk:
```
[persistence] LANGGRAPH_DISABLE_FILE_PERSISTENCE=true — FileSystemPersistence disk flush disabled (in-memory only)
RESULT bytes=4428903 files=.langgraphjs_api.checkpointer.json
DISK_GROWS=YES
```
GREEN — corrected fix, real `FileSystemPersistence.persist()` flush of
2000 threads:
```
[persistence] LANGGRAPH_DISABLE_FILE_PERSISTENCE enabled (value="true") — FileSystemPersistence disk flush disabled across all fs write surfaces (in-memory only)
[persistence] intercepted first persist-dir write via promises.mkdir (target=.../.langgraph_api); further hits silent
RESULT bytes=0 files=(none)
DISK_GROWS=NO
```
GREEN — corrected fix, the SAME bypass surface as RED is now closed:
```
RESULT bytes=0 files=(dir-not-created)
DISK_GROWS=NO
```
GREEN — real in-memory round-trip still works (conversation capability
preserved) while disk stays empty:
```
ASSISTANT_RESPONSE="Hello from LGT"
DISK bytes=0
ROUNDTRIP=OK_AND_NO_DISK
```
Version guard fails loudly on a simulated upgrade (installed 1.2.0 vs
expected 1.1.17):
```
[persistence] version/shape guard failed: @langchain/langgraph-api is 1.2.0, but the fs-write interception was verified against 1.1.17 ... Refusing to boot with an unverified writer shape.
```
`pytest tests/python/test_entrypoint_watchdog.py` → **23 passed** (incl.
the new real-package + real-reaper behavioral tests; they run, not skip,
when the agent's `node_modules` is installed).
---
## Round-3 CR fixes
Confirmation-CR round-2 raised two HIGH findings + three LOWs; all
resolved on this branch (commit `01d3fa7794`).
**HIGH-1 — ESM-namespace-snapshot fragility (silent bypass on
load-order).** The patch mutates `require("node:fs").promises.*`; a
consumer's `import * as fs from "node:fs/promises"` observes it only if
fs/promises links *after* the reassignment. If anything links it first,
the namespace snapshots the ORIGINAL fn and every persist write silently
bypasses the no-op. Added a runtime binding-identity net: after
patching, the preload `await import("node:fs/promises")` and asserts
`ns[name] === patchedFn` for every member it installed (writeFile,
appendFile, mkdir, rename, open, cp); on any mismatch it FAILS BOOT
naming the members, never continues silently. The source-string guard
could not catch this (it checks text, not runtime identity).
**HIGH-2 — behavioral test skipped in CI (green-but-unproven).** The
sole real-package proof `pytest.skip`ped when agent `node_modules` was
absent, and no CI installed them. Now the test reads
`LGT_REQUIRE_BEHAVIORAL`: `=1` (CI) turns a missing runtime into a
FAILURE; unset (local dev) still skips gracefully. The
`python-unit-tests` job in `showcase_validate.yml` now sets up Node 22,
`npm install`s the agent deps, and runs the langgraph-typescript pytest
with `LGT_REQUIRE_BEHAVIORAL=1`, so a green check genuinely proves the
interception fired.
**LOWs:** tolerant writer-shape guard regex (quote/whitespace/alias
agnostic; still trips on a named-import switch); read-only
`open`/`openSync` reject an ENOENT-shaped error for suppressed paths
(write-intent still no-ops) so a reader hits its missing-file branch;
`mkdir {recursive:true}` returns the topmost-created dir per the real fs
contract.
### Round-3 red-green — REAL `@langchain/langgraph-api@1.1.17`
RED (no preload; the real writer's flush lands on disk):
```
REAL-PACKAGE-RESULT {"readBack":"reply-0","bytes":32302}
```
GREEN (preload + `LANGGRAPH_DISABLE_FILE_PERSISTENCE=true`; dir stays
empty, round-trip intact):
```
[persistence] ... disabled across all fs write surfaces (in-memory only); namespace binding-identity verified for 6 members
REAL-PACKAGE-RESULT {"readBack":"reply-0","bytes":0}
```
HIGH-1 guard-fires (fs/promises linked before the patch → boot THROWS,
exit 1):
```
[persistence] FATAL: node:fs/promises namespace binding does NOT reflect the installed patch for: writeFile, appendFile, mkdir, rename, open, cp. ... Refusing to boot. ...
EXIT=1
```
`LGT_REQUIRE_BEHAVIORAL=1 pytest
tests/python/test_entrypoint_watchdog.py` → **24 passed** (adds the
HIGH-1 guard-fires regression test; the real-package behavioral proof
now runs, not skips, and is CI-gated).
Round-3 CR fixes for the LGT persistence-disable preload.
HIGH-1: after installing the fs-write patches, import the node:fs/promises
namespace and assert each patched member is identity-equal to the installed
function; throw (fail boot) naming any mismatched member. Catches the
load-order case where fs/promises was linked before the reassignment and the
namespace snapshotted the original fn (silent bypass -> disk-growth recurrence).
HIGH-2: make the real-package behavioral test non-skippable under
LGT_REQUIRE_BEHAVIORAL=1 (missing runtime fails, not skips), and wire the
python-unit-tests job to set up Node, npm install the agent deps, and run the
langgraph-typescript pytest with that flag so a green check proves interception.
LOW: tolerant writer-shape guard regex (quote/whitespace/alias agnostic; still
trips on a named-import switch); read-only open/openSync reject ENOENT for
suppressed paths (write-intent still no-ops); mkdir recursive returns the
topmost-created dir per the real fs contract.
Adds a HIGH-1 guard-fires regression test.
Merging main into the branch pushed the aimock exact-duplicate count to 301
(main added cross-demo fixture aliases of the runtime-disambiguated-by-fixtureFile
kind, e.g. ag2 headless-complete/gen-ui-headless-complete). Verified the +4 are
NOT in the mastra context — the Partner Refresh's new fixtures introduce zero new
exact dupes. Ratchet the ceiling to match; 828/828 aimock-fixtures tests pass.
Round-2 CR fixes for the LGT file-persistence disable. Empirically verified
against the real @langchain/langgraph-api@1.1.17: the FileSystemPersistence
writer uses namespace fs access (fs.writeFile via import * as fs), so the
CJS .promises patch IS observed — but the round-1 patch covered only
writeFile+mkdir with an unanchored substring, leaving atomic write-then-rename
/ appendFile / *Sync / stream surfaces as silent bypasses.
- Patch every fs write surface (promises + sync + open/openSync/createWriteStream)
for the persist dir, so no future writer shape can silently grow .langgraph_api.
- Anchor path matching to the .langgraph_api path segment (not a substring);
normalise string/Buffer/URL forms — no more over-match data loss.
- Add a version + writer-shape guard that fails loudly at boot if the package
is upgraded or switches to named imports (prevents a silent recurrence).
- Honour mkdir {recursive:true} return contract; accept 1/true/yes/on env
conventions and log on both enabled and not-enabled branches.
- Tests: assert the SHIPPED reaper (_reap_watchdog_children/$BASHPID walk) not
a stale comment-only trap; replace the mock-shape behavioral reaper test with
one that drives the real entrypoint helpers; fix a flaky single-shot poll with
a bounded retry loop; refresh the stale truncate-era docstring; add a
real-package behavioral test proving disk stays empty and in-memory round-trip
still returns an assistant response.
Jordan Ritter