The D5 gen-ui-custom probe branched on integrationSlug via a stale
CHART_INTEGRATIONS allowlist: ~5 slugs got the pie-chart prompt +
assertions, everyone else got an obsolete generate_haiku prompt +
HaikuCard assertion. That violates Showcase iron rule 1 (one shared
probe, no per-slug branching in the test) and no longer matches the
product — all 21 gen-ui-tool-based pages register render_bar_chart +
render_pie_chart, and every committed D6 render-a2ui fixture carries
the chart exchange.
Collapse to the single shared contract: every integration sends
"Show me a pie chart of revenue by category" and runs the SVG/pie-chart
shape + second-leg narration assertions. Remove the CHART_INTEGRATIONS
allowlist / isChartIntegration branch and the now-unused haiku prompt +
HaikuCard fallback (verified no other usage).
No fixture, backend, frontend, npm, or aimock changes.
## Problem
Fleet-wide "empty assistant response": the assistant-message container
mounts but never receives text. #5801 (first released 1.63.0) deferred
the runtime `/info` call to a React effect, widening the "provisional
agent" window; 1.63.2 exposed an `isReady` signal on `useAgent` but
`CopilotChat` never consumed it. A chat submitted during the provisional
window is committed to the provisional agent and then lost when `/info`
swaps in the real agent — the user message and streamed assistant text
disappear, so the assistant bubble renders empty.
This was confirmed with a controlled SSE A/B: stock 1.68.1 does forward
`TEXT_MESSAGE_START → TEXT_MESSAGE_CONTENT → TEXT_MESSAGE_END` (the
runtime is fine — not the in-memory runner / #5837), but the `/info`
agent swap drops the rendered messages; restoring a readiness guard
makes the identical SSE render correctly.
## Fix
`CopilotChat` now consumes `isReady` from `useAgent` and withholds
`onSubmitMessage` until the runtime is ready:
```
- const { agent } = useAgent({ ... });
+ const { agent, isReady } = useAgent({ ... });
...
- onSubmitMessage: onSubmitInput,
+ onSubmitMessage: isReady ? onSubmitInput : undefined,
```
`CopilotChatInput` already derives `canSend` (and its Enter handler)
from `onSubmitMessage`, so withholding it while not-ready (a) disables
the send control and (b) makes Enter a no-op that **preserves** the
composer text — the message can't be committed to the doomed provisional
agent. No runtime/runner changes; no fixture re-recording.
## Red–green proof
New test `CopilotChat.readinessGate.test.tsx` drives the real readiness
race against the real `CopilotChat` submit path: holds the runtime in
Connecting (deferred `/info`), sends during the provisional window, then
resolves `/info` (the real status-change re-render that flips `isReady`)
and asserts the message survives to render an assistant response.
- **RED** (fix reverted): the chat body contains only chrome text — no
user message, no assistant response (the empty-container symptom).
- **GREEN** (fix applied): assistant text renders; passes 3×
consecutively (deterministic).
- Mutation-verified: reverting the fix reproduces RED.
## Verification
- react-core: **1468 tests pass** (0 regressions; 3 pre-existing
web-inspector `localStorage` jsdom-env file errors are unrelated and
present with and without this change).
- react-ui: **69 tests pass**.
- react-core typecheck (`tsc --noEmit`): **0 errors**.
## Follow-up (not in this PR)
The showcase D4 probe driver
(`showcase/harness/src/probes/drivers/d4-chat-roundtrip.ts`) should wait
for the send control to be enabled before pressing Enter (poll
`[data-testid="copilot-send-button"]` `disabled === false` after
typing). Omitted here because it can't be red-green'd without a live
showcase backend. Note this fix makes the follow-up more relevant: with
send gated, a probe that types + Enters during the provisional window
now silently no-ops.
Ref: #5801🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_017t7HsmM31NHNmUQrHF47pm
Two probe false-red guards around the react-core readiness gate:
- Guard the readiness waitForSelector(SEND_ENABLED) against an exhausted
budget BEFORE issuing it. A type that drains the first-token envelope
would otherwise issue a doomed ~1ms readiness wait that Playwright
rejects and the outer catch mis-classifies as a generic level-error.
Below SEND_READY_MIN_BUDGET_MS it now throws ReadinessBudgetExhausted
(errorDesc: delayed-readiness) — an observable, specific red.
- Capture the per-attempt turn-lifecycle baseline AFTER type + the
enabled-send wait and immediately before Enter (for both the initial
attempt and retries). Taking it before the wait let a run completing
DURING the wait land its edge past the snapshot, so the poll mistook it
for THIS submitted turn finishing and false-red'd an empty container.
Adds a delayed-readiness/budget regression and a
counter-advances-during-wait regression; restructures the existing
press-guard test to drain during the readiness wait (via a new
sendEnableDelayMs fake option) so it still targets send-budget-exhausted.
The react-core readiness fix disables the send control and no-ops Enter
while useAgent().isReady is false (the provisional agent /info swaps out).
An early probe Enter during that window is a silent no-op, leaving an empty
assistant response that falsely reds the cell. sendTurn now waits for
[data-testid="copilot-send-button"]:not([disabled]) after typing and before
pressing Enter, so the send lands on the real bound agent. Adds a
driver-ordering test (type -> wait-for-enabled-send -> Enter) with a fake
page that models the readiness gate.
## What & why
Railway **memory** is the showcase project's largest cost line
(~$1.2k/mo; egress was already fixed by the July internal-URL flip).
This PR attacks it two ways:
1. **Cap langgraph integration backend memory.** The three `langgraph-*`
showcase backends run the in-memory `langgraph dev` server and were
drifting to 6–13 GB RSS each. Set `MALLOC_ARENA_MAX=2` +
`MALLOC_TRIM_THRESHOLD_` on the python/fastapi entrypoints (glibc arena
fragmentation on the many-core Railway host) and
`NODE_OPTIONS=--max-old-space-size=1536` scoped to the
langgraph-typescript **agent** process (not the sibling Next.js server)
to cap the V8 heap. All values use `${VAR:-default}` so explicit Railway
overrides win.
2. **Recycle harness workers** after `WORKER_MAX_JOBS` settled jobs
(default 100, per-replica jitter via `WORKER_MAX_JOBS_JITTER`) to
pre-empt slow Chromium/heap growth — the Gunicorn `--max-requests` /
Celery `max-tasks-per-child` pattern. The recycle exits non-zero
(`WORKER_RECYCLE_EXIT_CODE=42`) so Railway's `ON_FAILURE` policy
restarts a fresh container, and routes teardown through the **same
deregister-first `gracefulTeardown` → `drainFleetWorker`** SIGTERM uses
(made idempotent via a once-latch) so a recycling worker deletes its
roster row instead of stranding it ~180s (a transient dashboard
false-red).
## Commits (by area of concern)
- `perf(showcase): cap langgraph integration backend memory`
- `feat(showcase/harness): recycle workers after WORKER_MAX_JOBS to
pre-empt leaks`
- `feat(showcase/harness): deregister workers cleanly on recycle exit`
## Validation
- **207/207** harness tests pass (recycle counter/threshold/jitter,
disabled-when-0, deterministic jitter, idempotent teardown, recycle-exit
deregister — all red-green + mutation-checked).
- oxfmt clean, oxlint 0 errors, typecheck clean (the only `tsc` errors
are 4 pre-existing `frontend-matrix.test.ts` errors from a missing
generated catalog, unrelated to this diff).
- Reviewed via a 4-round code-review loop (converged: 0 mandatory
findings; Procedure-3 promotion audit `PROMOTE_TO_A: 0`).
## ⚠️ Deploy / rollout notes
- **`WORKER_MAX_JOBS` defaults to 100 (recycle ON).** Set
`WORKER_MAX_JOBS=0` to disable if you want a staged rollout. Recommend
enabling on **staging first**.
- **The langgraph memory savings are not yet empirically measured.**
`MALLOC_ARENA_MAX` reclaims glibc fragmentation on the many-core host
but the GB delta must be confirmed by a **staging A/B** (deploy, compare
RSS with/without) before quoting a dollar figure. The change itself is
safe/standard; only the magnitude is unverified.
- Precondition confirmed: harness-workers restart policy is `ON_FAILURE`
(in-repo docs) — the non-zero recycle exit will restart. Recommend
pinning `restartPolicyType: ON_FAILURE` explicitly in IaC as
defense-in-depth (follow-up).
## Follow-up (out of scope — pre-existing, surfaced by review)
The review surfaced a large pre-existing debt backlog in touched files,
none load-bearing on this PR's subject (all deferred). Notable
candidates for a separate PR:
- **Recycle-teardown hardening:** bound `pool.shutdown` in the recycle
path with a timeout so a hard-wedged Chromium can't delay the voluntary
exit (currently backstopped by the `/health` 503 healthcheck); make the
recycle-vs-SIGTERM exit-code deterministic; `latchOnce` sync-throw
guard.
- **Monitor write-only-red:** `onPidSaturation` writes RED with no clear
path (permanent false-red once tripped).
- **Entrypoint robustness:** `wait -n` under `set -e` making the
exit-code/diagnostic block dead code; `kill` of wrapper PIDs orphaning
node children; TS watchdog probing the `:8124` sidecar not `:8123`.
- **Control-plane parity:** `runControlPlane` missing the S3 backup cron
/ `hydrateProbeLastRuns` / `probe_runs` metric wiring that `boot()` has.
- Assorted test-hygiene (afterEach `doUnmock` leaks, a few vacuous
assertions, fixed-timer flakes).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Wire the recycle exit through the same deregister-first graceful teardown as
SIGTERM (a shared, latch-once gracefulTeardown -> drainFleetWorker), so a
recycling worker deletes its roster row instead of stranding it ~180s for
fleet-health to reclaim (a transient false-red on every recycle). The latch
makes the shared teardown run at most once even if a recycle races a SIGTERM.
A worker stops claiming after WORKER_MAX_JOBS settled jobs (default 100,
staggered per replica via WORKER_MAX_JOBS_JITTER so replicas do not recycle in
lockstep), then exits non-zero (WORKER_RECYCLE_EXIT_CODE=42) so Railway restarts
a fresh container. Mirrors Gunicorn --max-requests / Celery max-tasks-per-child;
0/unset disables. Caps slow Chromium/heap growth on long-lived harness workers.
## What
Adds a **LangSmith Platform** deploy guide to the CopilotKit docs,
modeled after the existing AWS AgentCore deploy page.
It's a self-contained, agent-side guide: deploy a **LangGraph** or
**Google ADK** agent to the LangSmith Platform, then point the
CopilotKit Runtime at it. LangSmith has no frontend-hosting offering, so
the guide covers only the agent side plus wiring the runtime.
## Pages
- **Canonical:** `deploy/langsmith.mdx` — renders in the Overview →
Deploy sidebar.
- **Per-framework wrappers** (thin, like AgentCore):
- `integrations/langgraph/deploy-langsmith.mdx` → `<Content
framework="langgraph" .../>`
- `integrations/adk/deploy-langsmith.mdx` → `<Content framework="adk"
.../>`
- Both registered in their `meta.json` under a new `---Deploy---`
section.
- **Shared walkthrough snippet:**
`snippets/integrations/langsmith/index.mdx` — single source of truth for
all three pages; framework-aware via the `Content` loader scope. Reuses
the existing `langgraph-platform-deployment-tabs` snippet for the "grab
your deployment URL" step.
## Structure (mirrors agentcore.mdx)
Intro → How it works (ASCII flow `Browser → CopilotKit Runtime →
LangSmith deployment → your agent`) → What you get → Quickstart
`<Steps>` inside a `<TailoredContent>` (deploy-new vs already-deployed)
→ `<Callout>`s for the API key/URL and the LangSmith docs authority →
framework tabs (LangGraph / Google ADK) for the deployable-app step →
Troubleshooting `<Accordions>` → What's next `<Cards>`.
## Registry glue
- Generalized the `Content` MDX component to accept an optional
`partial` prop (defaults to the AgentCore partial; existing AgentCore
wrappers unchanged).
- Registered a `LangGraphPlatformDeploymentTabs` stub so the existing
deployment-tabs snippet is reusable.
## Verification
- Commands/flags (`uv tool install langgraph-cli`, `langgraph new
--template new-langgraph-project-python`, `langgraph deploy
--name/--deployment-type dedicated`, deployment API URL) verified
against the live LangChain quickstart.
- ADK path (`pip install "deployments-wrap-sdk[google-adk]"`,
`saf_sdk.adk` `wrap()` + `LangsmithSessionService`, `langgraph.json`
export) verified against the live [Deploy Google ADK
agents](https://docs.langchain.com/langsmith/deploy-google-adk) guide.
- Runtime wiring (`LangGraphAgent` from `@copilotkit/runtime/langgraph`
with `deploymentUrl` / `graphId` / `langsmithApiKey`) matches the repo's
LangGraph quickstart.
- `oxfmt` (format) clean, `oxlint` exits 0, `tsc` clean; registry /
search-href / link-rewrite tests pass. (Pre-existing failures in this
worktree from an uninstalled `react-icons` and unfetched git-LFS assets
are unrelated.)
## Note (small extra)
The LangGraph `deploy-agentcore.mdx` wrapper already existed but was
orphaned (not in any `meta.json`). The new `---Deploy---` section
surfaces it alongside `deploy-langsmith`, matching how AWS Strands
already exposes it.
Ticket: GROW-540
worker-loop's claim gate had exactly one conditional --
`if (budget.available <= 0)` -- and `available` is free Playwright
browser-CONTEXT slots (maxContexts - liveContextCount), nothing else. A
worker whose container has leaked PIDs to the cgroup ceiling therefore
still advertises full capacity and keeps winning claims it cannot
possibly run: the driver cannot fork, so each job burns its entire
600000ms lease and lands as an abort. Measured 2026-08-10: workers at
pids=1000/1000 with 758-761 zombies, 320 `timeout after 600000ms` rows.
Decline the claim above 0.90 of pids.max. Deliberately far above the
control-plane's 0.75 saturation ALARM: the two thresholds do different
jobs. 0.75 pages an operator while the worker is still perfectly able to
run jobs; gating dispatch that early would convert a warning into ~36h
of withheld fleet capacity. 0.90 leaves 100 free PIDs at the prod
ceiling -- a chosen reserve, not a derived one (the per-job PID cost of
a chromium context tree is not measured), hence the env override. The
ordering that matters is that the alarm always fires before capacity is
withdrawn.
BLAST RADIUS -- reuses the SAME decline-and-idle path as the existing
no-budget branch, which is what makes an all-workers-saturated fleet
safe: the job is never claimed, so it is never dropped and never
requeued; it simply stays `pending`. The loop then sleeps a full
pollIntervalMs, so a fully-saturated fleet idles at the poll cadence
instead of spinning. The queue stalls VISIBLY -- pending rows pile up,
the rising-edge warn says why, and the control-plane's
system:worker-pid-saturation alarm fired at 0.75 before this gate
engaged at 0.90. A stalled, alarmed queue is recoverable by redeploy;
claim-fail-repeat silently burned the whole cadence.
The gate is INERT when the cgroup gauges are unreadable (pidUsageRatio
-> null: off-Linux, every macOS dev box), so local workers claim exactly
as before.
Prod harness-workers replicas reach the platform-fixed pids.max=1000
ceiling roughly every 7 days and NOTHING alarmed. Measured 2026-08-10
20:09-20:10Z: pids=1000/1000 zombies=758, 1000/1000 zombies=761,
837/1000 zombies=744, with 320 `timeout after 600000ms` abort rows and
an e2e-smoke run 42 minutes into a 15-minute cadence with 3 jobs still
pending. The documented pool-unrecoverable alarm did not fire: it only
trips when the self-heal breaker gives up, which a PID-starved but
still-heartbeating worker never reaches.
The signal was already on the wire and read by nobody. The worker's
~75s heartbeat writes capacity_pids_current / capacity_pids_max onto
its `workers` row (worker/registration.ts), and fleet-health lists that
whole roster every 15s — its own comment said "the capacity gauges are
ignored here". Nothing in non-test source compared them.
Alarm control-plane-side rather than worker-side, because status-writer
documents `fleet-cp` as the only authoritative fleet writer ("workers
never write status directly"), and because fleet-health already holds
the roster read this needs.
- fleet-health raises a rising-edge alarm at pids.current/pids.max >=
0.75 (~36h of lead time at the observed leak rate), evaluated BEFORE
the online/stale branch since the failing worker is fully ONLINE.
- Latched per worker with a 0.65 hysteresis clear: the monitor ticks
every 15s, so an unlatched alarm would page ~8600 times across the
lead-time window. Construction fails loud on a ratio pair with no gap.
- Unmeasured gauges (null off-Linux, unbounded pids.max) never alarm --
pidUsageRatio returns null and null is never read as zero.
- Routed to a system:worker-pid-saturation status row plus #oss-alerts
via SLACK_WEBHOOK_OSS_ALERTS, the target family-silence and the
D0-gone monitor already post to. Deliberately NOT
SLACK_WEBHOOK_BROWSER_POOL_UNRECOVERABLE, which appears nowhere in
this repo outside its own definition and is unset everywhere -- an
alarm nobody receives is the gap being closed.
harness-workers leaked one cgroup PID slot per orphaned chromium grandchild.
Node as PID 1 only waitpid()s processes it spawned, so every browser crash
stranded ~5 <defunct> renderers permanently. Prod climbed to
pids.current=1000/1000 with zombieCount=757 over 6d22h uptime, after which no
browser could launch and ~295 d6 cells went abort fleet-wide.
Install tini in the existing playwright apt layer and run it as PID 1 via
exec-form ENTRYPOINT. No -g, so signal delivery to node is unchanged and
orchestrator.ts's SIGTERM drain still runs; tini propagates the child's exit
status so a crash still exits non-zero and Railway still restarts.
Root package.json pins patchedDependencies (eventsource@3.0.7 ->
patches/eventsource@3.0.7.patch, added in #6334). The harness Dockerfile
selectively copies root manifests but omitted patches/, so the frozen
install hashes the patch file, ENOENTs, and exits 254. Broke the
showcase-harness build-check for any PR rebased onto post-#6334 main.
Copy patches/ before the install. Verified: image builds clean locally.
The `completeOnMount` gate added in d70d48a561 named the `mcp-app-iframe`
testid, which only Angular's `copilot-mcp-apps-widget` declared. react-core
and vue build the sandbox iframe imperatively with no testid, so every
React/Vue integration timed the turn out at 30s with
`reason=surface-missing` and never reached `assertIframePresent` — whose
`iframe[sandbox]` fallback would have passed. D5 + D6 `mcp-apps` went red on
all 18 integrations that support the feature (first_failure_at 2026-07-28
23:03Z) while the demos rendered correctly by hand.
Fixed on both sides of the contract:
- react-core and vue now set `data-testid="mcp-app-iframe"` and a `title` on
the host-created iframe, matching Angular. Pinned by a test in each package.
- `completeOnMount` accepts CSS `selectors` alongside `testIds`, so the probe
settles on the same cascade its module doc and assertion already use
(`[data-testid="mcp-app-iframe"], iframe[sandbox]`). A comma-joined entry is
one conjunctive surface whose branches `querySelectorAll` unions, so the
delta/`minNewMounts` semantics are unchanged and `testIds` is now sugar for
the equivalent selector. This half greens the fleet on the next sweep
without waiting for a package release, since the integrations pin
@copilotkit/react-core 1.61.2.
A spec naming no surface now throws instead of burning the turn budget and
reporting a misleading `surface-missing`.
Verified against live staging: after clicking the pill, the old gate matched
0 elements and the cascade matched 1 (the sandboxed iframe was there all
along). Also recorded in showcase/GOTCHAS.md.
## What
Brings the **ms-agent-dotnet** (Microsoft Agent Framework .NET) showcase
integration from D5 to **D6**, using **langgraph-python** as the
north-star reference.
### 1. Frontend parity with langgraph-python
Restores near-identical frontends where ms-agent-dotnet had drifted,
while **preserving the load-bearing .NET adaptations** (per the showcase
iron rules — differences belong in fixtures/minimal backend, not the
shared frontend):
- Root shell: `globals.css` (Tailwind `@theme` block + brand green),
manifest-driven index `page.tsx`, `layout.tsx`, new `middleware.ts`
(`x-pathname`), `tsconfig` include.
- `declarative-gen-ui` subtree restored (fixes divergent pill testids
the shared probe asserts).
- Doc-snippet `@region` markers, import-style normalization, `subagents`
revert, stale-file cleanup, `auth` inspector flag.
- **Kept** (load-bearing, not reverted): `parse-json-result` 3-layer
unwrap, multimodal legacy-shim, tool-based `hitl` (MAF has no
`interrupt()`), `agent-config` `properties=`.
### 2. shared-state-streaming → per-token (removed from
`not_supported_features`)
`write_document`'s `document` arg now streams into `state.document`
per-token via a `createSharedStateStreamingAgent` route shim (mirrors
the proven `createGenUiAgent` bridge, with a partial-JSON string
decoder), since the .NET AG-UI host has no `predict_state_config`.
### 3. a2ui-recovery cell (new)
First MS-Agent-Framework implementation of the A2UI
validate→retry→`a2ui_recovery_exhausted` recovery loop. Because the MAF
AG-UI adapter can't emit the custom `ACTIVITY_SNAPSHOT{status:"failed"}`
the exhausted card needs, it's a **raw-SSE `MapPost` endpoint**
(`RecoveryAgent.cs`) — the same adapter-bypass pattern already shipped
for `/multimodal`. Adds the demo frontend, API route, deterministic
aimock fixture (heal seq0-invalid→seq1-valid; exhaust always-invalid),
and a unique per-slug `PROMPTS` entry in the shared probe.
### 4. threadid-frontend-tool-roundtrip demo (parity)
Added for demo-set parity (reuses the `frontend_tools` passthrough; not
a D6-scored feature, mirroring the reference).
`gen-ui-interrupt` / `interrupt-headless` remain honestly quarantined
(upstream `@copilotkit/react-core` `useInterrupt` resume-path bug — not
a backend gap).
## Verification
- Code was authored in parallel worktree-isolated slots, each
cross-verified against the reference + the shared probe contracts; the
a2ui-recovery fixture was cross-checked against
`RecoveryAgent.ValidateComponents`.
- Local D6 harness: the image builds and the stack + probes run, but
**full local green was blocked by Windows-only harness friction**
(`core.symlinks=false` breaks `stage_shared`'s `[ -L ]` materialization;
`--direct` doesn't context-scope the `x-aimock-context` header so
context-keyed a2ui fixtures miss). These are environmental, not code
issues. **Relying on CI's Linux harness (real symlinks + fleet worker)
for authoritative D6.**
## Follow-up (not in this PR)
- `stage_shared()` should also materialize Windows symlink-as-file
entries (detect a regular file whose content is a relative path), so
forced local rebuilds work on `core.symlinks=false` checkouts.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The fail-safe polarity flip's widest case was frozen by nothing. A stripped-
`signal` red on a `health:`/`agent:` row flips a whole column green->red
(achieved 6->0, isRegression true) because those keys are integration-scoped,
but `red-signal-unknown` was only applied by `positionSweep` (D3-D6), and
`liveness-fresh-red-d1/-d2` use `row()`'s default `signal: null` -- which is
not `undefined`, so they classified FAIL_FRESH under the old polarity too.
Measured: restoring `!redSignalKnown -> NO_DATA` killed 4 golden-master entries,
all D3-D6, and left the entire D1/D2 leg byte-identical.
Underneath that sat a structural blind spot: every fixture variant maps ONE
uniform row shape over ALL keys of a rung, so no family anywhere contained two
DIFFERING red rows and no family contained a green row with an absent `signal`
key. A uniform set has nothing to aggregate, so three explicitly load-bearing
family folds were invisible at golden-master level -- `Math.max`->`Math.min` on
`maxNonInfraRedFailCount`, `allRedSoftClass` `every`->`any`, and re-widening the
provenance flag from red-row to family scope each left all 56 entries identical.
Adds 5 fixtures (56 -> 61 baseline entries):
- `liveness-red-signal-unknown-d1` / `-d2` -- a red D1/D2 row with `signal:
undefined` over a green D3-D6 ladder. Both now fail under a polarity
re-widening, joining the 4 existing entries.
- `hetero-d4-red-failcount-max` -- D4 {chat red fc1, tools red fc3}, straddling
D4_FIRST_STRIKE_THRESHOLD. MAX(1,3)=3 -> FAIL_FRESH/red; MIN(1,3)=1 ->
FIRST_STRIKE_FRESH/amber, i.e. a fresh sibling re-arming tolerance for an
already-confirmed failure, which the MAX doc explicitly forbids.
- `hetero-d4-green-stripped-infra-red` -- D4 {chat green signal-omitted, tools
red driver-error}, the production shape `classifyRung` cites verbatim and the
first fixture with a green row whose `signal` key is absent. Red-row scoping
keeps this gray; a family-scoped flag paints it red.
- `hetero-starter-mixed-soft-hard` -- starter {transport-error (soft),
smoke-failed (hard), rest green}, both below SOFT_MISS_TOLERANCE_THRESHOLD so
the soft-class quantifier is the only deciding factor.
Each mutation is now caught by exactly one fixture while its pre-existing
sibling stays blind, so the teeth come from the new shapes rather than
incidental coupling. Baseline entries were derived by hand from the source
first and inserted additively (140 insertions, 0 deletions) -- no wholesale
regeneration, so no result is self-approving.
Also closes the D1/D2 gap in the dashboard equivalence suite: the `fail-safe
polarity` fixtures seeded no `health:`/`agent:`/D5/D6 rows, so those rungs
contributed ABSENT -- also gray -- and the gray assertions passed via the
no-data path while naming the infra path. A `greenScaffold` helper makes every
rung but the one under test green-fresh, discriminating pill assertions prove
the gray sits over a present red pill, and a new test pins the whole-column
D1/D2 stripped-red case api-vs-browser (equal chip, achieved 0, regression
true).
Fixtures and baseline only -- no classification logic, `combine.ts`, or
coherence invariant touched.
INV7 (`assertChipStripCoherent`) fired on a legitimate engine shape: with no
`e2e:` row, `scanWorst` breaks at the ABSENT D3 rung (invariant I1) and never
consults D4, so a fresh product-red `chat:<slug>` renders a gray chip over a red
D4 pill. INV7 demanded infra evidence for a red the chip provably never read.
Round-1 finding a4(iii), unfixed by the round-1 INV7 fix; re-raised as a8.
Adds a third, narrow carve-out beside the U8 all-stale one: the gray is excused
only when the ladder walk stopped for want of any observation STRICTLY BELOW the
shallowest red pill. `ladderGapDepth` tests exactly that ("no row for any of the
rung's keys"), which is provably equivalent to `classifyRung`'s ABSENT for a
D3/D4/D5 rung — deliberately not the strip's `status === null`, which also
covers the `anyExpectedMissing` NO_DATA collapse (a rung that does have an
observation and does not stop the walk). `combine.ts` is untouched: the
gap-break's own masking is a separate change needing a golden-master re-freeze.
The allowance is one-directional and canaried. A gap ABOVE the red excuses
nothing, and when the gap-break is eventually fixed the new case's chip stops
being gray, so its precondition assertion fails loudly rather than letting the
carve-out excuse a shape the engine no longer produces.
Also removes INV7's unreachable starter branch (b24). A starter cell has no
depth strip, so INV7's precondition is structurally unsatisfiable there; the
branch implied coverage that does not exist. It is now a loud throw, and a new
test pins the structural reason and fails if the starter axis gains a strip.
New cases: the gap shape (green, plus an in-suite mutation proof that the
allowance is what passes it), a gap above the red, a gapless gray-over-red, and
an UNPERTURBED engine model INV7 still rejects (an infra-red D4 grays a
product-red D6 — D6 is outside `scanWorst`'s scan). That last one corrects the
file's claim that the engine no longer produces a gray-over-red incoherence.
Exhaustive sweep over 295,245 rung combinations: 25,504 INV7 failures without
the allowance, 1,456 with. Every remaining one has its gap at or above the
shallowest red pill, i.e. none is gap-induced; they are two other pre-existing
engine masking mechanisms and INV7 keeps its teeth on them.
This reverts the scratch commit. Its only purpose was to prove the two new jobs
fail on a real defect in product source — see the PR body for the RED run ID.
The branch is back to a strictly additive CI change with no product diff.
TEMPORARY — reverted in the next commit. A job that has never failed is not yet
a gate, so this proves both new jobs actually catch a defect in product source.
harness: D4_STALE_AFTER_MS 1h -> 10h in shared/cell-model/staleness.ts
(a frozen-green D4 row would credit D4 for ten hours instead of one)
dashboard: sortOrder mastra 8 -> 88 in lib/sort-order.ts
Both reproduce locally before pushing: harness 3 failed / 177 passed in
src/shared/cell-model, dashboard 4 failed / 1326 passed.
Wiring the harness unit suite into CI (next commit) found it already RED on
main with exactly three failures. A job that is red on arrival gets ignored or
disabled, so this adds a CI-only vitest config that excludes those three files
— and a ratchet that stops the exclusion from becoming permanent.
vitest.quarantine.json the three entries, each with a date, a reason,
and an explicit exit criterion
vitest.ci.config.ts `test` minus the quarantined files
scripts/quarantine-ratchet.ts re-runs each quarantined file and requires it
to STILL FAIL
The ratchet is what makes the exclusion defensible: the moment somebody fixes a
quarantined test, it goes red and names the entry to delete, so an entry can
never outlive the failure it excuses. It also fails on an entry that points at a
missing file, on an entry whose filter matches more than one file, and on a
malformed manifest.
No test is deleted and no assertion is changed. The three failures are a stale
hard-coded cell count, a drift test stranded by a completed single-source
refactor, and a genuine `D5_REPRESENTATIVES` gap for `browser-use-smoke` that
belongs to the D5 owner.
`test:ci` and `test:quarantine-ratchet` are declared in the harness package.json
`nx.targets` block (project-local, not a workspace default) with `cache: false`:
the nx `test` named-input covers `src/**` but not the quarantine manifest, so a
cached result could survive an edit to the exclusion list.
Local `pnpm test` still uses the unfiltered config — a developer should see the
quarantined failures.
Every numeric claim below is re-derived from production PocketBase
(showcase-pocketbase-production.up.railway.app) or from repo config on
2026-07-24, not copied from a neighbouring comment.
- Row/page counts: 2455 rows / 5 pages -> 3082 rows (~3100) / 7 pages,
with the exact source query recorded so the number can be refreshed.
The INITIAL_FANOUT_BATCH no-over-fetch note now says that it holds
because the CURRENT page count is odd, not as an algorithm property.
- Byte estimate: ~1.29 MB -> ~1.6 MB. The retired figure applied the
(correct) 371 KB -> 113 KB per-500-row basis to the OLD 2455-row
collection. Measured all 7 pages both ways: 2.34 MB full vs 0.70 MB
projected = 1.64 MB of signal.
- Worst-case re-delivery window: ~29 min -> ~60 min. staleness.ts and
harness/config/probes/*.yml schedule e2e-demos, starter_smoke and
d6-all-pills-e2e HOURLY (10/40 * * * *); aimock-wiring is 6-hourly and
the drift probes are weekly/monthly.
- Restored the coverage-gap acknowledgement this PR deleted, rewritten
for what is true today: the supplemental union is neither minimal nor
complete. It is green-blind outside clause 1 (2646 rows carry a signal
but sit outside it; the 731 green per-cell rows under the aggregate
dimensions are excluded by BOTH clauses). Currently latent (0
uncovered rows carry __fleetCommError) but structurally open, and it
is the hole that let the mis-scoped signal-provenance misreport ship
green.
- Anchored the 94%/6% product-vs-infra split with its query, date and
partitioning predicate (336 vs 21 of 357 red rows), and flagged it as
a snapshot the polarity argument does not depend on. Softened the
unreproducible "~20% of the matrix" to the derivable bounds.
- Version-qualified the STATUS_LIST_FIELDS omission-vs-null claim to the
pinned PocketBase 0.22.21 and recorded the upgrade hazard: PB >=0.23's
PublicExport isVisible/GetHidden path lets a field-level hidden flag
omit signal independently of our projection, reintroducing the
ambiguity the provenance flag exists to avoid.
INTEGRATION RECONCILE (folded in when this was cherry-picked on top of
the signalKnown-scope fix). Three conflicts in useLiveStatus.ts, all
row/page-count prose, resolved to the measured 3082-row / 7-page anchor
and to this commit's corrected INITIAL_FANOUT_BATCH note (the "page 7
ends its own wave" claim is only true at an ODD page count, which the
superseded text asserted as an algorithm property). Additionally, every
doc reference to `RawRung.signalKnown` — a field the scope fix DELETED —
was rewritten to name the surviving red-row-scoped
`FamilyFold.redSignalKnown`, so no comment describes a flag that no
longer exists: live-status.ts STATUS_LIST_FIELDS doc (3 sites),
useLiveStatus.ts coverage-gap note, api-matrix-equivalence.test.ts (2
sites).
Comments only - no logic, no reformatting, no reordering.
INV7 (added with the cold-load gray-masks-red fix) was unsound in two ways
that only failed to bite by fixture accident:
1. WRONG QUANTIFICATION. Its doc says "every CONTRIBUTING red row", but it
filtered over every row in `live.values()` — a whole-matrix map. Any red
row belonging to a different column or feature could fail the invariant
for a cell it contributes nothing to. Now quantified over the exact
keyspace `buildCellModel` collects (`contributingKeys`).
2. MISSING isStaleCell EXEMPTION. `buildCellModel` deliberately force-grays
an all-stale cell on every path (agent, starter, null-feature), folding
ANY stale colour — red included — to the "re-sweep pending" gray. That
gray is a recency claim about the cell, not an infra attribution for the
red pill, so an all-stale cell containing a red row legitimately renders
gray over a red pill. INV7 failed it. Exempted, with the reasoning inline.
Both holes were latent: the current `stale-*` fixture variants are
green/degraded, and the matrix has no cross-cell rows, so nothing hit them —
the next fixture added would have tripped a false positive.
Also adds an anti-vacuity guard (the narrowed quantification must still find
at least one contributing red row whenever the strip reads red, so a future
drift between `contributingKeys` and the engine's keyspace fails loudly
instead of silently passing everything), and fixes the synthetic contribution
helper `c()`, which set `freshestAgeMs: 0` for `ABSENT` — a shape
`classifyRung` never produces (it returns `null`: an absent rung has no
observation to age).
New tests cover both false positives, a control, and four teeth cases
proving INV7 still fails a genuine non-stale gray-chip-over-red-pill.
`classifyRung`'s U7 gray branch reads the `signal` blobs of a family's RED rows
and nothing else, but the precondition qualifying what it read — "was `signal`
actually delivered, or projected away?" — was `raw.signalKnown`, a boolean AND
over EVERY present row of the family. A predicate about red-row evidence was
answerable "no" by a row the branch never looks at.
That broke exactly the case the precondition was written to protect. The
dashboard's supplemental cold-load fetch restores `signal` for `state != "green"`
only (by design — the classifier needs it solely in the red branch), so in a
MIXED-state family the red rows arrive WITH attribution and the green siblings
arrive WITHOUT it. The family-wide flag therefore read `false` precisely when it
should have read `true`: D4 = green `chat` + infra-red `tools` skipped the gray
branch, fell through first-strike (its only red is infra, so
`maxNonInfraRedFailCount` is null), and landed on FAIL_FRESH — the browser
painting RED a cell that `/api/matrix`, which always has the full `signal`,
reports as gray. That is the §11.4 api-vs-render drift the read-model exists to
forbid, and it made INFRA_RED_FRESH effectively unreachable in the browser for
every multi-row family: D4, and any multi-pill D5/D6 (whose per-cell
`d5:<slug>/<pill>` keys the supplemental filter's clause-1 `key !~ "%/%"`
excludes). Only single-key D3 was unaffected — there the family's one row IS the
red row and the two scopes coincide, which is why every fixture missed it.
The fix derives the flag where the rows are, at the scope that consumes it:
`foldFamily` now reports `redSignalKnown` over the RED rows only, alongside the
red-scoped `hasNonInfraRed` / `maxNonInfraRedFailCount` / `allRedSoftClass` it
already computed. `RawRung.signalKnown` and its `gatherRows` plumbing are removed
rather than left dead — a family-scoped flag named `signalKnown` sitting next to
a red-scoped consumer is the trap that produced this bug. `anyExpectedMissing`
stays family-scoped, correctly: a missing sub-key IS a family property and it
gates a family-scoped verdict.
`redSignalKnown` is today IMPLIED by `!hasNonInfraRed`, since
`signalHasInfraErrorClass(undefined)` is false and a stripped red row forces
`hasNonInfraRed` on its own. It is kept explicit anyway — this is the
masks-real-red guard and it must not rest on a coincidental property of a helper
two modules away — and pinned directly on the fold so the scope cannot regress
silently.
Coverage. The previous tests named `signalKnown: false` but were insensitive to
it: `raw.signalKnown &&` could be deleted outright and all 181 harness
cell-model tests plus 58 dashboard equivalence tests stayed green, because every
fixture stripped `signal` from ALL rows of the target rung and never from a green
sibling. The new tests construct the mixed-state family that ships the bug, on
the real derivation surface (`buildCellModel`, not a hand-passed flag): D4 chip,
multi-key D5 chip, and the D6 `d6Effective` badge — D6 is the soft-parity top so
its chip is amber either way, but its badge drifts red-vs-null. Each asserts
browser == server == `/api/matrix`. `api-matrix-equivalence`'s
`coldLoadChip === serverChip` guard is extended off single-key D3 to the D4
family. Five `foldFamily` tests pin the flag's scope directly, and the fail-safe
polarity keeps its negative control: a red row whose OWN `signal` was stripped
still renders RED.
The dashboard's bulk initial fetch projects the heavy `signal` blob away
(`STATUS_LIST_FIELDS`) for a real payload win — measured at ~70% of the
response, 371 KB → 113 KB per 500-row page. A supplemental fetch then restored
`signal` for the rows that are read at render time, but it only ever covered
the comm-error AGGREGATE rows: dimensions `d6`/`d4`/`e2e-demos`/
`d5-single-pill-e2e`, narrowed by `key !~ "%/%"`.
`classifyRung` also reads `signal`, to tell an INFRA red from a PRODUCT red,
and the rows it needs were excluded twice over: they are mostly dimension
`d5`/`e2e`/`health` (absent from FLEET_COMM_AGGREGATE_DIMENSIONS entirely) AND
they are per-cell `<dim>:<slug>/<featureId>` keys that `key !~ "%/%"` filters
out. So on every cold load those rungs arrived with no attribution at all, and
self-corrected only when the probe's next sweep rewrote that specific row and
the SSE delta redelivered it with `signal` — up to a full sweep interval
(~29 min observed, ~14 min mean), again on every reload.
The supplemental filter becomes a UNION: the existing comm-error clause OR
`state != "green"`. Scoping by STATE is what keeps it cheap — `classifyRung`
consults `signal` only in its red branch, so green rows never need it, and in
production the non-green rows are ~360 of ~3100 (~430 KB, about a third of what
shipping `signal` on every row would cost) with no schema change. `!= "green"`
rather than an explicit red/degraded list also catches an out-of-vocabulary
state, which `rankOfState` ranks WORST — the rows that matter most can never
fall outside the fetch.
Widening the EXISTING fetch rather than adding a second one reuses its
freshness guard (`supplementalRowIsOlder`) and chimera-avoidance merge verbatim,
keeps first paint to one extra request, and keeps the fail-loud retry posture.
A dimension-scoped hook still narrows the comm-error clause to the matched
literal, and drops it entirely when the scope sits outside the aggregate set —
but the non-green clause always applies, so such a scope no longer skips the
supplemental fetch altogether. Its test is updated accordingly.
Also fixes two test doubles that identified the supplemental request by
`filter` containing `key !~`. That marker now disappears for an out-of-set
dimension scope, so both would silently misclassify a supplemental request as a
BULK page and corrupt the fan-out instrumentation. They key off the absence of
a `fields` projection instead — the property that structurally distinguishes
the two requests and the whole reason this one exists.
`classifyRung` opened its red branch with `if (!raw.signalKnown) return
NO_DATA` — so whenever the dashboard's bulk fetch had projected the `signal`
blob away, a genuinely-failing rung was classified "no data" and painted the
muted gray "nothing to see here" chip, while the depth strip beside it
correctly rendered `1P ✗`.
That polarity is backwards twice over:
- `signalKnown === false` never meant "known to have no signal". PocketBase
omits the `signal` key ONLY under a `fields=` projection; a row that
genuinely has no signal arrives as `null`, and `null !== undefined`. So the
flag means "this row came from a projected fetch" — a PENDING attribution,
the absence of evidence about WHY the rung failed. It was never evidence
that the failure was infra.
- Measured against production, ~94% of reds are product-class and only ~6%
are infra-class. Graying every unattributed red lost 94 real failures to
suppress 6 false alarms — and a masked red is never investigated, while a
false alarm is looked at once and closed.
Graying a red now requires POSITIVE infra evidence: `hasNonInfraRed` is false
only when every contributing red row's blob actually carries an
INFRA_ERROR_CLASSES attribution, and `signalKnown` is retained as an explicit
second precondition so a missing blob can never be read as an infra
attribution. The residual cost is over-reporting, which is the direction a
health dashboard has to fail in.
Tests: the harness `§7 I5` case and the dashboard's api/render equivalence
case both ASSERTED the buggy gray as intended behaviour — which is why CI
never caught this. Both are inverted, with the reasoning recorded inline.
Four golden-master fixtures (`pos-d{3,4,5,6}-red-signal-unknown`) are
re-frozen; the regenerated baseline diff touches those 4 of 56 and nothing
else.
Adds coherence invariant INV7: a gray chip sitting above a red depth pill
requires positive infra evidence on every red row. INV1-INV6 only relate
chipColor to the other CHIP-side outputs and never inspect the d3/d4/d5/d6
pills, so the engine could return one object saying both `d5.status === "red"`
and `chipColor === "gray"` with every invariant intact. INV7 fails on the old
polarity and closes the gap in the only safe direction — by fixing the chip
upward, never by muting the strip.
## What & why
Brings the **built-in-agent** showcase integration to parity with the
**LangGraph-Python (LGP)** reference: byte-identical demo frontends + a
named-agent backend registry (BuiltInAgent + TanStack AI), so every demo
climbs the D0–D6 ladder against the LGP gold standard.
## Changes (4 commits)
1. **P0 pattern** — `agentic-chat` byte-identical + named agent
(`agentic_chat`); proven D6-green locally. Fixed `gpt-4o` → `gpt-5.5` in
the shared factory.
2. **Frontend migration (all demos)** — every LGP `src/app/demos/*`
copied verbatim (`diff -r` clean), plus shared `components/ui` (25
shadcn primitives) + `lib/utils`, byte-identical. Added the 5 demos BIA
lacked (`a2ui-recovery`, `declarative-hashbrown`,
`declarative-json-render`, `shared-state-read`,
`threadid-frontend-tool-roundtrip`); added the frontend deps the copied
UI needs (radix-ui, cmdk, embla-carousel-react, react-markdown,
remark-gfm, yaml, …).
3. **Named-agent backend** — `/api/copilotkit` registers 22 named agents
(generic all-tools, fixture-driven; reasoning trio via the reasoning
adapter). 8 dedicated routes re-keyed `default` → LGP agent id;
`mcp-apps` also serves `headless-complete`; `ogui` serves both
open-gen-ui ids; `byoc-*` routes renamed to `declarative-*`; new
`a2ui-recovery` + `beautiful-chat` routes reuse existing agents. Dropped
BIA-only extras (`byoc-*`, `hitl-in-chat-booking`).
4. **Reconcile** — `manifest.yaml` (37 features / 40 demos;
`generate-registry` + `validate-parity` pass) + `PARITY_NOTES.md`.
> **Note on "byte-identical":** frontends are verbatim LGP **modulo
BIA's `consistent-type-imports` ESLint rule** (type imports split into
`import type {}`) — required for a green lint/PR, semantically & DOM
identical.
## D6 status (local sweep)
- **~33/40 demos GREEN** on the first sweep — byte-identical frontends +
named agents + existing fixtures work broadly.
- **4 RED locally are an aimock-infra issue, not this integration:** the
deployed `ghcr.io/copilotkit/aimock:latest` has no
`context`/`--context-field` fixture scoping, so cross-slug `userMessage`
collisions let earlier-loaded (`ag2`/`d4`) fixtures shadow BIA's own.
BIA's fixtures are **correct** and converge under a context-aware aimock
(present on aimock `origin/main`). Affects
`tool-rendering-custom-catchall`, `headless-complete`, `gen-ui-agent`,
`frontend-tools`. **Action for infra: redeploy aimock from a
context-aware build.** Details in `PARITY_NOTES.md`.
- **2 downstream-host RED (kept as features, informational — mirrors
LGP):** `declarative-gen-ui` (A2UI renderer host) and `mcp-apps` (MCP
iframe host).
- Quarantined NSF unchanged: `gen-ui-interrupt`, `interrupt-headless`,
`shared-state-streaming`, reasoning-trio.
D6 is informational/weekly (not a merge gate); these are documented for
parity tracking.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Implement the render/validate/retry recovery loop with a2ui_recovery_exhausted
hard-fail as a raw-SSE endpoint (RecoveryAgent.cs, mounted in Program.cs), add
the demo frontend + API route, the deterministic aimock fixture, and the unique
per-slug PROMPTS entry in the shared d5-a2ui-recovery probe.
Mark