The override docstring claimed it reproduces the upstream aggregate_tool_calls body byte-for-byte; it is functionally equivalent with two cosmetic diffs (Optional type hint, list comprehension). Reword to match reality.
(cherry picked from commit 5e91118b3a463dcefcd228f6343e472db6081c0f)
The _note asserted injectA2UITool:false (unchanged) and that flipping to true
would blank-render, and that generate_a2ui returns an a2ui_operations container
for the middleware to forward. Both are now false: this PR set injectA2UITool:true,
generate_a2ui returns raw planner args, and the surface mounts from a streamed
render_a2ui tool-call (START/ARGS/END) the agent re-emits, which the middleware
watches under injectA2UITool:true. Prose-only; no match keys or payloads changed.
(cherry picked from commit 5679b001580615f2e7d988d8c7063994076ace29)
The page header still described the runtime as configured with
`injectA2UITool: false` and the backend agent as owning `generate_a2ui`,
mirroring beautiful-chat. This PR inverted the route to
`injectA2UITool: true`, so the comment was stale. Rewrite the step-3 block
to describe the current mechanism: `injectA2UITool: true` populates the A2UI
middleware's watched-names set, which mounts the surface from a STREAMED
`render_a2ui` tool-call the agent re-emits via its `aggregate_tool_calls`
override in a2ui_dynamic.py. Drops the stale generate_a2ui framing and
matches the accurate header in route.ts.
(cherry picked from commit 407d755638ebe28418f1f8ce2c558f202995284e)
The InfoRow renderer was the only catalog component missing a
data-testid. The top-account pill's _design_a2ui_surface leg emits
7 InfoRow facts + a PieChart, and the d5-gen-ui-declarative probe
asserts declarative-info-row (minCount 1) as top-account's
distinguishing testid. Because the renderer never painted that
testid, the completeOnMount gate (whose surfaceTestIds include
declarative-info-row) never observed a mount, the turn never
completed, and the run reported reason=surface-missing. Every other
pill passed because its distinguishing testid (metric / status-badge /
data-table) was already emitted.
The A2UI middleware mounts the surface from a STREAMED render-tool CALL whose
name is in its watched set, not from a TOOL_CALL_RESULT. The prior approach
emitted a TOOL_CALL_RESULT carrying an a2ui_operations container, which the
middleware never inspects, so the surface stayed unmounted (surface-missing).
- route.ts: set a2ui.injectA2UITool: true so the middleware watches render_a2ui.
- a2ui_dynamic.py: generate_a2ui now returns the planner's render_a2ui args
(surfaceId/catalogId/components/data) as JSON; the workflow override
(_A2UIRenderToolCallWorkflow) parses each backend tool result and re-emits it
as a streamed render_a2ui tool-CALL (TOOL_CALL_START name=render_a2ui →
chunked TOOL_CALL_ARGS carrying the components JSON → TOOL_CALL_END), mirroring
how google-adk drives the middleware. These events are already in the upstream
AG_UI_EVENTS allow-list the SSE router streams against.
Backend still produces the components (no stub). Inner planner tool stays
_design_a2ui_surface, so the d6 fixture needs no re-keying.
Three stale, pre-layer-(b) prose sites still claimed drain() ABANDONS the
in-flight run and the loop skips queue.report: the drainFleetWorker JSDoc, the
runWorker stop() inline comment, AND the case-worker SIGTERM-handler block.
Layer (b) changed this — on SIGTERM the worker stops claiming, lets its
in-flight cell FINISH within the 90s grace, and REPORTS its real terminal
result via the runAbort/abortedWithoutResult discriminator; only a run that
OVERRUNS the grace is abandoned -> reclaimed by layer (a). Preserved the still-
accurate drainReason=shutdown red-side-emit soft-wind-down + deregister-vs-crash
distinction. Also refreshed the WHY-THIS-ORDER preamble (90s finish-and-report
grace, 180s platform stop / layer-c drainingSeconds). Comments only — no
behavior change.
Layer (c) deploy-rollover config for the harness-workers fleet — pure
Railway config, no custom rolling-restart code. Declares overlapSeconds=45
(capacity floor: old deployment serves until new workers register+claim, so
no staleness dip) and drainingSeconds=180 (SIGTERM->SIGKILL window >=
PLATFORM_STOP_GRACE_MS so the layer-(b) 3s+90s composed worker-drain finishes)
for both prod and staging in the railway-envs SSOT, regenerates the JSON
snapshot, and extends the harness-workers drift gate so CI fails if either
field drifts between SSOT and snapshot. Documents both knobs, their rationale
(incl. that Railway's draining default is 0s = immediate SIGKILL), the
composition with drain layers a+b, and how to apply them (GraphQL
serviceInstanceUpdate / dashboard) in RAILWAY.md.
CR P2 round on layer-(b) graceful worker drain.
P2-A (contested TOCTOU, reconciled empirically): a run that resolves with a
valid result in the same flush the grace setTimeout fires is REPORTED, not
spuriously abandoned. runAbort.abort() lives only in stop()'s Promise.race
TIMEOUT leg, which loses the race once `done` is resolvable — so a finished
run never trips the abortedWithoutResult discriminator. Reviewer crb6 was
correct; the TOCTOU is a non-bug, so no logic changed. Added a deterministic
fake-timer regression pin forcing both the run-completion timer and the grace
timer due in one advance (verified to BITE under an over-aggressive
abandon mutation), plus a clarifying comment at the discriminator.
P2-B: corrected the requestDrain() JSDoc — post-B2 the report-skip,
heartbeat-stop, and driver-cancel key on the grace-expiry signal
runAbort.signal, not stopAbort.signal (which now only stops claiming).
B5 — set the production drain grace T to 90s (was 6s). The grace stopped being
a bare TEARDOWN budget and is now the FINISH-AND-REPORT budget stop() waits for
an in-flight run to finish-and-report (layer b) before firing runAbort -> abandon
-> layer-(a) reclaim. T must BOUND a typical cell-job (so a normal in-flight job
finishes within grace) yet stay SHORTER than the platform SIGTERM->SIGKILL window
with headroom. Sized from in-repo cell-job signal: a single-service cell-job runs
~15s (light e2e-deep) up to ~200s (heavy d6-all-pills under contention); the
per-job lease ceiling is 300s. 90s covers the bulk and stays well under the lease
so a finishing job's lease never lapses. The tail (a job that cannot finish in
grace) falls back to layer (a) — grace is deliberately FINITE. 90s is a defensible
default; B-VAL confirms/retunes from staging p95. Env-overridable via
WORKER_DRAIN_GRACE_MS.
Introduce PLATFORM_STOP_GRACE_MS (180s) documenting the C3 requirement: layer-(c)
must set Railway terminationGracePeriodSeconds = 180 so the composed serial budget
DRAIN_DEREGISTER_TIMEOUT_MS (3s) + DEFAULT_WORKER_DRAIN_GRACE_MS (90s) fits with
>=30s headroom for the health-server-close + pool-shutdown remainder. The
composed-budget test now pins the relation 3s + grace < PLATFORM_STOP_GRACE_MS
(was hardcoded < 10s) and the concrete numbers.
Add a behavioral fake-clock test of the composed-budget invariant: a run that
finishes WITHIN T is reported (finish-and-report); a run still running AT
grace-expiry fires runAbort -> abandons WITHOUT a usable result -> not reported
(layer-a reclaim backstop). The raise also broke the existing wedged-driver
default-grace test (advancing a 90s fake span re-armed the 0ms-yielding heartbeat
into a runaway cascade); fixed by giving it a clock-honoring sleep so the
heartbeat stays quiet across the window and only the grace timer is crossed.
Post-B2/B3 a run can finish-and-report after a graceful drain (drain no
longer hard-cancels the run; only grace-expiry runAbort does, which is
ctx.abortSignal). The d6 red-suppression already AND-s drainReason with
ctx.abortSignal.aborted and errorClass==="abort", which makes it
aborted-only by construction — a finished run whose legitimate red is a
genuine failure (errorClass!="abort") is NOT suppressed. This adds a
pinning test for that aborted-only contract (a finished goto-error red
under drainReason=shutdown with an un-fired abort signal must still be
reported). Verified RED via a temporary mutation to the over-broad
"drainReason alone" suppression form, GREEN on the real code.
B3 (layer-b graceful drain): the heartbeat-abort previously keyed on the
DRAIN signal, stopping lease renewal at drain-start. With B2 decoupling drain
from run-abort, a still-finishing job now runs past drain-start until grace
expiry, so killing renewal at drain-start lets the lease lapse mid-finish and
the layer-(a) reaper could reclaim the row out from under the worker
(double-run / report-after-reclaim). Gate the heartbeat-abort on the
grace-expiry signal (runAbortSignal) instead: a finishing job keeps renewing
until it reports terminal; a genuinely-abandoned job (runAbort fired at
grace-expiry) still stops renewing so its lease lapses and the reaper reclaims
it. runAbortSignal defaults to drainSignal, preserving direct-call unit-test
semantics.
Decouple 'stop claiming new jobs' from 'abort the in-flight run': drain()
no longer fires the run's abortSignal. The run's hard cancel is now a
separate grace-expiry signal (runAbort), so a run seconds from done
finishes within grace and is reported instead of being abandoned to the
sweeper. The abandon break is now conditional on abortedWithoutResult
(runAbort fired at grace-expiry → no usable result); a finished run falls
through to the report path. A wedged run that overruns grace is still cut
(runAbort.abort() in stop()'s timeout leg) and abandons → layer (a)
reclaim is the backstop.
Reconcile the drain tests that encoded the old abort-on-drain coupling to
the new contract (drain no longer fires ctx.abortSignal; the grace-expiry
abort fires at grace, exercised via a short WORKER_DRAIN_GRACE_MS).
## What
Removes the `@copilotkit/bot-store-redis` and
`@copilotkit/bot-store-postgres` adapters, added in #5613. The pluggable
`StateStore` interface and the in-memory `MemoryStore` default stay;
durable backends can be reintroduced as a follow-up when there's a
concrete need.
## Why
Both adapter packages were merged but **never published to npm**, so
there are no consumers — removal is a clean delete with no migration or
deprecation cycle. Trimming the surface keeps the bot persistence story
to one well-tested in-memory default plus a documented "bring your own
`StateStore`" path.
## Changes
- Delete `packages/bot-store-redis` and `packages/bot-store-postgres`.
- Revert the `bot` release scope and the release drift-guard test to
`bot + bot-ui` (count 2).
- Strip the Redis dependency, `demo:restart` script, restart demo,
`docker-compose.yml`, and `REDIS_URL` env from `examples/slack`.
- Rewrite the bot persistence/transcripts docs around "MemoryStore
default + implement the `StateStore` interface yourself for durability"
(unrelated enterprise/Helm Redis/Postgres docs untouched).
## Verification
- `nx run @copilotkit/bot:build` — pass
- `nx run @copilotkit/bot:test` — 121/121 pass
- Release drift guard — pass at count 2
- Repo-wide grep for
`bot-store-redis|bot-store-postgres|createRedisStore|createPostgresStore`
— zero matches
## What
Replaces the queue reaper's delete-wins behavior for stale long-expired
orphaned in-flight rows with **reclaim-wins-until-cap**: an orphaned row
is re-queued (not deleted) up to a bounded number of CONSECUTIVE
re-orphans before final claim-delete. This is layer (a) of the
worker-reclamation + graceful-rollover redesign — it makes a worker
bounce mid-column non-lossy (the column's in-flight work is reclaimed by
surviving workers instead of dropped).
## How
- **Reclaim-wins-until-cap** in `runSweepExpired`
(`showcase/harness/src/fleet/queue-client.ts`): stale long-expired
orphaned in-flight rows are re-queued; the prior delete-wins carve-out
is inverted.
- **Consecutive-orphan budget** (`consecutive_orphan_count`, migration
`1779990400`): the reclaim cap (`MAX_RECLAIM_ATTEMPTS=3`) is scoped to
*consecutive* sweeper re-orphans — incremented only on the sweeper
re-queue path, **reset to 0 on terminal `done`/`failed`**. Peer-worker
expired-lease *steals* bump the lifetime `reclaim_count` (retained as a
dashboard diagnostic) but do NOT consume the reclaim budget.
- **Stale-age re-anchoring** (`requeued_at ?? created`, migration
`1779990300`): a reclaimed row's staleness clock restarts so it isn't
immediately re-expired.
## Review
7-agent CR + 7-agent confirmation round → **0 P0 / 0 P1**. Highlights:
- **Red-green GENUINE (empirically re-verified):** P1-A — a long-lived
job with lifetime `reclaim_count=3` but `consecutive_orphan_count=0` is
RE-QUEUED on a fresh orphan (pre-fix: wrongly deleted); P1-B — low
boundary at `MAX-1=2` re-queues, mutation-killed. Both non-tautological
(the test fake bumps `consecutive_orphan_count` only on reclaim, never
on steal; a dedicated pin test confirms steals don't consume budget).
- **No regression / terminates / fails-safe:** 248/248 tests, tsc +
oxlint clean; reclaim loop terminates at 3 consecutive orphans →
claim-delete; queue cannot grow unbounded; SweepResult metrics +
`reclaim_count` dashboard consumers (run-view `jobs.reclaimed`,
family-silence) unbroken.
- **Migration `1779990400`** additive, nullable, default-0;
pre-migration rows evaluate to 0 → safe re-queue path.
## Follow-on (not in this PR)
- Layer (b) graceful worker drain and layer (c) rolling restart are the
remaining phases of the redesign — see the Notion proposal:
https://app.notion.com/p/38b3aa381852817bacf5c9cda1f11cc0🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Tightens the **D4/BE chat-roundtrip probe gate** from `text.length > 0`
to a **provenance check**. A turn now only counts green if the assistant
text came from the `[data-testid="copilot-assistant-message"]` container
— not from the `<body>` fallback scrape (which picks up static page
chrome).
## Why
This is the gate weakness that **masked the BIA prod outage**: a dead
agent (RUN_STARTED→RUN_FINISHED with zero `TEXT_MESSAGE`) left the page
chrome on screen, the `<body>` scrape returned non-empty text, and the
old `text.length > 0` gate reported **green** while the agent was
actually broken. The new gate threads a `fromAssistantContainer` flag
(set true only on the testid-container read path) so a body-scrape-only
result fails (`!fromAssistantContainer` → red). The same guard is
applied at L4 (tool-rendering).
## Review
**7-agent CR → 0 P0 / 0 P1.** Highlights:
- **Red-green genuine** (empirical): old gate false-passes BIA (1/52
RED); post-fix 52/52. 3-way guard (green /
empty-via-unchanged-length-guard / BIA-via-provenance).
- **Blast radius: no false-fail risk** — all 20 integrations'
`agentic-chat` + `tool-rendering` demos use the shared v2 `CopilotChat`
which mounts the testid unconditionally; aimock `d4/<slug>/chat.json`
fixtures are text-bearing for all 20. Verdict: **ship as a blocking
gate, no soak.**
- **No lost coverage** — the `<body>`-scrape fallback *code* is retained
(diagnostics); only the gate *semantics* narrowed. No D4-probed cell
relied on the fallback to pass.
- **No cross-family regression** — change is confined to
`d4-chat-roundtrip.ts`; other probe drivers untouched; 169/0 tests pass;
tsc clean.
## Notes / follow-ups (non-blocking, P2)
- A *future* probed demo using a `CopilotChatAssistantMessage`
children-render override (which doesn't emit the testid) would
false-fail — worth a one-line note near the fallback comment if that
pattern is ever added to a D4 route.
Part of the 2026-06-26 incident remediation (companion to #5733); this
gate would have surfaced the BIA failure that the dashboard masked.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Bundles the four code fixes from the 2026-06-26 showcase prod incident
remediation. (Prod config reconciles — fleet image promotes, worker
scaling, CVDIAG keys — were applied directly to Railway and are not
code; the post-incident debugging lesson shipped separately in #5727.)
## Incident context (why these exist)
Prod's coverage dashboard cascaded to a wall of red. Root cause was
**staleness, not a feature break**: the harness worker pool was starved
(a deploy-bounce + under-provisioning), so probe sweeps couldn't
complete within the staleness windows → cells aged out → rendered
red/BE✗ even though the apps were fine. Compounded by a `llamaindex`
container crash (per-request log flood tripping Railway's log cap) and a
2-day-stale prod image fleet. See #5727 for the debugging lesson, and
the Notion proposal below for the durable worker-reclamation redesign.
## The four fixes
1. **Dashboard stale-while-revalidate**
(`shell-dashboard/depth-chip.tsx`) — a passing (green) cell keeps its
color + a non-destructive `⟳` refreshing affordance during a re-probe
instead of flapping to grey; never-run stays grey; failure/regression
keeps its color with no spinner; staleness bound preserved. + a11y
(`role=status`, flag-gated regression label, `motion-reduce`).
2. **Family-silence grace window** (`harness/fleet/control-plane/*` +
dashboard banner) — a normal harness deploy's post-bounce worker drain
no longer fires a false "worker family silent" banner/alert (suppressed
within a `2×period` bounce grace keyed on the freshest worker
registration), while genuine silence beyond the window still fires on
both the Slack-alert and banner paths. No silent failure (PB-down →
grace disabled → real outages still alert).
3. **llamaindex per-request log gate**
(`integrations/llamaindex/.../route.ts`) — gates the chatty
`[copilotkit/route]` per-request logs behind `SHOWCASE_ROUTE_DEBUG`
(default off) so they can't flood Railway's log cap and kill the
container. Error logging untouched.
4. **SSOT worker-provisioning** (`scripts/railway-envs.ts` + drift gate)
— brings harness-workers replica provisioning under SSOT so prod/staging
can't silently drift. Models the **effective** field
(`multiRegionConfig.<region>.numReplicas`) +
`BROWSER_POOL_MAX_CONTEXTS`, declares current reconciled reality
(prod=staging=6 replicas, 40 contexts), with a CI drift-detection test.
## Review
7-agent CR round + 7-agent confirmation round → converged to **0 P0 / 0
P1**. Two post-CR fixes (SSOT effective-field model after discovering
`multiRegionConfig` is the live knob; re-anchoring a grace-edge test to
genuinely pin the 2× boundary) each re-confirmed clean. Suites:
shell-dashboard 92 + harness 63 + scripts 12 = **167 passed, 0 failed**.
No new tsc errors (pre-existing only).
## Notes / follow-ups (non-blocking)
- llamaindex log-gate is logging-only and untested (acceptable); the
chatty pattern exists in ~16 integrations incl. the gold standard — a
**fleet-wide log-flood gate** is a worthwhile follow-up.
- `freshestBounceMs` has two equivalent impls (harness `parseIso` /
dashboard `Date.parse`) — flagged for future lockstep.
- The D4/BE probe gate weakness (asserts `text.length>0`, which masked
the BIA outage) is being addressed in a **separate** PR
(`fix/d4-gate-tighten`).
## Refs
- Post-incident debugging lesson: #5727
- Worker reclamation + graceful-rollover redesign (Notion proposal):
https://app.notion.com/p/38b3aa381852817bacf5c9cda1f11cc0🤖 Generated with [Claude Code](https://claude.com/claude-code)
P1-A (cap semantics): the MAX_RECLAIM_ATTEMPTS cap was keyed on
`reclaim_count`, a LIFETIME tally bumped by BOTH the sweeper re-queue
path AND the peer-worker expired-lease steal (claim CAS). A long-lived
job that accrues benign peer steals could exhaust its 3-budget and then
get claim-DELETED on its first real orphan rather than re-queued.
Fix: introduce a dedicated `consecutive_orphan_count` column (migration
1779990400) that is bumped ONLY by the sweeper re-queue path in the
fleet-claim release CAS, and reset to 0 on every terminal done|failed
release. The peer-worker steal (claim CAS wasExpiredSteal branch) does
NOT touch this counter. The reaper's cap check now uses
`consecutive_orphan_count` instead of `reclaim_count`. `reclaim_count`
is left intact as the lifetime dashboard diagnostic (jobs.reclaimed).
P1-B (low boundary): adds a test at consecutive_orphan_count = MAX-1
(= 2) asserting the row is RE-QUEUED, not deleted. The off-by-one
mutation `>= MAX` -> `>= MAX-1` causes this test to go RED.
Test-fake honesty: `makeReclaimClaim`'s claimJob now explicitly models
the steal-bump on `reclaim_count` (matching the real hook) while
intentionally NOT bumping `consecutive_orphan_count`, and adds an
explicit pin test confirming steals do not consume the reclaim budget.
JSDoc on MAX_RECLAIM_ATTEMPTS updated to describe the correct semantics:
consecutive re-orphans scoped by sweeper re-queue, reset on terminal.
Red-green proof:
- P1-A RED: revert cap to reclaim_count → "P1-A CAP SCOPE" fails with
`expect(undefined).toBeDefined()` (job deleted instead of re-queued)
- P1-A GREEN: consecutive_orphan_count cap → test passes (re-queued)
- P1-B RED: mutate `>= MAX` to `>= MAX-1` → low-boundary test fails
- P1-B GREEN: revert mutation → low-boundary test passes
Suite: 145 queue-client + 103 producer = 248 total, all green.
Layer (a) of the worker reclamation+rollover redesign: make a worker bounce
non-lossy. The reaper's long-expired carve-out (G1d) used to claim-DELETE an
orphaned in-flight (claimed/running) row whose lease expired beyond its
family's stale window AND whose created-age was past that window —
`reclaimed=0, expiredPending++` — silently dropping work an abrupt bounce
(SIGKILL past grace / OOM / crash) left mid-flight.
Invert it to RECLAIM-WINS-UNTIL-CAP: re-queue the orphan to pending (it
re-runs; idempotent probes make at-least-once safe) until its durable
`reclaim_count` reaches MAX_RECLAIM_ATTEMPTS (3), only then claim-deleting a
row that keeps re-orphaning so a poison job cannot loop forever.
The carve-out existed to dodge an honesty bind: re-queueing a `created`-stale
row emitted a "back in flight" gray the next sweep falsified by claim-deleting
it off the renewal-immune `created` age. Dissolve the bind with a new
`requeued_at` column (migration 1779990300) the release CAS stamps on every
pending re-queue; both stale phases now age off `staleAgeAnchorMs`
(`requeued_at ?? created`), so a reclaimed row is genuinely young again and the
next sweep does not delete it. `reclaim_count` (migration 1779990200) is reused
as the attempt counter — no second tally to drift.
Red-green proven on the real reaper (queue-client.test.ts): a stale-aged
long-expired orphan below the cap goes RED (deleted, reclaimed=0,
expiredPending=1) on delete-wins and GREEN (re-queued, reclaimed=1,
expiredPending=0, requeued_at stamped) on reclaim-wins; plus an attempt-cap
deletion test and a next-sweep no-falsification test.
The L3 chat-roundtrip gate only asserted `text.length > 0`, which the
`<body>` fallback scrape can satisfy with static page text (nav links,
footer copy, demo blurb) trailing the sent message. When the agent run
finished with ZERO assistant content (RUN_STARTED -> RUN_FINISHED, no
TEXT_MESSAGE) the assistant-message bubble never rendered, so the probe
fell back to scraping <body> and false-PASSED on incidental page chrome —
the mechanism that masked the BIA outage (dead agent reported green).
Track the provenance of the captured response and require it to come from
the [data-testid="copilot-assistant-message"] container (a genuine
assistant turn — the DOM-layer equivalent of RUN_FINISHED + a non-empty
TEXT_MESSAGE), not the body fallback. Apply the same guard to L4 so weather
content must also come from a real assistant turn. The instrumented probe
sends the aimock fixture headers and renders real content into the
container, so this does not affect a working agent; it only rejects the
empty-turn false-pass.
Adds a red-green regression test reproducing the BIA false-pass.
## Summary
Two pre-existing isolation-CLI bugs that #5730 (slot-reaping) surfaced
and correctly deferred as out-of-subject — both real, now fixed with
real-surface red-green.
1. **Port-conflict guard silent-failure** — `_slot_ports_free` consumed
`_slot_offset_ports` via process substitution (`done < <(...)`), so a
`die` on an out-of-range/non-numeric slot exited only the subshell; the
loop read zero ports and the function returned 0 ("all free"), silently
defeating the port-conflict guard for a bad slot. Now captures into a
variable with `|| die` so the failure propagates to the caller.
2. **`--isolate` arg-parsing** — `--isolate <name> <slug>` (name before
slug) mis-parsed the name as the slug then died "Unexpected argument"; a
bare `--isolate=` silently auto-picked. Now order-independent (deferred
`pending_iso_name` resolution after the parse loop), rejects an empty
value loudly, and `--isolate=<name>` (non-numeric) binds an explicit
name. `--isolate=<N>` numeric pin unchanged. **Intended behavior
change:** `--isolate=<name>` previously died in the picker; it now binds
a name.
## Tests
`bats showcase/scripts/__tests__/` = 170/0 — real slot dirs, real
bad-slot input, real parser end-to-end (no mocks). CR converged round 1
(zero bucket-a), Procedure 3 promotion audit clean.
## Deferred to a potential follow-up (pre-existing "port-enumeration
completeness")
CR surfaced, and correctly kept out of scope: `_slot_offset_ports`
hardcoded/incomplete infra-port list (phantom 8081, misses other
published ports); apply_isolation port-offset regex matches quoted
short-form only; `.last-test-ts` not repointed per-isolate (concurrent
clobber); `--repeat` treats a following flag as its count.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The StateStore interface and the in-memory MemoryStore default remain;
durable backends can be reintroduced as a follow-up. Both adapter packages
were merged in #5613 but never published to npm, so removal is a clean
delete with no consumer impact.
- Delete packages/bot-store-redis and packages/bot-store-postgres.
- Revert the bot release scope and drift guard to bot + bot-ui.
- Strip the Redis dep, demo:restart script, restart demo, docker-compose,
and REDIS_URL env from examples/slack.
- Rewrite the bot persistence/transcripts docs around "MemoryStore default
+ implement the StateStore interface yourself for durability".
The harness-workers SSOT modeled only the top-level numReplicas, whose drift
gate watched a field that does not drive the live replica count. harness-workers
is single-region (us-west2); Railway derives the live count from
multiRegionConfig.us-west2.numReplicas.
- WorkerProvisioning gains effectiveReplicas (= multiRegionConfig.us-west2.
numReplicas), the authoritative field the drift gate now asserts. Top-level
numReplicas is retained as a documented mirror.
- Declared values reflect current reconciled reality (verified live via the
Railway GraphQL environment.config staged-config read): prod and staging both
at effectiveReplicas=6 (parity achieved by B-reconcile scaling prod 3 -> 6 in
both the top-level field and multiRegionConfig). BROWSER_POOL_MAX_CONTEXTS is
40 on both envs (verified live).
- Regenerated railway-envs.generated.json; drift-gate test asserts
effectiveReplicas (RED-GREEN proven). RAILWAY.md documents multiRegionConfig
as the effective knob and the achieved parity.
Add WorkerProvisioning interface and workerProvisioning field to the
harness-workers ServiceEntry in railway-envs.ts. Declares current live
reality: prod=3 replicas, staging=6 replicas, BROWSER_POOL_MAX_CONTEXTS=40
per worker (both envs).
Worker model: 1-worker-per-replica (Railway runs one process per
container, keyed on HOSTNAME). HARNESS_POOL_COUNT is informational
only — not a fork factor. Authoritative concurrency knob per worker is
BROWSER_POOL_MAX_CONTEXTS.
Comments flag: staging config-field drift (Railway field=2, live=6) as
a follow-up item; and the prod/staging parity decision (prod=3 vs
staging=6) as deliberately deferred.
Extends emit-railway-envs-json.ts to emit workerProvisioning into the
generated JSON snapshot. Adds drift-gate test
(harness-workers-provisioning.test.ts) that fails if SSOT numReplicas
diverges from the committed JSON snapshot — no live Railway API calls.
Red-green-red-green verified locally.
Updates RAILWAY.md with the 1-worker-per-replica model, declared
values, manual apply procedure, and drift-gate reference.
The existing tooling is verify-only for numReplicas; applying a replica
count change to Railway remains a manual operation (Railway Dashboard or
GraphQL API).
The llamaindex copilotkit API route logged '[copilotkit/route] POST ...'
and '[copilotkit/route] Response status: 200' unconditionally on EVERY
request. Under d6 probe fan-out this exceeded Railway's 500-logs/sec cap
('Messages dropped' -> 'Stopping Container'), killing the replica.
Gate both per-request console.log lines behind a SHOWCASE_ROUTE_DEBUG env
flag (default off). Module-load logs and error logging are unchanged.
This chatty pattern is shared/copied across ~16 integrations (including
the langgraph-python gold standard); this commit scopes the fix to
llamaindex. The others are flagged as follow-up.
A normal harness deploy rebuilds the shared showcase-harness image and
bounces the pool workers (PR #5715). Immediately after the bounce the
workers re-register, the producers re-arm, and every family is mid-sweep:
lastSuccessAt still points at the pre-bounce success, so it reads stale
against the silence thresholds (banner 2x period, Slack alert 3x period +
3 consecutive ticks). The result was a FALSE "worker family X has not
completed successfully" banner AND Slack family-silence alert during the
expected post-bounce drain window.
Fix: a bounce-keyed grace window. The freshest worker registered_at across
the /api/runs workers strip is the fleet's most-recent bounce instant
(independent of CP boot — a worker can bounce while the CP stays up). While
now - bounce < 2 x period, a family with no success yet is DRAINING, not
silent, so neither the §7.4 banner, the §7.3 cell glyph, nor the §9 Slack
alert flags it. Beyond the window with still-no-success, genuine silence
fires exactly as before.
The determination lives in two surfaces (server monitor for the Slack
alert; client isFamilySilent for the banner + glyph), so both now consume
the same new SSOT field (WorkerView.registeredAt) and the same 2x-period
grace constant, keeping them consistent.
- run-view.ts: project registered_at -> WorkerView.registeredAt (server)
- family-silence-monitor.ts: BOUNCE_GRACE_PERIOD_MULTIPLIER + freshest-bounce
grace gate, keyed off body.workers
- worker-runs-context.tsx: freshestBounceMs + bounceAtMs grace arg on
isFamilySilent; banner + cell glyph pass it
- ops-api.ts: WorkerView.registeredAt on the client DTO
A GREEN coverage cell flapped green -> grey -> green every time its probe
job's worker lease lapsed and the control-plane sweeper re-queued the job
(worker-reclaimed-pending). The data layer already preserves the
last-known-good colour in chipColor while surfaceState flips to "pending",
but depth-chip.tsx's `if (pending)` branch did a destructive grey
early-return that never read chipColor.
Split the pending branch:
- prior-good (chipColor is a real colour, or depth > 0): render the normal
coloured chip via the default branch's exact ternary (threading regression
through) PLUS a non-destructive refreshing affordance -- a corner ⟳ glyph
and a subtle pulsing ring in the chip's own colour, with
data-refreshing="true" / data-has-prior="true". Colour preserved.
- no-prior (never-run / first load: gray + depth 0): keep today's honest grey
⟳ chip, now tagged data-has-prior="false".
The unreachable red ⚡ overlay and the "failure never masked" gate are
untouched; red/regression cells pass through with no spinner. The refreshing
cue is conveyed by shape (⟳) + motion (ring), never by colour alone, and the
pulse/spin respect prefers-reduced-motion (motion-reduce:animate-none), so the
static ⟳ carries the meaning for colour-blind and reduced-motion operators.
Renderer-only change; unified-cell.tsx is the sole caller passing `pending`.
--isolate <name> <slug> (name before slug) mis-parsed the name as the
slug, then died "Unexpected argument"; a bare --isolate= silently
fell through to auto-pick. Defer the ambiguous post---isolate token to
pending_iso_name and resolve it after the parse loop (slug present =>
token was the name; no slug => token was the slug); reject an empty
--isolate= loudly; --isolate=<name> (non-numeric) now binds an explicit
isolate name. The --isolate=<N> numeric pin is unchanged.
_slot_ports_free consumed _slot_offset_ports via process substitution
(done < <(...)), so a die on an out-of-range/non-numeric slot exited only
the subshell — the loop read zero ports, any_held stayed 0, and the
function returned 0 ("all free"), silently defeating the port-conflict
guard for a bad slot. Capture into a variable with || die so the failure
propagates to the caller. Real-surface bats prove a bad slot now fails
loudly while valid-slot free/held behavior is unchanged.
Dry-run by default (lists the plan, changes nothing); --force executes,
--all ignores TTL/keep, --include-live opts into reaping a live-owner
target, <name|slot> targets one. Identifies harness-owned projects via
the slot-record / run-dir / showcase-iso<N> / self-id-label union, and
never touches the base 'showcase' stack or BuildKit resources. Real
docker bats prove dry-run/--force/--all + the base/buildkit guards.
A --keep'd isolated stack whose owning process had exited (but whose
containers kept running) was classified 'live' forever and never reaped,
leaking Docker stacks indefinitely. Introduce a start-time-verified
_owner_liveness probe and a new 'kept' state, an ISOLATE_KEEP_TTL (4h,
SHOWCASE_ISOLATE_KEEP_TTL-overridable) that flips an over-age kept slot
to 'stale' so the sweep reclaims it, a com.copilotkit.showcase.isolate
self-id label stamped by apply_isolation, a 'slots --reapable' filter,
and a macOS lsof COMMAND-truncation fix in the own-project port filter.
Real-surface bats cover the liveness false-positive and TTL reaping.
A red/BE✗ dashboard cell is often staleness, not a broken feature. The
per-cell BE flag is resolveD4 = worst-of(chat,tools) folded by a staleness
window; a green row older than its window folds to stale-red even when the
app is healthy. Add D5 Strategy 10 with a diagnostic checklist (harness
/api/runs sweep-duration vs window, observed_at age vs *_STALE_AFTER_MS,
numReplicas × HARNESS_POOL_COUNT concurrency) plus an anti-pattern entry.
Evidence from the 2026-06-26 prod incident: D4 sweep (127m d6 / 97m e2e-demos)
blew past the 60m window with a starved worker pool, stale-reddening ~13
integration columns while apps were fine.
Emit CVDIAG backend boundary markers from the agent process for strands-typescript (byteLength fix on sseChunkByteLength), enable the emitter in docker-compose.local.yml, vendor src/cvdiag, and exclude tests from tsconfig.
Forward inbound X-AIMock-Strict header through the two-process strands-typescript hop (Next route -> agent -> sub-agent fetch), with null-guard on the forwarding proxy fetch and supporting unit tests.
Bring the agnostic root A2UI docs up to the catalog-on-provider model and
make every generated framework serve them consistently.
- Root /generative-ui/a2ui (index, fixed-schema, dynamic-schema): lead with
passing a catalog on the provider (auto-enables A2UI and auto-injects the
generate_a2ui tool), add a manual opt-out section explaining the two pieces
you wire yourself (the generate_a2ui agent tool and the A2UIMiddleware), and
set fixed-schema to injectA2UITool: false since the agent owns the tool.
- Flip langgraph-fastapi, strands, strands-typescript to docs_mode: generated
so they serve the shared root A2UI docs 1:1 with langgraph-python.
Generated frameworks covered: langgraph-python/fastapi/typescript, google-adk,
strands, strands-typescript. deepagents (authored) is handled separately.
## Root cause
Prod's `harness-workers` fleet worker runs a **stale `showcase-harness`
image** because it had **no `prod` env entry** in the railway-envs SSOT.
- The worker (`serviceId c2aa8a0b-350e-4b76-8541-3012dfac41d0`, prod
instance `7c48ee43-6df4-457b-b977-10f1f1ac1680`, `HARNESS_ROLE=worker`)
consumes the shared `showcase-harness` image via `imageOf: "harness"`.
- `expandImageConsumers(names, env)` is **env-aware**: a consumer only
enters an env's redeploy scope if it declares that env
(`redeploy-env.ts:278` — `if (!Object.hasOwn(entry.environments, env))
continue;`).
- Because the worker modeled **staging only**, a rebuilt
`showcase-harness:latest` bounced the prod control-plane but **silently
skipped the prod worker**, which kept its stale **2026-06-19** image.
- That stale image bakes a **1-demo `registry.json`** for
`ms-agent-harness-dotnet` (only `beautiful-chat`). The hourly
`e2e_demos` driver runs on the worker → resolves 1 demo → writes only
`e2e:ms-agent-harness-dotnet/beautiful-chat`. The other 38 feature rows
never exist in prod PocketBase → `resolveD3.exists === false` → `UI`
badge omitted → broken D3 rung collapses the ladder → **D0**.
(d5/d6 populate fully in prod because the D5/D6 drivers enumerate from a
**compiled-in** script registry, not the `registry.json` data file —
only `e2e_demos` is data-driven, which is why only `UI` was affected.)
## Fix
Backfill the live prod worker as a real `prod` env entry in
`scripts/railway-envs.ts` (real serviceInstance ID `7c48ee43…`), flip
`gateIgnore` off, and set `gateValidated: true`. The env-aware `imageOf`
expansion now pulls the prod worker into the **prod** redeploy scope on
every `showcase-harness` rebuild, so it can no longer drift onto a stale
image.
Also regenerates `railway-envs.generated.json` (Ruby/jq boundary
artifact) and the golden behavior-preservation fixture, and updates the
two gate-count assertions (`gateValidated` services 40→41;
`harness-workers` removed from the gateIgnore set).
## Local RED → GREEN proof
Failure surface: the real `expandImageConsumers("harness", "prod")`
against the real SSOT must include `harness-workers`.
**RED** (prod env entry absent from SSOT — the bug):
```
× includes harness-workers in the PROD redeploy scope when showcase-harness rebuilds
AssertionError: expected [ 'harness' ] to include 'harness-workers'
at __tests__/redeploy-env.harness-worker-prod-scope.test.ts:31:19
Tests 1 failed | 1 passed (2)
```
**GREEN** (after adding the prod `harness-workers` env entry):
```
✓ includes harness-workers in the PROD redeploy scope when showcase-harness rebuilds
✓ still includes harness-workers in the STAGING redeploy scope (no regression)
Tests 2 passed (2)
```
Full SSOT-dependent suite (golden snapshot, emit-json, image-ref gate,
promote closure, verify-matrix, redeploy-env): **140 passed**.
## Note (out of scope for this PR)
This SSOT change ensures the prod worker is bounced on **future**
rebuilds. The currently-live prod worker still needs a one-time
redeploy/restart onto the current `showcase-harness:latest` (39-demo
registry) to immediately backfill the 38 missing rows; that is an
operational step, not a code change.
## What
Brings the **Authentication demo** of three integrations into 1:1
conformance with the `langgraph-python` (LGP) gold standard, completing
the work started in #5713. The showcase Iron Law: LGP is the reference;
every integration must have (1) identical tests, (2) near-identical
frontends, (3) minimal backends, (4) per-integration fixtures.
A conformance audit against LGP found 3 violators (the other 17
integrations already conform):
| Integration | Violation | Fix |
|---|---|---|
| **claude-sdk-python** | Legacy auth-*first* shape: class
`ChatErrorBoundary` + `lastError`, no `handleAuthError`, missing
`sign-in-card.tsx`, divergent banner/hook | Ported
`page.tsx`/`use-demo-auth.ts`/`auth-banner.tsx` **byte-identical** to
LGP + new `sign-in-card.tsx`; added the shared shadcn primitives it
lacked (`lib/utils.ts`, `components/ui/{button,card}.tsx`) +
`radix-ui@^1.4.3` (matching the claude-sdk-typescript peer) |
| **built-in-agent** | Distinct legacy variant:
`ChatErrorBoundary`→`auth-demo-chat-boundary`, local 401-regex
`onError`, auth-first hook | Normalized error-handling shape + hook to
LGP; **preserved** the forced `<CopilotKitProvider>` (default-agent) +
raw-Tailwind divergences (documented in a new `README.md`) |
| **ms-agent-harness-dotnet** | Missing `tests/e2e/auth.spec.ts` (rule
1) | Added LGP's spec **byte-identical** (sha256 `603a68e5…`) |
After this PR, all auth `page.tsx`/hook files are byte-identical to LGP
except documented, forced per-integration wiring; all `auth.spec.ts`
share LGP's sha256.
## Red–green proof (per integration, on the real probe surface)
The shared `d5-auth.ts` probe accepts *either* `auth-demo-error` *or*
`auth-demo-chat-boundary`, so it passes leniently on the legacy shape —
the **discriminating gate is the byte-identical `auth.spec.ts`**
(asserts unauth-first `SignInCard` + `auth-authenticate-button` +
post-sign-out `auth-demo-error`):
- **claude-sdk-python:** legacy frontend → `auth.spec.ts` **6/6 FAIL**
(timeout on `auth-sign-in-button`); conformed → **6/6 PASS** (`next
build` clean).
- **built-in-agent:** legacy → 6/6 FAIL; conformed → 4 conformance
assertions flip FAIL→PASS incl. unauthenticated-send surfaces
`auth-demo-error` (`next build` clean).
- **ms-agent-harness-dotnet:** spec absent (coverage gap) → added →
`--d5 --isolate` green, full real-browser auth flow passes.
## Review
7-agent CR round + mandatory 7-agent confirmation round → **converged to
zero findings** (correctness, conformance, types/build, deps/lockfile,
tests, silent-failures, cross-integration regressions). 2 P2 conformance
nits found and fixed (import-style alignment; restored `DEMO_TOKEN` so
built-in-agent's hook is byte-identical to LGP).
## Known limitation (non-blocking, pre-existing infra)
The GHA workflow `test_e2e-showcase-on-demand.yml` runs Playwright only
for slugs with a Python agent, so the **built-in-agent /
ms-agent-harness-dotnet auth specs are not executed in PR CI**. This is
a pre-existing infra gap (those integrations have no Python agent), not
introduced here. Coverage **does** exist post-merge: the Railway staging
**d6 harness** enumerates services language-agnostically and runs the
auth probe against live `/demos/auth` for both — verified, and it's what
drives their dashboard cells green at D6. A follow-up to add a
non-Python e2e execution path is warranted.
## Notes (pre-existing, not introduced)
- `npm ci`/`npm install` in `showcase/integrations/claude-sdk-python`
shows a micromark/unified desync and a zod/openai ERESOLVE peer conflict
— both reproduce identically at the base commit `ab85b939ac`
(independent of the `radix-ui` add); handled by the existing
`--legacy-peer-deps` path.
Ref: #5713 (original post-sign-out auth rejection fix).
Add a d5-a2ui-recovery probe so the A2UI error-recovery demo runs on every
PR via the d5/d6 fleet harness, not only the manual on-demand workflow.
- New probe d5-a2ui-recovery.ts drives both pills in one session: HEAL
asserts >=2 newly-mounted declarative-metric tiles and no hard-failure
card; EXHAUST asserts the "Couldn't generate the UI" card appears and
no surface paints. Deltas (vs a pre-send baseline) keep the two
mutually-exclusive negatives correct across the shared session. The
transient "Retrying..." label is not asserted (timing-flaky).
- Prompts are sent as typed input, keyed per integration slug, mirroring
each slug's suggestions.ts message verbatim. The recovery prompts are
unique per slug because the inner render_a2ui calls carry no
x-aimock-context; a typed message is byte-identical to the pill
dispatch, so it matches the same fixture. Sending via input (not a
preFill pill click) lets the runner snapshot its run-lifecycle baseline
first, avoiding a false done-signal-missing failure.
- Register a2ui-recovery in d5-registry, map it in d5-feature-mapping,
add its representative fixture, and mirror the mapping in the dashboard
CATALOG_TO_D5_KEY (kept in lock-step via the drift test).
Verified green locally on both recovery paths: langgraph-python
(backend-owned get_a2ui_tools) and strands (auto-inject middleware).
Jordan Ritter