The promote workflow's verify-prod job calls verify-deploy.ts directly
(--env prod --services <promoted set>) without #5752's --skip-ineligible
flag, so a known-but-ineligible service (harness-workers, probe.prod=false)
hard-errored exit 2 and crashed the gate AFTER a successful promote
(CI run 28333317081: llamaindex landed, then verify-prod crashed).
Generalize the eligibility filter into verify-deploy.ts itself rather than
relying on each caller to pass a flag: flip skipIneligible to ON by default
in the CLI (parseArgs). A known-but-not-probe-eligible service for the
requested env is now SKIPPED with an `N/A — not probe-eligible ... skipped`
status line and the eligible subset is probed. Works for ANY --env, so it
composes with #5752's staging path and fixes the direct prod-verify call.
When EVERY requested service is ineligible (e.g. promoted set is just
harness-workers), runVerify exits 0 with a "nothing to probe" note instead
of the vacuous-green FAIL — distinct from the empty-filter fault, which
still fails loud. Unknown (non-SSOT) names STILL hard-error on every path
(a typo is a real fault). Added --strict-eligibility to opt back into the
hard-refuse; --skip-ineligible kept as an explicit no-op for back-compat.
Red: `verify-deploy.ts --env prod --services harness-workers` crashed
exit 2. Green: same command skips (N/A) and exits 0. Mixed set
harness-workers,showcase-llamaindex skips workers and still probes (and
red-gates) llamaindex.
P3 (the staging-live-green precondition in `bin/railway promote`) handed
EVERY service in the snapshot to verify-deploy.ts, including services the
SSOT marks `probe.staging=false` (harness-workers). verify-deploy.ts
hard-errors on a `--services` entry that is not probe-eligible, so P3
surfaced "verify-deploy crashed: ... not probe-eligible" as a REFUSE.
In a tier-ordered fleet promote (`bin/railway promote llamaindex`) that
REFUSE gated every later tier, so llamaindex was never promoted (CI run
28332775532).
Fix: derive STAGING_PROBE_INELIGIBLE from the same SSOT, and in
check_p3_staging_live_green drop those names BEFORE invoking the probe,
logging "P3 N/A (<svc>): not staging-probe-eligible". An ineligible-only
set returns no findings (clean skip); a mixed set still probes — and
still gates on — the eligible services. P3 is unchanged for eligible
services.
Renumbered the snapshot-ivar-lint allowlist for the shifted lines.
The bare-substring 'weather' fixture in aimock/d4/llamaindex/chat.json emitted a
get_weather tool call with no toolName gate. The tool-free voice agent's prompt
"What is the weather in Tokyo?" (substring "weather") leaked into this fixture,
emitting a get_weather call the voice agent could never resolve, so the voice D6
cell hung (done-signal-missing, body stuck on get_weather/Running).
Add toolName:"get_weather" so the fixture only fires when the requesting agent
actually registers get_weather (mirrors the gate in d6 tool-rendering.json). The
tool-free voice request now falls through to voice.json's exact content match.
Local red->green proof (showcase test llamaindex:voice --d6 --direct):
- RED: done-signal-missing; body "What is the weather in Tokyo? get_weather Running"
- GREEN: assistant settled "The weather in Tokyo is currently 22C with partly
cloudy skies and light easterly winds."; 1 passed (3.0s)
Direct aimock probes confirm the gate: tool-free -> content; with get_weather tool
-> tool call still fires. Regression: tool-rendering D6 still green; headless-complete
weather turn still passes (uses its own gen-ui-headless-complete.json fixture).
(cherry picked from commit cf6ff7c08153367239437d6c4fff425d546eb245)
The ADD-2 block suppresses the streamed TOOL_CALL_CHUNK for all frontend
tools, relying on the bare snapshot's ag_ui_tool_calls to deliver the call
(emitting both doubles the args and breaks scheduleTime / pie-bar
useComponent). But the open-generative-ui runtime middleware builds the
sandboxed iframe exclusively from streamed TOOL_CALL_* events and never
reads the snapshot, so open-gen-ui and open-gen-ui-advanced rendered 0
iframes.
Add a name-scoped exemption that streams the chunk only for
generateSandboxedUi, keeping snapshot-only delivery for every other
frontend tool. The exemption is intentionally narrow to preserve the
double-args fix.
Red->green (D6, --direct, real Docker page):
- open-gen-ui: RED "saw 0 iframe(s), longest srcdoc=0" -> GREEN (iframe + srcdoc)
- open-gen-ui-advanced: RED "selector cascade matched 0 elements" -> GREEN
Regression (all still green): beautiful-chat 5/5 (toggle-theme, pie-chart,
bar-chart, search-flights, schedule-meeting), agentic-chat, mcp-apps.
(cherry picked from commit 03f8d02cedbe737ec83aeefa708a9146f438a904)
The no-tools branch of ReasoningAGUIChatWorkflow.chat streams the answer via
astream_chat over OpenAIResponses, which (unlike astream_chat_with_tools on the
tools branch and the GREEN tool_rendering_reasoning_chain_agent) does not
accumulate resp.delta back onto the terminal resp.message.content. The
content-empty message was then snapshotted into MESSAGES_SNAPSHOT, clobbering
the ~284-char streamed answer and rendering an empty assistant bubble
(reasoning-display failed text-unstable: reasoning block painted, answer gone).
Accumulate the streamed text deltas in the no-tools path only (track_text =
not tools) and fold them onto resp.message before _finalize_chat snapshots it.
Strictly additive: only fills a message the stream left empty, never overwrites
content the LLM already accumulated, and is inert when tools are present so the
tools branch / reasoning-chain agent are untouched.
(cherry picked from commit 46afd0e040a669d14f91b8c136df9c94a91950d0)
The bare 'summarize' userMessage in d4/llamaindex/chat.json substring-matched
the D6 gen-ui-agent pill 'Research our top competitor and summarize their
strengths and weaknesses.', returning the sales-pipeline text fixture instead
of the gen-ui-agent set_steps tool call. The competitor pill then produced
no/duplicate steps, failing d6:llamaindex. Narrow the match to the verbatim D4
toolbar probe 'Summarize the sales pipeline' (langgraph-python parity), which
no demo pill contains as a substring. D4 llamaindex stays green (the bare entry
was unused by any D4 cell).
(cherry picked from commit c23801c8b32d292cacf6fb2e7e2a68270eebaa84)
The shared FixedAGUIChatWorkflow catch-all dropped request-injected query_notes,
so NotesCard never mounted. Give the cell its own agent (mirrors beautiful_chat_agent)
so request-time frontend tools forward. Verified GREEN via control-plane --direct.
(cherry picked from commit 9c1b8ce2c33afc89000355d83c995f03da208f0f)
Same root cause as the sibling declarative-gen-ui (A2UI Dynamic Schema) fix:
the A2UI middleware mounts the surface from a STREAMED render-tool CALL whose
name is in its watched set, not from a TOOL_CALL_RESULT. The prior approach had
display_flight return an a2ui_operations container in the tool RESULT, which the
llama-index AG-UI adapter only re-emits via MESSAGES_SNAPSHOT — a shape the
middleware never inspects — so the flight-card surface stayed unmounted
(reason=surface-missing; the a2ui-fixed-card testid never appeared).
- route.ts: set a2ui.injectA2UITool: true so the middleware watches render_a2ui.
- a2ui_fixed.py: display_flight now returns the fixed-schema render_a2ui args
(surfaceId/catalogId/components/data) as JSON; a workflow override
(_A2UIRenderToolCallWorkflow) parses each backend tool result and re-emits it
as a streamed render_a2ui tool-CALL (TOOL_CALL_START name=render_a2ui ->
chunked TOOL_CALL_ARGS carrying the components+data JSON -> TOOL_CALL_END),
mirroring how google-adk drives the middleware. These events are already in
the upstream AG_UI_EVENTS allow-list the SSE router streams against.
Backend still produces the pre-authored flight schema (no stub). Only
display_flight (one backend tool, name unchanged) is involved, so the d6 fixture
needs no re-keying. Integration-code only; no shared/@ag-ui package touched.
(cherry picked from commit 74d61eddf7eccf22bcc96c37f0e35a70dd823a2f)
## What was broken
`d6:llamaindex/gen-ui-declarative` (A2UI Dynamic Schema) was red, and
the whole **llamaindex D6 column** was crash-looping.
## Root cause (two stacked layers, both llamaindex-integration-level)
1. **OOM crash-loop.** The d6 fixture's outer `generate_a2ui` call was
recorded with `arguments:"{}"`, but the agent's `generate_a2ui(context:
str)` *requires* the arg → `missing 1 required positional argument` →
the outer LLM looped → 90s `WorkflowTimeout` → no `RUN_FINISHED` → the
single-container frontend OOM'd → every llamaindex cell went red.
2. **surface-missing.** Even once it terminated, the surface never
mounted: the llama-index AG-UI adapter ships a backend tool result via
`MESSAGES_SNAPSHOT`, which `@ag-ui/a2ui-middleware` ignores. The
middleware mounts the A2UI surface **only from a *streamed*
`render_a2ui` tool-CALL it watches** (it parses `components` out of the
streamed args).
## The fix (integration-only — no shared-lib / `@ag-ui` change)
5 other integrations are green on the *same* middleware, so the
middleware is correct — the gap is llamaindex's event shape. This PR:
- **Rebuilds** `aimock/d6/llamaindex/gen-ui-declarative.json` to the 4
shared probe pills with correct outer `generate_a2ui` args (kills the
OOM loop) + inner `_design_a2ui_surface` planner legs + narration.
- **Overrides `aggregate_tool_calls`** (`a2ui_dynamic.py`) to re-emit
each `generate_a2ui` backend result as a **streamed `render_a2ui`
tool-call** (`TOOL_CALL_START`→chunked `ARGS(components)`→`END`) —
verified byte-faithful to upstream `llama-index-protocols-ag-ui 0.2.2`
plus this one additive step.
- **Flips `injectA2UITool: true`** so the middleware watches
`render_a2ui`.
- Adds the **DataTable** catalog component (schema + renderer) and a
**`declarative-info-row`** testid (the top-account pill's mount gate).
- Aligns `suggestions.ts` to the shared probe pills.
## Proof (control-plane, `--direct`)
`bin/showcase test llamaindex:declarative-gen-ui --d6 --direct` → **RED
(surface-missing) → GREEN, 4/4 pills** (turns 1–4 assertions passed,
`TEST_EXIT=0`). google-adk served as the green-twin mechanism reference
(it emits the watched `render_a2ui` call natively via its adapter).
## Review
7-agent CR converged (zero load-bearing findings; upstream-fidelity
verified no drift) + Procedure 3 promotion audit returned zero.
Re-verified green after the CR doc/prompt/logging fixes.
## Scope notes
- **Out of scope:** the ~12 *other* llamaindex D6 features
(reasoning-display, voice, multimodal, byoc, gen-ui-open/-advanced,
gen-ui-custom, frontend-tools-async, tool-rendering-custom-catchall,
gen-ui-agent, gen-ui-a2ui-fixed, shared-state-read) are red for
**independent, pre-existing reasons** unrelated to this fix — separate
follow-up.
- **Follow-ups (non-blocking):** guard the inner planner `json.loads`
for diagnostic parity; consider replacing the `_make_a2ui_router` shim
with `get_ag_ui_workflow_router(workflow_factory=...)`.
- Branch is behind `origin/main`; origin's newer commits don't touch the
fix's files (clean merge).
The render re-emit override had three silent failure paths: a non-JSON tool
output (broad except swallowing TypeError/ValueError), the {"error": ...} dict
from generate_a2ui's no-tool-call branch, and a valid-JSON result missing
components. Each produced a blank UI with no diagnostic trail. Narrow the parse
except to json.JSONDecodeError (guarding that content is a str) and log a
contextual warning on each path. Happy path unchanged.
(cherry picked from commit 94b0a69aa4772758e3bcc05f67a6c28b1b0a503d)
The inlined planner SYSTEM_PROMPT listed every A2UI catalog component except
DataTable, even though the TS catalog and a team-performance suggestion pill
target a DataTable surface. Since the planner is a separate OpenAI call driven
solely by this hardcoded prompt (it never sees the TS Zod schema), DataTable
emission was unreliable. Add DataTable to the catalog list, mirroring the TS
definition (columns/rows shape) and the other entries' wording.
(cherry picked from commit a054e41b8d9394540e1cf7b84ccf9e8e0722c519)
The override docstring claimed it reproduces the upstream aggregate_tool_calls body byte-for-byte; it is functionally equivalent with two cosmetic diffs (Optional type hint, list comprehension). Reword to match reality.
(cherry picked from commit 5e91118b3a463dcefcd228f6343e472db6081c0f)
The _note asserted injectA2UITool:false (unchanged) and that flipping to true
would blank-render, and that generate_a2ui returns an a2ui_operations container
for the middleware to forward. Both are now false: this PR set injectA2UITool:true,
generate_a2ui returns raw planner args, and the surface mounts from a streamed
render_a2ui tool-call (START/ARGS/END) the agent re-emits, which the middleware
watches under injectA2UITool:true. Prose-only; no match keys or payloads changed.
(cherry picked from commit 5679b001580615f2e7d988d8c7063994076ace29)
The page header still described the runtime as configured with
`injectA2UITool: false` and the backend agent as owning `generate_a2ui`,
mirroring beautiful-chat. This PR inverted the route to
`injectA2UITool: true`, so the comment was stale. Rewrite the step-3 block
to describe the current mechanism: `injectA2UITool: true` populates the A2UI
middleware's watched-names set, which mounts the surface from a STREAMED
`render_a2ui` tool-call the agent re-emits via its `aggregate_tool_calls`
override in a2ui_dynamic.py. Drops the stale generate_a2ui framing and
matches the accurate header in route.ts.
(cherry picked from commit 407d755638ebe28418f1f8ce2c558f202995284e)
The InfoRow renderer was the only catalog component missing a
data-testid. The top-account pill's _design_a2ui_surface leg emits
7 InfoRow facts + a PieChart, and the d5-gen-ui-declarative probe
asserts declarative-info-row (minCount 1) as top-account's
distinguishing testid. Because the renderer never painted that
testid, the completeOnMount gate (whose surfaceTestIds include
declarative-info-row) never observed a mount, the turn never
completed, and the run reported reason=surface-missing. Every other
pill passed because its distinguishing testid (metric / status-badge /
data-table) was already emitted.
The A2UI middleware mounts the surface from a STREAMED render-tool CALL whose
name is in its watched set, not from a TOOL_CALL_RESULT. The prior approach
emitted a TOOL_CALL_RESULT carrying an a2ui_operations container, which the
middleware never inspects, so the surface stayed unmounted (surface-missing).
- route.ts: set a2ui.injectA2UITool: true so the middleware watches render_a2ui.
- a2ui_dynamic.py: generate_a2ui now returns the planner's render_a2ui args
(surfaceId/catalogId/components/data) as JSON; the workflow override
(_A2UIRenderToolCallWorkflow) parses each backend tool result and re-emits it
as a streamed render_a2ui tool-CALL (TOOL_CALL_START name=render_a2ui →
chunked TOOL_CALL_ARGS carrying the components JSON → TOOL_CALL_END), mirroring
how google-adk drives the middleware. These events are already in the upstream
AG_UI_EVENTS allow-list the SSE router streams against.
Backend still produces the components (no stub). Inner planner tool stays
_design_a2ui_surface, so the d6 fixture needs no re-keying.
Three stale, pre-layer-(b) prose sites still claimed drain() ABANDONS the
in-flight run and the loop skips queue.report: the drainFleetWorker JSDoc, the
runWorker stop() inline comment, AND the case-worker SIGTERM-handler block.
Layer (b) changed this — on SIGTERM the worker stops claiming, lets its
in-flight cell FINISH within the 90s grace, and REPORTS its real terminal
result via the runAbort/abortedWithoutResult discriminator; only a run that
OVERRUNS the grace is abandoned -> reclaimed by layer (a). Preserved the still-
accurate drainReason=shutdown red-side-emit soft-wind-down + deregister-vs-crash
distinction. Also refreshed the WHY-THIS-ORDER preamble (90s finish-and-report
grace, 180s platform stop / layer-c drainingSeconds). Comments only — no
behavior change.
Layer (c) deploy-rollover config for the harness-workers fleet — pure
Railway config, no custom rolling-restart code. Declares overlapSeconds=45
(capacity floor: old deployment serves until new workers register+claim, so
no staleness dip) and drainingSeconds=180 (SIGTERM->SIGKILL window >=
PLATFORM_STOP_GRACE_MS so the layer-(b) 3s+90s composed worker-drain finishes)
for both prod and staging in the railway-envs SSOT, regenerates the JSON
snapshot, and extends the harness-workers drift gate so CI fails if either
field drifts between SSOT and snapshot. Documents both knobs, their rationale
(incl. that Railway's draining default is 0s = immediate SIGKILL), the
composition with drain layers a+b, and how to apply them (GraphQL
serviceInstanceUpdate / dashboard) in RAILWAY.md.
CR P2 round on layer-(b) graceful worker drain.
P2-A (contested TOCTOU, reconciled empirically): a run that resolves with a
valid result in the same flush the grace setTimeout fires is REPORTED, not
spuriously abandoned. runAbort.abort() lives only in stop()'s Promise.race
TIMEOUT leg, which loses the race once `done` is resolvable — so a finished
run never trips the abortedWithoutResult discriminator. Reviewer crb6 was
correct; the TOCTOU is a non-bug, so no logic changed. Added a deterministic
fake-timer regression pin forcing both the run-completion timer and the grace
timer due in one advance (verified to BITE under an over-aggressive
abandon mutation), plus a clarifying comment at the discriminator.
P2-B: corrected the requestDrain() JSDoc — post-B2 the report-skip,
heartbeat-stop, and driver-cancel key on the grace-expiry signal
runAbort.signal, not stopAbort.signal (which now only stops claiming).
B5 — set the production drain grace T to 90s (was 6s). The grace stopped being
a bare TEARDOWN budget and is now the FINISH-AND-REPORT budget stop() waits for
an in-flight run to finish-and-report (layer b) before firing runAbort -> abandon
-> layer-(a) reclaim. T must BOUND a typical cell-job (so a normal in-flight job
finishes within grace) yet stay SHORTER than the platform SIGTERM->SIGKILL window
with headroom. Sized from in-repo cell-job signal: a single-service cell-job runs
~15s (light e2e-deep) up to ~200s (heavy d6-all-pills under contention); the
per-job lease ceiling is 300s. 90s covers the bulk and stays well under the lease
so a finishing job's lease never lapses. The tail (a job that cannot finish in
grace) falls back to layer (a) — grace is deliberately FINITE. 90s is a defensible
default; B-VAL confirms/retunes from staging p95. Env-overridable via
WORKER_DRAIN_GRACE_MS.
Introduce PLATFORM_STOP_GRACE_MS (180s) documenting the C3 requirement: layer-(c)
must set Railway terminationGracePeriodSeconds = 180 so the composed serial budget
DRAIN_DEREGISTER_TIMEOUT_MS (3s) + DEFAULT_WORKER_DRAIN_GRACE_MS (90s) fits with
>=30s headroom for the health-server-close + pool-shutdown remainder. The
composed-budget test now pins the relation 3s + grace < PLATFORM_STOP_GRACE_MS
(was hardcoded < 10s) and the concrete numbers.
Add a behavioral fake-clock test of the composed-budget invariant: a run that
finishes WITHIN T is reported (finish-and-report); a run still running AT
grace-expiry fires runAbort -> abandons WITHOUT a usable result -> not reported
(layer-a reclaim backstop). The raise also broke the existing wedged-driver
default-grace test (advancing a 90s fake span re-armed the 0ms-yielding heartbeat
into a runaway cascade); fixed by giving it a clock-honoring sleep so the
heartbeat stays quiet across the window and only the grace timer is crossed.
Post-B2/B3 a run can finish-and-report after a graceful drain (drain no
longer hard-cancels the run; only grace-expiry runAbort does, which is
ctx.abortSignal). The d6 red-suppression already AND-s drainReason with
ctx.abortSignal.aborted and errorClass==="abort", which makes it
aborted-only by construction — a finished run whose legitimate red is a
genuine failure (errorClass!="abort") is NOT suppressed. This adds a
pinning test for that aborted-only contract (a finished goto-error red
under drainReason=shutdown with an un-fired abort signal must still be
reported). Verified RED via a temporary mutation to the over-broad
"drainReason alone" suppression form, GREEN on the real code.
B3 (layer-b graceful drain): the heartbeat-abort previously keyed on the
DRAIN signal, stopping lease renewal at drain-start. With B2 decoupling drain
from run-abort, a still-finishing job now runs past drain-start until grace
expiry, so killing renewal at drain-start lets the lease lapse mid-finish and
the layer-(a) reaper could reclaim the row out from under the worker
(double-run / report-after-reclaim). Gate the heartbeat-abort on the
grace-expiry signal (runAbortSignal) instead: a finishing job keeps renewing
until it reports terminal; a genuinely-abandoned job (runAbort fired at
grace-expiry) still stops renewing so its lease lapses and the reaper reclaims
it. runAbortSignal defaults to drainSignal, preserving direct-call unit-test
semantics.
Decouple 'stop claiming new jobs' from 'abort the in-flight run': drain()
no longer fires the run's abortSignal. The run's hard cancel is now a
separate grace-expiry signal (runAbort), so a run seconds from done
finishes within grace and is reported instead of being abandoned to the
sweeper. The abandon break is now conditional on abortedWithoutResult
(runAbort fired at grace-expiry → no usable result); a finished run falls
through to the report path. A wedged run that overruns grace is still cut
(runAbort.abort() in stop()'s timeout leg) and abandons → layer (a)
reclaim is the backstop.
Reconcile the drain tests that encoded the old abort-on-drain coupling to
the new contract (drain no longer fires ctx.abortSignal; the grace-expiry
abort fires at grace, exercised via a short WORKER_DRAIN_GRACE_MS).
## What
Removes the `@copilotkit/bot-store-redis` and
`@copilotkit/bot-store-postgres` adapters, added in #5613. The pluggable
`StateStore` interface and the in-memory `MemoryStore` default stay;
durable backends can be reintroduced as a follow-up when there's a
concrete need.
## Why
Both adapter packages were merged but **never published to npm**, so
there are no consumers — removal is a clean delete with no migration or
deprecation cycle. Trimming the surface keeps the bot persistence story
to one well-tested in-memory default plus a documented "bring your own
`StateStore`" path.
## Changes
- Delete `packages/bot-store-redis` and `packages/bot-store-postgres`.
- Revert the `bot` release scope and the release drift-guard test to
`bot + bot-ui` (count 2).
- Strip the Redis dependency, `demo:restart` script, restart demo,
`docker-compose.yml`, and `REDIS_URL` env from `examples/slack`.
- Rewrite the bot persistence/transcripts docs around "MemoryStore
default + implement the `StateStore` interface yourself for durability"
(unrelated enterprise/Helm Redis/Postgres docs untouched).
## Verification
- `nx run @copilotkit/bot:build` — pass
- `nx run @copilotkit/bot:test` — 121/121 pass
- Release drift guard — pass at count 2
- Repo-wide grep for
`bot-store-redis|bot-store-postgres|createRedisStore|createPostgresStore`
— zero matches
## What
Replaces the queue reaper's delete-wins behavior for stale long-expired
orphaned in-flight rows with **reclaim-wins-until-cap**: an orphaned row
is re-queued (not deleted) up to a bounded number of CONSECUTIVE
re-orphans before final claim-delete. This is layer (a) of the
worker-reclamation + graceful-rollover redesign — it makes a worker
bounce mid-column non-lossy (the column's in-flight work is reclaimed by
surviving workers instead of dropped).
## How
- **Reclaim-wins-until-cap** in `runSweepExpired`
(`showcase/harness/src/fleet/queue-client.ts`): stale long-expired
orphaned in-flight rows are re-queued; the prior delete-wins carve-out
is inverted.
- **Consecutive-orphan budget** (`consecutive_orphan_count`, migration
`1779990400`): the reclaim cap (`MAX_RECLAIM_ATTEMPTS=3`) is scoped to
*consecutive* sweeper re-orphans — incremented only on the sweeper
re-queue path, **reset to 0 on terminal `done`/`failed`**. Peer-worker
expired-lease *steals* bump the lifetime `reclaim_count` (retained as a
dashboard diagnostic) but do NOT consume the reclaim budget.
- **Stale-age re-anchoring** (`requeued_at ?? created`, migration
`1779990300`): a reclaimed row's staleness clock restarts so it isn't
immediately re-expired.
## Review
7-agent CR + 7-agent confirmation round → **0 P0 / 0 P1**. Highlights:
- **Red-green GENUINE (empirically re-verified):** P1-A — a long-lived
job with lifetime `reclaim_count=3` but `consecutive_orphan_count=0` is
RE-QUEUED on a fresh orphan (pre-fix: wrongly deleted); P1-B — low
boundary at `MAX-1=2` re-queues, mutation-killed. Both non-tautological
(the test fake bumps `consecutive_orphan_count` only on reclaim, never
on steal; a dedicated pin test confirms steals don't consume budget).
- **No regression / terminates / fails-safe:** 248/248 tests, tsc +
oxlint clean; reclaim loop terminates at 3 consecutive orphans →
claim-delete; queue cannot grow unbounded; SweepResult metrics +
`reclaim_count` dashboard consumers (run-view `jobs.reclaimed`,
family-silence) unbroken.
- **Migration `1779990400`** additive, nullable, default-0;
pre-migration rows evaluate to 0 → safe re-queue path.
## Follow-on (not in this PR)
- Layer (b) graceful worker drain and layer (c) rolling restart are the
remaining phases of the redesign — see the Notion proposal:
https://app.notion.com/p/38b3aa381852817bacf5c9cda1f11cc0🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Tightens the **D4/BE chat-roundtrip probe gate** from `text.length > 0`
to a **provenance check**. A turn now only counts green if the assistant
text came from the `[data-testid="copilot-assistant-message"]` container
— not from the `<body>` fallback scrape (which picks up static page
chrome).
## Why
This is the gate weakness that **masked the BIA prod outage**: a dead
agent (RUN_STARTED→RUN_FINISHED with zero `TEXT_MESSAGE`) left the page
chrome on screen, the `<body>` scrape returned non-empty text, and the
old `text.length > 0` gate reported **green** while the agent was
actually broken. The new gate threads a `fromAssistantContainer` flag
(set true only on the testid-container read path) so a body-scrape-only
result fails (`!fromAssistantContainer` → red). The same guard is
applied at L4 (tool-rendering).
## Review
**7-agent CR → 0 P0 / 0 P1.** Highlights:
- **Red-green genuine** (empirical): old gate false-passes BIA (1/52
RED); post-fix 52/52. 3-way guard (green /
empty-via-unchanged-length-guard / BIA-via-provenance).
- **Blast radius: no false-fail risk** — all 20 integrations'
`agentic-chat` + `tool-rendering` demos use the shared v2 `CopilotChat`
which mounts the testid unconditionally; aimock `d4/<slug>/chat.json`
fixtures are text-bearing for all 20. Verdict: **ship as a blocking
gate, no soak.**
- **No lost coverage** — the `<body>`-scrape fallback *code* is retained
(diagnostics); only the gate *semantics* narrowed. No D4-probed cell
relied on the fallback to pass.
- **No cross-family regression** — change is confined to
`d4-chat-roundtrip.ts`; other probe drivers untouched; 169/0 tests pass;
tsc clean.
## Notes / follow-ups (non-blocking, P2)
- A *future* probed demo using a `CopilotChatAssistantMessage`
children-render override (which doesn't emit the testid) would
false-fail — worth a one-line note near the fallback comment if that
pattern is ever added to a D4 route.
Part of the 2026-06-26 incident remediation (companion to #5733); this
gate would have surfaced the BIA failure that the dashboard masked.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## What
Bundles the four code fixes from the 2026-06-26 showcase prod incident
remediation. (Prod config reconciles — fleet image promotes, worker
scaling, CVDIAG keys — were applied directly to Railway and are not
code; the post-incident debugging lesson shipped separately in #5727.)
## Incident context (why these exist)
Prod's coverage dashboard cascaded to a wall of red. Root cause was
**staleness, not a feature break**: the harness worker pool was starved
(a deploy-bounce + under-provisioning), so probe sweeps couldn't
complete within the staleness windows → cells aged out → rendered
red/BE✗ even though the apps were fine. Compounded by a `llamaindex`
container crash (per-request log flood tripping Railway's log cap) and a
2-day-stale prod image fleet. See #5727 for the debugging lesson, and
the Notion proposal below for the durable worker-reclamation redesign.
## The four fixes
1. **Dashboard stale-while-revalidate**
(`shell-dashboard/depth-chip.tsx`) — a passing (green) cell keeps its
color + a non-destructive `⟳` refreshing affordance during a re-probe
instead of flapping to grey; never-run stays grey; failure/regression
keeps its color with no spinner; staleness bound preserved. + a11y
(`role=status`, flag-gated regression label, `motion-reduce`).
2. **Family-silence grace window** (`harness/fleet/control-plane/*` +
dashboard banner) — a normal harness deploy's post-bounce worker drain
no longer fires a false "worker family silent" banner/alert (suppressed
within a `2×period` bounce grace keyed on the freshest worker
registration), while genuine silence beyond the window still fires on
both the Slack-alert and banner paths. No silent failure (PB-down →
grace disabled → real outages still alert).
3. **llamaindex per-request log gate**
(`integrations/llamaindex/.../route.ts`) — gates the chatty
`[copilotkit/route]` per-request logs behind `SHOWCASE_ROUTE_DEBUG`
(default off) so they can't flood Railway's log cap and kill the
container. Error logging untouched.
4. **SSOT worker-provisioning** (`scripts/railway-envs.ts` + drift gate)
— brings harness-workers replica provisioning under SSOT so prod/staging
can't silently drift. Models the **effective** field
(`multiRegionConfig.<region>.numReplicas`) +
`BROWSER_POOL_MAX_CONTEXTS`, declares current reconciled reality
(prod=staging=6 replicas, 40 contexts), with a CI drift-detection test.
## Review
7-agent CR round + 7-agent confirmation round → converged to **0 P0 / 0
P1**. Two post-CR fixes (SSOT effective-field model after discovering
`multiRegionConfig` is the live knob; re-anchoring a grace-edge test to
genuinely pin the 2× boundary) each re-confirmed clean. Suites:
shell-dashboard 92 + harness 63 + scripts 12 = **167 passed, 0 failed**.
No new tsc errors (pre-existing only).
## Notes / follow-ups (non-blocking)
- llamaindex log-gate is logging-only and untested (acceptable); the
chatty pattern exists in ~16 integrations incl. the gold standard — a
**fleet-wide log-flood gate** is a worthwhile follow-up.
- `freshestBounceMs` has two equivalent impls (harness `parseIso` /
dashboard `Date.parse`) — flagged for future lockstep.
- The D4/BE probe gate weakness (asserts `text.length>0`, which masked
the BIA outage) is being addressed in a **separate** PR
(`fix/d4-gate-tighten`).
## Refs
- Post-incident debugging lesson: #5727
- Worker reclamation + graceful-rollover redesign (Notion proposal):
https://app.notion.com/p/38b3aa381852817bacf5c9cda1f11cc0🤖 Generated with [Claude Code](https://claude.com/claude-code)
P1-A (cap semantics): the MAX_RECLAIM_ATTEMPTS cap was keyed on
`reclaim_count`, a LIFETIME tally bumped by BOTH the sweeper re-queue
path AND the peer-worker expired-lease steal (claim CAS). A long-lived
job that accrues benign peer steals could exhaust its 3-budget and then
get claim-DELETED on its first real orphan rather than re-queued.
Fix: introduce a dedicated `consecutive_orphan_count` column (migration
1779990400) that is bumped ONLY by the sweeper re-queue path in the
fleet-claim release CAS, and reset to 0 on every terminal done|failed
release. The peer-worker steal (claim CAS wasExpiredSteal branch) does
NOT touch this counter. The reaper's cap check now uses
`consecutive_orphan_count` instead of `reclaim_count`. `reclaim_count`
is left intact as the lifetime dashboard diagnostic (jobs.reclaimed).
P1-B (low boundary): adds a test at consecutive_orphan_count = MAX-1
(= 2) asserting the row is RE-QUEUED, not deleted. The off-by-one
mutation `>= MAX` -> `>= MAX-1` causes this test to go RED.
Test-fake honesty: `makeReclaimClaim`'s claimJob now explicitly models
the steal-bump on `reclaim_count` (matching the real hook) while
intentionally NOT bumping `consecutive_orphan_count`, and adds an
explicit pin test confirming steals do not consume the reclaim budget.
JSDoc on MAX_RECLAIM_ATTEMPTS updated to describe the correct semantics:
consecutive re-orphans scoped by sweeper re-queue, reset on terminal.
Red-green proof:
- P1-A RED: revert cap to reclaim_count → "P1-A CAP SCOPE" fails with
`expect(undefined).toBeDefined()` (job deleted instead of re-queued)
- P1-A GREEN: consecutive_orphan_count cap → test passes (re-queued)
- P1-B RED: mutate `>= MAX` to `>= MAX-1` → low-boundary test fails
- P1-B GREEN: revert mutation → low-boundary test passes
Suite: 145 queue-client + 103 producer = 248 total, all green.
Layer (a) of the worker reclamation+rollover redesign: make a worker bounce
non-lossy. The reaper's long-expired carve-out (G1d) used to claim-DELETE an
orphaned in-flight (claimed/running) row whose lease expired beyond its
family's stale window AND whose created-age was past that window —
`reclaimed=0, expiredPending++` — silently dropping work an abrupt bounce
(SIGKILL past grace / OOM / crash) left mid-flight.
Invert it to RECLAIM-WINS-UNTIL-CAP: re-queue the orphan to pending (it
re-runs; idempotent probes make at-least-once safe) until its durable
`reclaim_count` reaches MAX_RECLAIM_ATTEMPTS (3), only then claim-deleting a
row that keeps re-orphaning so a poison job cannot loop forever.
The carve-out existed to dodge an honesty bind: re-queueing a `created`-stale
row emitted a "back in flight" gray the next sweep falsified by claim-deleting
it off the renewal-immune `created` age. Dissolve the bind with a new
`requeued_at` column (migration 1779990300) the release CAS stamps on every
pending re-queue; both stale phases now age off `staleAgeAnchorMs`
(`requeued_at ?? created`), so a reclaimed row is genuinely young again and the
next sweep does not delete it. `reclaim_count` (migration 1779990200) is reused
as the attempt counter — no second tally to drift.
Red-green proven on the real reaper (queue-client.test.ts): a stale-aged
long-expired orphan below the cap goes RED (deleted, reclaimed=0,
expiredPending=1) on delete-wins and GREEN (re-queued, reclaimed=1,
expiredPending=0, requeued_at stamped) on reclaim-wins; plus an attempt-cap
deletion test and a next-sweep no-falsification test.
The L3 chat-roundtrip gate only asserted `text.length > 0`, which the
`<body>` fallback scrape can satisfy with static page text (nav links,
footer copy, demo blurb) trailing the sent message. When the agent run
finished with ZERO assistant content (RUN_STARTED -> RUN_FINISHED, no
TEXT_MESSAGE) the assistant-message bubble never rendered, so the probe
fell back to scraping <body> and false-PASSED on incidental page chrome —
the mechanism that masked the BIA outage (dead agent reported green).
Track the provenance of the captured response and require it to come from
the [data-testid="copilot-assistant-message"] container (a genuine
assistant turn — the DOM-layer equivalent of RUN_FINISHED + a non-empty
TEXT_MESSAGE), not the body fallback. Apply the same guard to L4 so weather
content must also come from a real assistant turn. The instrumented probe
sends the aimock fixture headers and renders real content into the
container, so this does not affect a working agent; it only rejects the
empty-turn false-pass.
Adds a red-green regression test reproducing the BIA false-pass.
## Summary
Two pre-existing isolation-CLI bugs that #5730 (slot-reaping) surfaced
and correctly deferred as out-of-subject — both real, now fixed with
real-surface red-green.
1. **Port-conflict guard silent-failure** — `_slot_ports_free` consumed
`_slot_offset_ports` via process substitution (`done < <(...)`), so a
`die` on an out-of-range/non-numeric slot exited only the subshell; the
loop read zero ports and the function returned 0 ("all free"), silently
defeating the port-conflict guard for a bad slot. Now captures into a
variable with `|| die` so the failure propagates to the caller.
2. **`--isolate` arg-parsing** — `--isolate <name> <slug>` (name before
slug) mis-parsed the name as the slug then died "Unexpected argument"; a
bare `--isolate=` silently auto-picked. Now order-independent (deferred
`pending_iso_name` resolution after the parse loop), rejects an empty
value loudly, and `--isolate=<name>` (non-numeric) binds an explicit
name. `--isolate=<N>` numeric pin unchanged. **Intended behavior
change:** `--isolate=<name>` previously died in the picker; it now binds
a name.
## Tests
`bats showcase/scripts/__tests__/` = 170/0 — real slot dirs, real
bad-slot input, real parser end-to-end (no mocks). CR converged round 1
(zero bucket-a), Procedure 3 promotion audit clean.
## Deferred to a potential follow-up (pre-existing "port-enumeration
completeness")
CR surfaced, and correctly kept out of scope: `_slot_offset_ports`
hardcoded/incomplete infra-port list (phantom 8081, misses other
published ports); apply_isolation port-offset regex matches quoted
short-form only; `.last-test-ts` not repointed per-isolate (concurrent
clobber); `--repeat` treats a following flag as its count.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The StateStore interface and the in-memory MemoryStore default remain;
durable backends can be reintroduced as a follow-up. Both adapter packages
were merged in #5613 but never published to npm, so removal is a clean
delete with no consumer impact.
- Delete packages/bot-store-redis and packages/bot-store-postgres.
- Revert the bot release scope and drift guard to bot + bot-ui.
- Strip the Redis dep, demo:restart script, restart demo, docker-compose,
and REDIS_URL env from examples/slack.
- Rewrite the bot persistence/transcripts docs around "MemoryStore default
+ implement the StateStore interface yourself for durability".
The harness-workers SSOT modeled only the top-level numReplicas, whose drift
gate watched a field that does not drive the live replica count. harness-workers
is single-region (us-west2); Railway derives the live count from
multiRegionConfig.us-west2.numReplicas.
- WorkerProvisioning gains effectiveReplicas (= multiRegionConfig.us-west2.
numReplicas), the authoritative field the drift gate now asserts. Top-level
numReplicas is retained as a documented mirror.
- Declared values reflect current reconciled reality (verified live via the
Railway GraphQL environment.config staged-config read): prod and staging both
at effectiveReplicas=6 (parity achieved by B-reconcile scaling prod 3 -> 6 in
both the top-level field and multiRegionConfig). BROWSER_POOL_MAX_CONTEXTS is
40 on both envs (verified live).
- Regenerated railway-envs.generated.json; drift-gate test asserts
effectiveReplicas (RED-GREEN proven). RAILWAY.md documents multiRegionConfig
as the effective knob and the achieved parity.
Add WorkerProvisioning interface and workerProvisioning field to the
harness-workers ServiceEntry in railway-envs.ts. Declares current live
reality: prod=3 replicas, staging=6 replicas, BROWSER_POOL_MAX_CONTEXTS=40
per worker (both envs).
Worker model: 1-worker-per-replica (Railway runs one process per
container, keyed on HOSTNAME). HARNESS_POOL_COUNT is informational
only — not a fork factor. Authoritative concurrency knob per worker is
BROWSER_POOL_MAX_CONTEXTS.
Comments flag: staging config-field drift (Railway field=2, live=6) as
a follow-up item; and the prod/staging parity decision (prod=3 vs
staging=6) as deliberately deferred.
Extends emit-railway-envs-json.ts to emit workerProvisioning into the
generated JSON snapshot. Adds drift-gate test
(harness-workers-provisioning.test.ts) that fails if SSOT numReplicas
diverges from the committed JSON snapshot — no live Railway API calls.
Red-green-red-green verified locally.
Updates RAILWAY.md with the 1-worker-per-replica model, declared
values, manual apply procedure, and drift-gate reference.
The existing tooling is verify-only for numReplicas; applying a replica
count change to Railway remains a manual operation (Railway Dashboard or
GraphQL API).
The llamaindex copilotkit API route logged '[copilotkit/route] POST ...'
and '[copilotkit/route] Response status: 200' unconditionally on EVERY
request. Under d6 probe fan-out this exceeded Railway's 500-logs/sec cap
('Messages dropped' -> 'Stopping Container'), killing the replica.
Gate both per-request console.log lines behind a SHOWCASE_ROUTE_DEBUG env
flag (default off). Module-load logs and error logging are unchanged.
This chatty pattern is shared/copied across ~16 integrations (including
the langgraph-python gold standard); this commit scopes the fix to
llamaindex. The others are flagged as follow-up.
A normal harness deploy rebuilds the shared showcase-harness image and
bounces the pool workers (PR #5715). Immediately after the bounce the
workers re-register, the producers re-arm, and every family is mid-sweep:
lastSuccessAt still points at the pre-bounce success, so it reads stale
against the silence thresholds (banner 2x period, Slack alert 3x period +
3 consecutive ticks). The result was a FALSE "worker family X has not
completed successfully" banner AND Slack family-silence alert during the
expected post-bounce drain window.
Fix: a bounce-keyed grace window. The freshest worker registered_at across
the /api/runs workers strip is the fleet's most-recent bounce instant
(independent of CP boot — a worker can bounce while the CP stays up). While
now - bounce < 2 x period, a family with no success yet is DRAINING, not
silent, so neither the §7.4 banner, the §7.3 cell glyph, nor the §9 Slack
alert flags it. Beyond the window with still-no-success, genuine silence
fires exactly as before.
The determination lives in two surfaces (server monitor for the Slack
alert; client isFamilySilent for the banner + glyph), so both now consume
the same new SSOT field (WorkerView.registeredAt) and the same 2x-period
grace constant, keeping them consistent.
- run-view.ts: project registered_at -> WorkerView.registeredAt (server)
- family-silence-monitor.ts: BOUNCE_GRACE_PERIOD_MULTIPLIER + freshest-bounce
grace gate, keyed off body.workers
- worker-runs-context.tsx: freshestBounceMs + bounceAtMs grace arg on
isFamilySilent; banner + cell glyph pass it
- ops-api.ts: WorkerView.registeredAt on the client DTO
A GREEN coverage cell flapped green -> grey -> green every time its probe
job's worker lease lapsed and the control-plane sweeper re-queued the job
(worker-reclaimed-pending). The data layer already preserves the
last-known-good colour in chipColor while surfaceState flips to "pending",
but depth-chip.tsx's `if (pending)` branch did a destructive grey
early-return that never read chipColor.
Split the pending branch:
- prior-good (chipColor is a real colour, or depth > 0): render the normal
coloured chip via the default branch's exact ternary (threading regression
through) PLUS a non-destructive refreshing affordance -- a corner ⟳ glyph
and a subtle pulsing ring in the chip's own colour, with
data-refreshing="true" / data-has-prior="true". Colour preserved.
- no-prior (never-run / first load: gray + depth 0): keep today's honest grey
⟳ chip, now tagged data-has-prior="false".
The unreachable red ⚡ overlay and the "failure never masked" gate are
untouched; red/regression cells pass through with no spinner. The refreshing
cue is conveyed by shape (⟳) + motion (ring), never by colour alone, and the
pulse/spin respect prefers-reduced-motion (motion-reduce:animate-none), so the
static ⟳ carries the meaning for colour-blind and reduced-motion operators.
Renderer-only change; unified-cell.tsx is the sole caller passing `pending`.
--isolate <name> <slug> (name before slug) mis-parsed the name as the
slug, then died "Unexpected argument"; a bare --isolate= silently
fell through to auto-pick. Defer the ambiguous post---isolate token to
pending_iso_name and resolve it after the parse loop (slug present =>
token was the name; no slug => token was the slug); reject an empty
--isolate= loudly; --isolate=<name> (non-numeric) now binds an explicit
isolate name. The --isolate=<N> numeric pin is unchanged.
_slot_ports_free consumed _slot_offset_ports via process substitution
(done < <(...)), so a die on an out-of-range/non-numeric slot exited only
the subshell — the loop read zero ports, any_held stayed 0, and the
function returned 0 ("all free"), silently defeating the port-conflict
guard for a bad slot. Capture into a variable with || die so the failure
propagates to the caller. Real-surface bats prove a bad slot now fails
loudly while valid-slot free/held behavior is unchanged.
Dry-run by default (lists the plan, changes nothing); --force executes,
--all ignores TTL/keep, --include-live opts into reaping a live-owner
target, <name|slot> targets one. Identifies harness-owned projects via
the slot-record / run-dir / showcase-iso<N> / self-id-label union, and
never touches the base 'showcase' stack or BuildKit resources. Real
docker bats prove dry-run/--force/--all + the base/buildkit guards.
Jordan Ritter