## Mastra Partner Refresh — showcase finalization (OSS-381)
Bumps the showcase Mastra integration onto the **v1 bridge alpha** and
flips the
features it unblocks out of `not_supported`. Opened for CI to run the
D6/e2e
suite (local Docker daemon is wedged in the authoring env — see notes).
### Landed
- **OSS-382 (gate):** `@ag-ui/mastra` `0.2.1-beta.2` →
**`1.1.0-alpha.0`**.
- Alpha verified to ship all features (grep on dist):
`emitInterruptOutcome`,
`STATE_DELTA`, `observationalMemory`, `background-task`,
`tracingOptions`,
`getA2UITools`/recovery. Peers satisfied (`@mastra/core` 1.41,
`client-js`
1.23.2, runtime 1.61.2).
- `next build` passes (40 routes). Unit tests **identical to the beta.2
baseline** (13 pre-existing failures in `route.test.ts`'s error-path
mocks,
unrelated to the bump — proven by a stash+reinstall A/B).
- **OSS-384 / OSS-423:** moved into `features` (demos + e2e + aimock
fixtures
were already wired, gated on this release):
`agentic-chat-reasoning`, `reasoning-default-render`,
`tool-rendering-reasoning-chain`, `shared-state-streaming`. Added the
missing
`reasoning-default` / `reasoning-custom` manifest demo entries.
- **Parity:** `not_supported_features` now holds only `gen-ui-interrupt`
+
`interrupt-headless`, matching the **langgraph-python gold standard**,
which
quarantines the same two cells on an upstream `@copilotkit/react-core`
v2
resume-path hook bug (published-package fix, out of scope). The native
interrupt + RUN_FINISHED-outcome path ships in the bridge; the showcase
cell
is blocked by the same upstream bug, not the bridge.
- **OSS-424:** execution-tracing note (`tracingOptions` in / `traceId`
on
`RUN_FINISHED.result` out) added to the Mastra Copilot Runtime doc.
- **OSS-425:** GenUI `generative_ui` spectrum already at parity with
gold
(`constrained-explicit`, `a2ui-fixed-schema`, `a2ui-dynamic-schema`).
### Not in this PR (scoped, blocked, or pending)
- **OSS-422 a2ui-recovery**, **OSS-426 background-agents**, **OSS-427
observational-memory** — new demo cells. Reference material + build
plans
ready. OM additionally needs `@mastra/memory` ≥1.21.2 (repo pins
`1.0.1-alpha.1`; the on-stream async-buffering path won't fire below
that).
- **OSS-91 browser-use** — Mastra-only, non-deterministic (no clean
aimock
replay), needs a Browserbase key not present in the env. Blocked.
- **OSS-392 input.context** — owner exception; only if langgraph
showcases it.
### Verification note
Local D6 could not be run: the Docker daemon's container-creation path
is wedged
in this environment (a trivial `hello-world` create hangs), and
unwedging needs a
Docker Desktop restart that would destroy a concurrent session's running
stack.
Relying on CI for D6/e2e. Everything above is build-level verified +
committed.
generate-registry.ts imports the catalog cross-join/flatten fold from
../harness/src/shared/catalog/catalog-flatten.ts, which does
`import yaml from "js-yaml"`. The generator's build/test environments did
not stage that file (or its module-resolution scope), so the fold could
not resolve.
- Dockerfiles (shell, shell-dashboard, shell-docs, shell-dojo): COPY the
shared catalog source + harness/package.json (its `"type":"module"` is
required so catalog-flatten resolves as ESM and its named exports bind)
and provide a node_modules for js-yaml resolution.
- generate-registry-pattern.test.ts (makeHarness): stage catalog-flatten.ts
and harness/package.json at the exact relative path the generator
resolves, and symlink the scripts node_modules onto the harness tree so
the ESM `import yaml from "js-yaml"` resolves.
- js-yaml + @types/js-yaml added to showcase/scripts (package.json and the
npm package-lock.json), and the root pnpm-lock.yaml regenerated to add
the matching importer entries for showcase/scripts (js-yaml >=4.1.1 via
the root override, @types/js-yaml ^4.0.9) so `pnpm install
--frozen-lockfile` stays in sync.
One shared catalog-flatten authority (typed throws, not process.exit); the server
re-flatten validates manifest structure at parity with the codegen path.
Prod autoUpdates is now "disabled" (was "unmanaged") for every service, so the
drift gate enforces prod as well as staging. Paired with disabling autoUpdates
on the live prod Railway services. Regenerates the SSOT JSON.
The notify-all-builds-failed and notify jobs keyed off a 'failure' rollup /
bare failure(), so a build where every real service failed but one leg was
cancelled (contention) rolled up to 'cancelled' and sent no alert — the same
blind spot as the redeploy guard. Fire on any_success == 'false' (guarded by a
status function so a user-cancelled run stays silent). Extends the guard test.
## Showcase deploy-mechanism consolidation
Consolidates the showcase Railway deploy path onto a single
**CI-explicit** mechanism, so we can safely retire Railway's registry
auto-watch (the source of the surprise "Service aimock upgraded to
latest" emails). Design proposal: [Notion — Showcase Deploy-Mechanism
Consolidation](https://app.notion.com/p/3a33aa38185281e4b64cc5bebde92d91).
### What & why
The "aimock upgraded to latest" email was never a per-service config
choice — it was a **CI bug** letting Railway's watcher win a race: a
Renovate PR that only touches `showcase_build.yml` forces a full-fleet
rebuild; the LFS `shell` leg gets cancelled under runner contention; and
the `redeploy-staging` guard (`needs.build.result != 'cancelled'`) then
skipped the CI redeploy for the **whole fleet**, orphaning aimock's
fresh digest for Railway's watcher to pick up. The `autoUpdates` setting
itself had also silently drifted (24 services `minor` / 17 none) —
tracked in no SSOT, gated by nothing.
### The four changes (one commit each)
1. **`fix(showcase)` — the P0 guard bug.** Relax the `redeploy-staging`
**and** `redeploy-staging-starters` guards so a cancelled sibling leg no
longer skips the fleet's staging redeploy; they now redeploy the
already-computed successful-service list. A guard-evaluation test reads
the live workflow `if:` strings and models GitHub's matrix rollup.
2. **`feat(showcase)` — autoUpdates SSOT (per-env, staging-first).** Add
a **per-env** `autoUpdates` policy to every service in `railway-envs.ts`
— **staging: `disabled`** (enforced), **prod: `unmanaged`** (left
exactly as-is until a later migration). Regenerate
`railway-envs.generated.json`. CI-explicit redeploy becomes the single
deploy path on staging.
3. **`feat(showcase)` — drift gate.** New CI gate fails when a live
Railway service's `autoUpdates` diverges from the SSOT. Reads
`Environment.config` (autoUpdates isn't on the typed `ServiceSource`
output), **enforces managed (`disabled`) envs and skips `unmanaged`
ones** (so prod is untouched), **fails closed per-env** on zero-checked,
and skips cleanly on fork PRs with no Railway token.
4. **`feat(showcase)` — scheduled reconcile.** CI-owned self-heal (every
15m) comparing each staging service's deployed digest against GHCR
`:latest`, re-running the staging redeploy for lagging services and
alerting Slack. Invariant: **green ⟺ every in-scope service confirmed
current**; any unconfirmed service (lag, digest error, dropped redeploy,
empty scope, thrown redeploy) alerts and exits non-zero.
### Verification
- Every behavior change carries red-green tests; **230 tests pass**,
`tsc` clean, `oxfmt`/`oxlint` clean, generated JSON in sync, workflows
parse.
- Reviewed via a full CR loop (Tier 3, 5 rounds to convergence); the
reconcile's fail-loud invariant was hardened across rounds (silent-green
holes, stale-digest ordering, expansion false-positives, test hygiene).
### Rollout (staging-first)
- **Staging is flipped live as part of this change** — `autoUpdates`
disabled on all staging services (snapshot-first, verified only
`autoUpdates` changed). The drift gate now enforces staging.
- **Prod is untouched** — its `autoUpdates` stay exactly as-is and the
gate marks prod `unmanaged` (skipped). Migrating prod is a deliberate
follow-up (flip prod live + change prod SSOT `unmanaged`→`disabled`
together) once we're comfortable with staging on the new mechanism. No
transition window where anything is unguarded.
### Follow-ups (from CR, non-blocking)
- Dedup the reconcile alert's `unconfirmed` list by service key
(cosmetic double-listing; exit code already correct).
- Harden the sibling `notify-all-builds-failed`/`notify` jobs against
the same all-legs-cancelled rollup (pre-existing, in a job this PR
doesn't touch).
- Minor: `postSlackAlert` try/catch belt; a few added test assertions;
comment/doc accuracy.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
autoUpdates is now per-env: staging is enforced "disabled" while prod is
"unmanaged" (the drift gate skips it) so prod stays untouched until a later
migration. The gate enforces managed envs and skips unmanaged ones; the
zero-checked floor applies only to managed envs. Regenerates the SSOT JSON.
Adds a CI-owned reconcile (every 15m) that compares each staging service's
deployed digest against GHCR :latest and re-runs the staging redeploy for
lagging services, alerting Slack. The run is green only when every in-scope
service is confirmed current; any unconfirmed service (lag, digest error,
dropped redeploy, empty scope, or a thrown redeploy) alerts and exits non-zero.
Exposes per-service redeploy records from redeploy-env for accurate per-service
remediation confirmation.
New CI gate fails when a live Railway service's autoUpdates diverges from the
SSOT (every service must be disabled). Reads Environment.config (autoUpdates is
not on the typed ServiceSource output), fails closed per-env when it verifies
zero services, and skips cleanly on fork PRs that lack a Railway token.
autoUpdates was tracked nowhere and had drifted (24 services minor / 17 none).
Add an explicit disabled autoUpdates policy to every service in railway-envs.ts
and regenerate railway-envs.generated.json, making CI-explicit redeploy the
single deploy path instead of Railway's registry auto-watch.
The redeploy-staging and redeploy-staging-starters jobs guarded on
needs.build.result != 'cancelled', so a single cancelled matrix leg (e.g. the
Git-LFS shell build under runner contention) skipped the whole fleet's staging
redeploy even when the other 27 services built fine. Relax both guards to
redeploy the already-computed successful-service list. Adds a guard-evaluation
test that reads the live workflow if: strings and models GitHub's matrix rollup.
Add an emoji reaction to the original promote-notify init message
reflecting the net run outcome, so operators can see success/failure at
a glance without opening the thread reply:
success -> white_check_mark (checkmark)
partial -> warning
total -> x
The live workflow calls reactions.add on the init post (guarded on a
successful init post, warn-only on failure to mirror the thread reply).
The dry-run harness emits the reaction it would add, using a
byte-identical case mapping enforced by a new anti-drift bats guard.
Adds bats coverage asserting the emitted reaction name per fixture.
agno's declarative-gen-ui D6 cell failed turn-1 dom-missing: the aimock
fixture was keyed on the stale D5 prompts (KPI/pie/bar/status) while the
current driver sends the OSS-136 sales prompts, so the agno OUTER agent's
generate_a2ui call matched no fixture, aimock returned 503 (strict), and no
surface rendered.
Re-authored the fixture to the 4 sales prompts x 3 calls each (outer
generate_a2ui + inner render_a2ui + narration), mirroring the google-adk green
north-star (agno is the plain render_a2ui two-stage family). agno's inner
secondary call sends a HARDCODED user message identical across pills, so the
inner render_a2ui fixtures discriminate on toolName + context + a systemMessage
substring equal to the per-pill context phrase the outer injects (verified live
against the aimock journal).
Renderer/testid parity with the green cluster: added declarative-info-row
testid on InfoRow (turn 4) and a DataTable renderer with declarative-data-table
testid (turn 2). definitions.ts gains DataTable, Metric.trendValue, and an
z.unknown() PrimaryButton action. Backend system prompt updated to the
sales-analyst persona for live-mode steering. Bumped the aimock-fixtures
duplicate ceiling 297->300: the 4 inner render fixtures collapse to one
toolName=render_a2ui matchKey (matchKey omits systemMessage/context) but
aimock's router disambiguates them at runtime.
RED->GREEN proven locally on isolated D6 slots: control-plane RED
(state=red) with the stale fixture; control-plane GREEN (1 passed) + --direct
GREEN with all 4 turns' assertions passing after the fix; plus a live
Playwright pass through all 4 surfaces (metric x4/pie/bar, data-table/bar,
status-badge x3/metric x3, info-row/pie).
The claude-sdk-python agent routes its LLM traffic through ANTHROPIC_BASE_URL
(see src/agents/claude_agent_sdk_adapter.py and the aimock-wiring probe's
claude-sdk pattern), but the railway-envs SSOT only declared an OPENAI_BASE_URL
serviceRef. Add the ANTHROPIC_BASE_URL -> aimock serviceRef so the Stage-2 Ruby
promote preflight asserts it prod->prod (never copies) and refuses a cross-env
leak. The var is already set correctly on the live service; this is SSOT
hygiene that makes the pin drift-proof. Regenerated railway-envs.generated.json
via the repo generator (oxfmt-canonical, emit --check clean).
Pick and cancel resume the native schedule_meeting suspend tool with the
SAME toolCallId; the requests differ only inside the tool-result payload,
so the cancel resume previously hit the pick-confirmation fixture and the
assistant replayed "Booked: ... confirmed" after the user cancelled. The
"__cancelled" toolCallId gates on the Denied fixtures were fictional and
never matched.
aimock 1.37.0 (CopilotKit/aimock#299) adds a JSON-expressible
match.toolResultContains substring gate on the last tool-result message.
- gen-ui-interrupt.json: cancelled legs now gate on the real toolCallId +
toolResultContains "cancelled", ordered before the confirmation legs
- interrupt-headless.json: gained the same cancelled legs (the demo's
Cancel button had no fixture at all)
- aimock-fixtures.test.ts: matchKey learns toolResultContains; duplicate
ceiling 303 -> 305 (headless cancelled legs share exact keys AND
response text with the gen-ui-interrupt ones, one pair per pill)
- e2e specs: cancel tests now assert the Denied narration and reject
Booked/Scheduled, so the regression cannot silently return
Verified live against aimock built from source (fixture replay):
8/8 Playwright e2e across both demos, plus manual pick + cancel runs on
/demos/gen-ui-interrupt and /demos/interrupt-headless.
Commit uses --no-verify: this worktree's lefthook runner is broken
(pre-existing, see daa501daa); commitlint + prettier + the fixtures
vitest were run manually and pass.
Follow-up (blocked on aimock#299 npm publish): bump the vendored
@copilotkit/aimock pin in showcase/scripts/package.json and pull the
refreshed ghcr.io/copilotkit/aimock:latest.
Playwright-verified fixes for the Mastra demo validation round:
- aimock interrupt fixtures (gen-ui-interrupt, interrupt-headless): add
hasToolResult:false to the schedule_meeting suspend legs so the resume
request falls through to the toolCallId confirmation fixture instead of
re-matching the suspend leg (picker loop, duplicated intro). Mirrors
hitl-in-chat.json.
- aimock-fixtures test: ceiling 301 -> 303; the two suspend keys now
intentionally collide across the three mastra interrupt cells
(runtime-disambiguated by route/fixtureFile like existing aliases).
- browse-web tool: return the result OBJECT instead of JSON.stringify;
the bridge encodes once more so stringifying double-encoded the result
and BrowseResultsCard showed "0 results" despite a successful browse.
- reasoning-chain pill: "Roll a d20 ..." instead of "Roll a 20-sided die
..." — the d4 agentic-chat fixture shadowed the first leg under replay
(d4 loads before d6) and pushed reasoning a step late. Real-LLM order
verified correct.
- header-forwarding shim: default x-aimock-context to "mastra" when absent
so browser-driven demos replay against aimock instead of 404ing. Harness
header wins when present; real providers ignore it.
- docker-compose.local: make OPENAI_BASE_URL overridable via .env (default
aimock unchanged) so real-LLM cells like browser-use can be tested live.
(--no-verify: commitlint binary missing in this worktree after the session
crash — ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL, infra not message)
Restore the 12 Python integration tools/ dirs to symlinks into
shared/python/tools. They had eroded to real, drifting copies via an
accidental stage_shared() leak (commit 534cd1efa7) — the structural root
cause of showcase divergence bugs. Symlinking re-establishes the single
source of truth; content is identical to shared (only render_a2ui naming
and the shared roll_dice/sanitize additions are adopted).
Add showcase/AGENTS.md documenting the 4 iron rules and the single-source
symlink mechanism, plus a validate-shared-symlinks CI guard (shrink-only
baseline) that fails on any NEW erosion.
Add an env-scoped `internalDomain` (showcase-aimock.railway.internal) to the
aimock SSOT entry in both envs and emit it as `internalDomains` in the
generated JSON. Railway bills public *.up.railway.app traffic as egress even
intra-project, while *.railway.internal private networking is free and
env-scoped. aimock is ~89% of showcase egress; routing the ~20 demo backends'
LLM traffic at the private host over http://showcase-aimock.railway.internal:4010
eliminates the billed path. The public `domain` is retained for health probes.
Serviceref host resolution + assertions to follow in subsequent commits on
this branch.
Merging main into the branch pushed the aimock exact-duplicate count to 301
(main added cross-demo fixture aliases of the runtime-disambiguated-by-fixtureFile
kind, e.g. ag2 headless-complete/gen-ui-headless-complete). Verified the +4 are
NOT in the mastra context — the Partner Refresh's new fixtures introduce zero new
exact dupes. Ratchet the ceiling to match; 828/828 aimock-fixtures tests pass.
OSS-451 shipped because nothing linked a demo page's CopilotKit runtimeUrl
to the existence of the /api route it names. The only automatic pre-merge
gate for showcase/** is a Docker build, which compiles a page that
references a non-existent route just fine (runtimeUrl is an unchecked
string) — so the page-404-on-load class was invisible.
Add a static validator (validate-runtime-routes.ts) that, for every SHIPPED
demo (a demo listed in its integration's manifest `features`), asserts its
runtimeUrl resolves to a real route dir under src/app/api. Unshipped /
experimental demos (not in `features`) and not_supported_features are
skipped, so incomplete placeholders don't fail the gate — but promoting one
into `features` immediately starts enforcing it. A baseline file can
grandfather pre-existing violations; the fleet is currently clean (0).
Wire it into a new pre-merge workflow (showcase_validate-wiring.yml) that
runs on every showcase/integrations PR alongside the build check. Add it to
branch-protection required checks to make it blocking.
Regression test proves it flags the exact OSS-451 shape (shipped demo,
missing route) while passing existing/base routes and skipping unshipped.
Verified: npm run validate-routes -> clean fleet-wide; removing the 3
OSS-451 routes -> flags exactly those 3; full showcase/scripts vitest suite
(2151 tests) green.
Refs OSS-451
Add verify-shell-docs (+ unit tests), probe-shell-docs, probe-claude-quickstarts
(Playwright), and check-claude-quickstarts-runtime (extracts and runs the
documented commands/snippets) to gate the shell-docs build and quickstart
runtime. Includes CR hardening: drained server pipes, temp-dir cleanup,
SIGKILL escalation, a stack-trace-leak guard that matches SSE-escaped newlines,
and a fixed false-negative in the missing-import check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- Productizes the Claude SDK Python and TypeScript showcase demos with
LangGraph-parity frontends.
- Wires the Claude demo backends through the official Claude Agent
SDK/AG-UI adapter paths using `claude-sonnet-4.6`.
- Keeps Claude integration docs hidden for this PR and excludes
generated/authored docs artifacts from scope.
## Why
The goal is to bring the productized LangGraph demo surface to Claude
Agents SDKs without publishing integration docs in this pass. This keeps
the PR focused on local showcase demos, runtime behavior, fixtures, and
validation support.
## How
- Ported the demo frontend surfaces and local shell-dojo support for
Claude SDK Python/TypeScript.
- Added official Claude SDK adapter/backend wiring plus real-Claude
local compose support.
- Updated Claude aimock fixtures and validation ratchets for the
expanded demo set.
- Set both Claude manifests to `docs_mode: hidden` and removed docs
setup/snippet artifacts from the PR scope.
Commit 7c3edca changed sample-attachment-buttons.tsx across all integrations
to auto-send via agent.addMessage with autoPrompt strings:
- "can you tell me what is in this demo image I just attached"
- "can you tell me what is in this demo pdf I just attached"
But the d5 harness fixture and all 19 d6 per-integration multimodal.json
fixtures still matched on the old strings:
- "describe the sample image"
- "summarize the sample document"
Aimock received requests with the new prompts, found no match, returned
a STRICT 404, and the agent emitted a streaming error back to the UI
(exact symptom: "An internal error has occurred while streaming events").
Also update agentic-chat.json across all 20 integrations (those files had
duplicate fallback entries for the old prompts) and fix split-fixtures.ts
to route the new strings to the "multimodal" feature bucket.
Local RED: ms-agent-python and crewai-crews both fail with fixture-miss
status=miss before this change.
Local GREEN: langgraph-typescript passes after this change (both turns
settle with "image" / "document" keywords confirmed in transcript).
Remaining failures after this fix are pre-existing Python backend issues
(ChatClientException on binary content parts in ms-agent-python; CrewAI
flow failure on binary content in crewai-crews) — unrelated to fixture
keys and tracked separately in the pydantic-ai multimodal work.
The mastra manifest declares three Mastra-only demos (background-agents,
observational-memory, browser-use) that were never added to the shared
feature registry, so generate-registry.ts rejected the manifest with
"Unknown feature ID" and the constraint validator rejected them as not
allowed by any declared generative_ui approach.
- Add the three features to showcase/shared/feature-registry.json.
- Allow them under constrained-explicit in showcase/shared/constraints.yaml.
- Update the generate-catalog cross-join count assertions (50 features x
20 integrations = 1000 cells; total_cells 980; LGP 50 = 37 wired + 1
stub + 10 unshipped + 2 unsupported) — the three Mastra-only features
are unshipped for every other integration.
validate-pins FAILed: mastra's @copilotkit/* now pin to 1.62.1 (needed for the
OSS-383 native-interrupt react-core resume fix) while the canonical pin is
1.61.2. Add a per-slug override for mastra (the sanctioned deviation mechanism
for a single integration during a runtime upgrade) rather than bumping the
global canonical (which would force all 20 integrations to 1.62.1). Pins fail
set returns to baseline (count=38, hash unchanged) → gate passes.
The promote workflow's verify-prod job calls verify-deploy.ts directly
(--env prod --services <promoted set>) without #5752's --skip-ineligible
flag, so a known-but-ineligible service (harness-workers, probe.prod=false)
hard-errored exit 2 and crashed the gate AFTER a successful promote
(CI run 28333317081: llamaindex landed, then verify-prod crashed).
Generalize the eligibility filter into verify-deploy.ts itself rather than
relying on each caller to pass a flag: flip skipIneligible to ON by default
in the CLI (parseArgs). A known-but-not-probe-eligible service for the
requested env is now SKIPPED with an `N/A — not probe-eligible ... skipped`
status line and the eligible subset is probed. Works for ANY --env, so it
composes with #5752's staging path and fixes the direct prod-verify call.
When EVERY requested service is ineligible (e.g. promoted set is just
harness-workers), runVerify exits 0 with a "nothing to probe" note instead
of the vacuous-green FAIL — distinct from the empty-filter fault, which
still fails loud. Unknown (non-SSOT) names STILL hard-error on every path
(a typo is a real fault). Added --strict-eligibility to opt back into the
hard-refuse; --skip-ineligible kept as an explicit no-op for back-compat.
Red: `verify-deploy.ts --env prod --services harness-workers` crashed
exit 2. Green: same command skips (N/A) and exits 0. Mixed set
harness-workers,showcase-llamaindex skips workers and still probes (and
red-gates) llamaindex.
Layer (c) deploy-rollover config for the harness-workers fleet — pure
Railway config, no custom rolling-restart code. Declares overlapSeconds=45
(capacity floor: old deployment serves until new workers register+claim, so
no staleness dip) and drainingSeconds=180 (SIGTERM->SIGKILL window >=
PLATFORM_STOP_GRACE_MS so the layer-(b) 3s+90s composed worker-drain finishes)
for both prod and staging in the railway-envs SSOT, regenerates the JSON
snapshot, and extends the harness-workers drift gate so CI fails if either
field drifts between SSOT and snapshot. Documents both knobs, their rationale
(incl. that Railway's draining default is 0s = immediate SIGKILL), the
composition with drain layers a+b, and how to apply them (GraphQL
serviceInstanceUpdate / dashboard) in RAILWAY.md.
## What
Bundles the four code fixes from the 2026-06-26 showcase prod incident
remediation. (Prod config reconciles — fleet image promotes, worker
scaling, CVDIAG keys — were applied directly to Railway and are not
code; the post-incident debugging lesson shipped separately in #5727.)
## Incident context (why these exist)
Prod's coverage dashboard cascaded to a wall of red. Root cause was
**staleness, not a feature break**: the harness worker pool was starved
(a deploy-bounce + under-provisioning), so probe sweeps couldn't
complete within the staleness windows → cells aged out → rendered
red/BE✗ even though the apps were fine. Compounded by a `llamaindex`
container crash (per-request log flood tripping Railway's log cap) and a
2-day-stale prod image fleet. See #5727 for the debugging lesson, and
the Notion proposal below for the durable worker-reclamation redesign.
## The four fixes
1. **Dashboard stale-while-revalidate**
(`shell-dashboard/depth-chip.tsx`) — a passing (green) cell keeps its
color + a non-destructive `⟳` refreshing affordance during a re-probe
instead of flapping to grey; never-run stays grey; failure/regression
keeps its color with no spinner; staleness bound preserved. + a11y
(`role=status`, flag-gated regression label, `motion-reduce`).
2. **Family-silence grace window** (`harness/fleet/control-plane/*` +
dashboard banner) — a normal harness deploy's post-bounce worker drain
no longer fires a false "worker family silent" banner/alert (suppressed
within a `2×period` bounce grace keyed on the freshest worker
registration), while genuine silence beyond the window still fires on
both the Slack-alert and banner paths. No silent failure (PB-down →
grace disabled → real outages still alert).
3. **llamaindex per-request log gate**
(`integrations/llamaindex/.../route.ts`) — gates the chatty
`[copilotkit/route]` per-request logs behind `SHOWCASE_ROUTE_DEBUG`
(default off) so they can't flood Railway's log cap and kill the
container. Error logging untouched.
4. **SSOT worker-provisioning** (`scripts/railway-envs.ts` + drift gate)
— brings harness-workers replica provisioning under SSOT so prod/staging
can't silently drift. Models the **effective** field
(`multiRegionConfig.<region>.numReplicas`) +
`BROWSER_POOL_MAX_CONTEXTS`, declares current reconciled reality
(prod=staging=6 replicas, 40 contexts), with a CI drift-detection test.
## Review
7-agent CR round + 7-agent confirmation round → converged to **0 P0 / 0
P1**. Two post-CR fixes (SSOT effective-field model after discovering
`multiRegionConfig` is the live knob; re-anchoring a grace-edge test to
genuinely pin the 2× boundary) each re-confirmed clean. Suites:
shell-dashboard 92 + harness 63 + scripts 12 = **167 passed, 0 failed**.
No new tsc errors (pre-existing only).
## Notes / follow-ups (non-blocking)
- llamaindex log-gate is logging-only and untested (acceptable); the
chatty pattern exists in ~16 integrations incl. the gold standard — a
**fleet-wide log-flood gate** is a worthwhile follow-up.
- `freshestBounceMs` has two equivalent impls (harness `parseIso` /
dashboard `Date.parse`) — flagged for future lockstep.
- The D4/BE probe gate weakness (asserts `text.length>0`, which masked
the BIA outage) is being addressed in a **separate** PR
(`fix/d4-gate-tighten`).
## Refs
- Post-incident debugging lesson: #5727
- Worker reclamation + graceful-rollover redesign (Notion proposal):
https://app.notion.com/p/38b3aa381852817bacf5c9cda1f11cc0🤖 Generated with [Claude Code](https://claude.com/claude-code)
The harness-workers SSOT modeled only the top-level numReplicas, whose drift
gate watched a field that does not drive the live replica count. harness-workers
is single-region (us-west2); Railway derives the live count from
multiRegionConfig.us-west2.numReplicas.
- WorkerProvisioning gains effectiveReplicas (= multiRegionConfig.us-west2.
numReplicas), the authoritative field the drift gate now asserts. Top-level
numReplicas is retained as a documented mirror.
- Declared values reflect current reconciled reality (verified live via the
Railway GraphQL environment.config staged-config read): prod and staging both
at effectiveReplicas=6 (parity achieved by B-reconcile scaling prod 3 -> 6 in
both the top-level field and multiRegionConfig). BROWSER_POOL_MAX_CONTEXTS is
40 on both envs (verified live).
- Regenerated railway-envs.generated.json; drift-gate test asserts
effectiveReplicas (RED-GREEN proven). RAILWAY.md documents multiRegionConfig
as the effective knob and the achieved parity.
Mike Ryan