autoUpdates was tracked nowhere and had drifted (24 services minor / 17 none).
Add an explicit disabled autoUpdates policy to every service in railway-envs.ts
and regenerate railway-envs.generated.json, making CI-explicit redeploy the
single deploy path instead of Railway's registry auto-watch.
The redeploy-staging and redeploy-staging-starters jobs guarded on
needs.build.result != 'cancelled', so a single cancelled matrix leg (e.g. the
Git-LFS shell build under runner contention) skipped the whole fleet's staging
redeploy even when the other 27 services built fine. Relax both guards to
redeploy the already-computed successful-service list. Adds a guard-evaluation
test that reads the live workflow if: strings and models GitHub's matrix rollup.
Add an emoji reaction to the original promote-notify init message
reflecting the net run outcome, so operators can see success/failure at
a glance without opening the thread reply:
success -> white_check_mark (checkmark)
partial -> warning
total -> x
The live workflow calls reactions.add on the init post (guarded on a
successful init post, warn-only on failure to mirror the thread reply).
The dry-run harness emits the reaction it would add, using a
byte-identical case mapping enforced by a new anti-drift bats guard.
Adds bats coverage asserting the emitted reaction name per fixture.
agno's declarative-gen-ui D6 cell failed turn-1 dom-missing: the aimock
fixture was keyed on the stale D5 prompts (KPI/pie/bar/status) while the
current driver sends the OSS-136 sales prompts, so the agno OUTER agent's
generate_a2ui call matched no fixture, aimock returned 503 (strict), and no
surface rendered.
Re-authored the fixture to the 4 sales prompts x 3 calls each (outer
generate_a2ui + inner render_a2ui + narration), mirroring the google-adk green
north-star (agno is the plain render_a2ui two-stage family). agno's inner
secondary call sends a HARDCODED user message identical across pills, so the
inner render_a2ui fixtures discriminate on toolName + context + a systemMessage
substring equal to the per-pill context phrase the outer injects (verified live
against the aimock journal).
Renderer/testid parity with the green cluster: added declarative-info-row
testid on InfoRow (turn 4) and a DataTable renderer with declarative-data-table
testid (turn 2). definitions.ts gains DataTable, Metric.trendValue, and an
z.unknown() PrimaryButton action. Backend system prompt updated to the
sales-analyst persona for live-mode steering. Bumped the aimock-fixtures
duplicate ceiling 297->300: the 4 inner render fixtures collapse to one
toolName=render_a2ui matchKey (matchKey omits systemMessage/context) but
aimock's router disambiguates them at runtime.
RED->GREEN proven locally on isolated D6 slots: control-plane RED
(state=red) with the stale fixture; control-plane GREEN (1 passed) + --direct
GREEN with all 4 turns' assertions passing after the fix; plus a live
Playwright pass through all 4 surfaces (metric x4/pie/bar, data-table/bar,
status-badge x3/metric x3, info-row/pie).
The claude-sdk-python agent routes its LLM traffic through ANTHROPIC_BASE_URL
(see src/agents/claude_agent_sdk_adapter.py and the aimock-wiring probe's
claude-sdk pattern), but the railway-envs SSOT only declared an OPENAI_BASE_URL
serviceRef. Add the ANTHROPIC_BASE_URL -> aimock serviceRef so the Stage-2 Ruby
promote preflight asserts it prod->prod (never copies) and refuses a cross-env
leak. The var is already set correctly on the live service; this is SSOT
hygiene that makes the pin drift-proof. Regenerated railway-envs.generated.json
via the repo generator (oxfmt-canonical, emit --check clean).
Pick and cancel resume the native schedule_meeting suspend tool with the
SAME toolCallId; the requests differ only inside the tool-result payload,
so the cancel resume previously hit the pick-confirmation fixture and the
assistant replayed "Booked: ... confirmed" after the user cancelled. The
"__cancelled" toolCallId gates on the Denied fixtures were fictional and
never matched.
aimock 1.37.0 (CopilotKit/aimock#299) adds a JSON-expressible
match.toolResultContains substring gate on the last tool-result message.
- gen-ui-interrupt.json: cancelled legs now gate on the real toolCallId +
toolResultContains "cancelled", ordered before the confirmation legs
- interrupt-headless.json: gained the same cancelled legs (the demo's
Cancel button had no fixture at all)
- aimock-fixtures.test.ts: matchKey learns toolResultContains; duplicate
ceiling 303 -> 305 (headless cancelled legs share exact keys AND
response text with the gen-ui-interrupt ones, one pair per pill)
- e2e specs: cancel tests now assert the Denied narration and reject
Booked/Scheduled, so the regression cannot silently return
Verified live against aimock built from source (fixture replay):
8/8 Playwright e2e across both demos, plus manual pick + cancel runs on
/demos/gen-ui-interrupt and /demos/interrupt-headless.
Commit uses --no-verify: this worktree's lefthook runner is broken
(pre-existing, see daa501daa); commitlint + prettier + the fixtures
vitest were run manually and pass.
Follow-up (blocked on aimock#299 npm publish): bump the vendored
@copilotkit/aimock pin in showcase/scripts/package.json and pull the
refreshed ghcr.io/copilotkit/aimock:latest.
Playwright-verified fixes for the Mastra demo validation round:
- aimock interrupt fixtures (gen-ui-interrupt, interrupt-headless): add
hasToolResult:false to the schedule_meeting suspend legs so the resume
request falls through to the toolCallId confirmation fixture instead of
re-matching the suspend leg (picker loop, duplicated intro). Mirrors
hitl-in-chat.json.
- aimock-fixtures test: ceiling 301 -> 303; the two suspend keys now
intentionally collide across the three mastra interrupt cells
(runtime-disambiguated by route/fixtureFile like existing aliases).
- browse-web tool: return the result OBJECT instead of JSON.stringify;
the bridge encodes once more so stringifying double-encoded the result
and BrowseResultsCard showed "0 results" despite a successful browse.
- reasoning-chain pill: "Roll a d20 ..." instead of "Roll a 20-sided die
..." — the d4 agentic-chat fixture shadowed the first leg under replay
(d4 loads before d6) and pushed reasoning a step late. Real-LLM order
verified correct.
- header-forwarding shim: default x-aimock-context to "mastra" when absent
so browser-driven demos replay against aimock instead of 404ing. Harness
header wins when present; real providers ignore it.
- docker-compose.local: make OPENAI_BASE_URL overridable via .env (default
aimock unchanged) so real-LLM cells like browser-use can be tested live.
(--no-verify: commitlint binary missing in this worktree after the session
crash — ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL, infra not message)
Restore the 12 Python integration tools/ dirs to symlinks into
shared/python/tools. They had eroded to real, drifting copies via an
accidental stage_shared() leak (commit 534cd1efa7) — the structural root
cause of showcase divergence bugs. Symlinking re-establishes the single
source of truth; content is identical to shared (only render_a2ui naming
and the shared roll_dice/sanitize additions are adopted).
Add showcase/AGENTS.md documenting the 4 iron rules and the single-source
symlink mechanism, plus a validate-shared-symlinks CI guard (shrink-only
baseline) that fails on any NEW erosion.
Add an env-scoped `internalDomain` (showcase-aimock.railway.internal) to the
aimock SSOT entry in both envs and emit it as `internalDomains` in the
generated JSON. Railway bills public *.up.railway.app traffic as egress even
intra-project, while *.railway.internal private networking is free and
env-scoped. aimock is ~89% of showcase egress; routing the ~20 demo backends'
LLM traffic at the private host over http://showcase-aimock.railway.internal:4010
eliminates the billed path. The public `domain` is retained for health probes.
Serviceref host resolution + assertions to follow in subsequent commits on
this branch.
Merging main into the branch pushed the aimock exact-duplicate count to 301
(main added cross-demo fixture aliases of the runtime-disambiguated-by-fixtureFile
kind, e.g. ag2 headless-complete/gen-ui-headless-complete). Verified the +4 are
NOT in the mastra context — the Partner Refresh's new fixtures introduce zero new
exact dupes. Ratchet the ceiling to match; 828/828 aimock-fixtures tests pass.
OSS-451 shipped because nothing linked a demo page's CopilotKit runtimeUrl
to the existence of the /api route it names. The only automatic pre-merge
gate for showcase/** is a Docker build, which compiles a page that
references a non-existent route just fine (runtimeUrl is an unchecked
string) — so the page-404-on-load class was invisible.
Add a static validator (validate-runtime-routes.ts) that, for every SHIPPED
demo (a demo listed in its integration's manifest `features`), asserts its
runtimeUrl resolves to a real route dir under src/app/api. Unshipped /
experimental demos (not in `features`) and not_supported_features are
skipped, so incomplete placeholders don't fail the gate — but promoting one
into `features` immediately starts enforcing it. A baseline file can
grandfather pre-existing violations; the fleet is currently clean (0).
Wire it into a new pre-merge workflow (showcase_validate-wiring.yml) that
runs on every showcase/integrations PR alongside the build check. Add it to
branch-protection required checks to make it blocking.
Regression test proves it flags the exact OSS-451 shape (shipped demo,
missing route) while passing existing/base routes and skipping unshipped.
Verified: npm run validate-routes -> clean fleet-wide; removing the 3
OSS-451 routes -> flags exactly those 3; full showcase/scripts vitest suite
(2151 tests) green.
Refs OSS-451
Add verify-shell-docs (+ unit tests), probe-shell-docs, probe-claude-quickstarts
(Playwright), and check-claude-quickstarts-runtime (extracts and runs the
documented commands/snippets) to gate the shell-docs build and quickstart
runtime. Includes CR hardening: drained server pipes, temp-dir cleanup,
SIGKILL escalation, a stack-trace-leak guard that matches SSE-escaped newlines,
and a fixed false-negative in the missing-import check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
## Summary
- Productizes the Claude SDK Python and TypeScript showcase demos with
LangGraph-parity frontends.
- Wires the Claude demo backends through the official Claude Agent
SDK/AG-UI adapter paths using `claude-sonnet-4.6`.
- Keeps Claude integration docs hidden for this PR and excludes
generated/authored docs artifacts from scope.
## Why
The goal is to bring the productized LangGraph demo surface to Claude
Agents SDKs without publishing integration docs in this pass. This keeps
the PR focused on local showcase demos, runtime behavior, fixtures, and
validation support.
## How
- Ported the demo frontend surfaces and local shell-dojo support for
Claude SDK Python/TypeScript.
- Added official Claude SDK adapter/backend wiring plus real-Claude
local compose support.
- Updated Claude aimock fixtures and validation ratchets for the
expanded demo set.
- Set both Claude manifests to `docs_mode: hidden` and removed docs
setup/snippet artifacts from the PR scope.
Commit 7c3edca changed sample-attachment-buttons.tsx across all integrations
to auto-send via agent.addMessage with autoPrompt strings:
- "can you tell me what is in this demo image I just attached"
- "can you tell me what is in this demo pdf I just attached"
But the d5 harness fixture and all 19 d6 per-integration multimodal.json
fixtures still matched on the old strings:
- "describe the sample image"
- "summarize the sample document"
Aimock received requests with the new prompts, found no match, returned
a STRICT 404, and the agent emitted a streaming error back to the UI
(exact symptom: "An internal error has occurred while streaming events").
Also update agentic-chat.json across all 20 integrations (those files had
duplicate fallback entries for the old prompts) and fix split-fixtures.ts
to route the new strings to the "multimodal" feature bucket.
Local RED: ms-agent-python and crewai-crews both fail with fixture-miss
status=miss before this change.
Local GREEN: langgraph-typescript passes after this change (both turns
settle with "image" / "document" keywords confirmed in transcript).
Remaining failures after this fix are pre-existing Python backend issues
(ChatClientException on binary content parts in ms-agent-python; CrewAI
flow failure on binary content in crewai-crews) — unrelated to fixture
keys and tracked separately in the pydantic-ai multimodal work.
The mastra manifest declares three Mastra-only demos (background-agents,
observational-memory, browser-use) that were never added to the shared
feature registry, so generate-registry.ts rejected the manifest with
"Unknown feature ID" and the constraint validator rejected them as not
allowed by any declared generative_ui approach.
- Add the three features to showcase/shared/feature-registry.json.
- Allow them under constrained-explicit in showcase/shared/constraints.yaml.
- Update the generate-catalog cross-join count assertions (50 features x
20 integrations = 1000 cells; total_cells 980; LGP 50 = 37 wired + 1
stub + 10 unshipped + 2 unsupported) — the three Mastra-only features
are unshipped for every other integration.
validate-pins FAILed: mastra's @copilotkit/* now pin to 1.62.1 (needed for the
OSS-383 native-interrupt react-core resume fix) while the canonical pin is
1.61.2. Add a per-slug override for mastra (the sanctioned deviation mechanism
for a single integration during a runtime upgrade) rather than bumping the
global canonical (which would force all 20 integrations to 1.62.1). Pins fail
set returns to baseline (count=38, hash unchanged) → gate passes.
The promote workflow's verify-prod job calls verify-deploy.ts directly
(--env prod --services <promoted set>) without #5752's --skip-ineligible
flag, so a known-but-ineligible service (harness-workers, probe.prod=false)
hard-errored exit 2 and crashed the gate AFTER a successful promote
(CI run 28333317081: llamaindex landed, then verify-prod crashed).
Generalize the eligibility filter into verify-deploy.ts itself rather than
relying on each caller to pass a flag: flip skipIneligible to ON by default
in the CLI (parseArgs). A known-but-not-probe-eligible service for the
requested env is now SKIPPED with an `N/A — not probe-eligible ... skipped`
status line and the eligible subset is probed. Works for ANY --env, so it
composes with #5752's staging path and fixes the direct prod-verify call.
When EVERY requested service is ineligible (e.g. promoted set is just
harness-workers), runVerify exits 0 with a "nothing to probe" note instead
of the vacuous-green FAIL — distinct from the empty-filter fault, which
still fails loud. Unknown (non-SSOT) names STILL hard-error on every path
(a typo is a real fault). Added --strict-eligibility to opt back into the
hard-refuse; --skip-ineligible kept as an explicit no-op for back-compat.
Red: `verify-deploy.ts --env prod --services harness-workers` crashed
exit 2. Green: same command skips (N/A) and exits 0. Mixed set
harness-workers,showcase-llamaindex skips workers and still probes (and
red-gates) llamaindex.
Layer (c) deploy-rollover config for the harness-workers fleet — pure
Railway config, no custom rolling-restart code. Declares overlapSeconds=45
(capacity floor: old deployment serves until new workers register+claim, so
no staleness dip) and drainingSeconds=180 (SIGTERM->SIGKILL window >=
PLATFORM_STOP_GRACE_MS so the layer-(b) 3s+90s composed worker-drain finishes)
for both prod and staging in the railway-envs SSOT, regenerates the JSON
snapshot, and extends the harness-workers drift gate so CI fails if either
field drifts between SSOT and snapshot. Documents both knobs, their rationale
(incl. that Railway's draining default is 0s = immediate SIGKILL), the
composition with drain layers a+b, and how to apply them (GraphQL
serviceInstanceUpdate / dashboard) in RAILWAY.md.
## What
Bundles the four code fixes from the 2026-06-26 showcase prod incident
remediation. (Prod config reconciles — fleet image promotes, worker
scaling, CVDIAG keys — were applied directly to Railway and are not
code; the post-incident debugging lesson shipped separately in #5727.)
## Incident context (why these exist)
Prod's coverage dashboard cascaded to a wall of red. Root cause was
**staleness, not a feature break**: the harness worker pool was starved
(a deploy-bounce + under-provisioning), so probe sweeps couldn't
complete within the staleness windows → cells aged out → rendered
red/BE✗ even though the apps were fine. Compounded by a `llamaindex`
container crash (per-request log flood tripping Railway's log cap) and a
2-day-stale prod image fleet. See #5727 for the debugging lesson, and
the Notion proposal below for the durable worker-reclamation redesign.
## The four fixes
1. **Dashboard stale-while-revalidate**
(`shell-dashboard/depth-chip.tsx`) — a passing (green) cell keeps its
color + a non-destructive `⟳` refreshing affordance during a re-probe
instead of flapping to grey; never-run stays grey; failure/regression
keeps its color with no spinner; staleness bound preserved. + a11y
(`role=status`, flag-gated regression label, `motion-reduce`).
2. **Family-silence grace window** (`harness/fleet/control-plane/*` +
dashboard banner) — a normal harness deploy's post-bounce worker drain
no longer fires a false "worker family silent" banner/alert (suppressed
within a `2×period` bounce grace keyed on the freshest worker
registration), while genuine silence beyond the window still fires on
both the Slack-alert and banner paths. No silent failure (PB-down →
grace disabled → real outages still alert).
3. **llamaindex per-request log gate**
(`integrations/llamaindex/.../route.ts`) — gates the chatty
`[copilotkit/route]` per-request logs behind `SHOWCASE_ROUTE_DEBUG`
(default off) so they can't flood Railway's log cap and kill the
container. Error logging untouched.
4. **SSOT worker-provisioning** (`scripts/railway-envs.ts` + drift gate)
— brings harness-workers replica provisioning under SSOT so prod/staging
can't silently drift. Models the **effective** field
(`multiRegionConfig.<region>.numReplicas`) +
`BROWSER_POOL_MAX_CONTEXTS`, declares current reconciled reality
(prod=staging=6 replicas, 40 contexts), with a CI drift-detection test.
## Review
7-agent CR round + 7-agent confirmation round → converged to **0 P0 / 0
P1**. Two post-CR fixes (SSOT effective-field model after discovering
`multiRegionConfig` is the live knob; re-anchoring a grace-edge test to
genuinely pin the 2× boundary) each re-confirmed clean. Suites:
shell-dashboard 92 + harness 63 + scripts 12 = **167 passed, 0 failed**.
No new tsc errors (pre-existing only).
## Notes / follow-ups (non-blocking)
- llamaindex log-gate is logging-only and untested (acceptable); the
chatty pattern exists in ~16 integrations incl. the gold standard — a
**fleet-wide log-flood gate** is a worthwhile follow-up.
- `freshestBounceMs` has two equivalent impls (harness `parseIso` /
dashboard `Date.parse`) — flagged for future lockstep.
- The D4/BE probe gate weakness (asserts `text.length>0`, which masked
the BIA outage) is being addressed in a **separate** PR
(`fix/d4-gate-tighten`).
## Refs
- Post-incident debugging lesson: #5727
- Worker reclamation + graceful-rollover redesign (Notion proposal):
https://app.notion.com/p/38b3aa381852817bacf5c9cda1f11cc0🤖 Generated with [Claude Code](https://claude.com/claude-code)
The harness-workers SSOT modeled only the top-level numReplicas, whose drift
gate watched a field that does not drive the live replica count. harness-workers
is single-region (us-west2); Railway derives the live count from
multiRegionConfig.us-west2.numReplicas.
- WorkerProvisioning gains effectiveReplicas (= multiRegionConfig.us-west2.
numReplicas), the authoritative field the drift gate now asserts. Top-level
numReplicas is retained as a documented mirror.
- Declared values reflect current reconciled reality (verified live via the
Railway GraphQL environment.config staged-config read): prod and staging both
at effectiveReplicas=6 (parity achieved by B-reconcile scaling prod 3 -> 6 in
both the top-level field and multiRegionConfig). BROWSER_POOL_MAX_CONTEXTS is
40 on both envs (verified live).
- Regenerated railway-envs.generated.json; drift-gate test asserts
effectiveReplicas (RED-GREEN proven). RAILWAY.md documents multiRegionConfig
as the effective knob and the achieved parity.
Add WorkerProvisioning interface and workerProvisioning field to the
harness-workers ServiceEntry in railway-envs.ts. Declares current live
reality: prod=3 replicas, staging=6 replicas, BROWSER_POOL_MAX_CONTEXTS=40
per worker (both envs).
Worker model: 1-worker-per-replica (Railway runs one process per
container, keyed on HOSTNAME). HARNESS_POOL_COUNT is informational
only — not a fork factor. Authoritative concurrency knob per worker is
BROWSER_POOL_MAX_CONTEXTS.
Comments flag: staging config-field drift (Railway field=2, live=6) as
a follow-up item; and the prod/staging parity decision (prod=3 vs
staging=6) as deliberately deferred.
Extends emit-railway-envs-json.ts to emit workerProvisioning into the
generated JSON snapshot. Adds drift-gate test
(harness-workers-provisioning.test.ts) that fails if SSOT numReplicas
diverges from the committed JSON snapshot — no live Railway API calls.
Red-green-red-green verified locally.
Updates RAILWAY.md with the 1-worker-per-replica model, declared
values, manual apply procedure, and drift-gate reference.
The existing tooling is verify-only for numReplicas; applying a replica
count change to Railway remains a manual operation (Railway Dashboard or
GraphQL API).
--isolate <name> <slug> (name before slug) mis-parsed the name as the
slug, then died "Unexpected argument"; a bare --isolate= silently
fell through to auto-pick. Defer the ambiguous post---isolate token to
pending_iso_name and resolve it after the parse loop (slug present =>
token was the name; no slug => token was the slug); reject an empty
--isolate= loudly; --isolate=<name> (non-numeric) now binds an explicit
isolate name. The --isolate=<N> numeric pin is unchanged.
_slot_ports_free consumed _slot_offset_ports via process substitution
(done < <(...)), so a die on an out-of-range/non-numeric slot exited only
the subshell — the loop read zero ports, any_held stayed 0, and the
function returned 0 ("all free"), silently defeating the port-conflict
guard for a bad slot. Capture into a variable with || die so the failure
propagates to the caller. Real-surface bats prove a bad slot now fails
loudly while valid-slot free/held behavior is unchanged.
Dry-run by default (lists the plan, changes nothing); --force executes,
--all ignores TTL/keep, --include-live opts into reaping a live-owner
target, <name|slot> targets one. Identifies harness-owned projects via
the slot-record / run-dir / showcase-iso<N> / self-id-label union, and
never touches the base 'showcase' stack or BuildKit resources. Real
docker bats prove dry-run/--force/--all + the base/buildkit guards.
A --keep'd isolated stack whose owning process had exited (but whose
containers kept running) was classified 'live' forever and never reaped,
leaking Docker stacks indefinitely. Introduce a start-time-verified
_owner_liveness probe and a new 'kept' state, an ISOLATE_KEEP_TTL (4h,
SHOWCASE_ISOLATE_KEEP_TTL-overridable) that flips an over-age kept slot
to 'stale' so the sweep reclaims it, a com.copilotkit.showcase.isolate
self-id label stamped by apply_isolation, a 'slots --reapable' filter,
and a macOS lsof COMMAND-truncation fix in the own-project port filter.
Real-surface bats cover the liveness false-positive and TTL reaping.
Emit CVDIAG backend boundary markers from the agent process for strands-typescript (byteLength fix on sseChunkByteLength), enable the emitter in docker-compose.local.yml, vendor src/cvdiag, and exclude tests from tsconfig.
## Root cause
Prod's `harness-workers` fleet worker runs a **stale `showcase-harness`
image** because it had **no `prod` env entry** in the railway-envs SSOT.
- The worker (`serviceId c2aa8a0b-350e-4b76-8541-3012dfac41d0`, prod
instance `7c48ee43-6df4-457b-b977-10f1f1ac1680`, `HARNESS_ROLE=worker`)
consumes the shared `showcase-harness` image via `imageOf: "harness"`.
- `expandImageConsumers(names, env)` is **env-aware**: a consumer only
enters an env's redeploy scope if it declares that env
(`redeploy-env.ts:278` — `if (!Object.hasOwn(entry.environments, env))
continue;`).
- Because the worker modeled **staging only**, a rebuilt
`showcase-harness:latest` bounced the prod control-plane but **silently
skipped the prod worker**, which kept its stale **2026-06-19** image.
- That stale image bakes a **1-demo `registry.json`** for
`ms-agent-harness-dotnet` (only `beautiful-chat`). The hourly
`e2e_demos` driver runs on the worker → resolves 1 demo → writes only
`e2e:ms-agent-harness-dotnet/beautiful-chat`. The other 38 feature rows
never exist in prod PocketBase → `resolveD3.exists === false` → `UI`
badge omitted → broken D3 rung collapses the ladder → **D0**.
(d5/d6 populate fully in prod because the D5/D6 drivers enumerate from a
**compiled-in** script registry, not the `registry.json` data file —
only `e2e_demos` is data-driven, which is why only `UI` was affected.)
## Fix
Backfill the live prod worker as a real `prod` env entry in
`scripts/railway-envs.ts` (real serviceInstance ID `7c48ee43…`), flip
`gateIgnore` off, and set `gateValidated: true`. The env-aware `imageOf`
expansion now pulls the prod worker into the **prod** redeploy scope on
every `showcase-harness` rebuild, so it can no longer drift onto a stale
image.
Also regenerates `railway-envs.generated.json` (Ruby/jq boundary
artifact) and the golden behavior-preservation fixture, and updates the
two gate-count assertions (`gateValidated` services 40→41;
`harness-workers` removed from the gateIgnore set).
## Local RED → GREEN proof
Failure surface: the real `expandImageConsumers("harness", "prod")`
against the real SSOT must include `harness-workers`.
**RED** (prod env entry absent from SSOT — the bug):
```
× includes harness-workers in the PROD redeploy scope when showcase-harness rebuilds
AssertionError: expected [ 'harness' ] to include 'harness-workers'
at __tests__/redeploy-env.harness-worker-prod-scope.test.ts:31:19
Tests 1 failed | 1 passed (2)
```
**GREEN** (after adding the prod `harness-workers` env entry):
```
✓ includes harness-workers in the PROD redeploy scope when showcase-harness rebuilds
✓ still includes harness-workers in the STAGING redeploy scope (no regression)
Tests 2 passed (2)
```
Full SSOT-dependent suite (golden snapshot, emit-json, image-ref gate,
promote closure, verify-matrix, redeploy-env): **140 passed**.
## Note (out of scope for this PR)
This SSOT change ensures the prod worker is bounced on **future**
rebuilds. The currently-live prod worker still needs a one-time
redeploy/restart onto the current `showcase-harness:latest` (39-demo
registry) to immediately backfill the 38 missing rows; that is an
operational step, not a code change.
Port the google-adk a2ui-recovery demo to langgraph (python, fastapi,
typescript) and aws-strands (python, typescript). Each ships a dedicated
recovery agent, route, demo page/chat/suggestions, manifest entry, aimock
d6 fixtures, e2e spec, and QA doc.
Backend-owned recovery on langgraph via get_a2ui_tools / getA2UITools
(injectA2UITool=false); auto-inject recovery on the strands adapter path.
Heal stages an invalid-then-valid render via aimock sequenceIndex (the
toolkit validate->retry loop rejects the whole surface, so a single-pass
parse_and_fix heal is ADK-specific and does not apply here). Recovery
prompts are unique per framework and the fixtures carry no context match
field, so they fire for real browser (dojo) traffic, not just the harness.
Also harden the strands declarative-gen-ui composition guide to name the
exact catalog component (Metric, not MetricTile) and update the
generate-catalog + aimock-fixtures test expectations.
The prod harness-workers backfill (e88d01a) inverts the old
"harness-workers is staging-only" invariant. Update the 8 stale
assertions across 3 test files that still encoded staging-only,
deriving the new expected values from the SSOT (railway-envs.ts) and
the regenerated railway-envs.generated.json:
- healthcheckPathFor/emit healthcheckPath: prod now /health (was undefined/omitted)
- repoNameFor(prod): now resolves showcase-harness (was throw)
- envsFor: now [prod, staging] (was [staging])
- the worker-shape test: dual-env, domainless+probe-disabled in BOTH
envs, gateValidated:true / gateIgnore dropped (per SSOT)
- computePromoteClosure: harness-workers now Tier-1 promoted, not
skipped; the always-Tier-1 set no longer filters it out
- expandImageConsumers(prod) / default prod redeploy scope (39->40):
the dual-env worker now joins the prod showcase-harness redeploy scope
The prod `harness-workers` fleet worker (serviceId
c2aa8a0b-350e-4b76-8541-3012dfac41d0, instance
7c48ee43-6df4-457b-b977-10f1f1ac1680) runs the shared `showcase-harness`
image (`imageOf: "harness"`) but had NO `prod` env entry in the
railway-envs SSOT. `expandImageConsumers` is env-aware — a consumer only
joins an env's redeploy scope if it declares that env — so a rebuilt
`showcase-harness:latest` bounced the prod control-plane but SILENTLY
SKIPPED the prod worker, leaving it pinned to a stale 2026-06-19 image.
That stale worker image carries a 1-demo `registry.json` for
`ms-agent-harness-dotnet` (only `beautiful-chat`), so the hourly
`e2e_demos` driver running on it produced only 1 of 39 `e2e:` rows in
prod PocketBase. The other 38 feature rows were absent → `resolveD3`
exists=false → `UI` badge omitted → broken D3 rung → D0.
Backfill the live prod worker as a `prod` env entry (real
serviceInstance ID), flip `gateIgnore` off, and set `gateValidated:
true` so the env-aware `imageOf` expansion now pulls the prod worker
into the prod redeploy scope on every `showcase-harness` rebuild.
Regenerate the emitted JSON + golden fixture and update the two
gate-count assertions accordingly.
The showcase-strands-typescript integration was staging-only: it had no
production Railway serviceInstance, so the prod D6 dashboard column showed
a uniform false-red (every cell errorClass=goto-error, backendUrl="") —
the probe navigated a bare relative path because the harness had no prod
health record / backendUrl to discover.
Provisions the prod serviceInstance (8a50728e-6119-43c4-b59c-d9535b6717a4,
domain showcase-strands-typescript-production.up.railway.app, healthcheck
/api/health, image pinned to the GHCR @sha256 digest, OPENAI_BASE_URL at
prod aimock) and brings the SSOT to the dual-env showcase-strands shape:
- railway-envs.ts: add the prod env entry with the real instanceId,
gateValidated:true, drop gateIgnore, remove the legacyJsonCompat
prod-domain placeholder.
- railway-envs.generated.json: regenerated (prod instanceId/domain, probe.prod
true, prod healthcheck; moved into the promote closure, tier 2).
- railway-envs.golden.json: regenerated to include the new prod (service,env)
pair (intentional behavior change, not a refactor regression).
- showcase_promote.yml: dropdown regenerated to list strands-typescript.
- verify-railway-image-refs.test.ts / redeploy-env.test.ts: update the
gateValidated/scope counts (39->40 gate targets, prod default 38->39) and the
now-stale staging-only comments.
RED->GREEN (live prod): BEFORE /api/health 404, prod PocketBase
health:strands-typescript totalItems:0, the 3 named D6 cells all
errorClass=goto-error backendUrl="". AFTER /api/health 200, prod PocketBase
health:strands-typescript present (status:200, valid url),
verify-railway-image-refs OK 80 instances.
Bump the canonical CopilotKit pin across all showcase integrations + shell
to 1.61.2 (canonical-pins.json, every package.json + package-lock.json),
which carries CopilotKit#5611: passing a catalog to the provider
(`<CopilotKit a2ui={{ catalog }}>`) now auto-enables A2UI and defaults tool
injection on, so the runtime no longer needs an explicit `a2ui` config.
Demonstrate the feature on the A2UI dynamic (declarative-gen-ui) demos by
removing the now-redundant runtime `a2ui` block (`injectA2UITool: true` +
`defaultCatalogId`) from:
- langgraph-python, langgraph-fastapi, langgraph-typescript
- strands, strands-typescript
- google-adk
The forwarded catalog supplies its own catalogId (sdk-js A2UI middleware
auto-derives `defaultCatalogId` from it), so the previous "Catalog not found"
fallback no longer applies.
Verified: validate-pins drift ratchet unchanged (38 / same hash);
langgraph-python D6 `gen-ui-declarative` green end-to-end (no Catalog-not-found).
The dojo's preview iframe built its src from `integration.backend_url`,
which generate-registry.ts bakes into registry.json at Docker BUILD time
(default `showcase-{slug}-production.up.railway.app`). So the staging
dojo iframed PROD integration backends — the exact staging->prod leakage
the shell's SU-13 runtime-derivation fix already prevents, but which was
never ported to shell-dojo.
Port the `backendHostPattern` slice of SU-13:
- copy shell's backend-url.ts verbatim (resolveBackendUrl + the
NEXT_PUBLIC_LOCAL_BACKENDS local-dev override); a scripts drift-guard
test keeps it byte-identical to the shell's and pins the default
pattern across backend-url.ts and generate-registry.ts.
- add `backendHostPattern` to shell-dojo's RuntimeConfig (server reads
SHOWCASE_BACKEND_HOST_PATTERN at request time; client carries the SSR
sentinel) — the existing layout injection picks it up automatically.
- page.tsx derives previewUrl via resolveBackendUrl at request time,
gated on a `mounted` flag so the SSR-phase sentinel host never reaches
an iframe src (shell-dojo loads the registry synchronously, so unlike
the shell it has no data-loading guard to defer the read past
hydration).
Staging dojo's SHOWCASE_BACKEND_HOST_PATTERN is set to
`showcase-{slug}-staging.up.railway.app`; prod stays unset (= default
prod pattern), so prod behavior is byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The strands-typescript SSOT entry is ciBuilt:true but staging-only (prod
instance not yet provisioned). redeploy-env.ts's default scope was the full
CI_BUILT_SERVICES set for BOTH envs, so a staging-only ciBuilt service would
wrongly enter the prod default scope and fail a manual `redeploy-env.ts prod`
(no prod instance). Filter the default scope by env declaration (explicit
--services stays unfiltered, preserving the contract-pin that an operator can
force a named service in an env it does not declare). imageOf expansion was
already env-aware; this extends the same invariant to the base scope.
Update the inventory-lock test counts for the new service (total 40->41,
CI_BUILT 38->39, staging default scope 39->40; prod default scope stays 38 as
the staging-only service is now correctly excluded).
Wire the strands-typescript showcase integration for staging deployment,
mirroring how the Python strands integration is deployed.
- manifest: flip deployed: true so the shell lists it in the integration menu
- railway-envs.ts: add showcase-strands-typescript SSOT entry (staging-only
for now: prod instance not yet provisioned, so it omits the prod env and is
gateIgnore'd until promoted dual-env); regenerate railway-envs.generated.json
- showcase_build.yml + showcase_build_check.yml: add the strands-typescript
build matrix entry, change-detection filter, and dispatch option (railway_id
is the new Railway service id)
- golden fixture + image-ref-gate inventory tests updated for the new service
Railway staging service showcase-strands-typescript provisioned
(showcase-strands-typescript-staging.up.railway.app, health /api/health,
OpenAI-via-aimock env). Prod is added later via the promote pipeline.
Fail loud on a dropped #oss-alerts page: the failure-alert cross-post no longer
swallows a 200/ok:false Slack response, so a dropped page-the-humans alert reds
the renderer job instead of vanishing on a green run. The thread reply stays
warn-only. Both posts capture the response via a shared slack_alert_posted_ok
predicate, mirrored byte-identically across the live workflow and the dry-run
helper.
Debt cleanup: drop a dead failed_count var, correct a misleading gha_url comment,
and validate the decoded blob run_id against ^[0-9a-f]{6}$ in the render step so
a malformed run_id can't reach Slack or the run name.
Tests: predicate edge cases (non-JSON, malformed, missing/null ok), an anti-drift
parity guard asserting the predicate is identical in both files, and call-site
tests locking the #oss-alerts fail-loud vs thread warn-only exit semantics.
Jordan Ritter