Restore the 12 Python integration tools/ dirs to symlinks into
shared/python/tools. They had eroded to real, drifting copies via an
accidental stage_shared() leak (commit 534cd1efa7) — the structural root
cause of showcase divergence bugs. Symlinking re-establishes the single
source of truth; content is identical to shared (only render_a2ui naming
and the shared roll_dice/sanitize additions are adopted).
Add showcase/AGENTS.md documenting the 4 iron rules and the single-source
symlink mechanism, plus a validate-shared-symlinks CI guard (shrink-only
baseline) that fails on any NEW erosion.
OSS-451 shipped because nothing linked a demo page's CopilotKit runtimeUrl
to the existence of the /api route it names. The only automatic pre-merge
gate for showcase/** is a Docker build, which compiles a page that
references a non-existent route just fine (runtimeUrl is an unchecked
string) — so the page-404-on-load class was invisible.
Add a static validator (validate-runtime-routes.ts) that, for every SHIPPED
demo (a demo listed in its integration's manifest `features`), asserts its
runtimeUrl resolves to a real route dir under src/app/api. Unshipped /
experimental demos (not in `features`) and not_supported_features are
skipped, so incomplete placeholders don't fail the gate — but promoting one
into `features` immediately starts enforcing it. A baseline file can
grandfather pre-existing violations; the fleet is currently clean (0).
Wire it into a new pre-merge workflow (showcase_validate-wiring.yml) that
runs on every showcase/integrations PR alongside the build check. Add it to
branch-protection required checks to make it blocking.
Regression test proves it flags the exact OSS-451 shape (shipped demo,
missing route) while passing existing/base routes and skipping unshipped.
Verified: npm run validate-routes -> clean fleet-wide; removing the 3
OSS-451 routes -> flags exactly those 3; full showcase/scripts vitest suite
(2151 tests) green.
Refs OSS-451
The promote workflow's verify-prod job calls verify-deploy.ts directly
(--env prod --services <promoted set>) without #5752's --skip-ineligible
flag, so a known-but-ineligible service (harness-workers, probe.prod=false)
hard-errored exit 2 and crashed the gate AFTER a successful promote
(CI run 28333317081: llamaindex landed, then verify-prod crashed).
Generalize the eligibility filter into verify-deploy.ts itself rather than
relying on each caller to pass a flag: flip skipIneligible to ON by default
in the CLI (parseArgs). A known-but-not-probe-eligible service for the
requested env is now SKIPPED with an `N/A — not probe-eligible ... skipped`
status line and the eligible subset is probed. Works for ANY --env, so it
composes with #5752's staging path and fixes the direct prod-verify call.
When EVERY requested service is ineligible (e.g. promoted set is just
harness-workers), runVerify exits 0 with a "nothing to probe" note instead
of the vacuous-green FAIL — distinct from the empty-filter fault, which
still fails loud. Unknown (non-SSOT) names STILL hard-error on every path
(a typo is a real fault). Added --strict-eligibility to opt back into the
hard-refuse; --skip-ineligible kept as an explicit no-op for back-compat.
Red: `verify-deploy.ts --env prod --services harness-workers` crashed
exit 2. Green: same command skips (N/A) and exits 0. Mixed set
harness-workers,showcase-llamaindex skips workers and still probes (and
red-gates) llamaindex.
Layer (c) deploy-rollover config for the harness-workers fleet — pure
Railway config, no custom rolling-restart code. Declares overlapSeconds=45
(capacity floor: old deployment serves until new workers register+claim, so
no staleness dip) and drainingSeconds=180 (SIGTERM->SIGKILL window >=
PLATFORM_STOP_GRACE_MS so the layer-(b) 3s+90s composed worker-drain finishes)
for both prod and staging in the railway-envs SSOT, regenerates the JSON
snapshot, and extends the harness-workers drift gate so CI fails if either
field drifts between SSOT and snapshot. Documents both knobs, their rationale
(incl. that Railway's draining default is 0s = immediate SIGKILL), the
composition with drain layers a+b, and how to apply them (GraphQL
serviceInstanceUpdate / dashboard) in RAILWAY.md.
## What
Bundles the four code fixes from the 2026-06-26 showcase prod incident
remediation. (Prod config reconciles — fleet image promotes, worker
scaling, CVDIAG keys — were applied directly to Railway and are not
code; the post-incident debugging lesson shipped separately in #5727.)
## Incident context (why these exist)
Prod's coverage dashboard cascaded to a wall of red. Root cause was
**staleness, not a feature break**: the harness worker pool was starved
(a deploy-bounce + under-provisioning), so probe sweeps couldn't
complete within the staleness windows → cells aged out → rendered
red/BE✗ even though the apps were fine. Compounded by a `llamaindex`
container crash (per-request log flood tripping Railway's log cap) and a
2-day-stale prod image fleet. See #5727 for the debugging lesson, and
the Notion proposal below for the durable worker-reclamation redesign.
## The four fixes
1. **Dashboard stale-while-revalidate**
(`shell-dashboard/depth-chip.tsx`) — a passing (green) cell keeps its
color + a non-destructive `⟳` refreshing affordance during a re-probe
instead of flapping to grey; never-run stays grey; failure/regression
keeps its color with no spinner; staleness bound preserved. + a11y
(`role=status`, flag-gated regression label, `motion-reduce`).
2. **Family-silence grace window** (`harness/fleet/control-plane/*` +
dashboard banner) — a normal harness deploy's post-bounce worker drain
no longer fires a false "worker family silent" banner/alert (suppressed
within a `2×period` bounce grace keyed on the freshest worker
registration), while genuine silence beyond the window still fires on
both the Slack-alert and banner paths. No silent failure (PB-down →
grace disabled → real outages still alert).
3. **llamaindex per-request log gate**
(`integrations/llamaindex/.../route.ts`) — gates the chatty
`[copilotkit/route]` per-request logs behind `SHOWCASE_ROUTE_DEBUG`
(default off) so they can't flood Railway's log cap and kill the
container. Error logging untouched.
4. **SSOT worker-provisioning** (`scripts/railway-envs.ts` + drift gate)
— brings harness-workers replica provisioning under SSOT so prod/staging
can't silently drift. Models the **effective** field
(`multiRegionConfig.<region>.numReplicas`) +
`BROWSER_POOL_MAX_CONTEXTS`, declares current reconciled reality
(prod=staging=6 replicas, 40 contexts), with a CI drift-detection test.
## Review
7-agent CR round + 7-agent confirmation round → converged to **0 P0 / 0
P1**. Two post-CR fixes (SSOT effective-field model after discovering
`multiRegionConfig` is the live knob; re-anchoring a grace-edge test to
genuinely pin the 2× boundary) each re-confirmed clean. Suites:
shell-dashboard 92 + harness 63 + scripts 12 = **167 passed, 0 failed**.
No new tsc errors (pre-existing only).
## Notes / follow-ups (non-blocking)
- llamaindex log-gate is logging-only and untested (acceptable); the
chatty pattern exists in ~16 integrations incl. the gold standard — a
**fleet-wide log-flood gate** is a worthwhile follow-up.
- `freshestBounceMs` has two equivalent impls (harness `parseIso` /
dashboard `Date.parse`) — flagged for future lockstep.
- The D4/BE probe gate weakness (asserts `text.length>0`, which masked
the BIA outage) is being addressed in a **separate** PR
(`fix/d4-gate-tighten`).
## Refs
- Post-incident debugging lesson: #5727
- Worker reclamation + graceful-rollover redesign (Notion proposal):
https://app.notion.com/p/38b3aa381852817bacf5c9cda1f11cc0🤖 Generated with [Claude Code](https://claude.com/claude-code)
The harness-workers SSOT modeled only the top-level numReplicas, whose drift
gate watched a field that does not drive the live replica count. harness-workers
is single-region (us-west2); Railway derives the live count from
multiRegionConfig.us-west2.numReplicas.
- WorkerProvisioning gains effectiveReplicas (= multiRegionConfig.us-west2.
numReplicas), the authoritative field the drift gate now asserts. Top-level
numReplicas is retained as a documented mirror.
- Declared values reflect current reconciled reality (verified live via the
Railway GraphQL environment.config staged-config read): prod and staging both
at effectiveReplicas=6 (parity achieved by B-reconcile scaling prod 3 -> 6 in
both the top-level field and multiRegionConfig). BROWSER_POOL_MAX_CONTEXTS is
40 on both envs (verified live).
- Regenerated railway-envs.generated.json; drift-gate test asserts
effectiveReplicas (RED-GREEN proven). RAILWAY.md documents multiRegionConfig
as the effective knob and the achieved parity.
Add WorkerProvisioning interface and workerProvisioning field to the
harness-workers ServiceEntry in railway-envs.ts. Declares current live
reality: prod=3 replicas, staging=6 replicas, BROWSER_POOL_MAX_CONTEXTS=40
per worker (both envs).
Worker model: 1-worker-per-replica (Railway runs one process per
container, keyed on HOSTNAME). HARNESS_POOL_COUNT is informational
only — not a fork factor. Authoritative concurrency knob per worker is
BROWSER_POOL_MAX_CONTEXTS.
Comments flag: staging config-field drift (Railway field=2, live=6) as
a follow-up item; and the prod/staging parity decision (prod=3 vs
staging=6) as deliberately deferred.
Extends emit-railway-envs-json.ts to emit workerProvisioning into the
generated JSON snapshot. Adds drift-gate test
(harness-workers-provisioning.test.ts) that fails if SSOT numReplicas
diverges from the committed JSON snapshot — no live Railway API calls.
Red-green-red-green verified locally.
Updates RAILWAY.md with the 1-worker-per-replica model, declared
values, manual apply procedure, and drift-gate reference.
The existing tooling is verify-only for numReplicas; applying a replica
count change to Railway remains a manual operation (Railway Dashboard or
GraphQL API).
_slot_ports_free consumed _slot_offset_ports via process substitution
(done < <(...)), so a die on an out-of-range/non-numeric slot exited only
the subshell — the loop read zero ports, any_held stayed 0, and the
function returned 0 ("all free"), silently defeating the port-conflict
guard for a bad slot. Capture into a variable with || die so the failure
propagates to the caller. Real-surface bats prove a bad slot now fails
loudly while valid-slot free/held behavior is unchanged.
Dry-run by default (lists the plan, changes nothing); --force executes,
--all ignores TTL/keep, --include-live opts into reaping a live-owner
target, <name|slot> targets one. Identifies harness-owned projects via
the slot-record / run-dir / showcase-iso<N> / self-id-label union, and
never touches the base 'showcase' stack or BuildKit resources. Real
docker bats prove dry-run/--force/--all + the base/buildkit guards.
A --keep'd isolated stack whose owning process had exited (but whose
containers kept running) was classified 'live' forever and never reaped,
leaking Docker stacks indefinitely. Introduce a start-time-verified
_owner_liveness probe and a new 'kept' state, an ISOLATE_KEEP_TTL (4h,
SHOWCASE_ISOLATE_KEEP_TTL-overridable) that flips an over-age kept slot
to 'stale' so the sweep reclaims it, a com.copilotkit.showcase.isolate
self-id label stamped by apply_isolation, a 'slots --reapable' filter,
and a macOS lsof COMMAND-truncation fix in the own-project port filter.
Real-surface bats cover the liveness false-positive and TTL reaping.
## Root cause
Prod's `harness-workers` fleet worker runs a **stale `showcase-harness`
image** because it had **no `prod` env entry** in the railway-envs SSOT.
- The worker (`serviceId c2aa8a0b-350e-4b76-8541-3012dfac41d0`, prod
instance `7c48ee43-6df4-457b-b977-10f1f1ac1680`, `HARNESS_ROLE=worker`)
consumes the shared `showcase-harness` image via `imageOf: "harness"`.
- `expandImageConsumers(names, env)` is **env-aware**: a consumer only
enters an env's redeploy scope if it declares that env
(`redeploy-env.ts:278` — `if (!Object.hasOwn(entry.environments, env))
continue;`).
- Because the worker modeled **staging only**, a rebuilt
`showcase-harness:latest` bounced the prod control-plane but **silently
skipped the prod worker**, which kept its stale **2026-06-19** image.
- That stale image bakes a **1-demo `registry.json`** for
`ms-agent-harness-dotnet` (only `beautiful-chat`). The hourly
`e2e_demos` driver runs on the worker → resolves 1 demo → writes only
`e2e:ms-agent-harness-dotnet/beautiful-chat`. The other 38 feature rows
never exist in prod PocketBase → `resolveD3.exists === false` → `UI`
badge omitted → broken D3 rung collapses the ladder → **D0**.
(d5/d6 populate fully in prod because the D5/D6 drivers enumerate from a
**compiled-in** script registry, not the `registry.json` data file —
only `e2e_demos` is data-driven, which is why only `UI` was affected.)
## Fix
Backfill the live prod worker as a real `prod` env entry in
`scripts/railway-envs.ts` (real serviceInstance ID `7c48ee43…`), flip
`gateIgnore` off, and set `gateValidated: true`. The env-aware `imageOf`
expansion now pulls the prod worker into the **prod** redeploy scope on
every `showcase-harness` rebuild, so it can no longer drift onto a stale
image.
Also regenerates `railway-envs.generated.json` (Ruby/jq boundary
artifact) and the golden behavior-preservation fixture, and updates the
two gate-count assertions (`gateValidated` services 40→41;
`harness-workers` removed from the gateIgnore set).
## Local RED → GREEN proof
Failure surface: the real `expandImageConsumers("harness", "prod")`
against the real SSOT must include `harness-workers`.
**RED** (prod env entry absent from SSOT — the bug):
```
× includes harness-workers in the PROD redeploy scope when showcase-harness rebuilds
AssertionError: expected [ 'harness' ] to include 'harness-workers'
at __tests__/redeploy-env.harness-worker-prod-scope.test.ts:31:19
Tests 1 failed | 1 passed (2)
```
**GREEN** (after adding the prod `harness-workers` env entry):
```
✓ includes harness-workers in the PROD redeploy scope when showcase-harness rebuilds
✓ still includes harness-workers in the STAGING redeploy scope (no regression)
Tests 2 passed (2)
```
Full SSOT-dependent suite (golden snapshot, emit-json, image-ref gate,
promote closure, verify-matrix, redeploy-env): **140 passed**.
## Note (out of scope for this PR)
This SSOT change ensures the prod worker is bounced on **future**
rebuilds. The currently-live prod worker still needs a one-time
redeploy/restart onto the current `showcase-harness:latest` (39-demo
registry) to immediately backfill the 38 missing rows; that is an
operational step, not a code change.
Port the google-adk a2ui-recovery demo to langgraph (python, fastapi,
typescript) and aws-strands (python, typescript). Each ships a dedicated
recovery agent, route, demo page/chat/suggestions, manifest entry, aimock
d6 fixtures, e2e spec, and QA doc.
Backend-owned recovery on langgraph via get_a2ui_tools / getA2UITools
(injectA2UITool=false); auto-inject recovery on the strands adapter path.
Heal stages an invalid-then-valid render via aimock sequenceIndex (the
toolkit validate->retry loop rejects the whole surface, so a single-pass
parse_and_fix heal is ADK-specific and does not apply here). Recovery
prompts are unique per framework and the fixtures carry no context match
field, so they fire for real browser (dojo) traffic, not just the harness.
Also harden the strands declarative-gen-ui composition guide to name the
exact catalog component (Metric, not MetricTile) and update the
generate-catalog + aimock-fixtures test expectations.
The prod `harness-workers` fleet worker (serviceId
c2aa8a0b-350e-4b76-8541-3012dfac41d0, instance
7c48ee43-6df4-457b-b977-10f1f1ac1680) runs the shared `showcase-harness`
image (`imageOf: "harness"`) but had NO `prod` env entry in the
railway-envs SSOT. `expandImageConsumers` is env-aware — a consumer only
joins an env's redeploy scope if it declares that env — so a rebuilt
`showcase-harness:latest` bounced the prod control-plane but SILENTLY
SKIPPED the prod worker, leaving it pinned to a stale 2026-06-19 image.
That stale worker image carries a 1-demo `registry.json` for
`ms-agent-harness-dotnet` (only `beautiful-chat`), so the hourly
`e2e_demos` driver running on it produced only 1 of 39 `e2e:` rows in
prod PocketBase. The other 38 feature rows were absent → `resolveD3`
exists=false → `UI` badge omitted → broken D3 rung → D0.
Backfill the live prod worker as a `prod` env entry (real
serviceInstance ID), flip `gateIgnore` off, and set `gateValidated:
true` so the env-aware `imageOf` expansion now pulls the prod worker
into the prod redeploy scope on every `showcase-harness` rebuild.
Regenerate the emitted JSON + golden fixture and update the two
gate-count assertions accordingly.
The showcase-strands-typescript integration was staging-only: it had no
production Railway serviceInstance, so the prod D6 dashboard column showed
a uniform false-red (every cell errorClass=goto-error, backendUrl="") —
the probe navigated a bare relative path because the harness had no prod
health record / backendUrl to discover.
Provisions the prod serviceInstance (8a50728e-6119-43c4-b59c-d9535b6717a4,
domain showcase-strands-typescript-production.up.railway.app, healthcheck
/api/health, image pinned to the GHCR @sha256 digest, OPENAI_BASE_URL at
prod aimock) and brings the SSOT to the dual-env showcase-strands shape:
- railway-envs.ts: add the prod env entry with the real instanceId,
gateValidated:true, drop gateIgnore, remove the legacyJsonCompat
prod-domain placeholder.
- railway-envs.generated.json: regenerated (prod instanceId/domain, probe.prod
true, prod healthcheck; moved into the promote closure, tier 2).
- railway-envs.golden.json: regenerated to include the new prod (service,env)
pair (intentional behavior change, not a refactor regression).
- showcase_promote.yml: dropdown regenerated to list strands-typescript.
- verify-railway-image-refs.test.ts / redeploy-env.test.ts: update the
gateValidated/scope counts (39->40 gate targets, prod default 38->39) and the
now-stale staging-only comments.
RED->GREEN (live prod): BEFORE /api/health 404, prod PocketBase
health:strands-typescript totalItems:0, the 3 named D6 cells all
errorClass=goto-error backendUrl="". AFTER /api/health 200, prod PocketBase
health:strands-typescript present (status:200, valid url),
verify-railway-image-refs OK 80 instances.
The dojo's preview iframe built its src from `integration.backend_url`,
which generate-registry.ts bakes into registry.json at Docker BUILD time
(default `showcase-{slug}-production.up.railway.app`). So the staging
dojo iframed PROD integration backends — the exact staging->prod leakage
the shell's SU-13 runtime-derivation fix already prevents, but which was
never ported to shell-dojo.
Port the `backendHostPattern` slice of SU-13:
- copy shell's backend-url.ts verbatim (resolveBackendUrl + the
NEXT_PUBLIC_LOCAL_BACKENDS local-dev override); a scripts drift-guard
test keeps it byte-identical to the shell's and pins the default
pattern across backend-url.ts and generate-registry.ts.
- add `backendHostPattern` to shell-dojo's RuntimeConfig (server reads
SHOWCASE_BACKEND_HOST_PATTERN at request time; client carries the SSR
sentinel) — the existing layout injection picks it up automatically.
- page.tsx derives previewUrl via resolveBackendUrl at request time,
gated on a `mounted` flag so the SSR-phase sentinel host never reaches
an iframe src (shell-dojo loads the registry synchronously, so unlike
the shell it has no data-loading guard to defer the read past
hydration).
Staging dojo's SHOWCASE_BACKEND_HOST_PATTERN is set to
`showcase-{slug}-staging.up.railway.app`; prod stays unset (= default
prod pattern), so prod behavior is byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wire the strands-typescript showcase integration for staging deployment,
mirroring how the Python strands integration is deployed.
- manifest: flip deployed: true so the shell lists it in the integration menu
- railway-envs.ts: add showcase-strands-typescript SSOT entry (staging-only
for now: prod instance not yet provisioned, so it omits the prod env and is
gateIgnore'd until promoted dual-env); regenerate railway-envs.generated.json
- showcase_build.yml + showcase_build_check.yml: add the strands-typescript
build matrix entry, change-detection filter, and dispatch option (railway_id
is the new Railway service id)
- golden fixture + image-ref-gate inventory tests updated for the new service
Railway staging service showcase-strands-typescript provisioned
(showcase-strands-typescript-staging.up.railway.app, health /api/health,
OpenAI-via-aimock env). Prod is added later via the promote pipeline.
Fail loud on a dropped #oss-alerts page: the failure-alert cross-post no longer
swallows a 200/ok:false Slack response, so a dropped page-the-humans alert reds
the renderer job instead of vanishing on a green run. The thread reply stays
warn-only. Both posts capture the response via a shared slack_alert_posted_ok
predicate, mirrored byte-identically across the live workflow and the dry-run
helper.
Debt cleanup: drop a dead failed_count var, correct a misleading gha_url comment,
and validate the decoded blob run_id against ^[0-9a-f]{6}$ in the render step so
a malformed run_id can't reach Slack or the run name.
Tests: predicate edge cases (non-JSON, malformed, missing/null ok), an anti-drift
parity guard asserting the predicate is identical in both files, and call-site
tests locking the #oss-alerts fail-loud vs thread warn-only exit semantics.
Adding a2ui-recovery to the shared feature-registry cross-joins it across all
20 integrations (wired for google-adk, unshipped elsewhere), so the catalog
generator's hardcoded counts move: cross-join 920->940, metadata.total_cells
900->920, LGP 46->47 (unshipped 7->8). docs_only stays 20; the relative
sum-checks are unchanged. Fixes the Validate Showcase failure on #5662.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The pool-fleet migration is complete: the control-plane harness plus the
prod workers (deployed 2026-06-19, HARNESS_ROLE=worker, pool count 2) now
cover every probe dimension the interim `harness-legacy` fleet-migration
bridge was holding live, so `harness-legacy` is dead config.
This is the code-side cleanup only:
- Remove the `harness-legacy` entry from the railway-envs SSOT and the
now-dead `key === "harness-legacy"` special-case in computePromoteClosure.
- Regenerate railway-envs.generated.json (41 -> 40 services).
- Drop harness-legacy from the golden snapshot, the gateIgnore expectation
sets, the promote-notify fixtures, and the redeploy-env doc comments;
update the service-count assertions (41 -> 40).
- Fix the stale "STAGING-ONLY" harness-workers comment: prod workers are
live on Railway, though this SSOT entry still models the staging
instance only (no prod env backfilled here yet).
The live Railway `harness-legacy` service is torn down separately as a
follow-up infra step.
Promote-notify Slack message: name the promoted AND failed services (one
Failed: header + bullets), legible "(N): <names>" count, real wall-clock
elapsed (integer-coerced), and drop the constant verify-prod legend line.
Durable healthcheckPath: track it per-service/env in the SSOT (railway-envs),
re-assert it on the promote pin path (omit-when-absent, never null), and route
deploy-to-railway provisioning through isTrackedService/resolveProvisionHealthcheck
so a tracked-null service omits the healthcheck while an untracked one keeps the
/api/health default — fixing the silent prod-healthcheck drift that refused aimock.
Tests: ruby pin-reassert spec + deploy-to-railway healthcheck spec + emit/golden/accessor.
Flip the 12 starter-* services to behave like every other managed
showcase service: always-on and staging-probed.
- railway-envs SSOT: staging probe false -> true for all 12 starters
(prod was already on); refresh the stale "staging probe OFF" /
fail-loud-placeholder doc-comments to reflect baseline-driver verification.
- provision-starter-fleet: sleepApplication true -> false so the
provisioner creates always-on (not sleepable) Railway instances; update
the "the whole point — sleepable" header/inline comments + log lines.
- Regenerate railway-envs.generated.json (emit-railway-envs-json.ts) and
the golden fixture; update the SSOT probe + provisioner sleep assertions.
Replace the fail-loud `case "starter"` stub in verify-deploy's dispatch
switch with a real baseline liveness driver. The starter-template fleet
EXPOSEs only its Next.js frontend (serving `/` and `/api/copilotkit`,
NO `/api/health`), so the driver healthchecks `/` via probeBaseline,
exactly like the Next.js shells: deployment-SUCCESS + HTTP 200 on `/`.
promote-fleet.sh now emits a base64 results JSON (schema_version=1) with
both succeeded[] and failed[] alongside the existing succeeded_csv /
staging_drift outputs. showcase_promote.yml's notify job replaces the old
inline two-state (success/failure) notifier — which dumped the full
requested CSV and mislabeled any partial promote as a blanket Failed — with
a single dispatch of the previously-orphaned showcase_promote_notify.yml
renderer (success / partial / total). The dispatch enriches promote-fleet's
results blob with run context (6-hex run_id, trigger=workflow, operator,
pre_staging) and authenticates via the devops-bot App token (actions:write),
mirroring canary.yml — the default GITHUB_TOKEN cannot start new workflow runs.
promote-fleet has no failure taxonomy, so each failed entry uses the default
category "promote-failed".
## Summary
An **on-demand** tool to answer "is prod caught up with staging right
now?". The showcase deploy model is **staging = mutable `:latest`**
(continuously rebuilt) and **prod = immutable `@sha256:`** (advances
only on an explicit promote), so a prod column can sit **behind** a
green staging.
**There is no scheduled drift alert — by design.** Prod lagging staging
is **often intentional**: changes are batched and promoted deliberately,
so a recurring "N columns stale" alert would be pure noise. This tool is
therefore manual-only: a maintainer runs it when they want to check, and
it tells them the current state.
- **`bin/railway reconcile-prod`** — for every prod-eligible
(`probe.prod == true`) service, compares the **prod serving digest**
(the `@sha256:` from
`SnapshotCommand.build_snapshot(PRODUCTION_ENV_ID)`) against the
**staging running digest** (reuses
`PromoteCommand#staging_running_digest`, the same source the promote pin
uses). Classifies each:
- `green` — prod == staging (in sync)
- `stale` — prod != staging **and** staging is resolvable (prod is
behind a green staging)
- `gray` — staging running digest not resolvable, or the service has no
prod snapshot entry yet — informational, **not** stale
- Prints a readable per-service table + summary; **exits nonzero iff any
service is stale**; `--json` for machine output. **Read-only — no
promotes/mutations.**
- **`showcase/scripts/reconcile-prod-gate.sh`** — wrapper mirroring
`lint-prod-gate.sh`: surfaces the table into `$GITHUB_STEP_SUMMARY`,
optionally captures `--json` to `RECONCILE_JSON`, and propagates the
exit-code verdict (never swallows a non-zero).
- **`.github/workflows/showcase_reconcile.yml`** — **`workflow_dispatch`
only** (no cron). Regenerates the SSOT JSON (`EMIT_SKIP_OXFMT=1`, same
as the promote workflow's resolve/promote jobs), runs the gate with the
Railway/GHCR auth env, renders the reconcile table to the **GH step
summary**, and uploads the `--json` as a `reconcile-json` artifact. **No
Slack.** The run exits nonzero on a stale column so a manual run visibly
flags drift. `timeout-minutes: 10`.
- **Tests** — Ruby minitest (`test_reconcile_prod.rb`: classification +
exit-code + `--json` shape + dispatcher registration) and a bats gate
test (`reconcile-prod-gate.bats`). Wired the gate script into the
`showcase_validate.yml` shellcheck list.
### What changed from the original scheduled-alert design
The first cut of this PR shipped a daily cron + auto-post to #oss-alerts
on any stale column. Per owner feedback, that was reshaped to on-demand
only: the `schedule:` trigger and the Slack-on-stale step were removed
(intentional/deliberate staleness is not a bug, so an unsolicited
recurring alert is noise). The CLI command, the gate wrapper, and all
tests are unchanged.
## Gates
- `ruby showcase/bin/spec/test_reconcile_prod.rb` → **9 runs, 20
assertions, 0 failures**
- `bats showcase/scripts/__tests__/reconcile-prod-gate.bats` → **6 ok**
- `shellcheck -s bash showcase/scripts/reconcile-prod-gate.sh` →
**clean**
- `actionlint .github/workflows/showcase_reconcile.yml` → **clean** (the
pre-existing `depot-ubuntu-24.04-4` custom-runner-label warning is on
`showcase_validate.yml`, predates this PR — my only change there is one
line in the shellcheck list)
## Test plan
- [ ] CI green (Ruby suite, bats suite, actionlint/shellcheck,
commitlint)
- [ ] Optional: read-only `workflow_dispatch` run of
`showcase_reconcile.yml` to confirm it runs against live prod/staging
(safe — no mutations)
A `shell-docs` promote expanded its dependency closure to include the
always-on Tier-1 verification set (harness, dashboard, …); harness then
tripped a P6 env-divergence WARN-refusal and the tier barrier gated docs
as NOT-ATTEMPTED. docs has no runtime dependency on the control plane.
Introduce a declarative `standalone` service class: a leaf that neither
depends on anything nor gates on anything. computePromoteClosure (TS) and
the resolve-promote-targets jq both skip the Tier-1 union when the request
is entirely standalone, so a standalone request promotes ONLY itself; the
fleet runner promotes `s:`-marked services ungated (never NOT-ATTEMPTED on
an unrelated failure, never gating a tier). Flag `docs` standalone.
Red-green: railway-envs.test.ts (4), resolve-promote-targets.bats (5),
promote-fleet.bats (3) fail on origin/main (docs closure pulls harness;
`s:` is an unknown tier) and pass after the fix.
Prod sitting behind staging is often intentional (changes are batched and
promoted deliberately), so a recurring drift alert is noise. Reshape the
reconcile workflow to manual-only:
- Remove the daily `schedule:` cron trigger — leave only `workflow_dispatch`.
- Remove the auto-Slack-on-stale step (and its SLACK_WEBHOOK env / stale_line
output derivation) — no unsolicited #oss-alerts post on mere staleness.
- A manual run surfaces the reconcile table to the GH step summary, keeps the
cheap `--json` capture as an uploaded artifact, and still exits nonzero on a
stale column so a manual run visibly flags drift.
- De-noise the gate script + bats comments that referenced the removed
scheduled/Slack behavior.
The on-demand CLI (`bin/railway reconcile-prod`), the gate wrapper, and the
Ruby + bats tests are unchanged.
Lever 1 of the promote-reliability hardening plan. The showcase deploy
model is staging=mutable :latest (continuously rebuilt), prod=immutable
@sha256: (advances only on explicit promote), so a prod column can
silently fall BEHIND a green staging — drift today is only noticed by
eyeballing a dead column. This adds proactive, automatic detection.
- bin/railway reconcile-prod: for every prod-eligible (probe.prod==true)
service, compares the prod SERVING digest (LintProd snapshot path) vs
the staging RUNNING digest (reuses PromoteCommand#staging_running_digest).
Classifies green/stale/gray, prints a table + summary, exits 1 iff any
stale. --json for machine output. Read-only: no promotes/mutations.
- scripts/reconcile-prod-gate.sh: wrapper mirroring lint-prod-gate.sh —
surfaces the table to the GH step summary, captures JSON for the Slack
builder, propagates the exit-code verdict.
- .github/workflows/showcase_reconcile.yml: daily cron + workflow_dispatch;
runs the gate; on stale services posts the stale-column list to
#oss-alerts (SLACK_WEBHOOK_OSS_ALERTS) via the fromJSON('"\n"') idiom.
- Tests: Ruby minitest (classification + exit-code, RED-anchored on a
drift-blind classifier) and a bats gate test. Wired the gate script
into the showcase_validate.yml shellcheck list.
Post-promote convergence verification is deferred to a fast-follow.
Two PR checks were red on the new integration:
- check-config-files: add strands-typescript/next.config.ts to the build-
config allowlist.
- Validate Showcase: update the new-integration guard pins that intentionally
trip when an integration is added — BORN_IN_SHOWCASE 6→7, calculator
_from-feature-parity count 18→19, catalog cross-join 874→920 / total_cells
855→900 / docs_only 19→20 (46 features × 20 integrations), and the aimock
substring-shadow ceiling 132→133 (+1 from the strands-typescript calculator
fixture).
Also drop the premature deploy wiring: strands-typescript is removed from
showcase_build.yml (matrix + path filter + ALL_SERVICES) because it has no
Railway service yet (deployed: false) and the railway-envs SSOT test requires
a real service entry. It re-enters the deploy pipeline when the Railway
service is provisioned (external setup per INTEGRATION-CHECKLIST).
Regenerated the stale committed showcase_promote.yml so the 12 starter-*
services (+ shell-docs and all existing targets) appear in the service
dispatch choice list (fixes HTTP 422 on
gh workflow run -f service=starter-*). Reverted isProdPromotable to
env-map-only (environments.prod.probe), equivalent to the workflow resolve
predicate. Added a regression test asserting shell-docs + all 12 starters
remain in the generated AND committed dropdown.
## Summary
Fixes two correctness bugs in the showcase staging→prod promote path
(`showcase/bin/railway`), discovered + live-validated while promoting
the full 19-service cluster.
**Bug #1 — promote pinned the wrong digest.** `resolved_prod_image`
re-resolved the mutable `:latest` tag against *current* GHCR instead of
pinning the digest staging is actually *running*
(`latestDeployment.meta.imageDigest`). When `:latest` drifted after
staging deployed, promote pushed an unvalidated (and once, regressed)
image to prod. Now pins staging's running digest. Adds a loud `⚠️
STAGING DRIFT` warning (promote stdout + `STAGING_DRIFT_MARKER:` →
`promote-fleet.sh` aggregation → both Slack payloads) when staging's
running digest ≠ current `:latest`, so the gap is visible without
blocking the promote.
**Bug #2 — the pin never activated.** `pin_and_verify` used
`serviceInstanceRedeploy`, which replays the *existing* deployment's old
image rather than the just-pinned `source.image`. Config showed the new
digest while prod kept serving the old one (this is why earlier promotes
"succeeded" while prod stayed broken). Switched to
`serviceInstanceDeployV2` + a new `verify_serving_digest!` gate that
polls the new deployment to SUCCESS and **fails loud** if the running
digest ≠ pinned.
Plus CR-round hardening: P2 in-flight race check now reads
`meta.imageDigest` (was dead on tag-form staging) and skips on
`--digest` override; `detect_staging_drift` fails loud (WARN) on
GHCR-resolve failure instead of swallowing; `--digest` override
suppresses spurious drift; `drift_line` LF-stripped at the
`GITHUB_OUTPUT` boundary; multi-service drift join fixed; fallback-log
drift preserved.
## Commits
1. `fix(showcase): pin prod to staging's running digest + loud
staging-drift warning`
2. `fix(showcase): activate prod pin via serviceInstanceDeployV2 +
verify running==pinned`
3. `fix(showcase): harden promote P2 race check + refresh ivar-lint
allowlist`
## Validation
- Live red-green on real Railway: claude-sdk-python prod flipped from
stale → pinned digest via the fixed CLI; all 19 cluster services
promoted green; 5 previously-degraded backends (ag2, llamaindex,
pydantic-ai, ms-agent-python, strands) recovered.
- 15/15 Ruby specs + 15/15 bats green; ruby -c, shellcheck (CI
invocation), actionlint clean.
- 3-round cr-loop (7 agents/round) converged to zero bucket-(a).
## Follow-ups (not in this PR)
- Defensive `.to_s.empty?` on `meta.imageDigest` extraction
(can't-happen on real Railway).
- Test-quality nits (capture_io scoping, weak bats glob, shared mock
fixture).
- Pre-existing verify-prod `succeeded_csv`/`GITHUB_OUTPUT` coupling
(graceful-degraded, audited STAY_IN_C).
- `deploy-to-railway.ts` births prod on `:latest` (the root provisioning
gap); CLI↔workflow notify equivalence (PR2).
## Test plan
- [ ] CI green on PR HEAD
- [ ] (post-merge) a real `gh workflow run showcase_promote.yml` shows
the drift line in the #team-showcase notification when staging is behind
:latest
Prod services are now born pinned to a resolved @sha256 digest (mirroring
the Ruby promote CLI's GHCR resolver) instead of the mutable :latest tag,
fail-loud on resolve failure, and goLive asserts the prod source.image is
digest-pinned. Adds DI-testable assertProdDigestPinned + coverage.
github-actions[bot]