Pin alignment fixes 9 validate-pins FAILs; ratchet the drift baseline
count and hash accordingly. Also tighten the _comment: document the exact
hash recipe (SHA-256 of the stderr-only [FAIL] lines, LC_ALL=C sort -u)
and correct baselineDemoCount semantics (exact expected demo count per
package; deviation either direction warns).
## Summary
Hardens the `--isolate` showcase verification flow across three areas:
**1. XDG state migration.** Isolate slot registry and per-run
rewritten-compose scratch dirs move off `/tmp` (wiped on reboot,
world-writable) to
`${XDG_STATE_HOME:-$HOME/.local/state}/copilotkit/showcase/` (`slots/` +
`runs/<name>/`). `/tmp` clearing silently destroyed a kept stack's
compose file and slot, making `--keep` unreliable. Run dirs are keyed by
the finalized project name (not PID) so a kept run is locatable for
manual teardown.
**2. Slot reaping + registry concurrency.** Since the state dir is now
persistent, slots are reaped by compose-project liveness (`docker ps
--filter label=com.docker.compose.project=<name>`), with PID/age
heuristics as fallback. The registry is made safe under concurrent
claimers: a sweep lock with heartbeat updates, own-pid lock release, and
tombstones; a claim-then-verify duplicate-name guard closing the TOCTOU
window; crash-safe reap ordering with compose-down of reap remnants and
a path-traversal guard. Failed `--isolate` setup no longer tears down
the default stack; half-initialized state is cleaned up on the way out.
Teardown uses `--volumes` everywhere, and a failed compose-down
preserves state for diagnosis. `--isolate` names are validated (must
start with lowercase letter/digit; `showcase` is reserved — it aliases
the default stack), and a fail-loud warning precedes pre-down of an
existing stack.
**3. `--keep` now actually persists an isolated stack.** Previously the
unconditional `trap restore_isolation EXIT` tore the stack down
regardless of `--keep`. Teardown is now gated on the keep flag
(`ISOLATE_KEEP` promoted to a global so it survives `cmd_test` return
into the trap scope): the slot + run dir are retained and a survival
notice prints the project name, the three offset host ports, and the
exact `docker compose -p <name> down` command — no silent port/slot
leak. A kept stack's live containers keep its slot from being reaped.
Shell-only — confined to `showcase/scripts/cli/_common.sh` +
`cmd-test.sh`; the harness TS only reads the env vars the shell exports
(unchanged). Follows up the `--keep` caveat documented in #5346.
## Review hardening
The branch went through an 8-round, 7-agent code-review loop with
red-green-verified fixes — that loop produced the state-machine
hardening commit (trap-scope fix, default-stack guards, registry
concurrency/teardown robustness, name validation) and grew the test
suite to pin every fix. A live end-to-end `--keep` verification run is
what surfaced the trap-scope bug (`--keep` silently not honored),
driving the `ISOLATE_KEEP` global fix.
## Test plan
- [x] `showcase/scripts/__tests__/isolate.bats` — 41 isolate tests
(red→green): XDG path resolution (+`XDG_STATE_HOME` override,
`~/.local/state` fallback, `runs/<name>`), liveness-based reaping (dead
project reaped/reclaimed, live project preserved), real-trap-path
`--keep` tests (no simulated-trap shortcuts), sweep/lock/tombstone race
pins (heartbeat resurrection, lock takeover, duplicate-name TOCTOU),
reap-order probe pinning live-slot protection, root/PID-reuse/DST
guards, and sentinel anti-vacuity discipline so trap tests cannot pass
vacuously.
- [x] Full `bats showcase/scripts/__tests__/` green, matching CI's Shell
script tests invocation.
- [x] shellcheck: no new warnings.
- [x] Live end-to-end: `bin/showcase test <slug> --d6 --isolate <name>
--keep` persists the stack under `~/.local/state/copilotkit/showcase`,
survival notice + manual teardown work, follow-up run reaps the stale
slot.
## Summary
- PR #5352's worker-side flap fixes never reached staging automatically:
`harness-workers` runs the same `showcase-harness` image as the
`harness` scheduler, but the SSOT's `ciBuilt: false` conflated "owns a
build slot" with "should be redeployed when its image is rebuilt" — so
main merges redeployed only the scheduler and the workers silently kept
running a stale image (a manual redeploy was required to ship the
fixes).
- This adds an `imageOf` field to the Railway SSOT so a rebuilt image
redeploys **all** of its consumers: the CI redeploy scope is now built
slots ∪ their `imageOf` consumers that declare the target env.
- Staging default scope becomes 27 (26 ciBuilt + `harness-workers` via
expansion); prod is unchanged at 26 (the worker is staging-only and the
expansion is env-aware).
## Design
- `imageOf: "<ssot-key>"` on consumer entries (`harness-workers` →
`harness`), enforced by a module-load invariant
`assertImageConsumersValid`: dangling targets, non-ciBuilt producers,
consumer chains, and consumer envs not a subset of the producer's all
fail loud at import; lookups are prototype-safe (`Object.hasOwn`).
- `expandImageConsumers` in `redeploy-env.ts` performs the env-aware,
single-level expansion and fails loud on unnormalized env names
(synonyms like `production` must go through `resolveEnv`) — the first
real consumer of `ENV_ID_BY_NAME`.
- Service-name resolution (`resolveTargetServices`/`runRedeploy`) now
rejects inherited `Object.prototype` keys with the proper
Unknown-service operator error.
- The explicit `--services` passthrough (a named service is attempted
even in an env it does not declare) is documented and contract-pinned by
a test.
## Review
- 5 unbiased 7-agent CR rounds plus a diff-attribution triage; every
diff-authored finding fixed with red-green proofs.
- ~30 pre-existing script-hygiene findings (env-registry consolidation,
accessor leniency, fetch timeout, parseArgs edges, coverage gaps in
`makeLiveRedeploy`/summary-JSON, etc.) deferred to the flap-fix
follow-up backlog.
## Test plan
- [x] 82/82 vitest (13 new tests: expansion, env-awareness, invariants
incl. prototype keys and env-subset, contract pins)
- [x] `tsc --noEmit -p showcase/scripts/tsconfig.json` clean
- [x] oxfmt clean on all changed files
- [ ] CI green on this PR
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The webhooks SSOT comment claimed the push-driven default scope is
"guaranteed" to leave webhooks untouched — false: a push touching the
build workflow files trips the workflow_config paths-filter disjunct,
which selects every matrix slot (webhooks included; its skip_build slot
still reports success and enters the redeploy CSV). Reworded to state
the actual behavior. imageOf doc now states the enforced NON-EMPTY
subset constraint; serviceEnvPairs doc now truthfully says it has no
consumers yet; file header notes the probe flag default and
bin/railway's Ruby-only "stage" synonym.
Test hygiene: drop the stale bin/railway line-number citation from a
test name, the _envConfigTypeAnchor (EnvironmentConfig is genuinely
referenced by the shape-compile test), a dead eslint-disable, and a
dead `as never` cast (env is an open string); align the webhooks
dispatch-name pin regex with the extraction regex's whitespace
tolerance.
assertEnvRegistryConsistent gains four clauses: (iv) a key present in
both ENV_IDS and ENV_ID_BY_NAME must carry the same env-id (ENV_IDS.prod
drifted to the staging id previously passed every clause while
resolveEnv("prod") silently returned staging); (v) every ENV_IDS env-id
must be carried by a canonical name (was only caught lazily in
resolveEnv); (vi) registry keys must be trim().toLowerCase()-normalized
(resolveEnv lowercases input, so a non-lowercase spelling is registered
but unreachable); (vii) no registry key may be an Object.prototype
property name.
expandImageConsumers' per-entry env skip-check becomes an own-property
test (Object.hasOwn) for uniformity with every other lookup in the file.
REDEPLOY_SUMMARY_JSON is trimmed before the set-but-empty branch so a
whitespace-only value hits the loud warn path instead of attempting a
JSON write against a garbage path.
- ciBuilt field doc: pocketbase IS showcase-CI-built — only webhooks
remains out-of-band; keep the MUST-NOT-touch claim for webhooks only.
- gateValidated field doc: true for every service EXCEPT the two
gateIgnore entries (harness-workers, harness-legacy), not "every
service".
- CI_BUILT_SERVICES comment: also names the excluded non-CI-built
harness-workers and harness-legacy alongside webhooks.
- legacyJsonCompat doc: it is bin/railway's EXPECTED_DOMAINS derivation
that filters out *.up.railway.app hosts (no "parity test rejects"
claim contradicting the placeholder data below).
- webhooks entry comment: a manual service=all build dispatch MAY
bounce webhooks staging (the skip_build slot still reports success,
entering the matrix ∩ success-set redeploy scope); only the
push-driven default scope is guaranteed to leave webhooks untouched.
- railway-envs: route envsFor/instanceIdFor/domainFor/probeEnabled (and
repoNameFor) through shared getEntry/getEnvCfg own-property helpers so
inherited Object.prototype keys on either axis produce the curated
error (or probeEnabled's contract false) instead of raw TypeErrors,
silent undefined, or a spurious probe=true.
- railway-envs: two new module-load invariants (synthetic-map
injectable): assertEnvRegistryConsistent (per-service env keys are
registered canonical names; ENV_ID_BY_NAME env-ids unique; every
canonical name has an ENV_IDS spelling) and
assertServiceAndInstanceIdsUnique (serviceId unique per entry,
instanceId globally unique).
- redeploy-env: invert the exit-code policy to fail-loud by default —
any env except the documented staging carve-out exits non-zero on
per-service failure (a future preview/canary env inherits fatal
semantics instead of silently swallowing failures).
- redeploy-env: sanitize per-service THROWN error messages through
sanitizeErrorBody before recording; flatten bare \r in the summary
table escape.
- redeploy-env: warn on set-but-empty REDEPLOY_SUMMARY_JSON; reject
flag-like --services CSV parts (both forms) and a flag-like first
argument (missing env); derive usage env lists from ENV_IDS.
- tests: prototype-key sweep across all accessors, invariant
positive/negative coverage, third-env exit-code pin, sanitization and
CLI-guard coverage, makeLiveRedeploy !res.ok and non-true mutation
branches.
Cross-session review fixes for the --isolate machinery (one concern:
source + test + docs).
1) Reaper reserved-name guard (critical): _reap_isolate_slot trusted
slot records — a record naming 'showcase' (corrupt, or written by an
older CLI version before apply_isolation reserved the name) passes
the charset regex, so the reap ran `docker compose -p showcase down
--remove-orphans --volumes` against the LIVE default stack,
destroying the PocketBase named volume. The reserved name now gets
the same treatment as the path-traversal guard: warn (naming the
record and why it is dangerous) and leave the slot intact for manual
inspection — no compose-down, no state removal.
Call-site enumeration: _reap_isolate_slot's sole caller is
_sweep_isolate_slots, at 3 sites (dead-PID reap, project-recorded/
no-owner reap, age-fallback reap), all passing
"$slot_entry" "$slot_proj" — all three flow through the new guard
identically.
Red-green: the new bats test ("a slot whose project record reads the
RESERVED 'showcase' is left intact...") was run against the UNFIXED
code first and FAILED — the sweep logged "Attempting to reclaim
stale slot 0 (project showcase has no live containers and no
recorded owner)" and reaped the slot. It passes with the guard.
2) .iso-bak restore race: two concurrent runs can both see a stale
backup; the loser's mv is the FINAL command of its `[ -f ] && mv`
AND-list, so its failure trips set -e and kills the CLI pre-claim
with a raw error. Both mv's now carry `2>/dev/null || true` — the
survivor's restore wins, the loser proceeds with restored originals.
3) Keep-test absence regexes greped only the `--project-name <name>
down` spelling; the reaper's own downs use `-p <name> down`, so a
keep-branch regression via the -p form passed undetected. Both keep
absence assertions now match `(--project-name|-p) <name> down`.
Mutation-verified: a temporary -p-form compose-down added to the
keep branch made BOTH broadened tests FAIL; reverted, suite green.
(All other absence assertions use the word-matched generic
`compose ... down` regex, which already covers both spellings.)
4) RUNBOOK.md/DEBUGGING.md contradicted shipped code: the manual
teardown was quoted without --volumes plus notes claiming
`down --remove-orphans` leaves named volumes (the shipped survival
notice and every teardown path include --volumes), and the name rule
was documented as `[a-z0-9_-]+` (actual: starts with [a-z0-9], then
[a-z0-9_-], uppercase normalized with a warn, 'showcase' reserved).
Both updated to the shipped semantics; the now-redundant separate
`down --volumes` snippets removed.
Verification: full `bats showcase/scripts/__tests__/` green (60 tests);
shellcheck on _common.sh shows no new warnings vs baseline
(pre-existing SC2034/SC2115 only, line-shifted).
Confirmation-CR bucket-(a) fixes for PR #5353 — one coherent concern:
env-name resolution has exactly ONE authority (the ENV_IDS /
ENV_ID_BY_NAME registries) and SSOT accessors fail loud instead of
silently returning wrong values.
- runRedeploy: resolve envId via ENV_ID_BY_NAME with an Object.hasOwn
guard + fail-loud throw listing the registered envs. Removes the
hardcoded `prod`/`staging` pair check and PRODUCTION/STAGING ternary
that contradicted the SSOT's documented open-env contract ("a new env
needs only a registry entry"); the registry lookup subsumes it.
- resolveEnv: derive resolution entirely from the registries (ENV_IDS
spellings -> env-id -> canonical ENV_ID_BY_NAME name) instead of its
own hardcoded synonym chain. Behavior identical for
prod/production/staging; still throws on unknowns, and now also
throws on a mis-wired registry (a spelling whose env-id has no
canonical name).
- serviceForDispatchName: fix the docstring's false "CI-built service"
claim — it does no ciBuilt filtering and tests pin the unfiltered
behavior (the non-CI-built webhooks resolves).
- repoNameFor: fail loud (consistent with instanceIdFor/domainFor)
instead of silently echoing the service name — the exact
silently-wrong-GHCR-name class this PR's hardening targets. Throws on
unknown service, on an env not registered in ENV_ID_BY_NAME
(unnormalized synonyms like "production"), and on a registered env
the service does not declare. Keeps the documented default (the
service name) for declared envs without an override.
Call-site enumeration confirming nothing relies on the old fallback:
- verify-railway-image-refs.ts:523 — iterates the entry's DECLARED
environments keys, registry-filtered, SSOT-matched service names
- __tests__/railway-envs.golden.test.ts:81 — iterates envsFor(name)
(declared envs only) over real SSOT keys
- railway-envs.test.ts repoNameFor cases — dual-env services,
prod/staging only
- __tests__/verify-railway-image-refs.test.ts:264 — FIVE_NEW keys,
all dual-env
- resolveTargetServices: throw when an explicitly-provided services
list resolves to zero entries (whitespace-only programmatic input)
instead of letting runRedeploy exit 0 having redeployed nothing; the
default undefined -> full CI-built scope is unchanged.
- makeLiveRedeploy: add signal: AbortSignal.timeout(30s) so a hung
Railway API records a per-service FAIL instead of stalling CI, and
pass GraphQL errors[].message through sanitizeErrorBody for
consistency with the HTTP-error path. Exported for direct unit tests.
Red-green: 11 new tests (open-env registry resolution incl. a
runtime-registered hypothetical env, repoNameFor negatives,
empty-resolution throw, abort-signal presence, GraphQL error
sanitization) all failed against the old code; full showcase/scripts
suite green (50 files, 1807 tests) + tsc --noEmit clean.
- ISOLATE_KEEP promoted to a global so --keep survives cmd_test return
into the trap scope
- early-die and default-stack protection: failed --isolate setup no
longer tears down the default stack; half-initialized state is
cleaned up on the way out
- liveness/PID/age reaping signals with a sweep lock: heartbeat
updates, own-pid lock release, tombstones, and a claim-then-verify
duplicate-name guard close slot-registry races (TOCTOU, lock
takeover, reap order)
- teardown robustness: --volumes on every compose down, failed-down
runs preserve state for diagnosis, reap remnants get a compose-down,
path-traversal guard, uniform rm guards under set -e
- name validation: --isolate names must start with a lowercase letter
or digit; reserved name 'showcase' rejected (it aliases the default
stack)
- fail-loud warning before pre-down of an existing stack; help text
updated
Result of an 8-round, 7-agent code-review loop with red-green
verified fixes.
harness-workers runs the SAME showcase-harness GHCR image as the harness
scheduler but has ciBuilt:false (it has no build slot of its own), and the
CI staging redeploy scope was derived purely from ciBuilt — so a main-merge
rebuild of showcase-harness:latest only bounced the scheduler while the
workers silently kept running the stale image (PR #5352's worker-side fixes
never reached staging).
Model image consumption explicitly in the SSOT instead:
- railway-envs.ts: new optional `imageOf` field on ServiceEntry — the SSOT
key of the ciBuilt service whose image this entry runs. Set
`imageOf: "harness"` on harness-workers. New module-load invariant
`assertImageConsumersValid` (fail-loud, same style as
assertDispatchNamesUnique): imageOf must name an existing SSOT key, the
target must be ciBuilt, and the consumer itself must not be ciBuilt.
- redeploy-env.ts: new `expandImageConsumers(names, env)` applied inside
runRedeploy — the redeploy set becomes the resolved scope PLUS any
service whose imageOf points at a service already in scope. Env-aware:
a consumer only joins envs it declares, so the staging-only worker
never enters a prod redeploy (prod behavior unchanged).
No workflow change needed: showcase_build.yml keeps passing the
built-and-successful dispatch_names; the script expands them. Gate
behavior (gateIgnore / image-ref gate), the generated JSON
(emit --check passes byte-identical), and the promote dropdown are all
untouched. harness-legacy deliberately gets no imageOf (pinned pre-fleet
digest; must not follow rebuilds).
Previously the EXIT-trap restore_isolation always tore down the isolated
stack, ignoring --keep. Now restore_isolation reads a keep flag (set in
cmd-test.sh when --keep is parsed): when kept it skips compose down, the run-dir
removal, and the slot release, and instead prints a survival notice with the
project, slot, the three offset host ports, and the exact manual teardown
command. The kept stack's live containers keep its slot from being reaped.
Persist the compose project name into each claimed slot dir, and at claim time
reap any slot whose recorded project has no live containers (queried via
docker ps --filter label=com.docker.compose.project). This correctly leaves a
--keep'd stack's slot alone since its containers are still up. The existing
PID/age heuristics remain as a fallback for slots predating the project file.
Move the --isolate slot registry and per-run scratch dir off /tmp (wiped on
reboot, world-writable) to $XDG_STATE_HOME/copilotkit/showcase (slots/ and
runs/<name>). The run dir is now keyed by the finalized project name instead
of the PID so a kept run is locatable for manual teardown. Adds a bats suite
covering the new state-base helper and run-dir location.
llamaindex, agno, and claude-sdk-python emit AG-UI REASONING_MESSAGE_*
events but pinned @ag-ui/client ^0.0.43, whose @ag-ui/core discriminated
union lacks the REASONING_MESSAGE_* variants — the frontend threw
invalid_union_discriminator and the reasoning demo broke. Pin all three
to exact 0.0.48 (matching the claude-sdk-typescript fix in #5323),
regenerate their lockfiles so @ag-ui/core resolves to 0.0.48 with zero
0.0.43 nodes, and ratchet the validate-pins drift baseline down from 60
to 57 to reflect the now-exact pins. Verified locally on the built-image
showcase rig: D5 green and the D6 reasoning-display probe passes for all
three backends with zero invalid_union_discriminator.
When a promote fails, the succeeded-service set is empty, so verify-prod
hits its skip branch (`exit 0`). The GitHub job result is therefore
`success`, and the notify step rendered `verify-prod=success` in the
#oss-alerts Slack message — a misleading green, since prod was never
probed.
verify-prod now exports a `status` output: `success` after a real probe
passes, `skipped` on the empty-CSV skip. notify reads that output (via
the new bats-tested verify-prod-display.sh) instead of the raw job
result, so the Slack line accurately reads `verify-prod=skipped` vs
`success` vs `failure`. A genuine probe failure / contract violation
exits non-zero (job result `failure`, status never written), and the
display falls back to the job result. Slack formatting is unchanged.
Extracts the display mapping into showcase/scripts/verify-prod-display.sh
(mirroring promote-fleet.sh) with red-green bats coverage, and adds it to
the showcase_validate.yml shellcheck step.
The pool-fleet worker's Railway service is named `harness-workers` (PLURAL),
but the SSOT keyed it `showcase-harness-worker` (singular). The image-ref gate
matches SSOT keys to Railway service names verbatim, so the gate reported
`harness-workers` as an untracked Railway service AND the stale singular key
matched nothing. Rename the SSOT key (and every test/fixture reference) to the
exact Railway name `harness-workers`.
It stays the staging-only, domainless, probe-disabled worker that runs the
shared `showcase-harness` image: serviceId c2aa8a0b-…, staging instance
362c1e37-…, ciBuilt:false, gateIgnore:true, no build slot (so no dispatchName),
single `staging` env with no domain. Add a focused test pinning that shape.
Counts are unchanged (29 services / 26 CI_BUILT) — this is a rename, not an
addition; both harness workers already existed on main.
Verified LOCALLY against Railway: verify-railway-image-refs reports
`54 env-scoped instances verified (2 skipped)` — 0 violations, 0 missing, 0
untracked (harness-workers reconciled, harness-workers + harness-legacy the 2
gateIgnore'd skips). emit --check zero drift, Ruby parity green (borrowed
.up.railway.app host is parity-excluded), full scripts suite + typecheck green.
Replace ServiceEntry's parallel prodInstanceId/stagingInstanceId/domains/probe/
repoNameOverride fields with a single environments: Record<string, {instanceId,
domain?, probe?, repoName?}> map plus a hoisted env-independent probeDriver.
EnvName becomes an open string backed by an ENV_ID_BY_NAME registry so
accessors resolve arbitrary env names; a single-env service (the staging-only
showcase-harness-worker) now simply omits the absent env instead of carrying a
placeholder ID/borrowed host.
Accessors instanceIdFor/domainFor/repoNameFor index environments[env]
(domainFor still throws on missing/scheme); add envsFor(name),
serviceEnvPairs(), and probeEnabled(name, env). Generalize the image-ref gate
(iterate each service's declared environments, resolve env-id via the
registry, sum/iterate missingByEnv over registry env names) and verify-deploy's
host->env reverse-map (envForTarget iterates environments).
Pure TS-internal: emit-railway-envs-json.ts projects the env-map back onto the
FROZEN legacy JSON shape (prodInstanceId/stagingInstanceId/domains/probe/
repoNameOverride) via a documented legacyJsonCompat shim for the two domainless
harness workers, so railway-envs.generated.json stays byte-identical and Ruby
(bin/railway) + workflow jq + the parity test are untouched. Verified: emit
--check zero drift, Ruby test_expected_domains_parity green, golden snapshot
toEqual proves byte-identical resolution for every real (service, env) pair,
full scripts vitest suite green, showcase scripts typecheck clean.
Serializes the fully-resolved {service -> env -> {instanceId, domain, probe,
driver, repoName}} projection for all 29 services x 2 envs via the public
accessors (instanceIdFor/domainFor/repoNameFor) + per-entry probe config,
frozen as a fixture. This is the behavior-preservation guard for the
forthcoming env-map (Option C) refactor: resolved values must stay
byte-identical before and after.
Move gen-ui-interrupt + interrupt-headless from features: to
not_supported_features: across affected integration manifests, and align
the generate-registry/generate-catalog scripts tests to the resulting
wired-feature counts (derive expected lengths from the parsed manifest
rather than hardcoding pre-quarantine numbers).
Make the showcase dev tool faithful to staging by construction. Two changes:
1. `showcase test --d5/--d6` now drives the fleet CONTROL-PLANE (producer ->
probe_jobs queue -> worker -> result-aggregator) instead of the legacy
in-process runLevel() driver. The new cli/control-plane-run.ts replicates
the deep/full producer tick exactly as runControlPlane wires it
(createE2eDeepServiceEnumerator / createServiceEnumerator over
createJobProducer + createFleetQueueClient), enqueues one operator-triggered
tick, and polls local PocketBase for the run's terminal cells. The running
worker fleet claims + runs the driver + the aggregator writes the d5/d6
status cells, so the dev tool exercises the IDENTICAL wiring + concurrency
as staging. The old in-process path stays available behind `--direct`.
2. `showcase up --dev` adds a docker-compose.dev.yml overlay that bind-mounts
each integration's source and overrides the run command with a stack-aware
hot-reload entrypoint (shared/dev/dev-entrypoint.sh: uvicorn --reload for
FastAPI agents, langgraph dev for graphs, next dev for the frontend). Edit a
source file and the component reloads in place with no image rebuild. The
built-image mode remains the faithful/staging-equivalent default.
The showcase_build verify-image-refs gate (SSOT = showcase/scripts/railway-envs.ts)
was failing with "1 untracked Railway services" because the interim
harness-legacy staging service (the legacy all-probe harness kept live during
the pool-fleet migration) exists on Railway but had no SSOT entry. That
Railway->SSOT drift check skips the build, so nothing deploys.
Adds a harness-legacy SERVICES entry mirroring the showcase-harness-worker
precedent (PR #5280): ciBuilt:false (not built by showcase_build, runs a pinned
out-of-band digest) and gateIgnore:true (deliberately-untracked for the image-ref
gate). findUntrackedServices treats any SSOT entry as known, so this clears the
untracked failure; gateValidated:false keeps findMissingServices from flagging
it. Real serviceInstance IDs for both envs recorded from Railway GraphQL.
Regenerates railway-envs.generated.json and updates the service-count /
gate-ignored carve-out assertions (28->29 services).
Verified: live verify-railway-image-refs.ts now exits 0 ("54 env-scoped
instances verified, 2 skipped"); without the entry it exits 1 with the
harness-legacy untracked failure. Full scripts test suite green (1771 passed).
The pool-fleet cutover manually created the staging-only
`showcase-harness-worker` Railway service (HARNESS_ROLE=worker, 2
replicas) and flipped the existing `harness` service to
HARNESS_ROLE=control-plane. The new service was untracked in the SSOT,
so verify-railway-image-refs.ts failed the "Showcase: Build & Push"
workflow on every push to main (1 untracked Railway service), which
skipped the harness `build` job and blocked harness image rebuilds.
Add the worker to SERVICES as a staging-only, domain-less queue worker:
- ciBuilt:false — it runs the SAME `showcase-harness` image the existing
harness build slot produces; there is no separate worker build.
- gateIgnore:true / gateValidated:false — no prod instance and no public
domain, so it does not fit the symmetric dual-env shape the image-ref
gate validates. gateIgnore clears the "untracked Railway service"
failure (any SSOT entry counts as known) without tripping a false
"missing from prod" failure.
- repoNameOverride → showcase-harness so the image-ref shape resolves.
- probe disabled in both envs (no externally-reachable health endpoint).
Regenerate railway-envs.generated.json and update the SSOT-count and
gate-coverage test invariants (27→28 services; worker is the sole
intentional gateIgnore/gateValidated:false entry).
Close the recurring "mutation result not validated" defect class in the
starter-fleet provisioner and make the existing-services snapshot fail loud
instead of silently feeding erroneous create decisions.
- Add a uniform assertMutationOk guard and route EVERY mutation through it:
serviceCreate (assert .id), serviceInstanceUpdate (was DISCARDED — a false
Boolean! return meant sleep/healthcheck/image/creds were never applied while
the script reported success; now asserted and "configured" is logged only
after verification), serviceDomainCreate (assert .domain on the create path),
and serviceInstanceRedeploy (routed through the same guard for consistency).
- Absorb a serviceCreate "already exists" rejection: on a snapshot-miss the
create path now re-fetches the service id by name and falls through to UPDATE
instead of aborting the whole fleet. Predicate renamed ALREADY_EXISTS_RE and
reused by the domain-create path.
- fetchExistingServices fails loud on page-drain truncation (hasNextPage still
true at the defensive bound) rather than returning a partial byName map.
- fetchExistingServices coalesces null serviceInstances/.edges (transitional
service nodes) so an unguarded .find can't TypeError and abort the fetch;
interface fields marked optional/nullable.
- TRANSIENT_ERROR_RE made single-line ([^\n]*? not [\s\S]*?) so a newline-joined
multi-error blob can't bridge "Service" and "not found" across lines.
- withRetry wraps the schedule-exhaustion rethrow with context and { cause }.
Three functional fixes to the starter-fleet provisioner found in CR:
- fetchExistingServices now drains the Relay ServiceConnection via
pageInfo.hasNextPage/endCursor. A single un-paginated query truncated the
snapshot (~27 SSOT + 12 starter services span >1 page), making an existing
starter look absent → CREATE path → serviceCreate "already exists" →
non-transient abort of the whole run.
- TRANSIENT_ERROR_RE now matches Railway's INTERPOLATED "Service <id> not
found" (id embedded), not just the contiguous "Service not found", so the
post-create eventual-consistency retry actually fires.
- serviceInstanceRedeploy result check: documented the verified Boolean!
contract (sources: redeploy-env.ts, bin/railway RestoreCommand) and now
gates on truthiness (rejects false/null, accepts truthy defensively).
Hardening: ABORT on the live path when GITHUB_TOKEN is unset (private GHCR
images would image-pull-backoff while reporting success); warn-and-continue
only under --dry-run. Benign domain "already exists" no-op now logs the actual
matched Railway message for a forensic trail.
Harden the committed starter-fleet Railway provisioner against the
partial-failure / mistyped-flag / un-deployed-image failure modes
surfaced in CR:
- Domain idempotency: a serviceDomainCreate that Railway rejects with an
"already exists" error (start-of-run snapshot missed the domain due to
eventual consistency, or a prior run died mid-fleet) is now caught as a
benign no-op (marked "existing", logged) so a re-run converges instead
of aborting the entire remaining fleet. A genuine non-transient error
still aborts.
- Explicit redeploy: serviceCreate + serviceInstanceUpdate(source.image)
only PINS the image; it does not start a deployment, and Railway's image
auto-updates fire only on a NEW digest push. Added serviceInstanceRedeploy
after the instance update on BOTH the create and update paths so the
pinned image actually runs (and starter_smoke can find the service up).
Mirrors the documented update+redeploy pattern in bin/railway and the
explicit redeploy showcase_deploy.yml issues after each GHCR push.
- argv validation: parseArgs() now rejects any unrecognized argument
(e.g. a mistyped --dry-rn) with a usage hint before any provisioning,
instead of silently ignoring it and proceeding to REAL live provisioning.
- Fail-fast safety: validate the Railway token AND registry credentials
up front in main() (token resolution no longer process.exit()s deep in
the GraphQL boundary; main().catch owns the exit). Broadened the
withRetry transient predicate to the domain/instance eventual-consistency
class via an overridable per-call predicate. Dry-run now reports a new
service's domain as "would-create" for a faithful preview.
Adds showcase/scripts/provision-starter-fleet.ts — a committed, idempotent
provisioner for the SSOT-decoupled "starter container fleet". It creates (or
updates) one sleepable Railway service per starter template in the STAGING
environment, deriving the 12 targets from STARTER_TO_COLUMN (the smoke-matrix
SSOT) so the fleet can never drift from the build matrix.
Per service: serviceCreate scoped to the STAGING env (environmentId on
ServiceCreateInput, so NO production instance is ever materialized) with
source.image=ghcr.io/copilotkit/starter-<slug>:latest (RAW starter slug) and
GHCR registryCredentials; then serviceInstanceUpdate against staging with
sleepApplication:true + healthcheckPath="/" + region=us-west1; then
serviceDomainCreate for a generated staging domain. A bounded retry absorbs
Railway's eventual-consistency "ServiceInstance not found" right after create.
Healthcheck is "/" not "/api/health": the starters' single deployable image
EXPOSEs 3000 running the Next.js frontend, which serves "/" and
"/api/copilotkit" but has no "/api/health" route; the agent's "/health" is on
the internal 8123 port Railway does not expose. region read-back is null on the
serviceInstance for ALL existing showcase services too — that is normal Railway
behavior, so the fleet matches the existing services.
The fleet is decoupled from the 27-service railway-envs SSOT (starter-* services
are auto-discovered by the starter_smoke probe). #5254 already made
verify-railway-image-refs.ts tolerate starter-* names, so provisioning does not
trip the image-ref gate / skip the showcase build.
Red-green tested against an injected Railway GraphQL mock: target derivation
(raw vs remapped slug), GHCR credential resolution, sleepApplication:true,
staging-env scoping on BOTH create and update (never prod), idempotent
update-vs-create, domain de-duplication, and transient-error retry.
The starter container fleet (starter-<slug>) is decoupled from the
27-service railway-envs SSOT: each starter-* service is auto-discovered
at runtime by the starter_smoke probe (railway-services discovery,
namePrefix "starter-") and is never read from railway-envs.ts.
verify-railway-image-refs.ts is a hard needs: of the build job and runs a
bidirectional live-Railway drift check. Before this change, provisioning
a starter-* service made it untracked in the SSOT, so findUntrackedServices
failed and the showcase build was SKIPPED (the canary regression).
Scope both drift checks (findUntrackedServices and, defensively,
findMissingServices) to exclude services matched by a single, well-named
predicate isStarterFleetService(name) => name.startsWith("starter-"),
mirroring the harness discovery filter convention. Real showcase-*/infra
services are still drift-checked exactly as before.
This is the prerequisite for Phase-3 starter provisioning — provisioning
must happen AFTER this merges.
PocketBase had no CI build path: `ghcr.io/copilotkit/showcase-pocketbase`
was a stale April `:latest`, and there was no way to ship pb_migrations /
pb_hooks changes without an ad-hoc manual build. Add a `pocketbase` slot to
showcase_build.yml's build matrix, mirroring the harness/aimock entries:
- dispatch_name `showcase-pocketbase`, context `showcase/pocketbase`, its
own Dockerfile, health `/api/health`, railway_id from the SSOT.
- a paths-filter key gated to `showcase/pocketbase/**` so the slot only
rebuilds when PB's own files change (the image is self-contained — no
shared-module copy), not on every showcase push.
- the workflow_dispatch service choice so PB is human-targetable.
Flip the SSOT entry (railway-envs.ts) to `ciBuilt: true` with
`dispatchName: "showcase-pocketbase"` so it is built+pushed (`:sha` +
`:latest`) and joins the default staging-redeploy scope; the build's
redeploy step only touches the matrix-intersect-success set, so PB still
only redeploys when its own files change. Regenerate
railway-envs.generated.json and the showcase_promote.yml service dropdown,
and update the SSOT/redeploy tests that pinned PB as out-of-band
(CI_BUILT_SERVICES 25 -> 26; webhooks stays the only non-CI-built service).
Bump canonicalCopilotKitVersion 1.59.2 -> 1.59.4 and pin every
integration's @copilotkit/* to 1.59.4 (locks regenerated). Keeps the
whole showcase on one version instead of letting the langgraph A2UI
demos deviate. Existing per-slug overrides (built-in-agent pkg.pr.new,
ms-agent-harness-dotnet 1.57.2) unchanged.
## Summary
Hardens a set of pre-existing `showcase` promote/`bin/railway` bugs
surfaced during review of the GHCR bearer fix (#5239) and the
#team-showcase notification (#5240). Each is an independent, real
defect; all changes are covered by tests (Ruby `bin/spec` **127 runs / 0
failures**, TS `verify-deploy` drivers **82 passing**, `actionlint`
clean).
## Fixes
1. **`rollback` could roll back to the wrong deployment.**
`find_previous_deployment` selected the second-newest SUCCESS from an
*unsorted* GraphQL result, and assumed the head deploy was always
SUCCESS — so when the latest deploy FAILED/CRASHED (the exact case
rollback is for) it rolled back one good deploy too far. Now sorts by
`createdAt` desc, selects the newest SUCCESS strictly older than the
current head (`sorted.drop(1).find { SUCCESS }`), and **fails loud**
(rather than silently mis-rolling) when the `first: N` window is
saturated with no valid target — telling the operator to pass `--to`.
2. **`env-diff` was dishonest.** It advertised custom-domain comparison
it never performed, and exposed a `--ignore-env-scoped` flag that was
parsed but never read. Now actually diffs `custom_domains` (the snapshot
already carried them), removes the dead flag/helper, and nil-guards
**every** accessor in `diff_services` (`services`, `env_keys`,
`custom_domains`) consistent with the rest of the file.
3. **`verify-prod` could vacuously pass.** Its empty-`succeeded_csv`
branch `exit 0`'d unconditionally. Now fails loud if `promote` reported
success but produced no succeeded set (contract violation), while still
skipping cleanly when promote genuinely failed.
4. **`verify-prod` raced the prod rollout.** It failed instantly when
the just-promoted deploy was still `DEPLOYING` (observed live: promote
succeeded, verify-prod failed ~17s later mid-rollout). `verify-deploy`
now polls in-progress statuses
(`DEPLOYING`/`BUILDING`/`INITIALIZING`/`WAITING`/`QUEUED`/`NEEDS_APPROVAL`)
until terminal (~150s budget), still failing fast on
`FAILED`/`CRASHED`/`REMOVED`.
5. **`npx tsx` ran from the wrong cwd** in the verify jobs (deps
installed in `showcase/scripts`, invoked from repo root → could fetch
`tsx` from the network). Now runs with `working-directory:
showcase/scripts`, matching the resolve/promote jobs.
6. **Slack payloads rendered literal `\n`.** Both promote Slack posts
used `toJSON(format('...\n...'))`, where the literal `\n` survives as
backslash-n (verified live in #team-showcase). Now uses the
`fromJSON('"\n"')` idiom for real line breaks.
## Test plan
- [x] `showcase/bin/spec` — 127 runs, 0 failures (new: rollback
head-FAILED + saturated-window, env-diff custom-domain + nil-guard
cases)
- [x] `showcase/scripts` `tsc --noEmit` clean; `verify-deploy` driver
tests 82 passing (in-progress→SUCCESS, →timeout, fast-fail-on-terminal)
- [x] `actionlint` clean
- [ ] CI green
## Out of scope (separate follow-up)
Review surfaced further pre-existing items intentionally NOT fixed here
(no diff overlap): `run_staging_probe`'s `IO.popen` nests its options
hash inside the argv array (stderr redirect verified working; the
`child_env` hash isn't applied but the probe inherits the parent env in
CI); `image_shape`/`parse_image_ref`/`PinCommand` colon-splitting for
registry-port refs (latent — portless `ghcr.io` only); the deeper
`DEPLOYMENTS_QUERY first:10` truncation beyond the new fail-loud guard;
`succeeded_csv` integrity under a 20-min promote-job timeout; and a few
comment/test-coverage nits.
verify-prod commonly runs seconds after a promote pins a new image digest,
while Railway is still rolling the container out. checkDeploymentSuccess (in
verify-deploy.drivers.baseline.ts — the SSOT for the deployment-SUCCESS gate
shared by every driver) treated a transient in-progress status as a hard FAIL:
promote run 26966193624's predecessor pinned the docs digest, then verify-prod
fired ~17s later and FAILED with status="DEPLOYING" — a race, not a failure.
Now the deployment-status check polls when the latest deployment sits in any
non-terminal Railway status (QUEUED/BUILDING/INITIALIZING/DEPLOYING/WAITING/
NEEDS_APPROVAL), re-querying every 5s up to a 150s budget until it reaches a
terminal state, then asserts SUCCESS. Terminal-failure statuses (FAILED/
CRASHED/REMOVED/any non-SUCCESS-non-in-progress) still fail FAST with the
original error-string shape — no waiting. Infra/contract errors (network,
GraphQL errors[], missing edge) also fail fast.
The poll loop is fully seamed for tests (injectable sleep/now/budget); the
signature stays backward-compatible (trailing optional pollOpts). probeBaseline
forwards an optional deployPoll through, and both --env staging and --env prod
go through the identical code path, so staging behavior is unchanged.
Red-green tests: in-progress-then-SUCCESS passes (and polls), in-progress-until-
timeout fails with "still in progress", terminal FAILED fails fast with zero
sleeps.
declarative-gen-ui moved to the CopilotKitMiddleware auto-A2UI path across
the 3 langgraph integrations. The middleware's inner forced tool is
render_a2ui, so each integration's gen-ui-declarative.json gained 4
render_a2ui fixtures that share match keys with the pre-existing render_a2ui
entries in that integration's render-a2ui.json (the a2ui_fixed demo). 4
pills x 3 integrations = 12. Same-context cross-demo overlap, disambiguated
at runtime by the probe fixtureFile.
Jordan Ritter