## Summary
Hardens the `--isolate` showcase verification flow across three areas:
**1. XDG state migration.** Isolate slot registry and per-run
rewritten-compose scratch dirs move off `/tmp` (wiped on reboot,
world-writable) to
`${XDG_STATE_HOME:-$HOME/.local/state}/copilotkit/showcase/` (`slots/` +
`runs/<name>/`). `/tmp` clearing silently destroyed a kept stack's
compose file and slot, making `--keep` unreliable. Run dirs are keyed by
the finalized project name (not PID) so a kept run is locatable for
manual teardown.
**2. Slot reaping + registry concurrency.** Since the state dir is now
persistent, slots are reaped by compose-project liveness (`docker ps
--filter label=com.docker.compose.project=<name>`), with PID/age
heuristics as fallback. The registry is made safe under concurrent
claimers: a sweep lock with heartbeat updates, own-pid lock release, and
tombstones; a claim-then-verify duplicate-name guard closing the TOCTOU
window; crash-safe reap ordering with compose-down of reap remnants and
a path-traversal guard. Failed `--isolate` setup no longer tears down
the default stack; half-initialized state is cleaned up on the way out.
Teardown uses `--volumes` everywhere, and a failed compose-down
preserves state for diagnosis. `--isolate` names are validated (must
start with lowercase letter/digit; `showcase` is reserved — it aliases
the default stack), and a fail-loud warning precedes pre-down of an
existing stack.
**3. `--keep` now actually persists an isolated stack.** Previously the
unconditional `trap restore_isolation EXIT` tore the stack down
regardless of `--keep`. Teardown is now gated on the keep flag
(`ISOLATE_KEEP` promoted to a global so it survives `cmd_test` return
into the trap scope): the slot + run dir are retained and a survival
notice prints the project name, the three offset host ports, and the
exact `docker compose -p <name> down` command — no silent port/slot
leak. A kept stack's live containers keep its slot from being reaped.
Shell-only — confined to `showcase/scripts/cli/_common.sh` +
`cmd-test.sh`; the harness TS only reads the env vars the shell exports
(unchanged). Follows up the `--keep` caveat documented in #5346.
## Review hardening
The branch went through an 8-round, 7-agent code-review loop with
red-green-verified fixes — that loop produced the state-machine
hardening commit (trap-scope fix, default-stack guards, registry
concurrency/teardown robustness, name validation) and grew the test
suite to pin every fix. A live end-to-end `--keep` verification run is
what surfaced the trap-scope bug (`--keep` silently not honored),
driving the `ISOLATE_KEEP` global fix.
## Test plan
- [x] `showcase/scripts/__tests__/isolate.bats` — 41 isolate tests
(red→green): XDG path resolution (+`XDG_STATE_HOME` override,
`~/.local/state` fallback, `runs/<name>`), liveness-based reaping (dead
project reaped/reclaimed, live project preserved), real-trap-path
`--keep` tests (no simulated-trap shortcuts), sweep/lock/tombstone race
pins (heartbeat resurrection, lock takeover, duplicate-name TOCTOU),
reap-order probe pinning live-slot protection, root/PID-reuse/DST
guards, and sentinel anti-vacuity discipline so trap tests cannot pass
vacuously.
- [x] Full `bats showcase/scripts/__tests__/` green, matching CI's Shell
script tests invocation.
- [x] shellcheck: no new warnings.
- [x] Live end-to-end: `bin/showcase test <slug> --d6 --isolate <name>
--keep` persists the stack under `~/.local/state/copilotkit/showcase`,
survival notice + manual teardown work, follow-up run reaps the stale
slot.
Cross-session review fixes for the --isolate machinery (one concern:
source + test + docs).
1) Reaper reserved-name guard (critical): _reap_isolate_slot trusted
slot records — a record naming 'showcase' (corrupt, or written by an
older CLI version before apply_isolation reserved the name) passes
the charset regex, so the reap ran `docker compose -p showcase down
--remove-orphans --volumes` against the LIVE default stack,
destroying the PocketBase named volume. The reserved name now gets
the same treatment as the path-traversal guard: warn (naming the
record and why it is dangerous) and leave the slot intact for manual
inspection — no compose-down, no state removal.
Call-site enumeration: _reap_isolate_slot's sole caller is
_sweep_isolate_slots, at 3 sites (dead-PID reap, project-recorded/
no-owner reap, age-fallback reap), all passing
"$slot_entry" "$slot_proj" — all three flow through the new guard
identically.
Red-green: the new bats test ("a slot whose project record reads the
RESERVED 'showcase' is left intact...") was run against the UNFIXED
code first and FAILED — the sweep logged "Attempting to reclaim
stale slot 0 (project showcase has no live containers and no
recorded owner)" and reaped the slot. It passes with the guard.
2) .iso-bak restore race: two concurrent runs can both see a stale
backup; the loser's mv is the FINAL command of its `[ -f ] && mv`
AND-list, so its failure trips set -e and kills the CLI pre-claim
with a raw error. Both mv's now carry `2>/dev/null || true` — the
survivor's restore wins, the loser proceeds with restored originals.
3) Keep-test absence regexes greped only the `--project-name <name>
down` spelling; the reaper's own downs use `-p <name> down`, so a
keep-branch regression via the -p form passed undetected. Both keep
absence assertions now match `(--project-name|-p) <name> down`.
Mutation-verified: a temporary -p-form compose-down added to the
keep branch made BOTH broadened tests FAIL; reverted, suite green.
(All other absence assertions use the word-matched generic
`compose ... down` regex, which already covers both spellings.)
4) RUNBOOK.md/DEBUGGING.md contradicted shipped code: the manual
teardown was quoted without --volumes plus notes claiming
`down --remove-orphans` leaves named volumes (the shipped survival
notice and every teardown path include --volumes), and the name rule
was documented as `[a-z0-9_-]+` (actual: starts with [a-z0-9], then
[a-z0-9_-], uppercase normalized with a warn, 'showcase' reserved).
Both updated to the shipped semantics; the now-redundant separate
`down --volumes` snippets removed.
Verification: full `bats showcase/scripts/__tests__/` green (60 tests);
shellcheck on _common.sh shows no new warnings vs baseline
(pre-existing SC2034/SC2115 only, line-shifted).
Previously the EXIT-trap restore_isolation always tore down the isolated
stack, ignoring --keep. Now restore_isolation reads a keep flag (set in
cmd-test.sh when --keep is parsed): when kept it skips compose down, the run-dir
removal, and the slot release, and instead prints a survival notice with the
project, slot, the three offset host ports, and the exact manual teardown
command. The kept stack's live containers keep its slot from being reaped.
Persist the compose project name into each claimed slot dir, and at claim time
reap any slot whose recorded project has no live containers (queried via
docker ps --filter label=com.docker.compose.project). This correctly leaves a
--keep'd stack's slot alone since its containers are still up. The existing
PID/age heuristics remain as a fallback for slots predating the project file.
Move the --isolate slot registry and per-run scratch dir off /tmp (wiped on
reboot, world-writable) to $XDG_STATE_HOME/copilotkit/showcase (slots/ and
runs/<name>). The run dir is now keyed by the finalized project name instead
of the PID so a kept run is locatable for manual teardown. Adds a bats suite
covering the new state-base helper and run-dir location.
When a promote fails, the succeeded-service set is empty, so verify-prod
hits its skip branch (`exit 0`). The GitHub job result is therefore
`success`, and the notify step rendered `verify-prod=success` in the
#oss-alerts Slack message — a misleading green, since prod was never
probed.
verify-prod now exports a `status` output: `success` after a real probe
passes, `skipped` on the empty-CSV skip. notify reads that output (via
the new bats-tested verify-prod-display.sh) instead of the raw job
result, so the Slack line accurately reads `verify-prod=skipped` vs
`success` vs `failure`. A genuine probe failure / contract violation
exits non-zero (job result `failure`, status never written), and the
display falls back to the job result. Slack formatting is unchanged.
Extracts the display mapping into showcase/scripts/verify-prod-display.sh
(mirroring promote-fleet.sh) with red-green bats coverage, and adds it to
the showcase_validate.yml shellcheck step.
The pool-fleet worker's Railway service is named `harness-workers` (PLURAL),
but the SSOT keyed it `showcase-harness-worker` (singular). The image-ref gate
matches SSOT keys to Railway service names verbatim, so the gate reported
`harness-workers` as an untracked Railway service AND the stale singular key
matched nothing. Rename the SSOT key (and every test/fixture reference) to the
exact Railway name `harness-workers`.
It stays the staging-only, domainless, probe-disabled worker that runs the
shared `showcase-harness` image: serviceId c2aa8a0b-…, staging instance
362c1e37-…, ciBuilt:false, gateIgnore:true, no build slot (so no dispatchName),
single `staging` env with no domain. Add a focused test pinning that shape.
Counts are unchanged (29 services / 26 CI_BUILT) — this is a rename, not an
addition; both harness workers already existed on main.
Verified LOCALLY against Railway: verify-railway-image-refs reports
`54 env-scoped instances verified (2 skipped)` — 0 violations, 0 missing, 0
untracked (harness-workers reconciled, harness-workers + harness-legacy the 2
gateIgnore'd skips). emit --check zero drift, Ruby parity green (borrowed
.up.railway.app host is parity-excluded), full scripts suite + typecheck green.
Replace ServiceEntry's parallel prodInstanceId/stagingInstanceId/domains/probe/
repoNameOverride fields with a single environments: Record<string, {instanceId,
domain?, probe?, repoName?}> map plus a hoisted env-independent probeDriver.
EnvName becomes an open string backed by an ENV_ID_BY_NAME registry so
accessors resolve arbitrary env names; a single-env service (the staging-only
showcase-harness-worker) now simply omits the absent env instead of carrying a
placeholder ID/borrowed host.
Accessors instanceIdFor/domainFor/repoNameFor index environments[env]
(domainFor still throws on missing/scheme); add envsFor(name),
serviceEnvPairs(), and probeEnabled(name, env). Generalize the image-ref gate
(iterate each service's declared environments, resolve env-id via the
registry, sum/iterate missingByEnv over registry env names) and verify-deploy's
host->env reverse-map (envForTarget iterates environments).
Pure TS-internal: emit-railway-envs-json.ts projects the env-map back onto the
FROZEN legacy JSON shape (prodInstanceId/stagingInstanceId/domains/probe/
repoNameOverride) via a documented legacyJsonCompat shim for the two domainless
harness workers, so railway-envs.generated.json stays byte-identical and Ruby
(bin/railway) + workflow jq + the parity test are untouched. Verified: emit
--check zero drift, Ruby test_expected_domains_parity green, golden snapshot
toEqual proves byte-identical resolution for every real (service, env) pair,
full scripts vitest suite green, showcase scripts typecheck clean.
Serializes the fully-resolved {service -> env -> {instanceId, domain, probe,
driver, repoName}} projection for all 29 services x 2 envs via the public
accessors (instanceIdFor/domainFor/repoNameFor) + per-entry probe config,
frozen as a fixture. This is the behavior-preservation guard for the
forthcoming env-map (Option C) refactor: resolved values must stay
byte-identical before and after.
Move gen-ui-interrupt + interrupt-headless from features: to
not_supported_features: across affected integration manifests, and align
the generate-registry/generate-catalog scripts tests to the resulting
wired-feature counts (derive expected lengths from the parsed manifest
rather than hardcoding pre-quarantine numbers).
The showcase_build verify-image-refs gate (SSOT = showcase/scripts/railway-envs.ts)
was failing with "1 untracked Railway services" because the interim
harness-legacy staging service (the legacy all-probe harness kept live during
the pool-fleet migration) exists on Railway but had no SSOT entry. That
Railway->SSOT drift check skips the build, so nothing deploys.
Adds a harness-legacy SERVICES entry mirroring the showcase-harness-worker
precedent (PR #5280): ciBuilt:false (not built by showcase_build, runs a pinned
out-of-band digest) and gateIgnore:true (deliberately-untracked for the image-ref
gate). findUntrackedServices treats any SSOT entry as known, so this clears the
untracked failure; gateValidated:false keeps findMissingServices from flagging
it. Real serviceInstance IDs for both envs recorded from Railway GraphQL.
Regenerates railway-envs.generated.json and updates the service-count /
gate-ignored carve-out assertions (28->29 services).
Verified: live verify-railway-image-refs.ts now exits 0 ("54 env-scoped
instances verified, 2 skipped"); without the entry it exits 1 with the
harness-legacy untracked failure. Full scripts test suite green (1771 passed).
The pool-fleet cutover manually created the staging-only
`showcase-harness-worker` Railway service (HARNESS_ROLE=worker, 2
replicas) and flipped the existing `harness` service to
HARNESS_ROLE=control-plane. The new service was untracked in the SSOT,
so verify-railway-image-refs.ts failed the "Showcase: Build & Push"
workflow on every push to main (1 untracked Railway service), which
skipped the harness `build` job and blocked harness image rebuilds.
Add the worker to SERVICES as a staging-only, domain-less queue worker:
- ciBuilt:false — it runs the SAME `showcase-harness` image the existing
harness build slot produces; there is no separate worker build.
- gateIgnore:true / gateValidated:false — no prod instance and no public
domain, so it does not fit the symmetric dual-env shape the image-ref
gate validates. gateIgnore clears the "untracked Railway service"
failure (any SSOT entry counts as known) without tripping a false
"missing from prod" failure.
- repoNameOverride → showcase-harness so the image-ref shape resolves.
- probe disabled in both envs (no externally-reachable health endpoint).
Regenerate railway-envs.generated.json and update the SSOT-count and
gate-coverage test invariants (27→28 services; worker is the sole
intentional gateIgnore/gateValidated:false entry).
Close the recurring "mutation result not validated" defect class in the
starter-fleet provisioner and make the existing-services snapshot fail loud
instead of silently feeding erroneous create decisions.
- Add a uniform assertMutationOk guard and route EVERY mutation through it:
serviceCreate (assert .id), serviceInstanceUpdate (was DISCARDED — a false
Boolean! return meant sleep/healthcheck/image/creds were never applied while
the script reported success; now asserted and "configured" is logged only
after verification), serviceDomainCreate (assert .domain on the create path),
and serviceInstanceRedeploy (routed through the same guard for consistency).
- Absorb a serviceCreate "already exists" rejection: on a snapshot-miss the
create path now re-fetches the service id by name and falls through to UPDATE
instead of aborting the whole fleet. Predicate renamed ALREADY_EXISTS_RE and
reused by the domain-create path.
- fetchExistingServices fails loud on page-drain truncation (hasNextPage still
true at the defensive bound) rather than returning a partial byName map.
- fetchExistingServices coalesces null serviceInstances/.edges (transitional
service nodes) so an unguarded .find can't TypeError and abort the fetch;
interface fields marked optional/nullable.
- TRANSIENT_ERROR_RE made single-line ([^\n]*? not [\s\S]*?) so a newline-joined
multi-error blob can't bridge "Service" and "not found" across lines.
- withRetry wraps the schedule-exhaustion rethrow with context and { cause }.
Three functional fixes to the starter-fleet provisioner found in CR:
- fetchExistingServices now drains the Relay ServiceConnection via
pageInfo.hasNextPage/endCursor. A single un-paginated query truncated the
snapshot (~27 SSOT + 12 starter services span >1 page), making an existing
starter look absent → CREATE path → serviceCreate "already exists" →
non-transient abort of the whole run.
- TRANSIENT_ERROR_RE now matches Railway's INTERPOLATED "Service <id> not
found" (id embedded), not just the contiguous "Service not found", so the
post-create eventual-consistency retry actually fires.
- serviceInstanceRedeploy result check: documented the verified Boolean!
contract (sources: redeploy-env.ts, bin/railway RestoreCommand) and now
gates on truthiness (rejects false/null, accepts truthy defensively).
Hardening: ABORT on the live path when GITHUB_TOKEN is unset (private GHCR
images would image-pull-backoff while reporting success); warn-and-continue
only under --dry-run. Benign domain "already exists" no-op now logs the actual
matched Railway message for a forensic trail.
Harden the committed starter-fleet Railway provisioner against the
partial-failure / mistyped-flag / un-deployed-image failure modes
surfaced in CR:
- Domain idempotency: a serviceDomainCreate that Railway rejects with an
"already exists" error (start-of-run snapshot missed the domain due to
eventual consistency, or a prior run died mid-fleet) is now caught as a
benign no-op (marked "existing", logged) so a re-run converges instead
of aborting the entire remaining fleet. A genuine non-transient error
still aborts.
- Explicit redeploy: serviceCreate + serviceInstanceUpdate(source.image)
only PINS the image; it does not start a deployment, and Railway's image
auto-updates fire only on a NEW digest push. Added serviceInstanceRedeploy
after the instance update on BOTH the create and update paths so the
pinned image actually runs (and starter_smoke can find the service up).
Mirrors the documented update+redeploy pattern in bin/railway and the
explicit redeploy showcase_deploy.yml issues after each GHCR push.
- argv validation: parseArgs() now rejects any unrecognized argument
(e.g. a mistyped --dry-rn) with a usage hint before any provisioning,
instead of silently ignoring it and proceeding to REAL live provisioning.
- Fail-fast safety: validate the Railway token AND registry credentials
up front in main() (token resolution no longer process.exit()s deep in
the GraphQL boundary; main().catch owns the exit). Broadened the
withRetry transient predicate to the domain/instance eventual-consistency
class via an overridable per-call predicate. Dry-run now reports a new
service's domain as "would-create" for a faithful preview.
Adds showcase/scripts/provision-starter-fleet.ts — a committed, idempotent
provisioner for the SSOT-decoupled "starter container fleet". It creates (or
updates) one sleepable Railway service per starter template in the STAGING
environment, deriving the 12 targets from STARTER_TO_COLUMN (the smoke-matrix
SSOT) so the fleet can never drift from the build matrix.
Per service: serviceCreate scoped to the STAGING env (environmentId on
ServiceCreateInput, so NO production instance is ever materialized) with
source.image=ghcr.io/copilotkit/starter-<slug>:latest (RAW starter slug) and
GHCR registryCredentials; then serviceInstanceUpdate against staging with
sleepApplication:true + healthcheckPath="/" + region=us-west1; then
serviceDomainCreate for a generated staging domain. A bounded retry absorbs
Railway's eventual-consistency "ServiceInstance not found" right after create.
Healthcheck is "/" not "/api/health": the starters' single deployable image
EXPOSEs 3000 running the Next.js frontend, which serves "/" and
"/api/copilotkit" but has no "/api/health" route; the agent's "/health" is on
the internal 8123 port Railway does not expose. region read-back is null on the
serviceInstance for ALL existing showcase services too — that is normal Railway
behavior, so the fleet matches the existing services.
The fleet is decoupled from the 27-service railway-envs SSOT (starter-* services
are auto-discovered by the starter_smoke probe). #5254 already made
verify-railway-image-refs.ts tolerate starter-* names, so provisioning does not
trip the image-ref gate / skip the showcase build.
Red-green tested against an injected Railway GraphQL mock: target derivation
(raw vs remapped slug), GHCR credential resolution, sleepApplication:true,
staging-env scoping on BOTH create and update (never prod), idempotent
update-vs-create, domain de-duplication, and transient-error retry.
The starter container fleet (starter-<slug>) is decoupled from the
27-service railway-envs SSOT: each starter-* service is auto-discovered
at runtime by the starter_smoke probe (railway-services discovery,
namePrefix "starter-") and is never read from railway-envs.ts.
verify-railway-image-refs.ts is a hard needs: of the build job and runs a
bidirectional live-Railway drift check. Before this change, provisioning
a starter-* service made it untracked in the SSOT, so findUntrackedServices
failed and the showcase build was SKIPPED (the canary regression).
Scope both drift checks (findUntrackedServices and, defensively,
findMissingServices) to exclude services matched by a single, well-named
predicate isStarterFleetService(name) => name.startsWith("starter-"),
mirroring the harness discovery filter convention. Real showcase-*/infra
services are still drift-checked exactly as before.
This is the prerequisite for Phase-3 starter provisioning — provisioning
must happen AFTER this merges.
PocketBase had no CI build path: `ghcr.io/copilotkit/showcase-pocketbase`
was a stale April `:latest`, and there was no way to ship pb_migrations /
pb_hooks changes without an ad-hoc manual build. Add a `pocketbase` slot to
showcase_build.yml's build matrix, mirroring the harness/aimock entries:
- dispatch_name `showcase-pocketbase`, context `showcase/pocketbase`, its
own Dockerfile, health `/api/health`, railway_id from the SSOT.
- a paths-filter key gated to `showcase/pocketbase/**` so the slot only
rebuilds when PB's own files change (the image is self-contained — no
shared-module copy), not on every showcase push.
- the workflow_dispatch service choice so PB is human-targetable.
Flip the SSOT entry (railway-envs.ts) to `ciBuilt: true` with
`dispatchName: "showcase-pocketbase"` so it is built+pushed (`:sha` +
`:latest`) and joins the default staging-redeploy scope; the build's
redeploy step only touches the matrix-intersect-success set, so PB still
only redeploys when its own files change. Regenerate
railway-envs.generated.json and the showcase_promote.yml service dropdown,
and update the SSOT/redeploy tests that pinned PB as out-of-band
(CI_BUILT_SERVICES 25 -> 26; webhooks stays the only non-CI-built service).
## Summary
Hardens a set of pre-existing `showcase` promote/`bin/railway` bugs
surfaced during review of the GHCR bearer fix (#5239) and the
#team-showcase notification (#5240). Each is an independent, real
defect; all changes are covered by tests (Ruby `bin/spec` **127 runs / 0
failures**, TS `verify-deploy` drivers **82 passing**, `actionlint`
clean).
## Fixes
1. **`rollback` could roll back to the wrong deployment.**
`find_previous_deployment` selected the second-newest SUCCESS from an
*unsorted* GraphQL result, and assumed the head deploy was always
SUCCESS — so when the latest deploy FAILED/CRASHED (the exact case
rollback is for) it rolled back one good deploy too far. Now sorts by
`createdAt` desc, selects the newest SUCCESS strictly older than the
current head (`sorted.drop(1).find { SUCCESS }`), and **fails loud**
(rather than silently mis-rolling) when the `first: N` window is
saturated with no valid target — telling the operator to pass `--to`.
2. **`env-diff` was dishonest.** It advertised custom-domain comparison
it never performed, and exposed a `--ignore-env-scoped` flag that was
parsed but never read. Now actually diffs `custom_domains` (the snapshot
already carried them), removes the dead flag/helper, and nil-guards
**every** accessor in `diff_services` (`services`, `env_keys`,
`custom_domains`) consistent with the rest of the file.
3. **`verify-prod` could vacuously pass.** Its empty-`succeeded_csv`
branch `exit 0`'d unconditionally. Now fails loud if `promote` reported
success but produced no succeeded set (contract violation), while still
skipping cleanly when promote genuinely failed.
4. **`verify-prod` raced the prod rollout.** It failed instantly when
the just-promoted deploy was still `DEPLOYING` (observed live: promote
succeeded, verify-prod failed ~17s later mid-rollout). `verify-deploy`
now polls in-progress statuses
(`DEPLOYING`/`BUILDING`/`INITIALIZING`/`WAITING`/`QUEUED`/`NEEDS_APPROVAL`)
until terminal (~150s budget), still failing fast on
`FAILED`/`CRASHED`/`REMOVED`.
5. **`npx tsx` ran from the wrong cwd** in the verify jobs (deps
installed in `showcase/scripts`, invoked from repo root → could fetch
`tsx` from the network). Now runs with `working-directory:
showcase/scripts`, matching the resolve/promote jobs.
6. **Slack payloads rendered literal `\n`.** Both promote Slack posts
used `toJSON(format('...\n...'))`, where the literal `\n` survives as
backslash-n (verified live in #team-showcase). Now uses the
`fromJSON('"\n"')` idiom for real line breaks.
## Test plan
- [x] `showcase/bin/spec` — 127 runs, 0 failures (new: rollback
head-FAILED + saturated-window, env-diff custom-domain + nil-guard
cases)
- [x] `showcase/scripts` `tsc --noEmit` clean; `verify-deploy` driver
tests 82 passing (in-progress→SUCCESS, →timeout, fast-fail-on-terminal)
- [x] `actionlint` clean
- [ ] CI green
## Out of scope (separate follow-up)
Review surfaced further pre-existing items intentionally NOT fixed here
(no diff overlap): `run_staging_probe`'s `IO.popen` nests its options
hash inside the argv array (stderr redirect verified working; the
`child_env` hash isn't applied but the probe inherits the parent env in
CI); `image_shape`/`parse_image_ref`/`PinCommand` colon-splitting for
registry-port refs (latent — portless `ghcr.io` only); the deeper
`DEPLOYMENTS_QUERY first:10` truncation beyond the new fail-loud guard;
`succeeded_csv` integrity under a 20-min promote-job timeout; and a few
comment/test-coverage nits.
verify-prod commonly runs seconds after a promote pins a new image digest,
while Railway is still rolling the container out. checkDeploymentSuccess (in
verify-deploy.drivers.baseline.ts — the SSOT for the deployment-SUCCESS gate
shared by every driver) treated a transient in-progress status as a hard FAIL:
promote run 26966193624's predecessor pinned the docs digest, then verify-prod
fired ~17s later and FAILED with status="DEPLOYING" — a race, not a failure.
Now the deployment-status check polls when the latest deployment sits in any
non-terminal Railway status (QUEUED/BUILDING/INITIALIZING/DEPLOYING/WAITING/
NEEDS_APPROVAL), re-querying every 5s up to a 150s budget until it reaches a
terminal state, then asserts SUCCESS. Terminal-failure statuses (FAILED/
CRASHED/REMOVED/any non-SUCCESS-non-in-progress) still fail FAST with the
original error-string shape — no waiting. Infra/contract errors (network,
GraphQL errors[], missing edge) also fail fast.
The poll loop is fully seamed for tests (injectable sleep/now/budget); the
signature stays backward-compatible (trailing optional pollOpts). probeBaseline
forwards an optional deployPoll through, and both --env staging and --env prod
go through the identical code path, so staging behavior is unchanged.
Red-green tests: in-progress-then-SUCCESS passes (and polls), in-progress-until-
timeout fails with "still in progress", terminal FAILED fails fast with zero
sleeps.
declarative-gen-ui moved to the CopilotKitMiddleware auto-A2UI path across
the 3 langgraph integrations. The middleware's inner forced tool is
render_a2ui, so each integration's gen-ui-declarative.json gained 4
render_a2ui fixtures that share match keys with the pre-existing render_a2ui
entries in that integration's render-a2ui.json (the a2ui_fixed demo). 4
pills x 3 integrations = 12. Same-context cross-demo overlap, disambiguated
at runtime by the probe fixtureFile.
the per-service promote loop ran under `set -euo pipefail`, so the first failing
service aborted the whole `all` fleet promote; extracted to promote-fleet.sh
which attempts every service, accumulates succeeded/failed sets, exits non-zero
only after attempting all, and exports succeeded_csv. verify-prod now runs
`if: !cancelled()` and scopes --services to the succeeded set; the staging
precondition is advisory (promote runs even when it reports red — bin/railway
enforces staging-green per-service); notify success keys on PROMOTE && PROD.
Adds a shell-script-tests CI job (bats + shellcheck) and input-validation
hardening (fail-loud on empty/all-empty CSV, RAILWAY_BIN check, whitespace trim).
Add showcase/scripts/sync-promote-service-options.ts: generates the
promote workflow's service `choice` options from the SSOT
(railway-envs.ts), spliced between BEGIN/END markers in
showcase_promote.yml. Fail-loud throughout — every emitted token must
resolve to exactly one service under the resolve-step predicate
(name|dispatchName match AND probe.prod), tokens are YAML-safe, args are
strict (a typo'd flag cannot trigger a destructive write), and markers
are validated before any rewrite.
Wire it into a lefthook pre-commit hook (regenerate + restage; set -e so
a failed regen blocks the commit) and an advisory (never-failing) drift
check in showcase_validate.yml. Vitest coverage for ordering, exclusion,
collision/ambiguity guards, marker errors, exit codes, idempotency, and
the import-side-effect guard.
The gen-ui-headless-complete probe
(showcase/harness/src/probes/scripts/d5-gen-ui-headless-complete.ts)
references fixtureFile "gen-ui-headless-complete.json", but that file
existed in no D6 slug, so the probe's first leg 503'd under strict.
Add gen-ui-headless-complete.json for all 18 D6 slugs, modeled on the
green langgraph-python headless-complete.json pattern: the four gen-UI
pills (weather/stock/highlight/revenue) with narration (toolCallId)
fixtures FIRST and toolcall (userMessage+context) fixtures AFTER, no
turnIndex gate, so every one of the probe's four sequential turns in one
chat thread matches regardless of prior assistant/tool history. Interrupt
fixtures are intentionally omitted (interrupt-headless is a separate
cluster item).
These 8 fixtures per slug share match keys with the pre-existing
headless-complete.json for the same context (the demos share pills and
are disambiguated at runtime by probe path), which raises the
aimock-fixtures collision-detection exact-duplicate count by 46. Bump
KNOWN_DUPLICATE_CEILING 230 -> 276 to match, consistent with how prior
per-integration feature fixtures bumped the baseline; the substring-shadow
ceiling is unchanged (no new shadows).
Validated: all 732 aimock-fixtures schema/collision tests pass, and a
local aimock --strict run returns 200 for each of the four pills across
turns (langgraph-python + pydantic-ai spearheads, plus spring-ai).
Reject ASCII control chars, a colon (port suffix), and any char outside the
DNS-label charset; switch the Host brand from a string-literal __brand to a
non-exported unique symbol so a stray `as Host` cast from outside the module
is a type error. asHost stays the sole runtime constructor. Adds positive +
negative test coverage incl. interior-tab rejection.
Introduce a branded Host produced by asHost (rejects scheme/path/empty/whitespace/userinfo/query/
fragment); brand at the resolveProbeTargets ingress; make ProbeTarget fields readonly; add asHost
+ override-seam tests.
Closing hardening pass on the showcase deploy-gate's verify-matrix
resolver. The 7-agent review confirmed the gate is correct; this
commit fixes the residual rough edges.
- showcase_deploy.yml: correct the false §3 ok-non-empty comment.
The empty-intersection case can coexist with redeploy_red=false
(every redeploy succeeded, just none probe-eligible) — that's a
correctly-green run, not a red one.
- showcase_deploy.yml: tighten the summary.json shape guard to catch
PARTIAL drift (TOTAL>0 && WITH_STATUS<TOTAL). The previous all-or-
nothing TOTAL>0 && WITH_STATUS==0 check silently dropped drifted
rows on a mixed summary. Validated locally on mixed/normal/empty/
total-drift jq samples.
- resolve-verify-matrix.ts: add asSupportedEventName narrowing helper
+ use it in the CLI. Replaces the unchecked `as` cast — type system
and runtime now tell one story. Resolver's internal eventName
guard becomes defense-in-depth for direct (test) callers.
- resolve-verify-matrix.ts: make the workflow_run boundary total —
summaryPresent MUST be exactly "true"/"false". Any other value
(including "" from a step-id-rename wiring break) throws now
instead of silently emitting has_services=false.
- resolve-verify-matrix.ts: drop the try/catch around
fileURLToPath(import.meta.url) in `invokedDirectly`. The catch
used to swallow ESM-interop failures and silently no-op the CLI
(exit 0, no GITHUB_OUTPUT write → verify skipped = false-green).
- resolve-verify-matrix.ts: reword parseSsotServices JSDoc to
distinguish schema-drift from truncation (the two are different
failure modes, not one conflated story).
- showcase_build.yml: comment addendum on the redeploy-summary
upload — swapping the guard to `if: always()` would red the
legitimate services=='' path (no summary written), trading the
already-closed false-green for a false-red on every non-buildable
push.
- resolve-verify-matrix.cli.test.ts: switch to spawnSync so stderr
is captured on both zero and non-zero exit (execFileSync only
exposes stderr on throw). Hard-code two stable probe-eligible
names ("aimock", "harness") for the sorted-CSV test rather than
picking probe[0]/probe[1] off the live SSOT — the prior test was
tautological (already-sorted in, sorted out) and would silently
pass if the resolver did nothing.
- resolve-verify-matrix.cli.test.ts: add CLI coverage for the
dropped-token ::warning:: path (FIX 3 — the entire drift-detection
contract had zero CLI coverage), the unexpected-EVENT_NAME error
(FIX 5), and the workflow_run-summary_present total boundary
(FIX 7, both "" and "True" inputs).
- resolve-verify-matrix.test.ts: add unit coverage for the new
workflow_run summaryPresent boundary (empty + "True" + the
workflow_dispatch ignores-summaryPresent regression).
Red-green: 6 tests RED before code changes (FIX 3 warning, FIX 5
unknown EVENT_NAME, FIX 7 unit + CLI ×2 for "" and "True"); 79
tests GREEN after.
Validation: 4 vitest files / 79 tests passing; 87/87 ruby specs
passing; actionlint findings unchanged vs integration baseline
(8 → 8, identical diff); yaml.safe_load OK on both workflows.
A 7-agent review of the verify-matrix resolver and its surrounding workflow plumbing found three
boundary surfaces that could silently produce a GREEN deploy on a broken release, plus an
untested CLI contract that CI compares against the literal strings 'true' / 'false'.
FIX 1 — Validate the SSOT shape in loadSsotServices(). The prior `JSON.parse(...) as
{services: SsotService[]}` was an unchecked cast: a truncated/drifted SSOT (emitter crashed
mid-write, or schema renamed) parses fine but silently shrinks/empties the probe-eligible set
→ some redeployed services go unverified, or verify is skipped on a real redeploy. Extract a
pure exported parseSsotServices(raw, path) that requires the shape we depend on (non-empty
services array; each entry has a non-empty string name, an optional string|null dispatchName,
and a probe object with a boolean staging). Throw `::error::SSOT <path> malformed: <detail>`
on any violation. Also re-check existsSync(SSOT_JSON) after the regenerate-if-missing
execFileSync — a regen that exits 0 without writing must not proceed to a useless JSON.parse
crash. Drop the defensive `probe?.staging` once shape is guaranteed.
FIX 2 — Validate summary.json shape in the redeploy-gate bash. The bullseye false-green
surface: if redeploy-env.ts's schema ever drifts (e.g. `status` → `state`, `ok` → `success`),
every `jq select(.status==...)` yields empty → redeploy_red=false AND ok_services="" →
resolver skips verify → GREEN CI on a real unverified redeploy. Add a TOTAL vs WITH_STATUS
shape guard right after loading the summary: if TOTAL > 0 && WITH_STATUS == 0, emit
::error::summary.json has $TOTAL entries but none with status ok|error (schema drift?) and
exit 1. The legitimate empty-array path (TOTAL=0) is preserved.
FIX 3 — Fail loud on unknown eventName in resolveVerifyMatrix. The prior code fell through to
the workflow_run intersection branch for ANY unrecognized eventName (typo, unexpected
trigger), silently emitting has_services=false → indistinguishable from a legit "summary
absent" skip. Add an explicit guard so only workflow_run / workflow_dispatch are accepted;
anything else throws ::error::resolve-verify-matrix: unexpected eventName '<value>'. Tighten
the eventName parameter type to the literal union.
FIX 4 — Trim ok tokens + warn on dropped tokens in okCsvToCanonicalNames. Split, then
.map(t => t.trim()).filter(Boolean) so "a, b" (spaces) matches. Collect tokens that match NO
SSOT service (by name or dispatchName) and have the CLI wrapper emit ::warning::ok_services
tokens dropped (no SSOT match): <list> on stderr when non-empty — surfaces SSOT/build drift.
The pure function stays IO-free; logging lives in the wrapper.
FIX 5 — CLI wrapper integration test. New resolve-verify-matrix.cli.test.ts spawns
`npx tsx showcase/scripts/resolve-verify-matrix.ts` with a temp $GITHUB_OUTPUT file across
four scenarios and asserts the temp file contents EXACTLY (the workflow YAML compares
has_services against the literal strings 'true'/'false', so the byte-for-byte format is part
of the contract). Uses the real railway-envs.generated.json so the loader exercise is real.
FIX 6 — Cleanup. Remove the dead `env: DISPATCH_SERVICE: ...` block on the redeploy-gate
step (the next step redeclares it — leftover from the extraction). Soften the §3
decision-table all-errors bullet to match resolve-verify-matrix.ts's careful wording, and
append that when the success-set is empty (or the intersection collapses to empty), verify
is skipped and the gate reds independently. Append to showcase_build.yml's "Upload redeploy
summary" path-(A) comment that `if-no-files-found: error` still reds path (A) even if a
future change adds `if: always()`.
Tests: red→green for FIX 1/3/4/5 verified locally. Resolve-verify-matrix vitest count:
12 → 28. Full requested suite (resolve-verify-matrix + cli + aggregate-build-results +
lint-rule-no-public-env): 72 passed. showcase/bin ruby specs: 87 runs / 0 failures / 0
errors / 0 skips. actionlint baseline preserved (8 findings, identical to integration tip).
Extract the inline bash+jq decision logic from showcase_deploy.yml's
resolve-matrix job into showcase/scripts/resolve-verify-matrix.ts, a
pure function with a vitest suite. The bash had produced two confirmed
bugs across prior CR rounds, so making it testable is the lasting fix.
Issue A (the bug this PR fixes): when summary_present=true but
ok_services is empty (every service errored on redeploy), the old bash
skipped the intersection and fell through to the full probe-eligible
fleet, gratuitously probing every service against stale :latest. The
resolver now returns has_services=false in that case — enforce-redeploy
-gate independently reds the workflow on redeploy_red=true, so this
case is already loud; there is nothing left to verify.
Parity preserved for unchanged cases:
- workflow_dispatch + 'all'/empty → full probe-eligible set
- workflow_dispatch + specific svc → that one (unknown → error exit)
- workflow_run + summary_present=false → has_services=false
- workflow_run + present + ok non-empty → intersection with probe-
eligible (SSOT key OR dispatchName aliases both resolve)
Also clarified the Upload-redeploy-summary comment in showcase_build.yml
to document both red paths (hard crash → redeploy step exits non-zero;
exit-0-but-no-file → if-no-files-found:error reds the step) so no
false-green path is possible.
Tests: 12-case vitest suite covers each decision-table row plus the
Issue A fix (written red-first; failed against a naive full-fleet
fallback, passed once the early return was added). CLI parity verified
against the real generated SSOT for the three representative env-var
combinations (workflow_run + present + ok=[a,c]; workflow_run + present
+ ok empty; workflow_dispatch + 'all').
D1 — showcase_deploy.yml false-red fix
======================================
The build workflow legitimately uploads no `redeploy-summary` artifact when it ran
(push touched `showcase/**` so `paths:` matched) but `detect-changes` found no
buildable service, so `redeploy-staging` was skipped. The build still concludes
`success`, so `showcase_deploy.yml` fires on `workflow_run` and `resolve-matrix`
runs. `actions/download-artifact@v4` with `name:` HARD-FAILS on a missing
artifact, so the unguarded download was failing the job, and a downstream guard
that trips `enforce-redeploy-gate` on `resolve-matrix.result == 'failure'` was
flipping the workflow RED — a false-red on a routine showcase-docs/script change.
Add an artifact-existence pre-check using `actions/github-script` (pinned by SHA,
matching the existing repo convention) that lists the artifacts for
`workflow_run.id` via `actions: read` (already granted to `resolve-matrix`) and
sets `summary_present=true|false`. Gate the existing download step on
`summary_present == 'true'`. Keep NO `continue-on-error`, so the C1 property
holds: when the artifact exists but the download genuinely fails, the job still
fails loud and `enforce-redeploy-gate` correctly reds the workflow. When the
artifact is legitimately absent, the bash gate's existing `[ ! -f "$SUMMARY" ]`
branch no-ops (`redeploy_red=false`, `ok_services=""`) — nothing was
redeployed, so there is nothing to gate.
Updated the step comment block to enumerate the three distinct cases now
handled: workflow_dispatch (no download); workflow_run + artifact absent
(graceful skip); workflow_run + artifact present (download with fail-loud).
L1-L5 — env lint rule hardening
===============================
- L1: route the destructuring (VariableDeclarator/ObjectPattern) branch through
the shared `staticKeyName()` helper so the computed-string-key form
`const { ["NEXT_PUBLIC_X"]: y } = process.env` and the no-expression
template-literal form `const { [\`NEXT_PUBLIC_X\`]: y } = process.env` are
caught with the same parity as the bracket-member read.
- L2: unwrap a wrapping `ChainExpression` at the top of `isProcessEnv()` so
`process.env?.X` is matched robustly across parser flavors; corrected the
helper's doc comment to describe the actual semantics.
- L3: export `BANNED_KEYS` from the rule module and have the table-driven test
dynamically import the rule's own Set instead of hand-mirroring it — the
test set now cannot drift from the rule.
- L4: added override-scoping fixtures for `showcase/shell/src/**` and
`showcase/shell-dojo/src/**`; the `.oxlintrc.json` override list already
includes these, but the test now exercises them so an accidental drop is
caught.
- L5: expanded the file-header "Out of scope" doc list to include bulk-iteration
reads (`Object.keys/values/entries(process.env)`, for-in, spread
`{...process.env}`), rest-pattern destructuring, compound-assignment LHS, and
update operators. Documentation-only — the deliberate non-coverage is now
auditable.
Validation
==========
- RED→GREEN confirmed for L1 (two new destructuring computed-key tests) and L3
(dynamic `await import(...)` of BANNED_KEYS failed pre-fix with
"Rule module did not export a non-empty BANNED_KEYS Set", green after export).
- vitest: 38 passed (was 34 baseline + 4 new); aggregate-build-results 6 passed.
- Ruby promote suite: 87 runs, 251 assertions, 0 failures (unchanged).
- python3 yaml.safe_load: showcase_deploy.yml + showcase_build.yml +
showcase_promote.yml all parse OK.
- actionlint: zero NEW findings on the changed file. The pre-existing
showcase_build.yml SC2086/SC2129/runner-label findings are identical on the
integration baseline (unchanged by this commit).
Seven-agent CR surfaced correctness defects in the build/deploy/promote
pipeline and in the no-public-env-shell-read oxlint rule. This commit
closes the false-green paths and broadens lint coverage.
Workflow fixes:
- showcase_deploy.yml: drop `continue-on-error: true` on the redeploy-summary
artifact download. The dispatch path is already guarded by the `if:
workflow_run` clause, so the bash "no summary" branch handles legitimate
manual dispatches. A genuine workflow_run download failure must now fail
loud instead of silently widening verify to the full service set against
stale `:latest`.
- showcase_build.yml: redeploy-staging now intersects the build matrix with
the aggregator success set (`needs.aggregate-build-results.outputs.results`,
status == "success") before producing the redeploy CSV. Failed/skipped
slots no longer get redeployed (which would just re-pull stale `:latest`
and look healthy).
- showcase_build.yml: `notify-all-builds-failed` now additionally requires
`needs.build.result == 'failure'` so it doesn't Slack-spam when the build
job was SKIPPED (verify-image-refs upstream failure).
- showcase_build.yml: `notify` now lists [build, aggregate-build-results,
redeploy-staging] in `needs:` so aggregator/redeploy failures still emit
a Slack signal. `if: failure()` still skips when none of the needs failed.
- showcase_build.yml: `set -euo pipefail` on the Prepare build args step
so a transient $GITHUB_OUTPUT write failure can't ship images without
COMMIT_SHA/BRANCH baked in.
- showcase_deploy.yml: `enforce-redeploy-gate` now also trips on a
resolve-matrix failure (`needs.resolve-matrix.result == 'failure'`) so
an upstream crash that leaves `redeploy_red` empty can't bypass the gate.
- Doc-comment accuracy: drop stale `(PR #5093)` reference; correct the
env-IDs source-of-truth comment; document the optional `skip_build` field
in ALL_SERVICES; clarify that health_path is informational and verify
uses per-service drivers; add the missing `resolve-targets` step 0 to the
promote workflow's "Order:" header.
Aggregator fix (RED-GREEN):
- aggregate-build-results.ts: throw on zero slot dirs. The job is gated
upstream on has_changes == 'true', so zero slot dirs is a broken artifact
download, not a legitimate empty build set. Silently emitting
any_success=false + results=[] is indistinguishable from "all builds
failed" and lets the deploy workflow fall back to probing the full
service set against stale `:latest`. Refuse the ambiguity.
- aggregate-build-results.test.ts: existing empty-INPUT_DIR test was
updated to assert the throw (was: return []).
Oxlint rule (RED-GREEN):
- no-public-env-shell-read.mjs: handle destructuring reads
(const { NEXT_PUBLIC_X } = process.env and aliased form), template-literal
computed keys (process.env[\`NEXT_PUBLIC_X\`]), and explicitly skip
assignment-LHS / `delete` targets (writes are not reads). Optional
chaining already worked through the existing MemberExpression path.
Aliasing (`const e = process.env; e.X`) is intentionally documented as
out of scope (needs scope tracking). Description sharpened to say the
rule guards a specific banned-key set, not all NEXT_PUBLIC_* reads.
- .oxlintrc.json: tighten the off-override glob from
`showcase/**/*runtime-config*` to
`showcase/**/lib/runtime-config*.{ts,tsx}` so it only silences the
intended implementation files, not arbitrary paths containing that
substring.
- lint-rule-no-public-env.test.ts: rewritten as table-driven coverage of
every BANNED_KEYS entry (dotted + bracket-string forms), every ALLOWED
key (asserting non-firing), all new variants from the rule expansion,
the assignment/delete non-fire cases, and override scoping
(runtime-config exempt; packages exempt; shell-tree non-runtime-config
flagged).
Validation:
- actionlint on all three workflows: 8 pre-existing findings (depot label,
pre-existing SC2086 infos in untouched steps); my edits add zero.
- python3 yaml.safe_load: all three workflows OK.
- vitest aggregate-build-results.test.ts: 6/6 pass (incl. new throw test).
- vitest lint-rule-no-public-env.test.ts: 34/34 pass.
- vitest full showcase/scripts suite: 1654/1654 pass across 46 files.
- ruby showcase/bin/spec/all_tests.rb: 87 runs, 0 failures.
- Intersection jq proof (matrix a,b,c × success a,c) → "a,c"; all-failed
→ ""; skipped status excluded.
Distinguish ENOENT (treat as drift) from other read errors in
emit-railway-envs-json.ts --check; non-ENOENT errors now exit 2 with the
real error on stderr instead of being silently coerced into a misleading
'stale' message or an overwrite on a false drift signal.
Add a --out=<path> override so tests can write to a temp directory and
never mutate the tracked railway-envs.generated.json artifact. Rewrite
the emit-railway-envs-json test to use mkdtempSync + --out, switch the
staleness assertion to spawnSync so it asserts on exit code 1 plus the
stale-diagnostic substring, and add coverage for the new EISDIR
fail-loud path. After this change git status is clean post-test.
New TS probe driven off railway-envs SSOT. Accepts --env staging|prod and
optional --services CSV; iterates SERVICES where probe[env]===true and
dispatches to per-driver feature-level verifiers. Refuses to start when a
probe-required service is missing a domain for the requested env (no
silent skip). HTTP 200 is necessary but not sufficient.
Adds verify-deploy.drivers.ts dispatch with exhaustive never check on
ProbeDriver, plus one stub module per ProbeDriver literal (shell, docs,
dashboard, dojo, harness, eval, aimock, pocketbase, webhooks, agent).
Stubs fail loud with an explicit "not yet implemented" error so any
accidental real-network invocation surfaces; per-driver feature-level
impls land as subsequent micro-tasks. Refs spec section 3 / section 3.5.
[BLITZ:A5]
Adds optional REDEPLOY_SUMMARY_JSON path; when set, redeploy-env writes
a structured per-service record array {service,status,error?}. PR #5093's
exit-code contract is preserved (staging=0, prod=1-on-failure).
Consumed by showcase_deploy.yml to fail the workflow on staging per-service
errors without changing the script's exit semantics. Refs spec §3.
Plan-B / Option-B migration moved every shell URL/analytics key off the
build-time NEXT_PUBLIC_* env channel and onto runtime config served via
__SHOWCASE_CONFIG__ + getRuntimeConfig(). To prevent a silent regression
where a future change reintroduces a direct process.env.NEXT_PUBLIC_*
read in shell code (which would re-freeze the value at build time and
break no-rebuild env switching), add a focused lint rule.
The rule (copilotkit/no-public-env-shell-read) is implemented as a
custom oxlint JS plugin rule in the existing copilotkit plugin and
enabled under shell-scoped overrides in .oxlintrc.json:
- Errors on process.env.NEXT_PUBLIC_<URL/ANALYTICS> reads in:
showcase/shell-dashboard/src/**, showcase/shell-docs/src/**,
showcase/shell/src/**, showcase/shell-dojo/src/**
- Banned keys: POCKETBASE_URL, SHELL_URL, BASE_URL, OPS_BASE_URL,
INTELLIGENCE_SIGNUP_URL, POSTHOG_KEY, POSTHOG_HOST, SCARF_PIXEL_ID,
GOOGLE_ANALYTICS_TRACKING_ID, REB2B_KEY, REO_KEY
- Intentionally allowed (NOT banned): NEXT_PUBLIC_COMMIT_SHA and
NEXT_PUBLIC_BRANCH (build-stamped artifact identifiers per B10/B11)
and NEXT_PUBLIC_LOCAL_BACKENDS (computed from shared/local-ports.json
at build, local-dev only).
- Excluded files (rule disabled via a follow-up override): MDX content
under shell-docs/src/content/**, runtime-config implementation files,
and *.test.{ts,tsx} / *.spec.{ts,tsx}. oxlint does not support
excludedFiles inside an override block, so the exclusion is expressed
as a later override that sets the rule to off.
Plan-B originally targeted oxlint's eslint/no-restricted-syntax with an
AST-selector regex. oxlint 1.x does not implement that rule (only
no-restricted-globals / no-restricted-imports), so the equivalent guard
is realized as a small custom rule in the existing copilotkit JS plugin
(meta.name=copilotkit), reusing the same plugin loader the repo already
has for require-cpk-prefix and no-single-arg-zod-record.
Verification (red-green): the rule fires on a fixture containing
process.env.NEXT_PUBLIC_POCKETBASE_URL and does NOT fire on a fixture
containing process.env.NEXT_PUBLIC_COMMIT_SHA. Test pins the config via
-c so it works inside git worktrees nested under .claude/worktrees/
where oxlint's automatic upward config search can miss the worktree's
own .oxlintrc.json.
All four shells lint clean: 0 errors of the new rule across
shell-dashboard (114 files), shell-docs (137), shell (29), shell-dojo (6).
Cover four malformed-ref shapes the gate must reject:
* `:sha256-<hex>` (missing the @ separator — looks like a digest
pin but is actually a tag)
* `@sha256:<too-short-hex>` (truncated digest hex)
* The 2026-04-21 `...atest` corruption shape from the script
docstring (the original reason this gate exists)
* Non-ghcr.io registries on both envs
These match the canonical PROD_SHAPE / STAGING_SHAPE regexes in
verify-railway-image-refs.ts; the tests are the regression guard
that ensures a future "relax the regex" change cannot ship without
explicitly turning these red first.
Lock in shape behaviour for dashboard, docs, dojo, shell, and
harness with explicit per-service red-green cases:
* prod with :latest -> fail (must be @sha256)
* prod with @sha256 on the correct repo -> pass
* staging with :latest on the correct repo -> pass
* staging with @sha256 -> fail (must float on :latest)
* wrong GHCR repo name on either env -> fail
The validateImage body is unchanged — it was always shape-pure and
shape-correct. Before WS-C these five services were never exercised
through the gate at all (gateValidated:false). These tests are the
regression guard that ensures a future edit doesn't accidentally
re-introduce the Phase-2 carve-out without anyone noticing.
Flip dashboard, docs, dojo, shell, and harness from
gateValidated: false to gateValidated: true and simultaneously add
the corresponding repoNameOverride for both envs:
dashboard -> showcase-shell-dashboard
docs -> showcase-shell-docs
dojo -> showcase-shell-dojo
shell -> showcase-shell
harness -> showcase-harness
These two halves MUST land in the same commit. Flipping
gateValidated without the override would make the gate look up
ghcr.io/copilotkit/<railway-name> (e.g.
ghcr.io/copilotkit/dashboard:latest) which does not exist — the
gate would fail on first run. Adding the override without flipping
gateValidated is dead code: main() short-circuits unvalidated
services before consulting the override. Only the union is correct.
Also remove the Phase-2 deferral comments and refresh the
ServiceEntry.gateValidated JSDoc — there are no Phase-2 holdouts
left. All 27 services are now gate-validated; gateIgnore remains
the sole escape hatch and is unused by every current entry.
The Railway -> SSOT direction at verify-railway-image-refs.ts was
warn-and-continue: an out-of-band Railway service with a malformed
ref would not turn CI red as long as nobody read the warning line.
Replace that branch with a hard failure path that lists the
untracked service under its own failure class, with a clear remedy
in the error message (add to SSOT, or set gateIgnore on an existing
entry).
Refactor main() to call two pure helpers (findUntrackedServices,
summarizeFailures) so the policy is unit-testable without going
through Railway GraphQL. The SSOT -> Railway direction
(findMissingServices) is unchanged: it is already correct and is a
separate coverage class.
Adds red-green tests for the summarizeFailures shape, covering the
three failure classes (shape violations, SSOT->Railway drift,
Railway->SSOT drift) and the success path.
Jordan Ritter