The showcase_build verify-image-refs gate (SSOT = showcase/scripts/railway-envs.ts)
was failing with "1 untracked Railway services" because the interim
harness-legacy staging service (the legacy all-probe harness kept live during
the pool-fleet migration) exists on Railway but had no SSOT entry. That
Railway->SSOT drift check skips the build, so nothing deploys.
Adds a harness-legacy SERVICES entry mirroring the showcase-harness-worker
precedent (PR #5280): ciBuilt:false (not built by showcase_build, runs a pinned
out-of-band digest) and gateIgnore:true (deliberately-untracked for the image-ref
gate). findUntrackedServices treats any SSOT entry as known, so this clears the
untracked failure; gateValidated:false keeps findMissingServices from flagging
it. Real serviceInstance IDs for both envs recorded from Railway GraphQL.
Regenerates railway-envs.generated.json and updates the service-count /
gate-ignored carve-out assertions (28->29 services).
Verified: live verify-railway-image-refs.ts now exits 0 ("54 env-scoped
instances verified, 2 skipped"); without the entry it exits 1 with the
harness-legacy untracked failure. Full scripts test suite green (1771 passed).
The pool-fleet cutover manually created the staging-only
`showcase-harness-worker` Railway service (HARNESS_ROLE=worker, 2
replicas) and flipped the existing `harness` service to
HARNESS_ROLE=control-plane. The new service was untracked in the SSOT,
so verify-railway-image-refs.ts failed the "Showcase: Build & Push"
workflow on every push to main (1 untracked Railway service), which
skipped the harness `build` job and blocked harness image rebuilds.
Add the worker to SERVICES as a staging-only, domain-less queue worker:
- ciBuilt:false — it runs the SAME `showcase-harness` image the existing
harness build slot produces; there is no separate worker build.
- gateIgnore:true / gateValidated:false — no prod instance and no public
domain, so it does not fit the symmetric dual-env shape the image-ref
gate validates. gateIgnore clears the "untracked Railway service"
failure (any SSOT entry counts as known) without tripping a false
"missing from prod" failure.
- repoNameOverride → showcase-harness so the image-ref shape resolves.
- probe disabled in both envs (no externally-reachable health endpoint).
Regenerate railway-envs.generated.json and update the SSOT-count and
gate-coverage test invariants (27→28 services; worker is the sole
intentional gateIgnore/gateValidated:false entry).
Close the recurring "mutation result not validated" defect class in the
starter-fleet provisioner and make the existing-services snapshot fail loud
instead of silently feeding erroneous create decisions.
- Add a uniform assertMutationOk guard and route EVERY mutation through it:
serviceCreate (assert .id), serviceInstanceUpdate (was DISCARDED — a false
Boolean! return meant sleep/healthcheck/image/creds were never applied while
the script reported success; now asserted and "configured" is logged only
after verification), serviceDomainCreate (assert .domain on the create path),
and serviceInstanceRedeploy (routed through the same guard for consistency).
- Absorb a serviceCreate "already exists" rejection: on a snapshot-miss the
create path now re-fetches the service id by name and falls through to UPDATE
instead of aborting the whole fleet. Predicate renamed ALREADY_EXISTS_RE and
reused by the domain-create path.
- fetchExistingServices fails loud on page-drain truncation (hasNextPage still
true at the defensive bound) rather than returning a partial byName map.
- fetchExistingServices coalesces null serviceInstances/.edges (transitional
service nodes) so an unguarded .find can't TypeError and abort the fetch;
interface fields marked optional/nullable.
- TRANSIENT_ERROR_RE made single-line ([^\n]*? not [\s\S]*?) so a newline-joined
multi-error blob can't bridge "Service" and "not found" across lines.
- withRetry wraps the schedule-exhaustion rethrow with context and { cause }.
Three functional fixes to the starter-fleet provisioner found in CR:
- fetchExistingServices now drains the Relay ServiceConnection via
pageInfo.hasNextPage/endCursor. A single un-paginated query truncated the
snapshot (~27 SSOT + 12 starter services span >1 page), making an existing
starter look absent → CREATE path → serviceCreate "already exists" →
non-transient abort of the whole run.
- TRANSIENT_ERROR_RE now matches Railway's INTERPOLATED "Service <id> not
found" (id embedded), not just the contiguous "Service not found", so the
post-create eventual-consistency retry actually fires.
- serviceInstanceRedeploy result check: documented the verified Boolean!
contract (sources: redeploy-env.ts, bin/railway RestoreCommand) and now
gates on truthiness (rejects false/null, accepts truthy defensively).
Hardening: ABORT on the live path when GITHUB_TOKEN is unset (private GHCR
images would image-pull-backoff while reporting success); warn-and-continue
only under --dry-run. Benign domain "already exists" no-op now logs the actual
matched Railway message for a forensic trail.
Harden the committed starter-fleet Railway provisioner against the
partial-failure / mistyped-flag / un-deployed-image failure modes
surfaced in CR:
- Domain idempotency: a serviceDomainCreate that Railway rejects with an
"already exists" error (start-of-run snapshot missed the domain due to
eventual consistency, or a prior run died mid-fleet) is now caught as a
benign no-op (marked "existing", logged) so a re-run converges instead
of aborting the entire remaining fleet. A genuine non-transient error
still aborts.
- Explicit redeploy: serviceCreate + serviceInstanceUpdate(source.image)
only PINS the image; it does not start a deployment, and Railway's image
auto-updates fire only on a NEW digest push. Added serviceInstanceRedeploy
after the instance update on BOTH the create and update paths so the
pinned image actually runs (and starter_smoke can find the service up).
Mirrors the documented update+redeploy pattern in bin/railway and the
explicit redeploy showcase_deploy.yml issues after each GHCR push.
- argv validation: parseArgs() now rejects any unrecognized argument
(e.g. a mistyped --dry-rn) with a usage hint before any provisioning,
instead of silently ignoring it and proceeding to REAL live provisioning.
- Fail-fast safety: validate the Railway token AND registry credentials
up front in main() (token resolution no longer process.exit()s deep in
the GraphQL boundary; main().catch owns the exit). Broadened the
withRetry transient predicate to the domain/instance eventual-consistency
class via an overridable per-call predicate. Dry-run now reports a new
service's domain as "would-create" for a faithful preview.
Adds showcase/scripts/provision-starter-fleet.ts — a committed, idempotent
provisioner for the SSOT-decoupled "starter container fleet". It creates (or
updates) one sleepable Railway service per starter template in the STAGING
environment, deriving the 12 targets from STARTER_TO_COLUMN (the smoke-matrix
SSOT) so the fleet can never drift from the build matrix.
Per service: serviceCreate scoped to the STAGING env (environmentId on
ServiceCreateInput, so NO production instance is ever materialized) with
source.image=ghcr.io/copilotkit/starter-<slug>:latest (RAW starter slug) and
GHCR registryCredentials; then serviceInstanceUpdate against staging with
sleepApplication:true + healthcheckPath="/" + region=us-west1; then
serviceDomainCreate for a generated staging domain. A bounded retry absorbs
Railway's eventual-consistency "ServiceInstance not found" right after create.
Healthcheck is "/" not "/api/health": the starters' single deployable image
EXPOSEs 3000 running the Next.js frontend, which serves "/" and
"/api/copilotkit" but has no "/api/health" route; the agent's "/health" is on
the internal 8123 port Railway does not expose. region read-back is null on the
serviceInstance for ALL existing showcase services too — that is normal Railway
behavior, so the fleet matches the existing services.
The fleet is decoupled from the 27-service railway-envs SSOT (starter-* services
are auto-discovered by the starter_smoke probe). #5254 already made
verify-railway-image-refs.ts tolerate starter-* names, so provisioning does not
trip the image-ref gate / skip the showcase build.
Red-green tested against an injected Railway GraphQL mock: target derivation
(raw vs remapped slug), GHCR credential resolution, sleepApplication:true,
staging-env scoping on BOTH create and update (never prod), idempotent
update-vs-create, domain de-duplication, and transient-error retry.
The starter container fleet (starter-<slug>) is decoupled from the
27-service railway-envs SSOT: each starter-* service is auto-discovered
at runtime by the starter_smoke probe (railway-services discovery,
namePrefix "starter-") and is never read from railway-envs.ts.
verify-railway-image-refs.ts is a hard needs: of the build job and runs a
bidirectional live-Railway drift check. Before this change, provisioning
a starter-* service made it untracked in the SSOT, so findUntrackedServices
failed and the showcase build was SKIPPED (the canary regression).
Scope both drift checks (findUntrackedServices and, defensively,
findMissingServices) to exclude services matched by a single, well-named
predicate isStarterFleetService(name) => name.startsWith("starter-"),
mirroring the harness discovery filter convention. Real showcase-*/infra
services are still drift-checked exactly as before.
This is the prerequisite for Phase-3 starter provisioning — provisioning
must happen AFTER this merges.
PocketBase had no CI build path: `ghcr.io/copilotkit/showcase-pocketbase`
was a stale April `:latest`, and there was no way to ship pb_migrations /
pb_hooks changes without an ad-hoc manual build. Add a `pocketbase` slot to
showcase_build.yml's build matrix, mirroring the harness/aimock entries:
- dispatch_name `showcase-pocketbase`, context `showcase/pocketbase`, its
own Dockerfile, health `/api/health`, railway_id from the SSOT.
- a paths-filter key gated to `showcase/pocketbase/**` so the slot only
rebuilds when PB's own files change (the image is self-contained — no
shared-module copy), not on every showcase push.
- the workflow_dispatch service choice so PB is human-targetable.
Flip the SSOT entry (railway-envs.ts) to `ciBuilt: true` with
`dispatchName: "showcase-pocketbase"` so it is built+pushed (`:sha` +
`:latest`) and joins the default staging-redeploy scope; the build's
redeploy step only touches the matrix-intersect-success set, so PB still
only redeploys when its own files change. Regenerate
railway-envs.generated.json and the showcase_promote.yml service dropdown,
and update the SSOT/redeploy tests that pinned PB as out-of-band
(CI_BUILT_SERVICES 25 -> 26; webhooks stays the only non-CI-built service).
## Summary
Hardens a set of pre-existing `showcase` promote/`bin/railway` bugs
surfaced during review of the GHCR bearer fix (#5239) and the
#team-showcase notification (#5240). Each is an independent, real
defect; all changes are covered by tests (Ruby `bin/spec` **127 runs / 0
failures**, TS `verify-deploy` drivers **82 passing**, `actionlint`
clean).
## Fixes
1. **`rollback` could roll back to the wrong deployment.**
`find_previous_deployment` selected the second-newest SUCCESS from an
*unsorted* GraphQL result, and assumed the head deploy was always
SUCCESS — so when the latest deploy FAILED/CRASHED (the exact case
rollback is for) it rolled back one good deploy too far. Now sorts by
`createdAt` desc, selects the newest SUCCESS strictly older than the
current head (`sorted.drop(1).find { SUCCESS }`), and **fails loud**
(rather than silently mis-rolling) when the `first: N` window is
saturated with no valid target — telling the operator to pass `--to`.
2. **`env-diff` was dishonest.** It advertised custom-domain comparison
it never performed, and exposed a `--ignore-env-scoped` flag that was
parsed but never read. Now actually diffs `custom_domains` (the snapshot
already carried them), removes the dead flag/helper, and nil-guards
**every** accessor in `diff_services` (`services`, `env_keys`,
`custom_domains`) consistent with the rest of the file.
3. **`verify-prod` could vacuously pass.** Its empty-`succeeded_csv`
branch `exit 0`'d unconditionally. Now fails loud if `promote` reported
success but produced no succeeded set (contract violation), while still
skipping cleanly when promote genuinely failed.
4. **`verify-prod` raced the prod rollout.** It failed instantly when
the just-promoted deploy was still `DEPLOYING` (observed live: promote
succeeded, verify-prod failed ~17s later mid-rollout). `verify-deploy`
now polls in-progress statuses
(`DEPLOYING`/`BUILDING`/`INITIALIZING`/`WAITING`/`QUEUED`/`NEEDS_APPROVAL`)
until terminal (~150s budget), still failing fast on
`FAILED`/`CRASHED`/`REMOVED`.
5. **`npx tsx` ran from the wrong cwd** in the verify jobs (deps
installed in `showcase/scripts`, invoked from repo root → could fetch
`tsx` from the network). Now runs with `working-directory:
showcase/scripts`, matching the resolve/promote jobs.
6. **Slack payloads rendered literal `\n`.** Both promote Slack posts
used `toJSON(format('...\n...'))`, where the literal `\n` survives as
backslash-n (verified live in #team-showcase). Now uses the
`fromJSON('"\n"')` idiom for real line breaks.
## Test plan
- [x] `showcase/bin/spec` — 127 runs, 0 failures (new: rollback
head-FAILED + saturated-window, env-diff custom-domain + nil-guard
cases)
- [x] `showcase/scripts` `tsc --noEmit` clean; `verify-deploy` driver
tests 82 passing (in-progress→SUCCESS, →timeout, fast-fail-on-terminal)
- [x] `actionlint` clean
- [ ] CI green
## Out of scope (separate follow-up)
Review surfaced further pre-existing items intentionally NOT fixed here
(no diff overlap): `run_staging_probe`'s `IO.popen` nests its options
hash inside the argv array (stderr redirect verified working; the
`child_env` hash isn't applied but the probe inherits the parent env in
CI); `image_shape`/`parse_image_ref`/`PinCommand` colon-splitting for
registry-port refs (latent — portless `ghcr.io` only); the deeper
`DEPLOYMENTS_QUERY first:10` truncation beyond the new fail-loud guard;
`succeeded_csv` integrity under a 20-min promote-job timeout; and a few
comment/test-coverage nits.
verify-prod commonly runs seconds after a promote pins a new image digest,
while Railway is still rolling the container out. checkDeploymentSuccess (in
verify-deploy.drivers.baseline.ts — the SSOT for the deployment-SUCCESS gate
shared by every driver) treated a transient in-progress status as a hard FAIL:
promote run 26966193624's predecessor pinned the docs digest, then verify-prod
fired ~17s later and FAILED with status="DEPLOYING" — a race, not a failure.
Now the deployment-status check polls when the latest deployment sits in any
non-terminal Railway status (QUEUED/BUILDING/INITIALIZING/DEPLOYING/WAITING/
NEEDS_APPROVAL), re-querying every 5s up to a 150s budget until it reaches a
terminal state, then asserts SUCCESS. Terminal-failure statuses (FAILED/
CRASHED/REMOVED/any non-SUCCESS-non-in-progress) still fail FAST with the
original error-string shape — no waiting. Infra/contract errors (network,
GraphQL errors[], missing edge) also fail fast.
The poll loop is fully seamed for tests (injectable sleep/now/budget); the
signature stays backward-compatible (trailing optional pollOpts). probeBaseline
forwards an optional deployPoll through, and both --env staging and --env prod
go through the identical code path, so staging behavior is unchanged.
Red-green tests: in-progress-then-SUCCESS passes (and polls), in-progress-until-
timeout fails with "still in progress", terminal FAILED fails fast with zero
sleeps.
declarative-gen-ui moved to the CopilotKitMiddleware auto-A2UI path across
the 3 langgraph integrations. The middleware's inner forced tool is
render_a2ui, so each integration's gen-ui-declarative.json gained 4
render_a2ui fixtures that share match keys with the pre-existing render_a2ui
entries in that integration's render-a2ui.json (the a2ui_fixed demo). 4
pills x 3 integrations = 12. Same-context cross-demo overlap, disambiguated
at runtime by the probe fixtureFile.
the per-service promote loop ran under `set -euo pipefail`, so the first failing
service aborted the whole `all` fleet promote; extracted to promote-fleet.sh
which attempts every service, accumulates succeeded/failed sets, exits non-zero
only after attempting all, and exports succeeded_csv. verify-prod now runs
`if: !cancelled()` and scopes --services to the succeeded set; the staging
precondition is advisory (promote runs even when it reports red — bin/railway
enforces staging-green per-service); notify success keys on PROMOTE && PROD.
Adds a shell-script-tests CI job (bats + shellcheck) and input-validation
hardening (fail-loud on empty/all-empty CSV, RAILWAY_BIN check, whitespace trim).
Add showcase/scripts/sync-promote-service-options.ts: generates the
promote workflow's service `choice` options from the SSOT
(railway-envs.ts), spliced between BEGIN/END markers in
showcase_promote.yml. Fail-loud throughout — every emitted token must
resolve to exactly one service under the resolve-step predicate
(name|dispatchName match AND probe.prod), tokens are YAML-safe, args are
strict (a typo'd flag cannot trigger a destructive write), and markers
are validated before any rewrite.
Wire it into a lefthook pre-commit hook (regenerate + restage; set -e so
a failed regen blocks the commit) and an advisory (never-failing) drift
check in showcase_validate.yml. Vitest coverage for ordering, exclusion,
collision/ambiguity guards, marker errors, exit codes, idempotency, and
the import-side-effect guard.
The gen-ui-headless-complete probe
(showcase/harness/src/probes/scripts/d5-gen-ui-headless-complete.ts)
references fixtureFile "gen-ui-headless-complete.json", but that file
existed in no D6 slug, so the probe's first leg 503'd under strict.
Add gen-ui-headless-complete.json for all 18 D6 slugs, modeled on the
green langgraph-python headless-complete.json pattern: the four gen-UI
pills (weather/stock/highlight/revenue) with narration (toolCallId)
fixtures FIRST and toolcall (userMessage+context) fixtures AFTER, no
turnIndex gate, so every one of the probe's four sequential turns in one
chat thread matches regardless of prior assistant/tool history. Interrupt
fixtures are intentionally omitted (interrupt-headless is a separate
cluster item).
These 8 fixtures per slug share match keys with the pre-existing
headless-complete.json for the same context (the demos share pills and
are disambiguated at runtime by probe path), which raises the
aimock-fixtures collision-detection exact-duplicate count by 46. Bump
KNOWN_DUPLICATE_CEILING 230 -> 276 to match, consistent with how prior
per-integration feature fixtures bumped the baseline; the substring-shadow
ceiling is unchanged (no new shadows).
Validated: all 732 aimock-fixtures schema/collision tests pass, and a
local aimock --strict run returns 200 for each of the four pills across
turns (langgraph-python + pydantic-ai spearheads, plus spring-ai).
Reject ASCII control chars, a colon (port suffix), and any char outside the
DNS-label charset; switch the Host brand from a string-literal __brand to a
non-exported unique symbol so a stray `as Host` cast from outside the module
is a type error. asHost stays the sole runtime constructor. Adds positive +
negative test coverage incl. interior-tab rejection.
Introduce a branded Host produced by asHost (rejects scheme/path/empty/whitespace/userinfo/query/
fragment); brand at the resolveProbeTargets ingress; make ProbeTarget fields readonly; add asHost
+ override-seam tests.
Closing hardening pass on the showcase deploy-gate's verify-matrix
resolver. The 7-agent review confirmed the gate is correct; this
commit fixes the residual rough edges.
- showcase_deploy.yml: correct the false §3 ok-non-empty comment.
The empty-intersection case can coexist with redeploy_red=false
(every redeploy succeeded, just none probe-eligible) — that's a
correctly-green run, not a red one.
- showcase_deploy.yml: tighten the summary.json shape guard to catch
PARTIAL drift (TOTAL>0 && WITH_STATUS<TOTAL). The previous all-or-
nothing TOTAL>0 && WITH_STATUS==0 check silently dropped drifted
rows on a mixed summary. Validated locally on mixed/normal/empty/
total-drift jq samples.
- resolve-verify-matrix.ts: add asSupportedEventName narrowing helper
+ use it in the CLI. Replaces the unchecked `as` cast — type system
and runtime now tell one story. Resolver's internal eventName
guard becomes defense-in-depth for direct (test) callers.
- resolve-verify-matrix.ts: make the workflow_run boundary total —
summaryPresent MUST be exactly "true"/"false". Any other value
(including "" from a step-id-rename wiring break) throws now
instead of silently emitting has_services=false.
- resolve-verify-matrix.ts: drop the try/catch around
fileURLToPath(import.meta.url) in `invokedDirectly`. The catch
used to swallow ESM-interop failures and silently no-op the CLI
(exit 0, no GITHUB_OUTPUT write → verify skipped = false-green).
- resolve-verify-matrix.ts: reword parseSsotServices JSDoc to
distinguish schema-drift from truncation (the two are different
failure modes, not one conflated story).
- showcase_build.yml: comment addendum on the redeploy-summary
upload — swapping the guard to `if: always()` would red the
legitimate services=='' path (no summary written), trading the
already-closed false-green for a false-red on every non-buildable
push.
- resolve-verify-matrix.cli.test.ts: switch to spawnSync so stderr
is captured on both zero and non-zero exit (execFileSync only
exposes stderr on throw). Hard-code two stable probe-eligible
names ("aimock", "harness") for the sorted-CSV test rather than
picking probe[0]/probe[1] off the live SSOT — the prior test was
tautological (already-sorted in, sorted out) and would silently
pass if the resolver did nothing.
- resolve-verify-matrix.cli.test.ts: add CLI coverage for the
dropped-token ::warning:: path (FIX 3 — the entire drift-detection
contract had zero CLI coverage), the unexpected-EVENT_NAME error
(FIX 5), and the workflow_run-summary_present total boundary
(FIX 7, both "" and "True" inputs).
- resolve-verify-matrix.test.ts: add unit coverage for the new
workflow_run summaryPresent boundary (empty + "True" + the
workflow_dispatch ignores-summaryPresent regression).
Red-green: 6 tests RED before code changes (FIX 3 warning, FIX 5
unknown EVENT_NAME, FIX 7 unit + CLI ×2 for "" and "True"); 79
tests GREEN after.
Validation: 4 vitest files / 79 tests passing; 87/87 ruby specs
passing; actionlint findings unchanged vs integration baseline
(8 → 8, identical diff); yaml.safe_load OK on both workflows.
A 7-agent review of the verify-matrix resolver and its surrounding workflow plumbing found three
boundary surfaces that could silently produce a GREEN deploy on a broken release, plus an
untested CLI contract that CI compares against the literal strings 'true' / 'false'.
FIX 1 — Validate the SSOT shape in loadSsotServices(). The prior `JSON.parse(...) as
{services: SsotService[]}` was an unchecked cast: a truncated/drifted SSOT (emitter crashed
mid-write, or schema renamed) parses fine but silently shrinks/empties the probe-eligible set
→ some redeployed services go unverified, or verify is skipped on a real redeploy. Extract a
pure exported parseSsotServices(raw, path) that requires the shape we depend on (non-empty
services array; each entry has a non-empty string name, an optional string|null dispatchName,
and a probe object with a boolean staging). Throw `::error::SSOT <path> malformed: <detail>`
on any violation. Also re-check existsSync(SSOT_JSON) after the regenerate-if-missing
execFileSync — a regen that exits 0 without writing must not proceed to a useless JSON.parse
crash. Drop the defensive `probe?.staging` once shape is guaranteed.
FIX 2 — Validate summary.json shape in the redeploy-gate bash. The bullseye false-green
surface: if redeploy-env.ts's schema ever drifts (e.g. `status` → `state`, `ok` → `success`),
every `jq select(.status==...)` yields empty → redeploy_red=false AND ok_services="" →
resolver skips verify → GREEN CI on a real unverified redeploy. Add a TOTAL vs WITH_STATUS
shape guard right after loading the summary: if TOTAL > 0 && WITH_STATUS == 0, emit
::error::summary.json has $TOTAL entries but none with status ok|error (schema drift?) and
exit 1. The legitimate empty-array path (TOTAL=0) is preserved.
FIX 3 — Fail loud on unknown eventName in resolveVerifyMatrix. The prior code fell through to
the workflow_run intersection branch for ANY unrecognized eventName (typo, unexpected
trigger), silently emitting has_services=false → indistinguishable from a legit "summary
absent" skip. Add an explicit guard so only workflow_run / workflow_dispatch are accepted;
anything else throws ::error::resolve-verify-matrix: unexpected eventName '<value>'. Tighten
the eventName parameter type to the literal union.
FIX 4 — Trim ok tokens + warn on dropped tokens in okCsvToCanonicalNames. Split, then
.map(t => t.trim()).filter(Boolean) so "a, b" (spaces) matches. Collect tokens that match NO
SSOT service (by name or dispatchName) and have the CLI wrapper emit ::warning::ok_services
tokens dropped (no SSOT match): <list> on stderr when non-empty — surfaces SSOT/build drift.
The pure function stays IO-free; logging lives in the wrapper.
FIX 5 — CLI wrapper integration test. New resolve-verify-matrix.cli.test.ts spawns
`npx tsx showcase/scripts/resolve-verify-matrix.ts` with a temp $GITHUB_OUTPUT file across
four scenarios and asserts the temp file contents EXACTLY (the workflow YAML compares
has_services against the literal strings 'true'/'false', so the byte-for-byte format is part
of the contract). Uses the real railway-envs.generated.json so the loader exercise is real.
FIX 6 — Cleanup. Remove the dead `env: DISPATCH_SERVICE: ...` block on the redeploy-gate
step (the next step redeclares it — leftover from the extraction). Soften the §3
decision-table all-errors bullet to match resolve-verify-matrix.ts's careful wording, and
append that when the success-set is empty (or the intersection collapses to empty), verify
is skipped and the gate reds independently. Append to showcase_build.yml's "Upload redeploy
summary" path-(A) comment that `if-no-files-found: error` still reds path (A) even if a
future change adds `if: always()`.
Tests: red→green for FIX 1/3/4/5 verified locally. Resolve-verify-matrix vitest count:
12 → 28. Full requested suite (resolve-verify-matrix + cli + aggregate-build-results +
lint-rule-no-public-env): 72 passed. showcase/bin ruby specs: 87 runs / 0 failures / 0
errors / 0 skips. actionlint baseline preserved (8 findings, identical to integration tip).
Extract the inline bash+jq decision logic from showcase_deploy.yml's
resolve-matrix job into showcase/scripts/resolve-verify-matrix.ts, a
pure function with a vitest suite. The bash had produced two confirmed
bugs across prior CR rounds, so making it testable is the lasting fix.
Issue A (the bug this PR fixes): when summary_present=true but
ok_services is empty (every service errored on redeploy), the old bash
skipped the intersection and fell through to the full probe-eligible
fleet, gratuitously probing every service against stale :latest. The
resolver now returns has_services=false in that case — enforce-redeploy
-gate independently reds the workflow on redeploy_red=true, so this
case is already loud; there is nothing left to verify.
Parity preserved for unchanged cases:
- workflow_dispatch + 'all'/empty → full probe-eligible set
- workflow_dispatch + specific svc → that one (unknown → error exit)
- workflow_run + summary_present=false → has_services=false
- workflow_run + present + ok non-empty → intersection with probe-
eligible (SSOT key OR dispatchName aliases both resolve)
Also clarified the Upload-redeploy-summary comment in showcase_build.yml
to document both red paths (hard crash → redeploy step exits non-zero;
exit-0-but-no-file → if-no-files-found:error reds the step) so no
false-green path is possible.
Tests: 12-case vitest suite covers each decision-table row plus the
Issue A fix (written red-first; failed against a naive full-fleet
fallback, passed once the early return was added). CLI parity verified
against the real generated SSOT for the three representative env-var
combinations (workflow_run + present + ok=[a,c]; workflow_run + present
+ ok empty; workflow_dispatch + 'all').
D1 — showcase_deploy.yml false-red fix
======================================
The build workflow legitimately uploads no `redeploy-summary` artifact when it ran
(push touched `showcase/**` so `paths:` matched) but `detect-changes` found no
buildable service, so `redeploy-staging` was skipped. The build still concludes
`success`, so `showcase_deploy.yml` fires on `workflow_run` and `resolve-matrix`
runs. `actions/download-artifact@v4` with `name:` HARD-FAILS on a missing
artifact, so the unguarded download was failing the job, and a downstream guard
that trips `enforce-redeploy-gate` on `resolve-matrix.result == 'failure'` was
flipping the workflow RED — a false-red on a routine showcase-docs/script change.
Add an artifact-existence pre-check using `actions/github-script` (pinned by SHA,
matching the existing repo convention) that lists the artifacts for
`workflow_run.id` via `actions: read` (already granted to `resolve-matrix`) and
sets `summary_present=true|false`. Gate the existing download step on
`summary_present == 'true'`. Keep NO `continue-on-error`, so the C1 property
holds: when the artifact exists but the download genuinely fails, the job still
fails loud and `enforce-redeploy-gate` correctly reds the workflow. When the
artifact is legitimately absent, the bash gate's existing `[ ! -f "$SUMMARY" ]`
branch no-ops (`redeploy_red=false`, `ok_services=""`) — nothing was
redeployed, so there is nothing to gate.
Updated the step comment block to enumerate the three distinct cases now
handled: workflow_dispatch (no download); workflow_run + artifact absent
(graceful skip); workflow_run + artifact present (download with fail-loud).
L1-L5 — env lint rule hardening
===============================
- L1: route the destructuring (VariableDeclarator/ObjectPattern) branch through
the shared `staticKeyName()` helper so the computed-string-key form
`const { ["NEXT_PUBLIC_X"]: y } = process.env` and the no-expression
template-literal form `const { [\`NEXT_PUBLIC_X\`]: y } = process.env` are
caught with the same parity as the bracket-member read.
- L2: unwrap a wrapping `ChainExpression` at the top of `isProcessEnv()` so
`process.env?.X` is matched robustly across parser flavors; corrected the
helper's doc comment to describe the actual semantics.
- L3: export `BANNED_KEYS` from the rule module and have the table-driven test
dynamically import the rule's own Set instead of hand-mirroring it — the
test set now cannot drift from the rule.
- L4: added override-scoping fixtures for `showcase/shell/src/**` and
`showcase/shell-dojo/src/**`; the `.oxlintrc.json` override list already
includes these, but the test now exercises them so an accidental drop is
caught.
- L5: expanded the file-header "Out of scope" doc list to include bulk-iteration
reads (`Object.keys/values/entries(process.env)`, for-in, spread
`{...process.env}`), rest-pattern destructuring, compound-assignment LHS, and
update operators. Documentation-only — the deliberate non-coverage is now
auditable.
Validation
==========
- RED→GREEN confirmed for L1 (two new destructuring computed-key tests) and L3
(dynamic `await import(...)` of BANNED_KEYS failed pre-fix with
"Rule module did not export a non-empty BANNED_KEYS Set", green after export).
- vitest: 38 passed (was 34 baseline + 4 new); aggregate-build-results 6 passed.
- Ruby promote suite: 87 runs, 251 assertions, 0 failures (unchanged).
- python3 yaml.safe_load: showcase_deploy.yml + showcase_build.yml +
showcase_promote.yml all parse OK.
- actionlint: zero NEW findings on the changed file. The pre-existing
showcase_build.yml SC2086/SC2129/runner-label findings are identical on the
integration baseline (unchanged by this commit).
Seven-agent CR surfaced correctness defects in the build/deploy/promote
pipeline and in the no-public-env-shell-read oxlint rule. This commit
closes the false-green paths and broadens lint coverage.
Workflow fixes:
- showcase_deploy.yml: drop `continue-on-error: true` on the redeploy-summary
artifact download. The dispatch path is already guarded by the `if:
workflow_run` clause, so the bash "no summary" branch handles legitimate
manual dispatches. A genuine workflow_run download failure must now fail
loud instead of silently widening verify to the full service set against
stale `:latest`.
- showcase_build.yml: redeploy-staging now intersects the build matrix with
the aggregator success set (`needs.aggregate-build-results.outputs.results`,
status == "success") before producing the redeploy CSV. Failed/skipped
slots no longer get redeployed (which would just re-pull stale `:latest`
and look healthy).
- showcase_build.yml: `notify-all-builds-failed` now additionally requires
`needs.build.result == 'failure'` so it doesn't Slack-spam when the build
job was SKIPPED (verify-image-refs upstream failure).
- showcase_build.yml: `notify` now lists [build, aggregate-build-results,
redeploy-staging] in `needs:` so aggregator/redeploy failures still emit
a Slack signal. `if: failure()` still skips when none of the needs failed.
- showcase_build.yml: `set -euo pipefail` on the Prepare build args step
so a transient $GITHUB_OUTPUT write failure can't ship images without
COMMIT_SHA/BRANCH baked in.
- showcase_deploy.yml: `enforce-redeploy-gate` now also trips on a
resolve-matrix failure (`needs.resolve-matrix.result == 'failure'`) so
an upstream crash that leaves `redeploy_red` empty can't bypass the gate.
- Doc-comment accuracy: drop stale `(PR #5093)` reference; correct the
env-IDs source-of-truth comment; document the optional `skip_build` field
in ALL_SERVICES; clarify that health_path is informational and verify
uses per-service drivers; add the missing `resolve-targets` step 0 to the
promote workflow's "Order:" header.
Aggregator fix (RED-GREEN):
- aggregate-build-results.ts: throw on zero slot dirs. The job is gated
upstream on has_changes == 'true', so zero slot dirs is a broken artifact
download, not a legitimate empty build set. Silently emitting
any_success=false + results=[] is indistinguishable from "all builds
failed" and lets the deploy workflow fall back to probing the full
service set against stale `:latest`. Refuse the ambiguity.
- aggregate-build-results.test.ts: existing empty-INPUT_DIR test was
updated to assert the throw (was: return []).
Oxlint rule (RED-GREEN):
- no-public-env-shell-read.mjs: handle destructuring reads
(const { NEXT_PUBLIC_X } = process.env and aliased form), template-literal
computed keys (process.env[\`NEXT_PUBLIC_X\`]), and explicitly skip
assignment-LHS / `delete` targets (writes are not reads). Optional
chaining already worked through the existing MemberExpression path.
Aliasing (`const e = process.env; e.X`) is intentionally documented as
out of scope (needs scope tracking). Description sharpened to say the
rule guards a specific banned-key set, not all NEXT_PUBLIC_* reads.
- .oxlintrc.json: tighten the off-override glob from
`showcase/**/*runtime-config*` to
`showcase/**/lib/runtime-config*.{ts,tsx}` so it only silences the
intended implementation files, not arbitrary paths containing that
substring.
- lint-rule-no-public-env.test.ts: rewritten as table-driven coverage of
every BANNED_KEYS entry (dotted + bracket-string forms), every ALLOWED
key (asserting non-firing), all new variants from the rule expansion,
the assignment/delete non-fire cases, and override scoping
(runtime-config exempt; packages exempt; shell-tree non-runtime-config
flagged).
Validation:
- actionlint on all three workflows: 8 pre-existing findings (depot label,
pre-existing SC2086 infos in untouched steps); my edits add zero.
- python3 yaml.safe_load: all three workflows OK.
- vitest aggregate-build-results.test.ts: 6/6 pass (incl. new throw test).
- vitest lint-rule-no-public-env.test.ts: 34/34 pass.
- vitest full showcase/scripts suite: 1654/1654 pass across 46 files.
- ruby showcase/bin/spec/all_tests.rb: 87 runs, 0 failures.
- Intersection jq proof (matrix a,b,c × success a,c) → "a,c"; all-failed
→ ""; skipped status excluded.
Distinguish ENOENT (treat as drift) from other read errors in
emit-railway-envs-json.ts --check; non-ENOENT errors now exit 2 with the
real error on stderr instead of being silently coerced into a misleading
'stale' message or an overwrite on a false drift signal.
Add a --out=<path> override so tests can write to a temp directory and
never mutate the tracked railway-envs.generated.json artifact. Rewrite
the emit-railway-envs-json test to use mkdtempSync + --out, switch the
staleness assertion to spawnSync so it asserts on exit code 1 plus the
stale-diagnostic substring, and add coverage for the new EISDIR
fail-loud path. After this change git status is clean post-test.
New TS probe driven off railway-envs SSOT. Accepts --env staging|prod and
optional --services CSV; iterates SERVICES where probe[env]===true and
dispatches to per-driver feature-level verifiers. Refuses to start when a
probe-required service is missing a domain for the requested env (no
silent skip). HTTP 200 is necessary but not sufficient.
Adds verify-deploy.drivers.ts dispatch with exhaustive never check on
ProbeDriver, plus one stub module per ProbeDriver literal (shell, docs,
dashboard, dojo, harness, eval, aimock, pocketbase, webhooks, agent).
Stubs fail loud with an explicit "not yet implemented" error so any
accidental real-network invocation surfaces; per-driver feature-level
impls land as subsequent micro-tasks. Refs spec section 3 / section 3.5.
[BLITZ:A5]
Adds optional REDEPLOY_SUMMARY_JSON path; when set, redeploy-env writes
a structured per-service record array {service,status,error?}. PR #5093's
exit-code contract is preserved (staging=0, prod=1-on-failure).
Consumed by showcase_deploy.yml to fail the workflow on staging per-service
errors without changing the script's exit semantics. Refs spec §3.
Plan-B / Option-B migration moved every shell URL/analytics key off the
build-time NEXT_PUBLIC_* env channel and onto runtime config served via
__SHOWCASE_CONFIG__ + getRuntimeConfig(). To prevent a silent regression
where a future change reintroduces a direct process.env.NEXT_PUBLIC_*
read in shell code (which would re-freeze the value at build time and
break no-rebuild env switching), add a focused lint rule.
The rule (copilotkit/no-public-env-shell-read) is implemented as a
custom oxlint JS plugin rule in the existing copilotkit plugin and
enabled under shell-scoped overrides in .oxlintrc.json:
- Errors on process.env.NEXT_PUBLIC_<URL/ANALYTICS> reads in:
showcase/shell-dashboard/src/**, showcase/shell-docs/src/**,
showcase/shell/src/**, showcase/shell-dojo/src/**
- Banned keys: POCKETBASE_URL, SHELL_URL, BASE_URL, OPS_BASE_URL,
INTELLIGENCE_SIGNUP_URL, POSTHOG_KEY, POSTHOG_HOST, SCARF_PIXEL_ID,
GOOGLE_ANALYTICS_TRACKING_ID, REB2B_KEY, REO_KEY
- Intentionally allowed (NOT banned): NEXT_PUBLIC_COMMIT_SHA and
NEXT_PUBLIC_BRANCH (build-stamped artifact identifiers per B10/B11)
and NEXT_PUBLIC_LOCAL_BACKENDS (computed from shared/local-ports.json
at build, local-dev only).
- Excluded files (rule disabled via a follow-up override): MDX content
under shell-docs/src/content/**, runtime-config implementation files,
and *.test.{ts,tsx} / *.spec.{ts,tsx}. oxlint does not support
excludedFiles inside an override block, so the exclusion is expressed
as a later override that sets the rule to off.
Plan-B originally targeted oxlint's eslint/no-restricted-syntax with an
AST-selector regex. oxlint 1.x does not implement that rule (only
no-restricted-globals / no-restricted-imports), so the equivalent guard
is realized as a small custom rule in the existing copilotkit JS plugin
(meta.name=copilotkit), reusing the same plugin loader the repo already
has for require-cpk-prefix and no-single-arg-zod-record.
Verification (red-green): the rule fires on a fixture containing
process.env.NEXT_PUBLIC_POCKETBASE_URL and does NOT fire on a fixture
containing process.env.NEXT_PUBLIC_COMMIT_SHA. Test pins the config via
-c so it works inside git worktrees nested under .claude/worktrees/
where oxlint's automatic upward config search can miss the worktree's
own .oxlintrc.json.
All four shells lint clean: 0 errors of the new rule across
shell-dashboard (114 files), shell-docs (137), shell (29), shell-dojo (6).
Cover four malformed-ref shapes the gate must reject:
* `:sha256-<hex>` (missing the @ separator — looks like a digest
pin but is actually a tag)
* `@sha256:<too-short-hex>` (truncated digest hex)
* The 2026-04-21 `...atest` corruption shape from the script
docstring (the original reason this gate exists)
* Non-ghcr.io registries on both envs
These match the canonical PROD_SHAPE / STAGING_SHAPE regexes in
verify-railway-image-refs.ts; the tests are the regression guard
that ensures a future "relax the regex" change cannot ship without
explicitly turning these red first.
Lock in shape behaviour for dashboard, docs, dojo, shell, and
harness with explicit per-service red-green cases:
* prod with :latest -> fail (must be @sha256)
* prod with @sha256 on the correct repo -> pass
* staging with :latest on the correct repo -> pass
* staging with @sha256 -> fail (must float on :latest)
* wrong GHCR repo name on either env -> fail
The validateImage body is unchanged — it was always shape-pure and
shape-correct. Before WS-C these five services were never exercised
through the gate at all (gateValidated:false). These tests are the
regression guard that ensures a future edit doesn't accidentally
re-introduce the Phase-2 carve-out without anyone noticing.
Flip dashboard, docs, dojo, shell, and harness from
gateValidated: false to gateValidated: true and simultaneously add
the corresponding repoNameOverride for both envs:
dashboard -> showcase-shell-dashboard
docs -> showcase-shell-docs
dojo -> showcase-shell-dojo
shell -> showcase-shell
harness -> showcase-harness
These two halves MUST land in the same commit. Flipping
gateValidated without the override would make the gate look up
ghcr.io/copilotkit/<railway-name> (e.g.
ghcr.io/copilotkit/dashboard:latest) which does not exist — the
gate would fail on first run. Adding the override without flipping
gateValidated is dead code: main() short-circuits unvalidated
services before consulting the override. Only the union is correct.
Also remove the Phase-2 deferral comments and refresh the
ServiceEntry.gateValidated JSDoc — there are no Phase-2 holdouts
left. All 27 services are now gate-validated; gateIgnore remains
the sole escape hatch and is unused by every current entry.
The Railway -> SSOT direction at verify-railway-image-refs.ts was
warn-and-continue: an out-of-band Railway service with a malformed
ref would not turn CI red as long as nobody read the warning line.
Replace that branch with a hard failure path that lists the
untracked service under its own failure class, with a clear remedy
in the error message (add to SSOT, or set gateIgnore on an existing
entry).
Refactor main() to call two pure helpers (findUntrackedServices,
summarizeFailures) so the policy is unit-testable without going
through Railway GraphQL. The SSOT -> Railway direction
(findMissingServices) is unchanged: it is already correct and is a
separate coverage class.
Adds red-green tests for the summarizeFailures shape, covering the
three failure classes (shape violations, SSOT->Railway drift,
Railway->SSOT drift) and the success path.
Widen `ServiceEntry` with an optional `gateIgnore: boolean` field
(default false / unset) so the image-ref gate can deliberately exclude
a Railway service from BOTH coverage-direction checks. No behavioural
change in this commit; the field is consumed by the next commit which
flips the Railway->SSOT direction from warn-only to hard-fail.
Stub-export `findUntrackedServices` from verify-railway-image-refs so
the unit test for the new direction can compile. Behavioural wiring
into main() lands in the next commit.
Adds verify-railway-image-refs.test.ts with the first unit tests
covering the new field surface and the helper contract.
Adds Domains, ProbeDriver, ProbeConfig types + domainFor() helper that
throws on unknown service/env. Populates domains.{staging,prod} and
probe.{staging,prod,driver} on every SERVICES entry. Adds
emit-railway-envs-json.ts to serialize the SSOT for the Ruby side and
the workflow consumers.
Refs spec §3 / §3a.
## Summary
Adds per-integration D6 aimock fixtures for 18 showcase integrations,
completing the D6 probe coverage that #5022 (Slice 3) scaffolded. Each
integration now has the standard set of D5 feature-type fixtures
(agent-config, auth, byoc, gen-ui-*, interrupt-headless, multimodal,
prebuilt-*, tool-rendering-*) under `showcase/aimock/d6/<integration>/`.
## Why
Slice 3 (#5022) shipped the per-integration directory structure + D6
probe driver + harness scoping. The first D6 probe run reported most
integrations RED because most integrations' D6 directories were missing
per-feature fixtures (only langgraph-python had any). This PR fills in
those gaps.
## Authoring rules applied
- Every fixture has `match.context = "<integration>"` for
cross-integration isolation
- toolCall responses use `hasToolResult: false` (or
`toolName`/`toolCallId` gates) to prevent re-match loops
- Conversation turns match the corresponding D5 probe scripts at
`showcase/harness/src/probes/scripts/d5-<feature>.ts`
- Reference shape: langgraph-python's fixtures + each integration's
existing D5/D6 fixture conventions
- Skipped features: each integration's `not_supported_features` list
(e.g., google-adk skips gen-ui-interrupt + interrupt-headless)
## CR
Two rounds of 7-agent unbiased CR converged. R1 surfaced 5 bucket-(a)
findings, all addressed in R2 fix commit. R2 confirmation converged with
no in-scope regressions.
## Test plan
- [ ] Showcase auto-redeploys on merge (path filter matches
`showcase/**`)
- [ ] Next D6 probe cycle on production harness writes
`d6:<slug>/<featureId>` rows for all 18 integrations
- [ ] Dashboard D6 chips reflect real per-integration state instead of
all-gray
Bumps exact-duplicate ceiling 11 → 230 and substring-shadow ceiling
126 → 151. The D6 per-integration fixtures naturally share match keys
with pre-existing demo fixtures in the same context scope, disambiguated
at runtime by the active demo/probe path.
The four fixture routing regression tests still referenced the old
monolithic d5-all.json / smoke.json / feature-parity.json files that
were reorganized into per-integration d4/ d6/ shared/ directories.
Changes:
- Load fixtures via glob from d6/langgraph-python/ (reference
integration) instead of deleted monolithic files
- Add _context to test requests (aimock 1.26.1 checks req._context
against match.context for per-integration scoping)
- Adapt subagents test from toolCallId-based chaining to
turnIndex-based chaining (matches new D6 fixture structure)
- Adapt state-context test to D6 context-scoping model (no
systemMessage matching; routing happens via X-AIMock-Context)
- Remove _migrated-from-*.json shared files (all fixtures already
exist in per-integration D6 dirs; the migration files caused
620 shared-vs-scoped collisions)
- Add KNOWN_DUPLICATE_CEILING=11 ratchet for pre-existing D6
intra-feature duplicate match keys
Add fixture analysis/split/merge scripts for the D6 restructure,
update aimock fixture collision detection tests for the new directory
layout, update GOTCHAS.md and QA-COVERAGE.md with D6 notes, and
add --d6 flag support to the showcase test CLI.
The previous fixture regression (HTML+CSS only, no jsFunctions) slipped
past CI because the e2e suite only asserted "iframe mounts with non-empty
srcdoc" — which passes whether or not the iframe is interactive. Adds
two layers of guard so the same regression cannot land silently:
1. showcase/scripts/__tests__/open-gen-ui-advanced-fixtures.test.ts
(vitest, runs in showcase_validate on every PR): asserts each of the
three interactive fixture entries in d5-all.json ships jsFunctions
referencing the matching host bridge (evaluateExpression / notifyHost).
Catches "someone removed jsFunctions" at PR-time with no
infrastructure dependencies.
2. showcase/integrations/langgraph-python/tests/e2e/open-gen-ui-advanced.spec.ts
(playwright, runs in test_e2e-showcase-on-demand): adds three
round-trip tests that drive the in-iframe controls and assert the
host-side handler ran by capturing its console.log + verifying the
iframe output element reflects the host response. Catches "the
renderer fails to inject jsFunctions into the sandbox" too.
The e2e tests also switch the existing smoke tests off pill-click and
onto a textarea-driven fill+Enter path, following the same precedent as
commit 15db0bbf3 (gen-ui-headless-complete) — chip mounts diverge
between EmptyState and SuggestionBar surfaces, and Playwright's pill
click races React hydration. Using [data-testid="copilot-chat-textarea"]
with an explicit click + waitForLoadState("networkidle") makes the
suite reliable end-to-end (7/7 passing locally against the aimock-driven
stack).
The "Draw a flowchart" suggestion pill in the mcp-apps demo sent
"Use Excalidraw to draw a simple flowchart with three steps." which
had no matching create_view fixture in d5-all.json. aimock walked
through to feature-parity.json's `{userMessage: "steps"}` substring
fixture and returned a generic "Here is my plan..." content blurb
with no MCP tool call, so the runtime never invoked create_view, the
MCP middleware never fetched the UI resource, and the sandboxed
iframe never mounted.
Add a fixture pair in d5-all.json (and its harness mirror) keyed on
"draw a simple flowchart": turn 1 emits create_view with a three-
step Start -> Process -> End flowchart, turn 2 emits the narration
after the tool result. The distinctive substring beats the generic
feature-parity catch-alls under first-match-wins.
Adds a regression test that loads the same fixture files in the same
order as docker-compose.local.yml and asserts via aimock's matchFixture
that each mcp-apps pill routes to its create_view fixture on turn 1 and
its narration fixture on turn 2.
Jordan Ritter