The strands-typescript SSOT entry is ciBuilt:true but staging-only (prod
instance not yet provisioned). redeploy-env.ts's default scope was the full
CI_BUILT_SERVICES set for BOTH envs, so a staging-only ciBuilt service would
wrongly enter the prod default scope and fail a manual `redeploy-env.ts prod`
(no prod instance). Filter the default scope by env declaration (explicit
--services stays unfiltered, preserving the contract-pin that an operator can
force a named service in an env it does not declare). imageOf expansion was
already env-aware; this extends the same invariant to the base scope.
Update the inventory-lock test counts for the new service (total 40->41,
CI_BUILT 38->39, staging default scope 39->40; prod default scope stays 38 as
the staging-only service is now correctly excluded).
The pool-fleet migration is complete: the control-plane harness plus the
prod workers (deployed 2026-06-19, HARNESS_ROLE=worker, pool count 2) now
cover every probe dimension the interim `harness-legacy` fleet-migration
bridge was holding live, so `harness-legacy` is dead config.
This is the code-side cleanup only:
- Remove the `harness-legacy` entry from the railway-envs SSOT and the
now-dead `key === "harness-legacy"` special-case in computePromoteClosure.
- Regenerate railway-envs.generated.json (41 -> 40 services).
- Drop harness-legacy from the golden snapshot, the gateIgnore expectation
sets, the promote-notify fixtures, and the redeploy-env doc comments;
update the service-count assertions (41 -> 40).
- Fix the stale "STAGING-ONLY" harness-workers comment: prod workers are
live on Railway, though this SSOT entry still models the staging
instance only (no prod env backfilled here yet).
The live Railway `harness-legacy` service is torn down separately as a
follow-up infra step.
Promote-notify Slack message: name the promoted AND failed services (one
Failed: header + bullets), legible "(N): <names>" count, real wall-clock
elapsed (integer-coerced), and drop the constant verify-prod legend line.
Durable healthcheckPath: track it per-service/env in the SSOT (railway-envs),
re-assert it on the promote pin path (omit-when-absent, never null), and route
deploy-to-railway provisioning through isTrackedService/resolveProvisionHealthcheck
so a tracked-null service omits the healthcheck while an untracked one keeps the
/api/health default — fixing the silent prod-healthcheck drift that refused aimock.
Tests: ruby pin-reassert spec + deploy-to-railway healthcheck spec + emit/golden/accessor.
Flip the 12 starter-* services to behave like every other managed
showcase service: always-on and staging-probed.
- railway-envs SSOT: staging probe false -> true for all 12 starters
(prod was already on); refresh the stale "staging probe OFF" /
fail-loud-placeholder doc-comments to reflect baseline-driver verification.
- provision-starter-fleet: sleepApplication true -> false so the
provisioner creates always-on (not sleepable) Railway instances; update
the "the whole point — sleepable" header/inline comments + log lines.
- Regenerate railway-envs.generated.json (emit-railway-envs-json.ts) and
the golden fixture; update the SSOT probe + provisioner sleep assertions.
A `shell-docs` promote expanded its dependency closure to include the
always-on Tier-1 verification set (harness, dashboard, …); harness then
tripped a P6 env-divergence WARN-refusal and the tier barrier gated docs
as NOT-ATTEMPTED. docs has no runtime dependency on the control plane.
Introduce a declarative `standalone` service class: a leaf that neither
depends on anything nor gates on anything. computePromoteClosure (TS) and
the resolve-promote-targets jq both skip the Tier-1 union when the request
is entirely standalone, so a standalone request promotes ONLY itself; the
fleet runner promotes `s:`-marked services ungated (never NOT-ATTEMPTED on
an unrelated failure, never gating a tier). Flag `docs` standalone.
Red-green: railway-envs.test.ts (4), resolve-promote-targets.bats (5),
promote-fleet.bats (3) fail on origin/main (docs closure pulls harness;
`s:` is an unknown tier) and pass after the fix.
The webhooks SSOT comment claimed the push-driven default scope is
"guaranteed" to leave webhooks untouched — false: a push touching the
build workflow files trips the workflow_config paths-filter disjunct,
which selects every matrix slot (webhooks included; its skip_build slot
still reports success and enters the redeploy CSV). Reworded to state
the actual behavior. imageOf doc now states the enforced NON-EMPTY
subset constraint; serviceEnvPairs doc now truthfully says it has no
consumers yet; file header notes the probe flag default and
bin/railway's Ruby-only "stage" synonym.
Test hygiene: drop the stale bin/railway line-number citation from a
test name, the _envConfigTypeAnchor (EnvironmentConfig is genuinely
referenced by the shape-compile test), a dead eslint-disable, and a
dead `as never` cast (env is an open string); align the webhooks
dispatch-name pin regex with the extraction regex's whitespace
tolerance.
assertEnvRegistryConsistent gains four clauses: (iv) a key present in
both ENV_IDS and ENV_ID_BY_NAME must carry the same env-id (ENV_IDS.prod
drifted to the staging id previously passed every clause while
resolveEnv("prod") silently returned staging); (v) every ENV_IDS env-id
must be carried by a canonical name (was only caught lazily in
resolveEnv); (vi) registry keys must be trim().toLowerCase()-normalized
(resolveEnv lowercases input, so a non-lowercase spelling is registered
but unreachable); (vii) no registry key may be an Object.prototype
property name.
expandImageConsumers' per-entry env skip-check becomes an own-property
test (Object.hasOwn) for uniformity with every other lookup in the file.
REDEPLOY_SUMMARY_JSON is trimmed before the set-but-empty branch so a
whitespace-only value hits the loud warn path instead of attempting a
JSON write against a garbage path.
- railway-envs: route envsFor/instanceIdFor/domainFor/probeEnabled (and
repoNameFor) through shared getEntry/getEnvCfg own-property helpers so
inherited Object.prototype keys on either axis produce the curated
error (or probeEnabled's contract false) instead of raw TypeErrors,
silent undefined, or a spurious probe=true.
- railway-envs: two new module-load invariants (synthetic-map
injectable): assertEnvRegistryConsistent (per-service env keys are
registered canonical names; ENV_ID_BY_NAME env-ids unique; every
canonical name has an ENV_IDS spelling) and
assertServiceAndInstanceIdsUnique (serviceId unique per entry,
instanceId globally unique).
- redeploy-env: invert the exit-code policy to fail-loud by default —
any env except the documented staging carve-out exits non-zero on
per-service failure (a future preview/canary env inherits fatal
semantics instead of silently swallowing failures).
- redeploy-env: sanitize per-service THROWN error messages through
sanitizeErrorBody before recording; flatten bare \r in the summary
table escape.
- redeploy-env: warn on set-but-empty REDEPLOY_SUMMARY_JSON; reject
flag-like --services CSV parts (both forms) and a flag-like first
argument (missing env); derive usage env lists from ENV_IDS.
- tests: prototype-key sweep across all accessors, invariant
positive/negative coverage, third-env exit-code pin, sanitization and
CLI-guard coverage, makeLiveRedeploy !res.ok and non-true mutation
branches.
Confirmation-CR bucket-(a) fixes for PR #5353 — one coherent concern:
env-name resolution has exactly ONE authority (the ENV_IDS /
ENV_ID_BY_NAME registries) and SSOT accessors fail loud instead of
silently returning wrong values.
- runRedeploy: resolve envId via ENV_ID_BY_NAME with an Object.hasOwn
guard + fail-loud throw listing the registered envs. Removes the
hardcoded `prod`/`staging` pair check and PRODUCTION/STAGING ternary
that contradicted the SSOT's documented open-env contract ("a new env
needs only a registry entry"); the registry lookup subsumes it.
- resolveEnv: derive resolution entirely from the registries (ENV_IDS
spellings -> env-id -> canonical ENV_ID_BY_NAME name) instead of its
own hardcoded synonym chain. Behavior identical for
prod/production/staging; still throws on unknowns, and now also
throws on a mis-wired registry (a spelling whose env-id has no
canonical name).
- serviceForDispatchName: fix the docstring's false "CI-built service"
claim — it does no ciBuilt filtering and tests pin the unfiltered
behavior (the non-CI-built webhooks resolves).
- repoNameFor: fail loud (consistent with instanceIdFor/domainFor)
instead of silently echoing the service name — the exact
silently-wrong-GHCR-name class this PR's hardening targets. Throws on
unknown service, on an env not registered in ENV_ID_BY_NAME
(unnormalized synonyms like "production"), and on a registered env
the service does not declare. Keeps the documented default (the
service name) for declared envs without an override.
Call-site enumeration confirming nothing relies on the old fallback:
- verify-railway-image-refs.ts:523 — iterates the entry's DECLARED
environments keys, registry-filtered, SSOT-matched service names
- __tests__/railway-envs.golden.test.ts:81 — iterates envsFor(name)
(declared envs only) over real SSOT keys
- railway-envs.test.ts repoNameFor cases — dual-env services,
prod/staging only
- __tests__/verify-railway-image-refs.test.ts:264 — FIVE_NEW keys,
all dual-env
- resolveTargetServices: throw when an explicitly-provided services
list resolves to zero entries (whitespace-only programmatic input)
instead of letting runRedeploy exit 0 having redeployed nothing; the
default undefined -> full CI-built scope is unchanged.
- makeLiveRedeploy: add signal: AbortSignal.timeout(30s) so a hung
Railway API records a per-service FAIL instead of stalling CI, and
pass GraphQL errors[].message through sanitizeErrorBody for
consistency with the HTTP-error path. Exported for direct unit tests.
Red-green: 11 new tests (open-env registry resolution incl. a
runtime-registered hypothetical env, repoNameFor negatives,
empty-resolution throw, abort-signal presence, GraphQL error
sanitization) all failed against the old code; full showcase/scripts
suite green (50 files, 1807 tests) + tsc --noEmit clean.
harness-workers runs the SAME showcase-harness GHCR image as the harness
scheduler but has ciBuilt:false (it has no build slot of its own), and the
CI staging redeploy scope was derived purely from ciBuilt — so a main-merge
rebuild of showcase-harness:latest only bounced the scheduler while the
workers silently kept running the stale image (PR #5352's worker-side fixes
never reached staging).
Model image consumption explicitly in the SSOT instead:
- railway-envs.ts: new optional `imageOf` field on ServiceEntry — the SSOT
key of the ciBuilt service whose image this entry runs. Set
`imageOf: "harness"` on harness-workers. New module-load invariant
`assertImageConsumersValid` (fail-loud, same style as
assertDispatchNamesUnique): imageOf must name an existing SSOT key, the
target must be ciBuilt, and the consumer itself must not be ciBuilt.
- redeploy-env.ts: new `expandImageConsumers(names, env)` applied inside
runRedeploy — the redeploy set becomes the resolved scope PLUS any
service whose imageOf points at a service already in scope. Env-aware:
a consumer only joins envs it declares, so the staging-only worker
never enters a prod redeploy (prod behavior unchanged).
No workflow change needed: showcase_build.yml keeps passing the
built-and-successful dispatch_names; the script expands them. Gate
behavior (gateIgnore / image-ref gate), the generated JSON
(emit --check passes byte-identical), and the promote dropdown are all
untouched. harness-legacy deliberately gets no imageOf (pinned pre-fleet
digest; must not follow rebuilds).
The pool-fleet worker's Railway service is named `harness-workers` (PLURAL),
but the SSOT keyed it `showcase-harness-worker` (singular). The image-ref gate
matches SSOT keys to Railway service names verbatim, so the gate reported
`harness-workers` as an untracked Railway service AND the stale singular key
matched nothing. Rename the SSOT key (and every test/fixture reference) to the
exact Railway name `harness-workers`.
It stays the staging-only, domainless, probe-disabled worker that runs the
shared `showcase-harness` image: serviceId c2aa8a0b-…, staging instance
362c1e37-…, ciBuilt:false, gateIgnore:true, no build slot (so no dispatchName),
single `staging` env with no domain. Add a focused test pinning that shape.
Counts are unchanged (29 services / 26 CI_BUILT) — this is a rename, not an
addition; both harness workers already existed on main.
Verified LOCALLY against Railway: verify-railway-image-refs reports
`54 env-scoped instances verified (2 skipped)` — 0 violations, 0 missing, 0
untracked (harness-workers reconciled, harness-workers + harness-legacy the 2
gateIgnore'd skips). emit --check zero drift, Ruby parity green (borrowed
.up.railway.app host is parity-excluded), full scripts suite + typecheck green.
Replace ServiceEntry's parallel prodInstanceId/stagingInstanceId/domains/probe/
repoNameOverride fields with a single environments: Record<string, {instanceId,
domain?, probe?, repoName?}> map plus a hoisted env-independent probeDriver.
EnvName becomes an open string backed by an ENV_ID_BY_NAME registry so
accessors resolve arbitrary env names; a single-env service (the staging-only
showcase-harness-worker) now simply omits the absent env instead of carrying a
placeholder ID/borrowed host.
Accessors instanceIdFor/domainFor/repoNameFor index environments[env]
(domainFor still throws on missing/scheme); add envsFor(name),
serviceEnvPairs(), and probeEnabled(name, env). Generalize the image-ref gate
(iterate each service's declared environments, resolve env-id via the
registry, sum/iterate missingByEnv over registry env names) and verify-deploy's
host->env reverse-map (envForTarget iterates environments).
Pure TS-internal: emit-railway-envs-json.ts projects the env-map back onto the
FROZEN legacy JSON shape (prodInstanceId/stagingInstanceId/domains/probe/
repoNameOverride) via a documented legacyJsonCompat shim for the two domainless
harness workers, so railway-envs.generated.json stays byte-identical and Ruby
(bin/railway) + workflow jq + the parity test are untouched. Verified: emit
--check zero drift, Ruby test_expected_domains_parity green, golden snapshot
toEqual proves byte-identical resolution for every real (service, env) pair,
full scripts vitest suite green, showcase scripts typecheck clean.
The showcase_build verify-image-refs gate (SSOT = showcase/scripts/railway-envs.ts)
was failing with "1 untracked Railway services" because the interim
harness-legacy staging service (the legacy all-probe harness kept live during
the pool-fleet migration) exists on Railway but had no SSOT entry. That
Railway->SSOT drift check skips the build, so nothing deploys.
Adds a harness-legacy SERVICES entry mirroring the showcase-harness-worker
precedent (PR #5280): ciBuilt:false (not built by showcase_build, runs a pinned
out-of-band digest) and gateIgnore:true (deliberately-untracked for the image-ref
gate). findUntrackedServices treats any SSOT entry as known, so this clears the
untracked failure; gateValidated:false keeps findMissingServices from flagging
it. Real serviceInstance IDs for both envs recorded from Railway GraphQL.
Regenerates railway-envs.generated.json and updates the service-count /
gate-ignored carve-out assertions (28->29 services).
Verified: live verify-railway-image-refs.ts now exits 0 ("54 env-scoped
instances verified, 2 skipped"); without the entry it exits 1 with the
harness-legacy untracked failure. Full scripts test suite green (1771 passed).
The pool-fleet cutover manually created the staging-only
`showcase-harness-worker` Railway service (HARNESS_ROLE=worker, 2
replicas) and flipped the existing `harness` service to
HARNESS_ROLE=control-plane. The new service was untracked in the SSOT,
so verify-railway-image-refs.ts failed the "Showcase: Build & Push"
workflow on every push to main (1 untracked Railway service), which
skipped the harness `build` job and blocked harness image rebuilds.
Add the worker to SERVICES as a staging-only, domain-less queue worker:
- ciBuilt:false — it runs the SAME `showcase-harness` image the existing
harness build slot produces; there is no separate worker build.
- gateIgnore:true / gateValidated:false — no prod instance and no public
domain, so it does not fit the symmetric dual-env shape the image-ref
gate validates. gateIgnore clears the "untracked Railway service"
failure (any SSOT entry counts as known) without tripping a false
"missing from prod" failure.
- repoNameOverride → showcase-harness so the image-ref shape resolves.
- probe disabled in both envs (no externally-reachable health endpoint).
Regenerate railway-envs.generated.json and update the SSOT-count and
gate-coverage test invariants (27→28 services; worker is the sole
intentional gateIgnore/gateValidated:false entry).
PocketBase had no CI build path: `ghcr.io/copilotkit/showcase-pocketbase`
was a stale April `:latest`, and there was no way to ship pb_migrations /
pb_hooks changes without an ad-hoc manual build. Add a `pocketbase` slot to
showcase_build.yml's build matrix, mirroring the harness/aimock entries:
- dispatch_name `showcase-pocketbase`, context `showcase/pocketbase`, its
own Dockerfile, health `/api/health`, railway_id from the SSOT.
- a paths-filter key gated to `showcase/pocketbase/**` so the slot only
rebuilds when PB's own files change (the image is self-contained — no
shared-module copy), not on every showcase push.
- the workflow_dispatch service choice so PB is human-targetable.
Flip the SSOT entry (railway-envs.ts) to `ciBuilt: true` with
`dispatchName: "showcase-pocketbase"` so it is built+pushed (`:sha` +
`:latest`) and joins the default staging-redeploy scope; the build's
redeploy step only touches the matrix-intersect-success set, so PB still
only redeploys when its own files change. Regenerate
railway-envs.generated.json and the showcase_promote.yml service dropdown,
and update the SSOT/redeploy tests that pinned PB as out-of-band
(CI_BUILT_SERVICES 25 -> 26; webhooks stays the only non-CI-built service).
deploy.yml moved SSOT consumption into resolve-verify-matrix.ts; the test now asserts
deploy.yml invokes that script AND the script reads railway-envs.generated.json + filters
probe.staging===true, rather than scanning the YAML for literals that moved one hop down.
A.7 rewrote showcase_deploy.yml to be SSOT-driven (no hardcoded options
list, no inline ALL_SERVICES). Webhooks remains canonically wired via
the SSOT (probe.staging:true, dispatchName:"webhooks"); test now pins
that contract instead of the now-obsolete literal regexes.
Adds Domains, ProbeDriver, ProbeConfig types + domainFor() helper that
throws on unknown service/env. Populates domains.{staging,prod} and
probe.{staging,prod,driver} on every SERVICES entry. Adds
emit-railway-envs-json.ts to serialize the SSOT for the Ruby side and
the workflow consumers.
Refs spec §3 / §3a.
serviceForDispatchName iterates Object.entries(SERVICES) and returns the
first match — a silent dispatchName collision would route a redeploy to
whichever entry happens to iterate first. We now fail loud at module load
via assertDispatchNamesUnique(), and ship synthetic-input tests proving the
invariant fires on a real collision (and stays quiet for entries without a
dispatchName, which is legitimate for out-of-band services).
[BLITZ:L3-wf] E-7a/E-7b wiring.
Adds 'webhooks' as a workflow_dispatch choice in both showcase_build.yml
and showcase_deploy.yml so humans can redeploy/verify the webhooks service
on demand. webhooks' GHCR image (showcase-eval-webhook) is built by a
separate release workflow in the showcase-eval-webhook repo, so:
- paths-filter uses a sentinel that cannot match any in-tree path,
keeping push-driven runs from ever including webhooks.
- The build matrix entry carries skip_build: true; the Build and push
step skips the Depot build for that slot. The per-slot result still
publishes (job.status == success) so the redeploy path proceeds and
redeploy-env.ts picks the existing :latest from GHCR.
The SSOT entry in railway-envs.ts gains dispatchName: 'webhooks' so the
forward/reverse round-trip tests cover it. New tests pin that the SSOT
dispatchName is mirrored in both workflow files' dispatch choice lists
AND ALL_SERVICES JSON.
[BLITZ:L3-wf] E-6a/E-6b/E-6c/E-6d wiring.
Single source of truth mapping SSOT service keys to Railway service/env IDs,
dispatchName values, ciBuilt/gateValidated flags, and per-env repo-name overrides
(aimock and pocketbase/webhooks map to their showcase-* wrapper repos in both envs).
Adds dispatchName round-trip + workflow YAML forward-guard tests.
Ran Shem Tov