CLI accepts optional positional SERVICE + --digest REF (the showcase_promote.yml per-service
loop contract); validates against the SSOT; narrows the promotion target while fleet-scoped
preflight (service-set parity, expected prod domains) keeps reading full snapshots so a
healthy fleet isn't spuriously refused; adds red-green specs.
Nine correctness fixes to bin/railway PromoteCommand, each red-green tested.
- P2 in-flight race-check now compares deployed digest against the digest
captured in @promote_refs (P1-resolved), not svc["digest"] which is nil
for tag-form staging — the check was dead code. Also: parse JSON-string
Deployment.meta; sort fetch_latest_staging_deployments by createdAt desc.
- @promote_refs is RESET (not memoized) at the top of check_p1_ghcr_digests,
so a reused command instance cannot carry stale A-era refs into a B-era
promote. execute_promotion hard-guards against a nil @promote_refs.
- execute_promotion pre-validates that every prod-matched service has a
digest-shaped @promote_refs entry BEFORE pinning anything, eliminating
the partial-promotion-on-missing-ref hazard.
- execute_promotion rescue broadens to MutationError + GraphQL::Error +
StandardError so a transient mid-loop failure still surfaces the
PARTIAL-PROMOTION recovery report; dedup the duplicate warn line and
note that source.image may already be partially advanced on Railway.
- check_p1_ghcr_digests emits REFUSE: P1 ... "no image" for an imageless
staging service (instead of a silent skip that surfaced later as a
misleading "internal error").
- check_p1_ghcr_digests per-service rescue broadens to StandardError so a
non-GHCR error (e.g. ArgumentError, network) does not bypass the rescue
and crash the loop, discarding earlier services' findings.
- pin_and_verify raises ArgumentError immediately if called with a
tag-form image (instead of 30s of futile retries + misleading error).
- pin_and_verify timestamp gate is non-vacuous: a non-nil observed
updatedAt is ALWAYS required, even when pre_update_ts is nil
(which previously collapsed the gate to digest-equality alone).
- run_staging_probe rescues Errno::ENOENT / StandardError around the
IO.popen launch so a missing npx produces a clean ok:false summary
instead of a raw stack trace bubbling out of P3.
Spec hygiene: drop the unused FakeGQL class in test_promote_execute.rb
(it referenced an uninitialized @after_image); give the unresolvable-tag
fixture a placeholder digest so it never builds a malformed "...@" ref;
test_promote_p2.rb tests now capture both streams and assert against the
combined output, matching the convention used elsewhere in the suite.
PromoteCommand had a TOCTOU window: resolved_prod_image(svc) was called
twice for every staging service — once in check_p1_ghcr_digests (where
the resolved digest was manifest_exists-verified), and again in
execute_promotion (whose result is what actually got pinned). Because
staging is a mutable :latest tag, a concurrent push between P1 and
execute could make the two resolutions return different digests, and
prod would be pinned to a digest P1 never verified. It also doubled
the GHCR round-trip per service.
Resolve+verify each staging service's digest exactly once during P1,
store the result on @promote_refs (service_name => digest-pinned ref),
and reuse that exact ref in execute_promotion. If a service has no
entry (P1 didn't run or didn't pass), refuse rather than silently fall
back to a tag.
Also:
- check_p1_ghcr_digests had a method-level rescue Railway::GHCR::Error
that replaced the entire findings array with one entry — so a GHCR
error on service N discarded findings already accumulated for
services 1..N-1. Move the rescue inside the per-service iteration
so each error becomes its own REFUSE finding and the loop continues.
- self.pin_and_verify asserted serviceInstanceUpdate == true but
discarded the serviceInstanceRedeploy result. A failed redeploy
could pass verification because the update mutation had already
advanced source.image+updatedAt. Require truthy redeploy result;
raise MutationError otherwise, symmetric with the update check.
- check_p2_staging_deployments already guarded meta.is_a?(Hash) so it
doesn't crash on a String meta, but the silent skip of the in-flight
race-check was invisible. Add a WARN finding so the skip is visible.
SUCCESS status remains the real gate (still REFUSE).
- execute_promotion now tracks already-pinned services and, on a
mid-loop MutationError, emits a loud PARTIAL PROMOTION report
naming both the already-pinned services and the failing one with
a pointer at bin/railway rollback-commit. Auto-rollback is left as
a follow-up — the goal here is just to make the mixed-state loud
and actionable rather than a quiet exit 1.
The showcase deploy model is STAGING = mutable :latest tag, PROD = immutable
@sha256: digest (P6 enforces both shapes). SnapshotCommand#build_snapshot
stored the raw serviceInstance.source.image, so for staging svc["image"] was
the :latest TAG. execute_promotion was pinning THAT mutable tag to prod via
serviceInstanceUpdate, defeating the immutable-prod invariant before
pin_and_verify raised on the nil expected_digest.
Fix: add PromoteCommand#resolved_prod_image — returns the staging svc as
@sha256:-pinned (pass-through if already pinned; resolves the tag via the
shared GHCR client otherwise; returns nil if the tag cannot be resolved).
execute_promotion now refuses (P0) rather than pin a mutable tag, and
check_p1_ghcr_digests verifies the resolved digest (it previously SKIPPED
tag-form images entirely, so :latest was never P1-checked).
Also:
- P2 race-check guards latest["meta"] when Railway returns a JSON String
(deserialized as Ruby String, not Hash) — .dig used to crash with
NoMethodError. SUCCESS status remains the real gate.
- Remove dead --include-startcommand flag (never read; doubly inert because
P6 REFUSEs on any startCommand divergence).
- Spec hygiene: P3 skip-test raises if probe runs under --no-require-staging-
green; P6 warn-proceed stubs execute_promotion to isolate the gate and
asserts rc==0; test_ghcr_token teardown unconditionally deletes
GITHUB_TOKEN/GHCR_TOKEN/RAILWAY_TOKEN before restoring priors.
70 runs, 204 assertions, 0 failures (up from 66/188 baseline).
P3 is the live re-probe gate spec §7.2 requires: CI history is not
authoritative for staging-green because showcase_deploy.yml uses
cancel-in-progress (the most recent CI run may have aborted before
the probe ran). Promote shells out to Workstream A's parameterized
verify-deploy.ts entrypoint at promote time and refuses on a red
result. Default-on; can be disabled with --no-require-staging-green
(prints "P3 SKIPPED" so the bypass is visible in logs).
run_staging_probe builds a clean child env (RAILWAY_TOKEN, GHCR_TOKEN,
GITHUB_TOKEN, PATH, HOME) and IO.popens `npx --yes tsx
showcase/scripts/verify-deploy.ts --env staging --services <csv>`.
Exit 0 = green, non-zero = red; the last 10 lines of stdout become
the human summary in the REFUSE message.
3 new P3 tests (red probe REFUSE, skip when flag off, green probe
pass). Also stubs run_staging_probe in the P1 and P2 test fakes so
those tests don't shell out to tsx (full suite stays sub-10ms).
Adds the P6 staging/prod parity matrix to promote:
- REFUSE on startCommand, healthcheckPath, or image-shape divergence
(staging is expected :tag/mutable, prod is expected :digest/pinned).
- WARN on region, replicas, restartPolicy, or env-var KEY-set divergence.
WARN findings refuse the promote unless --confirm-divergence is set,
at which point they print "[--confirm-divergence set] proceeding past
N WARN finding(s)" and continue.
- Env var VALUES are never compared (staging/prod hold different
secrets/URLs by design); the NOTE wired in commit #2's
run_with_preflight_only is preserved.
Snapshot schema bumped to v2:
- SERVICE_INSTANCE_QUERY adds healthcheckPath, region, numReplicas,
restartPolicyType.
- SnapshotCommand#build_snapshot maps them onto healthcheck_path,
region, replicas, restart_policy in the snapshot hash.
- SnapshotIO.SCHEMA_VERSION = 2 with SUPPORTED_VERSIONS = [1, 2] so
rollback-commit can still replay v1 snapshots from historical SHAs.
PromoteCommand.image_shape classifies a ref as :digest / :tag /
:missing / :other.
New tests: 6 P6 cases (startCommand REFUSE, healthcheckPath REFUSE,
image-shape REFUSE, WARN-without-confirm refusal, WARN-with-confirm
proceed, every-run NOTE). Two new snapshot tests: v2 captures new
fields end-to-end, and SnapshotIO.read accepts both v1 and v2.
serviceInstanceUpdate returns a Boolean scalar — the spec requires we
confirm that boolean is true AND re-query serviceInstance to confirm
BOTH source.image advanced to the new digest AND updatedAt strictly
advanced past the pre-mutation value. Image-equality alone is
insufficient: a no-op re-pin to the current value would otherwise
appear green.
Implementation:
- PromoteCommand::SERVICE_INSTANCE_RECHECK_QUERY: minimal query adding
updatedAt (kept separate from snapshot's SERVICE_INSTANCE_QUERY to
avoid disturbing snapshot behavior).
- PromoteCommand.pin_and_verify: pre-query updatedAt, run
serviceInstanceUpdate, assert boolean true, run serviceInstanceRedeploy,
then re-query up to RETRY_COUNT=3 times with RETRY_DELAY_SEC=10s
apart. Each retry must observe BOTH gates green (image match AND
updatedAt > pre_update_ts). Otherwise raises PromoteCommand::MutationError.
- execute_promotion now calls pin_and_verify (instead of
RestoreCommand.pin_and_redeploy) so promote inherits the verification.
MutationError is caught and converted to exit 1.
5 new P5 tests: boolean=false refusal, happy-path success,
image-advanced-but-ts-stale refusal, all-retries-stale refusal, and
late-third-retry success.
Implements check_p2_staging_deployments via
RollbackCommand::DEPLOYMENTS_QUERY (input:{serviceId,environmentId})
against STAGING_ENV_ID. For each staging service we are promoting,
requires that the most recent staging deployment is SUCCESS and its
deployed image digest matches the digest we are about to promote.
A mismatch indicates an in-flight build that landed mid-promote;
refusing prevents racing a newer digest into prod.
Refuse messages:
- "no staging deployments found" when edges is empty.
- "latest staging deployment status is <STATUS>, not SUCCESS".
- "in-flight race - latest staging deployment is <NEW> but snapshot
has <OLD>. Re-snapshot and retry."
Also adjusts test_promote_p1 fakes: preflight checks accumulate
findings before any short-circuit, so P2 still issues its gql query
even on a P1 REFUSE. Replaces the raise-on-call FakeGQL with a
benign empty-deployments FakeGQLEmpty so P1 cases stay focused.
3 new P2 tests (FAILED status, digest race, clean SUCCESS).
Full suite: 50 runs, 141 assertions, 0 failures.
Splits the previously-monolithic PromoteCommand#run into:
- capture_snapshots: pulls staging + prod snapshots (test-injectable).
- run_with_preflight_only: runs the P1..P6 preconditions, prints the
mandatory "env var VALUES are not compared" NOTE, gates on
REFUSE/WARN findings, then defers to execute_promotion.
- execute_promotion: the actual pin+redeploy loop.
Adds the P1 GHCR digest existence gate: every staging-side image
(@sha256:...) is HEAD-checked via GHCR.manifest_exists before any
serviceInstanceUpdate is issued. Tri-state result drives explicit
REFUSE messages (:missing -> garbage-collected hint; :auth_failed ->
GHCR_TOKEN / GITHUB_TOKEN hint). P2/P3/P6 land as []-returning stubs
for later phases (D.2 / D.3 / D.5).
Also adds parser flags --confirm-divergence,
--require-staging-green / --no-require-staging-green and
default_options that flips require_staging_green default-on per
spec §7.2 P3.
3 new P1 tests (REFUSE on :missing, REFUSE on :auth_failed with the
expected hint, clean pass on :exists). Existing suite stays green.
Adds two foundational helpers for the promote-hardening P1 check:
- Railway::Auth.ghcr_token: resolves a GHCR bearer separately from the
Railway API token. Prefers GHCR_TOKEN, falls back to GITHUB_TOKEN
(CI workflow token with packages:read). Returns nil if neither is set
so callers can refuse rather than silently fall through to anonymous.
- Railway::GHCR#manifest_exists: digest-existence HEAD against
ghcr.io/v2/<repo>/manifests/<sha256:...>. Returns tri-state
(:exists/:missing/:auth_failed); raises on 5xx; raises ArgumentError
if caller passes an unpinned tag (programmer error — P1 verifies the
concrete bytes about to ship).
GHCR.new default token source switched from ENV["GHCR_TOKEN"] direct to
Railway::Auth.ghcr_token, which adds the GITHUB_TOKEN fallback. Audited
all GHCR.new callers: BaseCommand#ghcr uses the default (intended new
behavior); all existing spec callers pass explicit token: kwarg so they
are unaffected.
Tests: 4 ghcr_token cases + 5 manifest_exists cases.
bin/railway now derives EXPECTED_DOMAINS from
showcase/scripts/railway-envs.generated.json instead of maintaining a
parallel Ruby hash. The TS railway-envs.ts is canonical; CI guards drift
via `emit-railway-envs-json.ts --check`. Adds a Minitest parity test that
boots Ruby and asserts its derived EXPECTED_DOMAINS matches the SSOT
JSON (public hosts only, env-id keys match SSOT envIds). Refs spec §3a.
[BLITZ:A6]
The tool was written from a spec but never exercised against live Railway,
so several queries reference fields that do not exist in the public
schema. Live runs (including the CI lint-prod step) failed with errors
like `Cannot query field "domains" on type "Project"`. Unit tests passed
because they only covered parsing and IO, not the GraphQL shape.
Verified the live schema via introspection (2026-05) and corrected
every mismatch:
- Project has no `domains` field. The previous `customDomains: domains
{ customDomains { ... } }` block on the Project selection is gone.
Custom domains now come from `serviceInstance.domains.customDomains`
for the env we are inspecting.
- Service has no `serviceInstances` field. We can no longer enumerate
per-env instances by nesting under Service. The new flow is:
1. SERVICES_LIST_QUERY -> list services in the project
2. SERVICE_INSTANCE_QUERY -> per (service, env), fetch source,
startCommand, latestDeployment, domains
3. ENVIRONMENT_VARIABLES_QUERY -> all variables in the env, then
group keys by Variable.serviceId for per-service env_keys
- `serviceInstanceDeployV2` does not accept an `image` argument; its
signature is (commitSha, environmentId, serviceId). To pin a service
to a specific image we now use
serviceInstanceUpdate(input: { source: { image } })
followed by serviceInstanceRedeploy. RestoreCommand exposes a
pin_and_redeploy class method that PromoteCommand and PinCommand
reuse.
- `deploymentRollback` returns scalar Boolean, so the previous
`deploymentRollback(id: $id) { id }` was invalid GraphQL — selection
sets are not allowed on scalars. Dropped the selection set.
- Auth.token now reads `user.accessToken` from ~/.railway/config.json
first. The legacy `user.token` field is a short CLI session token
(4 chars on a fresh login) that does not authenticate against the
public GraphQL API and was producing silent "Not Authorized" errors.
Added spec/test_snapshot_graphql.rb with a FakeGQL that returns realistic
shapes, plus regression guards that fail the build if anyone reintroduces
`Project.domains`, `Service.serviceInstances`, `serviceInstanceDeployV2`
with an image arg, or a selection set on `deploymentRollback`.
Smoke-tested live (read-only) against the showcase project:
- lint-prod: "OK: all production services digest-pinned." (27 services)
- snapshot --env staging: full YAML with images, startCommands, env_keys
- env-diff staging production: 32 drift findings (expected, staging is
not digest-pinned)
- resolve-digest ghcr.io/copilotkit/showcase-aimock:latest: digest
returned successfully
No mutating subcommands (restore, rollback, promote, pin) were run live.
Make the lint-prod audit result legible without having to click into the
workflow logs. Two surfaces, both rendered from the same JSON payload:
1. `$GITHUB_STEP_SUMMARY` — structured markdown block at the top of every
workflow run page. Shows on every event (push, pull_request,
workflow_dispatch).
2. Sticky PR comment — one comment per PR, keyed by the HTML marker
`<!-- lint-prod-sticky-comment -->`. Re-runs update the same comment via
`gh api -X PATCH` instead of creating duplicates. Plain `gh` CLI only,
no third-party action.
Both surfaces show: one-line status, a table of the unpinned services only
(not all 27), and a Pacific-time run timestamp with the finding count.
To support the renderer, add `--format json` to `lint-prod`:
{services:[{name,source,status}], findings:N, timestamp:"ISO8601"}
The workflow consumes this shape and also writes `findings` to
`$GITHUB_OUTPUT` so downstream jobs (future Slack alert) can compare runs.
Idempotent: re-running the workflow finds the existing comment by marker
and PATCHes it — never duplicates.
Add --exit-zero flag to bin/railway lint-prod that makes the command exit 0
even when digest-pinning findings exist. Findings still print to stdout so
they remain visible in the CI step log.
Wire the showcase_lint_prod.yml workflow to pass --exit-zero so the job
cannot block PRs while we soak the check against real production state.
Once we have confidence the findings are clean, remove --exit-zero from
the workflow step to flip lint-prod to enforcing.
Updates README to document the advisory-mode behavior and the path to
flipping the check to enforcing.
PR #4449 removed the eval tier system while unaware #4448 had just
merged. Restoring orchestrator, matrix, scope, config, GHA workflow,
and CLI wiring with all 8 CR fixes intact.
Jordan Ritter