CR-round hardening that builds on the bug #1/#2 fixes: the P2 in-flight race
guard now reads the running digest from meta.imageDigest (the dead guard
never REFUSEd before) and is skipped on the --digest override path. Refresh
the snapshot ivar-lint allowlist line numbers after the surrounding CR line
drift.
Nine correctness fixes to bin/railway PromoteCommand, each red-green tested.
- P2 in-flight race-check now compares deployed digest against the digest
captured in @promote_refs (P1-resolved), not svc["digest"] which is nil
for tag-form staging — the check was dead code. Also: parse JSON-string
Deployment.meta; sort fetch_latest_staging_deployments by createdAt desc.
- @promote_refs is RESET (not memoized) at the top of check_p1_ghcr_digests,
so a reused command instance cannot carry stale A-era refs into a B-era
promote. execute_promotion hard-guards against a nil @promote_refs.
- execute_promotion pre-validates that every prod-matched service has a
digest-shaped @promote_refs entry BEFORE pinning anything, eliminating
the partial-promotion-on-missing-ref hazard.
- execute_promotion rescue broadens to MutationError + GraphQL::Error +
StandardError so a transient mid-loop failure still surfaces the
PARTIAL-PROMOTION recovery report; dedup the duplicate warn line and
note that source.image may already be partially advanced on Railway.
- check_p1_ghcr_digests emits REFUSE: P1 ... "no image" for an imageless
staging service (instead of a silent skip that surfaced later as a
misleading "internal error").
- check_p1_ghcr_digests per-service rescue broadens to StandardError so a
non-GHCR error (e.g. ArgumentError, network) does not bypass the rescue
and crash the loop, discarding earlier services' findings.
- pin_and_verify raises ArgumentError immediately if called with a
tag-form image (instead of 30s of futile retries + misleading error).
- pin_and_verify timestamp gate is non-vacuous: a non-nil observed
updatedAt is ALWAYS required, even when pre_update_ts is nil
(which previously collapsed the gate to digest-equality alone).
- run_staging_probe rescues Errno::ENOENT / StandardError around the
IO.popen launch so a missing npx produces a clean ok:false summary
instead of a raw stack trace bubbling out of P3.
Spec hygiene: drop the unused FakeGQL class in test_promote_execute.rb
(it referenced an uninitialized @after_image); give the unresolvable-tag
fixture a placeholder digest so it never builds a malformed "...@" ref;
test_promote_p2.rb tests now capture both streams and assert against the
combined output, matching the convention used elsewhere in the suite.
The showcase deploy model is STAGING = mutable :latest tag, PROD = immutable
@sha256: digest (P6 enforces both shapes). SnapshotCommand#build_snapshot
stored the raw serviceInstance.source.image, so for staging svc["image"] was
the :latest TAG. execute_promotion was pinning THAT mutable tag to prod via
serviceInstanceUpdate, defeating the immutable-prod invariant before
pin_and_verify raised on the nil expected_digest.
Fix: add PromoteCommand#resolved_prod_image — returns the staging svc as
@sha256:-pinned (pass-through if already pinned; resolves the tag via the
shared GHCR client otherwise; returns nil if the tag cannot be resolved).
execute_promotion now refuses (P0) rather than pin a mutable tag, and
check_p1_ghcr_digests verifies the resolved digest (it previously SKIPPED
tag-form images entirely, so :latest was never P1-checked).
Also:
- P2 race-check guards latest["meta"] when Railway returns a JSON String
(deserialized as Ruby String, not Hash) — .dig used to crash with
NoMethodError. SUCCESS status remains the real gate.
- Remove dead --include-startcommand flag (never read; doubly inert because
P6 REFUSEs on any startCommand divergence).
- Spec hygiene: P3 skip-test raises if probe runs under --no-require-staging-
green; P6 warn-proceed stubs execute_promotion to isolate the gate and
asserts rc==0; test_ghcr_token teardown unconditionally deletes
GITHUB_TOKEN/GHCR_TOKEN/RAILWAY_TOKEN before restoring priors.
70 runs, 204 assertions, 0 failures (up from 66/188 baseline).
P3 is the live re-probe gate spec §7.2 requires: CI history is not
authoritative for staging-green because showcase_deploy.yml uses
cancel-in-progress (the most recent CI run may have aborted before
the probe ran). Promote shells out to Workstream A's parameterized
verify-deploy.ts entrypoint at promote time and refuses on a red
result. Default-on; can be disabled with --no-require-staging-green
(prints "P3 SKIPPED" so the bypass is visible in logs).
run_staging_probe builds a clean child env (RAILWAY_TOKEN, GHCR_TOKEN,
GITHUB_TOKEN, PATH, HOME) and IO.popens `npx --yes tsx
showcase/scripts/verify-deploy.ts --env staging --services <csv>`.
Exit 0 = green, non-zero = red; the last 10 lines of stdout become
the human summary in the REFUSE message.
3 new P3 tests (red probe REFUSE, skip when flag off, green probe
pass). Also stubs run_staging_probe in the P1 and P2 test fakes so
those tests don't shell out to tsx (full suite stays sub-10ms).
Implements check_p2_staging_deployments via
RollbackCommand::DEPLOYMENTS_QUERY (input:{serviceId,environmentId})
against STAGING_ENV_ID. For each staging service we are promoting,
requires that the most recent staging deployment is SUCCESS and its
deployed image digest matches the digest we are about to promote.
A mismatch indicates an in-flight build that landed mid-promote;
refusing prevents racing a newer digest into prod.
Refuse messages:
- "no staging deployments found" when edges is empty.
- "latest staging deployment status is <STATUS>, not SUCCESS".
- "in-flight race - latest staging deployment is <NEW> but snapshot
has <OLD>. Re-snapshot and retry."
Also adjusts test_promote_p1 fakes: preflight checks accumulate
findings before any short-circuit, so P2 still issues its gql query
even on a P1 REFUSE. Replaces the raise-on-call FakeGQL with a
benign empty-deployments FakeGQLEmpty so P1 cases stay focused.
3 new P2 tests (FAILED status, digest race, clean SUCCESS).
Full suite: 50 runs, 141 assertions, 0 failures.
Jordan Ritter