Implements check_p2_staging_deployments via
RollbackCommand::DEPLOYMENTS_QUERY (input:{serviceId,environmentId})
against STAGING_ENV_ID. For each staging service we are promoting,
requires that the most recent staging deployment is SUCCESS and its
deployed image digest matches the digest we are about to promote.
A mismatch indicates an in-flight build that landed mid-promote;
refusing prevents racing a newer digest into prod.
Refuse messages:
- "no staging deployments found" when edges is empty.
- "latest staging deployment status is <STATUS>, not SUCCESS".
- "in-flight race - latest staging deployment is <NEW> but snapshot
has <OLD>. Re-snapshot and retry."
Also adjusts test_promote_p1 fakes: preflight checks accumulate
findings before any short-circuit, so P2 still issues its gql query
even on a P1 REFUSE. Replaces the raise-on-call FakeGQL with a
benign empty-deployments FakeGQLEmpty so P1 cases stay focused.
3 new P2 tests (FAILED status, digest race, clean SUCCESS).
Full suite: 50 runs, 141 assertions, 0 failures.
Splits the previously-monolithic PromoteCommand#run into:
- capture_snapshots: pulls staging + prod snapshots (test-injectable).
- run_with_preflight_only: runs the P1..P6 preconditions, prints the
mandatory "env var VALUES are not compared" NOTE, gates on
REFUSE/WARN findings, then defers to execute_promotion.
- execute_promotion: the actual pin+redeploy loop.
Adds the P1 GHCR digest existence gate: every staging-side image
(@sha256:...) is HEAD-checked via GHCR.manifest_exists before any
serviceInstanceUpdate is issued. Tri-state result drives explicit
REFUSE messages (:missing -> garbage-collected hint; :auth_failed ->
GHCR_TOKEN / GITHUB_TOKEN hint). P2/P3/P6 land as []-returning stubs
for later phases (D.2 / D.3 / D.5).
Also adds parser flags --confirm-divergence,
--require-staging-green / --no-require-staging-green and
default_options that flips require_staging_green default-on per
spec §7.2 P3.
3 new P1 tests (REFUSE on :missing, REFUSE on :auth_failed with the
expected hint, clean pass on :exists). Existing suite stays green.
Adds two foundational helpers for the promote-hardening P1 check:
- Railway::Auth.ghcr_token: resolves a GHCR bearer separately from the
Railway API token. Prefers GHCR_TOKEN, falls back to GITHUB_TOKEN
(CI workflow token with packages:read). Returns nil if neither is set
so callers can refuse rather than silently fall through to anonymous.
- Railway::GHCR#manifest_exists: digest-existence HEAD against
ghcr.io/v2/<repo>/manifests/<sha256:...>. Returns tri-state
(:exists/:missing/:auth_failed); raises on 5xx; raises ArgumentError
if caller passes an unpinned tag (programmer error — P1 verifies the
concrete bytes about to ship).
GHCR.new default token source switched from ENV["GHCR_TOKEN"] direct to
Railway::Auth.ghcr_token, which adds the GITHUB_TOKEN fallback. Audited
all GHCR.new callers: BaseCommand#ghcr uses the default (intended new
behavior); all existing spec callers pass explicit token: kwarg so they
are unaffected.
Tests: 4 ghcr_token cases + 5 manifest_exists cases.
bin/railway now derives EXPECTED_DOMAINS from
showcase/scripts/railway-envs.generated.json instead of maintaining a
parallel Ruby hash. The TS railway-envs.ts is canonical; CI guards drift
via `emit-railway-envs-json.ts --check`. Adds a Minitest parity test that
boots Ruby and asserts its derived EXPECTED_DOMAINS matches the SSOT
JSON (public hosts only, env-id keys match SSOT envIds). Refs spec §3a.
[BLITZ:A6]
The tool was written from a spec but never exercised against live Railway,
so several queries reference fields that do not exist in the public
schema. Live runs (including the CI lint-prod step) failed with errors
like `Cannot query field "domains" on type "Project"`. Unit tests passed
because they only covered parsing and IO, not the GraphQL shape.
Verified the live schema via introspection (2026-05) and corrected
every mismatch:
- Project has no `domains` field. The previous `customDomains: domains
{ customDomains { ... } }` block on the Project selection is gone.
Custom domains now come from `serviceInstance.domains.customDomains`
for the env we are inspecting.
- Service has no `serviceInstances` field. We can no longer enumerate
per-env instances by nesting under Service. The new flow is:
1. SERVICES_LIST_QUERY -> list services in the project
2. SERVICE_INSTANCE_QUERY -> per (service, env), fetch source,
startCommand, latestDeployment, domains
3. ENVIRONMENT_VARIABLES_QUERY -> all variables in the env, then
group keys by Variable.serviceId for per-service env_keys
- `serviceInstanceDeployV2` does not accept an `image` argument; its
signature is (commitSha, environmentId, serviceId). To pin a service
to a specific image we now use
serviceInstanceUpdate(input: { source: { image } })
followed by serviceInstanceRedeploy. RestoreCommand exposes a
pin_and_redeploy class method that PromoteCommand and PinCommand
reuse.
- `deploymentRollback` returns scalar Boolean, so the previous
`deploymentRollback(id: $id) { id }` was invalid GraphQL — selection
sets are not allowed on scalars. Dropped the selection set.
- Auth.token now reads `user.accessToken` from ~/.railway/config.json
first. The legacy `user.token` field is a short CLI session token
(4 chars on a fresh login) that does not authenticate against the
public GraphQL API and was producing silent "Not Authorized" errors.
Added spec/test_snapshot_graphql.rb with a FakeGQL that returns realistic
shapes, plus regression guards that fail the build if anyone reintroduces
`Project.domains`, `Service.serviceInstances`, `serviceInstanceDeployV2`
with an image arg, or a selection set on `deploymentRollback`.
Smoke-tested live (read-only) against the showcase project:
- lint-prod: "OK: all production services digest-pinned." (27 services)
- snapshot --env staging: full YAML with images, startCommands, env_keys
- env-diff staging production: 32 drift findings (expected, staging is
not digest-pinned)
- resolve-digest ghcr.io/copilotkit/showcase-aimock:latest: digest
returned successfully
No mutating subcommands (restore, rollback, promote, pin) were run live.
Make the lint-prod audit result legible without having to click into the
workflow logs. Two surfaces, both rendered from the same JSON payload:
1. `$GITHUB_STEP_SUMMARY` — structured markdown block at the top of every
workflow run page. Shows on every event (push, pull_request,
workflow_dispatch).
2. Sticky PR comment — one comment per PR, keyed by the HTML marker
`<!-- lint-prod-sticky-comment -->`. Re-runs update the same comment via
`gh api -X PATCH` instead of creating duplicates. Plain `gh` CLI only,
no third-party action.
Both surfaces show: one-line status, a table of the unpinned services only
(not all 27), and a Pacific-time run timestamp with the finding count.
To support the renderer, add `--format json` to `lint-prod`:
{services:[{name,source,status}], findings:N, timestamp:"ISO8601"}
The workflow consumes this shape and also writes `findings` to
`$GITHUB_OUTPUT` so downstream jobs (future Slack alert) can compare runs.
Idempotent: re-running the workflow finds the existing comment by marker
and PATCHes it — never duplicates.
Add --exit-zero flag to bin/railway lint-prod that makes the command exit 0
even when digest-pinning findings exist. Findings still print to stdout so
they remain visible in the CI step log.
Wire the showcase_lint_prod.yml workflow to pass --exit-zero so the job
cannot block PRs while we soak the check against real production state.
Once we have confidence the findings are clean, remove --exit-zero from
the workflow step to flip lint-prod to enforcing.
Updates README to document the advisory-mode behavior and the path to
flipping the check to enforcing.
PR #4449 removed the eval tier system while unaware #4448 had just
merged. Restoring orchestrator, matrix, scope, config, GHA workflow,
and CLI wiring with all 8 CR fixes intact.
Jordan Ritter