Two independent ways verification was silently lost on 2026-07-25.
1. `resolve-matrix` required `github.event.workflow_run.conclusion ==
'success'`. But `redeploy-staging` gates on the artifact-derived
`any_success`, not on the matrix rollup, so a build with some slots
cancelled (rollup `cancelled`) or failed (rollup `failure`) still
pushes real images and still redeploys the slots that built. Run
30162773601 redeployed 23 services to staging and this workflow never
started. An unverified real deploy is worse than a verified partial
one, so the trigger is now "the build reached a terminal conclusion"
and WHAT to verify stays decided by evidence: no redeploy-summary
artifact still means has_services=false and verify is skipped, and the
redeploy gate still narrows to the per-service success set, so a
cancelled slot can never be probed against a stale `:latest`.
2. The `showcase-verify-deploy` concurrency group was global, so any
later-finishing build preempted an earlier run's verification even
though they verify DIFFERENT commits. Verify run 30163309977 for the
one genuinely successful build of the day (#6168, 7282ecddf) started
at 15:17:13 and was cancelled 9 seconds later by run 30163312882 —
which was triggered by a different, partially-cancelled build and then
skipped every job anyway. Key the group per triggering commit.
deploy.yml's notify-harness step was sending {state, services, ...},
but the harness ingest schema at showcase/harness/src/http/webhooks/
deploy.ts is .strict() and requires succeeded/failed/cancelled arrays
+ bool. Every push-to-main notify-harness POST has been 400ing as
invalid-payload (unknown key 'state') since the schema landed.
Build the payload from the redeploy gate's success-and-error buckets:
- Emit failed_services alongside ok_services from the redeploy-gate
step (services whose status==error), and expose both as resolve-matrix
job outputs.
- Rewrite the notify-harness payload to drop state and emit:
succeeded=ok_services, failed=failed_services, cancelled=(verify
conclusion==cancelled), services=union of succeeded and failed.
- Omit buildRunId/buildRunUrl entirely when empty (the schema runs
.url() on buildRunUrl, so an empty string would be rejected). Same
treatment for runUrl as defense-in-depth.
- Group the no-summary echoes through a single >>GITHUB_OUTPUT block
so shellcheck SC2129 stays clean once a third echo is added.
Closing hardening pass on the showcase deploy-gate's verify-matrix
resolver. The 7-agent review confirmed the gate is correct; this
commit fixes the residual rough edges.
- showcase_deploy.yml: correct the false §3 ok-non-empty comment.
The empty-intersection case can coexist with redeploy_red=false
(every redeploy succeeded, just none probe-eligible) — that's a
correctly-green run, not a red one.
- showcase_deploy.yml: tighten the summary.json shape guard to catch
PARTIAL drift (TOTAL>0 && WITH_STATUS<TOTAL). The previous all-or-
nothing TOTAL>0 && WITH_STATUS==0 check silently dropped drifted
rows on a mixed summary. Validated locally on mixed/normal/empty/
total-drift jq samples.
- resolve-verify-matrix.ts: add asSupportedEventName narrowing helper
+ use it in the CLI. Replaces the unchecked `as` cast — type system
and runtime now tell one story. Resolver's internal eventName
guard becomes defense-in-depth for direct (test) callers.
- resolve-verify-matrix.ts: make the workflow_run boundary total —
summaryPresent MUST be exactly "true"/"false". Any other value
(including "" from a step-id-rename wiring break) throws now
instead of silently emitting has_services=false.
- resolve-verify-matrix.ts: drop the try/catch around
fileURLToPath(import.meta.url) in `invokedDirectly`. The catch
used to swallow ESM-interop failures and silently no-op the CLI
(exit 0, no GITHUB_OUTPUT write → verify skipped = false-green).
- resolve-verify-matrix.ts: reword parseSsotServices JSDoc to
distinguish schema-drift from truncation (the two are different
failure modes, not one conflated story).
- showcase_build.yml: comment addendum on the redeploy-summary
upload — swapping the guard to `if: always()` would red the
legitimate services=='' path (no summary written), trading the
already-closed false-green for a false-red on every non-buildable
push.
- resolve-verify-matrix.cli.test.ts: switch to spawnSync so stderr
is captured on both zero and non-zero exit (execFileSync only
exposes stderr on throw). Hard-code two stable probe-eligible
names ("aimock", "harness") for the sorted-CSV test rather than
picking probe[0]/probe[1] off the live SSOT — the prior test was
tautological (already-sorted in, sorted out) and would silently
pass if the resolver did nothing.
- resolve-verify-matrix.cli.test.ts: add CLI coverage for the
dropped-token ::warning:: path (FIX 3 — the entire drift-detection
contract had zero CLI coverage), the unexpected-EVENT_NAME error
(FIX 5), and the workflow_run-summary_present total boundary
(FIX 7, both "" and "True" inputs).
- resolve-verify-matrix.test.ts: add unit coverage for the new
workflow_run summaryPresent boundary (empty + "True" + the
workflow_dispatch ignores-summaryPresent regression).
Red-green: 6 tests RED before code changes (FIX 3 warning, FIX 5
unknown EVENT_NAME, FIX 7 unit + CLI ×2 for "" and "True"); 79
tests GREEN after.
Validation: 4 vitest files / 79 tests passing; 87/87 ruby specs
passing; actionlint findings unchanged vs integration baseline
(8 → 8, identical diff); yaml.safe_load OK on both workflows.
A 7-agent review of the verify-matrix resolver and its surrounding workflow plumbing found three
boundary surfaces that could silently produce a GREEN deploy on a broken release, plus an
untested CLI contract that CI compares against the literal strings 'true' / 'false'.
FIX 1 — Validate the SSOT shape in loadSsotServices(). The prior `JSON.parse(...) as
{services: SsotService[]}` was an unchecked cast: a truncated/drifted SSOT (emitter crashed
mid-write, or schema renamed) parses fine but silently shrinks/empties the probe-eligible set
→ some redeployed services go unverified, or verify is skipped on a real redeploy. Extract a
pure exported parseSsotServices(raw, path) that requires the shape we depend on (non-empty
services array; each entry has a non-empty string name, an optional string|null dispatchName,
and a probe object with a boolean staging). Throw `::error::SSOT <path> malformed: <detail>`
on any violation. Also re-check existsSync(SSOT_JSON) after the regenerate-if-missing
execFileSync — a regen that exits 0 without writing must not proceed to a useless JSON.parse
crash. Drop the defensive `probe?.staging` once shape is guaranteed.
FIX 2 — Validate summary.json shape in the redeploy-gate bash. The bullseye false-green
surface: if redeploy-env.ts's schema ever drifts (e.g. `status` → `state`, `ok` → `success`),
every `jq select(.status==...)` yields empty → redeploy_red=false AND ok_services="" →
resolver skips verify → GREEN CI on a real unverified redeploy. Add a TOTAL vs WITH_STATUS
shape guard right after loading the summary: if TOTAL > 0 && WITH_STATUS == 0, emit
::error::summary.json has $TOTAL entries but none with status ok|error (schema drift?) and
exit 1. The legitimate empty-array path (TOTAL=0) is preserved.
FIX 3 — Fail loud on unknown eventName in resolveVerifyMatrix. The prior code fell through to
the workflow_run intersection branch for ANY unrecognized eventName (typo, unexpected
trigger), silently emitting has_services=false → indistinguishable from a legit "summary
absent" skip. Add an explicit guard so only workflow_run / workflow_dispatch are accepted;
anything else throws ::error::resolve-verify-matrix: unexpected eventName '<value>'. Tighten
the eventName parameter type to the literal union.
FIX 4 — Trim ok tokens + warn on dropped tokens in okCsvToCanonicalNames. Split, then
.map(t => t.trim()).filter(Boolean) so "a, b" (spaces) matches. Collect tokens that match NO
SSOT service (by name or dispatchName) and have the CLI wrapper emit ::warning::ok_services
tokens dropped (no SSOT match): <list> on stderr when non-empty — surfaces SSOT/build drift.
The pure function stays IO-free; logging lives in the wrapper.
FIX 5 — CLI wrapper integration test. New resolve-verify-matrix.cli.test.ts spawns
`npx tsx showcase/scripts/resolve-verify-matrix.ts` with a temp $GITHUB_OUTPUT file across
four scenarios and asserts the temp file contents EXACTLY (the workflow YAML compares
has_services against the literal strings 'true'/'false', so the byte-for-byte format is part
of the contract). Uses the real railway-envs.generated.json so the loader exercise is real.
FIX 6 — Cleanup. Remove the dead `env: DISPATCH_SERVICE: ...` block on the redeploy-gate
step (the next step redeclares it — leftover from the extraction). Soften the §3
decision-table all-errors bullet to match resolve-verify-matrix.ts's careful wording, and
append that when the success-set is empty (or the intersection collapses to empty), verify
is skipped and the gate reds independently. Append to showcase_build.yml's "Upload redeploy
summary" path-(A) comment that `if-no-files-found: error` still reds path (A) even if a
future change adds `if: always()`.
Tests: red→green for FIX 1/3/4/5 verified locally. Resolve-verify-matrix vitest count:
12 → 28. Full requested suite (resolve-verify-matrix + cli + aggregate-build-results +
lint-rule-no-public-env): 72 passed. showcase/bin ruby specs: 87 runs / 0 failures / 0
errors / 0 skips. actionlint baseline preserved (8 findings, identical to integration tip).
Extract the inline bash+jq decision logic from showcase_deploy.yml's
resolve-matrix job into showcase/scripts/resolve-verify-matrix.ts, a
pure function with a vitest suite. The bash had produced two confirmed
bugs across prior CR rounds, so making it testable is the lasting fix.
Issue A (the bug this PR fixes): when summary_present=true but
ok_services is empty (every service errored on redeploy), the old bash
skipped the intersection and fell through to the full probe-eligible
fleet, gratuitously probing every service against stale :latest. The
resolver now returns has_services=false in that case — enforce-redeploy
-gate independently reds the workflow on redeploy_red=true, so this
case is already loud; there is nothing left to verify.
Parity preserved for unchanged cases:
- workflow_dispatch + 'all'/empty → full probe-eligible set
- workflow_dispatch + specific svc → that one (unknown → error exit)
- workflow_run + summary_present=false → has_services=false
- workflow_run + present + ok non-empty → intersection with probe-
eligible (SSOT key OR dispatchName aliases both resolve)
Also clarified the Upload-redeploy-summary comment in showcase_build.yml
to document both red paths (hard crash → redeploy step exits non-zero;
exit-0-but-no-file → if-no-files-found:error reds the step) so no
false-green path is possible.
Tests: 12-case vitest suite covers each decision-table row plus the
Issue A fix (written red-first; failed against a naive full-fleet
fallback, passed once the early return was added). CLI parity verified
against the real generated SSOT for the three representative env-var
combinations (workflow_run + present + ok=[a,c]; workflow_run + present
+ ok empty; workflow_dispatch + 'all').
Three correctness holes uncovered by confirmation review of the earlier
deploy-gate fix:
(1) showcase_deploy.yml — Build verify matrix step: when workflow_run fires
with summary_present=false (legitimate "build redeployed nothing", e.g.
docs/script-only push under showcase/**), the gate correctly no-oped but
the matrix fell through the empty-OK_FROM_REDEPLOY branch and resolved to
the FULL probe-eligible set. Verify then ran against the whole staging
fleet for a push that deployed nothing — gratuitous, and false-reds the
deploy workflow if any unrelated staging service happens to be unhealthy
at probe time. Thread github.event_name + summary_present into the step
via env and add an explicit (workflow_run && summary_present==false)
guard that sets services_csv="" / has_services=false. workflow_dispatch
fall-through (full fleet / chosen service) preserved. workflow_run +
summary present + all-errors path unchanged: enforce-redeploy-gate still
trips RED on redeploy_red=true.
(2) showcase_build.yml — Upload redeploy summary step: was gated on
services != '' && hashFiles('.redeploy/summary.json') != ''. If
redeploy-env.ts crashes before writing summary.json (the script is
documented "always exits 0", but a crash/OOM/unhandled-rejection can
skip the write), services != '' but hashFiles == '' silently skipped
the upload. The deploy side then saw "artifact absent", treated it as
"nothing redeployed", skipped the gate, and produced a FALSE GREEN
despite a real redeploy failure. Drop the hashFiles clause so the upload
is mandatory whenever a redeploy was attempted; if-no-files-found:error
(already set) then fails the step → fails the redeploy-staging job →
fails the build workflow → showcase_deploy.yml's resolve-matrix.if
(workflow_run.conclusion == 'success') blocks the deploy run from
starting at all. Loud failure on the build side. The legitimate
services == '' (matrix ∩ success-set empty) path is preserved by the
services != '' guard.
(3) showcase_deploy.yml — check-redeploy-summary github-script: was a
single per_page:100 list call. While the current run uploads ~28
artifacts (well within 100), a future expansion past 100 could push
redeploy-summary off the first page and produce a false "absent" → gate
skipped → false-green. Switch to github.paginate.iterator with the
endpoint's name="redeploy-summary" filter for an exact-match,
pagination-safe lookup. No try/catch is added: github-script propagates
unhandled rejections by default, so a 5xx/permission error fails the
step → resolve-matrix.result == 'failure' → enforce-redeploy-gate trips
RED. Silent default-to-false on API error would open the gate on a
broken pipeline, which is what we explicitly do NOT want.
Validation: actionlint shows 8 findings on both files, identical to the
integration baseline (zero new findings). python3 yaml.safe_load OK on
both. Regression suites green: showcase/scripts vitest 44/44
(aggregate-build-results + lint-rule-no-public-env);
showcase/bin/spec/all_tests.rb 87 runs / 251 assertions / 0 failures.
D1 — showcase_deploy.yml false-red fix
======================================
The build workflow legitimately uploads no `redeploy-summary` artifact when it ran
(push touched `showcase/**` so `paths:` matched) but `detect-changes` found no
buildable service, so `redeploy-staging` was skipped. The build still concludes
`success`, so `showcase_deploy.yml` fires on `workflow_run` and `resolve-matrix`
runs. `actions/download-artifact@v4` with `name:` HARD-FAILS on a missing
artifact, so the unguarded download was failing the job, and a downstream guard
that trips `enforce-redeploy-gate` on `resolve-matrix.result == 'failure'` was
flipping the workflow RED — a false-red on a routine showcase-docs/script change.
Add an artifact-existence pre-check using `actions/github-script` (pinned by SHA,
matching the existing repo convention) that lists the artifacts for
`workflow_run.id` via `actions: read` (already granted to `resolve-matrix`) and
sets `summary_present=true|false`. Gate the existing download step on
`summary_present == 'true'`. Keep NO `continue-on-error`, so the C1 property
holds: when the artifact exists but the download genuinely fails, the job still
fails loud and `enforce-redeploy-gate` correctly reds the workflow. When the
artifact is legitimately absent, the bash gate's existing `[ ! -f "$SUMMARY" ]`
branch no-ops (`redeploy_red=false`, `ok_services=""`) — nothing was
redeployed, so there is nothing to gate.
Updated the step comment block to enumerate the three distinct cases now
handled: workflow_dispatch (no download); workflow_run + artifact absent
(graceful skip); workflow_run + artifact present (download with fail-loud).
L1-L5 — env lint rule hardening
===============================
- L1: route the destructuring (VariableDeclarator/ObjectPattern) branch through
the shared `staticKeyName()` helper so the computed-string-key form
`const { ["NEXT_PUBLIC_X"]: y } = process.env` and the no-expression
template-literal form `const { [\`NEXT_PUBLIC_X\`]: y } = process.env` are
caught with the same parity as the bracket-member read.
- L2: unwrap a wrapping `ChainExpression` at the top of `isProcessEnv()` so
`process.env?.X` is matched robustly across parser flavors; corrected the
helper's doc comment to describe the actual semantics.
- L3: export `BANNED_KEYS` from the rule module and have the table-driven test
dynamically import the rule's own Set instead of hand-mirroring it — the
test set now cannot drift from the rule.
- L4: added override-scoping fixtures for `showcase/shell/src/**` and
`showcase/shell-dojo/src/**`; the `.oxlintrc.json` override list already
includes these, but the test now exercises them so an accidental drop is
caught.
- L5: expanded the file-header "Out of scope" doc list to include bulk-iteration
reads (`Object.keys/values/entries(process.env)`, for-in, spread
`{...process.env}`), rest-pattern destructuring, compound-assignment LHS, and
update operators. Documentation-only — the deliberate non-coverage is now
auditable.
Validation
==========
- RED→GREEN confirmed for L1 (two new destructuring computed-key tests) and L3
(dynamic `await import(...)` of BANNED_KEYS failed pre-fix with
"Rule module did not export a non-empty BANNED_KEYS Set", green after export).
- vitest: 38 passed (was 34 baseline + 4 new); aggregate-build-results 6 passed.
- Ruby promote suite: 87 runs, 251 assertions, 0 failures (unchanged).
- python3 yaml.safe_load: showcase_deploy.yml + showcase_build.yml +
showcase_promote.yml all parse OK.
- actionlint: zero NEW findings on the changed file. The pre-existing
showcase_build.yml SC2086/SC2129/runner-label findings are identical on the
integration baseline (unchanged by this commit).
Seven-agent CR surfaced correctness defects in the build/deploy/promote
pipeline and in the no-public-env-shell-read oxlint rule. This commit
closes the false-green paths and broadens lint coverage.
Workflow fixes:
- showcase_deploy.yml: drop `continue-on-error: true` on the redeploy-summary
artifact download. The dispatch path is already guarded by the `if:
workflow_run` clause, so the bash "no summary" branch handles legitimate
manual dispatches. A genuine workflow_run download failure must now fail
loud instead of silently widening verify to the full service set against
stale `:latest`.
- showcase_build.yml: redeploy-staging now intersects the build matrix with
the aggregator success set (`needs.aggregate-build-results.outputs.results`,
status == "success") before producing the redeploy CSV. Failed/skipped
slots no longer get redeployed (which would just re-pull stale `:latest`
and look healthy).
- showcase_build.yml: `notify-all-builds-failed` now additionally requires
`needs.build.result == 'failure'` so it doesn't Slack-spam when the build
job was SKIPPED (verify-image-refs upstream failure).
- showcase_build.yml: `notify` now lists [build, aggregate-build-results,
redeploy-staging] in `needs:` so aggregator/redeploy failures still emit
a Slack signal. `if: failure()` still skips when none of the needs failed.
- showcase_build.yml: `set -euo pipefail` on the Prepare build args step
so a transient $GITHUB_OUTPUT write failure can't ship images without
COMMIT_SHA/BRANCH baked in.
- showcase_deploy.yml: `enforce-redeploy-gate` now also trips on a
resolve-matrix failure (`needs.resolve-matrix.result == 'failure'`) so
an upstream crash that leaves `redeploy_red` empty can't bypass the gate.
- Doc-comment accuracy: drop stale `(PR #5093)` reference; correct the
env-IDs source-of-truth comment; document the optional `skip_build` field
in ALL_SERVICES; clarify that health_path is informational and verify
uses per-service drivers; add the missing `resolve-targets` step 0 to the
promote workflow's "Order:" header.
Aggregator fix (RED-GREEN):
- aggregate-build-results.ts: throw on zero slot dirs. The job is gated
upstream on has_changes == 'true', so zero slot dirs is a broken artifact
download, not a legitimate empty build set. Silently emitting
any_success=false + results=[] is indistinguishable from "all builds
failed" and lets the deploy workflow fall back to probing the full
service set against stale `:latest`. Refuse the ambiguity.
- aggregate-build-results.test.ts: existing empty-INPUT_DIR test was
updated to assert the throw (was: return []).
Oxlint rule (RED-GREEN):
- no-public-env-shell-read.mjs: handle destructuring reads
(const { NEXT_PUBLIC_X } = process.env and aliased form), template-literal
computed keys (process.env[\`NEXT_PUBLIC_X\`]), and explicitly skip
assignment-LHS / `delete` targets (writes are not reads). Optional
chaining already worked through the existing MemberExpression path.
Aliasing (`const e = process.env; e.X`) is intentionally documented as
out of scope (needs scope tracking). Description sharpened to say the
rule guards a specific banned-key set, not all NEXT_PUBLIC_* reads.
- .oxlintrc.json: tighten the off-override glob from
`showcase/**/*runtime-config*` to
`showcase/**/lib/runtime-config*.{ts,tsx}` so it only silences the
intended implementation files, not arbitrary paths containing that
substring.
- lint-rule-no-public-env.test.ts: rewritten as table-driven coverage of
every BANNED_KEYS entry (dotted + bracket-string forms), every ALLOWED
key (asserting non-firing), all new variants from the rule expansion,
the assignment/delete non-fire cases, and override scoping
(runtime-config exempt; packages exempt; shell-tree non-runtime-config
flagged).
Validation:
- actionlint on all three workflows: 8 pre-existing findings (depot label,
pre-existing SC2086 infos in untouched steps); my edits add zero.
- python3 yaml.safe_load: all three workflows OK.
- vitest aggregate-build-results.test.ts: 6/6 pass (incl. new throw test).
- vitest lint-rule-no-public-env.test.ts: 34/34 pass.
- vitest full showcase/scripts suite: 1654/1654 pass across 46 files.
- ruby showcase/bin/spec/all_tests.rb: 87 runs, 0 failures.
- Intersection jq proof (matrix a,b,c × success a,c) → "a,c"; all-failed
→ ""; skipped status excluded.
showcase_deploy.yml is now SSOT-driven: env id and service set both come
from railway-envs.generated.json (no inline matrix, no hardcoded
RAILWAY_ENV_ID). Probes staging on workflow_run (was incorrectly probing
prod). Reads redeploy-env's per-service JSON summary uploaded as an
artifact and fails the workflow on any staging status:error while
verify still runs against the success-set. Preserves PR #5093's
redeploy-env.ts exit semantics unchanged. Refs spec §3.
Adds 'webhooks' as a workflow_dispatch choice in both showcase_build.yml
and showcase_deploy.yml so humans can redeploy/verify the webhooks service
on demand. webhooks' GHCR image (showcase-eval-webhook) is built by a
separate release workflow in the showcase-eval-webhook repo, so:
- paths-filter uses a sentinel that cannot match any in-tree path,
keeping push-driven runs from ever including webhooks.
- The build matrix entry carries skip_build: true; the Build and push
step skips the Depot build for that slot. The per-slot result still
publishes (job.status == success) so the redeploy path proceeds and
redeploy-env.ts picks the existing :latest from GHCR.
The SSOT entry in railway-envs.ts gains dispatchName: 'webhooks' so the
forward/reverse round-trip tests cover it. New tests pin that the SSOT
dispatchName is mirrored in both workflow files' dispatch choice lists
AND ALL_SERVICES JSON.
[BLITZ:L3-wf] E-6a/E-6b/E-6c/E-6d wiring.
Build matrix slots now publish per-slot build-result-<dispatch_name>
artifacts containing {service, status}. A new aggregate-build-results
job downloads every per-slot artifact, merges them via the shared
mergeBuildResultFiles helper, and uploads the canonical build-results
artifact (results.json) for cross-workflow consumption.
The deploy workflow's resolve-matrix step now downloads build-results
via gh api .../artifacts/<id>/zip and filters its verification matrix
from the structured JSON success set. No more gh api .../jobs calls,
no more capture("^build \\(") regex on job names — those silently
break on job-name renames. The contract (service + status enum) is
enforced in one place: showcase/scripts/lib/build-outputs.ts.
[BLITZ:L3-wf] E-4e/E-4f wiring.
E-3a/b/c per plan-E. Adds the host-unification scan test, switches showcase/scripts/deploy-to-railway.ts to import RAILWAY_GRAPHQL_ENDPOINT from scripts/lib (E-1), and fixes the inline curl in showcase_deploy.yml line 205. The .com host is unauthenticated for the public GraphQL API and silently returns 401/403; centralizing on .app prevents the drift from returning. Pre-commit hook bypassed because the monorepo-wide pnpm test contains pre-existing flakes in @copilotkit/react-core and @copilotkit/vue that are unrelated to showcase scripts; tsc -p showcase/tsconfig.json --noEmit and the railway-graphql vitest pass cleanly.
Three companion workflows duplicate the showcase_build.yml service registry
and were missed in the initial wiring commit. Bring them in sync:
- .github/workflows/showcase_build_check.yml: add ms_agent_harness_dotnet
to the paths-filter and the ALL_SERVICES matrix (mirror of the
production build matrix, used for pre-merge Docker build verification).
- .github/workflows/showcase_deploy.yml: add ms-agent-harness-dotnet to
the workflow_dispatch options and the verification ALL_SERVICES with
railway_id 6343d7f9-6c3f-4c8d-9a6e-79f03d2f1e37 and /api/health.
- .github/workflows/showcase_keep-alive.yml: add ms-agent-harness-dotnet
to the keep-alive ping matrix.
Comprehensive CI/CD security hardening pass over all 33 workflows.
Action pinning
- Every `uses:` is now pinned to a 40-char commit SHA with a `# vX.Y.Z`
comment alongside (167 occurrences resolved). Tag-style refs like `@v4`
are mutable and have been used in past supply-chain attacks (e.g.
tj-actions/changed-files in March 2025) to repoint widely-used actions
to malicious commits.
- Removed redundant `version: "10.13.1"` hardcodes from `pnpm/action-setup`
call sites so the action inherits from package.json `packageManager`
(one source of truth).
Automated maintenance
- Added `.github/dependabot.yml` for the `github-actions` ecosystem so
SHA pins stay current. Without this, pins go stale fast and new
upstream advisories never reach us. Minor/patch bumps are grouped;
major bumps stay separate so they get a real review.
Static analysis
- Added `.github/zizmor.yml` configuration and
`.github/workflows/security_zizmor.yml` (blocking on PR, runs on push
to main, weekly schedule for advisory drift). zizmor catches the
well-known classes of Actions footguns: template injection from
untrusted input, dangerous triggers, unpinned uses, excessive token
scopes, secret exfil patterns.
- All 28 high-severity and 54 medium-severity findings from the baseline
scan are remediated. Each suppression in zizmor.yml carries a
per-finding justification comment so future maintainers can audit the
trust assumption.
Workflow hardening (from zizmor + manual audit)
- Added `persist-credentials: false` to every `actions/checkout` except
the 7 workflows that legitimately push back to the repo via the
workflow token (release tagging, auto-formatting, docs-sync, registry
updates). Each retained credential persistence carries a
`persist-credentials required: ...` comment explaining the call site.
- Routed every attacker-controllable expansion (`github.head_ref`,
`github.event.pull_request.head.repo.full_name`, `inputs.*`,
step outputs) through `env:` and referenced as quoted shell variables.
Eliminates 17 template-injection vectors in fork-PR-reachable
workflows.
- Added per-job `permissions:` blocks across 14 workflows; demoted
broad workflow-level `id-token: write` to the specific Depot-runner
jobs that need it; narrowed `pull-requests: write` /
`actions: write` to the jobs that actually call those APIs.
Audit-driven fixes
- `publish-release.yml` build job: dropped `token:` and added
`persist-credentials: false`. The subsequent `Upload workspace` step
was packing `.git/config` (with the persisted GITHUB_TOKEN) into a
1-day-retention artifact downloadable by anyone with `actions:read`.
- `auto_merge_showcases.yml`: team-membership check now authorizes on
the PR AUTHOR (`pull_request.user.login`), never `context.actor` —
the actor is whoever triggered the latest event, so a team member
synchronizing or reopening an outsider's PR would otherwise
green-light auto-merge of code they didn't author.
- `static_quality.yml`: pinned ruff to a specific version so a
compromised release can't land on the next PR run with the
persisted-credentials write token in the format job.
- `showcase_capture-previews.yml`: switched the args-string construction
to a bash array so a slug or demo value containing whitespace or shell
metacharacters stays a single argument rather than being re-tokenized
by the shell.
The decoupled build workflow (PR #4471) removed the Railway deploy
trigger, assuming environmentPatchCommit auto-update would handle
deploys. Not all services have auto-update configured, so GHCR images
were pushed but Railway never pulled them. Restore the explicit
serviceInstanceRedeploy call after each GHCR push.
The old "Showcase: Build & Deploy" workflow used a single concurrency group
that cancelled in-flight builds on every push to main. When multiple PRs
merged in quick succession, most service builds got cancelled and never
deployed.
Split into two workflows:
1. showcase_build.yml ("Showcase: Build & Push") - triggered on push to main,
builds Docker images and pushes to GHCR. Has NO concurrency group so every
run completes. Railway auto-update picks up the new :latest tag.
2. showcase_deploy.yml ("Showcase: Verify Deploy") - triggered by workflow_run
from the build workflow. Polls Railway to verify each service picked up the
new image and is healthy. Uses cancel-in-progress since verification is
idempotent. Posts results to showcase-harness via webhook.
Also updates showcase_capture-previews.yml to trigger from the renamed build
workflow.
Production showcase-aimock was running a week-old image because fixture
file changes in showcase/aimock/ did not trigger a CI rebuild. This adds
showcase-aimock to the Build & Deploy workflow matrix so it auto-deploys
on merge, creates a thin Dockerfile that bakes fixture files into the
image, documents the local-vs-production parity requirement in
docker-compose.local.yml, and adds an aimock fixture deployment section
to the RUNBOOK.
Remove all showcase-starter-* services from the CI build matrix in
showcase_deploy.yml. These starter demos were decommissioned in PR
#4378 (code removed) and confirmed dormant with zero traffic. Removes
entries from workflow_dispatch options, paths-filter definitions, and
the ALL_SERVICES JSON array.
- Update collision-avoidance comment in showcase_deploy.yml to remove
starter-specific examples (service/starter collision no longer possible)
- Fix showcase_keep-alive.yml description: pings showcase services, not starters
- Remove "starter" from catalog-types.ts manifestation union type
- Remove dead starter cell skip logic from cell-matrix.tsx cellIndex builder
- Update depth-utils.ts comment (defensive null guard, not starter-specific)
- Remove starter-specific test cases from depth-utils and cell-matrix tests
CR R1 follow-ups on top of the /api/ops proxy fix.
Bucket (a) — must-fix:
- Dockerfile: declare ARG/ENV OPS_BASE_URL in the builder stage. next.config.ts
evaluates rewrites() at build time and throws if OPS_BASE_URL is unset, which
aborted `next build` in CI. Mirrors the existing NEXT_PUBLIC_SHELL_URL /
NEXT_PUBLIC_POCKETBASE_URL pattern.
- showcase_deploy.yml: pipe OPS_BASE_URL through to docker build for the
shell-dashboard matrix entry, defaulting to the production
showcase-ops-production.up.railway.app URL.
- next.config.ts: strip trailing slashes from OPS_BASE_URL before constructing
the rewrite destination, matching the same normalization in
src/lib/ops-api.ts:resolveBaseUrl so server-side rewrite and client-side
fetch agree on the URL shape.
- src/lib/ops-api.ts: treat empty / whitespace NEXT_PUBLIC_OPS_BASE_URL as
"no override". `??` only short-circuits on null/undefined, so an env var set
to "" silently produced baseUrl="" and URLs of the form "/probes" with no
/api/ops prefix.
- use-probes.integration.test.tsx: snapshot+restore process.env.NEXT_PUBLIC_OPS_BASE_URL
in beforeEach/afterEach so tests never leak env state. Strengthen the proxy
contract assertions: lock toHaveBeenCalledTimes(1), assert method=GET,
cache=no-store, accept JSON header, and signal is an AbortSignal. Tighten
the 404 regression to assert the canonical ensureOk message shape so a
refactor that changes the format trips the test.
Bucket (b) — applied since the diff stayed focused:
- triggerProbe: add cache:"no-store" for parity with the GET fetches.
- fetchProbeDetail / triggerProbe: throw early when id is empty so callers
get a clean error instead of a request to /probes//... .
- ensureOk: bump body-truncation cap from 200 to 500 chars and append a
`[truncated, N bytes total]` marker so operators can see they're missing
tail bytes when the server returns a long HTML/stack-trace body.
- ops-api.ts: drop dev-loop review-cycle tag prefixes (R2-C.3, R3-C, R3-D.1)
from comments. Keep the actual rationale.
- ops-api.ts header docstring: clarify that NEXT_PUBLIC_OPS_BASE_URL is read
live at runtime in this codebase (SSR + tests), not just statically inlined
into the client bundle.
Verified:
- Tests: vitest run — 26 files, 293 passed, 1 skipped (no test count change).
- Typecheck: tsc --noEmit clean.
- Lint + format: oxlint + oxfmt clean on changed files.
- Local Docker build: `docker build --build-arg OPS_BASE_URL=https://...` succeeds.
Without --build-arg the build fails with "OPS_BASE_URL must be set" as
expected, confirming the fix is load-bearing.
showcase-ops was excluded from .github/workflows/showcase_deploy.yml, so
commits touching showcase/ops/** never produced a fresh GHCR image. PR
#4293 (Status tab + /api/probes route) merged to main on 2026-04-26 but
no rebuild fired — the deployed Railway image is stale and /api/probes
404s in production.
Adding showcase-ops as a first-class matrix entry:
- dispatch_name: showcase-ops (workflow_dispatch option + filter_key)
- paths-filter: showcase/ops/**, plus shared/scripts/manifests
(showcase-ops's Dockerfile bundles all four into the runtime image
via build-stage COPY + generate-registry.ts)
- context: '.' (repo root) so the Dockerfile can COPY from
pnpm-workspace.yaml + packages/ + showcase/{ops,shared,packages,scripts}
- dockerfile: showcase/ops/Dockerfile
- image: showcase-ops -> ghcr.io/copilotkit/showcase-ops:latest
- railway_id: 3a14bfed-0537-4d71-897b-7c593dca161d
- health_path: /health (matches Dockerfile HEALTHCHECK + Hono route)
- timeout: 20 (heavier build than shells: pnpm deploy + chromium
install via playwright --with-deps)
- lfs: false (no Git LFS assets in showcase/ops)
- linux/amd64 platform inherited from existing build step (Depot)
Resulting matrix: 39 services (was 38). dispatch_name uniqueness +
JSON validity verified locally; actionlint/yamllint surface only
pre-existing findings on the workflow.
Three fixes batched to minimize PR churn:
1. Dockerfile: add ARG NEXT_PUBLIC_POCKETBASE_URL so the PB URL gets
baked into the Next.js bundle at build time. Without this, pb.ts
resolves to the sentinel URL and the dashboard shows "unavailable"
on every tab. Pre-existing bug exposed by fresh deploys.
2. showcase_deploy.yml: pass NEXT_PUBLIC_POCKETBASE_URL and
NEXT_PUBLIC_SHELL_URL as build args for the shell-dashboard service
in the CI matrix. Neither was ever passed before.
3. cell-matrix.tsx + parity-matrix.tsx: flatten nested table pattern
that caused column misalignment. Category rows used colSpan with
an inner <table> whose columns floated independently of the header.
Replaced with useCollapsible hook + flat sibling <tr> rows.
Also regenerates package-lock.json for the plugin-react downgrade
from PR #4241 (npm ci was failing in Docker).
Local Docker build verified with --build-arg for both NEXT_PUBLIC vars.
- deploy workflow: add shared/scripts/manifest paths to shell-dashboard
and shell-docs filters (previously triggered implicitly by committed
JSON diffs in those directories)
- capture-previews: add generate-registry step before capture; use
git add -f for the gitignored registry.json
- e2e smoke test: document generator dependency in import comment
aimock is no longer a Docker-built showcase service. Railway
pulls the pre-built upstream image directly from GHCR. Remove:
- aimock from workflow_dispatch service options
- aimock paths-filter (showcase/aimock/**)
- aimock entry from ALL_SERVICES matrix
Add platforms: linux/amd64 to the depot/build-push-action invocation in
showcase_deploy.yml. Railway and GHCR serve x86 hosts, so an arm64-only
image crashes on pull. Matches the platform requirement enforced for
local docker build invocations (documented in
showcase/starters/template/README.md).
The dojo app was missing items under the langgraph column because
shell-dojo shipped a stale committed registry.json. The generator
only wrote to shell/, the dojo Dockerfile didn't run the generator
at build, and the CI path filter didn't rebuild the dojo when
manifest files changed.
Fix: emit from generate-registry.ts to shell, shell-dojo, and
shell-docs; add the generator step to shell-dojo's Dockerfile;
expand the deploy workflow's path filter to include packages/**
and shared/**; and refresh the committed registry/demo-content
JSON so files on disk match what the generator produces today.
The shell-dashboard app baked http://localhost:3000 into every demo and code link because NEXT_PUBLIC_SHELL_URL was never provided at build time and the source defaulted to localhost. Next.js inlines NEXT_PUBLIC_* at next build, so setting the value on Railway at runtime does nothing.
Fix: remove the silent localhost fallback, pass NEXT_PUBLIC_SHELL_URL as a Docker build arg from showcase_deploy.yml, and fail loudly if it's unset at build so this can't regress silently.
Regression from the 2026-04-21 incident: 18 production Railway services
were found with malformed image refs of the form
`ghcr.io/copilotkit/showcase-<slug>atest` (missing the `:` before
`latest`, so Docker treats `...atest` as the tag). Root cause was an
out-of-band MCP/manual mutation — no committed code touched those refs,
so the data has been fixed but no source-controlled guardrail exists.
Add a standalone script that queries Railway's GraphQL API for every
service in the CopilotKit Showcase project and asserts each image ref
matches the canonical shape `ghcr.io/copilotkit/<service-name>:latest`.
Wire it into showcase_deploy.yml as a pre-build job so any drift aborts
the workflow before the build matrix fans out.
On violation the script prints the service name, the current image, the
expected shape, and the reason, so the fix is obvious in the run log.
Slack classification in the notify job distinguishes a drift failure
from other pre-build failures.
Verified locally: 41 services pass against current Railway state; the
exported `validateImage` function rejects the exact `...atest`
corruption, mismatched service/image names, missing tags, wrong
registries, wrong tag values, and null sources (9/9 simulated cases).
matrix.service.timeout is already a number in the generated matrix
JSON, so wrapping it in fromJSON() was a no-op that only obscured
the expression. Drop the wrapper.
The HTTP_CODE="000" sentinel assigned immediately before the probe
loop is dead: the loop's first iteration unconditionally overwrites
HTTP_CODE before any reader runs. Remove it to avoid implying a
meaningful default where there isn't one.
The shell-family starters bundle shared demo content and tooling at
build time via showcase/scripts/bundle-demo-content.ts, which walks
showcase/shared and showcase/packages/*/manifest.yaml. A change to
any of those inputs can alter the generated bundle without touching
the package directory, so path-filter was under-reporting changes
and skipping deploys that actually needed to rebuild. Extend the
filters to shared/**, scripts/**, and packages/*/manifest.yaml so
those inputs trigger the correct downstream deploys.
- Promote SLACK_WEBHOOK_OSS_ALERTS to job-level env so the step-level
if: gate can actually see it. Step-level env: is not visible to the
same step's if:, which silently disabled every Slack post.
- Drop the service suffix from the concurrency group so two manual
dispatches for different services don't serialize unexpectedly and
push-triggered runs cancel each other deterministically.
- Rewrite the shared-module copy skip guard as an allowlist scoped to
showcase/packages/*. Starters, shell-family, aimock, and shell-dojo
don't consume shared modules, so the previous denylist missed every
new self-contained context (context=".") and polluted the repo root.
- Add a comment to the prior-deploy Railway GraphQL query explaining
that deployments(first: 1) relies on the API default of
createdAt DESC ordering. Railway has no explicit orderBy argument
on this field; documenting the assumption means a future schema
change (or an operator reading the file cold) has something to
point at before debugging stale-deployment guard regressions
(finding #15).
- Harden both truncate_csv shell helpers (one per run block): disable
pathname globbing via set -f for the iteration so a slug
containing a glob char (*, ?, [) cannot expand against the
filesystem, and lock IFS to whitespace for deterministic
word-splitting regardless of caller env. Restore the prior glob
state on the way out so other shell blocks see no side effect
(finding #16).
The main-deploy step already uses '// empty' on PRIOR_DEPLOY_ID but
the verify step extracted DEPLOY_ID / STATUS / DOMAIN without it, so
when Railway's edges array was transiently empty these became the
literal string 'null'. That breaks the prior-vs-fresh comparison and
the DOMAIN emptiness guard in inconsistent ways. Add '// empty' to
all three jq filters so empty-edge responses normalize to '' the
same way everywhere.
Two small but load-bearing hygiene fixes in the notify job's shell:
1) The matrix-legs collection pipeline used `jq -r '...' 2>/dev/null`,
swallowing any jq parse or schema errors before the surrounding
pipeline substitution could see them. If the gh-api payload shape
ever shifts (schema rev, partial response, encoding glitch), we
now want the error in the job log so an operator knows WHY notify
fell back to softer wording — the existing `-z "$pairs"` guard
still handles the empty-output path without relying on suppression.
2) The duplicated inline `truncate_csv` helper was declaring
`local ws=...` on its OWN line AFTER `local budget=...` on the
function's first line. Redeclaring `local` inside the same
function body is legal but prints 'local: not in a function'
warnings on some bash versions (3.x, noexec edge cases) and is
outright forbidden by `set -eu` hardening profiles. Collapse `ws`
into the single `local` declaration at the top and use a plain
assignment where it's computed.
The BUILD outcome classifier read:
elif [ "$BUILD" = "failure" ] || [ -n "$BUILD" ]; then
After the earlier `success` + `skipped && detect=success` branches
had claimed the obvious green cases, any non-empty BUILD string fell
through the `-n` test and was tagged FAILURE — including values like
"skipped" that reach this branch when detect-changes did NOT emit
`success` (e.g. cancelled pre-stage races or a partial matrix skip).
That means the Slack bot would occasionally cry red at runs that never
failed.
Simplify to an exact string match on "failure"; the final `else`
branch already handles the empty/unknown case by emitting OUTCOME=skip,
which is the correct disposition for indeterminate state.
Jordan Ritter