The decoupled build workflow (PR #4471) removed the Railway deploy
trigger, assuming environmentPatchCommit auto-update would handle
deploys. Not all services have auto-update configured, so GHCR images
were pushed but Railway never pulled them. Restore the explicit
serviceInstanceRedeploy call after each GHCR push.
The old "Showcase: Build & Deploy" workflow used a single concurrency group
that cancelled in-flight builds on every push to main. When multiple PRs
merged in quick succession, most service builds got cancelled and never
deployed.
Split into two workflows:
1. showcase_build.yml ("Showcase: Build & Push") - triggered on push to main,
builds Docker images and pushes to GHCR. Has NO concurrency group so every
run completes. Railway auto-update picks up the new :latest tag.
2. showcase_deploy.yml ("Showcase: Verify Deploy") - triggered by workflow_run
from the build workflow. Polls Railway to verify each service picked up the
new image and is healthy. Uses cancel-in-progress since verification is
idempotent. Posts results to showcase-harness via webhook.
Also updates showcase_capture-previews.yml to trigger from the renamed build
workflow.
Production showcase-aimock was running a week-old image because fixture
file changes in showcase/aimock/ did not trigger a CI rebuild. This adds
showcase-aimock to the Build & Deploy workflow matrix so it auto-deploys
on merge, creates a thin Dockerfile that bakes fixture files into the
image, documents the local-vs-production parity requirement in
docker-compose.local.yml, and adds an aimock fixture deployment section
to the RUNBOOK.
Remove all showcase-starter-* services from the CI build matrix in
showcase_deploy.yml. These starter demos were decommissioned in PR
#4378 (code removed) and confirmed dormant with zero traffic. Removes
entries from workflow_dispatch options, paths-filter definitions, and
the ALL_SERVICES JSON array.
- Update collision-avoidance comment in showcase_deploy.yml to remove
starter-specific examples (service/starter collision no longer possible)
- Fix showcase_keep-alive.yml description: pings showcase services, not starters
- Remove "starter" from catalog-types.ts manifestation union type
- Remove dead starter cell skip logic from cell-matrix.tsx cellIndex builder
- Update depth-utils.ts comment (defensive null guard, not starter-specific)
- Remove starter-specific test cases from depth-utils and cell-matrix tests
CR R1 follow-ups on top of the /api/ops proxy fix.
Bucket (a) — must-fix:
- Dockerfile: declare ARG/ENV OPS_BASE_URL in the builder stage. next.config.ts
evaluates rewrites() at build time and throws if OPS_BASE_URL is unset, which
aborted `next build` in CI. Mirrors the existing NEXT_PUBLIC_SHELL_URL /
NEXT_PUBLIC_POCKETBASE_URL pattern.
- showcase_deploy.yml: pipe OPS_BASE_URL through to docker build for the
shell-dashboard matrix entry, defaulting to the production
showcase-ops-production.up.railway.app URL.
- next.config.ts: strip trailing slashes from OPS_BASE_URL before constructing
the rewrite destination, matching the same normalization in
src/lib/ops-api.ts:resolveBaseUrl so server-side rewrite and client-side
fetch agree on the URL shape.
- src/lib/ops-api.ts: treat empty / whitespace NEXT_PUBLIC_OPS_BASE_URL as
"no override". `??` only short-circuits on null/undefined, so an env var set
to "" silently produced baseUrl="" and URLs of the form "/probes" with no
/api/ops prefix.
- use-probes.integration.test.tsx: snapshot+restore process.env.NEXT_PUBLIC_OPS_BASE_URL
in beforeEach/afterEach so tests never leak env state. Strengthen the proxy
contract assertions: lock toHaveBeenCalledTimes(1), assert method=GET,
cache=no-store, accept JSON header, and signal is an AbortSignal. Tighten
the 404 regression to assert the canonical ensureOk message shape so a
refactor that changes the format trips the test.
Bucket (b) — applied since the diff stayed focused:
- triggerProbe: add cache:"no-store" for parity with the GET fetches.
- fetchProbeDetail / triggerProbe: throw early when id is empty so callers
get a clean error instead of a request to /probes//... .
- ensureOk: bump body-truncation cap from 200 to 500 chars and append a
`[truncated, N bytes total]` marker so operators can see they're missing
tail bytes when the server returns a long HTML/stack-trace body.
- ops-api.ts: drop dev-loop review-cycle tag prefixes (R2-C.3, R3-C, R3-D.1)
from comments. Keep the actual rationale.
- ops-api.ts header docstring: clarify that NEXT_PUBLIC_OPS_BASE_URL is read
live at runtime in this codebase (SSR + tests), not just statically inlined
into the client bundle.
Verified:
- Tests: vitest run — 26 files, 293 passed, 1 skipped (no test count change).
- Typecheck: tsc --noEmit clean.
- Lint + format: oxlint + oxfmt clean on changed files.
- Local Docker build: `docker build --build-arg OPS_BASE_URL=https://...` succeeds.
Without --build-arg the build fails with "OPS_BASE_URL must be set" as
expected, confirming the fix is load-bearing.
showcase-ops was excluded from .github/workflows/showcase_deploy.yml, so
commits touching showcase/ops/** never produced a fresh GHCR image. PR
#4293 (Status tab + /api/probes route) merged to main on 2026-04-26 but
no rebuild fired — the deployed Railway image is stale and /api/probes
404s in production.
Adding showcase-ops as a first-class matrix entry:
- dispatch_name: showcase-ops (workflow_dispatch option + filter_key)
- paths-filter: showcase/ops/**, plus shared/scripts/manifests
(showcase-ops's Dockerfile bundles all four into the runtime image
via build-stage COPY + generate-registry.ts)
- context: '.' (repo root) so the Dockerfile can COPY from
pnpm-workspace.yaml + packages/ + showcase/{ops,shared,packages,scripts}
- dockerfile: showcase/ops/Dockerfile
- image: showcase-ops -> ghcr.io/copilotkit/showcase-ops:latest
- railway_id: 3a14bfed-0537-4d71-897b-7c593dca161d
- health_path: /health (matches Dockerfile HEALTHCHECK + Hono route)
- timeout: 20 (heavier build than shells: pnpm deploy + chromium
install via playwright --with-deps)
- lfs: false (no Git LFS assets in showcase/ops)
- linux/amd64 platform inherited from existing build step (Depot)
Resulting matrix: 39 services (was 38). dispatch_name uniqueness +
JSON validity verified locally; actionlint/yamllint surface only
pre-existing findings on the workflow.
Three fixes batched to minimize PR churn:
1. Dockerfile: add ARG NEXT_PUBLIC_POCKETBASE_URL so the PB URL gets
baked into the Next.js bundle at build time. Without this, pb.ts
resolves to the sentinel URL and the dashboard shows "unavailable"
on every tab. Pre-existing bug exposed by fresh deploys.
2. showcase_deploy.yml: pass NEXT_PUBLIC_POCKETBASE_URL and
NEXT_PUBLIC_SHELL_URL as build args for the shell-dashboard service
in the CI matrix. Neither was ever passed before.
3. cell-matrix.tsx + parity-matrix.tsx: flatten nested table pattern
that caused column misalignment. Category rows used colSpan with
an inner <table> whose columns floated independently of the header.
Replaced with useCollapsible hook + flat sibling <tr> rows.
Also regenerates package-lock.json for the plugin-react downgrade
from PR #4241 (npm ci was failing in Docker).
Local Docker build verified with --build-arg for both NEXT_PUBLIC vars.
- deploy workflow: add shared/scripts/manifest paths to shell-dashboard
and shell-docs filters (previously triggered implicitly by committed
JSON diffs in those directories)
- capture-previews: add generate-registry step before capture; use
git add -f for the gitignored registry.json
- e2e smoke test: document generator dependency in import comment
aimock is no longer a Docker-built showcase service. Railway
pulls the pre-built upstream image directly from GHCR. Remove:
- aimock from workflow_dispatch service options
- aimock paths-filter (showcase/aimock/**)
- aimock entry from ALL_SERVICES matrix
Add platforms: linux/amd64 to the depot/build-push-action invocation in
showcase_deploy.yml. Railway and GHCR serve x86 hosts, so an arm64-only
image crashes on pull. Matches the platform requirement enforced for
local docker build invocations (documented in
showcase/starters/template/README.md).
The dojo app was missing items under the langgraph column because
shell-dojo shipped a stale committed registry.json. The generator
only wrote to shell/, the dojo Dockerfile didn't run the generator
at build, and the CI path filter didn't rebuild the dojo when
manifest files changed.
Fix: emit from generate-registry.ts to shell, shell-dojo, and
shell-docs; add the generator step to shell-dojo's Dockerfile;
expand the deploy workflow's path filter to include packages/**
and shared/**; and refresh the committed registry/demo-content
JSON so files on disk match what the generator produces today.
The shell-dashboard app baked http://localhost:3000 into every demo and code link because NEXT_PUBLIC_SHELL_URL was never provided at build time and the source defaulted to localhost. Next.js inlines NEXT_PUBLIC_* at next build, so setting the value on Railway at runtime does nothing.
Fix: remove the silent localhost fallback, pass NEXT_PUBLIC_SHELL_URL as a Docker build arg from showcase_deploy.yml, and fail loudly if it's unset at build so this can't regress silently.
Regression from the 2026-04-21 incident: 18 production Railway services
were found with malformed image refs of the form
`ghcr.io/copilotkit/showcase-<slug>atest` (missing the `:` before
`latest`, so Docker treats `...atest` as the tag). Root cause was an
out-of-band MCP/manual mutation — no committed code touched those refs,
so the data has been fixed but no source-controlled guardrail exists.
Add a standalone script that queries Railway's GraphQL API for every
service in the CopilotKit Showcase project and asserts each image ref
matches the canonical shape `ghcr.io/copilotkit/<service-name>:latest`.
Wire it into showcase_deploy.yml as a pre-build job so any drift aborts
the workflow before the build matrix fans out.
On violation the script prints the service name, the current image, the
expected shape, and the reason, so the fix is obvious in the run log.
Slack classification in the notify job distinguishes a drift failure
from other pre-build failures.
Verified locally: 41 services pass against current Railway state; the
exported `validateImage` function rejects the exact `...atest`
corruption, mismatched service/image names, missing tags, wrong
registries, wrong tag values, and null sources (9/9 simulated cases).
matrix.service.timeout is already a number in the generated matrix
JSON, so wrapping it in fromJSON() was a no-op that only obscured
the expression. Drop the wrapper.
The HTTP_CODE="000" sentinel assigned immediately before the probe
loop is dead: the loop's first iteration unconditionally overwrites
HTTP_CODE before any reader runs. Remove it to avoid implying a
meaningful default where there isn't one.
The shell-family starters bundle shared demo content and tooling at
build time via showcase/scripts/bundle-demo-content.ts, which walks
showcase/shared and showcase/packages/*/manifest.yaml. A change to
any of those inputs can alter the generated bundle without touching
the package directory, so path-filter was under-reporting changes
and skipping deploys that actually needed to rebuild. Extend the
filters to shared/**, scripts/**, and packages/*/manifest.yaml so
those inputs trigger the correct downstream deploys.
- Promote SLACK_WEBHOOK_OSS_ALERTS to job-level env so the step-level
if: gate can actually see it. Step-level env: is not visible to the
same step's if:, which silently disabled every Slack post.
- Drop the service suffix from the concurrency group so two manual
dispatches for different services don't serialize unexpectedly and
push-triggered runs cancel each other deterministically.
- Rewrite the shared-module copy skip guard as an allowlist scoped to
showcase/packages/*. Starters, shell-family, aimock, and shell-dojo
don't consume shared modules, so the previous denylist missed every
new self-contained context (context=".") and polluted the repo root.
- Add a comment to the prior-deploy Railway GraphQL query explaining
that deployments(first: 1) relies on the API default of
createdAt DESC ordering. Railway has no explicit orderBy argument
on this field; documenting the assumption means a future schema
change (or an operator reading the file cold) has something to
point at before debugging stale-deployment guard regressions
(finding #15).
- Harden both truncate_csv shell helpers (one per run block): disable
pathname globbing via set -f for the iteration so a slug
containing a glob char (*, ?, [) cannot expand against the
filesystem, and lock IFS to whitespace for deterministic
word-splitting regardless of caller env. Restore the prior glob
state on the way out so other shell blocks see no side effect
(finding #16).
The main-deploy step already uses '// empty' on PRIOR_DEPLOY_ID but
the verify step extracted DEPLOY_ID / STATUS / DOMAIN without it, so
when Railway's edges array was transiently empty these became the
literal string 'null'. That breaks the prior-vs-fresh comparison and
the DOMAIN emptiness guard in inconsistent ways. Add '// empty' to
all three jq filters so empty-edge responses normalize to '' the
same way everywhere.
Two small but load-bearing hygiene fixes in the notify job's shell:
1) The matrix-legs collection pipeline used `jq -r '...' 2>/dev/null`,
swallowing any jq parse or schema errors before the surrounding
pipeline substitution could see them. If the gh-api payload shape
ever shifts (schema rev, partial response, encoding glitch), we
now want the error in the job log so an operator knows WHY notify
fell back to softer wording — the existing `-z "$pairs"` guard
still handles the empty-output path without relying on suppression.
2) The duplicated inline `truncate_csv` helper was declaring
`local ws=...` on its OWN line AFTER `local budget=...` on the
function's first line. Redeclaring `local` inside the same
function body is legal but prints 'local: not in a function'
warnings on some bash versions (3.x, noexec edge cases) and is
outright forbidden by `set -eu` hardening profiles. Collapse `ws`
into the single `local` declaration at the top and use a plain
assignment where it's computed.
The BUILD outcome classifier read:
elif [ "$BUILD" = "failure" ] || [ -n "$BUILD" ]; then
After the earlier `success` + `skipped && detect=success` branches
had claimed the obvious green cases, any non-empty BUILD string fell
through the `-n` test and was tagged FAILURE — including values like
"skipped" that reach this branch when detect-changes did NOT emit
`success` (e.g. cancelled pre-stage races or a partial matrix skip).
That means the Slack bot would occasionally cry red at runs that never
failed.
Simplify to an exact string match on "failure"; the final `else`
branch already handles the empty/unknown case by emitting OUTCOME=skip,
which is the correct disposition for indeterminate state.
Remove every trace of the defunct test-integration-tmp service from the
showcase deploy workflow:
- drop the `test-integration-tmp` entry from the workflow_dispatch
inputs.service dropdown,
- drop the detect-changes `test_integration_tmp` output + paths-filter
entry, and
- delete the entire build-test_integration_tmp job (including its
placeholder RAILWAY_SERVICE_ID literal).
Matches the jpr5/test_integration_tmp_leak cleanup — the test harness
leak has been patched at its source (create-integration.test.ts), but
these workflow vestiges were never part of that fix and only survived
because nothing read them after the regression guard landed.
Adds the shell-docs package to the deploy matrix, paths-filter, and
dispatch options. Railway service `showcase-shell-docs` (ID
7badfb8d-4228-414c-9145-b4026803714f) has been provisioned with the
custom domain `docs.showcase.copilotkit.ai` and the Railway-generated
CNAME target `t6ge0qwv.up.railway.app`.
Env vars seeded on the Railway service: OPENAI_API_KEY,
ANTHROPIC_API_KEY, GOOGLE_API_KEY, POSTHOG_PROJECT_KEY, PORT=10000,
NEXT_PUBLIC_BASE_URL=https://docs.showcase.copilotkit.ai,
NEXT_PUBLIC_SHELL_URL=https://showcase.copilotkit.ai.
Matrix entry uses lfs:true + context:"." + explicit Dockerfile path
(mirrors shell) because the image pulls in shared/ scripts/ packages/
at build time. build_args_sha/branch are included so the build surface
embeds commit metadata, matching shell's pattern.
- Renamed showcase/shell-internal/ → showcase/shell-dashboard/ (git mv for history).
- Updated package name to @copilotkit/showcase-shell-dashboard (+ lockfile).
- Updated external references: showcase/scripts/{probe-docs,generate-status}.ts comments and showcase/README.md.
- Added dispatch + filter + ALL_SERVICES entry for shell-dashboard in .github/workflows/showcase_deploy.yml.
- Provisioned Railway service showcase-shell-dashboard (id 4d5dfd74-be61-40b2-8564-b53b7dd4c15b):
- image source ghcr.io/copilotkit/showcase-shell-dashboard:latest
- env vars OPENAI_API_KEY, ANTHROPIC_API_KEY, PORT copied from showcase-shell
- custom domain dashboard.showcase.copilotkit.ai attached (CNAME → 6u9icvje.up.railway.app)
- image will be built and pushed via showcase_deploy.yml on merge
Teams get pinged on every failure but never see a positive recovery
signal when these workflows flip back to passing. Mirror the per-service
transition pattern already implemented in showcase_smoke-monitor.yml.
showcase_deploy.yml (notify job):
- Restore previous run's status from actions/cache (key pattern
matches smoke-monitor: prefix-based restore-keys + run_id save key)
- Classify outcome as failure | success | skip
- failure → post existing red alert + update state
- success AND prev=failure → post recovery + clear state
- success AND prev=ok → silent (green→green, matches prior behavior)
- skip (no-changes or cancellation) → no state change, no post
starter_deployed_smoke.yml:
- Restore state before tests run
- After test step, classify via steps.playwright.outcome
- On red→green, post recovery message alongside existing failure path
- On any terminal outcome, save new state to cache
First-ever run (or cache miss): state initializes to ok so we don't
emit a false recovery on the first green run after this ships.
Recovery wording:
✅ *Showcase deploy*: recovered (was down since <iso8601>)
✅ *Starter Deployed Smoke Tests*: recovered (was down since <iso8601>)
Existing red-alert wording is unchanged.
NIT 3: collapse the per-leg jq pipeline in the notify step. The
previous version went `.jobs // [] | map(filter) | map(project) | .[]`
— three full passes over the list plus a flatten. Rewrite as a single
`.jobs[]? | select | "…"` stream: functionally identical, noticeably
easier to skim. No observable output change.
MEDIUM 3: keep the template/ exclusion cross-referenced in both places
it is enforced — validate-workflow-starters.ts EXCLUDED_DIRS and
showcase_smoke-monitor.yml's per-slug continue. If a future non-service
sibling (e.g. shared/, docs/) gets added, the pair of comments makes
it obvious that both sites need the new entry.
`cut -c1-200` truncated mid-slug, producing reader-hostile output
like `starter-claude-sdk-pyth` or `succeeded: starter-cl`. Replace
with a `truncate_csv` helper that iterates items, appends until the
next item would blow the budget, then emits `…` to signal truncation.
Applied to three list-cap sites (services summary, succeeded_list,
failed_list). Falls back to a hard character cut only when the very
first item is already over budget, so a single pathological 250-char
slug still emits something rather than an empty string.
Helper is inlined per step because YAML run: blocks don't share shell
functions. Comment cross-references steps.legs so the two copies stay
recognisably linked.
`shell` and `shell-dojolike` use `health_path="/"` because neither
ships an `/api/health` endpoint — they're Next.js app shells serving
the homepage as the liveness signal. With the new `exit 1` on
persistent unhealth, a single transient 5xx or 301-chain glitch at
the homepage during a cold-start would reset the HEALTHY_STREAK and,
at the edge of the 24-attempt budget, could page us for nothing.
Wrap the HTTP probe in a 3-attempt inner retry (2s apart) before the
streak logic observes the code. Services whose homepage is genuinely
dead still fail — all 3 tries must return non-200 — but a one-off
blip no longer trashes accumulated progress. Non-shell services
(which hit `/api/health`, `/health`, etc.) benefit from the same
transient-tolerance for free.
The notify job's failure branch posted "FAILED — N service(s) targeted"
whenever `needs.build.result == 'failure'`, which rolls up to 'failure'
if even a single matrix leg failed. With fail-fast: false, one failed
leg out of 18 can roll up as 'failure' while 17 others shipped fine —
the old wording falsely implied all N failed.
Add a "Compute per-leg build results" step that queries the Actions
API for this run's jobs, buckets each `build (<leg>)` leg by its
conclusion, and exposes failed/succeeded counts + lists as outputs.
The payload step now picks between three shapes:
- Partial failure: "N/M service(s) failed (<failed>) — <succeeded> ok"
- Full failure: existing "FAILED — M service(s) targeted (<list>)"
- API lookup lost: softer "1+ of M service(s) failed (<list> targeted)"
so the alert never lies about how many were affected.
Also reword the stale "sleep-on-idle waking up" comment on the
success-but-no-HTTP-200 branch — Railway is on the Pro tier (see
rationale near line 412) and does not sleep on idle. Clarify the
notify cancel-handling comment accordingly.
Previously, when a deployed service never returned a 200 on its health
path within the 360s window, the workflow emitted `::warning::` and
exited 0. That masked silently-broken deploys: Slack saw green, smoke
monitor saw no failing deploy run, and the stale image sat live.
Emit `::error::` and exit 1 so the deploy job fails and the alert
actually fires. Railway is on the Pro tier with no sleep-on-idle, so a
persistent failure to become healthy in 6 minutes is a real failure,
not a cold start.
Evidence of silent failures: https://github.com/CopilotKit/CopilotKit/actions/runs/24616467033
The showcase_deploy.yml notify job posted a green success message on
every run. Bulk drift rebuilds fan out one showcase_deploy.yml run per
stale service (up to ~18), so a single drift cycle produced ~18
"Showcase deploy: 1 service(s) deployed to Railway" messages in
#oss-alerts — pure noise.
Align with the channel policy: surface only actionable state
(failures + state transitions). The bulk-rebuild aggregate is already
posted by showcase_smoke-monitor.yml as "📦 Image drift detected
— N rebuilds triggered". Ad-hoc single-service pushes/dispatches stay
quiet on success — the Actions UI is the source of truth.
Kept as-is: failure messages (pre-build and build), mid-matrix
cancellation info (state transition worth humans seeing). Also added an
empty-webhook guard on the Post step so a missing SLACK_WEBHOOK_OSS_ALERTS
secret fails closed instead of erroring out the step.
Jordan Ritter