promote-fleet.sh now emits a base64 results JSON (schema_version=1) with
both succeeded[] and failed[] alongside the existing succeeded_csv /
staging_drift outputs. showcase_promote.yml's notify job replaces the old
inline two-state (success/failure) notifier — which dumped the full
requested CSV and mislabeled any partial promote as a blanket Failed — with
a single dispatch of the previously-orphaned showcase_promote_notify.yml
renderer (success / partial / total). The dispatch enriches promote-fleet's
results blob with run context (6-hex run_id, trigger=workflow, operator,
pre_staging) and authenticates via the devops-bot App token (actions:write),
mirroring canary.yml — the default GITHUB_TOKEN cannot start new workflow runs.
promote-fleet has no failure taxonomy, so each failed entry uses the default
category "promote-failed".
## Summary
An **on-demand** tool to answer "is prod caught up with staging right
now?". The showcase deploy model is **staging = mutable `:latest`**
(continuously rebuilt) and **prod = immutable `@sha256:`** (advances
only on an explicit promote), so a prod column can sit **behind** a
green staging.
**There is no scheduled drift alert — by design.** Prod lagging staging
is **often intentional**: changes are batched and promoted deliberately,
so a recurring "N columns stale" alert would be pure noise. This tool is
therefore manual-only: a maintainer runs it when they want to check, and
it tells them the current state.
- **`bin/railway reconcile-prod`** — for every prod-eligible
(`probe.prod == true`) service, compares the **prod serving digest**
(the `@sha256:` from
`SnapshotCommand.build_snapshot(PRODUCTION_ENV_ID)`) against the
**staging running digest** (reuses
`PromoteCommand#staging_running_digest`, the same source the promote pin
uses). Classifies each:
- `green` — prod == staging (in sync)
- `stale` — prod != staging **and** staging is resolvable (prod is
behind a green staging)
- `gray` — staging running digest not resolvable, or the service has no
prod snapshot entry yet — informational, **not** stale
- Prints a readable per-service table + summary; **exits nonzero iff any
service is stale**; `--json` for machine output. **Read-only — no
promotes/mutations.**
- **`showcase/scripts/reconcile-prod-gate.sh`** — wrapper mirroring
`lint-prod-gate.sh`: surfaces the table into `$GITHUB_STEP_SUMMARY`,
optionally captures `--json` to `RECONCILE_JSON`, and propagates the
exit-code verdict (never swallows a non-zero).
- **`.github/workflows/showcase_reconcile.yml`** — **`workflow_dispatch`
only** (no cron). Regenerates the SSOT JSON (`EMIT_SKIP_OXFMT=1`, same
as the promote workflow's resolve/promote jobs), runs the gate with the
Railway/GHCR auth env, renders the reconcile table to the **GH step
summary**, and uploads the `--json` as a `reconcile-json` artifact. **No
Slack.** The run exits nonzero on a stale column so a manual run visibly
flags drift. `timeout-minutes: 10`.
- **Tests** — Ruby minitest (`test_reconcile_prod.rb`: classification +
exit-code + `--json` shape + dispatcher registration) and a bats gate
test (`reconcile-prod-gate.bats`). Wired the gate script into the
`showcase_validate.yml` shellcheck list.
### What changed from the original scheduled-alert design
The first cut of this PR shipped a daily cron + auto-post to #oss-alerts
on any stale column. Per owner feedback, that was reshaped to on-demand
only: the `schedule:` trigger and the Slack-on-stale step were removed
(intentional/deliberate staleness is not a bug, so an unsolicited
recurring alert is noise). The CLI command, the gate wrapper, and all
tests are unchanged.
## Gates
- `ruby showcase/bin/spec/test_reconcile_prod.rb` → **9 runs, 20
assertions, 0 failures**
- `bats showcase/scripts/__tests__/reconcile-prod-gate.bats` → **6 ok**
- `shellcheck -s bash showcase/scripts/reconcile-prod-gate.sh` →
**clean**
- `actionlint .github/workflows/showcase_reconcile.yml` → **clean** (the
pre-existing `depot-ubuntu-24.04-4` custom-runner-label warning is on
`showcase_validate.yml`, predates this PR — my only change there is one
line in the shellcheck list)
## Test plan
- [ ] CI green (Ruby suite, bats suite, actionlint/shellcheck,
commitlint)
- [ ] Optional: read-only `workflow_dispatch` run of
`showcase_reconcile.yml` to confirm it runs against live prod/staging
(safe — no mutations)
A `shell-docs` promote expanded its dependency closure to include the
always-on Tier-1 verification set (harness, dashboard, …); harness then
tripped a P6 env-divergence WARN-refusal and the tier barrier gated docs
as NOT-ATTEMPTED. docs has no runtime dependency on the control plane.
Introduce a declarative `standalone` service class: a leaf that neither
depends on anything nor gates on anything. computePromoteClosure (TS) and
the resolve-promote-targets jq both skip the Tier-1 union when the request
is entirely standalone, so a standalone request promotes ONLY itself; the
fleet runner promotes `s:`-marked services ungated (never NOT-ATTEMPTED on
an unrelated failure, never gating a tier). Flag `docs` standalone.
Red-green: railway-envs.test.ts (4), resolve-promote-targets.bats (5),
promote-fleet.bats (3) fail on origin/main (docs closure pulls harness;
`s:` is an unknown tier) and pass after the fix.
Prod sitting behind staging is often intentional (changes are batched and
promoted deliberately), so a recurring drift alert is noise. Reshape the
reconcile workflow to manual-only:
- Remove the daily `schedule:` cron trigger — leave only `workflow_dispatch`.
- Remove the auto-Slack-on-stale step (and its SLACK_WEBHOOK env / stale_line
output derivation) — no unsolicited #oss-alerts post on mere staleness.
- A manual run surfaces the reconcile table to the GH step summary, keeps the
cheap `--json` capture as an uploaded artifact, and still exits nonzero on a
stale column so a manual run visibly flags drift.
- De-noise the gate script + bats comments that referenced the removed
scheduled/Slack behavior.
The on-demand CLI (`bin/railway reconcile-prod`), the gate wrapper, and the
Ruby + bats tests are unchanged.
Lever 1 of the promote-reliability hardening plan. The showcase deploy
model is staging=mutable :latest (continuously rebuilt), prod=immutable
@sha256: (advances only on explicit promote), so a prod column can
silently fall BEHIND a green staging — drift today is only noticed by
eyeballing a dead column. This adds proactive, automatic detection.
- bin/railway reconcile-prod: for every prod-eligible (probe.prod==true)
service, compares the prod SERVING digest (LintProd snapshot path) vs
the staging RUNNING digest (reuses PromoteCommand#staging_running_digest).
Classifies green/stale/gray, prints a table + summary, exits 1 iff any
stale. --json for machine output. Read-only: no promotes/mutations.
- scripts/reconcile-prod-gate.sh: wrapper mirroring lint-prod-gate.sh —
surfaces the table to the GH step summary, captures JSON for the Slack
builder, propagates the exit-code verdict.
- .github/workflows/showcase_reconcile.yml: daily cron + workflow_dispatch;
runs the gate; on stale services posts the stale-column list to
#oss-alerts (SLACK_WEBHOOK_OSS_ALERTS) via the fromJSON('"\n"') idiom.
- Tests: Ruby minitest (classification + exit-code, RED-anchored on a
drift-blind classifier) and a bats gate test. Wired the gate script
into the showcase_validate.yml shellcheck list.
Post-promote convergence verification is deferred to a fast-follow.
Two PR checks were red on the new integration:
- check-config-files: add strands-typescript/next.config.ts to the build-
config allowlist.
- Validate Showcase: update the new-integration guard pins that intentionally
trip when an integration is added — BORN_IN_SHOWCASE 6→7, calculator
_from-feature-parity count 18→19, catalog cross-join 874→920 / total_cells
855→900 / docs_only 19→20 (46 features × 20 integrations), and the aimock
substring-shadow ceiling 132→133 (+1 from the strands-typescript calculator
fixture).
Also drop the premature deploy wiring: strands-typescript is removed from
showcase_build.yml (matrix + path filter + ALL_SERVICES) because it has no
Railway service yet (deployed: false) and the railway-envs SSOT test requires
a real service entry. It re-enters the deploy pipeline when the Railway
service is provisioned (external setup per INTEGRATION-CHECKLIST).
Regenerated the stale committed showcase_promote.yml so the 12 starter-*
services (+ shell-docs and all existing targets) appear in the service
dispatch choice list (fixes HTTP 422 on
gh workflow run -f service=starter-*). Reverted isProdPromotable to
env-map-only (environments.prod.probe), equivalent to the workflow resolve
predicate. Added a regression test asserting shell-docs + all 12 starters
remain in the generated AND committed dropdown.
## Summary
Fixes two correctness bugs in the showcase staging→prod promote path
(`showcase/bin/railway`), discovered + live-validated while promoting
the full 19-service cluster.
**Bug #1 — promote pinned the wrong digest.** `resolved_prod_image`
re-resolved the mutable `:latest` tag against *current* GHCR instead of
pinning the digest staging is actually *running*
(`latestDeployment.meta.imageDigest`). When `:latest` drifted after
staging deployed, promote pushed an unvalidated (and once, regressed)
image to prod. Now pins staging's running digest. Adds a loud `⚠️
STAGING DRIFT` warning (promote stdout + `STAGING_DRIFT_MARKER:` →
`promote-fleet.sh` aggregation → both Slack payloads) when staging's
running digest ≠ current `:latest`, so the gap is visible without
blocking the promote.
**Bug #2 — the pin never activated.** `pin_and_verify` used
`serviceInstanceRedeploy`, which replays the *existing* deployment's old
image rather than the just-pinned `source.image`. Config showed the new
digest while prod kept serving the old one (this is why earlier promotes
"succeeded" while prod stayed broken). Switched to
`serviceInstanceDeployV2` + a new `verify_serving_digest!` gate that
polls the new deployment to SUCCESS and **fails loud** if the running
digest ≠ pinned.
Plus CR-round hardening: P2 in-flight race check now reads
`meta.imageDigest` (was dead on tag-form staging) and skips on
`--digest` override; `detect_staging_drift` fails loud (WARN) on
GHCR-resolve failure instead of swallowing; `--digest` override
suppresses spurious drift; `drift_line` LF-stripped at the
`GITHUB_OUTPUT` boundary; multi-service drift join fixed; fallback-log
drift preserved.
## Commits
1. `fix(showcase): pin prod to staging's running digest + loud
staging-drift warning`
2. `fix(showcase): activate prod pin via serviceInstanceDeployV2 +
verify running==pinned`
3. `fix(showcase): harden promote P2 race check + refresh ivar-lint
allowlist`
## Validation
- Live red-green on real Railway: claude-sdk-python prod flipped from
stale → pinned digest via the fixed CLI; all 19 cluster services
promoted green; 5 previously-degraded backends (ag2, llamaindex,
pydantic-ai, ms-agent-python, strands) recovered.
- 15/15 Ruby specs + 15/15 bats green; ruby -c, shellcheck (CI
invocation), actionlint clean.
- 3-round cr-loop (7 agents/round) converged to zero bucket-(a).
## Follow-ups (not in this PR)
- Defensive `.to_s.empty?` on `meta.imageDigest` extraction
(can't-happen on real Railway).
- Test-quality nits (capture_io scoping, weak bats glob, shared mock
fixture).
- Pre-existing verify-prod `succeeded_csv`/`GITHUB_OUTPUT` coupling
(graceful-degraded, audited STAY_IN_C).
- `deploy-to-railway.ts` births prod on `:latest` (the root provisioning
gap); CLI↔workflow notify equivalence (PR2).
## Test plan
- [ ] CI green on PR HEAD
- [ ] (post-merge) a real `gh workflow run showcase_promote.yml` shows
the drift line in the #team-showcase notification when staging is behind
:latest
Prod services are now born pinned to a resolved @sha256 digest (mirroring
the Ruby promote CLI's GHCR resolver) instead of the mutable :latest tag,
fail-loud on resolve failure, and goLive asserts the prod source.image is
digest-pinned. Adds DI-testable assertProdDigestPinned + coverage.
Bug #1: resolved_prod_image re-resolved :latest at promote time, so prod
could be pinned to a digest different from what staging is actually serving.
Now resolve via staging_running_digest (latestDeployment meta.imageDigest)
and REFUSE when unavailable. Add detect_staging_drift +
emit_staging_drift_warnings to surface (non-fatally) when :latest has moved
past staging's running digest; aggregate markers across the fleet in
promote-fleet.sh (printf join) and plumb the drift_line through
showcase_promote.yml (both Slack payloads + fallback log + GITHUB_OUTPUT
single-line guard). Skip drift detection on the --digest override path.
Adds a sugar form of the --isolate flag that pins the isolation slot
directly from the command line:
bin/showcase test agno --d5 --isolate=9
# equivalent to:
SHOWCASE_ISO_SLOT=9 bin/showcase test agno --d5 --isolate
The arg parser splits --isolate=<N> into setting use_isolate=true plus
exporting SHOWCASE_ISO_SLOT=<N>; the existing picker
(_claim_isolate_slot in _common.sh) handles all validation — positive
integer, slot 0 reserved, 1<=N<=ISOLATE_MAX_SLOT, port probe, liveness.
No validation logic is duplicated.
Tests:
- replays the parser branch and verifies SHOWCASE_ISO_SLOT export +
picker pinning
- drives the picker's reserved-slot (N=0) and out-of-range (N=99)
rejections through the arg form to pin the parser->env->picker wiring
- drift guard: sources the REAL cmd-test.sh, stubs apply_isolation, and
snapshots SHOWCASE_ISO_SLOT to catch any future regression of the
parser branch
Help text and TESTING.md updated in a follow-up commit.
## Summary
Built-in Agent docs now render at bare root URLs, such as `/quickstart`,
instead of under `/built-in-agent`. Legacy `/built-in-agent/*` and
`/integrations/built-in-agent/*` paths continue to redirect to canonical
root or backend URLs.
## Why
The Built-in Agent is the default docs surface, so public docs URLs
should not expose it as a sub-slug. Existing links still need to keep
working, and root-page navigation must stay stable regardless of a
visitor's stored framework selection.
## How
- Root docs resolve Built-in Agent authored pages first while preserving
reserved routes like `/ag-ui` and framework-prefixed docs.
- Redirect rules in `next.config.ts` and `seo-redirects.ts` point
retired Built-in Agent and unselected paths directly at canonical
destinations, with regression coverage that prevents redirect
destinations from targeting `/built-in-agent`.
- The sidebar/framework provider treats bare URLs as the default
Built-in Agent surface instead of letting a stored framework value
rewrite root-page chrome.
- MDX link and search-result href rewriting strip retired Built-in Agent
prefixes on root-rendered pages while preserving explicit
cross-framework links.
- Stale docs links, sitemap, `llms.txt`, markdown exports, and OG
resolution now align with the root-served Built-in Agent surface.
- Showcase generated-data tests now serialize shared fixture restoration
to avoid concurrent drift in CI.
## Verification
- GitHub checks are green on `f655013dd2576a46dc17b5901eb4dc501cb21028`.
- `npm --prefix showcase/shell-docs run test --
src/lib/__tests__/search-hrefs.test.ts
src/lib/__tests__/docs-link-rewrite.test.ts
src/lib/__tests__/seo-redirects.test.ts
src/lib/__tests__/next-config-redirects.test.ts
src/components/__tests__/docs-landing-next.test.tsx
'src/app/[framework]/[[...slug]]/__tests__/framework-root-shell-layout.test.ts'`
- `npm --prefix showcase/shell-docs run test -- --exclude
src/app/__tests__/public-assets.test.ts`
- `npm --prefix showcase/shell-docs run typecheck`
- `npm --prefix showcase/shell-docs run lint`
- `npm --prefix showcase/shell-docs run build`
- `npm --prefix showcase/scripts run test`
- `pnpm exec nx run @copilotkit/bot-slack:build --skip-nx-cache
--verbose`
* duplicate ceiling 290→291: tool-rendering.json's tightened 'current
price of AAPL' matchers now share two match keys with the existing
tool-rendering-custom-catchall.json entries in the same BIA context,
runtime-disambiguated by feature route.
* shadow ceiling 134→132 (ratchet down): the bare 'AAPL' vs 'current
price of AAPL' shadow pair on the tool-rendering.json side is gone.
* PARITY_NOTES: replaces the 'headless-complete turns 3+4 server-tool
reprompt loop' known-issue section with a resolved-via-sequenceIndex
description; the architectural reprompt loop now converges via the
sequenceIndex-gated emitter + narration-fallback pattern in
gen-ui-headless-complete.json.
The 5 new substring shadows are in d6 fixtures landed by this PR:
- d6/ag2/gen-ui-declarative.json: 'Show me a quick KPI dashboard' inner-call mirror overlaps with pre-existing 'KPI dashboard' entry in render-a2ui.json (same toolName=render_a2ui, context=ag2). Runtime-disambiguated by load order (inner-call mirrors ordered BEFORE outer generate_a2ui fixtures per the _meta._note in that file).
- d6/claude-sdk-typescript/gen-ui-declarative.json: same pattern as ag2.
- d6/claude-sdk-typescript/tool-rendering.json: dropped userMessage/turnIndex gates on toolCallId-keyed follow-up fixtures (per _shape_note in that file — Anthropic /v1/messages shape needs toolCallId-only gating for multi-pill loop safety). The remaining ungated userMessage fixtures ('weather in Tokyo', 'AAPL') now substring-overlap with gen-ui-headless-complete and tool-rendering-reasoning-chain prompts in the same context. Runtime-disambiguated by toolCallId chains and first-match-wins ordering.
These overlaps are exactly the runtime-disambiguated pattern the test comment endorses for ceiling bumps.
Prevents a file-parallelism race where this suite's beforeAll generator
call atomically renamed catalog.json while generate-registry.test.ts was
mid-sentinel-test, clobbering the appended sentinel.
Adds a 689-line integration test that exercises the full queue lifecycle
to the /api/runs projection (enqueue → claim → terminal → projection)
across all four families. Updates the railway-envs golden + verify-deploy
drivers regression test to account for the new fleet-runs route surface.
## Summary
End-to-end hardening of the showcase shell's URL plane:
- Carries `backendHostPattern` + `docsHost` in the shell runtime config
(no longer baked from `registry.json` at Docker build time). Derives
demo backend URLs at runtime from the pattern, so a new pattern via env
reconfigures every integration on the next deploy without a registry
rebuild.
- Issues docs-host 301/308 redirects from middleware with a runtime
`DOCS_HOST`; misconfigured values no longer 500 every docs route — they
fall back to a sentinel that disables the docs-redirect step.
- Hardens the redirect table builder + matcher: first-match-wins for
duplicate exact sources, deduped wildcard prefixes with warn, malformed
entries rejected at lookup-build time, case-insensitive matching parity,
trailing-slash normalization, structural `/integrations` namespace guard
above the docs-host redirect (closes R15/R17 hijack class).
- Hardens runtime-config + backend-url env readers:
scheme/whitespace/control-char normalization, query/fragment/userinfo
rejection on `DOCS_HOST` / `POSTHOG_HOST` / backend-host pattern /
local-override URLs, present-but-empty `posthogKey` rejection, prod
loopback `BASE_URL`/`DOCS_HOST` rejection (no silent `http://` prepend),
once-guarded FATAL logging (no per-request spam).
- Brings the build-time twin in `showcase/scripts/generate-registry.ts`
to parity with the runtime normalizer (scheme/trailing-slash strip,
degenerate fallback, `NEXT_PUBLIC` fallback, slug validation).
- Surfaces PostHog capture failures once per failure class; keeps
capture alive across the redirect via `event.waitUntil`; missing
`POSTHOG_KEY` is `console.error` in production and surfaces at
config-resolution time (not first redirect).
- Open-redirect hardening on `/shared//evil.com` (SU-18); `//` rejected
at both source and destination; root `/` exact-source and `/:path*`
wildcard sources rejected; non-printable-ASCII source/destination
rejected.
Stream: SU (shell-runtime-urls). Subject groups
SU-2/8/11/13/14/15/16/17/18/19/20 (initial), SU2-A/B (env-hardening),
CR2-C (test infra), SU5-A1..A7 (registry/builder/lint/matcher boundary),
SU6-A1..A6 (request-time normalization parity), SU6-B1..B7
(parsed-normalized return forms, SHOWCASE_LOCAL states, generator
`{slug}` validation), SU7-F1..F3 (final-round backend-pattern + POSTHOG
+ table + script-side parity).
## Test plan
- [x] `pnpm test` in `showcase/shell` — 7 files, 235/235 pass
- [x] `pnpm test` in `showcase/scripts` — 51 files, 1857/1857 pass (one
pre-existing flake in `generate-registry.test.ts` unrelated to this
diff; passes on subsequent runs, classic vitest fork-reuse cross-file
pollution)
- [x] `pnpm exec tsc --noEmit` in `showcase/shell` — clean
- [x] `oxlint` on every changed file — 0 warnings, 0 errors
- [x] `oxfmt --check` on every changed file — clean
- [x] `pnpm build` in `showcase/shell` (full Next 15.x build with
registry+demo-content+starter-content+search-index generation) — clean
- [ ] CI green on the PR — confirm via `gh pr checks` after push
SU7-F1 — backend host pattern hardening:
- F1.1 Reject bare trailing ?/# in the backend host pattern
- F1.2 Strip internal tab/CR/LF from the backend host pattern
- F1.3 Warn when ignoring an empty-string local backend override
- F1.4 Reject empty-userinfo @ in the backend host pattern authority
- F1.5 Keep __proto__ keys as data in local-backend maps
- F1.6 Commit the local-backends memo key only after the value computes
- F1.7 Trim local backend overrides before validation and name the real
rejection
- F1.8 Honest FATAL when the pattern host is a stray scheme fragment
- F1.9 Canonicalize the pattern authority for parity with the override
path
- F1.10 Acknowledge the staging-to-prod fail-open in the pattern fallback
- F1.11 Harden backend-url/local-backends-env test hygiene
SU7-F2 — runtime-config & client-config edge cases:
- F2.1 Branch POSTHOG_HOST rejection reasons (scheme/degenerate/parse-
failure) instead of the catch-all mislabel
- F2.2 Reject loopback BASE_URL/DOCS_HOST in production instead of the
silent http:// prepend
- F2.3 Key the DOCS_HOST fallback once-guard on (mode, shellHost, value)
and mode-prefix all value-only guard keys
- F2.4 Reject a present-but-empty posthogKey in the client config reader
- F2.5 Drop the trailing slash from SSR_PLACEHOLDER_URL for structural
parity with server values
- F2.6 Attribute the DOCS_HOST slash-strip to readDocsHost itself
- F2.7 Normalize trailing-dot FQDN spellings in the docs self-host loop
guard (both compare sides)
- F2.8 Harden console spies to capture all log args; pin the full all-env
config shape; converge SSR simulation on vi.stubGlobal
SU7-F3 — script-side parity, table classification & test isolation:
- F3 #1 Handle a missing reference integration per the error contract
- F3 #2 Port the runtime backend-host-pattern normalization into the
generator — scheme/trailing-slash strip, degenerate fallback,
NEXT_PUBLIC fallback
- F3 #3 Treat non-mapping manifest parses (empty/null/scalar/array YAML)
as validation errors, not TypeErrors
- F3 #4 Label a missing/unreadable constraints.yaml per the stderr+exit(1)
error contract
- F3 #5 Align atomic-write tmp naming with the test harness straggler-
sweep convention; guard main() on direct invocation
- F3 #6 Correct the determineCellStatus unshipped docstring; replace
stale hardcoded cell counts with formulas
- F3 #7 Isolate the pattern suite on a per-suite tmpdir harness; snapshot
the generator's full write set
- F3 #8 Classify discarded duplicate wildcards as duplicates — hoist the
owner check above the destination warns
- F3 #9 Reject a root ("/") EXACT seo-redirect source — homepage-hijack
twin of the root-wildcard guard
- F3 #10 Reject seo-redirect entries with non-printable-ASCII source/
destination — close the silent-dead-entry class
- F3 #11 Strip trailing slashes in normalizePosthogHost before the scheme
test
- F3 #12 Message-filter the empty-slug-set error count; pin the single
matcher entry
Round-by-round CR convergence covering the redirect builder, the middleware
matcher, the docs-host self-loop guard, and the runtime-config env readers.
Highlights:
- Clear module-load warns after fresh middleware import
- Validate SET BASE_URL values (scheme-less/degenerate/garbage) with
sentinel fallback + once-guarded FATAL log
- Normalize path/query/fragment-bearing DOCS_HOST to origin; reject
non-http(s) schemes; branch rejection reasons
- Harden POSTHOG_HOST (degenerate-host/scheme rejection); expose
posthogKey via readEnvPair semantics
- Reject a DOCS_HOST equal to the shell's own host (redirect-loop guard,
authority compare)
- Warn on missing local-ports.json under SHOWCASE_LOCAL=1 and validate
TCP port range; extract helper for tests
- backend-url hardening — slug charset guard, frozen local-backends memo,
pattern path-segment warn, local-override URL validation
- Client config fail-loud covers all four URL fields with type checks
- Make RuntimeConfig.posthogKey optional — absence is a valid state, not
a wiring bug
- Drop R15/R17 and guard /integrations from SEO redirects
- Dedup duplicate wildcard prefixes with first-match-wins warn
- Validate malformed SEO entries at lookup-build time
- Restore case-insensitive redirect matching parity
- Normalize trailing slashes before redirect matching
- Keep the framework segment on F13, pin MG3 case fix
- Read posthogKey from runtime config in middleware, not raw process.env
- Fall back to the default backend host pattern for degenerate values
- Disable docs redirects when the default fallback collides with the
shell host
- Bring validateBaseUrl to parity with its sibling readers
- Strip query/fragment from POSTHOG_HOST while keeping reverse-proxy paths
- Restrict local backend overrides to http(s) URLs
- Add server-only guard to runtime-config
- Harden localBackendsEnv failure posture
- Hoist /integrations namespace guard above the docs-host redirect
- Validate seo-redirect sources and cross-kind shadowing in
buildRedirectLookup
- Unify slash normalization for middleware matching
- Lowercase-normalize REGISTRY_FRAMEWORK_SLUGS at construction
- Escalate missing POSTHOG_KEY to console.error in production
- Skip all redirect steps when docs redirects are disabled (sentinel
consumer)
- Reject userinfo credentials in DOCS_HOST, POSTHOG_HOST, and the backend
host pattern
- Branch dev-vs-prod logging in readDocsHost and fatalPatternOnce
- Prepend http:// (not https://) to scheme-less loopback hosts
- Round-5 micro-finding batch across the URL config libs
SU5-A1..A7 — registry safety, // reject, builder lint batch (case-
insensitive :path*, same-destination twin allowlist, original-case
divergence remainder), matcher api boundary, generator+vitest infra, test
hygiene + empty docs-host guard, comment batch.
SU6-A1..A6 — reject miscased :path* tokens, warn on tokenless wildcards,
normalize redirect-destination comparisons like request time, reject
destinations containing "//", surface missing POSTHOG_KEY at config-
resolution time, compile matcher harness like Next's runtime, type
parse/tokensToRegexp in the path-to-regexp shim, keep buildRedirectLookup
JSDoc attached.
SU6-B1..B7 — reject query/fragment/userinfo in pattern and local-override
URL gates, return parsed-normalized URL form from validation success
paths, distinguish unset/blank/padded SHOWCASE_LOCAL states, warn when
SHOWCASE_LOCAL is set to a value other than 1, validate {slug} placeholder
in generate-registry, mirror middleware drop semantics in the wiring
test's registry re-derivation, pin the noStore spy and calls to one fresh
module instance in the Edge-path test.
- structural + behavioral pins for all 18 integrations (four ordering invariants; click walk
_001->_002->_003->_003 with follow-ups mirroring the server match/increment flow)
- fail-loud guards against loader error-swallowing and vacuous ordering passes
- full production pill text
- oxfmt applied
## Summary
Hardens the `--isolate` showcase verification flow across three areas:
**1. XDG state migration.** Isolate slot registry and per-run
rewritten-compose scratch dirs move off `/tmp` (wiped on reboot,
world-writable) to
`${XDG_STATE_HOME:-$HOME/.local/state}/copilotkit/showcase/` (`slots/` +
`runs/<name>/`). `/tmp` clearing silently destroyed a kept stack's
compose file and slot, making `--keep` unreliable. Run dirs are keyed by
the finalized project name (not PID) so a kept run is locatable for
manual teardown.
**2. Slot reaping + registry concurrency.** Since the state dir is now
persistent, slots are reaped by compose-project liveness (`docker ps
--filter label=com.docker.compose.project=<name>`), with PID/age
heuristics as fallback. The registry is made safe under concurrent
claimers: a sweep lock with heartbeat updates, own-pid lock release, and
tombstones; a claim-then-verify duplicate-name guard closing the TOCTOU
window; crash-safe reap ordering with compose-down of reap remnants and
a path-traversal guard. Failed `--isolate` setup no longer tears down
the default stack; half-initialized state is cleaned up on the way out.
Teardown uses `--volumes` everywhere, and a failed compose-down
preserves state for diagnosis. `--isolate` names are validated (must
start with lowercase letter/digit; `showcase` is reserved — it aliases
the default stack), and a fail-loud warning precedes pre-down of an
existing stack.
**3. `--keep` now actually persists an isolated stack.** Previously the
unconditional `trap restore_isolation EXIT` tore the stack down
regardless of `--keep`. Teardown is now gated on the keep flag
(`ISOLATE_KEEP` promoted to a global so it survives `cmd_test` return
into the trap scope): the slot + run dir are retained and a survival
notice prints the project name, the three offset host ports, and the
exact `docker compose -p <name> down` command — no silent port/slot
leak. A kept stack's live containers keep its slot from being reaped.
Shell-only — confined to `showcase/scripts/cli/_common.sh` +
`cmd-test.sh`; the harness TS only reads the env vars the shell exports
(unchanged). Follows up the `--keep` caveat documented in #5346.
## Review hardening
The branch went through an 8-round, 7-agent code-review loop with
red-green-verified fixes — that loop produced the state-machine
hardening commit (trap-scope fix, default-stack guards, registry
concurrency/teardown robustness, name validation) and grew the test
suite to pin every fix. A live end-to-end `--keep` verification run is
what surfaced the trap-scope bug (`--keep` silently not honored),
driving the `ISOLATE_KEEP` global fix.
## Test plan
- [x] `showcase/scripts/__tests__/isolate.bats` — 41 isolate tests
(red→green): XDG path resolution (+`XDG_STATE_HOME` override,
`~/.local/state` fallback, `runs/<name>`), liveness-based reaping (dead
project reaped/reclaimed, live project preserved), real-trap-path
`--keep` tests (no simulated-trap shortcuts), sweep/lock/tombstone race
pins (heartbeat resurrection, lock takeover, duplicate-name TOCTOU),
reap-order probe pinning live-slot protection, root/PID-reuse/DST
guards, and sentinel anti-vacuity discipline so trap tests cannot pass
vacuously.
- [x] Full `bats showcase/scripts/__tests__/` green, matching CI's Shell
script tests invocation.
- [x] shellcheck: no new warnings.
- [x] Live end-to-end: `bin/showcase test <slug> --d6 --isolate <name>
--keep` persists the stack under `~/.local/state/copilotkit/showcase`,
survival notice + manual teardown work, follow-up run reaps the stale
slot.
Cross-session review fixes for the --isolate machinery (one concern:
source + test + docs).
1) Reaper reserved-name guard (critical): _reap_isolate_slot trusted
slot records — a record naming 'showcase' (corrupt, or written by an
older CLI version before apply_isolation reserved the name) passes
the charset regex, so the reap ran `docker compose -p showcase down
--remove-orphans --volumes` against the LIVE default stack,
destroying the PocketBase named volume. The reserved name now gets
the same treatment as the path-traversal guard: warn (naming the
record and why it is dangerous) and leave the slot intact for manual
inspection — no compose-down, no state removal.
Call-site enumeration: _reap_isolate_slot's sole caller is
_sweep_isolate_slots, at 3 sites (dead-PID reap, project-recorded/
no-owner reap, age-fallback reap), all passing
"$slot_entry" "$slot_proj" — all three flow through the new guard
identically.
Red-green: the new bats test ("a slot whose project record reads the
RESERVED 'showcase' is left intact...") was run against the UNFIXED
code first and FAILED — the sweep logged "Attempting to reclaim
stale slot 0 (project showcase has no live containers and no
recorded owner)" and reaped the slot. It passes with the guard.
2) .iso-bak restore race: two concurrent runs can both see a stale
backup; the loser's mv is the FINAL command of its `[ -f ] && mv`
AND-list, so its failure trips set -e and kills the CLI pre-claim
with a raw error. Both mv's now carry `2>/dev/null || true` — the
survivor's restore wins, the loser proceeds with restored originals.
3) Keep-test absence regexes greped only the `--project-name <name>
down` spelling; the reaper's own downs use `-p <name> down`, so a
keep-branch regression via the -p form passed undetected. Both keep
absence assertions now match `(--project-name|-p) <name> down`.
Mutation-verified: a temporary -p-form compose-down added to the
keep branch made BOTH broadened tests FAIL; reverted, suite green.
(All other absence assertions use the word-matched generic
`compose ... down` regex, which already covers both spellings.)
4) RUNBOOK.md/DEBUGGING.md contradicted shipped code: the manual
teardown was quoted without --volumes plus notes claiming
`down --remove-orphans` leaves named volumes (the shipped survival
notice and every teardown path include --volumes), and the name rule
was documented as `[a-z0-9_-]+` (actual: starts with [a-z0-9], then
[a-z0-9_-], uppercase normalized with a warn, 'showcase' reserved).
Both updated to the shipped semantics; the now-redundant separate
`down --volumes` snippets removed.
Verification: full `bats showcase/scripts/__tests__/` green (60 tests);
shellcheck on _common.sh shows no new warnings vs baseline
(pre-existing SC2034/SC2115 only, line-shifted).
Previously the EXIT-trap restore_isolation always tore down the isolated
stack, ignoring --keep. Now restore_isolation reads a keep flag (set in
cmd-test.sh when --keep is parsed): when kept it skips compose down, the run-dir
removal, and the slot release, and instead prints a survival notice with the
project, slot, the three offset host ports, and the exact manual teardown
command. The kept stack's live containers keep its slot from being reaped.
Persist the compose project name into each claimed slot dir, and at claim time
reap any slot whose recorded project has no live containers (queried via
docker ps --filter label=com.docker.compose.project). This correctly leaves a
--keep'd stack's slot alone since its containers are still up. The existing
PID/age heuristics remain as a fallback for slots predating the project file.
Move the --isolate slot registry and per-run scratch dir off /tmp (wiped on
reboot, world-writable) to $XDG_STATE_HOME/copilotkit/showcase (slots/ and
runs/<name>). The run dir is now keyed by the finalized project name instead
of the PID so a kept run is locatable for manual teardown. Adds a bats suite
covering the new state-base helper and run-dir location.
When a promote fails, the succeeded-service set is empty, so verify-prod
hits its skip branch (`exit 0`). The GitHub job result is therefore
`success`, and the notify step rendered `verify-prod=success` in the
#oss-alerts Slack message — a misleading green, since prod was never
probed.
verify-prod now exports a `status` output: `success` after a real probe
passes, `skipped` on the empty-CSV skip. notify reads that output (via
the new bats-tested verify-prod-display.sh) instead of the raw job
result, so the Slack line accurately reads `verify-prod=skipped` vs
`success` vs `failure`. A genuine probe failure / contract violation
exits non-zero (job result `failure`, status never written), and the
display falls back to the job result. Slack formatting is unchanged.
Extracts the display mapping into showcase/scripts/verify-prod-display.sh
(mirroring promote-fleet.sh) with red-green bats coverage, and adds it to
the showcase_validate.yml shellcheck step.
The pool-fleet worker's Railway service is named `harness-workers` (PLURAL),
but the SSOT keyed it `showcase-harness-worker` (singular). The image-ref gate
matches SSOT keys to Railway service names verbatim, so the gate reported
`harness-workers` as an untracked Railway service AND the stale singular key
matched nothing. Rename the SSOT key (and every test/fixture reference) to the
exact Railway name `harness-workers`.
It stays the staging-only, domainless, probe-disabled worker that runs the
shared `showcase-harness` image: serviceId c2aa8a0b-…, staging instance
362c1e37-…, ciBuilt:false, gateIgnore:true, no build slot (so no dispatchName),
single `staging` env with no domain. Add a focused test pinning that shape.
Counts are unchanged (29 services / 26 CI_BUILT) — this is a rename, not an
addition; both harness workers already existed on main.
Verified LOCALLY against Railway: verify-railway-image-refs reports
`54 env-scoped instances verified (2 skipped)` — 0 violations, 0 missing, 0
untracked (harness-workers reconciled, harness-workers + harness-legacy the 2
gateIgnore'd skips). emit --check zero drift, Ruby parity green (borrowed
.up.railway.app host is parity-excluded), full scripts suite + typecheck green.
Replace ServiceEntry's parallel prodInstanceId/stagingInstanceId/domains/probe/
repoNameOverride fields with a single environments: Record<string, {instanceId,
domain?, probe?, repoName?}> map plus a hoisted env-independent probeDriver.
EnvName becomes an open string backed by an ENV_ID_BY_NAME registry so
accessors resolve arbitrary env names; a single-env service (the staging-only
showcase-harness-worker) now simply omits the absent env instead of carrying a
placeholder ID/borrowed host.
Accessors instanceIdFor/domainFor/repoNameFor index environments[env]
(domainFor still throws on missing/scheme); add envsFor(name),
serviceEnvPairs(), and probeEnabled(name, env). Generalize the image-ref gate
(iterate each service's declared environments, resolve env-id via the
registry, sum/iterate missingByEnv over registry env names) and verify-deploy's
host->env reverse-map (envForTarget iterates environments).
Pure TS-internal: emit-railway-envs-json.ts projects the env-map back onto the
FROZEN legacy JSON shape (prodInstanceId/stagingInstanceId/domains/probe/
repoNameOverride) via a documented legacyJsonCompat shim for the two domainless
harness workers, so railway-envs.generated.json stays byte-identical and Ruby
(bin/railway) + workflow jq + the parity test are untouched. Verified: emit
--check zero drift, Ruby test_expected_domains_parity green, golden snapshot
toEqual proves byte-identical resolution for every real (service, env) pair,
full scripts vitest suite green, showcase scripts typecheck clean.
Serializes the fully-resolved {service -> env -> {instanceId, domain, probe,
driver, repoName}} projection for all 29 services x 2 envs via the public
accessors (instanceIdFor/domainFor/repoNameFor) + per-entry probe config,
frozen as a fixture. This is the behavior-preservation guard for the
forthcoming env-map (Option C) refactor: resolved values must stay
byte-identical before and after.
Move gen-ui-interrupt + interrupt-headless from features: to
not_supported_features: across affected integration manifests, and align
the generate-registry/generate-catalog scripts tests to the resulting
wired-feature counts (derive expected lengths from the parsed manifest
rather than hardcoding pre-quarantine numbers).
The showcase_build verify-image-refs gate (SSOT = showcase/scripts/railway-envs.ts)
was failing with "1 untracked Railway services" because the interim
harness-legacy staging service (the legacy all-probe harness kept live during
the pool-fleet migration) exists on Railway but had no SSOT entry. That
Railway->SSOT drift check skips the build, so nothing deploys.
Adds a harness-legacy SERVICES entry mirroring the showcase-harness-worker
precedent (PR #5280): ciBuilt:false (not built by showcase_build, runs a pinned
out-of-band digest) and gateIgnore:true (deliberately-untracked for the image-ref
gate). findUntrackedServices treats any SSOT entry as known, so this clears the
untracked failure; gateValidated:false keeps findMissingServices from flagging
it. Real serviceInstance IDs for both envs recorded from Railway GraphQL.
Regenerates railway-envs.generated.json and updates the service-count /
gate-ignored carve-out assertions (28->29 services).
Verified: live verify-railway-image-refs.ts now exits 0 ("54 env-scoped
instances verified, 2 skipped"); without the entry it exits 1 with the
harness-legacy untracked failure. Full scripts test suite green (1771 passed).
Jordan Ritter