declarative-gen-ui moved to the CopilotKitMiddleware auto-A2UI path across
the 3 langgraph integrations. The middleware's inner forced tool is
render_a2ui, so each integration's gen-ui-declarative.json gained 4
render_a2ui fixtures that share match keys with the pre-existing render_a2ui
entries in that integration's render-a2ui.json (the a2ui_fixed demo). 4
pills x 3 integrations = 12. Same-context cross-demo overlap, disambiguated
at runtime by the probe fixtureFile.
langgraph-fastapi bumped ag-ui-langgraph[fastapi] 0.0.35->0.0.37 (auto-A2UI
needs get_a2ui_tools). Both non-exact, so the FAIL count stays 63 but the
FAIL-set hash shifted. Update validatePinsFailHash to the new set.
the per-service promote loop ran under `set -euo pipefail`, so the first failing
service aborted the whole `all` fleet promote; extracted to promote-fleet.sh
which attempts every service, accumulates succeeded/failed sets, exits non-zero
only after attempting all, and exports succeeded_csv. verify-prod now runs
`if: !cancelled()` and scopes --services to the succeeded set; the staging
precondition is advisory (promote runs even when it reports red — bin/railway
enforces staging-green per-service); notify success keys on PROMOTE && PROD.
Adds a shell-script-tests CI job (bats + shellcheck) and input-validation
hardening (fail-loud on empty/all-empty CSV, RAILWAY_BIN check, whitespace trim).
Add showcase/scripts/sync-promote-service-options.ts: generates the
promote workflow's service `choice` options from the SSOT
(railway-envs.ts), spliced between BEGIN/END markers in
showcase_promote.yml. Fail-loud throughout — every emitted token must
resolve to exactly one service under the resolve-step predicate
(name|dispatchName match AND probe.prod), tokens are YAML-safe, args are
strict (a typo'd flag cannot trigger a destructive write), and markers
are validated before any rewrite.
Wire it into a lefthook pre-commit hook (regenerate + restage; set -e so
a failed regen blocks the commit) and an advisory (never-failing) drift
check in showcase_validate.yml. Vitest coverage for ordering, exclusion,
collision/ambiguity guards, marker errors, exit codes, idempotency, and
the import-side-effect guard.
The gen-ui-headless-complete probe
(showcase/harness/src/probes/scripts/d5-gen-ui-headless-complete.ts)
references fixtureFile "gen-ui-headless-complete.json", but that file
existed in no D6 slug, so the probe's first leg 503'd under strict.
Add gen-ui-headless-complete.json for all 18 D6 slugs, modeled on the
green langgraph-python headless-complete.json pattern: the four gen-UI
pills (weather/stock/highlight/revenue) with narration (toolCallId)
fixtures FIRST and toolcall (userMessage+context) fixtures AFTER, no
turnIndex gate, so every one of the probe's four sequential turns in one
chat thread matches regardless of prior assistant/tool history. Interrupt
fixtures are intentionally omitted (interrupt-headless is a separate
cluster item).
These 8 fixtures per slug share match keys with the pre-existing
headless-complete.json for the same context (the demos share pills and
are disambiguated at runtime by probe path), which raises the
aimock-fixtures collision-detection exact-duplicate count by 46. Bump
KNOWN_DUPLICATE_CEILING 230 -> 276 to match, consistent with how prior
per-integration feature fixtures bumped the baseline; the substring-shadow
ceiling is unchanged (no new shadows).
Validated: all 732 aimock-fixtures schema/collision tests pass, and a
local aimock --strict run returns 200 for each of the four pills across
turns (langgraph-python + pydantic-ai spearheads, plus spring-ai).
Count is correct at 104 (down from 106 — net improvement from the
@copilotkit 1.59.2 exact-pin + manifest highlight fixes). The prior hash
was computed in a local env whose FAIL-line set differed from CI's
canonical pipeline; sync to CI's printed actual hash.
Previously pydantic-ai and claude-sdk-typescript used 'latest' for all
@copilotkit/* dependencies (and the @copilotkit/web-inspector pnpm
overrides on @copilotkit/core), which the showcase validate-pins ratchet
counts as non-exact pin drift. Intended design is to pin to exact 1.59.2.
Changes:
- integrations/pydantic-ai/package.json: @copilotkit/{a2ui-renderer,react-core,runtime,shared,voice} 'latest' -> '1.59.2'; npm + pnpm overrides on @copilotkit/web-inspector>@copilotkit/core 'latest' -> '1.59.2'
- integrations/claude-sdk-typescript/package.json: same set as above
- Regenerated both package-lock.json files via npm install --legacy-peer-deps --package-lock-only
- scripts/fail-baseline.json: ratcheted DOWN validatePinsFailCount 106 -> 104; updated validatePinsFailHash to dde7950e8d691de5a7b2c0c16ca64b3e550221cb6072d2c29e24dcb497515cf6 (matches local sort-uniq + shasum-256 of the new [FAIL] set)
Verified locally: npx tsx validate-pins.ts reports Summary FAIL=104
(2 fewer than baseline because pydantic-ai's @copilotkit/react-core and
@copilotkit/runtime moved from 'latest' (non-exact) to '1.59.2' (exact);
CST already had unrelated FAILs that remain). Non-@copilotkit deps
(lucide-react, cmdk, openai, @ag-ui/*) intentionally left unchanged.
FAIL count held at 106; only the rendered text of one FAIL line changed when
langgraph-python bumped copilotkit 0.1.92->0.1.93. Routine hash rotation, no
pin changes.
## Status
WIP / not ready to merge. Preserves in-flight D6 work so it isn't lost
mid-rollout. LGP is fixture-complete; other integrations are
mid-rollout.
### Latest banked work
- **langgraph-python — 185 / 0 / 2** (green). Achieved by narrowing a d4
chat matcher that was shadowing the d6 beautiful-chat search_flights
fixture (load order is shared -> d4 -> d6, first-match-wins) plus
refreshing the d6 tool-rendering and tool-rendering-custom-catchall AAPL
fixtures (turnIndex:0 -> hasToolResult:false so the first leg fires in
multi-pill threads). 2 skips are the by-design mcp-apps iframe gap.
- **langgraph-typescript — 185 / 0 / 2** (green). Mirrored the LGP d4
narrowing on the LGT side (3 matchers) and across the
d6/langgraph-typescript suite: replaced fragile turnIndex:0 gates with
hasToolResult:false, fixed em-dash escaping that broke literal matches
in multi-pill threads, and added jsFunctions payloads to the three
sandboxed-ui fixtures (`_from-feature-parity`, `headless-complete`,
`gen-ui-open-advanced`). The final fix removed the chain-tools
`hasToolResult` match gate (which checked the whole thread and made the
chain pill fall through to the broad weather matcher mid-thread),
mirroring LGP.
- **google-adk: 174/7/2 (was 134/52/5)** — conveyance + test-parity +
fixtures + pill-wiring rebuild; 7 residual (6 default-catchall framework
default-renderer testid version question, 1 beautiful-chat fixture).
- **Pill-parity staged across 13 integrations** (ag2, agno, mastra,
pydantic-ai, claude-sdk-python, claude-sdk-typescript, llamaindex,
langroid, strands, spring-ai, built-in-agent, crewai-crews,
langgraph-fastapi). The canonical LGP suggestion pill set is now
mirrored as `src/app/demos/*/suggestions.ts` files in each integration,
with targeted edits to existing `open-gen-ui-advanced` and
`byoc-hashbrown` files. **These new files are currently UNWIRED** — each
integration's `page.tsx` still defines its pill list inline via
`useConfigureSuggestions`. Banked so the canonical source survives; a
follow-up will rewire `page.tsx` to import from `suggestions.ts` and
drop the inline copies.
- **Fleet test-parity sweep**: 576 e2e specs across 15 integrations
aligned to LGP canonical (SHA-verified); 2 orphan specs removed.
- ms-agent-dotnet 177/5/7, ms-agent-python 174/11/2 (post
fixture-mirror); default-catchall green (page-level renderer, not
react-core-gated).
## Scope
### Conveyance (foundation)
Inbound `x-aimock-context` (and friends) must ride along on outbound LLM
HTTP calls so aimock fixture matching sees the inflight test's context.
Without this the call lands on the default project's aimock and silently
picks the wrong fixture. New per-integration
`_header_forwarding.{py,ts}` shim plus matching `agent_server` / route /
factory wiring covers: ag2, agno, built-in-agent, claude-sdk-python,
claude-sdk-typescript, crewai-crews, google-adk, langgraph-fastapi,
langgraph-python, langgraph-typescript, langroid, llamaindex, mastra,
ms-agent-python, pydantic-ai, strands.
For ADK/Gemini the global httpx hook is installed BEFORE any `agents.*`
import (google-genai constructs its client at module-import time).
### langgraph-python — 185 / 0 / 2
Fixture-complete via the conveyance shim + refreshed d6/langgraph-python
fixtures + copilotkit 0.1.93 bump + the latest d4-matcher-narrowing fix
(see banked work above).
### Per-integration fixtures
Mid-rollout snapshot of d6 fixtures across the cohort plus narrowing of
`aimock/shared/common.json`'s generic 'hello' fixture to 'hello world'
so it no longer shadows D6 pills whose prompts contain 'hello' as a
substring.
### Harness `--isolate` patch
`scripts/cli/_common.sh apply_isolation` now rewrites compose-file
relative paths to absolute (build/context/dockerfile/volumes/env_file),
enforces the docker compose `[a-z0-9_-]` project-name rule, and exports
`SHOWCASE_COMPOSE_FILE` / `SHOWCASE_INFRA_PORT_OFFSET` plus offset host
URLs. The TS harness CLI (`aimock-rebuild` / `config` / `doctor` /
`lifecycle`) honors the new env so concurrent isolated stacks stop
reporting each other's services as healthy.
## Lockfile decision flagged
`showcase/integrations/langgraph-python/pnpm-lock.yaml` was deleted in
this branch. Decision: keep the deletion. Rationale:
- 03bed3b76 (fix(showcase): regenerate 18 lockfiles in isolation; switch
to npm ci) migrated all showcase integrations off pnpm onto npm ci.
- The integration's Dockerfile uses `npm ci --legacy-peer-deps`.
- Every sibling integration committed only `package-lock.json` after
03bed3b76.
- The orphan pnpm-lock.yaml only risks tooling drift.
If anyone wants it restored: `git checkout origin/main --
showcase/integrations/langgraph-python/pnpm-lock.yaml`.
## Commits
- feat(showcase): D6 conveyance — forward x-aimock-context headers to
LLM clients
- feat(showcase/langgraph-python): D6 conveyance shim + copilotkit
0.1.93 bump
- feat(showcase/langgraph-typescript): D6 conveyance — propagate request
headers into ChatOpenAI
- feat(showcase/built-in-agent): D6 conveyance — header-forwarding shim
+ factory wiring
- feat(showcase/harness): support concurrent --isolate runs
- test(showcase): D6 langgraph-python fixtures — drive to 180/5/2
- test(showcase): D6 per-integration aimock fixtures + shared narrowing
- docs(showcase): GOTCHAS entry for D6 conveyance + --isolate notes
- feat(showcase): D6 conveyance — wire header-forwarding shims into
remaining entrypoints
- fix(showcase): unblock LGP D6 beautiful-chat + custom-catchall via d4
matcher narrowing
- fix(showcase): narrow LGT D6 d4 shadows + wire sandboxed-ui
jsFunctions
- chore(showcase): copy LGP canonical suggestion pills into 13
integrations
- fix(showcase): forward x-aimock-context per-request in google-adk
routes
- test(showcase): align google-adk e2e specs to langgraph-python
canonical
- fix(showcase): align google-adk D6 fixtures to LGP contract
- fix(showcase): wire google-adk default-catchall to shared 4-pill
suggestions
Memoize getToken so the Railway config is not re-read and the deprecation
warning is not re-emitted on every GraphQL request; failures are never cached.
Resolve the GHCR username with || so an empty GHCR_USERNAME falls through to
GITHUB_ACTOR, and use a truthy cache-hit guard so an empty token is never
memoized.
Reject ASCII control chars, a colon (port suffix), and any char outside the
DNS-label charset; switch the Host brand from a string-literal __brand to a
non-exported unique symbol so a stray `as Host` cast from outside the module
is a type error. asHost stays the sole runtime constructor. Adds positive +
negative test coverage incl. interior-tab rejection.
- scripts/cli/_common.sh apply_isolation rewrites compose-file relative
paths to absolute (build/context/dockerfile/volumes/env_file), enforces
the docker compose [a-z0-9_-] project-name rule (normalize-with-warn or
hard fail), and exports SHOWCASE_COMPOSE_FILE / SHOWCASE_INFRA_PORT_OFFSET
plus offset host URLs (AIMOCK_URL_LOCAL / DASHBOARD_URL_LOCAL /
POCKETBASE_URL_LOCAL) so the TS harness CLI talks to THIS project's
aimock instead of the default :4010
- harness/src/cli/{aimock-rebuild,config,doctor,lifecycle}.ts honor the
new env vars; lifecycle picks up the offset infra port for health
probes so concurrent stacks no longer report each other's services as
healthy
- Commit a generated harness/package-lock.json (new file) so npm ci
resolves deterministically
Replace the divergent getToken with resolveRailwayToken; source project/env IDs from the
railway-envs SSOT; read the GHCR username from env (trimmed, fail-loud) instead of a hardcoded
handle.
Introduce a branded Host produced by asHost (rejects scheme/path/empty/whitespace/userinfo/query/
fragment); brand at the resolveProbeTargets ingress; make ProbeTarget fields readonly; add asHost
+ override-seam tests.
deploy.yml moved SSOT consumption into resolve-verify-matrix.ts; the test now asserts
deploy.yml invokes that script AND the script reads railway-envs.generated.json + filters
probe.staging===true, rather than scanning the YAML for literals that moved one hop down.
Non-functional cleanup pass on the showcase deploy-pipeline integration
branch. All changes are scoped to comment rot, log severity for already-
demoted runtime-config fields, length-aware env-name coalescing (a
deliberately-empty primary no longer masks a populated alternate), and
test-quality tightening. No production behavior change beyond the
specific items below.
Changes by area:
- shell/shell-dashboard/shell-docs runtime-config.ts: factor the
`process.env[primary] ?? process.env[alt]` chain into a shared
length-aware `readEnvPair` helper. The prior `??` form treated
`PRIMARY=""` as set, masking a populated alternate; the helper now
treats empty-string as unset and falls through to the alternate.
- shell-docs runtime-config.ts: demote the two recoverable URL fields
(`intelligenceSignupUrl`, `posthogHost`) from console.info to
console.warn. The `FATAL-CONFIG:` Sentry-alert prefix is preserved
only on the true sentinels; the demoted fields now clear prod log-
aggregation thresholds without raising ops alerts.
- All three shells' runtime-config.ts: prefix log lines with the shell
name (e.g. `[shell-docs runtime-config]`) so the shared log stream
identifies which shell emitted the line.
- shell-docs runtime-config-serialize.ts: rewrite the U+2028 / U+2029
RegExp arguments using six-character ASCII backslash-u escape
sequences (was: literal codepoints in the string arg). The literal
codepoints are line terminators that a formatter or editor could
silently strip, breaking the security-critical XSS escape. The
ASCII form is robust to any such pass.
- shell-docs use-google-analytics.test.ts: de-tautologize the hook-
order test. It now asserts `usePathname(` and `useEffect(` both
exist in the source, so deleting all hooks would fail the test
rather than trivially satisfying the early-return path.
- shell-dashboard baseline-types.test.ts: update the partner-count
expectation from 25 to 26 -- the 26th entry (Cloudflare) is a
legitimate integration that landed independently; the test was
stale and had nothing to do with this branch.
- scripts/resolve-verify-matrix.ts: drop the `FIX 7 --` plan-
internal prefix from a comment; keep the explanation.
- shell-docs/.env.example: correct the `NEXT_PUBLIC_SHELL_URL`
fallback claim (sentinel, not canonical prod host) and document
the remaining 7 consumed env vars with their FATAL/warn/silent
semantics so the example matches runtime-config.ts.
Skipped:
- C-SENTINEL-DEDUP (`http://ops.invalid` shared constant across
shell-dashboard's next.config.ts and runtime-config.ts): both
files are at different module levels (root vs src/lib) and the
string appears once in each; extracting to a shared module would
widen the diff into a refactor for marginal benefit. Skipped per
the spec's "if it widens diff awkwardly, skip" guidance.
- C-SSRTEST: already exhaustively covered. Each of the three shells
has an SSR placeholder test that exercises every URL field via
`new URL()` parseability and (for shell-docs) the analytics-key
empty-string semantics. Treated as a no-op.
Validation: shell + shell-dashboard + shell-docs runtime-config /
serialize / GA tests green; bin/showcase Ruby suite green (87 runs);
showcase/scripts resolve-verify-matrix + aggregate-build-results +
lint-rule-no-public-env green (79 runs).
Closing hardening pass on the showcase deploy-gate's verify-matrix
resolver. The 7-agent review confirmed the gate is correct; this
commit fixes the residual rough edges.
- showcase_deploy.yml: correct the false §3 ok-non-empty comment.
The empty-intersection case can coexist with redeploy_red=false
(every redeploy succeeded, just none probe-eligible) — that's a
correctly-green run, not a red one.
- showcase_deploy.yml: tighten the summary.json shape guard to catch
PARTIAL drift (TOTAL>0 && WITH_STATUS<TOTAL). The previous all-or-
nothing TOTAL>0 && WITH_STATUS==0 check silently dropped drifted
rows on a mixed summary. Validated locally on mixed/normal/empty/
total-drift jq samples.
- resolve-verify-matrix.ts: add asSupportedEventName narrowing helper
+ use it in the CLI. Replaces the unchecked `as` cast — type system
and runtime now tell one story. Resolver's internal eventName
guard becomes defense-in-depth for direct (test) callers.
- resolve-verify-matrix.ts: make the workflow_run boundary total —
summaryPresent MUST be exactly "true"/"false". Any other value
(including "" from a step-id-rename wiring break) throws now
instead of silently emitting has_services=false.
- resolve-verify-matrix.ts: drop the try/catch around
fileURLToPath(import.meta.url) in `invokedDirectly`. The catch
used to swallow ESM-interop failures and silently no-op the CLI
(exit 0, no GITHUB_OUTPUT write → verify skipped = false-green).
- resolve-verify-matrix.ts: reword parseSsotServices JSDoc to
distinguish schema-drift from truncation (the two are different
failure modes, not one conflated story).
- showcase_build.yml: comment addendum on the redeploy-summary
upload — swapping the guard to `if: always()` would red the
legitimate services=='' path (no summary written), trading the
already-closed false-green for a false-red on every non-buildable
push.
- resolve-verify-matrix.cli.test.ts: switch to spawnSync so stderr
is captured on both zero and non-zero exit (execFileSync only
exposes stderr on throw). Hard-code two stable probe-eligible
names ("aimock", "harness") for the sorted-CSV test rather than
picking probe[0]/probe[1] off the live SSOT — the prior test was
tautological (already-sorted in, sorted out) and would silently
pass if the resolver did nothing.
- resolve-verify-matrix.cli.test.ts: add CLI coverage for the
dropped-token ::warning:: path (FIX 3 — the entire drift-detection
contract had zero CLI coverage), the unexpected-EVENT_NAME error
(FIX 5), and the workflow_run-summary_present total boundary
(FIX 7, both "" and "True" inputs).
- resolve-verify-matrix.test.ts: add unit coverage for the new
workflow_run summaryPresent boundary (empty + "True" + the
workflow_dispatch ignores-summaryPresent regression).
Red-green: 6 tests RED before code changes (FIX 3 warning, FIX 5
unknown EVENT_NAME, FIX 7 unit + CLI ×2 for "" and "True"); 79
tests GREEN after.
Validation: 4 vitest files / 79 tests passing; 87/87 ruby specs
passing; actionlint findings unchanged vs integration baseline
(8 → 8, identical diff); yaml.safe_load OK on both workflows.
A 7-agent review of the verify-matrix resolver and its surrounding workflow plumbing found three
boundary surfaces that could silently produce a GREEN deploy on a broken release, plus an
untested CLI contract that CI compares against the literal strings 'true' / 'false'.
FIX 1 — Validate the SSOT shape in loadSsotServices(). The prior `JSON.parse(...) as
{services: SsotService[]}` was an unchecked cast: a truncated/drifted SSOT (emitter crashed
mid-write, or schema renamed) parses fine but silently shrinks/empties the probe-eligible set
→ some redeployed services go unverified, or verify is skipped on a real redeploy. Extract a
pure exported parseSsotServices(raw, path) that requires the shape we depend on (non-empty
services array; each entry has a non-empty string name, an optional string|null dispatchName,
and a probe object with a boolean staging). Throw `::error::SSOT <path> malformed: <detail>`
on any violation. Also re-check existsSync(SSOT_JSON) after the regenerate-if-missing
execFileSync — a regen that exits 0 without writing must not proceed to a useless JSON.parse
crash. Drop the defensive `probe?.staging` once shape is guaranteed.
FIX 2 — Validate summary.json shape in the redeploy-gate bash. The bullseye false-green
surface: if redeploy-env.ts's schema ever drifts (e.g. `status` → `state`, `ok` → `success`),
every `jq select(.status==...)` yields empty → redeploy_red=false AND ok_services="" →
resolver skips verify → GREEN CI on a real unverified redeploy. Add a TOTAL vs WITH_STATUS
shape guard right after loading the summary: if TOTAL > 0 && WITH_STATUS == 0, emit
::error::summary.json has $TOTAL entries but none with status ok|error (schema drift?) and
exit 1. The legitimate empty-array path (TOTAL=0) is preserved.
FIX 3 — Fail loud on unknown eventName in resolveVerifyMatrix. The prior code fell through to
the workflow_run intersection branch for ANY unrecognized eventName (typo, unexpected
trigger), silently emitting has_services=false → indistinguishable from a legit "summary
absent" skip. Add an explicit guard so only workflow_run / workflow_dispatch are accepted;
anything else throws ::error::resolve-verify-matrix: unexpected eventName '<value>'. Tighten
the eventName parameter type to the literal union.
FIX 4 — Trim ok tokens + warn on dropped tokens in okCsvToCanonicalNames. Split, then
.map(t => t.trim()).filter(Boolean) so "a, b" (spaces) matches. Collect tokens that match NO
SSOT service (by name or dispatchName) and have the CLI wrapper emit ::warning::ok_services
tokens dropped (no SSOT match): <list> on stderr when non-empty — surfaces SSOT/build drift.
The pure function stays IO-free; logging lives in the wrapper.
FIX 5 — CLI wrapper integration test. New resolve-verify-matrix.cli.test.ts spawns
`npx tsx showcase/scripts/resolve-verify-matrix.ts` with a temp $GITHUB_OUTPUT file across
four scenarios and asserts the temp file contents EXACTLY (the workflow YAML compares
has_services against the literal strings 'true'/'false', so the byte-for-byte format is part
of the contract). Uses the real railway-envs.generated.json so the loader exercise is real.
FIX 6 — Cleanup. Remove the dead `env: DISPATCH_SERVICE: ...` block on the redeploy-gate
step (the next step redeclares it — leftover from the extraction). Soften the §3
decision-table all-errors bullet to match resolve-verify-matrix.ts's careful wording, and
append that when the success-set is empty (or the intersection collapses to empty), verify
is skipped and the gate reds independently. Append to showcase_build.yml's "Upload redeploy
summary" path-(A) comment that `if-no-files-found: error` still reds path (A) even if a
future change adds `if: always()`.
Tests: red→green for FIX 1/3/4/5 verified locally. Resolve-verify-matrix vitest count:
12 → 28. Full requested suite (resolve-verify-matrix + cli + aggregate-build-results +
lint-rule-no-public-env): 72 passed. showcase/bin ruby specs: 87 runs / 0 failures / 0
errors / 0 skips. actionlint baseline preserved (8 findings, identical to integration tip).
Extract the inline bash+jq decision logic from showcase_deploy.yml's
resolve-matrix job into showcase/scripts/resolve-verify-matrix.ts, a
pure function with a vitest suite. The bash had produced two confirmed
bugs across prior CR rounds, so making it testable is the lasting fix.
Issue A (the bug this PR fixes): when summary_present=true but
ok_services is empty (every service errored on redeploy), the old bash
skipped the intersection and fell through to the full probe-eligible
fleet, gratuitously probing every service against stale :latest. The
resolver now returns has_services=false in that case — enforce-redeploy
-gate independently reds the workflow on redeploy_red=true, so this
case is already loud; there is nothing left to verify.
Parity preserved for unchanged cases:
- workflow_dispatch + 'all'/empty → full probe-eligible set
- workflow_dispatch + specific svc → that one (unknown → error exit)
- workflow_run + summary_present=false → has_services=false
- workflow_run + present + ok non-empty → intersection with probe-
eligible (SSOT key OR dispatchName aliases both resolve)
Also clarified the Upload-redeploy-summary comment in showcase_build.yml
to document both red paths (hard crash → redeploy step exits non-zero;
exit-0-but-no-file → if-no-files-found:error reds the step) so no
false-green path is possible.
Tests: 12-case vitest suite covers each decision-table row plus the
Issue A fix (written red-first; failed against a naive full-fleet
fallback, passed once the early return was added). CLI parity verified
against the real generated SSOT for the three representative env-var
combinations (workflow_run + present + ok=[a,c]; workflow_run + present
+ ok empty; workflow_dispatch + 'all').
D1 — showcase_deploy.yml false-red fix
======================================
The build workflow legitimately uploads no `redeploy-summary` artifact when it ran
(push touched `showcase/**` so `paths:` matched) but `detect-changes` found no
buildable service, so `redeploy-staging` was skipped. The build still concludes
`success`, so `showcase_deploy.yml` fires on `workflow_run` and `resolve-matrix`
runs. `actions/download-artifact@v4` with `name:` HARD-FAILS on a missing
artifact, so the unguarded download was failing the job, and a downstream guard
that trips `enforce-redeploy-gate` on `resolve-matrix.result == 'failure'` was
flipping the workflow RED — a false-red on a routine showcase-docs/script change.
Add an artifact-existence pre-check using `actions/github-script` (pinned by SHA,
matching the existing repo convention) that lists the artifacts for
`workflow_run.id` via `actions: read` (already granted to `resolve-matrix`) and
sets `summary_present=true|false`. Gate the existing download step on
`summary_present == 'true'`. Keep NO `continue-on-error`, so the C1 property
holds: when the artifact exists but the download genuinely fails, the job still
fails loud and `enforce-redeploy-gate` correctly reds the workflow. When the
artifact is legitimately absent, the bash gate's existing `[ ! -f "$SUMMARY" ]`
branch no-ops (`redeploy_red=false`, `ok_services=""`) — nothing was
redeployed, so there is nothing to gate.
Updated the step comment block to enumerate the three distinct cases now
handled: workflow_dispatch (no download); workflow_run + artifact absent
(graceful skip); workflow_run + artifact present (download with fail-loud).
L1-L5 — env lint rule hardening
===============================
- L1: route the destructuring (VariableDeclarator/ObjectPattern) branch through
the shared `staticKeyName()` helper so the computed-string-key form
`const { ["NEXT_PUBLIC_X"]: y } = process.env` and the no-expression
template-literal form `const { [\`NEXT_PUBLIC_X\`]: y } = process.env` are
caught with the same parity as the bracket-member read.
- L2: unwrap a wrapping `ChainExpression` at the top of `isProcessEnv()` so
`process.env?.X` is matched robustly across parser flavors; corrected the
helper's doc comment to describe the actual semantics.
- L3: export `BANNED_KEYS` from the rule module and have the table-driven test
dynamically import the rule's own Set instead of hand-mirroring it — the
test set now cannot drift from the rule.
- L4: added override-scoping fixtures for `showcase/shell/src/**` and
`showcase/shell-dojo/src/**`; the `.oxlintrc.json` override list already
includes these, but the test now exercises them so an accidental drop is
caught.
- L5: expanded the file-header "Out of scope" doc list to include bulk-iteration
reads (`Object.keys/values/entries(process.env)`, for-in, spread
`{...process.env}`), rest-pattern destructuring, compound-assignment LHS, and
update operators. Documentation-only — the deliberate non-coverage is now
auditable.
Validation
==========
- RED→GREEN confirmed for L1 (two new destructuring computed-key tests) and L3
(dynamic `await import(...)` of BANNED_KEYS failed pre-fix with
"Rule module did not export a non-empty BANNED_KEYS Set", green after export).
- vitest: 38 passed (was 34 baseline + 4 new); aggregate-build-results 6 passed.
- Ruby promote suite: 87 runs, 251 assertions, 0 failures (unchanged).
- python3 yaml.safe_load: showcase_deploy.yml + showcase_build.yml +
showcase_promote.yml all parse OK.
- actionlint: zero NEW findings on the changed file. The pre-existing
showcase_build.yml SC2086/SC2129/runner-label findings are identical on the
integration baseline (unchanged by this commit).
Seven-agent CR surfaced correctness defects in the build/deploy/promote
pipeline and in the no-public-env-shell-read oxlint rule. This commit
closes the false-green paths and broadens lint coverage.
Workflow fixes:
- showcase_deploy.yml: drop `continue-on-error: true` on the redeploy-summary
artifact download. The dispatch path is already guarded by the `if:
workflow_run` clause, so the bash "no summary" branch handles legitimate
manual dispatches. A genuine workflow_run download failure must now fail
loud instead of silently widening verify to the full service set against
stale `:latest`.
- showcase_build.yml: redeploy-staging now intersects the build matrix with
the aggregator success set (`needs.aggregate-build-results.outputs.results`,
status == "success") before producing the redeploy CSV. Failed/skipped
slots no longer get redeployed (which would just re-pull stale `:latest`
and look healthy).
- showcase_build.yml: `notify-all-builds-failed` now additionally requires
`needs.build.result == 'failure'` so it doesn't Slack-spam when the build
job was SKIPPED (verify-image-refs upstream failure).
- showcase_build.yml: `notify` now lists [build, aggregate-build-results,
redeploy-staging] in `needs:` so aggregator/redeploy failures still emit
a Slack signal. `if: failure()` still skips when none of the needs failed.
- showcase_build.yml: `set -euo pipefail` on the Prepare build args step
so a transient $GITHUB_OUTPUT write failure can't ship images without
COMMIT_SHA/BRANCH baked in.
- showcase_deploy.yml: `enforce-redeploy-gate` now also trips on a
resolve-matrix failure (`needs.resolve-matrix.result == 'failure'`) so
an upstream crash that leaves `redeploy_red` empty can't bypass the gate.
- Doc-comment accuracy: drop stale `(PR #5093)` reference; correct the
env-IDs source-of-truth comment; document the optional `skip_build` field
in ALL_SERVICES; clarify that health_path is informational and verify
uses per-service drivers; add the missing `resolve-targets` step 0 to the
promote workflow's "Order:" header.
Aggregator fix (RED-GREEN):
- aggregate-build-results.ts: throw on zero slot dirs. The job is gated
upstream on has_changes == 'true', so zero slot dirs is a broken artifact
download, not a legitimate empty build set. Silently emitting
any_success=false + results=[] is indistinguishable from "all builds
failed" and lets the deploy workflow fall back to probing the full
service set against stale `:latest`. Refuse the ambiguity.
- aggregate-build-results.test.ts: existing empty-INPUT_DIR test was
updated to assert the throw (was: return []).
Oxlint rule (RED-GREEN):
- no-public-env-shell-read.mjs: handle destructuring reads
(const { NEXT_PUBLIC_X } = process.env and aliased form), template-literal
computed keys (process.env[\`NEXT_PUBLIC_X\`]), and explicitly skip
assignment-LHS / `delete` targets (writes are not reads). Optional
chaining already worked through the existing MemberExpression path.
Aliasing (`const e = process.env; e.X`) is intentionally documented as
out of scope (needs scope tracking). Description sharpened to say the
rule guards a specific banned-key set, not all NEXT_PUBLIC_* reads.
- .oxlintrc.json: tighten the off-override glob from
`showcase/**/*runtime-config*` to
`showcase/**/lib/runtime-config*.{ts,tsx}` so it only silences the
intended implementation files, not arbitrary paths containing that
substring.
- lint-rule-no-public-env.test.ts: rewritten as table-driven coverage of
every BANNED_KEYS entry (dotted + bracket-string forms), every ALLOWED
key (asserting non-firing), all new variants from the rule expansion,
the assignment/delete non-fire cases, and override scoping
(runtime-config exempt; packages exempt; shell-tree non-runtime-config
flagged).
Validation:
- actionlint on all three workflows: 8 pre-existing findings (depot label,
pre-existing SC2086 infos in untouched steps); my edits add zero.
- python3 yaml.safe_load: all three workflows OK.
- vitest aggregate-build-results.test.ts: 6/6 pass (incl. new throw test).
- vitest lint-rule-no-public-env.test.ts: 34/34 pass.
- vitest full showcase/scripts suite: 1654/1654 pass across 46 files.
- ruby showcase/bin/spec/all_tests.rb: 87 runs, 0 failures.
- Intersection jq proof (matrix a,b,c × success a,c) → "a,c"; all-failed
→ ""; skipped status excluded.
A RAILWAY_TOKEN secret with trailing whitespace/newline (common from op
read, heredoc, shell export) was returned verbatim and produced invalid
Authorization: Bearer headers and silent Railway 401s. Trim the env-var
lane and treat whitespace-only as UNSET so the config-file fallback runs.
Also adds a missing should-have-thrown guard in the NO_HOME test and
removes a stale gateIgnore clause from findUntrackedServices docstring.
Distinguish ENOENT (treat as drift) from other read errors in
emit-railway-envs-json.ts --check; non-ENOENT errors now exit 2 with the
real error on stderr instead of being silently coerced into a misleading
'stale' message or an overwrite on a false drift signal.
Add a --out=<path> override so tests can write to a temp directory and
never mutate the tracked railway-envs.generated.json artifact. Rewrite
the emit-railway-envs-json test to use mkdtempSync + --out, switch the
staleness assertion to spawnSync so it asserts on exit code 1 plus the
stale-diagnostic substring, and add coverage for the new EISDIR
fail-loud path. After this change git status is clean post-test.
A token read from ~/.railway/config.json with surrounding whitespace or a
trailing newline passed nonEmpty() but was returned verbatim, so an
'Authorization: Bearer <token>' header could carry CR/LF or stray spaces
(Node HTTP rejects invalid header chars; Railway 401s otherwise). Trim
each return path so the canonical token is always emitted; whitespace-only
values still fall through. Also clarify the JSDoc that this resolver does
not consult process.env.RAILWAY_TOKEN.
- Reject empty/whitespace-only service in parseBuildOutputs,
mergeBuildResultFiles, and buildResultArtifactName (was only
buildResultArtifactName, and only for length===0).
- mergeBuildResultFiles now throws on duplicate service names across
slots so an upstream dispatch-name collision surfaces instead of
letting a failure+success pair spuriously look like a success in
successSet (fail-loud discipline).
- Derive BuildOutcome union and VALID_STATUSES set from a single
BUILD_OUTCOMES as-const tuple plus a compile-time exhaustiveness
assignment so they cannot drift.
- Extract validateServiceBuildResult shared validator used by both
parseBuildOutputs (context: 'entry[i]') and mergeBuildResultFiles
(context: 'slot[i]') — one set of rules, one error format, both
now include the offending index.
- Trim shouldRedeployStaging JSDoc to what/why only (dropped the
external-caller enumeration) and note in the module header that
the single-result.json-per-slot invariant is enforced workflow-side,
not by the parser. shouldRedeployStaging([]) === false behavior
unchanged.
Tests: 16 -> 23 passing, all new red-then-green; tsc 0.
Treat whitespace-only token strings as empty so the resolver falls through
to the next candidate instead of returning a bearer that fails at Railway
with a confusing 401.
Guard against non-object JSON input (null/undefined/string/number/array)
defensively before property access — the config originates from
JSON.parse of ~/.railway/config.json (untrusted).
Replace non-null assertions on re-accessed expressions with
locally-captured values so the nonEmpty narrowing actually applies; no
behavior change for the happy path.
Comment cleanups: replace stale line-number reference with a symbolic
one, reword the Ruby-parity note to acknowledge the per-project token
fallback that is intentionally not honored, drop the rotting
"(43+ chars)" parenthetical, and soften the deprecation timeline to
"a future release."
A.7 rewrote showcase_deploy.yml to be SSOT-driven (no hardcoded options
list, no inline ALL_SERVICES). Webhooks remains canonically wired via
the SSOT (probe.staging:true, dispatchName:"webhooks"); test now pins
that contract instead of the now-obsolete literal regexes.
New TS probe driven off railway-envs SSOT. Accepts --env staging|prod and
optional --services CSV; iterates SERVICES where probe[env]===true and
dispatches to per-driver feature-level verifiers. Refuses to start when a
probe-required service is missing a domain for the requested env (no
silent skip). HTTP 200 is necessary but not sufficient.
Adds verify-deploy.drivers.ts dispatch with exhaustive never check on
ProbeDriver, plus one stub module per ProbeDriver literal (shell, docs,
dashboard, dojo, harness, eval, aimock, pocketbase, webhooks, agent).
Stubs fail loud with an explicit "not yet implemented" error so any
accidental real-network invocation surfaces; per-driver feature-level
impls land as subsequent micro-tasks. Refs spec section 3 / section 3.5.
[BLITZ:A5]
Adds optional REDEPLOY_SUMMARY_JSON path; when set, redeploy-env writes
a structured per-service record array {service,status,error?}. PR #5093's
exit-code contract is preserved (staging=0, prod=1-on-failure).
Consumed by showcase_deploy.yml to fail the workflow on staging per-service
errors without changing the script's exit semantics. Refs spec §3.
Plan-B / Option-B migration moved every shell URL/analytics key off the
build-time NEXT_PUBLIC_* env channel and onto runtime config served via
__SHOWCASE_CONFIG__ + getRuntimeConfig(). To prevent a silent regression
where a future change reintroduces a direct process.env.NEXT_PUBLIC_*
read in shell code (which would re-freeze the value at build time and
break no-rebuild env switching), add a focused lint rule.
The rule (copilotkit/no-public-env-shell-read) is implemented as a
custom oxlint JS plugin rule in the existing copilotkit plugin and
enabled under shell-scoped overrides in .oxlintrc.json:
- Errors on process.env.NEXT_PUBLIC_<URL/ANALYTICS> reads in:
showcase/shell-dashboard/src/**, showcase/shell-docs/src/**,
showcase/shell/src/**, showcase/shell-dojo/src/**
- Banned keys: POCKETBASE_URL, SHELL_URL, BASE_URL, OPS_BASE_URL,
INTELLIGENCE_SIGNUP_URL, POSTHOG_KEY, POSTHOG_HOST, SCARF_PIXEL_ID,
GOOGLE_ANALYTICS_TRACKING_ID, REB2B_KEY, REO_KEY
- Intentionally allowed (NOT banned): NEXT_PUBLIC_COMMIT_SHA and
NEXT_PUBLIC_BRANCH (build-stamped artifact identifiers per B10/B11)
and NEXT_PUBLIC_LOCAL_BACKENDS (computed from shared/local-ports.json
at build, local-dev only).
- Excluded files (rule disabled via a follow-up override): MDX content
under shell-docs/src/content/**, runtime-config implementation files,
and *.test.{ts,tsx} / *.spec.{ts,tsx}. oxlint does not support
excludedFiles inside an override block, so the exclusion is expressed
as a later override that sets the rule to off.
Plan-B originally targeted oxlint's eslint/no-restricted-syntax with an
AST-selector regex. oxlint 1.x does not implement that rule (only
no-restricted-globals / no-restricted-imports), so the equivalent guard
is realized as a small custom rule in the existing copilotkit JS plugin
(meta.name=copilotkit), reusing the same plugin loader the repo already
has for require-cpk-prefix and no-single-arg-zod-record.
Verification (red-green): the rule fires on a fixture containing
process.env.NEXT_PUBLIC_POCKETBASE_URL and does NOT fire on a fixture
containing process.env.NEXT_PUBLIC_COMMIT_SHA. Test pins the config via
-c so it works inside git worktrees nested under .claude/worktrees/
where oxlint's automatic upward config search can miss the worktree's
own .oxlintrc.json.
All four shells lint clean: 0 errors of the new rule across
shell-dashboard (114 files), shell-docs (137), shell (29), shell-dojo (6).
Cover four malformed-ref shapes the gate must reject:
* `:sha256-<hex>` (missing the @ separator — looks like a digest
pin but is actually a tag)
* `@sha256:<too-short-hex>` (truncated digest hex)
* The 2026-04-21 `...atest` corruption shape from the script
docstring (the original reason this gate exists)
* Non-ghcr.io registries on both envs
These match the canonical PROD_SHAPE / STAGING_SHAPE regexes in
verify-railway-image-refs.ts; the tests are the regression guard
that ensures a future "relax the regex" change cannot ship without
explicitly turning these red first.
Lock in shape behaviour for dashboard, docs, dojo, shell, and
harness with explicit per-service red-green cases:
* prod with :latest -> fail (must be @sha256)
* prod with @sha256 on the correct repo -> pass
* staging with :latest on the correct repo -> pass
* staging with @sha256 -> fail (must float on :latest)
* wrong GHCR repo name on either env -> fail
The validateImage body is unchanged — it was always shape-pure and
shape-correct. Before WS-C these five services were never exercised
through the gate at all (gateValidated:false). These tests are the
regression guard that ensures a future edit doesn't accidentally
re-introduce the Phase-2 carve-out without anyone noticing.
Flip dashboard, docs, dojo, shell, and harness from
gateValidated: false to gateValidated: true and simultaneously add
the corresponding repoNameOverride for both envs:
dashboard -> showcase-shell-dashboard
docs -> showcase-shell-docs
dojo -> showcase-shell-dojo
shell -> showcase-shell
harness -> showcase-harness
These two halves MUST land in the same commit. Flipping
gateValidated without the override would make the gate look up
ghcr.io/copilotkit/<railway-name> (e.g.
ghcr.io/copilotkit/dashboard:latest) which does not exist — the
gate would fail on first run. Adding the override without flipping
gateValidated is dead code: main() short-circuits unvalidated
services before consulting the override. Only the union is correct.
Also remove the Phase-2 deferral comments and refresh the
ServiceEntry.gateValidated JSDoc — there are no Phase-2 holdouts
left. All 27 services are now gate-validated; gateIgnore remains
the sole escape hatch and is unused by every current entry.
The Railway -> SSOT direction at verify-railway-image-refs.ts was
warn-and-continue: an out-of-band Railway service with a malformed
ref would not turn CI red as long as nobody read the warning line.
Replace that branch with a hard failure path that lists the
untracked service under its own failure class, with a clear remedy
in the error message (add to SSOT, or set gateIgnore on an existing
entry).
Refactor main() to call two pure helpers (findUntrackedServices,
summarizeFailures) so the policy is unit-testable without going
through Railway GraphQL. The SSOT -> Railway direction
(findMissingServices) is unchanged: it is already correct and is a
separate coverage class.
Adds red-green tests for the summarizeFailures shape, covering the
three failure classes (shape violations, SSOT->Railway drift,
Railway->SSOT drift) and the success path.
Widen `ServiceEntry` with an optional `gateIgnore: boolean` field
(default false / unset) so the image-ref gate can deliberately exclude
a Railway service from BOTH coverage-direction checks. No behavioural
change in this commit; the field is consumed by the next commit which
flips the Railway->SSOT direction from warn-only to hard-fail.
Stub-export `findUntrackedServices` from verify-railway-image-refs so
the unit test for the new direction can compile. Behavioural wiring
into main() lands in the next commit.
Adds verify-railway-image-refs.test.ts with the first unit tests
covering the new field surface and the helper contract.
Ran Shem Tov