Closing hardening pass on the showcase deploy-gate's verify-matrix
resolver. The 7-agent review confirmed the gate is correct; this
commit fixes the residual rough edges.
- showcase_deploy.yml: correct the false §3 ok-non-empty comment.
The empty-intersection case can coexist with redeploy_red=false
(every redeploy succeeded, just none probe-eligible) — that's a
correctly-green run, not a red one.
- showcase_deploy.yml: tighten the summary.json shape guard to catch
PARTIAL drift (TOTAL>0 && WITH_STATUS<TOTAL). The previous all-or-
nothing TOTAL>0 && WITH_STATUS==0 check silently dropped drifted
rows on a mixed summary. Validated locally on mixed/normal/empty/
total-drift jq samples.
- resolve-verify-matrix.ts: add asSupportedEventName narrowing helper
+ use it in the CLI. Replaces the unchecked `as` cast — type system
and runtime now tell one story. Resolver's internal eventName
guard becomes defense-in-depth for direct (test) callers.
- resolve-verify-matrix.ts: make the workflow_run boundary total —
summaryPresent MUST be exactly "true"/"false". Any other value
(including "" from a step-id-rename wiring break) throws now
instead of silently emitting has_services=false.
- resolve-verify-matrix.ts: drop the try/catch around
fileURLToPath(import.meta.url) in `invokedDirectly`. The catch
used to swallow ESM-interop failures and silently no-op the CLI
(exit 0, no GITHUB_OUTPUT write → verify skipped = false-green).
- resolve-verify-matrix.ts: reword parseSsotServices JSDoc to
distinguish schema-drift from truncation (the two are different
failure modes, not one conflated story).
- showcase_build.yml: comment addendum on the redeploy-summary
upload — swapping the guard to `if: always()` would red the
legitimate services=='' path (no summary written), trading the
already-closed false-green for a false-red on every non-buildable
push.
- resolve-verify-matrix.cli.test.ts: switch to spawnSync so stderr
is captured on both zero and non-zero exit (execFileSync only
exposes stderr on throw). Hard-code two stable probe-eligible
names ("aimock", "harness") for the sorted-CSV test rather than
picking probe[0]/probe[1] off the live SSOT — the prior test was
tautological (already-sorted in, sorted out) and would silently
pass if the resolver did nothing.
- resolve-verify-matrix.cli.test.ts: add CLI coverage for the
dropped-token ::warning:: path (FIX 3 — the entire drift-detection
contract had zero CLI coverage), the unexpected-EVENT_NAME error
(FIX 5), and the workflow_run-summary_present total boundary
(FIX 7, both "" and "True" inputs).
- resolve-verify-matrix.test.ts: add unit coverage for the new
workflow_run summaryPresent boundary (empty + "True" + the
workflow_dispatch ignores-summaryPresent regression).
Red-green: 6 tests RED before code changes (FIX 3 warning, FIX 5
unknown EVENT_NAME, FIX 7 unit + CLI ×2 for "" and "True"); 79
tests GREEN after.
Validation: 4 vitest files / 79 tests passing; 87/87 ruby specs
passing; actionlint findings unchanged vs integration baseline
(8 → 8, identical diff); yaml.safe_load OK on both workflows.
A 7-agent review of the verify-matrix resolver and its surrounding workflow plumbing found three
boundary surfaces that could silently produce a GREEN deploy on a broken release, plus an
untested CLI contract that CI compares against the literal strings 'true' / 'false'.
FIX 1 — Validate the SSOT shape in loadSsotServices(). The prior `JSON.parse(...) as
{services: SsotService[]}` was an unchecked cast: a truncated/drifted SSOT (emitter crashed
mid-write, or schema renamed) parses fine but silently shrinks/empties the probe-eligible set
→ some redeployed services go unverified, or verify is skipped on a real redeploy. Extract a
pure exported parseSsotServices(raw, path) that requires the shape we depend on (non-empty
services array; each entry has a non-empty string name, an optional string|null dispatchName,
and a probe object with a boolean staging). Throw `::error::SSOT <path> malformed: <detail>`
on any violation. Also re-check existsSync(SSOT_JSON) after the regenerate-if-missing
execFileSync — a regen that exits 0 without writing must not proceed to a useless JSON.parse
crash. Drop the defensive `probe?.staging` once shape is guaranteed.
FIX 2 — Validate summary.json shape in the redeploy-gate bash. The bullseye false-green
surface: if redeploy-env.ts's schema ever drifts (e.g. `status` → `state`, `ok` → `success`),
every `jq select(.status==...)` yields empty → redeploy_red=false AND ok_services="" →
resolver skips verify → GREEN CI on a real unverified redeploy. Add a TOTAL vs WITH_STATUS
shape guard right after loading the summary: if TOTAL > 0 && WITH_STATUS == 0, emit
::error::summary.json has $TOTAL entries but none with status ok|error (schema drift?) and
exit 1. The legitimate empty-array path (TOTAL=0) is preserved.
FIX 3 — Fail loud on unknown eventName in resolveVerifyMatrix. The prior code fell through to
the workflow_run intersection branch for ANY unrecognized eventName (typo, unexpected
trigger), silently emitting has_services=false → indistinguishable from a legit "summary
absent" skip. Add an explicit guard so only workflow_run / workflow_dispatch are accepted;
anything else throws ::error::resolve-verify-matrix: unexpected eventName '<value>'. Tighten
the eventName parameter type to the literal union.
FIX 4 — Trim ok tokens + warn on dropped tokens in okCsvToCanonicalNames. Split, then
.map(t => t.trim()).filter(Boolean) so "a, b" (spaces) matches. Collect tokens that match NO
SSOT service (by name or dispatchName) and have the CLI wrapper emit ::warning::ok_services
tokens dropped (no SSOT match): <list> on stderr when non-empty — surfaces SSOT/build drift.
The pure function stays IO-free; logging lives in the wrapper.
FIX 5 — CLI wrapper integration test. New resolve-verify-matrix.cli.test.ts spawns
`npx tsx showcase/scripts/resolve-verify-matrix.ts` with a temp $GITHUB_OUTPUT file across
four scenarios and asserts the temp file contents EXACTLY (the workflow YAML compares
has_services against the literal strings 'true'/'false', so the byte-for-byte format is part
of the contract). Uses the real railway-envs.generated.json so the loader exercise is real.
FIX 6 — Cleanup. Remove the dead `env: DISPATCH_SERVICE: ...` block on the redeploy-gate
step (the next step redeclares it — leftover from the extraction). Soften the §3
decision-table all-errors bullet to match resolve-verify-matrix.ts's careful wording, and
append that when the success-set is empty (or the intersection collapses to empty), verify
is skipped and the gate reds independently. Append to showcase_build.yml's "Upload redeploy
summary" path-(A) comment that `if-no-files-found: error` still reds path (A) even if a
future change adds `if: always()`.
Tests: red→green for FIX 1/3/4/5 verified locally. Resolve-verify-matrix vitest count:
12 → 28. Full requested suite (resolve-verify-matrix + cli + aggregate-build-results +
lint-rule-no-public-env): 72 passed. showcase/bin ruby specs: 87 runs / 0 failures / 0
errors / 0 skips. actionlint baseline preserved (8 findings, identical to integration tip).
Extract the inline bash+jq decision logic from showcase_deploy.yml's
resolve-matrix job into showcase/scripts/resolve-verify-matrix.ts, a
pure function with a vitest suite. The bash had produced two confirmed
bugs across prior CR rounds, so making it testable is the lasting fix.
Issue A (the bug this PR fixes): when summary_present=true but
ok_services is empty (every service errored on redeploy), the old bash
skipped the intersection and fell through to the full probe-eligible
fleet, gratuitously probing every service against stale :latest. The
resolver now returns has_services=false in that case — enforce-redeploy
-gate independently reds the workflow on redeploy_red=true, so this
case is already loud; there is nothing left to verify.
Parity preserved for unchanged cases:
- workflow_dispatch + 'all'/empty → full probe-eligible set
- workflow_dispatch + specific svc → that one (unknown → error exit)
- workflow_run + summary_present=false → has_services=false
- workflow_run + present + ok non-empty → intersection with probe-
eligible (SSOT key OR dispatchName aliases both resolve)
Also clarified the Upload-redeploy-summary comment in showcase_build.yml
to document both red paths (hard crash → redeploy step exits non-zero;
exit-0-but-no-file → if-no-files-found:error reds the step) so no
false-green path is possible.
Tests: 12-case vitest suite covers each decision-table row plus the
Issue A fix (written red-first; failed against a naive full-fleet
fallback, passed once the early return was added). CLI parity verified
against the real generated SSOT for the three representative env-var
combinations (workflow_run + present + ok=[a,c]; workflow_run + present
+ ok empty; workflow_dispatch + 'all').
D1 — showcase_deploy.yml false-red fix
======================================
The build workflow legitimately uploads no `redeploy-summary` artifact when it ran
(push touched `showcase/**` so `paths:` matched) but `detect-changes` found no
buildable service, so `redeploy-staging` was skipped. The build still concludes
`success`, so `showcase_deploy.yml` fires on `workflow_run` and `resolve-matrix`
runs. `actions/download-artifact@v4` with `name:` HARD-FAILS on a missing
artifact, so the unguarded download was failing the job, and a downstream guard
that trips `enforce-redeploy-gate` on `resolve-matrix.result == 'failure'` was
flipping the workflow RED — a false-red on a routine showcase-docs/script change.
Add an artifact-existence pre-check using `actions/github-script` (pinned by SHA,
matching the existing repo convention) that lists the artifacts for
`workflow_run.id` via `actions: read` (already granted to `resolve-matrix`) and
sets `summary_present=true|false`. Gate the existing download step on
`summary_present == 'true'`. Keep NO `continue-on-error`, so the C1 property
holds: when the artifact exists but the download genuinely fails, the job still
fails loud and `enforce-redeploy-gate` correctly reds the workflow. When the
artifact is legitimately absent, the bash gate's existing `[ ! -f "$SUMMARY" ]`
branch no-ops (`redeploy_red=false`, `ok_services=""`) — nothing was
redeployed, so there is nothing to gate.
Updated the step comment block to enumerate the three distinct cases now
handled: workflow_dispatch (no download); workflow_run + artifact absent
(graceful skip); workflow_run + artifact present (download with fail-loud).
L1-L5 — env lint rule hardening
===============================
- L1: route the destructuring (VariableDeclarator/ObjectPattern) branch through
the shared `staticKeyName()` helper so the computed-string-key form
`const { ["NEXT_PUBLIC_X"]: y } = process.env` and the no-expression
template-literal form `const { [\`NEXT_PUBLIC_X\`]: y } = process.env` are
caught with the same parity as the bracket-member read.
- L2: unwrap a wrapping `ChainExpression` at the top of `isProcessEnv()` so
`process.env?.X` is matched robustly across parser flavors; corrected the
helper's doc comment to describe the actual semantics.
- L3: export `BANNED_KEYS` from the rule module and have the table-driven test
dynamically import the rule's own Set instead of hand-mirroring it — the
test set now cannot drift from the rule.
- L4: added override-scoping fixtures for `showcase/shell/src/**` and
`showcase/shell-dojo/src/**`; the `.oxlintrc.json` override list already
includes these, but the test now exercises them so an accidental drop is
caught.
- L5: expanded the file-header "Out of scope" doc list to include bulk-iteration
reads (`Object.keys/values/entries(process.env)`, for-in, spread
`{...process.env}`), rest-pattern destructuring, compound-assignment LHS, and
update operators. Documentation-only — the deliberate non-coverage is now
auditable.
Validation
==========
- RED→GREEN confirmed for L1 (two new destructuring computed-key tests) and L3
(dynamic `await import(...)` of BANNED_KEYS failed pre-fix with
"Rule module did not export a non-empty BANNED_KEYS Set", green after export).
- vitest: 38 passed (was 34 baseline + 4 new); aggregate-build-results 6 passed.
- Ruby promote suite: 87 runs, 251 assertions, 0 failures (unchanged).
- python3 yaml.safe_load: showcase_deploy.yml + showcase_build.yml +
showcase_promote.yml all parse OK.
- actionlint: zero NEW findings on the changed file. The pre-existing
showcase_build.yml SC2086/SC2129/runner-label findings are identical on the
integration baseline (unchanged by this commit).
Seven-agent CR surfaced correctness defects in the build/deploy/promote
pipeline and in the no-public-env-shell-read oxlint rule. This commit
closes the false-green paths and broadens lint coverage.
Workflow fixes:
- showcase_deploy.yml: drop `continue-on-error: true` on the redeploy-summary
artifact download. The dispatch path is already guarded by the `if:
workflow_run` clause, so the bash "no summary" branch handles legitimate
manual dispatches. A genuine workflow_run download failure must now fail
loud instead of silently widening verify to the full service set against
stale `:latest`.
- showcase_build.yml: redeploy-staging now intersects the build matrix with
the aggregator success set (`needs.aggregate-build-results.outputs.results`,
status == "success") before producing the redeploy CSV. Failed/skipped
slots no longer get redeployed (which would just re-pull stale `:latest`
and look healthy).
- showcase_build.yml: `notify-all-builds-failed` now additionally requires
`needs.build.result == 'failure'` so it doesn't Slack-spam when the build
job was SKIPPED (verify-image-refs upstream failure).
- showcase_build.yml: `notify` now lists [build, aggregate-build-results,
redeploy-staging] in `needs:` so aggregator/redeploy failures still emit
a Slack signal. `if: failure()` still skips when none of the needs failed.
- showcase_build.yml: `set -euo pipefail` on the Prepare build args step
so a transient $GITHUB_OUTPUT write failure can't ship images without
COMMIT_SHA/BRANCH baked in.
- showcase_deploy.yml: `enforce-redeploy-gate` now also trips on a
resolve-matrix failure (`needs.resolve-matrix.result == 'failure'`) so
an upstream crash that leaves `redeploy_red` empty can't bypass the gate.
- Doc-comment accuracy: drop stale `(PR #5093)` reference; correct the
env-IDs source-of-truth comment; document the optional `skip_build` field
in ALL_SERVICES; clarify that health_path is informational and verify
uses per-service drivers; add the missing `resolve-targets` step 0 to the
promote workflow's "Order:" header.
Aggregator fix (RED-GREEN):
- aggregate-build-results.ts: throw on zero slot dirs. The job is gated
upstream on has_changes == 'true', so zero slot dirs is a broken artifact
download, not a legitimate empty build set. Silently emitting
any_success=false + results=[] is indistinguishable from "all builds
failed" and lets the deploy workflow fall back to probing the full
service set against stale `:latest`. Refuse the ambiguity.
- aggregate-build-results.test.ts: existing empty-INPUT_DIR test was
updated to assert the throw (was: return []).
Oxlint rule (RED-GREEN):
- no-public-env-shell-read.mjs: handle destructuring reads
(const { NEXT_PUBLIC_X } = process.env and aliased form), template-literal
computed keys (process.env[\`NEXT_PUBLIC_X\`]), and explicitly skip
assignment-LHS / `delete` targets (writes are not reads). Optional
chaining already worked through the existing MemberExpression path.
Aliasing (`const e = process.env; e.X`) is intentionally documented as
out of scope (needs scope tracking). Description sharpened to say the
rule guards a specific banned-key set, not all NEXT_PUBLIC_* reads.
- .oxlintrc.json: tighten the off-override glob from
`showcase/**/*runtime-config*` to
`showcase/**/lib/runtime-config*.{ts,tsx}` so it only silences the
intended implementation files, not arbitrary paths containing that
substring.
- lint-rule-no-public-env.test.ts: rewritten as table-driven coverage of
every BANNED_KEYS entry (dotted + bracket-string forms), every ALLOWED
key (asserting non-firing), all new variants from the rule expansion,
the assignment/delete non-fire cases, and override scoping
(runtime-config exempt; packages exempt; shell-tree non-runtime-config
flagged).
Validation:
- actionlint on all three workflows: 8 pre-existing findings (depot label,
pre-existing SC2086 infos in untouched steps); my edits add zero.
- python3 yaml.safe_load: all three workflows OK.
- vitest aggregate-build-results.test.ts: 6/6 pass (incl. new throw test).
- vitest lint-rule-no-public-env.test.ts: 34/34 pass.
- vitest full showcase/scripts suite: 1654/1654 pass across 46 files.
- ruby showcase/bin/spec/all_tests.rb: 87 runs, 0 failures.
- Intersection jq proof (matrix a,b,c × success a,c) → "a,c"; all-failed
→ ""; skipped status excluded.
Distinguish ENOENT (treat as drift) from other read errors in
emit-railway-envs-json.ts --check; non-ENOENT errors now exit 2 with the
real error on stderr instead of being silently coerced into a misleading
'stale' message or an overwrite on a false drift signal.
Add a --out=<path> override so tests can write to a temp directory and
never mutate the tracked railway-envs.generated.json artifact. Rewrite
the emit-railway-envs-json test to use mkdtempSync + --out, switch the
staleness assertion to spawnSync so it asserts on exit code 1 plus the
stale-diagnostic substring, and add coverage for the new EISDIR
fail-loud path. After this change git status is clean post-test.
New TS probe driven off railway-envs SSOT. Accepts --env staging|prod and
optional --services CSV; iterates SERVICES where probe[env]===true and
dispatches to per-driver feature-level verifiers. Refuses to start when a
probe-required service is missing a domain for the requested env (no
silent skip). HTTP 200 is necessary but not sufficient.
Adds verify-deploy.drivers.ts dispatch with exhaustive never check on
ProbeDriver, plus one stub module per ProbeDriver literal (shell, docs,
dashboard, dojo, harness, eval, aimock, pocketbase, webhooks, agent).
Stubs fail loud with an explicit "not yet implemented" error so any
accidental real-network invocation surfaces; per-driver feature-level
impls land as subsequent micro-tasks. Refs spec section 3 / section 3.5.
[BLITZ:A5]
Adds optional REDEPLOY_SUMMARY_JSON path; when set, redeploy-env writes
a structured per-service record array {service,status,error?}. PR #5093's
exit-code contract is preserved (staging=0, prod=1-on-failure).
Consumed by showcase_deploy.yml to fail the workflow on staging per-service
errors without changing the script's exit semantics. Refs spec §3.
Plan-B / Option-B migration moved every shell URL/analytics key off the
build-time NEXT_PUBLIC_* env channel and onto runtime config served via
__SHOWCASE_CONFIG__ + getRuntimeConfig(). To prevent a silent regression
where a future change reintroduces a direct process.env.NEXT_PUBLIC_*
read in shell code (which would re-freeze the value at build time and
break no-rebuild env switching), add a focused lint rule.
The rule (copilotkit/no-public-env-shell-read) is implemented as a
custom oxlint JS plugin rule in the existing copilotkit plugin and
enabled under shell-scoped overrides in .oxlintrc.json:
- Errors on process.env.NEXT_PUBLIC_<URL/ANALYTICS> reads in:
showcase/shell-dashboard/src/**, showcase/shell-docs/src/**,
showcase/shell/src/**, showcase/shell-dojo/src/**
- Banned keys: POCKETBASE_URL, SHELL_URL, BASE_URL, OPS_BASE_URL,
INTELLIGENCE_SIGNUP_URL, POSTHOG_KEY, POSTHOG_HOST, SCARF_PIXEL_ID,
GOOGLE_ANALYTICS_TRACKING_ID, REB2B_KEY, REO_KEY
- Intentionally allowed (NOT banned): NEXT_PUBLIC_COMMIT_SHA and
NEXT_PUBLIC_BRANCH (build-stamped artifact identifiers per B10/B11)
and NEXT_PUBLIC_LOCAL_BACKENDS (computed from shared/local-ports.json
at build, local-dev only).
- Excluded files (rule disabled via a follow-up override): MDX content
under shell-docs/src/content/**, runtime-config implementation files,
and *.test.{ts,tsx} / *.spec.{ts,tsx}. oxlint does not support
excludedFiles inside an override block, so the exclusion is expressed
as a later override that sets the rule to off.
Plan-B originally targeted oxlint's eslint/no-restricted-syntax with an
AST-selector regex. oxlint 1.x does not implement that rule (only
no-restricted-globals / no-restricted-imports), so the equivalent guard
is realized as a small custom rule in the existing copilotkit JS plugin
(meta.name=copilotkit), reusing the same plugin loader the repo already
has for require-cpk-prefix and no-single-arg-zod-record.
Verification (red-green): the rule fires on a fixture containing
process.env.NEXT_PUBLIC_POCKETBASE_URL and does NOT fire on a fixture
containing process.env.NEXT_PUBLIC_COMMIT_SHA. Test pins the config via
-c so it works inside git worktrees nested under .claude/worktrees/
where oxlint's automatic upward config search can miss the worktree's
own .oxlintrc.json.
All four shells lint clean: 0 errors of the new rule across
shell-dashboard (114 files), shell-docs (137), shell (29), shell-dojo (6).
Cover four malformed-ref shapes the gate must reject:
* `:sha256-<hex>` (missing the @ separator — looks like a digest
pin but is actually a tag)
* `@sha256:<too-short-hex>` (truncated digest hex)
* The 2026-04-21 `...atest` corruption shape from the script
docstring (the original reason this gate exists)
* Non-ghcr.io registries on both envs
These match the canonical PROD_SHAPE / STAGING_SHAPE regexes in
verify-railway-image-refs.ts; the tests are the regression guard
that ensures a future "relax the regex" change cannot ship without
explicitly turning these red first.
Lock in shape behaviour for dashboard, docs, dojo, shell, and
harness with explicit per-service red-green cases:
* prod with :latest -> fail (must be @sha256)
* prod with @sha256 on the correct repo -> pass
* staging with :latest on the correct repo -> pass
* staging with @sha256 -> fail (must float on :latest)
* wrong GHCR repo name on either env -> fail
The validateImage body is unchanged — it was always shape-pure and
shape-correct. Before WS-C these five services were never exercised
through the gate at all (gateValidated:false). These tests are the
regression guard that ensures a future edit doesn't accidentally
re-introduce the Phase-2 carve-out without anyone noticing.
Flip dashboard, docs, dojo, shell, and harness from
gateValidated: false to gateValidated: true and simultaneously add
the corresponding repoNameOverride for both envs:
dashboard -> showcase-shell-dashboard
docs -> showcase-shell-docs
dojo -> showcase-shell-dojo
shell -> showcase-shell
harness -> showcase-harness
These two halves MUST land in the same commit. Flipping
gateValidated without the override would make the gate look up
ghcr.io/copilotkit/<railway-name> (e.g.
ghcr.io/copilotkit/dashboard:latest) which does not exist — the
gate would fail on first run. Adding the override without flipping
gateValidated is dead code: main() short-circuits unvalidated
services before consulting the override. Only the union is correct.
Also remove the Phase-2 deferral comments and refresh the
ServiceEntry.gateValidated JSDoc — there are no Phase-2 holdouts
left. All 27 services are now gate-validated; gateIgnore remains
the sole escape hatch and is unused by every current entry.
The Railway -> SSOT direction at verify-railway-image-refs.ts was
warn-and-continue: an out-of-band Railway service with a malformed
ref would not turn CI red as long as nobody read the warning line.
Replace that branch with a hard failure path that lists the
untracked service under its own failure class, with a clear remedy
in the error message (add to SSOT, or set gateIgnore on an existing
entry).
Refactor main() to call two pure helpers (findUntrackedServices,
summarizeFailures) so the policy is unit-testable without going
through Railway GraphQL. The SSOT -> Railway direction
(findMissingServices) is unchanged: it is already correct and is a
separate coverage class.
Adds red-green tests for the summarizeFailures shape, covering the
three failure classes (shape violations, SSOT->Railway drift,
Railway->SSOT drift) and the success path.
Widen `ServiceEntry` with an optional `gateIgnore: boolean` field
(default false / unset) so the image-ref gate can deliberately exclude
a Railway service from BOTH coverage-direction checks. No behavioural
change in this commit; the field is consumed by the next commit which
flips the Railway->SSOT direction from warn-only to hard-fail.
Stub-export `findUntrackedServices` from verify-railway-image-refs so
the unit test for the new direction can compile. Behavioural wiring
into main() lands in the next commit.
Adds verify-railway-image-refs.test.ts with the first unit tests
covering the new field surface and the helper contract.
Adds Domains, ProbeDriver, ProbeConfig types + domainFor() helper that
throws on unknown service/env. Populates domains.{staging,prod} and
probe.{staging,prod,driver} on every SERVICES entry. Adds
emit-railway-envs-json.ts to serialize the SSOT for the Ruby side and
the workflow consumers.
Refs spec §3 / §3a.
## Summary
Adds per-integration D6 aimock fixtures for 18 showcase integrations,
completing the D6 probe coverage that #5022 (Slice 3) scaffolded. Each
integration now has the standard set of D5 feature-type fixtures
(agent-config, auth, byoc, gen-ui-*, interrupt-headless, multimodal,
prebuilt-*, tool-rendering-*) under `showcase/aimock/d6/<integration>/`.
## Why
Slice 3 (#5022) shipped the per-integration directory structure + D6
probe driver + harness scoping. The first D6 probe run reported most
integrations RED because most integrations' D6 directories were missing
per-feature fixtures (only langgraph-python had any). This PR fills in
those gaps.
## Authoring rules applied
- Every fixture has `match.context = "<integration>"` for
cross-integration isolation
- toolCall responses use `hasToolResult: false` (or
`toolName`/`toolCallId` gates) to prevent re-match loops
- Conversation turns match the corresponding D5 probe scripts at
`showcase/harness/src/probes/scripts/d5-<feature>.ts`
- Reference shape: langgraph-python's fixtures + each integration's
existing D5/D6 fixture conventions
- Skipped features: each integration's `not_supported_features` list
(e.g., google-adk skips gen-ui-interrupt + interrupt-headless)
## CR
Two rounds of 7-agent unbiased CR converged. R1 surfaced 5 bucket-(a)
findings, all addressed in R2 fix commit. R2 confirmation converged with
no in-scope regressions.
## Test plan
- [ ] Showcase auto-redeploys on merge (path filter matches
`showcase/**`)
- [ ] Next D6 probe cycle on production harness writes
`d6:<slug>/<featureId>` rows for all 18 integrations
- [ ] Dashboard D6 chips reflect real per-integration state instead of
all-gray
Bumps exact-duplicate ceiling 11 → 230 and substring-shadow ceiling
126 → 151. The D6 per-integration fixtures naturally share match keys
with pre-existing demo fixtures in the same context scope, disambiguated
at runtime by the active demo/probe path.
The four fixture routing regression tests still referenced the old
monolithic d5-all.json / smoke.json / feature-parity.json files that
were reorganized into per-integration d4/ d6/ shared/ directories.
Changes:
- Load fixtures via glob from d6/langgraph-python/ (reference
integration) instead of deleted monolithic files
- Add _context to test requests (aimock 1.26.1 checks req._context
against match.context for per-integration scoping)
- Adapt subagents test from toolCallId-based chaining to
turnIndex-based chaining (matches new D6 fixture structure)
- Adapt state-context test to D6 context-scoping model (no
systemMessage matching; routing happens via X-AIMock-Context)
- Remove _migrated-from-*.json shared files (all fixtures already
exist in per-integration D6 dirs; the migration files caused
620 shared-vs-scoped collisions)
- Add KNOWN_DUPLICATE_CEILING=11 ratchet for pre-existing D6
intra-feature duplicate match keys
Add fixture analysis/split/merge scripts for the D6 restructure,
update aimock fixture collision detection tests for the new directory
layout, update GOTCHAS.md and QA-COVERAGE.md with D6 notes, and
add --d6 flag support to the showcase test CLI.
The previous fixture regression (HTML+CSS only, no jsFunctions) slipped
past CI because the e2e suite only asserted "iframe mounts with non-empty
srcdoc" — which passes whether or not the iframe is interactive. Adds
two layers of guard so the same regression cannot land silently:
1. showcase/scripts/__tests__/open-gen-ui-advanced-fixtures.test.ts
(vitest, runs in showcase_validate on every PR): asserts each of the
three interactive fixture entries in d5-all.json ships jsFunctions
referencing the matching host bridge (evaluateExpression / notifyHost).
Catches "someone removed jsFunctions" at PR-time with no
infrastructure dependencies.
2. showcase/integrations/langgraph-python/tests/e2e/open-gen-ui-advanced.spec.ts
(playwright, runs in test_e2e-showcase-on-demand): adds three
round-trip tests that drive the in-iframe controls and assert the
host-side handler ran by capturing its console.log + verifying the
iframe output element reflects the host response. Catches "the
renderer fails to inject jsFunctions into the sandbox" too.
The e2e tests also switch the existing smoke tests off pill-click and
onto a textarea-driven fill+Enter path, following the same precedent as
commit 15db0bbf3 (gen-ui-headless-complete) — chip mounts diverge
between EmptyState and SuggestionBar surfaces, and Playwright's pill
click races React hydration. Using [data-testid="copilot-chat-textarea"]
with an explicit click + waitForLoadState("networkidle") makes the
suite reliable end-to-end (7/7 passing locally against the aimock-driven
stack).
The "Draw a flowchart" suggestion pill in the mcp-apps demo sent
"Use Excalidraw to draw a simple flowchart with three steps." which
had no matching create_view fixture in d5-all.json. aimock walked
through to feature-parity.json's `{userMessage: "steps"}` substring
fixture and returned a generic "Here is my plan..." content blurb
with no MCP tool call, so the runtime never invoked create_view, the
MCP middleware never fetched the UI resource, and the sandboxed
iframe never mounted.
Add a fixture pair in d5-all.json (and its harness mirror) keyed on
"draw a simple flowchart": turn 1 emits create_view with a three-
step Start -> Process -> End flowchart, turn 2 emits the narration
after the tool result. The distinctive substring beats the generic
feature-parity catch-alls under first-match-wins.
Adds a regression test that loads the same fixture files in the same
order as docker-compose.local.yml and asserts via aimock's matchFixture
that each mcp-apps pill routes to its create_view fixture on turn 1 and
its narration fixture on turn 2.
User-facing renames so the showcase reads the way a cold visitor would
expect:
- `byoc-hashbrown` → `declarative-hashbrown` (and `byoc-json-render` →
`declarative-json-render`). The display titles already said
"Declarative UI: …"; only the URL slugs and folder paths still
leaked the internal BYOC ("Bring Your Own Components") jargon.
Renamed:
/demos/byoc-hashbrown → /demos/declarative-hashbrown
/demos/byoc-json-render → /demos/declarative-json-render
/api/copilotkit-byoc-* → /api/copilotkit-declarative-*
src/app/demos/byoc-* → src/app/demos/declarative-*
qa/byoc-*.md → qa/declarative-*.md
tests/e2e/byoc-*.spec.ts → tests/e2e/declarative-*.spec.ts
Internal Python module names + langgraph graph IDs stay legacy
(`byoc_hashbrown_agent.py`, `byoc_hashbrown`) — those are not
user-facing and renaming them is a separate cross-codebase pass.
- `a2ui-fixed-schema` slug intentionally unchanged.
- Tool Rendering trio parenthetical rename (Default → Catch-all →
Custom progression reads clearly as "how much do I customize?"):
Tool Rendering (Default) — unchanged
Tool Rendering (Custom default) → Tool Rendering (Catch-all)
Tool Rendering (Specific) → Tool Rendering (Custom)
- `tool-rendering-reasoning-chain` cell renamed from
"Generative UI: Rendering multiple tools" to
"Generative UI: Tool calls + reasoning" (the demo is about combining
reasoning + tool rendering, not about quantity of tools).
- `Open Generative UI: Default` / `Open Generative UI: Custom`
descriptions expanded so a visitor understands how Open Generative UI
differs from Tool Rendering (agent composes UI from a registered
library vs. attaching a renderer to a *named* backend tool).
- Showcase index now sorts demos within each tag by `manifest.features`
order. Previously demos appeared in manifest declaration order, which
ignored the team's curated "polished flagship → simplest start →
variants" arc.
Cross-cutting registry / harness / dashboard updates that fall out of
the rename:
- `shared/feature-registry.json` adds the two new IDs alongside the
legacy `byoc-*` (so the catalog stays valid; the other 17
integrations still declare `byoc-*` in their manifests).
- `shared/constraints.yaml` adds the new IDs to the
generative-ui-approach allow-list.
- `scripts/__tests__/generate-catalog.test.ts` updates the cell-count
expectations (45 features × 18 integrations = 810; 792 after docs-
only exclusion; 45 LGP cells = 38 wired + 1 stub + 6 unshipped).
- Harness probe `d5-byoc.ts` + `d5-byoc.test.ts` now route both slug
families through `preNavigateRoute` and exercise the new branches.
- `d5-feature-mapping.ts` and `shell-dashboard/live-status.ts` mirror
the dual-ID mapping so both legacy and renamed slugs roll up under
the same `byoc` D5 featureType.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related changes that bring the dashboard's gold-standard view in
line with the desired naming convention and surface deprecated rows
behind a toggle (instead of hiding them at catalog generation).
## Naming alignment
Applied 28 renames in feature-registry.json + 20 in LGP manifest per
the user-provided mapping. Highlights:
- "Pre-Built CopilotChat" -> "Pre-Built: CopilotChat"
- "Headless Chat (Simple/Complete)" -> "Headless UI: Simple/Complete"
- "Multi-modal / File Uploads" -> "Attachements" (intentional spelling)
- "Controlled Gen-UI (Display)" -> "Generative UI: useComponent"
- "In-Chat HITL (use*)" -> "Human In/in the Loop: In-chat / Interrupts"
- "Headless Interrupt" -> "Human in the Loop: Headless Interrupts"
- "Declarative Generative UI (A2UI - *)" -> "Declarative UI: */* A2UI"
- "Fully Open-Ended Generative UI" -> "Open Generative UI: Default"
- "Tool Rendering ..." -> "Generative UI: Tool Rendering (...)"
- "Tool Rendering + Reasoning Chain" -> "Generative UI: Rendering multiple tools"
- "Agentic Generative UI ..." -> "Generative UI: Agent State"
- "Frontend Tools (...)" -> "Frontend Tools: ..."
- "Shared State (...)" -> "Shared State: ..."
- "State Streaming" -> "Shared State: Streaming"
- "Readonly State (Agent Context)" -> "Shared State: Frontend Context"
- "BYOC Hashbrown <-> json-render" -- labels intentionally swapped per
user instruction (demos were historically reversed; new labels
reflect what they actually do).
LGP manifest demos[].name updated to match feature-registry names so
the dojo and dashboard surface the same human-readable label.
## Show Deprecated toggle (feature-grid.tsx)
Added a checkbox in the matrix header -- default OFF -- that filters
feature rows where `feature.deprecated === true`. Toggle ON shows all
deprecated features across all integrations (audit trail); toggle OFF
hides those rows entirely so the gold-standard view stays clean.
Reverted the catalog-side filter from PR #4744 (which dropped LGP
cells for deprecated features at catalog-generation time). Now the
catalog emits cells uniformly for all (integration x feature) pairs,
and visibility is controlled at the dashboard layer. Toggling on
shows complete cross-integration data without missing-cell artifacts.
Affects 4 features marked deprecated:true in feature-registry.json:
agentic-chat-reasoning, hitl, hitl-in-chat-booking,
reasoning-default-render.
LGP cell count: back to 43 (38 wired + 1 stub + 4 unshipped). The 4
unshipped rows are hidden by default; toggle to surface them.
Tests: 18/18 catalog tests + 1588/1588 harness vitest passing.
validate-fixture-tool-surface clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three fixes that follow up on PR #4743 to bring LGP closer to fully-green
on the dashboard:
1. tool-rendering-reasoning-chain probe: was failing with
`expected [data-testid="reasoning-block"] to mount within 30000ms`.
Root cause: the demo's `<ReasoningBlock>` slot only mounts when a
reasoning-role message lands in the transcript, which requires
aimock to emit REASONING_MESSAGE_* events, which in turn requires
the fixture's first-leg response to carry a `reasoning` field. The
weather/Tokyo and SFO/JFK first-leg fixtures were missing it.
Mirrors the convention documented in reasoning-display.json:2.
Patched both source (harness/fixtures/d5/) and bundle (aimock/d5-all.json).
2. gen-ui-interrupt source fixture: the source fixture file was missing
the resume-leg toolCallId entries that already existed in the bundle.
Cosmetic mirror so re-bundling stays consistent. Same chip prompts +
same toolCallIds as interrupt-headless.json (both probes share the
same agent and aimock fixture set; the difference is the FRONTEND
rendering — useInterrupt inline vs useHeadlessInterrupt separate-pane).
3. Dashboard gold-standard filter: 4 deprecated/legacy features
(agentic-chat-reasoning, hitl, hitl-in-chat-booking,
reasoning-default-render) used to render as X-marked rows in the
LGP gold-standard dashboard view because LGP intentionally does
NOT implement them — they were consolidated into the modern shape
(reasoning-custom + reasoning-default; hitl-in-chat with
useHumanInTheLoop). Other 17 integrations still serve those legacy
demos, so we don't yank the features from feature-registry.json
entirely. Instead: marked them `deprecated: true` and updated
generate-registry.ts to skip emitting cells when a deprecated
feature is unshipped for an integration. LGP cells: 43 → 39 (the
4 deprecated rows disappear). Other integrations: unchanged
(audit trail preserved). Catalog total: 774 → 770.
Tests:
- 1588/1588 harness vitest passing
- 19/19 generate-catalog + generate-registry tests passing
(counts updated for the 4 dropped LGP cells + new deprecated-
feature filter test)
- validate-fixture-tool-surface clean (282 fixtures × 627 demos)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three highlight paths in langgraph-python's manifest pointed at files
that don't exist after the PR #4694 reorganization:
- hitl-in-chat → src/agents/hitl_in_chat.py (actually hitl_in_chat_agent.py)
- chat-slots → custom-welcome-screen.tsx (file doesn't exist; use slot-wrappers.tsx)
- mcp-apps → copilotkit-mcp-apps/route.ts (actually .../[[...slug]]/route.ts)
The bundler walks every highlight at build time; one missing path aborts
the whole CI step. Fix all three.
Test snapshot counts in generate-catalog and generate-registry hardcoded
40 features / 720 cells / 702 total. With the two new feature IDs added
to the registry (reasoning-default + reasoning-custom), counts shift to
42 / 756 / 738; the LGP-specific cell distribution moved from
39 wired + 1 stub + 0 unshipped to 35 wired + 1 stub + 6 unshipped, and
the registry-side LGP feature/demo count drops to 36 (PR #4694 trimmed
4 items from the manifest's features list).
docs-only features (e.g. cli-start) exist purely for documentation
coverage tracking -- they have no route, no depth probes, and no
health signals. The catalog metadata was counting their 18 stub cells
in the headline wired/stub/unshipped/unsupported breakdown, inflating
the total and making the stats bar misleading.
Exclude docs-only cells from the headline counts. A new docs_only
field tracks the excluded count separately so the invariant
(wired + stub + unshipped + unsupported + docs_only == cells.length)
holds.
Before: total_cells=720 wired=673 stub=18 unsupported=29
After: total_cells=702 wired=673 stub=0 unsupported=29 docs_only=18
- claude-sdk-typescript manifest: G3's headless-complete and a2ui-fixed-schema entries pointed at -agent.ts paths but the actual files are -prompt.ts. Bundle-demo-content blew up on the missing files.
- generate-catalog test: update reference and crewai-crews wired-count assertions for the post-blitz state. Multiple integrations now tie at 39 wired; auto-detection picks the alphabetical winner (langgraph-fastapi) over langgraph-python. crewai wired count moved to 37.
Adds a fourth cell status, "unsupported", to the showcase dashboard for
features whose framework cannot architecturally support them (e.g.,
no graph-interrupt API, no MCP tool runtime). This is distinct from
"unshipped" (just unbuilt).
- Manifest schema gains optional `not_supported_features: string[]`.
- generate-registry.ts: determineCellStatus checks `not_supported_features`
first; CatalogCell/CatalogMetadata gain `unsupported`; max_depth=0 for
unsupported cells; parity tier is computed against the supportable subset
of the reference (unsupported features no longer drag a framework's tier
down for gaps it legitimately cannot fill).
- depth-utils.ts: deriveDepth returns D0 with no regression for
unsupported cells.
- DepthChip: renders a dashed gray border + 🚫 glyph with
"Not supported by this framework" tooltip; data-status attribute
distinguishes unshipped vs unsupported.
- CoverageBar gains diagonal-stripe gray segment for unsupported.
- StatsBar / AdaptiveStatsBar / CellsView: separate "Unsupported" chip.
- CellMatrix gaps filter excludes unsupported cells (not work to do).
- Tests: cover unsupported in DepthChip, depth-utils, cell-matrix, and
catalog generator metadata/max_depth assertions.
remove dead starter scripts
Update probe configs, fixtures, and test drivers to reference
showcase/integrations/ instead of showcase/packages/.
Remove obsolete generate-starters test, starter-consistency
test, validate-workflow-starters, and rebuild-coord.sh.
Delete the pre-generated starters/ directory tree (previously
synced from packages/ by generate-starters.ts). Add
extract-starter.ts which produces a starter tarball on demand
from any integration. Move shared starter template files to
showcase/shared/starter-template/.
Jordan Ritter