Seven-agent CR surfaced correctness defects in the build/deploy/promote
pipeline and in the no-public-env-shell-read oxlint rule. This commit
closes the false-green paths and broadens lint coverage.
Workflow fixes:
- showcase_deploy.yml: drop `continue-on-error: true` on the redeploy-summary
artifact download. The dispatch path is already guarded by the `if:
workflow_run` clause, so the bash "no summary" branch handles legitimate
manual dispatches. A genuine workflow_run download failure must now fail
loud instead of silently widening verify to the full service set against
stale `:latest`.
- showcase_build.yml: redeploy-staging now intersects the build matrix with
the aggregator success set (`needs.aggregate-build-results.outputs.results`,
status == "success") before producing the redeploy CSV. Failed/skipped
slots no longer get redeployed (which would just re-pull stale `:latest`
and look healthy).
- showcase_build.yml: `notify-all-builds-failed` now additionally requires
`needs.build.result == 'failure'` so it doesn't Slack-spam when the build
job was SKIPPED (verify-image-refs upstream failure).
- showcase_build.yml: `notify` now lists [build, aggregate-build-results,
redeploy-staging] in `needs:` so aggregator/redeploy failures still emit
a Slack signal. `if: failure()` still skips when none of the needs failed.
- showcase_build.yml: `set -euo pipefail` on the Prepare build args step
so a transient $GITHUB_OUTPUT write failure can't ship images without
COMMIT_SHA/BRANCH baked in.
- showcase_deploy.yml: `enforce-redeploy-gate` now also trips on a
resolve-matrix failure (`needs.resolve-matrix.result == 'failure'`) so
an upstream crash that leaves `redeploy_red` empty can't bypass the gate.
- Doc-comment accuracy: drop stale `(PR #5093)` reference; correct the
env-IDs source-of-truth comment; document the optional `skip_build` field
in ALL_SERVICES; clarify that health_path is informational and verify
uses per-service drivers; add the missing `resolve-targets` step 0 to the
promote workflow's "Order:" header.
Aggregator fix (RED-GREEN):
- aggregate-build-results.ts: throw on zero slot dirs. The job is gated
upstream on has_changes == 'true', so zero slot dirs is a broken artifact
download, not a legitimate empty build set. Silently emitting
any_success=false + results=[] is indistinguishable from "all builds
failed" and lets the deploy workflow fall back to probing the full
service set against stale `:latest`. Refuse the ambiguity.
- aggregate-build-results.test.ts: existing empty-INPUT_DIR test was
updated to assert the throw (was: return []).
Oxlint rule (RED-GREEN):
- no-public-env-shell-read.mjs: handle destructuring reads
(const { NEXT_PUBLIC_X } = process.env and aliased form), template-literal
computed keys (process.env[\`NEXT_PUBLIC_X\`]), and explicitly skip
assignment-LHS / `delete` targets (writes are not reads). Optional
chaining already worked through the existing MemberExpression path.
Aliasing (`const e = process.env; e.X`) is intentionally documented as
out of scope (needs scope tracking). Description sharpened to say the
rule guards a specific banned-key set, not all NEXT_PUBLIC_* reads.
- .oxlintrc.json: tighten the off-override glob from
`showcase/**/*runtime-config*` to
`showcase/**/lib/runtime-config*.{ts,tsx}` so it only silences the
intended implementation files, not arbitrary paths containing that
substring.
- lint-rule-no-public-env.test.ts: rewritten as table-driven coverage of
every BANNED_KEYS entry (dotted + bracket-string forms), every ALLOWED
key (asserting non-firing), all new variants from the rule expansion,
the assignment/delete non-fire cases, and override scoping
(runtime-config exempt; packages exempt; shell-tree non-runtime-config
flagged).
Validation:
- actionlint on all three workflows: 8 pre-existing findings (depot label,
pre-existing SC2086 infos in untouched steps); my edits add zero.
- python3 yaml.safe_load: all three workflows OK.
- vitest aggregate-build-results.test.ts: 6/6 pass (incl. new throw test).
- vitest lint-rule-no-public-env.test.ts: 34/34 pass.
- vitest full showcase/scripts suite: 1654/1654 pass across 46 files.
- ruby showcase/bin/spec/all_tests.rb: 87 runs, 0 failures.
- Intersection jq proof (matrix a,b,c × success a,c) → "a,c"; all-failed
→ ""; skipped status excluded.
A RAILWAY_TOKEN secret with trailing whitespace/newline (common from op
read, heredoc, shell export) was returned verbatim and produced invalid
Authorization: Bearer headers and silent Railway 401s. Trim the env-var
lane and treat whitespace-only as UNSET so the config-file fallback runs.
Also adds a missing should-have-thrown guard in the NO_HOME test and
removes a stale gateIgnore clause from findUntrackedServices docstring.
Distinguish ENOENT (treat as drift) from other read errors in
emit-railway-envs-json.ts --check; non-ENOENT errors now exit 2 with the
real error on stderr instead of being silently coerced into a misleading
'stale' message or an overwrite on a false drift signal.
Add a --out=<path> override so tests can write to a temp directory and
never mutate the tracked railway-envs.generated.json artifact. Rewrite
the emit-railway-envs-json test to use mkdtempSync + --out, switch the
staleness assertion to spawnSync so it asserts on exit code 1 plus the
stale-diagnostic substring, and add coverage for the new EISDIR
fail-loud path. After this change git status is clean post-test.
A token read from ~/.railway/config.json with surrounding whitespace or a
trailing newline passed nonEmpty() but was returned verbatim, so an
'Authorization: Bearer <token>' header could carry CR/LF or stray spaces
(Node HTTP rejects invalid header chars; Railway 401s otherwise). Trim
each return path so the canonical token is always emitted; whitespace-only
values still fall through. Also clarify the JSDoc that this resolver does
not consult process.env.RAILWAY_TOKEN.
- Reject empty/whitespace-only service in parseBuildOutputs,
mergeBuildResultFiles, and buildResultArtifactName (was only
buildResultArtifactName, and only for length===0).
- mergeBuildResultFiles now throws on duplicate service names across
slots so an upstream dispatch-name collision surfaces instead of
letting a failure+success pair spuriously look like a success in
successSet (fail-loud discipline).
- Derive BuildOutcome union and VALID_STATUSES set from a single
BUILD_OUTCOMES as-const tuple plus a compile-time exhaustiveness
assignment so they cannot drift.
- Extract validateServiceBuildResult shared validator used by both
parseBuildOutputs (context: 'entry[i]') and mergeBuildResultFiles
(context: 'slot[i]') — one set of rules, one error format, both
now include the offending index.
- Trim shouldRedeployStaging JSDoc to what/why only (dropped the
external-caller enumeration) and note in the module header that
the single-result.json-per-slot invariant is enforced workflow-side,
not by the parser. shouldRedeployStaging([]) === false behavior
unchanged.
Tests: 16 -> 23 passing, all new red-then-green; tsc 0.
Treat whitespace-only token strings as empty so the resolver falls through
to the next candidate instead of returning a bearer that fails at Railway
with a confusing 401.
Guard against non-object JSON input (null/undefined/string/number/array)
defensively before property access — the config originates from
JSON.parse of ~/.railway/config.json (untrusted).
Replace non-null assertions on re-accessed expressions with
locally-captured values so the nonEmpty narrowing actually applies; no
behavior change for the happy path.
Comment cleanups: replace stale line-number reference with a symbolic
one, reword the Ruby-parity note to acknowledge the per-project token
fallback that is intentionally not honored, drop the rotting
"(43+ chars)" parenthetical, and soften the deprecation timeline to
"a future release."
A.7 rewrote showcase_deploy.yml to be SSOT-driven (no hardcoded options
list, no inline ALL_SERVICES). Webhooks remains canonically wired via
the SSOT (probe.staging:true, dispatchName:"webhooks"); test now pins
that contract instead of the now-obsolete literal regexes.
New TS probe driven off railway-envs SSOT. Accepts --env staging|prod and
optional --services CSV; iterates SERVICES where probe[env]===true and
dispatches to per-driver feature-level verifiers. Refuses to start when a
probe-required service is missing a domain for the requested env (no
silent skip). HTTP 200 is necessary but not sufficient.
Adds verify-deploy.drivers.ts dispatch with exhaustive never check on
ProbeDriver, plus one stub module per ProbeDriver literal (shell, docs,
dashboard, dojo, harness, eval, aimock, pocketbase, webhooks, agent).
Stubs fail loud with an explicit "not yet implemented" error so any
accidental real-network invocation surfaces; per-driver feature-level
impls land as subsequent micro-tasks. Refs spec section 3 / section 3.5.
[BLITZ:A5]
Adds optional REDEPLOY_SUMMARY_JSON path; when set, redeploy-env writes
a structured per-service record array {service,status,error?}. PR #5093's
exit-code contract is preserved (staging=0, prod=1-on-failure).
Consumed by showcase_deploy.yml to fail the workflow on staging per-service
errors without changing the script's exit semantics. Refs spec §3.
Plan-B / Option-B migration moved every shell URL/analytics key off the
build-time NEXT_PUBLIC_* env channel and onto runtime config served via
__SHOWCASE_CONFIG__ + getRuntimeConfig(). To prevent a silent regression
where a future change reintroduces a direct process.env.NEXT_PUBLIC_*
read in shell code (which would re-freeze the value at build time and
break no-rebuild env switching), add a focused lint rule.
The rule (copilotkit/no-public-env-shell-read) is implemented as a
custom oxlint JS plugin rule in the existing copilotkit plugin and
enabled under shell-scoped overrides in .oxlintrc.json:
- Errors on process.env.NEXT_PUBLIC_<URL/ANALYTICS> reads in:
showcase/shell-dashboard/src/**, showcase/shell-docs/src/**,
showcase/shell/src/**, showcase/shell-dojo/src/**
- Banned keys: POCKETBASE_URL, SHELL_URL, BASE_URL, OPS_BASE_URL,
INTELLIGENCE_SIGNUP_URL, POSTHOG_KEY, POSTHOG_HOST, SCARF_PIXEL_ID,
GOOGLE_ANALYTICS_TRACKING_ID, REB2B_KEY, REO_KEY
- Intentionally allowed (NOT banned): NEXT_PUBLIC_COMMIT_SHA and
NEXT_PUBLIC_BRANCH (build-stamped artifact identifiers per B10/B11)
and NEXT_PUBLIC_LOCAL_BACKENDS (computed from shared/local-ports.json
at build, local-dev only).
- Excluded files (rule disabled via a follow-up override): MDX content
under shell-docs/src/content/**, runtime-config implementation files,
and *.test.{ts,tsx} / *.spec.{ts,tsx}. oxlint does not support
excludedFiles inside an override block, so the exclusion is expressed
as a later override that sets the rule to off.
Plan-B originally targeted oxlint's eslint/no-restricted-syntax with an
AST-selector regex. oxlint 1.x does not implement that rule (only
no-restricted-globals / no-restricted-imports), so the equivalent guard
is realized as a small custom rule in the existing copilotkit JS plugin
(meta.name=copilotkit), reusing the same plugin loader the repo already
has for require-cpk-prefix and no-single-arg-zod-record.
Verification (red-green): the rule fires on a fixture containing
process.env.NEXT_PUBLIC_POCKETBASE_URL and does NOT fire on a fixture
containing process.env.NEXT_PUBLIC_COMMIT_SHA. Test pins the config via
-c so it works inside git worktrees nested under .claude/worktrees/
where oxlint's automatic upward config search can miss the worktree's
own .oxlintrc.json.
All four shells lint clean: 0 errors of the new rule across
shell-dashboard (114 files), shell-docs (137), shell (29), shell-dojo (6).
Cover four malformed-ref shapes the gate must reject:
* `:sha256-<hex>` (missing the @ separator — looks like a digest
pin but is actually a tag)
* `@sha256:<too-short-hex>` (truncated digest hex)
* The 2026-04-21 `...atest` corruption shape from the script
docstring (the original reason this gate exists)
* Non-ghcr.io registries on both envs
These match the canonical PROD_SHAPE / STAGING_SHAPE regexes in
verify-railway-image-refs.ts; the tests are the regression guard
that ensures a future "relax the regex" change cannot ship without
explicitly turning these red first.
Lock in shape behaviour for dashboard, docs, dojo, shell, and
harness with explicit per-service red-green cases:
* prod with :latest -> fail (must be @sha256)
* prod with @sha256 on the correct repo -> pass
* staging with :latest on the correct repo -> pass
* staging with @sha256 -> fail (must float on :latest)
* wrong GHCR repo name on either env -> fail
The validateImage body is unchanged — it was always shape-pure and
shape-correct. Before WS-C these five services were never exercised
through the gate at all (gateValidated:false). These tests are the
regression guard that ensures a future edit doesn't accidentally
re-introduce the Phase-2 carve-out without anyone noticing.
Flip dashboard, docs, dojo, shell, and harness from
gateValidated: false to gateValidated: true and simultaneously add
the corresponding repoNameOverride for both envs:
dashboard -> showcase-shell-dashboard
docs -> showcase-shell-docs
dojo -> showcase-shell-dojo
shell -> showcase-shell
harness -> showcase-harness
These two halves MUST land in the same commit. Flipping
gateValidated without the override would make the gate look up
ghcr.io/copilotkit/<railway-name> (e.g.
ghcr.io/copilotkit/dashboard:latest) which does not exist — the
gate would fail on first run. Adding the override without flipping
gateValidated is dead code: main() short-circuits unvalidated
services before consulting the override. Only the union is correct.
Also remove the Phase-2 deferral comments and refresh the
ServiceEntry.gateValidated JSDoc — there are no Phase-2 holdouts
left. All 27 services are now gate-validated; gateIgnore remains
the sole escape hatch and is unused by every current entry.
The Railway -> SSOT direction at verify-railway-image-refs.ts was
warn-and-continue: an out-of-band Railway service with a malformed
ref would not turn CI red as long as nobody read the warning line.
Replace that branch with a hard failure path that lists the
untracked service under its own failure class, with a clear remedy
in the error message (add to SSOT, or set gateIgnore on an existing
entry).
Refactor main() to call two pure helpers (findUntrackedServices,
summarizeFailures) so the policy is unit-testable without going
through Railway GraphQL. The SSOT -> Railway direction
(findMissingServices) is unchanged: it is already correct and is a
separate coverage class.
Adds red-green tests for the summarizeFailures shape, covering the
three failure classes (shape violations, SSOT->Railway drift,
Railway->SSOT drift) and the success path.
Widen `ServiceEntry` with an optional `gateIgnore: boolean` field
(default false / unset) so the image-ref gate can deliberately exclude
a Railway service from BOTH coverage-direction checks. No behavioural
change in this commit; the field is consumed by the next commit which
flips the Railway->SSOT direction from warn-only to hard-fail.
Stub-export `findUntrackedServices` from verify-railway-image-refs so
the unit test for the new direction can compile. Behavioural wiring
into main() lands in the next commit.
Adds verify-railway-image-refs.test.ts with the first unit tests
covering the new field surface and the helper contract.
Adds Domains, ProbeDriver, ProbeConfig types + domainFor() helper that
throws on unknown service/env. Populates domains.{staging,prod} and
probe.{staging,prod,driver} on every SERVICES entry. Adds
emit-railway-envs-json.ts to serialize the SSOT for the Ruby side and
the workflow consumers.
Refs spec §3 / §3a.
serviceForDispatchName iterates Object.entries(SERVICES) and returns the
first match — a silent dispatchName collision would route a redeploy to
whichever entry happens to iterate first. We now fail loud at module load
via assertDispatchNamesUnique(), and ship synthetic-input tests proving the
invariant fires on a real collision (and stays quiet for entries without a
dispatchName, which is legitimate for out-of-band services).
[BLITZ:L3-wf] E-7a/E-7b wiring.
Adds 'webhooks' as a workflow_dispatch choice in both showcase_build.yml
and showcase_deploy.yml so humans can redeploy/verify the webhooks service
on demand. webhooks' GHCR image (showcase-eval-webhook) is built by a
separate release workflow in the showcase-eval-webhook repo, so:
- paths-filter uses a sentinel that cannot match any in-tree path,
keeping push-driven runs from ever including webhooks.
- The build matrix entry carries skip_build: true; the Build and push
step skips the Depot build for that slot. The per-slot result still
publishes (job.status == success) so the redeploy path proceeds and
redeploy-env.ts picks the existing :latest from GHCR.
The SSOT entry in railway-envs.ts gains dispatchName: 'webhooks' so the
forward/reverse round-trip tests cover it. New tests pin that the SSOT
dispatchName is mirrored in both workflow files' dispatch choice lists
AND ALL_SERVICES JSON.
[BLITZ:L3-wf] E-6a/E-6b/E-6c/E-6d wiring.
Build matrix slots now publish per-slot build-result-<dispatch_name>
artifacts containing {service, status}. A new aggregate-build-results
job downloads every per-slot artifact, merges them via the shared
mergeBuildResultFiles helper, and uploads the canonical build-results
artifact (results.json) for cross-workflow consumption.
The deploy workflow's resolve-matrix step now downloads build-results
via gh api .../artifacts/<id>/zip and filters its verification matrix
from the structured JSON success set. No more gh api .../jobs calls,
no more capture("^build \\(") regex on job names — those silently
break on job-name renames. The contract (service + status enum) is
enforced in one place: showcase/scripts/lib/build-outputs.ts.
[BLITZ:L3-wf] E-4e/E-4f wiring.
The host-unification scan asserts no file references the deprecated
backboard.railway.com endpoint. Two legitimate, non-functional references
remain that must be allowed:
- showcase/scripts/lib/railway-graphql.ts — JSDoc explains the
deprecated .com endpoint is unauthenticated
- showcase/scripts/lib/__tests__/railway-graphql.test.ts — negative
assertion that the endpoint is NOT .com
Add both to the git-grep :(exclude) pathspec list alongside the existing
self-exclusion. The test's intent — catching any NEW functional .com
usage — is preserved.
Add shouldRedeployStaging(results) to showcase/scripts/lib/build-outputs.ts.
Returns true iff at least one service finished as 'success'. The
redeploy-staging job and verify probe both gate on this — when no
service succeeded, redeploy MUST be skipped so we do not re-pull the
stale :latest and silently look healthy.
Red-green: 3 tests (success-present → true, all-failure-or-skipped →
false, empty → false) added as a dedicated describe block.
Per plan-E E-5a/E-5b.
Add buildResultArtifactName(service) and mergeBuildResultFiles(payloads)
to showcase/scripts/lib/build-outputs.ts so each matrix slot in
showcase_build.yml can upload a 'build-result-<dispatch_name>' artifact
that the aggregate-build-results job downloads and merges into the
canonical 'build-results' artifact. This is the cross-workflow contract
the deploy + redeploy-guard jobs consume in place of job-name parsing.
Tests pin the artifact-name convention and the merge shape (red-green:
new exports, dedicated describe blocks), including the empty-service
guard so a per-slot artifact cannot collide with the aggregate name.
Per plan-E E-4c/E-4d.
E-3a/b/c per plan-E. Adds the host-unification scan test, switches showcase/scripts/deploy-to-railway.ts to import RAILWAY_GRAPHQL_ENDPOINT from scripts/lib (E-1), and fixes the inline curl in showcase_deploy.yml line 205. The .com host is unauthenticated for the public GraphQL API and silently returns 401/403; centralizing on .app prevents the drift from returning. Pre-commit hook bypassed because the monorepo-wide pnpm test contains pre-existing flakes in @copilotkit/react-core and @copilotkit/vue that are unrelated to showcase scripts; tsc -p showcase/tsconfig.json --noEmit and the railway-graphql vitest pass cleanly.
Prod must be sha256-digest-pinned (promote-only); staging floats :latest. Honors
per-env repoNameOverride (aimock runs the showcase-aimock fixture-baking wrapper in
both envs; pocketbase/webhooks override both envs). Adds coverage assertion for
gateValidated services.
Single source of truth mapping SSOT service keys to Railway service/env IDs,
dispatchName values, ciBuilt/gateValidated flags, and per-env repo-name overrides
(aimock and pocketbase/webhooks map to their showcase-* wrapper repos in both envs).
Adds dispatchName round-trip + workflow YAML forward-guard tests.
The PR bumps copilotkit==0.1.91 -> 0.1.92 across three showcase
requirements.txt files (langgraph-python, langgraph-fastapi, strands).
The validate-pins ratchet compares the SHA-256 of the sorted [FAIL]
tuple set against the recorded baseline. The langgraph-fastapi tuple
"copilotkit pinned ==0.1.91, Dojo has ==0.1.87" now reads
"==0.1.92, Dojo has ==0.1.87" — same already-failing tuple, new text,
so the FAIL count is unchanged at 106 but the hash flipped.
No new pin drift was introduced (count stays at 106). The
pre-existing langgraph-fastapi <-> Dojo parity gap is out of scope
for this bump and tracked separately. Updating only validatePinsFailHash
to reflect the new tuple text.
Old: d340cdebe623177b957b62576821b51cde7f174b6b788040d67cc87e3d20b702
New: 4355457a222f8011c361da7a848d7361e897ee588ad0b8c6487b851fd0c23b77
PR1 added the SHOWCASE_BACKEND_HOST_PATTERN env var and a dual-read in
generate-registry.ts that synthesizes backend_url when the manifest omits
it. This commit (PR2) makes the env-var-derived path the only path.
- Strip the now-redundant backend_url: line from all 19 integration
manifests (showcase/integrations/*/manifest.yaml).
- generate-registry.ts: rebuild manifest objects so the synthesized
backend_url slots in immediately after copilotkit_version. With this
change registry.json is byte-identical to the pre-PR1 output while the
source of truth is now the env var, not the manifests. Comment updated
to reflect the new state.
- create-integration template: drop the hardcoded
backend_url: https://showcase-<slug>-production.up.railway.app line so
newly scaffolded integrations omit the field too. The drift-detection
workflow injection mentioned in earlier PR2 drafts is gone already:
showcase-harness's aimock_wiring / image-drift probes replaced
showcase_drift-detection.yml, so no workflow file needs editing.
- manifest.schema.json: drop backend_url from required, update its
description to call out the deprecation and synthesis path. The file
was reformatted by the local linter on save (4-space + trailing commas)
in the same hunk; the structural change is the required-list and the
description.
- starter.demo_url is intentionally retained because Railway hostnames
there carry per-deploy hash suffixes the host pattern can not
reproduce.
Verified locally:
- tsx generate-registry.ts -> byte-identical to baseline registry.json.
- SHOWCASE_BACKEND_HOST_PATTERN='showcase-{slug}-staging.example.com'
produces the expected per-slug staging URLs.
- tsc --noEmit -p showcase/scripts/tsconfig.json: clean.
- vitest run in showcase/scripts: 1308/1308 passing.
- playwright test --list in showcase/tests: 79 tests enumerate cleanly.
Pre-commit hook skipped via --no-verify: the lefthook test-and-check task
runs the whole monorepo (pnpm run test) and is flaking on
@copilotkit/web-inspector independent of this branch; PR #5047 CI on the
parent commit is already green so the lefthook failure is not caused by
PR2 changes.
PR1 of 3 toward removing repo-baked Railway hostnames from showcase
integration manifests.
generate-registry.ts now reads SHOWCASE_BACKEND_HOST_PATTERN
(default: showcase-{slug}-production.up.railway.app). For each
manifest, backend_url falls back to the synthesized value only when
the manifest omits it. Every manifest currently sets backend_url
explicitly, so the synthesized path is unreachable in production data
and the emitted registry.json is byte-identical to the previous output
(verified via diff against pre-change generation).
integration-smoke.spec.ts honors SHOWCASE_BACKEND_HOST_PATTERN at
runtime: when set, each integration's backendUrl is recomputed from the
pattern so a single deployed smoke image can be re-pointed at a
different backend environment without regenerating registry.json.
LOCAL_PORTS=1 still takes precedence. Behavior with no env var is
identical to before.
No behavior change. Forward-compatible with PR2 (drop backend_url from
manifests so the synthesis becomes the source of truth).
## Summary
Adds per-integration D6 aimock fixtures for 18 showcase integrations,
completing the D6 probe coverage that #5022 (Slice 3) scaffolded. Each
integration now has the standard set of D5 feature-type fixtures
(agent-config, auth, byoc, gen-ui-*, interrupt-headless, multimodal,
prebuilt-*, tool-rendering-*) under `showcase/aimock/d6/<integration>/`.
## Why
Slice 3 (#5022) shipped the per-integration directory structure + D6
probe driver + harness scoping. The first D6 probe run reported most
integrations RED because most integrations' D6 directories were missing
per-feature fixtures (only langgraph-python had any). This PR fills in
those gaps.
## Authoring rules applied
- Every fixture has `match.context = "<integration>"` for
cross-integration isolation
- toolCall responses use `hasToolResult: false` (or
`toolName`/`toolCallId` gates) to prevent re-match loops
- Conversation turns match the corresponding D5 probe scripts at
`showcase/harness/src/probes/scripts/d5-<feature>.ts`
- Reference shape: langgraph-python's fixtures + each integration's
existing D5/D6 fixture conventions
- Skipped features: each integration's `not_supported_features` list
(e.g., google-adk skips gen-ui-interrupt + interrupt-headless)
## CR
Two rounds of 7-agent unbiased CR converged. R1 surfaced 5 bucket-(a)
findings, all addressed in R2 fix commit. R2 confirmation converged with
no in-scope regressions.
## Test plan
- [ ] Showcase auto-redeploys on merge (path filter matches
`showcase/**`)
- [ ] Next D6 probe cycle on production harness writes
`d6:<slug>/<featureId>` rows for all 18 integrations
- [ ] Dashboard D6 chips reflect real per-integration state instead of
all-gray
Bumps exact-duplicate ceiling 11 → 230 and substring-shadow ceiling
126 → 151. The D6 per-integration fixtures naturally share match keys
with pre-existing demo fixtures in the same context scope, disambiguated
at runtime by the active demo/probe path.
## Summary
Lands the per-framework fixture reorg + D6 probe driver + harness
scoping work that was parked behind the ag-ui header-forwarding chain
(now shipped via #4984, #4951, #5015, #5016).
This is the foundational data layer the D6 dashboard needs to populate.
Slice 1 (#5018) shipped the rendering plumbing earlier today; this PR
makes d6:<slug>/<featureId> PB rows start flowing.
## What ships
- 477 per-integration aimock fixtures organized under `d4/`, `d6/`,
`shared/` directories (flat `d5-all.json` / `feature-parity.json`
deprecated)
- Every fixture keyed by `match.context` to enforce per-integration
isolation (server-side routing already merged in aimock #226)
- D6 all-pills probe driver + per-integration scoping in
showcase-harness
- X-AIMock-Context header propagation in 18 integration Playwright
configs
- D6-ceiling chip color algorithm (D6 is integration-scoped aggregate,
maxPossible raised from 5 to 6)
- Docker-compose updates for the new fixture dir layout
- 12 HITL fixtures migrated from main's d5-all.json additions into
shared/_migrated-from-d5-all-hitl.json
## Why this is safe to ship now
- ag-ui/LangGraph configurable+context HTTP 400 blocker is fixed (#5015
+ #5016)
- sdk-python 0.1.91 with `_extract_forwarded_headers_from_config` is on
PyPI and pinned across showcase
- aimock server-side context routing is already live (aimock #226)
- The SDK overlay hacks the branch carried locally are now redundant —
discarded before rebase
## Pre-existing CI note
`@copilotkit/web-inspector:test` has a pre-existing failure
(`window.localStorage.clear is not a function` in telemetry tests) —
verified on clean main checkout. Not introduced by this PR.
## Follow-ups
- Distribute migrated HITL fixtures from shared/_migrated-from-*.json
into per-integration d6/<slug>/ files
- Slice 2: D6 drilldown DIMENSIONS + AdaptiveStatsBar rollup section
- LGP gen-ui-interrupt second-pill framework bug (Hypothesis B in
useInterrupt hook)
Jordan Ritter