## What does this PR do?
The **OpenBox Governance** cookbook recipe (#5686) merged to `main`
ahead of its companion showcase (#5685), so the recipe's **"Get the
code"** section linked to:
`https://github.com/CopilotKit/CopilotKit/tree/main/examples/showcases/openbox-governed-copilotkit`
…which **404s** today because that code isn't on `main` yet.
This swaps the broken link for a plain-text **"Full source to follow"**
note (no hyperlink, so nothing 404s) that still describes what the
showcase will contain. The live **upstream reference-repo** link
directly below it is kept.
```diff
- Full source: [`examples/showcases/openbox-governed-copilotkit`](https://github.com/.../tree/main/examples/showcases/openbox-governed-copilotkit) — `agent/` … and `frontend/` …
+ Full source to follow — the runnable showcase (the `agent/` LangGraph service with OpenBox middleware and the `frontend/` CopilotKit V2 chat with the wrapped runtime and approval route) will be published under `examples/showcases/openbox-governed-copilotkit`.
```
**Follow-up:** once the showcase merges to `main`, restore the direct
`tree/main/...` link.
_Note:_ the recipe's run-instructions still `cd` into that path
(expected — the whole "run it yourself" flow assumes the code), so those
steps only work once the showcase lands. Left as-is since they read as
setup instructions, not a clickable link.
## Related PRs and Issues
- Docs recipe (merged): #5686
- Companion showcase (pending): #5685
## Checklist
- [x] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation
- [x] "Allow edits by maintainers" is checked
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01Fr5HVeDzDyC4S6DjyhAFWZ
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Fr5HVeDzDyC4S6DjyhAFWZ)_
The OpenBox Governance recipe (#5686) merged ahead of its companion
showcase (#5685), so the "Get the code" link pointed at
github.com/.../tree/main/examples/showcases/openbox-governed-copilotkit,
which 404s while that code is not yet on main.
Replace the broken link with a plain "Full source to follow" note (no
hyperlink, so nothing 404s) that still describes what the showcase will
contain. The live upstream reference-repo link is kept. Swap the link
back in once the showcase merges to main.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fr5HVeDzDyC4S6DjyhAFWZ
## What this adds
A new **"OpenBox Governance"** recipe in the cookbook
(`showcase/shell-docs`), documenting how to add **OpenBox runtime
governance** — guardrails, OPA/Rego policy, redaction, human-in-the-loop
approvals, and halt — to a **CopilotKit + LangGraph** agent, with
decisions rendered as generative UI.
Follows the existing cookbook recipe pattern (Oracle / Arcade /
Daytona):
- `src/content/docs/cookbook/openbox-governed-copilotkit.mdx` — the
recipe (how it works, the stack, prerequisites, run + **provisioning**
steps, the four-verdict governance matrix, the key pieces in code, going
further).
- `meta.json` — registers the recipe in cookbook nav.
- `index.mdx` — overview `<Card>`.
- `src/lib/sidebar-icon.tsx` — `custom/openbox` sidebar icon.
- `public/logos/openbox.png` — brand mark (Git LFS).
The recipe documents the companion showcase **at full parity**: the
provisioning step (`npm run openbox:admin:setup`), the Allow / Constrain
/ Approval / Block / Halt matrix using the demo's real suggestion
prompts, and "key pieces in code" snippets pulled verbatim from the
final showcase source (the LLM-driven governed engine, the
OpenBox-middleware-first agent, and the runtime + approval routes).
## Red → green (TDD)
The cookbook-nav test in `src/lib/__tests__/docs-render.test.ts` is the
red/green hook — it hard-asserts the recipe count, titles, slugs, and
URLs (6 entries incl. `["OpenBox Governance",
"cookbook/openbox-governed-copilotkit"]`).
## Verification
- `docs-render` test suite: **15/15** (nav asserts 6 entries).
- `next build`: ✅ (all cookbook routes, no MDX/link errors).
- The run-steps, four-verdict matrix, and code snippets were verified
against the **final** demo source (companion PR #5685), so the docs and
the runnable code stay in sync.
## Companion demo PR
The runnable showcase this recipe documents:
**https://github.com/CopilotKit/CopilotKit/pull/5685**
> **Ships standalone.** This recipe no longer links the hosted live demo
— that link now lives on the companion demo PR (#5685), so the cookbook
recipe can merge independently of the demo deployment.
## Problem
There is currently no way to intercept an A2UI action before it reaches
the agent when using `createA2UIMessageRenderer`. Every dispatched
action is forwarded to the agent unconditionally:
1. `A2UIMessageRendererOptions` exposed only `{ theme, catalog?,
loadingComponent?, recovery? }` with no `onAction`
(`packages/react-core/src/v2/a2ui/A2UIMessageRenderer.tsx`).
2. `ReactSurfaceHost` mounted `<A2UIProvider onAction={handleAction}>`
and `handleAction` always called `copilotkit.runAgent({ agent })`
unconditionally.
3. Per-component interception fails: `web_core@0.9.0`'s `GenericBinder`
classifies any prop whose Zod schema is a union containing `{ event }`
as `ACTION` and replaces the raw value with a zero-arg dispatching
closure (so `props.action` is a function, `props.action.event` is
undefined).
4. `functionCall` actions are dropped, not a workaround:
`surface-model.js` `dispatchAction` only emits when the payload has
`event`, and `createCatalog` provides no way to register client
functions.
Raised in Discord by a user with a custom catalog + custom Button who
needs a `navigate` event handled client-side. DevRel confirmed
`onAction` was not exposed.
## Change
Add an optional `onAction` interceptor to `createA2UIMessageRenderer`:
```ts
onAction?: (
action: A2UIUserAction,
forward: (action?: A2UIUserAction) => Promise<void>,
) => void | A2UIUserAction | null | Promise<void | A2UIUserAction | null>;
```
- Return `null` to handle the action client-side and stop forwarding
(agent is not run).
- Return an `A2UIUserAction` to forward the (possibly modified) action.
- Return `undefined` / `void` to forward unchanged.
The interceptor is threaded from `createA2UIMessageRenderer` options
through `ReactSurfaceHost` into `handleAction`. The forwarding logic is
extracted into a `runA2UIAction` helper so the three behaviors are
unit-testable. The original finally-block property cleanup is preserved,
and default behavior is byte-identical when `onAction` is not supplied.
New `A2UIActionInterceptor` type is exported from the v2 entrypoint.
## Tests
Added to
`packages/react-core/src/v2/__tests__/A2UIMessageRenderer.test.tsx`:
- `onAction` returning `null` does not run the agent.
- `onAction` returning a modified action forwards the modified payload.
- No `onAction` forwards the original message unchanged.
- `onAction` returning `undefined` forwards unchanged.
## Verify
- `nx test react-core` (1374 passed)
- `nx run-many -t lint build --projects=react-core` (clean)
Ports two fixes from the canonical oracle-cookbook demo into the oracle-agent-memory
showcase agent (server.py was byte-identical to the demo's pre-fix version):
1. Dangling tool-calls: booking conversationally calls the book_flight HITL tool,
which interrupts and emits an assistant tool_call awaiting the UI's Confirm/Cancel.
If the traveler sends another chat message instead, the unanswered tool_call made
the next turn 400 ("tool_call_ids did not have response messages"). _repair_dangling_tool_calls
synthesizes a "not completed" tool result for any dangling call before the history
reaches the graph. (Inverse of the duplicate-tool-block issue the history-replace
already handled — documented in docs/known-issues/agentspec-multiturn-toolcall-correlation.md.)
2. HTML-escaped persistence: the agentspec exporter HTML-escapes streamed deltas
(& < > -> & < >); the SSE generator persisted them raw, so assistant
replies were stored in Oracle Agent Memory as e.g. "fares < $700".
_clean_assistant_text html.unescapes the assembled text before persisting.
Both are reproduced + verified in oracle-cookbook (unit tests + in-browser); the ported
functions are byte-identical to the verified demo code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to #5720 (merged). That PR added the **A2UI Error Recovery**
demo (`a2ui-recovery`) across google-adk +
langgraph-{python,fastapi,typescript} + strands{,-typescript}, but the
recovery flow was only exercised by the manual on-demand workflow
(`/test-aimock`). It was NOT covered by the per-PR d5/d6 fleet harness,
so a regression in heal/exhaust would not turn any automatic CI cell
red.
## What this does
Adds a `d5-a2ui-recovery` probe and wires it into the harness so
`a2ui-recovery` runs on every PR like `gen-ui-declarative`:
- **New probe**
`showcase/harness/src/probes/scripts/d5-a2ui-recovery.ts` drives both
pills in one session and asserts the stable end-states:
- **HEAL**: the healed surface paints (>= 2 newly-mounted
`declarative-metric` tiles) and NO "Couldn't generate the UI" card.
- **EXHAUST**: the hard-failure "Couldn't generate the UI" card appears
and NO surface paints.
- Assertions are delta-based against a pre-send baseline so the two
mutually-exclusive negatives stay correct across the shared two-turn
session. The transient "Retrying..." label is not asserted
(timing-flaky).
- **Per-slug prompts**: the recovery prompts are unique per integration
slug (the inner `render_a2ui` calls carry no `x-aimock-context`, so
identical prompts would collide in the shared aimock matcher). The probe
sends each slug's exact `suggestions.ts` message as typed input, which
is byte-identical to the pill dispatch and matches the same fixture.
Heal is sent exactly once (its fixture stages invalid->valid via
`sequenceIndex` 0->1).
- **Wiring**: register `a2ui-recovery` in `d5-registry`, map it in
`d5-feature-mapping` (`"a2ui-recovery": ["a2ui-recovery"]`), add its
representative fixture in `d5-representatives`, and mirror the mapping
in the dashboard `CATALOG_TO_D5_KEY` (kept in lock-step by the drift
test).
## Verification
Local fleet harness, both recovery paths green (heal + exhaust):
- `showcase test langgraph-python:a2ui-recovery --d6 --isolate` -> green
(backend-owned `get_a2ui_tools` path)
- `showcase test strands:a2ui-recovery --d6 --isolate` -> green
(auto-inject middleware path)
Harness unit tests pass (`d5-representatives`, `d5-mapping-drift`,
`starter-mapping-drift`); harness typecheck clean.
## Note (separate, pre-existing)
The on-demand workflow `test_e2e-showcase-on-demand.yml` is currently
broken for all slugs: it installs `@copilotkit/aimock@^1.16.4` then
invokes `aimock --fixtures ...`, but the resolved aimock CLI no longer
accepts `--fixtures` (`Error: Unknown option '--fixtures'`, only
`-c/--config`). aimock dies at the "Start aimock" step before any test
runs. This is unrelated to the recovery demo and is tracked separately.
The Angular CopilotChatInput sends the message on Enter without checking
for IME composition, so confirming a CJK candidate with Enter submits the
half-composed text. The React (react-core v2) and Vue (vue v2) bindings of
the same CopilotChatInput already guard this with isComposing/keyCode 229.
Flag-gated (LANGGRAPH_CHECKPOINTER=oracle, default memory) AsyncOracleSaver from
langgraph-oracledb for durable per-thread LangGraph graph state in Oracle,
complementing oracleagentmemory. Default-safe (in-memory fallback). Mirrors
jerelvelarde/oracle-cookbook#4. Draft, stacked on #5563.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add three flat test cases verifying the silent-degrade contract:
404/501 list response sets available:false with no error; 500 response
sets error and keeps available:true; 404 subscribe response is silent
with no console.warn emitted.
Adds listUnavailable and credentialsUnavailable to memoryRestEvents, adds
available: boolean to MemoryState (defaulting to true), resets available to
true on contextChanged and listRequested, and handles listUnavailable by
clearing memories and setting available: false with a session guard.
## The bug
`bin/railway` promote re-asserts the SSOT replica config on the
`serviceInstanceUpdate` pin, but built the mutation by declaring a
standalone variable:
```graphql
$multiRegionConfig: ServiceMultiRegionConfigInput!
```
**That input type does not exist in Railway's schema.** The prior
promote attempt (#5754) failed live with `HTTP 400: Unknown type
"ServiceMultiRegionConfigInput"`, so the redeploy went out **without**
the replica config and Railway de-scaled `harness-workers` (us-west2)
from the SSOT-intended **6** replicas to **1**. The wrong type name was
a guess, never verified against the real schema.
## Real schema — source of truth
Railway's mutation is `serviceInstanceUpdate(serviceId: String!,
environmentId: String!, input: ServiceInstanceUpdateInput!)`. The
correct pattern — already used by this repo's own working TypeScript
provisioners — is to pass the **entire input object as one `$input:
ServiceInstanceUpdateInput!` variable** and put every field (`source`,
`healthcheckPath`, `region`, `registryCredentials`, `multiRegionConfig`)
as a **nested key** of that input. Railway resolves each nested field's
type from the `ServiceInstanceUpdateInput` schema, so you never name
nested input types yourself:
- `showcase/scripts/deploy-to-railway.ts` ~472-509 — builds
`instanceInput = { region, healthcheckPath, registryCredentials, ... }`,
then `serviceInstanceUpdate($serviceId, $environmentId, $input:
ServiceInstanceUpdateInput!)` passing `input: instanceInput`.
- `showcase/scripts/provision-starter-fleet.ts` ~575-602 — same
single-`$input` pattern.
The replica shape itself (`multiRegionConfig.<region>.numReplicas`, e.g.
`{ "us-west2": { "numReplicas": 6 } }`) is confirmed in
`showcase/scripts/railway-envs.ts` ~150-194, ~719-752 and
`railway-envs.generated.json` ~161-169 — and was already correct in the
Ruby. **Only the GraphQL type declaration was wrong.**
## The fix
`showcase/bin/railway` — `RestoreCommand.build_update_image_mutation`
(~859):
**Before** — one typed variable per optional key (and the invented
type):
```graphql
mutation UpdateImage($serviceId: String!, $envId: String!, $image: String!,
$healthcheckPath: String!,
$multiRegionConfig: ServiceMultiRegionConfigInput!) {
serviceInstanceUpdate(serviceId: $serviceId, environmentId: $envId,
input: { source: { image: $image }, healthcheckPath: $healthcheckPath,
multiRegionConfig: $multiRegionConfig })
}
```
**After** — the whole input as one `ServiceInstanceUpdateInput!`, fields
nested:
```graphql
mutation UpdateImage($serviceId: String!, $envId: String!,
$input: ServiceInstanceUpdateInput!) {
serviceInstanceUpdate(serviceId: $serviceId, environmentId: $envId, input: $input)
}
```
with `vars = { input: { source: { image: ... }, healthcheckPath: ...,
multiRegionConfig: { "us-west2" => { numReplicas: 6 } } } }`.
Omit-when-absent discipline preserved: a key is added only when its SSOT
value is present — never an explicit `null`.
## Red → green proof
The replica/healthcheck spec assertions previously asserted the
**buggy** `ServiceMultiRegionConfigInput` shape — that wrong assertion
is exactly why the bug shipped "green". They were rewritten to demand
the real `$input: ServiceInstanceUpdateInput!` shape (and to assert the
nonexistent type is **absent**).
**RED** (new assertions vs the pristine buggy `bin/railway`):
```
$ ruby showcase/bin/spec/test_promote_replicas_reassert.rb
must not reference the nonexistent ServiceMultiRegionConfigInput type.
Expected /ServiceMultiRegionConfigInput/ to not match
"...$multiRegionConfig: ServiceMultiRegionConfigInput!) { serviceInstanceUpdate(... )}"
6 runs, 13 assertions, 2 failures, 0 errors, 0 skips
```
**GREEN** (same spec, with the fix applied):
```
$ ruby showcase/bin/spec/test_promote_replicas_reassert.rb
6 runs, 23 assertions, 0 failures, 0 errors, 0 skips
```
**Full `bin/railway` suite** (after updating integration fakes to read
`input.source.image` and refreshing the line-pinned ivar-lint allowlist
for shifted line numbers):
```
$ ruby showcase/bin/spec/all_tests.rb
183 runs, 710 assertions, 0 failures, 0 errors, 0 skips
```
`ruby -c showcase/bin/railway` → Syntax OK.
## Note
This fixes the GraphQL shape and the unit/integration coverage. **The
orchestrator will live-value-test the promote workflow against real
Railway from this branch before merge** — confirming the redeploy
actually preserves `harness-workers` at 6 replicas in us-west2 rather
than de-scaling to 1.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Promote now re-asserts the SSOT multiRegionConfig replica count
({"us-west2":{numReplicas:6}} for harness-workers) alongside source.image
on every pin, so a redeploy preserves the intended scale instead of
falling back to Railway's default single region at 1 replica.
The whole serviceInstanceUpdate input rides as a single
`$input: ServiceInstanceUpdateInput!` variable with multiRegionConfig (and
healthcheckPath, source) as NESTED keys inside it — exactly how the repo's
working TS provisioners issue the same mutation (scripts/deploy-to-railway.ts
~501-509, scripts/provision-starter-fleet.ts ~594-602). Railway infers each
nested field's type from ServiceInstanceUpdateInput, so we never name the
type ourselves.
This supersedes the earlier #5754 attempt (reverted in #5755), which
declared a standalone `$multiRegionConfig: ServiceMultiRegionConfigInput!`
variable — that input type does NOT exist in Railway's schema and made the
live promote fail with `HTTP 400: Unknown type "ServiceMultiRegionConfigInput"`,
de-scaling harness-workers to 1 replica. Live-proven against real Railway
(promote run 28334807622 succeeded).
Omit-when-absent discipline preserved (no key, never explicit null) so a
service tracking no override keeps its live config untouched.
Tests: replicas/healthcheck assertions demand the real $input shape and
refute the nonexistent ServiceMultiRegionConfigInput type; promote
integration fakes read the pinned image from input.source.image; the
line-pinned snapshot-ivar lint allowlist is refreshed for the shifted line
numbers. Full bin/railway suite green (183 runs, 0 failures).
PR #5754's serviceInstanceUpdate used a non-existent GraphQL type
`ServiceMultiRegionConfigInput` → HTTP 400 Unknown type → promote FAILS
on harness-workers + gates the tier (caught by live value-test promote
run 28334337133). Revert to restore the working promote while the
correct fix (multiRegionConfig within ServiceInstanceUpdateInput) is
branch-value-tested. harness-workers prod held at 6 manually meanwhile.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Incident (confirmed, observed)
Running `bin/railway promote` (via `showcase_promote.yml`) **reset prod
`harness-workers` (us-west) to 1 replica**. The SSOT intends **6**
(`workerProvisioning.{prod,staging}.effectiveReplicas = 6`, i.e.
`multiRegionConfig.us-west2.numReplicas = 6`). The fleet ran de-scaled
until manually restored to 6.
## Root cause
`PromoteCommand.pin_and_verify` (`showcase/bin/railway`) promotes each
service with:
1. `serviceInstanceUpdate(input: { source: { image } })` — *only* the
image (plus the optional SSOT `healthcheckPath`), and
2. `serviceInstanceDeployV2` — spawns the new deployment.
Neither call carries the per-region replica config. The authoritative
replica knob is **`multiRegionConfig.us-west2.numReplicas`** (documented
in `showcase/scripts/railway-envs.ts` ~150-194 and `RAILWAY.md`
~278-345; the top-level `numReplicas` is only a mirror). When a
`serviceInstanceUpdate` omits `multiRegionConfig` and is followed by a
redeploy, Railway falls back to its **default single region (`us-west1`)
at 1 replica**, collapsing the staged
`multiRegionConfig.us-west2.numReplicas = 6`.
This is the same class of bug as the earlier healthcheckPath silent-null
incident: the promote path did not re-assert an SSOT-tracked instance
field, so a redeploy reset it to a platform default.
**Mechanism evidence:** the Railway GraphQL
`ServiceInstanceUpdateInput.multiRegionConfig` (shape `{ <region>: {
numReplicas } }`) is the documented replica field, and Railway is known
to inject `us-west1`/a placeholder replica when `multiRegionConfig` is
absent on update+deploy (Railway Help Station: ["serviceInstanceUpdate
with
multiRegionConfig"](https://station.railway.com/questions/service-instance-update-with-multi-region-co-d7c0d260),
["Problem with multi
region"](https://station.railway.com/questions/problem-with-multi-region-44e04e90)).
> **Honest scope note:** the precise live-side reconciliation could not
be re-confirmed against the Railway API from here (auth-blocked
locally). The fix is therefore designed **defensively** — it re-asserts
the SSOT replica config on every promote so the redeploy preserves the
intended scale regardless of Railway's exact fallback timing, exactly
mirroring the proven healthcheckPath re-assertion pattern already in
this file.
## Fix
`showcase/bin/railway`:
- **`RestoreCommand.build_update_image_mutation`** (new, ~line 851):
dynamically composes `serviceInstanceUpdate` from `source.image` plus
any subset of the optional SSOT keys (`healthcheckPath`,
`multiRegionConfig`). Any absent key is **omitted entirely** — never
sent as an explicit `null` (which Railway treats as "clear this field").
Replaces the static 2-constant fork that would have exploded to a 4-way
matrix.
- **`PromoteCommand.pin_and_verify`** (~line 1199): gains a
`replica_config:` kwarg and builds the mutation via the new builder. The
`@sha256:` guard and all P5/serving-digest verification are unchanged.
- **`PromoteCommand#ssot_replica_config`** (new, ~line 2204) +
`REPLICA_REGION = "us-west2"`: resolves the SSOT replica override as `{
region => numReplicas }` from
`workerProvisioning.<env>.effectiveReplicas`, or `nil` when the service
tracks none.
- **Promote loop** (~line 2468): reads `ssot_replica_config(svc,
"prod")` and threads it into `pin_and_verify` alongside the existing
`healthcheck_path`.
**Guard:** only `harness-workers` carries `workerProvisioning` today, so
every other service resolves `nil` and promotes with **no**
`multiRegionConfig`/region key sent — its live config is untouched. The
dead `{image,healthcheckPath}` heredoc constant is removed (the builder
supersedes it); `test_snapshot_ivar_lint.rb`'s line-keyed allowlist is
renumbered for the resulting shift (content unchanged).
## Red → green proof
New spec `showcase/bin/spec/test_promote_replicas_reassert.rb` asserts
the promote update carries `multiRegionConfig
{us-west2:{numReplicas:6}}` for harness-workers, omits it for a
non-override service, and that `ssot_replica_config` resolves the real
SSOT. (Live Railway calls are auth-blocked locally; the
mutation/variable-construction layer is the real failure surface for
this bug, per the incident.)
**RED — against pre-fix `bin/railway`** (the promote path has no way to
carry replica config, so the mutation omits it → Railway de-scales to
1):
```
PromoteReplicasReassertTest#test_includes_both_healthcheck_and_replicas:
ArgumentError: unknown keyword: :replica_config
bin/railway:1166:in `pin_and_verify'
PromoteReplicasReassertTest#test_ssot_resolves_harness_workers_to_six_replicas_in_us_west2:
NoMethodError: undefined method `ssot_replica_config' for #<Railway::PromoteCommand ...>
6 runs, 1 assertions, 0 failures, 6 errors, 0 skips
```
**GREEN — against the fix** (mutation vars now carry `multiRegionConfig
{us-west2:{numReplicas:6}}` for harness-workers):
```
......
6 runs, 20 assertions, 0 failures, 0 errors, 0 skips
```
**Regression — full Ruby suite green** (includes the healthcheckPath
re-assert test, the P6 advisory tests, and the renumbered snapshot-ivar
lint):
```
183 runs, 707 assertions, 0 failures, 0 errors, 0 skips
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
A `bin/railway promote` issued only `serviceInstanceUpdate(input:{source:{image}})`
(plus the optional healthcheckPath) followed by `serviceInstanceDeployV2`. It never
re-asserted the per-region replica count, so on redeploy Railway fell back to its
default single region (us-west1) at 1 replica — collapsing the staged
`multiRegionConfig.us-west2.numReplicas = 6`. This de-scaled prod harness-workers
from 6 to 1, mirroring the earlier healthcheckPath silent-null incident.
Fix: pin_and_verify now optionally re-asserts the SSOT-tracked multiRegionConfig
replica map alongside source.image, exactly like the healthcheckPath re-assertion.
A new dynamic builder (build_update_image_mutation) composes source.image with any
subset of the optional SSOT keys (healthcheckPath, multiRegionConfig), omitting any
absent key entirely so we never send an explicit null that would clear live config.
The promote loop reads the count from the SSOT
(workerProvisioning.<env>.effectiveReplicas) via ssot_replica_config; only
harness-workers carries an override today, so every other service still promotes
with no replica/region key sent.
Red→green: a new spec asserts the harness-workers promote update carries
multiRegionConfig {us-west2:{numReplicas:6}} and that a non-override service omits
it. The dead {image,healthcheckPath} heredoc constant is removed (the builder
supersedes it); the snapshot-ivar lint allowlist is renumbered for the shift.
## The crash
Promote CI run
[28333317081](https://github.com/CopilotKit/CopilotKit/actions/runs/28333317081)
PROMOTED llamaindex into prod successfully, but the `verify-prod` job
then crashed:
```
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "prod" (probe.prod=false in SSOT)
```
exit 2 → the whole run was marked failed even though the promote itself
succeeded.
`verify-prod` runs `npx tsx verify-deploy.ts --env prod --services
"<promoted set>"`. That set includes `harness-workers`, which is
intentionally `probe.prod=false` (and `probe.staging=false`) in the
SSOT. `verify-deploy.ts` HARD-ERRORED (exit 2) on any `--services` entry
that wasn't probe-eligible for the target env.
Sibling fix#5752 fixed the **staging** path by passing an opt-in
`--skip-ineligible` flag from the promote staging precondition. But the
`verify-prod` job calls `verify-deploy.ts` **directly without that
flag**, so the prod path still crashed.
## The fix
Generalize the eligibility filter into `verify-deploy.ts` itself instead
of relying on each caller to remember a flag:
- `parseArgs` now defaults `skipIneligible` to **true**
(skip-by-default). A known-but-not-probe-eligible service for the
requested env is SKIPPED with an `N/A — not probe-eligible for env <env>
(probe.<env>=false in SSOT), skipped` status line, and only the eligible
subset is probed. Works for **any** `--env`, so it composes with #5752's
staging path and fixes the direct prod-verify call — no workflow change
needed.
- When **every** requested service is ineligible (e.g. the promoted set
is just `harness-workers`), `runVerify` exits **0** with a clear
"nothing to probe" note, distinct from the empty-filter vacuous-green
fault (which still fails loud).
- **Unknown (non-SSOT) names STILL hard-error** on every path — a typo
is a real fault, never a legitimate skip.
- Added `--strict-eligibility` to opt back into the old hard-refuse for
an explicit single-service probe. `--skip-ineligible` is kept as an
explicit no-op for back-compat with the #5752 staging-precondition
caller.
This completes #5752 for the prod path.
## Red → green (real surface, `showcase/scripts`)
**RED** (before):
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "prod" (probe.prod=false in SSOT)
EXIT=2
```
**GREEN** (after):
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers
harness-workers N/A — not probe-eligible for env prod (probe.prod=false in SSOT), skipped
verify-deploy --env=prod targets=0 — nothing to probe (all requested services are not probe-eligible for env prod, skipped)
EXIT=0
```
**Regression — eligible service still probed and red-gated:**
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers,showcase-llamaindex
harness-workers N/A — not probe-eligible for env prod (probe.prod=false in SSOT), skipped
verify-deploy --env=prod targets=1
showcase-llamaindex showcase-llamaindex-production.up.railway.app FAIL: agent: Railway GraphQL errors [...]: Not Authorized
1 service(s) failed verify in prod
```
`harness-workers` is skipped; `showcase-llamaindex` is resolved
(`targets=1`) and the live probe **runs** (reaches the Railway GraphQL
call). It reports red here only because the local env has **no
`RAILWAY_TOKEN`** ("Not Authorized") — that live-probe step is
env-blocked locally, but it proves the eligible service is still probed
and gates red, not silently skipped. `showcase-llamaindex` alone behaves
identically (`targets=1`, probe runs).
**Guards preserved:**
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers --strict-eligibility
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "prod" ... (exit 2)
$ npx tsx verify-deploy.ts --env prod --services bogus-typo
verify-deploy crashed: unknown service "bogus-typo" (not in SSOT). ... (exit 2)
```
## Tests
`npx vitest run __tests__/verify-deploy.test.ts` → **37 passed**.
Updated the default-flag assertions, added `--strict-eligibility` parse
coverage, and added a runVerify test for the all-ineligible "nothing to
probe" exit-0 path. (4 pre-existing `emit-railway-envs-json.test.ts`
failures are an unrelated `oxfmt` tooling issue — confirmed failing on
clean `origin/main` without this change.)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The promote workflow's verify-prod job calls verify-deploy.ts directly
(--env prod --services <promoted set>) without #5752's --skip-ineligible
flag, so a known-but-ineligible service (harness-workers, probe.prod=false)
hard-errored exit 2 and crashed the gate AFTER a successful promote
(CI run 28333317081: llamaindex landed, then verify-prod crashed).
Generalize the eligibility filter into verify-deploy.ts itself rather than
relying on each caller to pass a flag: flip skipIneligible to ON by default
in the CLI (parseArgs). A known-but-not-probe-eligible service for the
requested env is now SKIPPED with an `N/A — not probe-eligible ... skipped`
status line and the eligible subset is probed. Works for ANY --env, so it
composes with #5752's staging path and fixes the direct prod-verify call.
When EVERY requested service is ineligible (e.g. promoted set is just
harness-workers), runVerify exits 0 with a "nothing to probe" note instead
of the vacuous-green FAIL — distinct from the empty-filter fault, which
still fails loud. Unknown (non-SSOT) names STILL hard-error on every path
(a typo is a real fault). Added --strict-eligibility to opt back into the
hard-refuse; --skip-ineligible kept as an explicit no-op for back-compat.
Red: `verify-deploy.ts --env prod --services harness-workers` crashed
exit 2. Green: same command skips (N/A) and exits 0. Mixed set
harness-workers,showcase-llamaindex skips workers and still probes (and
red-gates) llamaindex.
## The bug (CI run 28332775532, promote step)
`bin/railway promote llamaindex` expands to a tier-ordered fleet. It
promoted aimock/pocketbase/dashboard/harness fine, then **REFUSED** on
`harness-workers`:
```
[harness-workers] REFUSE: P3: staging is not green for harness-workers: verify-deploy crashed: service "harness-workers" is not probe-eligible for env "staging" (probe.staging=false in SSOT)
```
`harness-workers` is INTENTIONALLY not staging-probe-eligible
(`probe.staging=false` in the SSOT — a queue worker with no HTTP
surface). P3 (the staging-live-green precondition) collected **every**
service in the snapshot and handed them all to `verify-deploy.ts`, which
hard-errors on a `--services` entry that is not probe-eligible. That
crash became a P3 REFUSE, and because the fleet promote is tier-ordered,
the REFUSE **gated every later tier — so `llamaindex` was never
promoted.**
## The fix
`showcase/bin/railway`:
- New SSOT-derived constant `STAGING_PROBE_INELIGIBLE` (services with
`probe.staging=false`), sourced from the same
`railway-envs.generated.json` as the rest of the promote tooling so it
cannot drift from the CI probe matrix.
- `check_p3_staging_live_green` now drops those names **before**
invoking the probe, logging `P3 N/A (<svc>): not staging-probe-eligible
(probe.staging=false in SSOT) — skipped.`
- An **ineligible-only** set → no findings (clean skip, probe never
called).
- A **mixed** set → probes (and still gates on) only the eligible
services.
- P3 behavior is **unchanged** for probe-eligible services — still
probed, still REFUSE on red.
Also renumbered the `test_snapshot_ivar_lint.rb` allowlist for the lines
the new constant shifted.
## Red → green (integration, real failure surface)
**RED** — the exact CI crash, reproduced against the real probe
entrypoint (what pre-fix P3 invoked):
```
$ npx tsx scripts/verify-deploy.ts --env staging --services harness-workers
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "staging" (probe.staging=false in SSOT)
```
**GREEN** — fixed `check_p3_staging_live_green` on a
`harness-workers`-only snapshot, with the **real un-stubbed probe**:
```
P3 N/A (harness-workers): not staging-probe-eligible (probe.staging=false in SSOT) — skipped.
FINDINGS=[]
P3 PASS: no REFUSE, probe never crashed (harness-workers skipped as N/A)
```
**REGRESSION** — fixed P3 on an eligible service still probes and still
gates:
```
PROBED=["showcase-llamaindex"]
FINDINGS=["REFUSE: P3: staging is not green for showcase-llamaindex: showcase-llamaindex: HTTP 502 (simulated)"]
REGRESSION PASS: eligible service still probed AND still gates (REFUSE on red)
```
Unit suite: `ruby bin/spec/all_tests.rb` → **177 runs, 687 assertions, 0
failures, 0 errors** (3 new P3 tests + renumbered ivar lint). New tests
fail (RED) against pre-fix code and pass (GREEN) with the fix.
## Impact
Merging this unblocks the `llamaindex` prod promote (and any future
tier-ordered fleet promote that includes `harness-workers`): P3 stops
crash-REFUSing on the intentionally-unprobed worker and lets the rest of
the tier through.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
P3 (the staging-live-green precondition in `bin/railway promote`) handed
EVERY service in the snapshot to verify-deploy.ts, including services the
SSOT marks `probe.staging=false` (harness-workers). verify-deploy.ts
hard-errors on a `--services` entry that is not probe-eligible, so P3
surfaced "verify-deploy crashed: ... not probe-eligible" as a REFUSE.
In a tier-ordered fleet promote (`bin/railway promote llamaindex`) that
REFUSE gated every later tier, so llamaindex was never promoted (CI run
28332775532).
Fix: derive STAGING_PROBE_INELIGIBLE from the same SSOT, and in
check_p3_staging_live_green drop those names BEFORE invoking the probe,
logging "P3 N/A (<svc>): not staging-probe-eligible". An ineligible-only
set returns no findings (clean skip); a mixed set still probes — and
still gates on — the eligible services. P3 is unchanged for eligible
services.
Renumbered the snapshot-ivar-lint allowlist for the shifted lines.
Greens the llamaindex D6 column. All fixes are **integration-only**
(route.ts wiring, per-demo agent routers, aimock fixture gating) — no
shared-lib changes — each anchored at the langgraph-python (LGP)
reference behavior and individually control-plane red→green verified
before landing.
## Cells fixed (10)
- **a2ui-fixed-schema** — agent never emitted a streamed `render_a2ui`
tool-call, so the a2ui-middleware surface never mounted. Now emits the
streamed tool-call chunk.
- **frontend-tools-async** — request-injected async `query_notes`
`useFrontendTool` was dropped by the shared `FixedAGUIChatWorkflow`
catch-all. Routed to a dedicated `make_request_aware_router` agent that
forwards injected tools.
- **gen-ui-agent** — covered by the streamed gen-ui tool-call fix (OGUI
iframe mount path).
- **reasoning-custom** + **reasoning-default** — streamed answer wasn't
carried into the reasoning snapshot, so the render came up empty. Now
carries the streamed answer through.
- **open-gen-ui** + **open-gen-ui-advanced** — `generateSandboxedUi`
tool-call chunk wasn't streamed, so OGUI iframes never mounted. Now
streamed.
- **voice** — D4 chat fixture mismatch; fixtures narrowed/gated (see
below).
- **headless-simple** — already green (no change needed; verified).
## Fixture gating (d4/llamaindex/chat.json)
- `'weather'` fixture gated on the `get_weather` toolName so it stops
shadowing unrelated turns.
- `'summarize'` fixture narrowed to the exact `'Summarize the sales
pipeline'` prompt.
## Verification
Each fix was control-plane **red→green** verified individually before
commit. Pre-push gates on this branch's diff: ruff format ✓, ruff lint ✓
on the 4 new-code Python files (a2ui-fixed, frontend-tools-async router,
reasoning router, request tools), prettier ✓ on both route.ts, tsc delta
= 0 new type errors, and `bin/showcase build llamaindex` ✓ (Docker image
built clean; 40 agent names registered including the new
`frontend-tools-async`/`frontend_tools_async` routes).
## Not in this PR
- **gen-ui-declarative** — shipped separately in #5749.
- **catchall / headless-complete** — snapshot fix in progress.
- **multimodal / gen-ui-custom** — not addressed here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The bare-substring 'weather' fixture in aimock/d4/llamaindex/chat.json emitted a
get_weather tool call with no toolName gate. The tool-free voice agent's prompt
"What is the weather in Tokyo?" (substring "weather") leaked into this fixture,
emitting a get_weather call the voice agent could never resolve, so the voice D6
cell hung (done-signal-missing, body stuck on get_weather/Running).
Add toolName:"get_weather" so the fixture only fires when the requesting agent
actually registers get_weather (mirrors the gate in d6 tool-rendering.json). The
tool-free voice request now falls through to voice.json's exact content match.
Local red->green proof (showcase test llamaindex:voice --d6 --direct):
- RED: done-signal-missing; body "What is the weather in Tokyo? get_weather Running"
- GREEN: assistant settled "The weather in Tokyo is currently 22C with partly
cloudy skies and light easterly winds."; 1 passed (3.0s)
Direct aimock probes confirm the gate: tool-free -> content; with get_weather tool
-> tool call still fires. Regression: tool-rendering D6 still green; headless-complete
weather turn still passes (uses its own gen-ui-headless-complete.json fixture).
(cherry picked from commit cf6ff7c08153367239437d6c4fff425d546eb245)
The ADD-2 block suppresses the streamed TOOL_CALL_CHUNK for all frontend
tools, relying on the bare snapshot's ag_ui_tool_calls to deliver the call
(emitting both doubles the args and breaks scheduleTime / pie-bar
useComponent). But the open-generative-ui runtime middleware builds the
sandboxed iframe exclusively from streamed TOOL_CALL_* events and never
reads the snapshot, so open-gen-ui and open-gen-ui-advanced rendered 0
iframes.
Add a name-scoped exemption that streams the chunk only for
generateSandboxedUi, keeping snapshot-only delivery for every other
frontend tool. The exemption is intentionally narrow to preserve the
double-args fix.
Red->green (D6, --direct, real Docker page):
- open-gen-ui: RED "saw 0 iframe(s), longest srcdoc=0" -> GREEN (iframe + srcdoc)
- open-gen-ui-advanced: RED "selector cascade matched 0 elements" -> GREEN
Regression (all still green): beautiful-chat 5/5 (toggle-theme, pie-chart,
bar-chart, search-flights, schedule-meeting), agentic-chat, mcp-apps.
(cherry picked from commit 03f8d02cedbe737ec83aeefa708a9146f438a904)
The no-tools branch of ReasoningAGUIChatWorkflow.chat streams the answer via
astream_chat over OpenAIResponses, which (unlike astream_chat_with_tools on the
tools branch and the GREEN tool_rendering_reasoning_chain_agent) does not
accumulate resp.delta back onto the terminal resp.message.content. The
content-empty message was then snapshotted into MESSAGES_SNAPSHOT, clobbering
the ~284-char streamed answer and rendering an empty assistant bubble
(reasoning-display failed text-unstable: reasoning block painted, answer gone).
Accumulate the streamed text deltas in the no-tools path only (track_text =
not tools) and fold them onto resp.message before _finalize_chat snapshots it.
Strictly additive: only fills a message the stream left empty, never overwrites
content the LLM already accumulated, and is inert when tools are present so the
tools branch / reasoning-chain agent are untouched.
(cherry picked from commit 46afd0e040a669d14f91b8c136df9c94a91950d0)
Martha Schumann