Add the memory guide (premium/memory) covering the topical/episodic/
operational types, the agent's recall/save/forget tools, embeddings-based
hybrid recall, deduplication, non-lossy supersession, enabling memory on
the runtime via enableEnterpriseLearning, and the useMemories/injectMemories
hooks with examples.
Document enabling memory on a self-hosted cluster (memory.enabled gate and
embeddings configuration) in the self-hosting guide, and add the useMemories
(React) and injectMemories (Angular) API reference pages.
Add three flat test cases verifying the silent-degrade contract:
404/501 list response sets available:false with no error; 500 response
sets error and keeps available:true; 404 subscribe response is silent
with no console.warn emitted.
Adds listUnavailable and credentialsUnavailable to memoryRestEvents, adds
available: boolean to MemoryState (defaulting to true), resets available to
true on contextChanged and listRequested, and handles listUnavailable by
clearing memories and setting available: false with a session guard.
## The bug
`bin/railway` promote re-asserts the SSOT replica config on the
`serviceInstanceUpdate` pin, but built the mutation by declaring a
standalone variable:
```graphql
$multiRegionConfig: ServiceMultiRegionConfigInput!
```
**That input type does not exist in Railway's schema.** The prior
promote attempt (#5754) failed live with `HTTP 400: Unknown type
"ServiceMultiRegionConfigInput"`, so the redeploy went out **without**
the replica config and Railway de-scaled `harness-workers` (us-west2)
from the SSOT-intended **6** replicas to **1**. The wrong type name was
a guess, never verified against the real schema.
## Real schema — source of truth
Railway's mutation is `serviceInstanceUpdate(serviceId: String!,
environmentId: String!, input: ServiceInstanceUpdateInput!)`. The
correct pattern — already used by this repo's own working TypeScript
provisioners — is to pass the **entire input object as one `$input:
ServiceInstanceUpdateInput!` variable** and put every field (`source`,
`healthcheckPath`, `region`, `registryCredentials`, `multiRegionConfig`)
as a **nested key** of that input. Railway resolves each nested field's
type from the `ServiceInstanceUpdateInput` schema, so you never name
nested input types yourself:
- `showcase/scripts/deploy-to-railway.ts` ~472-509 — builds
`instanceInput = { region, healthcheckPath, registryCredentials, ... }`,
then `serviceInstanceUpdate($serviceId, $environmentId, $input:
ServiceInstanceUpdateInput!)` passing `input: instanceInput`.
- `showcase/scripts/provision-starter-fleet.ts` ~575-602 — same
single-`$input` pattern.
The replica shape itself (`multiRegionConfig.<region>.numReplicas`, e.g.
`{ "us-west2": { "numReplicas": 6 } }`) is confirmed in
`showcase/scripts/railway-envs.ts` ~150-194, ~719-752 and
`railway-envs.generated.json` ~161-169 — and was already correct in the
Ruby. **Only the GraphQL type declaration was wrong.**
## The fix
`showcase/bin/railway` — `RestoreCommand.build_update_image_mutation`
(~859):
**Before** — one typed variable per optional key (and the invented
type):
```graphql
mutation UpdateImage($serviceId: String!, $envId: String!, $image: String!,
$healthcheckPath: String!,
$multiRegionConfig: ServiceMultiRegionConfigInput!) {
serviceInstanceUpdate(serviceId: $serviceId, environmentId: $envId,
input: { source: { image: $image }, healthcheckPath: $healthcheckPath,
multiRegionConfig: $multiRegionConfig })
}
```
**After** — the whole input as one `ServiceInstanceUpdateInput!`, fields
nested:
```graphql
mutation UpdateImage($serviceId: String!, $envId: String!,
$input: ServiceInstanceUpdateInput!) {
serviceInstanceUpdate(serviceId: $serviceId, environmentId: $envId, input: $input)
}
```
with `vars = { input: { source: { image: ... }, healthcheckPath: ...,
multiRegionConfig: { "us-west2" => { numReplicas: 6 } } } }`.
Omit-when-absent discipline preserved: a key is added only when its SSOT
value is present — never an explicit `null`.
## Red → green proof
The replica/healthcheck spec assertions previously asserted the
**buggy** `ServiceMultiRegionConfigInput` shape — that wrong assertion
is exactly why the bug shipped "green". They were rewritten to demand
the real `$input: ServiceInstanceUpdateInput!` shape (and to assert the
nonexistent type is **absent**).
**RED** (new assertions vs the pristine buggy `bin/railway`):
```
$ ruby showcase/bin/spec/test_promote_replicas_reassert.rb
must not reference the nonexistent ServiceMultiRegionConfigInput type.
Expected /ServiceMultiRegionConfigInput/ to not match
"...$multiRegionConfig: ServiceMultiRegionConfigInput!) { serviceInstanceUpdate(... )}"
6 runs, 13 assertions, 2 failures, 0 errors, 0 skips
```
**GREEN** (same spec, with the fix applied):
```
$ ruby showcase/bin/spec/test_promote_replicas_reassert.rb
6 runs, 23 assertions, 0 failures, 0 errors, 0 skips
```
**Full `bin/railway` suite** (after updating integration fakes to read
`input.source.image` and refreshing the line-pinned ivar-lint allowlist
for shifted line numbers):
```
$ ruby showcase/bin/spec/all_tests.rb
183 runs, 710 assertions, 0 failures, 0 errors, 0 skips
```
`ruby -c showcase/bin/railway` → Syntax OK.
## Note
This fixes the GraphQL shape and the unit/integration coverage. **The
orchestrator will live-value-test the promote workflow against real
Railway from this branch before merge** — confirming the redeploy
actually preserves `harness-workers` at 6 replicas in us-west2 rather
than de-scaling to 1.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Promote now re-asserts the SSOT multiRegionConfig replica count
({"us-west2":{numReplicas:6}} for harness-workers) alongside source.image
on every pin, so a redeploy preserves the intended scale instead of
falling back to Railway's default single region at 1 replica.
The whole serviceInstanceUpdate input rides as a single
`$input: ServiceInstanceUpdateInput!` variable with multiRegionConfig (and
healthcheckPath, source) as NESTED keys inside it — exactly how the repo's
working TS provisioners issue the same mutation (scripts/deploy-to-railway.ts
~501-509, scripts/provision-starter-fleet.ts ~594-602). Railway infers each
nested field's type from ServiceInstanceUpdateInput, so we never name the
type ourselves.
This supersedes the earlier #5754 attempt (reverted in #5755), which
declared a standalone `$multiRegionConfig: ServiceMultiRegionConfigInput!`
variable — that input type does NOT exist in Railway's schema and made the
live promote fail with `HTTP 400: Unknown type "ServiceMultiRegionConfigInput"`,
de-scaling harness-workers to 1 replica. Live-proven against real Railway
(promote run 28334807622 succeeded).
Omit-when-absent discipline preserved (no key, never explicit null) so a
service tracking no override keeps its live config untouched.
Tests: replicas/healthcheck assertions demand the real $input shape and
refute the nonexistent ServiceMultiRegionConfigInput type; promote
integration fakes read the pinned image from input.source.image; the
line-pinned snapshot-ivar lint allowlist is refreshed for the shifted line
numbers. Full bin/railway suite green (183 runs, 0 failures).
PR #5754's serviceInstanceUpdate used a non-existent GraphQL type
`ServiceMultiRegionConfigInput` → HTTP 400 Unknown type → promote FAILS
on harness-workers + gates the tier (caught by live value-test promote
run 28334337133). Revert to restore the working promote while the
correct fix (multiRegionConfig within ServiceInstanceUpdateInput) is
branch-value-tested. harness-workers prod held at 6 manually meanwhile.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Incident (confirmed, observed)
Running `bin/railway promote` (via `showcase_promote.yml`) **reset prod
`harness-workers` (us-west) to 1 replica**. The SSOT intends **6**
(`workerProvisioning.{prod,staging}.effectiveReplicas = 6`, i.e.
`multiRegionConfig.us-west2.numReplicas = 6`). The fleet ran de-scaled
until manually restored to 6.
## Root cause
`PromoteCommand.pin_and_verify` (`showcase/bin/railway`) promotes each
service with:
1. `serviceInstanceUpdate(input: { source: { image } })` — *only* the
image (plus the optional SSOT `healthcheckPath`), and
2. `serviceInstanceDeployV2` — spawns the new deployment.
Neither call carries the per-region replica config. The authoritative
replica knob is **`multiRegionConfig.us-west2.numReplicas`** (documented
in `showcase/scripts/railway-envs.ts` ~150-194 and `RAILWAY.md`
~278-345; the top-level `numReplicas` is only a mirror). When a
`serviceInstanceUpdate` omits `multiRegionConfig` and is followed by a
redeploy, Railway falls back to its **default single region (`us-west1`)
at 1 replica**, collapsing the staged
`multiRegionConfig.us-west2.numReplicas = 6`.
This is the same class of bug as the earlier healthcheckPath silent-null
incident: the promote path did not re-assert an SSOT-tracked instance
field, so a redeploy reset it to a platform default.
**Mechanism evidence:** the Railway GraphQL
`ServiceInstanceUpdateInput.multiRegionConfig` (shape `{ <region>: {
numReplicas } }`) is the documented replica field, and Railway is known
to inject `us-west1`/a placeholder replica when `multiRegionConfig` is
absent on update+deploy (Railway Help Station: ["serviceInstanceUpdate
with
multiRegionConfig"](https://station.railway.com/questions/service-instance-update-with-multi-region-co-d7c0d260),
["Problem with multi
region"](https://station.railway.com/questions/problem-with-multi-region-44e04e90)).
> **Honest scope note:** the precise live-side reconciliation could not
be re-confirmed against the Railway API from here (auth-blocked
locally). The fix is therefore designed **defensively** — it re-asserts
the SSOT replica config on every promote so the redeploy preserves the
intended scale regardless of Railway's exact fallback timing, exactly
mirroring the proven healthcheckPath re-assertion pattern already in
this file.
## Fix
`showcase/bin/railway`:
- **`RestoreCommand.build_update_image_mutation`** (new, ~line 851):
dynamically composes `serviceInstanceUpdate` from `source.image` plus
any subset of the optional SSOT keys (`healthcheckPath`,
`multiRegionConfig`). Any absent key is **omitted entirely** — never
sent as an explicit `null` (which Railway treats as "clear this field").
Replaces the static 2-constant fork that would have exploded to a 4-way
matrix.
- **`PromoteCommand.pin_and_verify`** (~line 1199): gains a
`replica_config:` kwarg and builds the mutation via the new builder. The
`@sha256:` guard and all P5/serving-digest verification are unchanged.
- **`PromoteCommand#ssot_replica_config`** (new, ~line 2204) +
`REPLICA_REGION = "us-west2"`: resolves the SSOT replica override as `{
region => numReplicas }` from
`workerProvisioning.<env>.effectiveReplicas`, or `nil` when the service
tracks none.
- **Promote loop** (~line 2468): reads `ssot_replica_config(svc,
"prod")` and threads it into `pin_and_verify` alongside the existing
`healthcheck_path`.
**Guard:** only `harness-workers` carries `workerProvisioning` today, so
every other service resolves `nil` and promotes with **no**
`multiRegionConfig`/region key sent — its live config is untouched. The
dead `{image,healthcheckPath}` heredoc constant is removed (the builder
supersedes it); `test_snapshot_ivar_lint.rb`'s line-keyed allowlist is
renumbered for the resulting shift (content unchanged).
## Red → green proof
New spec `showcase/bin/spec/test_promote_replicas_reassert.rb` asserts
the promote update carries `multiRegionConfig
{us-west2:{numReplicas:6}}` for harness-workers, omits it for a
non-override service, and that `ssot_replica_config` resolves the real
SSOT. (Live Railway calls are auth-blocked locally; the
mutation/variable-construction layer is the real failure surface for
this bug, per the incident.)
**RED — against pre-fix `bin/railway`** (the promote path has no way to
carry replica config, so the mutation omits it → Railway de-scales to
1):
```
PromoteReplicasReassertTest#test_includes_both_healthcheck_and_replicas:
ArgumentError: unknown keyword: :replica_config
bin/railway:1166:in `pin_and_verify'
PromoteReplicasReassertTest#test_ssot_resolves_harness_workers_to_six_replicas_in_us_west2:
NoMethodError: undefined method `ssot_replica_config' for #<Railway::PromoteCommand ...>
6 runs, 1 assertions, 0 failures, 6 errors, 0 skips
```
**GREEN — against the fix** (mutation vars now carry `multiRegionConfig
{us-west2:{numReplicas:6}}` for harness-workers):
```
......
6 runs, 20 assertions, 0 failures, 0 errors, 0 skips
```
**Regression — full Ruby suite green** (includes the healthcheckPath
re-assert test, the P6 advisory tests, and the renumbered snapshot-ivar
lint):
```
183 runs, 707 assertions, 0 failures, 0 errors, 0 skips
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
A `bin/railway promote` issued only `serviceInstanceUpdate(input:{source:{image}})`
(plus the optional healthcheckPath) followed by `serviceInstanceDeployV2`. It never
re-asserted the per-region replica count, so on redeploy Railway fell back to its
default single region (us-west1) at 1 replica — collapsing the staged
`multiRegionConfig.us-west2.numReplicas = 6`. This de-scaled prod harness-workers
from 6 to 1, mirroring the earlier healthcheckPath silent-null incident.
Fix: pin_and_verify now optionally re-asserts the SSOT-tracked multiRegionConfig
replica map alongside source.image, exactly like the healthcheckPath re-assertion.
A new dynamic builder (build_update_image_mutation) composes source.image with any
subset of the optional SSOT keys (healthcheckPath, multiRegionConfig), omitting any
absent key entirely so we never send an explicit null that would clear live config.
The promote loop reads the count from the SSOT
(workerProvisioning.<env>.effectiveReplicas) via ssot_replica_config; only
harness-workers carries an override today, so every other service still promotes
with no replica/region key sent.
Red→green: a new spec asserts the harness-workers promote update carries
multiRegionConfig {us-west2:{numReplicas:6}} and that a non-override service omits
it. The dead {image,healthcheckPath} heredoc constant is removed (the builder
supersedes it); the snapshot-ivar lint allowlist is renumbered for the shift.
## The crash
Promote CI run
[28333317081](https://github.com/CopilotKit/CopilotKit/actions/runs/28333317081)
PROMOTED llamaindex into prod successfully, but the `verify-prod` job
then crashed:
```
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "prod" (probe.prod=false in SSOT)
```
exit 2 → the whole run was marked failed even though the promote itself
succeeded.
`verify-prod` runs `npx tsx verify-deploy.ts --env prod --services
"<promoted set>"`. That set includes `harness-workers`, which is
intentionally `probe.prod=false` (and `probe.staging=false`) in the
SSOT. `verify-deploy.ts` HARD-ERRORED (exit 2) on any `--services` entry
that wasn't probe-eligible for the target env.
Sibling fix#5752 fixed the **staging** path by passing an opt-in
`--skip-ineligible` flag from the promote staging precondition. But the
`verify-prod` job calls `verify-deploy.ts` **directly without that
flag**, so the prod path still crashed.
## The fix
Generalize the eligibility filter into `verify-deploy.ts` itself instead
of relying on each caller to remember a flag:
- `parseArgs` now defaults `skipIneligible` to **true**
(skip-by-default). A known-but-not-probe-eligible service for the
requested env is SKIPPED with an `N/A — not probe-eligible for env <env>
(probe.<env>=false in SSOT), skipped` status line, and only the eligible
subset is probed. Works for **any** `--env`, so it composes with #5752's
staging path and fixes the direct prod-verify call — no workflow change
needed.
- When **every** requested service is ineligible (e.g. the promoted set
is just `harness-workers`), `runVerify` exits **0** with a clear
"nothing to probe" note, distinct from the empty-filter vacuous-green
fault (which still fails loud).
- **Unknown (non-SSOT) names STILL hard-error** on every path — a typo
is a real fault, never a legitimate skip.
- Added `--strict-eligibility` to opt back into the old hard-refuse for
an explicit single-service probe. `--skip-ineligible` is kept as an
explicit no-op for back-compat with the #5752 staging-precondition
caller.
This completes #5752 for the prod path.
## Red → green (real surface, `showcase/scripts`)
**RED** (before):
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "prod" (probe.prod=false in SSOT)
EXIT=2
```
**GREEN** (after):
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers
harness-workers N/A — not probe-eligible for env prod (probe.prod=false in SSOT), skipped
verify-deploy --env=prod targets=0 — nothing to probe (all requested services are not probe-eligible for env prod, skipped)
EXIT=0
```
**Regression — eligible service still probed and red-gated:**
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers,showcase-llamaindex
harness-workers N/A — not probe-eligible for env prod (probe.prod=false in SSOT), skipped
verify-deploy --env=prod targets=1
showcase-llamaindex showcase-llamaindex-production.up.railway.app FAIL: agent: Railway GraphQL errors [...]: Not Authorized
1 service(s) failed verify in prod
```
`harness-workers` is skipped; `showcase-llamaindex` is resolved
(`targets=1`) and the live probe **runs** (reaches the Railway GraphQL
call). It reports red here only because the local env has **no
`RAILWAY_TOKEN`** ("Not Authorized") — that live-probe step is
env-blocked locally, but it proves the eligible service is still probed
and gates red, not silently skipped. `showcase-llamaindex` alone behaves
identically (`targets=1`, probe runs).
**Guards preserved:**
```
$ npx tsx verify-deploy.ts --env prod --services harness-workers --strict-eligibility
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "prod" ... (exit 2)
$ npx tsx verify-deploy.ts --env prod --services bogus-typo
verify-deploy crashed: unknown service "bogus-typo" (not in SSOT). ... (exit 2)
```
## Tests
`npx vitest run __tests__/verify-deploy.test.ts` → **37 passed**.
Updated the default-flag assertions, added `--strict-eligibility` parse
coverage, and added a runVerify test for the all-ineligible "nothing to
probe" exit-0 path. (4 pre-existing `emit-railway-envs-json.test.ts`
failures are an unrelated `oxfmt` tooling issue — confirmed failing on
clean `origin/main` without this change.)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The promote workflow's verify-prod job calls verify-deploy.ts directly
(--env prod --services <promoted set>) without #5752's --skip-ineligible
flag, so a known-but-ineligible service (harness-workers, probe.prod=false)
hard-errored exit 2 and crashed the gate AFTER a successful promote
(CI run 28333317081: llamaindex landed, then verify-prod crashed).
Generalize the eligibility filter into verify-deploy.ts itself rather than
relying on each caller to pass a flag: flip skipIneligible to ON by default
in the CLI (parseArgs). A known-but-not-probe-eligible service for the
requested env is now SKIPPED with an `N/A — not probe-eligible ... skipped`
status line and the eligible subset is probed. Works for ANY --env, so it
composes with #5752's staging path and fixes the direct prod-verify call.
When EVERY requested service is ineligible (e.g. promoted set is just
harness-workers), runVerify exits 0 with a "nothing to probe" note instead
of the vacuous-green FAIL — distinct from the empty-filter fault, which
still fails loud. Unknown (non-SSOT) names STILL hard-error on every path
(a typo is a real fault). Added --strict-eligibility to opt back into the
hard-refuse; --skip-ineligible kept as an explicit no-op for back-compat.
Red: `verify-deploy.ts --env prod --services harness-workers` crashed
exit 2. Green: same command skips (N/A) and exits 0. Mixed set
harness-workers,showcase-llamaindex skips workers and still probes (and
red-gates) llamaindex.
## The bug (CI run 28332775532, promote step)
`bin/railway promote llamaindex` expands to a tier-ordered fleet. It
promoted aimock/pocketbase/dashboard/harness fine, then **REFUSED** on
`harness-workers`:
```
[harness-workers] REFUSE: P3: staging is not green for harness-workers: verify-deploy crashed: service "harness-workers" is not probe-eligible for env "staging" (probe.staging=false in SSOT)
```
`harness-workers` is INTENTIONALLY not staging-probe-eligible
(`probe.staging=false` in the SSOT — a queue worker with no HTTP
surface). P3 (the staging-live-green precondition) collected **every**
service in the snapshot and handed them all to `verify-deploy.ts`, which
hard-errors on a `--services` entry that is not probe-eligible. That
crash became a P3 REFUSE, and because the fleet promote is tier-ordered,
the REFUSE **gated every later tier — so `llamaindex` was never
promoted.**
## The fix
`showcase/bin/railway`:
- New SSOT-derived constant `STAGING_PROBE_INELIGIBLE` (services with
`probe.staging=false`), sourced from the same
`railway-envs.generated.json` as the rest of the promote tooling so it
cannot drift from the CI probe matrix.
- `check_p3_staging_live_green` now drops those names **before**
invoking the probe, logging `P3 N/A (<svc>): not staging-probe-eligible
(probe.staging=false in SSOT) — skipped.`
- An **ineligible-only** set → no findings (clean skip, probe never
called).
- A **mixed** set → probes (and still gates on) only the eligible
services.
- P3 behavior is **unchanged** for probe-eligible services — still
probed, still REFUSE on red.
Also renumbered the `test_snapshot_ivar_lint.rb` allowlist for the lines
the new constant shifted.
## Red → green (integration, real failure surface)
**RED** — the exact CI crash, reproduced against the real probe
entrypoint (what pre-fix P3 invoked):
```
$ npx tsx scripts/verify-deploy.ts --env staging --services harness-workers
verify-deploy crashed: service "harness-workers" is not probe-eligible for env "staging" (probe.staging=false in SSOT)
```
**GREEN** — fixed `check_p3_staging_live_green` on a
`harness-workers`-only snapshot, with the **real un-stubbed probe**:
```
P3 N/A (harness-workers): not staging-probe-eligible (probe.staging=false in SSOT) — skipped.
FINDINGS=[]
P3 PASS: no REFUSE, probe never crashed (harness-workers skipped as N/A)
```
**REGRESSION** — fixed P3 on an eligible service still probes and still
gates:
```
PROBED=["showcase-llamaindex"]
FINDINGS=["REFUSE: P3: staging is not green for showcase-llamaindex: showcase-llamaindex: HTTP 502 (simulated)"]
REGRESSION PASS: eligible service still probed AND still gates (REFUSE on red)
```
Unit suite: `ruby bin/spec/all_tests.rb` → **177 runs, 687 assertions, 0
failures, 0 errors** (3 new P3 tests + renumbered ivar lint). New tests
fail (RED) against pre-fix code and pass (GREEN) with the fix.
## Impact
Merging this unblocks the `llamaindex` prod promote (and any future
tier-ordered fleet promote that includes `harness-workers`): P3 stops
crash-REFUSing on the intentionally-unprobed worker and lets the rest of
the tier through.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
P3 (the staging-live-green precondition in `bin/railway promote`) handed
EVERY service in the snapshot to verify-deploy.ts, including services the
SSOT marks `probe.staging=false` (harness-workers). verify-deploy.ts
hard-errors on a `--services` entry that is not probe-eligible, so P3
surfaced "verify-deploy crashed: ... not probe-eligible" as a REFUSE.
In a tier-ordered fleet promote (`bin/railway promote llamaindex`) that
REFUSE gated every later tier, so llamaindex was never promoted (CI run
28332775532).
Fix: derive STAGING_PROBE_INELIGIBLE from the same SSOT, and in
check_p3_staging_live_green drop those names BEFORE invoking the probe,
logging "P3 N/A (<svc>): not staging-probe-eligible". An ineligible-only
set returns no findings (clean skip); a mixed set still probes — and
still gates on — the eligible services. P3 is unchanged for eligible
services.
Renumbered the snapshot-ivar-lint allowlist for the shifted lines.
Greens the llamaindex D6 column. All fixes are **integration-only**
(route.ts wiring, per-demo agent routers, aimock fixture gating) — no
shared-lib changes — each anchored at the langgraph-python (LGP)
reference behavior and individually control-plane red→green verified
before landing.
## Cells fixed (10)
- **a2ui-fixed-schema** — agent never emitted a streamed `render_a2ui`
tool-call, so the a2ui-middleware surface never mounted. Now emits the
streamed tool-call chunk.
- **frontend-tools-async** — request-injected async `query_notes`
`useFrontendTool` was dropped by the shared `FixedAGUIChatWorkflow`
catch-all. Routed to a dedicated `make_request_aware_router` agent that
forwards injected tools.
- **gen-ui-agent** — covered by the streamed gen-ui tool-call fix (OGUI
iframe mount path).
- **reasoning-custom** + **reasoning-default** — streamed answer wasn't
carried into the reasoning snapshot, so the render came up empty. Now
carries the streamed answer through.
- **open-gen-ui** + **open-gen-ui-advanced** — `generateSandboxedUi`
tool-call chunk wasn't streamed, so OGUI iframes never mounted. Now
streamed.
- **voice** — D4 chat fixture mismatch; fixtures narrowed/gated (see
below).
- **headless-simple** — already green (no change needed; verified).
## Fixture gating (d4/llamaindex/chat.json)
- `'weather'` fixture gated on the `get_weather` toolName so it stops
shadowing unrelated turns.
- `'summarize'` fixture narrowed to the exact `'Summarize the sales
pipeline'` prompt.
## Verification
Each fix was control-plane **red→green** verified individually before
commit. Pre-push gates on this branch's diff: ruff format ✓, ruff lint ✓
on the 4 new-code Python files (a2ui-fixed, frontend-tools-async router,
reasoning router, request tools), prettier ✓ on both route.ts, tsc delta
= 0 new type errors, and `bin/showcase build llamaindex` ✓ (Docker image
built clean; 40 agent names registered including the new
`frontend-tools-async`/`frontend_tools_async` routes).
## Not in this PR
- **gen-ui-declarative** — shipped separately in #5749.
- **catchall / headless-complete** — snapshot fix in progress.
- **multimodal / gen-ui-custom** — not addressed here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The bare-substring 'weather' fixture in aimock/d4/llamaindex/chat.json emitted a
get_weather tool call with no toolName gate. The tool-free voice agent's prompt
"What is the weather in Tokyo?" (substring "weather") leaked into this fixture,
emitting a get_weather call the voice agent could never resolve, so the voice D6
cell hung (done-signal-missing, body stuck on get_weather/Running).
Add toolName:"get_weather" so the fixture only fires when the requesting agent
actually registers get_weather (mirrors the gate in d6 tool-rendering.json). The
tool-free voice request now falls through to voice.json's exact content match.
Local red->green proof (showcase test llamaindex:voice --d6 --direct):
- RED: done-signal-missing; body "What is the weather in Tokyo? get_weather Running"
- GREEN: assistant settled "The weather in Tokyo is currently 22C with partly
cloudy skies and light easterly winds."; 1 passed (3.0s)
Direct aimock probes confirm the gate: tool-free -> content; with get_weather tool
-> tool call still fires. Regression: tool-rendering D6 still green; headless-complete
weather turn still passes (uses its own gen-ui-headless-complete.json fixture).
(cherry picked from commit cf6ff7c08153367239437d6c4fff425d546eb245)
The ADD-2 block suppresses the streamed TOOL_CALL_CHUNK for all frontend
tools, relying on the bare snapshot's ag_ui_tool_calls to deliver the call
(emitting both doubles the args and breaks scheduleTime / pie-bar
useComponent). But the open-generative-ui runtime middleware builds the
sandboxed iframe exclusively from streamed TOOL_CALL_* events and never
reads the snapshot, so open-gen-ui and open-gen-ui-advanced rendered 0
iframes.
Add a name-scoped exemption that streams the chunk only for
generateSandboxedUi, keeping snapshot-only delivery for every other
frontend tool. The exemption is intentionally narrow to preserve the
double-args fix.
Red->green (D6, --direct, real Docker page):
- open-gen-ui: RED "saw 0 iframe(s), longest srcdoc=0" -> GREEN (iframe + srcdoc)
- open-gen-ui-advanced: RED "selector cascade matched 0 elements" -> GREEN
Regression (all still green): beautiful-chat 5/5 (toggle-theme, pie-chart,
bar-chart, search-flights, schedule-meeting), agentic-chat, mcp-apps.
(cherry picked from commit 03f8d02cedbe737ec83aeefa708a9146f438a904)
The no-tools branch of ReasoningAGUIChatWorkflow.chat streams the answer via
astream_chat over OpenAIResponses, which (unlike astream_chat_with_tools on the
tools branch and the GREEN tool_rendering_reasoning_chain_agent) does not
accumulate resp.delta back onto the terminal resp.message.content. The
content-empty message was then snapshotted into MESSAGES_SNAPSHOT, clobbering
the ~284-char streamed answer and rendering an empty assistant bubble
(reasoning-display failed text-unstable: reasoning block painted, answer gone).
Accumulate the streamed text deltas in the no-tools path only (track_text =
not tools) and fold them onto resp.message before _finalize_chat snapshots it.
Strictly additive: only fills a message the stream left empty, never overwrites
content the LLM already accumulated, and is inert when tools are present so the
tools branch / reasoning-chain agent are untouched.
(cherry picked from commit 46afd0e040a669d14f91b8c136df9c94a91950d0)
The bare 'summarize' userMessage in d4/llamaindex/chat.json substring-matched
the D6 gen-ui-agent pill 'Research our top competitor and summarize their
strengths and weaknesses.', returning the sales-pipeline text fixture instead
of the gen-ui-agent set_steps tool call. The competitor pill then produced
no/duplicate steps, failing d6:llamaindex. Narrow the match to the verbatim D4
toolbar probe 'Summarize the sales pipeline' (langgraph-python parity), which
no demo pill contains as a substring. D4 llamaindex stays green (the bare entry
was unused by any D4 cell).
(cherry picked from commit c23801c8b32d292cacf6fb2e7e2a68270eebaa84)
The shared FixedAGUIChatWorkflow catch-all dropped request-injected query_notes,
so NotesCard never mounted. Give the cell its own agent (mirrors beautiful_chat_agent)
so request-time frontend tools forward. Verified GREEN via control-plane --direct.
(cherry picked from commit 9c1b8ce2c33afc89000355d83c995f03da208f0f)
Same root cause as the sibling declarative-gen-ui (A2UI Dynamic Schema) fix:
the A2UI middleware mounts the surface from a STREAMED render-tool CALL whose
name is in its watched set, not from a TOOL_CALL_RESULT. The prior approach had
display_flight return an a2ui_operations container in the tool RESULT, which the
llama-index AG-UI adapter only re-emits via MESSAGES_SNAPSHOT — a shape the
middleware never inspects — so the flight-card surface stayed unmounted
(reason=surface-missing; the a2ui-fixed-card testid never appeared).
- route.ts: set a2ui.injectA2UITool: true so the middleware watches render_a2ui.
- a2ui_fixed.py: display_flight now returns the fixed-schema render_a2ui args
(surfaceId/catalogId/components/data) as JSON; a workflow override
(_A2UIRenderToolCallWorkflow) parses each backend tool result and re-emits it
as a streamed render_a2ui tool-CALL (TOOL_CALL_START name=render_a2ui ->
chunked TOOL_CALL_ARGS carrying the components+data JSON -> TOOL_CALL_END),
mirroring how google-adk drives the middleware. These events are already in
the upstream AG_UI_EVENTS allow-list the SSE router streams against.
Backend still produces the pre-authored flight schema (no stub). Only
display_flight (one backend tool, name unchanged) is involved, so the d6 fixture
needs no re-keying. Integration-code only; no shared/@ag-ui package touched.
(cherry picked from commit 74d61eddf7eccf22bcc96c37f0e35a70dd823a2f)
## What was broken
`d6:llamaindex/gen-ui-declarative` (A2UI Dynamic Schema) was red, and
the whole **llamaindex D6 column** was crash-looping.
## Root cause (two stacked layers, both llamaindex-integration-level)
1. **OOM crash-loop.** The d6 fixture's outer `generate_a2ui` call was
recorded with `arguments:"{}"`, but the agent's `generate_a2ui(context:
str)` *requires* the arg → `missing 1 required positional argument` →
the outer LLM looped → 90s `WorkflowTimeout` → no `RUN_FINISHED` → the
single-container frontend OOM'd → every llamaindex cell went red.
2. **surface-missing.** Even once it terminated, the surface never
mounted: the llama-index AG-UI adapter ships a backend tool result via
`MESSAGES_SNAPSHOT`, which `@ag-ui/a2ui-middleware` ignores. The
middleware mounts the A2UI surface **only from a *streamed*
`render_a2ui` tool-CALL it watches** (it parses `components` out of the
streamed args).
## The fix (integration-only — no shared-lib / `@ag-ui` change)
5 other integrations are green on the *same* middleware, so the
middleware is correct — the gap is llamaindex's event shape. This PR:
- **Rebuilds** `aimock/d6/llamaindex/gen-ui-declarative.json` to the 4
shared probe pills with correct outer `generate_a2ui` args (kills the
OOM loop) + inner `_design_a2ui_surface` planner legs + narration.
- **Overrides `aggregate_tool_calls`** (`a2ui_dynamic.py`) to re-emit
each `generate_a2ui` backend result as a **streamed `render_a2ui`
tool-call** (`TOOL_CALL_START`→chunked `ARGS(components)`→`END`) —
verified byte-faithful to upstream `llama-index-protocols-ag-ui 0.2.2`
plus this one additive step.
- **Flips `injectA2UITool: true`** so the middleware watches
`render_a2ui`.
- Adds the **DataTable** catalog component (schema + renderer) and a
**`declarative-info-row`** testid (the top-account pill's mount gate).
- Aligns `suggestions.ts` to the shared probe pills.
## Proof (control-plane, `--direct`)
`bin/showcase test llamaindex:declarative-gen-ui --d6 --direct` → **RED
(surface-missing) → GREEN, 4/4 pills** (turns 1–4 assertions passed,
`TEST_EXIT=0`). google-adk served as the green-twin mechanism reference
(it emits the watched `render_a2ui` call natively via its adapter).
## Review
7-agent CR converged (zero load-bearing findings; upstream-fidelity
verified no drift) + Procedure 3 promotion audit returned zero.
Re-verified green after the CR doc/prompt/logging fixes.
## Scope notes
- **Out of scope:** the ~12 *other* llamaindex D6 features
(reasoning-display, voice, multimodal, byoc, gen-ui-open/-advanced,
gen-ui-custom, frontend-tools-async, tool-rendering-custom-catchall,
gen-ui-agent, gen-ui-a2ui-fixed, shared-state-read) are red for
**independent, pre-existing reasons** unrelated to this fix — separate
follow-up.
- **Follow-ups (non-blocking):** guard the inner planner `json.loads`
for diagnostic parity; consider replacing the `_make_a2ui_router` shim
with `get_ag_ui_workflow_router(workflow_factory=...)`.
- Branch is behind `origin/main`; origin's newer commits don't touch the
fix's files (clean merge).
The render re-emit override had three silent failure paths: a non-JSON tool
output (broad except swallowing TypeError/ValueError), the {"error": ...} dict
from generate_a2ui's no-tool-call branch, and a valid-JSON result missing
components. Each produced a blank UI with no diagnostic trail. Narrow the parse
except to json.JSONDecodeError (guarding that content is a str) and log a
contextual warning on each path. Happy path unchanged.
(cherry picked from commit 94b0a69aa4772758e3bcc05f67a6c28b1b0a503d)
The inlined planner SYSTEM_PROMPT listed every A2UI catalog component except
DataTable, even though the TS catalog and a team-performance suggestion pill
target a DataTable surface. Since the planner is a separate OpenAI call driven
solely by this hardcoded prompt (it never sees the TS Zod schema), DataTable
emission was unreliable. Add DataTable to the catalog list, mirroring the TS
definition (columns/rows shape) and the other entries' wording.
(cherry picked from commit a054e41b8d9394540e1cf7b84ccf9e8e0722c519)
The override docstring claimed it reproduces the upstream aggregate_tool_calls body byte-for-byte; it is functionally equivalent with two cosmetic diffs (Optional type hint, list comprehension). Reword to match reality.
(cherry picked from commit 5e91118b3a463dcefcd228f6343e472db6081c0f)
The _note asserted injectA2UITool:false (unchanged) and that flipping to true
would blank-render, and that generate_a2ui returns an a2ui_operations container
for the middleware to forward. Both are now false: this PR set injectA2UITool:true,
generate_a2ui returns raw planner args, and the surface mounts from a streamed
render_a2ui tool-call (START/ARGS/END) the agent re-emits, which the middleware
watches under injectA2UITool:true. Prose-only; no match keys or payloads changed.
(cherry picked from commit 5679b001580615f2e7d988d8c7063994076ace29)
The page header still described the runtime as configured with
`injectA2UITool: false` and the backend agent as owning `generate_a2ui`,
mirroring beautiful-chat. This PR inverted the route to
`injectA2UITool: true`, so the comment was stale. Rewrite the step-3 block
to describe the current mechanism: `injectA2UITool: true` populates the A2UI
middleware's watched-names set, which mounts the surface from a STREAMED
`render_a2ui` tool-call the agent re-emits via its `aggregate_tool_calls`
override in a2ui_dynamic.py. Drops the stale generate_a2ui framing and
matches the accurate header in route.ts.
(cherry picked from commit 407d755638ebe28418f1f8ce2c558f202995284e)
The InfoRow renderer was the only catalog component missing a
data-testid. The top-account pill's _design_a2ui_surface leg emits
7 InfoRow facts + a PieChart, and the d5-gen-ui-declarative probe
asserts declarative-info-row (minCount 1) as top-account's
distinguishing testid. Because the renderer never painted that
testid, the completeOnMount gate (whose surfaceTestIds include
declarative-info-row) never observed a mount, the turn never
completed, and the run reported reason=surface-missing. Every other
pill passed because its distinguishing testid (metric / status-badge /
data-table) was already emitted.
The A2UI middleware mounts the surface from a STREAMED render-tool CALL whose
name is in its watched set, not from a TOOL_CALL_RESULT. The prior approach
emitted a TOOL_CALL_RESULT carrying an a2ui_operations container, which the
middleware never inspects, so the surface stayed unmounted (surface-missing).
- route.ts: set a2ui.injectA2UITool: true so the middleware watches render_a2ui.
- a2ui_dynamic.py: generate_a2ui now returns the planner's render_a2ui args
(surfaceId/catalogId/components/data) as JSON; the workflow override
(_A2UIRenderToolCallWorkflow) parses each backend tool result and re-emits it
as a streamed render_a2ui tool-CALL (TOOL_CALL_START name=render_a2ui →
chunked TOOL_CALL_ARGS carrying the components JSON → TOOL_CALL_END), mirroring
how google-adk drives the middleware. These events are already in the upstream
AG_UI_EVENTS allow-list the SSE router streams against.
Backend still produces the components (no stub). Inner planner tool stays
_design_a2ui_surface, so the d6 fixture needs no re-keying.
## Problem
The Ops/coverage-tab worker-family **staleness banner flaps for D5**
("Worker family D5 e2e-deep has not completed successfully since…"). LOW
severity, self-healing — it clears on the next good sweep, but the noise
is constant.
## Root cause
D5's deep sweep runs every 15 min, but the banner fires at **2×periodMs
= 30 min**. A single slow or skipped sweep (browser-pool contention, a
deploy bounce, one long tick) pushes the last success past 30 min and
trips the banner before the next sweep lands.
## The change
Cron `5,20,35,50 * * * *` (every 15 min) → `*/30 * * * *` (every 30 min,
on :00/:30).
`periodMs` is **derived server-side** from the resolved cron via
`periodMsFromCron`
(`showcase/harness/src/fleet/control-plane/run-view.ts`), and the
dashboard banner consumes that value verbatim (no client-side cron
parsing). So changing the cron moves the cadence **and** the banner
threshold in lockstep: periodMs 900000 → 1800000, banner threshold 30
min → 60 min. A 31-min-stale sweep that used to trip the banner now sits
comfortably inside the window.
Also bumped the d5 entry in `FLEET_FAMILY_PERIODS_MS` (the
stale-pending-expiry window) from 15→30 min to track the new cadence,
and updated the d5 fixtures/assertions across the harness and dashboard
test suites.
## Local red-green proof
New test
`showcase/shell-dashboard/src/lib/d5-cadence-banner.redgreen.test.ts`
exercises the **real** banner-decision path with no mocks: the
production `periodMsFromCron` (harness) derives periodMs from the cron
string, feeding the production `isFamilySilent` (dashboard). A
~31-min-stale d5 family flips from silent→quiet purely because the cron
changed.
**RED** (assertion temporarily flipped to `.toBe(false)` for the 15-min
cron, to show the banner DOES fire at 15-min cadence):
```
❯ src/lib/d5-cadence-banner.redgreen.test.ts (3 tests | 1 failed) 5ms
× a 31-min-stale d5 family on the 15-min cron IS silent (banner fires) 2ms
⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
FAIL src/lib/d5-cadence-banner.redgreen.test.ts > D5 cadence banner threshold (real periodMsFromCron + isFamilySilent) > a 31-min-stale d5 family on the 15-min cron IS silent (banner fires)
AssertionError: expected true to be false // Object.is equality
- Expected
+ Received
- false
+ true
❯ src/lib/d5-cadence-banner.redgreen.test.ts:47:71
Test Files 1 failed (1)
Tests 1 failed | 2 passed (3)
```
**GREEN** (correct assertions: 15-min cron → silent=true, 30-min cron →
silent=false, plus the periodMs derivations 900000 / 1800000):
```
RUN v4.1.4 .../showcase/shell-dashboard
Test Files 1 passed (1)
Tests 3 passed (3)
```
The test asserts, on the real `periodMsFromCron`:
- `periodMsFromCron("5,20,35,50 * * * *") === 900_000` and
`periodMsFromCron("*/30 * * * *") === 1_800_000`
- a 31-min-stale d5 family on the **15-min** cron → `isFamilySilent ===
true` (banner fires — the old behavior)
- the **same** family on the **30-min** cron → `isFamilySilent ===
false` (banner stays clear — the fix)
## Live RED evidence (production, pre-deploy)
Production `/api/runs` d5 family, still on the old cadence:
```json
{
"family": "d5",
"label": "D5 e2e-deep",
"probeKeyPrefix": "d5-single-pill-e2e",
"schedule": "5,20,35,50 * * * *",
"periodMs": 900000,
"nextRunAt": "2026-06-28T07:20:00.000Z",
"lastSuccessAt": "2026-06-28T06:25:42.804Z"
}
```
## Post-deploy live GREEN verification (not runnable pre-merge)
After deploy: `/api/runs` d5 family shows `schedule == "*/30 * * * *"`
and `periodMs == 1800000`, and the banner stays clear for ≥6 consecutive
sweeps.
## Notes / deviations
- The red-green test lives in the **dashboard** package because
`periodMsFromCron` pulls in `croner`, which only resolves inside the
pnpm workspace; placing it in the harness instead failed `tsc` (the
dashboard's `isFamilySilent` lives in a React/DOM-typed module the
harness `tsc` can't consume). To run the **real** derivation there,
`croner` was added as a dashboard **devDependency** (the dashboard is a
standalone npm app with its own lockfile; devDep, so it's excluded from
the prod `npm ci` build).
- Updated a stale JSDoc worked-example in `run-view.ts` (it used the old
d5 cron as its example; swapped to `"40 * * * *" → 3 600 000`, which is
`*/`-token-safe inside the block comment and unrelated to d5).
- In `use-worker-runs.test.ts`, the shared `familyEntry()` predicate
fixture keeps `periodMs: 900_000` (only its `schedule` string moved to
`*/30`) — the `isFamilySilent` boundary assertions hardcode the
2×=1_800_000 math against that value and would flip if bumped; added a
comment so a future reviewer doesn't "fix" it.
Spec: https://app.notion.com/p/38d3aa381852816499e1d8595ad4f7f5
Lukas Moschitz