The /proc PPID walk forked an awk process for every entry in the process
table on every scan pass. Replace the `echo "${stat##*) }" | awk '{print $2}'`
pipeline with the `read` builtin, which word-splits the post-comm remainder
("STATE PPID PGRP …") on IFS and captures the 2nd field with no subprocess.
Byte-identical across the langgraph-typescript and strands-typescript
entrypoints. Non-behavioral; bash -n + shellcheck --severity=warning clean.
## What & why
Two showcase agent containers (**langgraph-typescript**,
**strands-typescript**) could enter a *running-but-dead* state: Railway
showed the service `● Online` while `/api/health` returned **HTTP 502**.
This took down all 36 LGT dashboard cells on staging **and** prod
(prod's `.langgraph_api` had crossed the 200 MB size-watchdog
threshold).
**Root cause:** the agent is launched via process substitution (`... &>
>(awk …) &`), so `$AGENT_PID` (`=$!`) is the **wrapper subshell**, not
the real `npm`→`node` server that holds the port. Every watchdog/cleanup
did a bare `kill -9 $AGENT_PID`, which reaped only the subshell and
**orphaned the real server** (reparented to PID 1, still bound to the
port). The watchdog's "kill agent → container restart → boot-purge"
contract therefore never fired: the frontend kept proxying to a dead
agent → 502 forever.
## Fixes (each with local red-green on the real entrypoint in
`node:22-slim`)
1. **cleanup() EXIT trap** → routes through `_kill_agent_tree` (was
orphaning the agent on every SIGTERM/redeploy).
2. **`_kill_agent_tree`** → `/proc`-based tree-kill with a bounded
re-scan (root killed last) so mid-walk forks can't escape; refuses PID ≤
1 (fail-closed).
3. **size-watchdog** hardened against non-numeric `du` and transient
errors (no silent gate-disable, no permanent loop death).
4. **strands health-watchdog** → 180 s startup-grace window (parity with
langgraph); the now-effective kill would otherwise loop a slow cold
start.
5. **`wait -n` under `set -e`** → capture exit code so the restart
diagnostic isn't dead code on the primary (137) path.
6. **structural:** one `_require_int` validator over *every*
operator-overridable numeric knob (fail-safe to default), and **every**
wrapped-PID kill (incl. `NEXTJS_PID`) routed through the guarded
tree-kill; dangerous `${AGENT_PID:-0}` sentinel removed.
7. **`_require_int`** requires a positive integer (rejects `0` and
leading-zero/octal).
## Incident status
Staging **and** prod LGT were restored immediately via redeploy
(boot-purge cleared the oversized state) — both `/api/health` → 200.
This PR stops the recurrence.
## Review
Converged through a 5-round unbiased review-fix loop (1 + 4
confirmation), zero mandatory findings at close, all load-bearing guards
independently re-verified. `bash -n` + shellcheck (`-S warning`) clean;
170/170 shell bats pass.
## Follow-up (tracked, separate PR — non-load-bearing)
`_require_int` upper-bound clamp (LOW arith-overflow, needs a 20+-digit
value); a stale `cleanup()` comment; size-guard unarmed during the
startup-grace window; SIZE_PID trap-registration micro-window;
diagnostic label on near-simultaneous exit; cosmetic log nits; startup
readiness `sleep 3`+`kill -0` probes the wrapper subshell; no dedicated
Next.js frontend watchdog.
The _require_int validator in the langgraph-typescript and strands-typescript
entrypoints accepted '0' and leading-zero/octal forms like '010'/'08'. Operator
typos on any numeric knob then broke a guard:
- SIZE_THRESHOLD_MB=0 kills the agent on cycle 1 (instant restart loop)
- HEALTH_STRIKE_LIMIT=0 kills on first probe miss
- SIZE_CHECK_INTERVAL=0 / HEALTH_CHECK_INTERVAL=0 busy-spin on 'while sleep 0'
- '010' is read as OCTAL (8) in arithmetic; '08'/'09' abort under set -e
Tighten the predicate to accept only a positive integer with no leading zero
([1-9][0-9]*). Invalid values keep the existing fail-safe behavior: WARN and
fall back to the documented default. Helper stays byte-identical across both
files.
CLASS 1 (guard silently disabled by a bad numeric override): add a reusable
_require_int validator and run it at startup over EVERY operator-overridable
numeric knob in both entrypoints (size threshold/interval, startup grace,
health-probe interval, strike limit). A non-integer/empty override now WARNs
and falls back to the documented default instead of breaking a sleep/loop/
arithmetic test. Closes instance #3 (LANGGRAPH_SIZE_CHECK_INTERVAL='60s'
killing the size-monitor loop on its first iteration).
CLASS 2 (wrapped-PID orphan + kill-0 footgun): route the cleanup() NEXTJS_PID
kill through _kill_agent_tree (it is process-sub-wrapped like the agent, so a
bare kill orphaned the real Next.js node server holding $PORT across redeploy).
Harden _kill_agent_tree and _agent_descendants to refuse a PID that is empty,
non-numeric, 0, or 1 (fail closed), making kill -9 0 / kill -9 1 structurally
impossible. Remove the ${AGENT_PID:-0} sentinel in the --check-size-once seam;
skip with a warning when AGENT_PID is unset instead of defaulting to 0.
Shared helper code kept byte-identical between the two entrypoints.
Both entrypoints run under set -e. The tail `wait -n $AGENT_PID $NEXTJS_PID`
returns non-zero on the PRIMARY designed exit path (137 = size-gate/watchdog
SIGKILL of the agent tree, or an agent crash), so set -e aborted the script AT
that line — making EXIT_CODE=$?, the entire 'which process exited with code N'
diagnostic, and the final `exit $EXIT_CODE` dead code on exactly the
interesting exits. Capture the code with `EXIT_CODE=0; wait -n ... || EXIT_CODE=$?`
so the diagnostic and explicit exit run and preserve the exact code (incl. 137);
the container-restart path is unchanged.
Same class: langgraph's LANGGRAPH_SIZE_THRESHOLD_MB was used in
`[ "$DIR_SIZE_MB" -ge "$threshold" ]` with no numericity guard, so a
non-integer operator override made the test error and silently no-op the size
gate every cycle. Validate the threshold the same way DIR_SIZE_MB already is
(numeric case guard + 'size guard inactive' WARNING, then skip safely).
The health-watchdog armed its 3-strike/~90s kill counter immediately with no
startup-grace window. langgraph-typescript has a 180s grace precisely to keep a
slow cold start from being killed mid-boot into a restart loop. Now that the
tree-kill makes the strands kill effective (the orphan bug previously made it
cosmetic), a slow tsx cold start (>90s) would be genuinely killed and loop.
Port langgraph's grace mechanism verbatim (GRACE=180, no env override, poll
every 5s, exit 0 on agent death during startup, arm anyway if grace elapses),
adapted to strands' :8000/health probe.
Two tightly-related defects in the size-gated restart machinery in
entrypoint.sh:
1. _watchdog_check_size_once validated the du/awk result only for
emptiness, not numericity. A non-integer value (junk du output, a
transient read error, a test-seam stub) reached the
`[ "$DIR_SIZE_MB" -ge ... ]` comparison and threw "integer expression
expected"; sitting inside an `if`, set -e was suppressed so the test
evaluated false and the size gate was SILENTLY skipped with no
warning (unlike the empty-string branch). Now match ^[0-9]+$ via a
case and emit the same "size guard inactive" WARNING, so the gate can
never silently disappear.
2. The size sub-loop used `_watchdog_check_size_once || break`, treating
ANY non-zero (including a transient check error) as a kill and
permanently ending the monitor for the container's lifetime. Now
break ONLY on the real-kill signal (rc==1) — preserving the
kill -> wait -n -> container-restart -> boot-purge contract — while a
transient non-zero keeps the monitor live and re-checks next cycle.
Verified RED->GREEN against the real entrypoint in node:22-slim with a
stubbed du seam: non-numeric du now warns and keeps the gate active; a
transient error no longer permanently disables the loop.
The tree-kill enumerated agent descendants in a single /proc snapshot then
killed. A child that forks a new child (or reparents) between the scan and the
kill escaped the walk, reparented to PID 1, and kept the agent port bound —
defeating the tree-kill's whole purpose of freeing the port before the
container restart.
Replace the single snapshot with a BOUNDED re-scan loop: keep the root alive as
the walk anchor, re-enumerate and SIGKILL live descendants deepest-first each
pass (up to 5 passes, 0.2s apart) until a scan comes back empty, then kill the
root last. Killing the root FIRST would immediately reparent every descendant to
PID 1 and make them unreachable by the root-anchored PPID walk, so root-last is
required for the re-scan to reap late/mid-walk descendants. A descendant that
fully daemonizes (double-fork to PID 1) before we reach it remains out of reach
— documented as an inherent limit of PPID-based reaping without job control; the
agent's npm->node tree does not daemonize.
Also document why the ${stat##*) } PPID parse is safe against a comm containing
") " (longest-prefix to the last ") " always lands on the true terminator).
Applied identically to langgraph-typescript (:8123) and strands-typescript
(:8000). Proven via local RED-GREEN in node:22-slim: pre-fix leaks a mid-walk
escapee (port stays bound), post-fix reaps it (0 orphans, port freed).
The cleanup() EXIT/SIGTERM trap in both langgraph-typescript and
strands-typescript entrypoints did a bare `kill $AGENT_PID`. Because
$AGENT_PID is the outer process-substitution subshell (not the real
npm->node server), this reaped only the subshell and orphaned the node
server (reparented to PID 1, still holding :8123 / :8000) on every
graceful/SIGTERM shutdown -- e.g. every Railway redeploy/rollover.
Route cleanup() through the existing _kill_agent_tree helper (as the
size-watchdog and health-strike kill sites already do), and move the
_agent_descendants/_kill_agent_tree helpers above cleanup()/the trap so
they are defined whenever the trap can first fire.
Brings the 499-commit-stale foundations branch up to date with main so #5761
has a clean diff and no stale reverts (e.g. forwardHeaders). Conflicts:
- CopilotThreadsDrawer.tsx: took main's (main renamed CopilotDrawer -> ThreadsDrawer
+ added the collapse feature; the branch's edit was a no-op import-type split).
- pnpm-lock.yaml: regenerated with the pinned pnpm 10.33.4 (adds @copilotkit/bot-intelligence).
strands-typescript/entrypoint.sh carries the identical latent trap fixed in
langgraph-typescript by this branch. The agent is launched through a process
substitution — `cd /app/src/agent && npm start &> >(awk …) &` — so
$AGENT_PID (=$!) is the outer subshell wrapping that pipeline, NOT the npm→node
tree it forks (`npm start` runs `node --import tsx server.ts`, which stays a
child of npm, not an exec-replacement). The health-strike `kill -9 $AGENT_PID`
therefore reaps only the subshell; npm and node reparent to PID 1 and KEEP
RUNNING, still bound to :8000. `wait -n` never observes the real server die →
container never restarts → the frontend proxies to a dead-but-not-restarted
agent forever (edge 502s). strands-typescript has no size-gate, so this only
fires after the 3-strike (~90s) health counter exhausts, but it is a real
latent footgun with the same root cause.
Fix: reuse the /proc-based `_kill_agent_tree` helper (node:22-slim ships
neither ps nor pgrep, and job control is off so a group kill would take out
the whole entrypoint) at the single health-strike kill site. The whole
npm→node tree now dies, :8000 is freed, `wait -n` returns and the container
restarts.
Red-green proven in a real node:22-slim container against the exact
process-sub → npm → node structure: RED (bare kill -9 $AGENT_PID) leaves node
orphaned and :8000 still LISTENing; GREEN (_kill_agent_tree) reaps the tree,
0 orphans, port freed.
The size-gated watchdog added in ef103f5f5 does `kill -9 $AGENT_PID`
expecting the container to exit and Railway to restart (re-running the
boot-purge). But $AGENT_PID is the process-substitution subshell wrapping
`cd /app/src/agent && npm start &> >(awk …)`, NOT the npm→node tree it
forks. A single-PID kill reaps only that subshell; npm and node reparent
to PID 1 and keep running, still bound to :8123 with the bloated in-memory
state resident. The frontend then proxies to a dead-but-not-restarted
agent and the edge 502s forever.
Fix: add a /proc-based `_kill_agent_tree` (node:22-slim ships neither ps
nor pgrep, and job control is off so a group kill would take out the whole
entrypoint) and use it at both watchdog kill sites (size gate + health
strikes). The whole npm→node tree now dies, :8123 is freed, `wait -n`
returns, the container restarts and re-runs boot-purge — preserving the
original RangeError-prevention intent.
Two bugs fixed:
1. Tool result event path: the `a2ui_operations` container was emitted
inside a `TextMessageContentEvent` block. The A2UI middleware only scans
`TOOL_CALL_RESULT` events for the container, so the card never mounted
and the raw JSON appeared as plain text in the chat. Fixed by emitting a
`ToolCallResultEvent` (matching the claude-sdk-python peer).
2. Operation shape: the ops used the legacy flat form
(`{"type": "create_surface", ...}`) which the renderer silently ignores.
Updated to the v0.9 nested form (`{"version": "v0.9", "createSurface":
{...}}`) used by every other working peer (claude-sdk-python, strands,
google-adk).
Also adds the missing langroid aimock D6 fixture for `gen-ui-a2ui-fixed`
(`display_flight` → tool result → confirmation text) so the D6 probe has
a mock response to drive the full surface-render assertion.
D6 cell: d6:langroid/gen-ui-a2ui-fixed red → green
## Summary
Spring-ai's `DisplayFlightTool` was emitting legacy flat A2UI operations
(`{"type":"create_surface",...}`) but the A2UI middleware expects v0.9
nested operations (`{"version":"v0.9","createSurface":{...}}`). This is
the same flat→nested migration done for Python/TS in #5832 and langroid
in #5839. The flat shape was silently ignored by the middleware, so the
flight card never mounted and the `a2ui-fixed-schema` D6 cell was
permanently red.
**Root cause** (confirmed by prior local red-green on the disproven TS
fix):
`DisplayFlightTool.apply()` emitted `a2ui_operations` in the legacy flat
format. The middleware's `tryParseA2UIOperations` parses the container
correctly, but the op dispatchers inside require the v0.9 shape. No
surface
was ever created → card never mounted.
## Fix
Updated the three operations in `DisplayFlightTool.java` to v0.9 nested
format:
| Before (flat, ignored) | After (v0.9 nested, works) |
|---|---|
| `{"type":"create_surface","surfaceId":...,"catalogId":...}` |
`{"version":"v0.9","createSurface":{"surfaceId":...,"catalogId":...}}` |
| `{"type":"update_components","surfaceId":...,"components":...}` |
`{"version":"v0.9","updateComponents":{"surfaceId":...,"components":...}}`
|
| `{"type":"update_data_model","surfaceId":...,"data":{...}}` |
`{"version":"v0.9","updateDataModel":{"surfaceId":...,"path":"/","value":{...}}}`
|
Shape matches `sdk-python/copilotkit/a2ui.py` and the shared Python
tools.
## Local Red-Green Proof (real control-plane probe, `--rebuild` both
runs)
**RED — original flat ops (`{"type":"create_surface",...}`):**
```
$ showcase test spring-ai:a2ui-fixed-schema --d6 --rebuild --keep
✗ d6:spring-ai/gen-ui-a2ui-fixed red (0.0s)
state=red
⚠ Tests failed for spring-ai:a2ui-fixed-schema (exit 1)
```
**GREEN — v0.9 nested ops
(`{"version":"v0.9","createSurface":{...}}`):**
```
$ showcase test spring-ai:a2ui-fixed-schema --d6 --rebuild --keep
✓ d6:spring-ai/gen-ui-a2ui-fixed green (0.0s)
1 passed
✓ Tests passed for spring-ai:a2ui-fixed-schema
```
## Java Build & Tests
All 64 existing spring-ai JUnit tests pass after the change (`mvn test`:
64 run, 0 failures, 0 errors). Code compiles cleanly (`mvn compile -q`).
## Files Changed
-
`showcase/integrations/spring-ai/src/main/java/com/copilotkit/showcase/springai/tools/DisplayFlightTool.java`
— v0.9 nested op format
The `route.ts` file is unchanged from main (the prior no-op `a2ui: {
injectA2UITool: true }` addition has been reverted — it was disproven as
a fix by a real local red-green).
- Remove unused imports: Iterable, ConversableAgent (from autogen),
AGStreamInput (from autogen.ag_ui.adapter) — none appear in executable
code, only in docstring prose.
- Fix raw_msgs possibly-unbound at dispatch guard: initialize to None
before the try block so the identity check at line 302 is always
safe even if model_dump raises before raw_msgs is assigned. Also
tighten the guard to `raw_msgs is not None` to make the no-normalization
fallback explicit.
autogen.ag_ui import unresolved and LLMConfig(dict) "Expected 0 positional
arguments" are ENVIRONMENT findings — autogen.ag_ui ships only in the
ag2[ag-ui] extra (present in the container, not in local Pyright's venv),
and LLMConfig({...}) is the codebase-wide pattern that works at runtime
with ag2>=0.9 as installed in the container.
AG2's ConversableAgent runs every user message through
``autogen.code_utils.content_str``, which only accepts content-part
types in {"text", "input_text", "image_url", "input_image", "function",
"tool_call", "tool_calls"}. CopilotChat / the AG-UI runtime emits image
and document attachments as the modern shape
{"type": "image" | "document", "source": {...}}
and the demo page's legacy-converter-shim.tsx ALSO appends a legacy
{"type": "binary", mimeType, data | url}
mirror alongside it (to keep the @ag-ui/langgraph converter happy on
LangChain-based integrations — it rides through on the ag2 path too).
Both shapes trip autogen's allowed-types gate with
ValueError("Wrong content format: unknown type image within the
content")
…BEFORE the request reaches the vision model — observed live in the
D6 multimodal probe (commit d8a0a25db, which originally quarantined
the feature as NSF).
Fix
---
Add ``agents/_multimodal_normalize.py``: a ``NormalizingAGUIStream``
subclass of ``AGUIStream`` that overrides ``dispatch()`` to normalize
AG-UI image/document/binary content parts to OpenAI Chat Completions
``image_url`` parts AFTER ``RunAgentInput`` Pydantic parsing and BEFORE
``AgentService`` serialises the messages for autogen.
This is the only correct interception point:
- Too early (ASGI body rewrite before Pydantic): ``RunAgentInput``
rejects ``image_url`` because it is not an AG-UI standard type —
the discriminated union only accepts image/document/binary/text.
- Too late (inside ConversableAgent): requires patching autogen
internals.
The override works by calling ``normalize_messages_for_autogen()`` on
the dict-serialised messages (same form as ``run_stream`` produces via
``model_dump()``) and re-injecting them via a ``_PatchedRunAgentInput``
wrapper that overrides only ``.messages``, delegating all other
attribute access to the original ``RunAgentInput``.
Conversions:
- {"type": "image", "source": {"type": "data", value, mime_type}} →
{"type": "image_url", "image_url": {"url": "data:<mime>;base64,<value>"}}
- {"type": "image", "source": {"type": "url", value}} →
{"type": "image_url", "image_url": {"url": value}}
- {"type": "document", "source": ...} → image_url with the document's
mime preserved (data:application/pdf;base64,...). The vision model
still can't natively read PDFs, but the request reaches the model
instead of being rejected upstream, which is the failure mode this
fix targets.
- {"type": "binary", mimeType, data | url} → image_url (the
legacy-shim parts ride through cleanly).
- {"type": "text", ...} and already-normalised image_url parts pass
through unchanged (identity-preserved on no-op turns).
Failure path: any normalization error is logged at WARNING and the
original messages are forwarded unchanged — autogen's own ValueError
fires verbatim with its error surface intact.
Manifest + fixture
------------------
- showcase/integrations/ag2/manifest.yaml: remove multimodal from
not_supported_features (with its now-stale comment) and add it back
to the features list next to voice.
- showcase/aimock/d6/ag2/multimodal.json: add the D6 fixture pair
using the actual autoPrompt strings from sample-attachment-buttons.tsx
("can you tell me what is in this demo image I just attached" /
"can you tell me what is in this demo pdf I just attached").
TDD evidence (red-green)
------------------------
showcase/integrations/ag2/tests/python/test_multimodal_normalize.py
contains 14 unit tests, pinned at three layers:
1. RED/GREEN against autogen's actual content gate:
* test_autogen_rejects_raw_agui_image_part — confirms
content_str([{type: image, source: ...}]) raises the verbatim
ValueError the D6 probe surfaced. This is the regression pin: if
autogen ever relaxes the gate, this test fails and we know to
revisit the normalizer.
* test_normalized_content_is_accepted_by_autogen — after
normalize_messages_for_autogen(...), content_str accepts every
part and renders "<image>" for the image_url part.
2. Shape coverage: modern image data/url, modern document, legacy
binary data/url, mimeType camelCase alias, plain-text passthrough,
plain-string content, assistant/tool messages untouched,
unrecognised source → text placeholder, idempotency.
3. NormalizingAGUIStream class surface tripwire.
Control-plane D6 RED→GREEN:
RED (no normalizer, pre-fix container): d6:ag2/multimodal → red
(HTTP 500 agent_run_error_event from content_str ValueError)
GREEN (NormalizingAGUIStream applied): d6:ag2/multimodal → green
The DisplayFlightTool was emitting legacy flat A2UI operations
({"type":"create_surface",...}) but the A2UI middleware expects v0.9
nested operations ({"version":"v0.9","createSurface":{...}}).
This is the same flat→nested migration done for Python/TS in #5832 and
langroid in #5839. The flat shape was silently ignored by the middleware,
so the flight card never mounted and the a2ui-fixed-schema D6 cell was
permanently red.
Fix: update the three ops to the v0.9 nested format:
- createSurface (was type:create_surface)
- updateComponents (was type:update_components)
- updateDataModel with value: (was type:update_data_model with data:)
Local red-green proof (real control-plane probe, --rebuild both times):
RED: d6:spring-ai/gen-ui-a2ui-fixed red (flat ops, card never mounts)
GREEN: d6:spring-ai/gen-ui-a2ui-fixed green (v0.9 nested ops, card mounts)
## Summary
- Productizes the Claude SDK Python and TypeScript showcase demos with
LangGraph-parity frontends.
- Wires the Claude demo backends through the official Claude Agent
SDK/AG-UI adapter paths using `claude-sonnet-4.6`.
- Keeps Claude integration docs hidden for this PR and excludes
generated/authored docs artifacts from scope.
## Why
The goal is to bring the productized LangGraph demo surface to Claude
Agents SDKs without publishing integration docs in this pass. This keeps
the PR focused on local showcase demos, runtime behavior, fixtures, and
validation support.
## How
- Ported the demo frontend surfaces and local shell-dojo support for
Claude SDK Python/TypeScript.
- Added official Claude SDK adapter/backend wiring plus real-Claude
local compose support.
- Updated Claude aimock fixtures and validation ratchets for the
expanded demo set.
- Set both Claude manifests to `docs_mode: hidden` and removed docs
setup/snippet artifacts from the PR scope.
## Root Cause
`agno 2.6.20` removed `agno.os.interfaces.agui.utils`. The floating
`agno>=2.5.17` pin in `requirements.txt` caused staging to pull the
breaking version on the next build, causing a startup failure.
## Red-Green Proof
**RED** — with `agno>=2.6.20` installed:
```
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'agno.os.interfaces.agui.utils'
```
**GREEN** — with `agno==2.6.19` installed:
```
GREEN: all 3 symbols OK
```
(Symbols confirmed: `async_stream_agno_response_as_agui_events`,
`extract_agui_user_input`, `validate_agui_state`)
## Changes
- `showcase/integrations/agno/requirements.txt`: pinned `agno>=2.5.17` →
`agno==2.6.19` (exact pin, last version with `agui.utils`)
- `showcase/integrations/agno/src/agent_server.py`: added TODO comment
at line 73 import site noting migration to agno 2.6.20+ API is a
follow-up; no structural changes to imports
## Follow-up
Migration of `agent_server.py` imports to the agno 2.6.20+ API (once the
replacement for `agui.utils` is identified) is tracked in the TODO
comment at line 73.
## Note on CI
The `validate-pins` CI check will likely flag pre-existing non-exact
pins across ~15 other integrations (`openai ^5.9.0`, `crewai` ranges,
etc.). This is pre-existing debt not introduced by this PR.
## What & why
Resolves [OSS-132](https://linear.app/copilotkit/issue/OSS-132).
Investigated with systematic-debugging; every conclusion verified
against the **real** OpenAI Responses API.
**Net change: a TanStack version bump only.** No showcase schema change.
- `@tanstack/ai` `0.18.0` → `0.35.0`
- `@tanstack/ai-openai` `0.9.1` → `0.15.6`
- `package-lock.json` regenerated (Dockerfile uses `npm ci
--legacy-peer-deps`)
## The bug
The built-in-agent showcase 400s on every prompt against real OpenAI.
The state tools (`AGUISendStateSnapshot` / `AGUISendStateDelta` /
`set_steps`) declare arbitrary payloads as `z.any()`, which serializes
to a **typeless** JSON-Schema property (`{ "description": ... }`, no
`"type"`).
The old `@tanstack/openai-base`'s `isStrictModeCompatible()` only
screened for `oneOf/allOf/not/$ref/$defs`, so it missed the missing
`type`, sent the tool with `strict: true`, and OpenAI rejected it:
```
400 Invalid schema for function 'AGUISendStateSnapshot':
In context=('properties','snapshot'), schema must have a 'type' key.
```
This was **masked in production** because the deployed showcase runs
against aimock, which replays fixtures without validating the request
schema — a raw `curl` to prod returns a clean `RUN_FINISHED`, green for
the wrong reason.
The ticket's original framing (zod3/zod4 drift → typeless *root*, `got
"None"`) was already fixed by the zod-4 migration; this is the same
symptom one layer down (typeless *property*).
## The fix is upstream
`@tanstack/ai-openai@0.15.6` (via `@tanstack/openai-base@0.9.2`) fixes
`isStrictModeCompatible`: it now detects typeless / `z.any()` properties
and sends `strict: false`. OpenAI accepts typeless properties under
`strict: false` — so `z.any()` works again with no schema change on our
side.
(`@tanstack/ai-openai@0.15.5` also dropped `@tanstack/ai-client` from
its peerDependencies, so no `ai-client` dep is added.)
## Verification (real OpenAI, gpt-4o)
| Probe | Result |
|---|---|
| Typeless property, `strict: true` (raw OpenAI) | **400** — `schema
must have a 'type' key` |
| Typeless property, `strict: false` (raw OpenAI) | **ACCEPTED** —
confirms it was the strict flag, not the schema |
| `z.any()` tool on old adapter (0.9.1/0.15.4) | adapter sends `strict:
true` → **400** |
| `z.any()` tool on new adapter (0.15.6) | adapter sends **`strict:
false`** → **ACCEPTED**, model calls the tool |
| All 3 `z.any()` state tools attached, new adapter | **ACCEPTED**, no
400 |
## Not covered here
The showcase's aimock + Playwright e2e suite was **not** run locally
(this worktree has no installed toolchain). CI runs it on this PR;
please confirm the gen-ui / shared-state demos still pass before merge.
---
_Branch history shows an interim `z.string()` workaround that was
reverted once the upstream fix shipped; the net diff is the version bump
only. Squash-merge recommended._
agno 2.6.20 removed agno.os.interfaces.agui.utils; the floating
agno>=2.5.17 pin in requirements.txt caused staging to pull the
breaking version. Pinned to 2.6.19 (last version with the module).
Added TODO comment at the import site for future migration.
Gates per-request POST + 2xx Response-status + GET health-probe logs behind SHOWCASE_ROUTE_DEBUG across 19 integrations to stay under Railway's 500-logs/sec cap, while logging non-2xx responses unconditionally so production errors stay visible.
Boot-purge of stale .langgraph_api state plus a size-gated restart (du > threshold -> kill agent -> container restart -> purge), replacing an in-flight-wiping periodic truncate loop. Adds mutation-sensitive subprocess tests for the watchdog.
The v1 bridge is now published stable (npm latest). Swap the alpha pin for the
GA release. Resolves to @ag-ui/mastra 1.1.0; next build clean (all routes).
The reasoning demos (reasoning-default, reasoning-custom,
tool-rendering-reasoning-chain) never rendered a reasoning message. Two
root causes, both showcase-side wiring (the @ag-ui/mastra bridge forwards
reasoning correctly):
1. Agent-not-found: the reasoning-default and reasoning-custom pages request
agent="reasoning-default"/"reasoning-custom", but the runtime registry
listed the stale feature keys "reasoning-default-render" and
"agentic-chat-reasoning" instead, so the chat never started. Register the
real agent names (the demoAgentNames parity test enforces this).
2. Non-reasoning model: all reasoning demos mapped to the default weatherAgent
(gpt-4o), which the OpenAI Responses API never emits reasoning-summary
items for, so the reasoning slot stayed dark. Add a dedicated reasoningAgent
(gpt-5-mini via OPENAI_REASONING_MODEL) with
providerOptions.openai.{reasoningEffort,reasoningSummary:"detailed"} on the
agent's default stream options, mirroring langgraph-python's reasoning_agent.
Map reasoning-default and reasoning-custom to it.
For tool-rendering-reasoning-chain, add a dedicated reasoningChainAgent that
registers the four chain tools (get_weather, search_flights, get_stock_price,
roll_dice, the latter new) under the exact tool-call names the aimock fixtures
emit, on the reasoning model, so Mastra executes each leg and the multi-turn
chain advances through its toolCallId-keyed fixtures to the closing narration.
Fixture parity fixes so the chain's gold toolCallId scheme wins on the
Responses path (matching langgraph-python):
- d4/mastra/chat.json: rename the broad "weather" and "flights from SFO to JFK"
probes to the non-colliding "_d4_unused_*" sentinels gold uses.
- d6/mastra/tool-rendering.json: the basic AAPL fixture used turnIndex:0, which
matched as a behind-count turn and stole the stock chain's later turns; switch
to hasToolResult:false (gold parity) so it only answers the no-tool-result turn.
Manifest features aligned to gold (reasoning-default, reasoning-custom). Removed
the stray agentic-chat-reasoning e2e spec (no page, no gold equivalent) and
renamed the reasoning QA docs to match the cell ids.
Verified via Playwright against the up --dev mastra container + aimock:
reasoning-default 2/2, reasoning-custom 7/7, tool-rendering-reasoning-chain 5/5.
Every other ACTIVE mastra cell has a per-integration e2e spec; interrupt-headless
had only the D6 harness probe (d5-interrupt-headless.ts). Adds a committed spec
mirroring that probe + the gen-ui-interrupt style: page-load/empty, both pills,
pick-a-slot resume (app-surface popup -> slot -> empty re-mounts + assistant
confirmation), cancel path. 8/8 green with gen-ui-interrupt on the host-prod
aimock rig (next start :3199 + container aimock, workers=1).
Validated the OSS-426 background-agents cell against the running dev stack
(showcase up mastra --dev + Playwright e2e on :3104). Findings:
- Under getLocalAgents({untilIdle:true}) the activity card NEVER reaches
'Completed': a probe polling the card status for 45s showed it stuck on
'Working…' the whole time, and the e2e was flaky (first attempt timed out
at 60s before the card even painted).
- Root cause: instrumenting run_deep_research.execute proved it NEVER fires
within the run. Mastra dispatches the backgroundable tool but nothing
executes it — this single-process Next.js demo has no background worker to
pick the task up, so no background-task-completed chunk is ever produced.
untilIdle only holds the stream open for the full idle timeout (many wasted
re-entry LLM calls, slow/flaky render) with zero completion benefit.
- Reverted to plain getLocalAgent: the run closes right after
background-task-started, the 'working' card paints fast and deterministically,
completion is out of band exactly as the tool/renderer/e2e/qa already
document. Full spec now green: 4 passed in ~3.7s, stable across 3 runs.
The OM activity card turned out to be deterministically reproducible under
aimock, not just via a real LLM as previously assumed. Measured against the
sanctioned rig (showcase up mastra --dev + Playwright vs :3104):
- one sizable pill click always paints exactly one card in buffering/running
in-turn (token-accounting driven, no Observer-LLM dependency) — 11/11
- a second sizable turn settles the first cycle to activation/activated and
opens a fresh buffering/running card — 5/5
Strengthen the e2e from a best-effort probe to hard assertions on that
lifecycle, and correct the QA doc: structure is aimock-deterministic (now
e2e-covered); only the observation CONTENT still needs a real-LLM QA pass.
Refs OSS-427
The mastra runtime image stripped two build-only artifacts that next dev
needs to re-resolve at request time: tsconfig.json (the @/* and
@copilotkit/showcase-shared-tools path aliases) and shared-tools/. Under
'showcase up mastra --dev' the dev overlay bind-mounts only ./src and runs
next dev, which recompiles from source and then failed with 'Module not
found: Can't resolve @/mastra' / '@copilotkit/showcase-shared-tools', so
every agent route (incl. /api/copilotkit-a2ui-recovery) returned 500 and no
agent run ever fired. next start (prod) is unaffected since .next has the
aliases pre-resolved. Mirrors langgraph-typescript's runner, which likewise
bakes shared-tools/ for dev.
Verified: with the fix baked, the recovery route compiles and POSTs 200 in
dev (LLM calls fire, aimock recovery fixtures match) instead of 500. The
a2ui-recovery e2e (heal paints Column+2 Metric, exhaust shows the fallback)
is 3/3 green on the committed-state built image.
The shared-state-streaming demo mapped to the generic weatherAgent (state
{proverbs}, no `document` field) and its aimock fixture returned plain text
gated on a stale "stream the counter to 5" match, so nothing ever streamed
into `state.document` and the e2e failed.
Add a dedicated `sharedStateStreamingAgent` with working memory enabled on a
`{ document: string }` schema. It writes drafts through Mastra's built-in
`updateWorkingMemory` tool; the @ag-ui/mastra bridge intercepts the streamed
tool-call args (OSS-414) and emits a leading STATE_SNAPSHOT followed by
incremental STATE_DELTA on /document, so the UI renders the document
token-by-token. This is the Mastra-native equivalent of langgraph-python's
StateStreamingMiddleware / predictive-state pattern.
- agents: add sharedStateStreamingAgent + SharedStateStreamingAgentState
- index: register the agent on the Mastra instance
- route: map shared-state-streaming -> sharedStateStreamingAgent with a
dedicated resourceId, build guard, and LocalMastraAgentName entry
- fixture: drive updateWorkingMemory with the streamed document for all three
pills (poem / email / quantum) plus a confirmation turn on hasToolResult
Parity with langgraph-python gold, which ships qa/gen-ui-interrupt.md +
qa/interrupt-headless.md; mastra had neither. Mastra-accurate: native
schedule_meeting suspend() -> @ag-ui/mastra bridge -> useInterrupt, the
TimePickerCard testids/states (time-picker-card/picked/cancelled, slot,
cancel), and the headless app-surface variant (interrupt-headless-popup/
empty/slot-<iso>). No shell-docs change — human-in-the-loop/interrupt-flow.mdx
already documents the native path and the correct context.agent destructure.
The native interrupt tool destructured `suspend`/`resumeData` off the top level
of the ToolExecutionContext, but in @mastra/core 1.48 they live under
`executionContext.agent` (the AgentToolExecutionContext sub-object). So
`suspend` was `undefined`, `return suspend(...)` threw `suspend is not a
function`, the model got a tool-error and re-called schedule_meeting, and the
agentic loop spun to the step cap (~30 LLM calls) with NO `tool-call-suspended`
chunk -> no `on_interrupt` -> the time-picker never rendered.
Reading off `executionContext.agent` (matching the proven @ag-ui/mastra dojo
tool) makes the loop pause at the interrupt. Verified e2e against aimock:
gen-ui-interrupt 4/4 and the interrupt-headless probe flow both green
(picker renders backend suspend-payload slots -> pick -> resume -> confirmation).
Two fixes surfaced by running the actual Playwright e2e (per showcase-demo-debugging):
- interrupt.ts: Mastra createTool execute is (inputData, executionContext) —
suspend/resumeData live on arg2, not arg1. Was destructured off arg1 →
suspend was undefined → suspend() threw → loop never paused. Fixed signature.
(NOTE: e2e still shows the loop not pausing — a deeper suspend-path issue
remains; tracked to the interrupt fix chip.)
- background-agents route: use getLocalAgents({ untilIdle: true }) (plural; the
singular getLocalAgent lacks the toggle) so the background-task lifecycle
(incl. background-task-completed + result) pipes in-band per Mastra's
resolution-order docs — card can complete in-turn instead of stuck 'working'.
Honest status: the advanced Mastra cells (interrupt, a2ui-recovery, reasoning,
shared-state-streaming) are NOT yet e2e-green under aimock; 7 focused follow-up
tasks spawned (one per item) with repro + the sanctioned up --dev validation rig.
Integrates 3 gap demos (authored in parallel worktrees, ported onto the
upgraded branch):
- background-agents (OSS-426): run_deep_research tool flagged
background:{enabled:true} + Mastra backgroundTasks:{enabled:true} → the bridge
maps background-task-started → a live 'working' activity card. Completion is
out-of-band by design (not asserted). Dedicated route + fixture + e2e + qa.
- observational-memory (OSS-427): OM enabled on the agent Memory
(scope:thread, observation 600/300) + surfaced via getLocalAgents({
observationalMemory:true }). SIZABLE pills trip the token-size trigger.
NOTE: OM data-om-* chunks come from the OM processor + observer LLM, not the
mocked completion — so it does NOT replay deterministically under aimock; the
e2e asserts the deterministic subset (page + pills + completing turn) and
full OM-card verification needs a real-LLM run (documented in qa).
- browser-use (OSS-91): Mastra-only, real-LLM. browse_web tool drives a LOCAL
headless Playwright Chromium (NO Browserbase) — top HN / page read, rendered
as in-chat cards. Non-deterministic → no aimock D6 fixture; smoke e2e only.
Needs 'npx playwright install chromium' at runtime (documented in Dockerfile+qa).
Shared: 3 agents + registrations + backgroundTasks toggle, 2 tool exports,
manifest features+demos (now 42 demos, not_supported_features still []),
playwright dep, demoAgentNames excludes for the 4 dedicated-route cells.
next build clean (all routes); validate-parity 0-fail; validate-pins baseline (38).
The @copilotkit/react-core v2 resume-path bug that quarantined gen-ui-interrupt
+ interrupt-headless is fixed as of 1.62.1, so migrate both cells to the native
interrupt path and move them out of not_supported_features.
- Bump @copilotkit/* 1.61.2 -> 1.62.1 (react-core/runtime/shared/voice/
a2ui-renderer + web-inspector core override). next build clean (40 routes).
- Backend: new src/mastra/tools/interrupt.ts — a real Mastra suspend tool
(schedule_meeting) with suspend/resume schemas; returns suspend() directly so
the agentic loop pauses. Wired into interruptAgent (was tools:{}). Instance
storage (src/mastra/index.ts) already satisfies the resume snapshot prereq;
emitInterruptOutcome defaults true in the v1 bridge so the standard
RUN_FINISHED outcome fires (resumable on client >=1.61.2).
- gen-ui-interrupt: useHumanInTheLoop workaround -> native useInterrupt
(renderInChat), reading the Mastra suspend wrapper's suspendPayload.
- interrupt-headless: hand-rolled on_interrupt subscription -> native
useInterrupt({renderInChat:false}) placed in the app surface (the hook handles
both the standard outcome and legacy on_interrupt + the correct resume array).
- manifest: not_supported_features now []; gen-ui-interrupt + interrupt-headless
in features; interrupt_pattern: native (parity with langgraph-python).
- fixture: gen-ui-interrupt.json userMessage keys (introductory sales team call
/ one-on-one with Alice) didn't substring-match the D6 probe + e2e prompts
(never caught while skipped) -> aligned to 'intro call with the sales team' /
'1:1 with Alice'. interrupt-headless.json already correct.
Local D6 e2e not run (Docker daemon container-create wedged in this env);
build-verified + relying on CI/harness. Native suspend/resume path is
dojo-proven on CopilotKit >=1.61.2.
Jordan Ritter