The watchdog test cluster in test_entrypoint_watchdog.py had three soundness
defects that let tests pass without exercising real behavior.
1. test_size_pid_reaped_by_in_subshell_trap asserted the literal
trap kill $SIZE_PID string, which exists ONLY in an explanatory comment
describing the old design. The shipped code registers
trap _reap_watchdog_children EXIT, so the test passed even when the real
trap registration was removed. Now asserts the shipped registration on an
executable (comment-stripped) line and that the named handler is defined.
2. test_size_pid_reaped_on_watchdog_exit_behavioral wrote its own mock
watchdog using the older SIZE_PID trap shape and never invoked shipped
code, so it passed even when the shipped reaper was gutted. Now extracts
the real _agent_descendants and _reap_watchdog_children definitions
verbatim from entrypoint.sh and drives them with the shipped order.
3. dummy_was_killed used a single-shot dummy.poll() immediately after the
run returned, racing the asynchronous SIGKILL delivery. Replaced with
bounded _wait_process_exited / _assert_still_alive helpers selected by
return code, making the killed/alive assertion deterministic.
entrypoint.sh unchanged. 22 passed; determinism confirmed by repeated runs.
The langgraph-typescript backend's @langchain/langgraph-api FileSystemPersistence
serialises all accumulated thread/run/checkpoint state to .langgraph_api on a
3-second timer. Under the D6 probe fan-out (36 parallel probes) the dir filled
past the 200MB size-watchdog threshold in ~90s, the watchdog killed the agent,
and on rapid restart the D6 cron refilled and re-tripped it until Railway
crash-loop backoff stopped restarting the container (2026-07-13 outage, staging
and prod).
Mirror PR #5825's langgraph-python fix, which exported
LANGGRAPH_DISABLE_FILE_PERSISTENCE=true so the python inmem runtime skips its
flush-to-disk loop. The TS package has no such switch and its persistence
writers are unexported module singletons behind an exports-map wall, so ship a
node --import preload (src/agent/disable-file-persistence.mjs) that, gated on the
same env var, no-ops node:fs/promises writeFile/mkdir for .langgraph_api paths
while leaving in-memory state (the real runtime state) intact. Wire it into
npm start and export the env var in entrypoint.sh.
Behavior preserved: runs still execute and thread state reads back from the
in-memory checkpointer within the container lifetime; only disk persistence is
removed, so the size-watchdog has nothing to fill and never trips under load.
Three wired demos 404'd on load: their pages point <CopilotKit runtimeUrl>
at /api/copilotkit-<demo>, but those route handlers were never created in the
Mastra integration (the pages were mirrored from langgraph-python without
porting the routes). The runtime-info fetch 404'd, so the page never mounted
(runtime_info_fetch_failed).
Add the three dedicated routes, mirroring the proven copilotkit-beautiful-chat
pattern. The two A2UI demos set a2ui.injectA2UITool:false (weatherAgent already
owns generate_a2ui — avoid a double-bind) and pin defaultCatalogId to the
catalog the page registers. agent-config registers the agent id the page
requests (agent-config-demo).
Page-load fix only; full behavioral parity (dedicated Mastra agents) is OSS-381.
Verified: next build compiles all three into the route manifest; POST returns
400 (route resolves) identically to copilotkit-beautiful-chat, vs 404 for a
nonexistent route.
Refs OSS-451
Align the claude-sdk-python and claude-sdk-typescript demo integrations behind
the published quickstarts: move the @region markers used for doc snippet
extraction, add the state-streaming and weather-tool snippet files, and add
setup-doc content. Runtime alignment: the TS agent handlers consistently emit
text/event-stream; the streaming snippets emit a fresh STATE_SNAPSHOT per delta
and drop the undeclared partial-json dependency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The launch-site comment claimed process substitution leaves $! pointing at
the real node process. That is false and contradicts the file header: $!/
AGENT_PID is the wrapping subshell, and the real npm->node server is a
descendant reached only via the tree-kill (the reason _kill_agent_tree exists).
Rewritten in both langgraph-typescript and strands-typescript entrypoints so no
maintainer reintroduces a bare kill.
Also in langgraph-typescript: clarify that the SIZE_PID kill is a retained
belt-and-suspenders backstop to the $BASHPID PPID-walk (not dead code), and
re-anchor the startup-grace rationale to the real cause (the top-level
@langchain/langgraph-api import cost), since the prod path no longer uses
langgraph-cli dev.
Comment-only; no executable code changed.
The bounded re-scan loop's sleep 0.2 was unguarded. Under set -e on a base image
whose sleep can return non-zero (e.g. a future busybox/Alpine rebase), a failed
sleep would abort the tree-kill mid-walk — root never killed, real npm→node
server left orphaned. Add || true so the walk completes regardless of sleep's
exit status. No behavior change on the current Debian base (coreutils sleep
succeeds). Helper kept byte-identical across both entrypoints.
Both langgraph-typescript and strands-typescript entrypoints inferred which
process exited via a post-hoc kill -0 if/elif after wait -n. That inference is
racy: on a near-simultaneous exit both PIDs are dead by probe time, so the first
kill -0 branch always wins and mislabels the diagnostic (naming the agent when
Next.js actually exited, attaching the wrong code to the wrong name). Use
bash's wait -n -p REAPED_PID (bash >= 5.1; node:22-slim ships 5.2) to capture
the actual reaped PID and key the message off it. Exit code (incl. 137) and the
final exit $EXIT_CODE are preserved; || EXIT_CODE=$? guard is unchanged.
Three related size-watchdog hardening changes in the langgraph entrypoint:
- Trap-order leak window: the size sub-loop was backgrounded (`( … ) &`,
SIZE_PID=$!) BEFORE its reaping `trap … EXIT` was registered, so an outer
SIGTERM landing in that window exited the watchdog subshell with no trap
armed and orphaned the sub-loop (reparented to PID 1, spinning for the
container's life). Arm the reaping trap FIRST, and reap via a $BASHPID
PPID-walk that finds the child regardless of whether SIZE_PID is assigned
yet — no ordering-dependent leak.
- Startup-grace size coverage: the size monitor started only AFTER the
up-to-180s startup-grace loop, leaving the size ceiling unguarded during a
pathological cold boot. Start it BEFORE the grace loop; safe because
_watchdog_check_size_once already fail-closes on every not-yet-ready
condition (agent PID not alive, PERSIST_DIR missing, non-numeric size/
threshold), so early cycles are harmless no-ops until the dir grows.
- cleanup() comment accuracy: corrected the note claiming WATCHDOG_PID
"forks nothing that outlives it" — it DOES fork the size sub-loop; the
bare `kill $WATCHDOG_PID` is safe because the watchdog's own inner EXIT
trap reaps that child, not because it forks nothing.
Proven RED->GREEN on the real entrypoint in node:22-slim: pre-fix the size
sub-loop keeps ticking after the watchdog exits (orphan) and the size monitor
spawns after the grace loop (unguarded); post-fix the sub-loop is reaped (0
ticks) and the monitor runs during grace (size-check fires within the grace
window). --check-size-once seam re-verified under/over threshold.
The numeric-config validator accepted any positive integer (`[1-9][0-9]*`),
so a 20+ digit override overflowed bash's signed-64-bit arithmetic and either
wrapped to a negative/garbage magnitude or aborted the `[ -ge ]` test with
"value too great for base" — which, suppressed to false inside the guard's
`if`, silently disabled the guard for the container's lifetime (the exact
fail-open class this validator exists to prevent).
Add a 10-digit length cap (max 9,999,999,999 — comfortably inside int64,
far above any real interval/threshold/strike knob) checked BEFORE the digit
`case`, since an all-digit 23-char value would otherwise pass validation.
A too-long value now takes the same WARN + fall-back-to-default fail-safe
path as every other bad override. Byte-identical across both entrypoints.
Proven RED->GREEN on the real entrypoints in node:22-slim: pre-fix a 23-digit
value survives validation and `$(( x * 3 ))` yields int64-wrapped garbage;
post-fix it WARNs, clamps to the default, and arithmetic is correct.
The /proc PPID walk forked an awk process for every entry in the process
table on every scan pass. Replace the `echo "${stat##*) }" | awk '{print $2}'`
pipeline with the `read` builtin, which word-splits the post-comm remainder
("STATE PPID PGRP …") on IFS and captures the 2nd field with no subprocess.
Byte-identical across the langgraph-typescript and strands-typescript
entrypoints. Non-behavioral; bash -n + shellcheck --severity=warning clean.
## What & why
Two showcase agent containers (**langgraph-typescript**,
**strands-typescript**) could enter a *running-but-dead* state: Railway
showed the service `● Online` while `/api/health` returned **HTTP 502**.
This took down all 36 LGT dashboard cells on staging **and** prod
(prod's `.langgraph_api` had crossed the 200 MB size-watchdog
threshold).
**Root cause:** the agent is launched via process substitution (`... &>
>(awk …) &`), so `$AGENT_PID` (`=$!`) is the **wrapper subshell**, not
the real `npm`→`node` server that holds the port. Every watchdog/cleanup
did a bare `kill -9 $AGENT_PID`, which reaped only the subshell and
**orphaned the real server** (reparented to PID 1, still bound to the
port). The watchdog's "kill agent → container restart → boot-purge"
contract therefore never fired: the frontend kept proxying to a dead
agent → 502 forever.
## Fixes (each with local red-green on the real entrypoint in
`node:22-slim`)
1. **cleanup() EXIT trap** → routes through `_kill_agent_tree` (was
orphaning the agent on every SIGTERM/redeploy).
2. **`_kill_agent_tree`** → `/proc`-based tree-kill with a bounded
re-scan (root killed last) so mid-walk forks can't escape; refuses PID ≤
1 (fail-closed).
3. **size-watchdog** hardened against non-numeric `du` and transient
errors (no silent gate-disable, no permanent loop death).
4. **strands health-watchdog** → 180 s startup-grace window (parity with
langgraph); the now-effective kill would otherwise loop a slow cold
start.
5. **`wait -n` under `set -e`** → capture exit code so the restart
diagnostic isn't dead code on the primary (137) path.
6. **structural:** one `_require_int` validator over *every*
operator-overridable numeric knob (fail-safe to default), and **every**
wrapped-PID kill (incl. `NEXTJS_PID`) routed through the guarded
tree-kill; dangerous `${AGENT_PID:-0}` sentinel removed.
7. **`_require_int`** requires a positive integer (rejects `0` and
leading-zero/octal).
## Incident status
Staging **and** prod LGT were restored immediately via redeploy
(boot-purge cleared the oversized state) — both `/api/health` → 200.
This PR stops the recurrence.
## Review
Converged through a 5-round unbiased review-fix loop (1 + 4
confirmation), zero mandatory findings at close, all load-bearing guards
independently re-verified. `bash -n` + shellcheck (`-S warning`) clean;
170/170 shell bats pass.
## Follow-up (tracked, separate PR — non-load-bearing)
`_require_int` upper-bound clamp (LOW arith-overflow, needs a 20+-digit
value); a stale `cleanup()` comment; size-guard unarmed during the
startup-grace window; SIZE_PID trap-registration micro-window;
diagnostic label on near-simultaneous exit; cosmetic log nits; startup
readiness `sleep 3`+`kill -0` probes the wrapper subshell; no dedicated
Next.js frontend watchdog.
The _require_int validator in the langgraph-typescript and strands-typescript
entrypoints accepted '0' and leading-zero/octal forms like '010'/'08'. Operator
typos on any numeric knob then broke a guard:
- SIZE_THRESHOLD_MB=0 kills the agent on cycle 1 (instant restart loop)
- HEALTH_STRIKE_LIMIT=0 kills on first probe miss
- SIZE_CHECK_INTERVAL=0 / HEALTH_CHECK_INTERVAL=0 busy-spin on 'while sleep 0'
- '010' is read as OCTAL (8) in arithmetic; '08'/'09' abort under set -e
Tighten the predicate to accept only a positive integer with no leading zero
([1-9][0-9]*). Invalid values keep the existing fail-safe behavior: WARN and
fall back to the documented default. Helper stays byte-identical across both
files.
CLASS 1 (guard silently disabled by a bad numeric override): add a reusable
_require_int validator and run it at startup over EVERY operator-overridable
numeric knob in both entrypoints (size threshold/interval, startup grace,
health-probe interval, strike limit). A non-integer/empty override now WARNs
and falls back to the documented default instead of breaking a sleep/loop/
arithmetic test. Closes instance #3 (LANGGRAPH_SIZE_CHECK_INTERVAL='60s'
killing the size-monitor loop on its first iteration).
CLASS 2 (wrapped-PID orphan + kill-0 footgun): route the cleanup() NEXTJS_PID
kill through _kill_agent_tree (it is process-sub-wrapped like the agent, so a
bare kill orphaned the real Next.js node server holding $PORT across redeploy).
Harden _kill_agent_tree and _agent_descendants to refuse a PID that is empty,
non-numeric, 0, or 1 (fail closed), making kill -9 0 / kill -9 1 structurally
impossible. Remove the ${AGENT_PID:-0} sentinel in the --check-size-once seam;
skip with a warning when AGENT_PID is unset instead of defaulting to 0.
Shared helper code kept byte-identical between the two entrypoints.
Both entrypoints run under set -e. The tail `wait -n $AGENT_PID $NEXTJS_PID`
returns non-zero on the PRIMARY designed exit path (137 = size-gate/watchdog
SIGKILL of the agent tree, or an agent crash), so set -e aborted the script AT
that line — making EXIT_CODE=$?, the entire 'which process exited with code N'
diagnostic, and the final `exit $EXIT_CODE` dead code on exactly the
interesting exits. Capture the code with `EXIT_CODE=0; wait -n ... || EXIT_CODE=$?`
so the diagnostic and explicit exit run and preserve the exact code (incl. 137);
the container-restart path is unchanged.
Same class: langgraph's LANGGRAPH_SIZE_THRESHOLD_MB was used in
`[ "$DIR_SIZE_MB" -ge "$threshold" ]` with no numericity guard, so a
non-integer operator override made the test error and silently no-op the size
gate every cycle. Validate the threshold the same way DIR_SIZE_MB already is
(numeric case guard + 'size guard inactive' WARNING, then skip safely).
The health-watchdog armed its 3-strike/~90s kill counter immediately with no
startup-grace window. langgraph-typescript has a 180s grace precisely to keep a
slow cold start from being killed mid-boot into a restart loop. Now that the
tree-kill makes the strands kill effective (the orphan bug previously made it
cosmetic), a slow tsx cold start (>90s) would be genuinely killed and loop.
Port langgraph's grace mechanism verbatim (GRACE=180, no env override, poll
every 5s, exit 0 on agent death during startup, arm anyway if grace elapses),
adapted to strands' :8000/health probe.
Two tightly-related defects in the size-gated restart machinery in
entrypoint.sh:
1. _watchdog_check_size_once validated the du/awk result only for
emptiness, not numericity. A non-integer value (junk du output, a
transient read error, a test-seam stub) reached the
`[ "$DIR_SIZE_MB" -ge ... ]` comparison and threw "integer expression
expected"; sitting inside an `if`, set -e was suppressed so the test
evaluated false and the size gate was SILENTLY skipped with no
warning (unlike the empty-string branch). Now match ^[0-9]+$ via a
case and emit the same "size guard inactive" WARNING, so the gate can
never silently disappear.
2. The size sub-loop used `_watchdog_check_size_once || break`, treating
ANY non-zero (including a transient check error) as a kill and
permanently ending the monitor for the container's lifetime. Now
break ONLY on the real-kill signal (rc==1) — preserving the
kill -> wait -n -> container-restart -> boot-purge contract — while a
transient non-zero keeps the monitor live and re-checks next cycle.
Verified RED->GREEN against the real entrypoint in node:22-slim with a
stubbed du seam: non-numeric du now warns and keeps the gate active; a
transient error no longer permanently disables the loop.
The tree-kill enumerated agent descendants in a single /proc snapshot then
killed. A child that forks a new child (or reparents) between the scan and the
kill escaped the walk, reparented to PID 1, and kept the agent port bound —
defeating the tree-kill's whole purpose of freeing the port before the
container restart.
Replace the single snapshot with a BOUNDED re-scan loop: keep the root alive as
the walk anchor, re-enumerate and SIGKILL live descendants deepest-first each
pass (up to 5 passes, 0.2s apart) until a scan comes back empty, then kill the
root last. Killing the root FIRST would immediately reparent every descendant to
PID 1 and make them unreachable by the root-anchored PPID walk, so root-last is
required for the re-scan to reap late/mid-walk descendants. A descendant that
fully daemonizes (double-fork to PID 1) before we reach it remains out of reach
— documented as an inherent limit of PPID-based reaping without job control; the
agent's npm->node tree does not daemonize.
Also document why the ${stat##*) } PPID parse is safe against a comm containing
") " (longest-prefix to the last ") " always lands on the true terminator).
Applied identically to langgraph-typescript (:8123) and strands-typescript
(:8000). Proven via local RED-GREEN in node:22-slim: pre-fix leaks a mid-walk
escapee (port stays bound), post-fix reaps it (0 orphans, port freed).
The cleanup() EXIT/SIGTERM trap in both langgraph-typescript and
strands-typescript entrypoints did a bare `kill $AGENT_PID`. Because
$AGENT_PID is the outer process-substitution subshell (not the real
npm->node server), this reaped only the subshell and orphaned the node
server (reparented to PID 1, still holding :8123 / :8000) on every
graceful/SIGTERM shutdown -- e.g. every Railway redeploy/rollover.
Route cleanup() through the existing _kill_agent_tree helper (as the
size-watchdog and health-strike kill sites already do), and move the
_agent_descendants/_kill_agent_tree helpers above cleanup()/the trap so
they are defined whenever the trap can first fire.
Brings the 499-commit-stale foundations branch up to date with main so #5761
has a clean diff and no stale reverts (e.g. forwardHeaders). Conflicts:
- CopilotThreadsDrawer.tsx: took main's (main renamed CopilotDrawer -> ThreadsDrawer
+ added the collapse feature; the branch's edit was a no-op import-type split).
- pnpm-lock.yaml: regenerated with the pinned pnpm 10.33.4 (adds @copilotkit/bot-intelligence).
strands-typescript/entrypoint.sh carries the identical latent trap fixed in
langgraph-typescript by this branch. The agent is launched through a process
substitution — `cd /app/src/agent && npm start &> >(awk …) &` — so
$AGENT_PID (=$!) is the outer subshell wrapping that pipeline, NOT the npm→node
tree it forks (`npm start` runs `node --import tsx server.ts`, which stays a
child of npm, not an exec-replacement). The health-strike `kill -9 $AGENT_PID`
therefore reaps only the subshell; npm and node reparent to PID 1 and KEEP
RUNNING, still bound to :8000. `wait -n` never observes the real server die →
container never restarts → the frontend proxies to a dead-but-not-restarted
agent forever (edge 502s). strands-typescript has no size-gate, so this only
fires after the 3-strike (~90s) health counter exhausts, but it is a real
latent footgun with the same root cause.
Fix: reuse the /proc-based `_kill_agent_tree` helper (node:22-slim ships
neither ps nor pgrep, and job control is off so a group kill would take out
the whole entrypoint) at the single health-strike kill site. The whole
npm→node tree now dies, :8000 is freed, `wait -n` returns and the container
restarts.
Red-green proven in a real node:22-slim container against the exact
process-sub → npm → node structure: RED (bare kill -9 $AGENT_PID) leaves node
orphaned and :8000 still LISTENing; GREEN (_kill_agent_tree) reaps the tree,
0 orphans, port freed.
The size-gated watchdog added in ef103f5f5 does `kill -9 $AGENT_PID`
expecting the container to exit and Railway to restart (re-running the
boot-purge). But $AGENT_PID is the process-substitution subshell wrapping
`cd /app/src/agent && npm start &> >(awk …)`, NOT the npm→node tree it
forks. A single-PID kill reaps only that subshell; npm and node reparent
to PID 1 and keep running, still bound to :8123 with the bloated in-memory
state resident. The frontend then proxies to a dead-but-not-restarted
agent and the edge 502s forever.
Fix: add a /proc-based `_kill_agent_tree` (node:22-slim ships neither ps
nor pgrep, and job control is off so a group kill would take out the whole
entrypoint) and use it at both watchdog kill sites (size gate + health
strikes). The whole npm→node tree now dies, :8123 is freed, `wait -n`
returns, the container restarts and re-runs boot-purge — preserving the
original RangeError-prevention intent.
Two bugs fixed:
1. Tool result event path: the `a2ui_operations` container was emitted
inside a `TextMessageContentEvent` block. The A2UI middleware only scans
`TOOL_CALL_RESULT` events for the container, so the card never mounted
and the raw JSON appeared as plain text in the chat. Fixed by emitting a
`ToolCallResultEvent` (matching the claude-sdk-python peer).
2. Operation shape: the ops used the legacy flat form
(`{"type": "create_surface", ...}`) which the renderer silently ignores.
Updated to the v0.9 nested form (`{"version": "v0.9", "createSurface":
{...}}`) used by every other working peer (claude-sdk-python, strands,
google-adk).
Also adds the missing langroid aimock D6 fixture for `gen-ui-a2ui-fixed`
(`display_flight` → tool result → confirmation text) so the D6 probe has
a mock response to drive the full surface-render assertion.
D6 cell: d6:langroid/gen-ui-a2ui-fixed red → green
## Summary
Spring-ai's `DisplayFlightTool` was emitting legacy flat A2UI operations
(`{"type":"create_surface",...}`) but the A2UI middleware expects v0.9
nested operations (`{"version":"v0.9","createSurface":{...}}`). This is
the same flat→nested migration done for Python/TS in #5832 and langroid
in #5839. The flat shape was silently ignored by the middleware, so the
flight card never mounted and the `a2ui-fixed-schema` D6 cell was
permanently red.
**Root cause** (confirmed by prior local red-green on the disproven TS
fix):
`DisplayFlightTool.apply()` emitted `a2ui_operations` in the legacy flat
format. The middleware's `tryParseA2UIOperations` parses the container
correctly, but the op dispatchers inside require the v0.9 shape. No
surface
was ever created → card never mounted.
## Fix
Updated the three operations in `DisplayFlightTool.java` to v0.9 nested
format:
| Before (flat, ignored) | After (v0.9 nested, works) |
|---|---|
| `{"type":"create_surface","surfaceId":...,"catalogId":...}` |
`{"version":"v0.9","createSurface":{"surfaceId":...,"catalogId":...}}` |
| `{"type":"update_components","surfaceId":...,"components":...}` |
`{"version":"v0.9","updateComponents":{"surfaceId":...,"components":...}}`
|
| `{"type":"update_data_model","surfaceId":...,"data":{...}}` |
`{"version":"v0.9","updateDataModel":{"surfaceId":...,"path":"/","value":{...}}}`
|
Shape matches `sdk-python/copilotkit/a2ui.py` and the shared Python
tools.
## Local Red-Green Proof (real control-plane probe, `--rebuild` both
runs)
**RED — original flat ops (`{"type":"create_surface",...}`):**
```
$ showcase test spring-ai:a2ui-fixed-schema --d6 --rebuild --keep
✗ d6:spring-ai/gen-ui-a2ui-fixed red (0.0s)
state=red
⚠ Tests failed for spring-ai:a2ui-fixed-schema (exit 1)
```
**GREEN — v0.9 nested ops
(`{"version":"v0.9","createSurface":{...}}`):**
```
$ showcase test spring-ai:a2ui-fixed-schema --d6 --rebuild --keep
✓ d6:spring-ai/gen-ui-a2ui-fixed green (0.0s)
1 passed
✓ Tests passed for spring-ai:a2ui-fixed-schema
```
## Java Build & Tests
All 64 existing spring-ai JUnit tests pass after the change (`mvn test`:
64 run, 0 failures, 0 errors). Code compiles cleanly (`mvn compile -q`).
## Files Changed
-
`showcase/integrations/spring-ai/src/main/java/com/copilotkit/showcase/springai/tools/DisplayFlightTool.java`
— v0.9 nested op format
The `route.ts` file is unchanged from main (the prior no-op `a2ui: {
injectA2UITool: true }` addition has been reverted — it was disproven as
a fix by a real local red-green).
- Remove unused imports: Iterable, ConversableAgent (from autogen),
AGStreamInput (from autogen.ag_ui.adapter) — none appear in executable
code, only in docstring prose.
- Fix raw_msgs possibly-unbound at dispatch guard: initialize to None
before the try block so the identity check at line 302 is always
safe even if model_dump raises before raw_msgs is assigned. Also
tighten the guard to `raw_msgs is not None` to make the no-normalization
fallback explicit.
autogen.ag_ui import unresolved and LLMConfig(dict) "Expected 0 positional
arguments" are ENVIRONMENT findings — autogen.ag_ui ships only in the
ag2[ag-ui] extra (present in the container, not in local Pyright's venv),
and LLMConfig({...}) is the codebase-wide pattern that works at runtime
with ag2>=0.9 as installed in the container.
AG2's ConversableAgent runs every user message through
``autogen.code_utils.content_str``, which only accepts content-part
types in {"text", "input_text", "image_url", "input_image", "function",
"tool_call", "tool_calls"}. CopilotChat / the AG-UI runtime emits image
and document attachments as the modern shape
{"type": "image" | "document", "source": {...}}
and the demo page's legacy-converter-shim.tsx ALSO appends a legacy
{"type": "binary", mimeType, data | url}
mirror alongside it (to keep the @ag-ui/langgraph converter happy on
LangChain-based integrations — it rides through on the ag2 path too).
Both shapes trip autogen's allowed-types gate with
ValueError("Wrong content format: unknown type image within the
content")
…BEFORE the request reaches the vision model — observed live in the
D6 multimodal probe (commit d8a0a25db, which originally quarantined
the feature as NSF).
Fix
---
Add ``agents/_multimodal_normalize.py``: a ``NormalizingAGUIStream``
subclass of ``AGUIStream`` that overrides ``dispatch()`` to normalize
AG-UI image/document/binary content parts to OpenAI Chat Completions
``image_url`` parts AFTER ``RunAgentInput`` Pydantic parsing and BEFORE
``AgentService`` serialises the messages for autogen.
This is the only correct interception point:
- Too early (ASGI body rewrite before Pydantic): ``RunAgentInput``
rejects ``image_url`` because it is not an AG-UI standard type —
the discriminated union only accepts image/document/binary/text.
- Too late (inside ConversableAgent): requires patching autogen
internals.
The override works by calling ``normalize_messages_for_autogen()`` on
the dict-serialised messages (same form as ``run_stream`` produces via
``model_dump()``) and re-injecting them via a ``_PatchedRunAgentInput``
wrapper that overrides only ``.messages``, delegating all other
attribute access to the original ``RunAgentInput``.
Conversions:
- {"type": "image", "source": {"type": "data", value, mime_type}} →
{"type": "image_url", "image_url": {"url": "data:<mime>;base64,<value>"}}
- {"type": "image", "source": {"type": "url", value}} →
{"type": "image_url", "image_url": {"url": value}}
- {"type": "document", "source": ...} → image_url with the document's
mime preserved (data:application/pdf;base64,...). The vision model
still can't natively read PDFs, but the request reaches the model
instead of being rejected upstream, which is the failure mode this
fix targets.
- {"type": "binary", mimeType, data | url} → image_url (the
legacy-shim parts ride through cleanly).
- {"type": "text", ...} and already-normalised image_url parts pass
through unchanged (identity-preserved on no-op turns).
Failure path: any normalization error is logged at WARNING and the
original messages are forwarded unchanged — autogen's own ValueError
fires verbatim with its error surface intact.
Manifest + fixture
------------------
- showcase/integrations/ag2/manifest.yaml: remove multimodal from
not_supported_features (with its now-stale comment) and add it back
to the features list next to voice.
- showcase/aimock/d6/ag2/multimodal.json: add the D6 fixture pair
using the actual autoPrompt strings from sample-attachment-buttons.tsx
("can you tell me what is in this demo image I just attached" /
"can you tell me what is in this demo pdf I just attached").
TDD evidence (red-green)
------------------------
showcase/integrations/ag2/tests/python/test_multimodal_normalize.py
contains 14 unit tests, pinned at three layers:
1. RED/GREEN against autogen's actual content gate:
* test_autogen_rejects_raw_agui_image_part — confirms
content_str([{type: image, source: ...}]) raises the verbatim
ValueError the D6 probe surfaced. This is the regression pin: if
autogen ever relaxes the gate, this test fails and we know to
revisit the normalizer.
* test_normalized_content_is_accepted_by_autogen — after
normalize_messages_for_autogen(...), content_str accepts every
part and renders "<image>" for the image_url part.
2. Shape coverage: modern image data/url, modern document, legacy
binary data/url, mimeType camelCase alias, plain-text passthrough,
plain-string content, assistant/tool messages untouched,
unrecognised source → text placeholder, idempotency.
3. NormalizingAGUIStream class surface tripwire.
Control-plane D6 RED→GREEN:
RED (no normalizer, pre-fix container): d6:ag2/multimodal → red
(HTTP 500 agent_run_error_event from content_str ValueError)
GREEN (NormalizingAGUIStream applied): d6:ag2/multimodal → green
The DisplayFlightTool was emitting legacy flat A2UI operations
({"type":"create_surface",...}) but the A2UI middleware expects v0.9
nested operations ({"version":"v0.9","createSurface":{...}}).
This is the same flat→nested migration done for Python/TS in #5832 and
langroid in #5839. The flat shape was silently ignored by the middleware,
so the flight card never mounted and the a2ui-fixed-schema D6 cell was
permanently red.
Fix: update the three ops to the v0.9 nested format:
- createSurface (was type:create_surface)
- updateComponents (was type:update_components)
- updateDataModel with value: (was type:update_data_model with data:)
Local red-green proof (real control-plane probe, --rebuild both times):
RED: d6:spring-ai/gen-ui-a2ui-fixed red (flat ops, card never mounts)
GREEN: d6:spring-ai/gen-ui-a2ui-fixed green (v0.9 nested ops, card mounts)
## Summary
- Productizes the Claude SDK Python and TypeScript showcase demos with
LangGraph-parity frontends.
- Wires the Claude demo backends through the official Claude Agent
SDK/AG-UI adapter paths using `claude-sonnet-4.6`.
- Keeps Claude integration docs hidden for this PR and excludes
generated/authored docs artifacts from scope.
## Why
The goal is to bring the productized LangGraph demo surface to Claude
Agents SDKs without publishing integration docs in this pass. This keeps
the PR focused on local showcase demos, runtime behavior, fixtures, and
validation support.
## How
- Ported the demo frontend surfaces and local shell-dojo support for
Claude SDK Python/TypeScript.
- Added official Claude SDK adapter/backend wiring plus real-Claude
local compose support.
- Updated Claude aimock fixtures and validation ratchets for the
expanded demo set.
- Set both Claude manifests to `docs_mode: hidden` and removed docs
setup/snippet artifacts from the PR scope.
## Root Cause
`agno 2.6.20` removed `agno.os.interfaces.agui.utils`. The floating
`agno>=2.5.17` pin in `requirements.txt` caused staging to pull the
breaking version on the next build, causing a startup failure.
## Red-Green Proof
**RED** — with `agno>=2.6.20` installed:
```
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'agno.os.interfaces.agui.utils'
```
**GREEN** — with `agno==2.6.19` installed:
```
GREEN: all 3 symbols OK
```
(Symbols confirmed: `async_stream_agno_response_as_agui_events`,
`extract_agui_user_input`, `validate_agui_state`)
## Changes
- `showcase/integrations/agno/requirements.txt`: pinned `agno>=2.5.17` →
`agno==2.6.19` (exact pin, last version with `agui.utils`)
- `showcase/integrations/agno/src/agent_server.py`: added TODO comment
at line 73 import site noting migration to agno 2.6.20+ API is a
follow-up; no structural changes to imports
## Follow-up
Migration of `agent_server.py` imports to the agno 2.6.20+ API (once the
replacement for `agui.utils` is identified) is tracked in the TODO
comment at line 73.
## Note on CI
The `validate-pins` CI check will likely flag pre-existing non-exact
pins across ~15 other integrations (`openai ^5.9.0`, `crewai` ranges,
etc.). This is pre-existing debt not introduced by this PR.
## What & why
Resolves [OSS-132](https://linear.app/copilotkit/issue/OSS-132).
Investigated with systematic-debugging; every conclusion verified
against the **real** OpenAI Responses API.
**Net change: a TanStack version bump only.** No showcase schema change.
- `@tanstack/ai` `0.18.0` → `0.35.0`
- `@tanstack/ai-openai` `0.9.1` → `0.15.6`
- `package-lock.json` regenerated (Dockerfile uses `npm ci
--legacy-peer-deps`)
## The bug
The built-in-agent showcase 400s on every prompt against real OpenAI.
The state tools (`AGUISendStateSnapshot` / `AGUISendStateDelta` /
`set_steps`) declare arbitrary payloads as `z.any()`, which serializes
to a **typeless** JSON-Schema property (`{ "description": ... }`, no
`"type"`).
The old `@tanstack/openai-base`'s `isStrictModeCompatible()` only
screened for `oneOf/allOf/not/$ref/$defs`, so it missed the missing
`type`, sent the tool with `strict: true`, and OpenAI rejected it:
```
400 Invalid schema for function 'AGUISendStateSnapshot':
In context=('properties','snapshot'), schema must have a 'type' key.
```
This was **masked in production** because the deployed showcase runs
against aimock, which replays fixtures without validating the request
schema — a raw `curl` to prod returns a clean `RUN_FINISHED`, green for
the wrong reason.
The ticket's original framing (zod3/zod4 drift → typeless *root*, `got
"None"`) was already fixed by the zod-4 migration; this is the same
symptom one layer down (typeless *property*).
## The fix is upstream
`@tanstack/ai-openai@0.15.6` (via `@tanstack/openai-base@0.9.2`) fixes
`isStrictModeCompatible`: it now detects typeless / `z.any()` properties
and sends `strict: false`. OpenAI accepts typeless properties under
`strict: false` — so `z.any()` works again with no schema change on our
side.
(`@tanstack/ai-openai@0.15.5` also dropped `@tanstack/ai-client` from
its peerDependencies, so no `ai-client` dep is added.)
## Verification (real OpenAI, gpt-4o)
| Probe | Result |
|---|---|
| Typeless property, `strict: true` (raw OpenAI) | **400** — `schema
must have a 'type' key` |
| Typeless property, `strict: false` (raw OpenAI) | **ACCEPTED** —
confirms it was the strict flag, not the schema |
| `z.any()` tool on old adapter (0.9.1/0.15.4) | adapter sends `strict:
true` → **400** |
| `z.any()` tool on new adapter (0.15.6) | adapter sends **`strict:
false`** → **ACCEPTED**, model calls the tool |
| All 3 `z.any()` state tools attached, new adapter | **ACCEPTED**, no
400 |
## Not covered here
The showcase's aimock + Playwright e2e suite was **not** run locally
(this worktree has no installed toolchain). CI runs it on this PR;
please confirm the gen-ui / shared-state demos still pass before merge.
---
_Branch history shows an interim `z.string()` workaround that was
reverted once the upstream fix shipped; the net diff is the version bump
only. Squash-merge recommended._
agno 2.6.20 removed agno.os.interfaces.agui.utils; the floating
agno>=2.5.17 pin in requirements.txt caused staging to pull the
breaking version. Pinned to 2.6.19 (last version with the module).
Added TODO comment at the import site for future migration.
Gates per-request POST + 2xx Response-status + GET health-probe logs behind SHOWCASE_ROUTE_DEBUG across 19 integrations to stay under Railway's 500-logs/sec cap, while logging non-2xx responses unconditionally so production errors stay visible.
Boot-purge of stale .langgraph_api state plus a size-gated restart (du > threshold -> kill agent -> container restart -> purge), replacing an in-flight-wiping periodic truncate loop. Adds mutation-sensitive subprocess tests for the watchdog.
The ADD-2 block suppresses the streamed TOOL_CALL_CHUNK for all frontend
tools, relying on the bare snapshot's ag_ui_tool_calls to deliver the call
(emitting both doubles the args and breaks scheduleTime / pie-bar
useComponent). But the open-generative-ui runtime middleware builds the
sandboxed iframe exclusively from streamed TOOL_CALL_* events and never
reads the snapshot, so open-gen-ui and open-gen-ui-advanced rendered 0
iframes.
Add a name-scoped exemption that streams the chunk only for
generateSandboxedUi, keeping snapshot-only delivery for every other
frontend tool. The exemption is intentionally narrow to preserve the
double-args fix.
Red->green (D6, --direct, real Docker page):
- open-gen-ui: RED "saw 0 iframe(s), longest srcdoc=0" -> GREEN (iframe + srcdoc)
- open-gen-ui-advanced: RED "selector cascade matched 0 elements" -> GREEN
Regression (all still green): beautiful-chat 5/5 (toggle-theme, pie-chart,
bar-chart, search-flights, schedule-meeting), agentic-chat, mcp-apps.
(cherry picked from commit 03f8d02cedbe737ec83aeefa708a9146f438a904)
The no-tools branch of ReasoningAGUIChatWorkflow.chat streams the answer via
astream_chat over OpenAIResponses, which (unlike astream_chat_with_tools on the
tools branch and the GREEN tool_rendering_reasoning_chain_agent) does not
accumulate resp.delta back onto the terminal resp.message.content. The
content-empty message was then snapshotted into MESSAGES_SNAPSHOT, clobbering
the ~284-char streamed answer and rendering an empty assistant bubble
(reasoning-display failed text-unstable: reasoning block painted, answer gone).
Accumulate the streamed text deltas in the no-tools path only (track_text =
not tools) and fold them onto resp.message before _finalize_chat snapshots it.
Strictly additive: only fills a message the stream left empty, never overwrites
content the LLM already accumulated, and is inert when tools are present so the
tools branch / reasoning-chain agent are untouched.
(cherry picked from commit 46afd0e040a669d14f91b8c136df9c94a91950d0)
The shared FixedAGUIChatWorkflow catch-all dropped request-injected query_notes,
so NotesCard never mounted. Give the cell its own agent (mirrors beautiful_chat_agent)
so request-time frontend tools forward. Verified GREEN via control-plane --direct.
(cherry picked from commit 9c1b8ce2c33afc89000355d83c995f03da208f0f)
Same root cause as the sibling declarative-gen-ui (A2UI Dynamic Schema) fix:
the A2UI middleware mounts the surface from a STREAMED render-tool CALL whose
name is in its watched set, not from a TOOL_CALL_RESULT. The prior approach had
display_flight return an a2ui_operations container in the tool RESULT, which the
llama-index AG-UI adapter only re-emits via MESSAGES_SNAPSHOT — a shape the
middleware never inspects — so the flight-card surface stayed unmounted
(reason=surface-missing; the a2ui-fixed-card testid never appeared).
- route.ts: set a2ui.injectA2UITool: true so the middleware watches render_a2ui.
- a2ui_fixed.py: display_flight now returns the fixed-schema render_a2ui args
(surfaceId/catalogId/components/data) as JSON; a workflow override
(_A2UIRenderToolCallWorkflow) parses each backend tool result and re-emits it
as a streamed render_a2ui tool-CALL (TOOL_CALL_START name=render_a2ui ->
chunked TOOL_CALL_ARGS carrying the components+data JSON -> TOOL_CALL_END),
mirroring how google-adk drives the middleware. These events are already in
the upstream AG_UI_EVENTS allow-list the SSE router streams against.
Backend still produces the pre-authored flight schema (no stub). Only
display_flight (one backend tool, name unchanged) is involved, so the d6 fixture
needs no re-keying. Integration-code only; no shared/@ag-ui package touched.
(cherry picked from commit 74d61eddf7eccf22bcc96c37f0e35a70dd823a2f)
The render re-emit override had three silent failure paths: a non-JSON tool
output (broad except swallowing TypeError/ValueError), the {"error": ...} dict
from generate_a2ui's no-tool-call branch, and a valid-JSON result missing
components. Each produced a blank UI with no diagnostic trail. Narrow the parse
except to json.JSONDecodeError (guarding that content is a str) and log a
contextual warning on each path. Happy path unchanged.
(cherry picked from commit 94b0a69aa4772758e3bcc05f67a6c28b1b0a503d)
Jordan Ritter