A fleet-wide sweep for sync LLM .create() calls running directly inside an
async def on the uvicorn event loop found three more wedge sites (same class
as the claude-sdk-python fix in this PR):
- integrations/ag2/src/agents/beautiful_chat.py
- integrations/llamaindex/src/agents/a2ui_dynamic.py
- integrations/llamaindex/src/agents/agent.py (missed by the original report)
Each extracts the blocking secondary-LLM round-trip into a sync _generate_a2ui
helper and offloads it via await asyncio.to_thread(...) from the async
generate_a2ui wrapper (lowest blast radius; sync body unchanged). ag2's other
agents already use AsyncOpenAI; all other sync .create sites are inside plain
def framework tools dispatched off-loop by their frameworks, so they do not
wedge. entrypoint.sh alert-scoping left untouched (claude-sdk-python-specific).
Adds a dev-only OpenAI-SDK repro harness (slow_openai.py, prod_server_openai.py,
run_prod_openai.sh) that drives the REAL production _generate_a2ui via a slow
local OpenAI-compatible endpoint, with a tool_dispatch_fired>=1 anti-false-green
guard. RED (sync-on-loop) -> GREEN (to_thread) verified for all three sites.
The claude-sdk-python agent :8000 wedges under D6/LLM load: two synchronous
anthropic.Anthropic().messages.create() calls run directly on the uvicorn
asyncio event loop, freezing it for the full LLM round-trip so /health stops
responding. The watchdog counts 3 consecutive failures (~90s) and kill-restarts
the container, dropping active sessions.
Root cause (sync-in-async), all in integrations/claude-sdk-python/:
- src/agents/agent.py: _execute_tool's generate_a2ui branch builds a sync
anthropic.Anthropic() and calls messages.create() synchronously; invoked on
the loop from run_agent's agentic loop AND from the Claude-Agent-SDK MCP tool
handler in claude_agent_sdk_adapter.py.
- src/agents/a2ui_dynamic.py: _generate_a2ui, same sync pattern, invoked on the
loop from the run_a2ui_dynamic_agent generator.
Fix: wrap every async call site in `await asyncio.to_thread(...)` (lowest blast
radius — the sync functions and the shared ExecuteTool callback type are
unchanged, and the whole tool-dispatch path is fixed uniformly, not just
generate_a2ui):
- agent.py run_agent call site
- claude_agent_sdk_adapter.py MCP tool handler
- a2ui_dynamic.py secondary call site
Blast radius: claude-sdk-python only. a2ui_dynamic.py is per-integration (each
framework has its own copy); every other claude-sdk-python agent already uses
AsyncAnthropic. The `tools` symlink to shared/python was not touched.
entrypoint.sh: drop the Slack alert from the :8000 agent watchdog branch (keep
the kill-restart — it self-heals silently now that the root cause is fixed);
keep the LOUD #oss-alerts page on the public $PORT /api/health branch.
Adds showcase/tests/repro/async-wedge/ — a faithful RED/GREEN harness driving
the real anthropic sync client against a controllable slow endpoint, plus a
mutation guard on the real _generate_a2ui.
Self-activate CVDIAG_LOG_STDOUT=0 when CVDIAG_PB_URL is wired (safe: only silences stdout when PB is receiving; explicit override preserved). Extend the watchdog to poll the public $PORT /api/health and, on sustained failure, POST a loud #oss-alerts Slack alert BEFORE kill-restart; add the same alert to the agent-\:8000 branch (no silent recovery). Add uvicorn --no-access-log to cut the access-log flood. Together these keep the shared log stream under the 500/sec cap so the pipe never backs up.
Shared cvdiag_bootstrap: gate the per-LLM-call breadcrumb capture handler and the emit_cvdiag stdout write behind CVDIAG_LOG_STDOUT (default ON so every other integration is byte-for-byte unchanged; opt-out per service). Enqueue to the non-blocking PocketBase sink BEFORE the stdout write so a wedged fd1 cannot cost the durable breadcrumb. No sampling; full fidelity to PB. Red-green unit tests incl. a hostile-stdout durability test.
## Summary
Companion structural PR to #5971. It fixes the **root cause** behind the
a2ui divergence #5971 patched: the showcase's single-source symlinks
eroded to real, drifting copies.
`showcase/integrations/*/tools`, `*/shared-tools`, `*/_shared` are meant
to be **symlinks into `showcase/shared/...`** — `stage_shared()`
dereferences them for the Docker build, `restore_symlinks()` restores
them. An **accidental `stage_shared()` leak** (commit `534cd1efa7`, PR
#4449 "D5 all-green") committed the dereferenced real files instead of
restoring the symlinks. Once they were real files, they drifted — which
is exactly how the a2ui `render_a2ui` vs `_design_a2ui_surface` split
(fixed in #5971) arose.
## Changes
- **Restore 12 Python `tools/` dirs to symlinks** →
`../../shared/python/tools` (ag2, agno, claude-sdk-python, crewai-crews,
google-adk, langgraph-fastapi, langgraph-python, langroid, llamaindex,
ms-agent-python, pydantic-ai, strands). Content is byte-adopted from
shared — verified no load-bearing per-integration code is lost (only
`render_a2ui` naming + shared `roll_dice`/sanitize additions).
Integrations' intentional internal-planner names (llamaindex,
ms-agent-python) live in `src/`, not `tools/`, and are untouched.
- **`showcase/AGENTS.md` (+ `CLAUDE.md`, root pointers,
INTEGRATION-CHECKLIST section)** — canonical statement of the 4 iron
rules (identical tests, near-identical frontends, minimal backends,
per-integration fixtures) + the single-source symlink mechanism ("edit
the shared source only; a real file there is a bug"). These were
previously written down nowhere.
- **`validate-shared-symlinks` CI guard** — fails on any NEW erosion
(real dir where a symlink belongs), with a shrink-only baseline that
tightens to fully-enforcing as symlinks are restored. Mirrors the
existing `validate-*` ratchet pattern.
## Scope / independence
- **No overlap with #5971** — this PR touches nothing under
`showcase/shared/typescript/` and does not modify the 3 TS integration
`shared-tools/` dirs (verified: empty file-set intersection). Mergeable
independently.
- Build-safe: `stage_shared()` correctly dereferences the restored
symlinks (targets resolve within the build context);
`restore_symlinks()` recreates them post-build.
## Verified
- `validate-shared-symlinks` test suite: 7/7 pass; validator EXIT 0 (no
new erosion).
- Reviewed by a full panel (correctness, content-integrity, build/CI,
docs, scope, silent-failure, simplicity) — zero mandatory findings.
## Follow-ups (deliberately out of scope)
1. **3 TS `shared-tools` dirs** (mastra, claude-sdk-typescript,
langgraph-typescript) remain real (baselined) — symlink them in a
follow-up **after #5971 merges**, to avoid overlapping its TS edits.
2. **Guard hardening**: validate the symlink *target* (not just that
it's a symlink), fail-loud on a malformed baseline, and code-enforce the
shrink-only ratchet. (This PR's guard catches the real-file erosion —
the actual failure mode; these are robustness extras.)
3. Pre-existing `shared/python` a2ui test failures (#5971-adjacent) and
a couple of stale doc line-refs, noted during review.
Companion: #5971.
Restore the 12 Python integration tools/ dirs to symlinks into
shared/python/tools. They had eroded to real, drifting copies via an
accidental stage_shared() leak (commit 534cd1efa7) — the structural root
cause of showcase divergence bugs. Symlinking re-establishes the single
source of truth; content is identical to shared (only render_a2ui naming
and the shared roll_dice/sanitize additions are adopted).
Add showcase/AGENTS.md documenting the 4 iron rules and the single-source
symlink mechanism, plus a validate-shared-symlinks CI guard (shrink-only
baseline) that fails on any NEW erosion.
Flip the 4 TS a2ui builders (shared/typescript + mastra,
claude-sdk-typescript, langgraph-typescript) from the legacy flat
operation shape to v0.9 nested (createSurface / updateComponents /
updateDataModel), matching the Python builder and what A2UI consumers
process. Flat ops were never processed as valid nested operations, so
the surface schema and components were never applied.
Also align the empty-data guard to Python's `if data:` semantics (empty
object -> no updateDataModel), add a v0.9 parity guard test to all 4
test files, and add 12 gen-ui-a2ui-fixed aimock fixtures.
Round-3 CR fixes for the LGT persistence-disable preload.
HIGH-1: after installing the fs-write patches, import the node:fs/promises
namespace and assert each patched member is identity-equal to the installed
function; throw (fail boot) naming any mismatched member. Catches the
load-order case where fs/promises was linked before the reassignment and the
namespace snapshotted the original fn (silent bypass -> disk-growth recurrence).
HIGH-2: make the real-package behavioral test non-skippable under
LGT_REQUIRE_BEHAVIORAL=1 (missing runtime fails, not skips), and wire the
python-unit-tests job to set up Node, npm install the agent deps, and run the
langgraph-typescript pytest with that flag so a green check proves interception.
LOW: tolerant writer-shape guard regex (quote/whitespace/alias agnostic; still
trips on a named-import switch); read-only open/openSync reject ENOENT for
suppressed paths (write-intent still no-ops); mkdir recursive returns the
topmost-created dir per the real fs contract.
Adds a HIGH-1 guard-fires regression test.
Round-2 CR fixes for the LGT file-persistence disable. Empirically verified
against the real @langchain/langgraph-api@1.1.17: the FileSystemPersistence
writer uses namespace fs access (fs.writeFile via import * as fs), so the
CJS .promises patch IS observed — but the round-1 patch covered only
writeFile+mkdir with an unanchored substring, leaving atomic write-then-rename
/ appendFile / *Sync / stream surfaces as silent bypasses.
- Patch every fs write surface (promises + sync + open/openSync/createWriteStream)
for the persist dir, so no future writer shape can silently grow .langgraph_api.
- Anchor path matching to the .langgraph_api path segment (not a substring);
normalise string/Buffer/URL forms — no more over-match data loss.
- Add a version + writer-shape guard that fails loudly at boot if the package
is upgraded or switches to named imports (prevents a silent recurrence).
- Honour mkdir {recursive:true} return contract; accept 1/true/yes/on env
conventions and log on both enabled and not-enabled branches.
- Tests: assert the SHIPPED reaper (_reap_watchdog_children/$BASHPID walk) not
a stale comment-only trap; replace the mock-shape behavioral reaper test with
one that drives the real entrypoint helpers; fix a flaky single-shot poll with
a bounded retry loop; refresh the stale truncate-era docstring; add a
real-package behavioral test proving disk stays empty and in-memory round-trip
still returns an assistant response.
The langgraph-typescript backend's @langchain/langgraph-api FileSystemPersistence
serialises all accumulated thread/run/checkpoint state to .langgraph_api on a
3-second timer. Under the D6 probe fan-out (36 parallel probes) the dir filled
past the 200MB size-watchdog threshold in ~90s, the watchdog killed the agent,
and on rapid restart the D6 cron refilled and re-tripped it until Railway
crash-loop backoff stopped restarting the container (2026-07-13 outage, staging
and prod).
Mirror PR #5825's langgraph-python fix, which exported
LANGGRAPH_DISABLE_FILE_PERSISTENCE=true so the python inmem runtime skips its
flush-to-disk loop. The TS package has no such switch and its persistence
writers are unexported module singletons behind an exports-map wall, so ship a
node --import preload (src/agent/disable-file-persistence.mjs) that, gated on the
same env var, no-ops node:fs/promises writeFile/mkdir for .langgraph_api paths
while leaving in-memory state (the real runtime state) intact. Wire it into
npm start and export the env var in entrypoint.sh.
Behavior preserved: runs still execute and thread state reads back from the
in-memory checkpointer within the container lifetime; only disk persistence is
removed, so the size-watchdog has nothing to fill and never trips under load.
Three wired demos 404'd on load: their pages point <CopilotKit runtimeUrl>
at /api/copilotkit-<demo>, but those route handlers were never created in the
Mastra integration (the pages were mirrored from langgraph-python without
porting the routes). The runtime-info fetch 404'd, so the page never mounted
(runtime_info_fetch_failed).
Add the three dedicated routes, mirroring the proven copilotkit-beautiful-chat
pattern. The two A2UI demos set a2ui.injectA2UITool:false (weatherAgent already
owns generate_a2ui — avoid a double-bind) and pin defaultCatalogId to the
catalog the page registers. agent-config registers the agent id the page
requests (agent-config-demo).
Page-load fix only; full behavioral parity (dedicated Mastra agents) is OSS-381.
Verified: next build compiles all three into the route manifest; POST returns
400 (route resolves) identically to copilotkit-beautiful-chat, vs 404 for a
nonexistent route.
Refs OSS-451
Align the claude-sdk-python and claude-sdk-typescript demo integrations behind
the published quickstarts: move the @region markers used for doc snippet
extraction, add the state-streaming and weather-tool snippet files, and add
setup-doc content. Runtime alignment: the TS agent handlers consistently emit
text/event-stream; the streaming snippets emit a fresh STATE_SNAPSHOT per delta
and drop the undeclared partial-json dependency.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The launch-site comment claimed process substitution leaves $! pointing at
the real node process. That is false and contradicts the file header: $!/
AGENT_PID is the wrapping subshell, and the real npm->node server is a
descendant reached only via the tree-kill (the reason _kill_agent_tree exists).
Rewritten in both langgraph-typescript and strands-typescript entrypoints so no
maintainer reintroduces a bare kill.
Also in langgraph-typescript: clarify that the SIZE_PID kill is a retained
belt-and-suspenders backstop to the $BASHPID PPID-walk (not dead code), and
re-anchor the startup-grace rationale to the real cause (the top-level
@langchain/langgraph-api import cost), since the prod path no longer uses
langgraph-cli dev.
Comment-only; no executable code changed.
The bounded re-scan loop's sleep 0.2 was unguarded. Under set -e on a base image
whose sleep can return non-zero (e.g. a future busybox/Alpine rebase), a failed
sleep would abort the tree-kill mid-walk — root never killed, real npm→node
server left orphaned. Add || true so the walk completes regardless of sleep's
exit status. No behavior change on the current Debian base (coreutils sleep
succeeds). Helper kept byte-identical across both entrypoints.
Both langgraph-typescript and strands-typescript entrypoints inferred which
process exited via a post-hoc kill -0 if/elif after wait -n. That inference is
racy: on a near-simultaneous exit both PIDs are dead by probe time, so the first
kill -0 branch always wins and mislabels the diagnostic (naming the agent when
Next.js actually exited, attaching the wrong code to the wrong name). Use
bash's wait -n -p REAPED_PID (bash >= 5.1; node:22-slim ships 5.2) to capture
the actual reaped PID and key the message off it. Exit code (incl. 137) and the
final exit $EXIT_CODE are preserved; || EXIT_CODE=$? guard is unchanged.
Three related size-watchdog hardening changes in the langgraph entrypoint:
- Trap-order leak window: the size sub-loop was backgrounded (`( … ) &`,
SIZE_PID=$!) BEFORE its reaping `trap … EXIT` was registered, so an outer
SIGTERM landing in that window exited the watchdog subshell with no trap
armed and orphaned the sub-loop (reparented to PID 1, spinning for the
container's life). Arm the reaping trap FIRST, and reap via a $BASHPID
PPID-walk that finds the child regardless of whether SIZE_PID is assigned
yet — no ordering-dependent leak.
- Startup-grace size coverage: the size monitor started only AFTER the
up-to-180s startup-grace loop, leaving the size ceiling unguarded during a
pathological cold boot. Start it BEFORE the grace loop; safe because
_watchdog_check_size_once already fail-closes on every not-yet-ready
condition (agent PID not alive, PERSIST_DIR missing, non-numeric size/
threshold), so early cycles are harmless no-ops until the dir grows.
- cleanup() comment accuracy: corrected the note claiming WATCHDOG_PID
"forks nothing that outlives it" — it DOES fork the size sub-loop; the
bare `kill $WATCHDOG_PID` is safe because the watchdog's own inner EXIT
trap reaps that child, not because it forks nothing.
Proven RED->GREEN on the real entrypoint in node:22-slim: pre-fix the size
sub-loop keeps ticking after the watchdog exits (orphan) and the size monitor
spawns after the grace loop (unguarded); post-fix the sub-loop is reaped (0
ticks) and the monitor runs during grace (size-check fires within the grace
window). --check-size-once seam re-verified under/over threshold.
The numeric-config validator accepted any positive integer (`[1-9][0-9]*`),
so a 20+ digit override overflowed bash's signed-64-bit arithmetic and either
wrapped to a negative/garbage magnitude or aborted the `[ -ge ]` test with
"value too great for base" — which, suppressed to false inside the guard's
`if`, silently disabled the guard for the container's lifetime (the exact
fail-open class this validator exists to prevent).
Add a 10-digit length cap (max 9,999,999,999 — comfortably inside int64,
far above any real interval/threshold/strike knob) checked BEFORE the digit
`case`, since an all-digit 23-char value would otherwise pass validation.
A too-long value now takes the same WARN + fall-back-to-default fail-safe
path as every other bad override. Byte-identical across both entrypoints.
Proven RED->GREEN on the real entrypoints in node:22-slim: pre-fix a 23-digit
value survives validation and `$(( x * 3 ))` yields int64-wrapped garbage;
post-fix it WARNs, clamps to the default, and arithmetic is correct.
The /proc PPID walk forked an awk process for every entry in the process
table on every scan pass. Replace the `echo "${stat##*) }" | awk '{print $2}'`
pipeline with the `read` builtin, which word-splits the post-comm remainder
("STATE PPID PGRP …") on IFS and captures the 2nd field with no subprocess.
Byte-identical across the langgraph-typescript and strands-typescript
entrypoints. Non-behavioral; bash -n + shellcheck --severity=warning clean.
## What & why
Two showcase agent containers (**langgraph-typescript**,
**strands-typescript**) could enter a *running-but-dead* state: Railway
showed the service `● Online` while `/api/health` returned **HTTP 502**.
This took down all 36 LGT dashboard cells on staging **and** prod
(prod's `.langgraph_api` had crossed the 200 MB size-watchdog
threshold).
**Root cause:** the agent is launched via process substitution (`... &>
>(awk …) &`), so `$AGENT_PID` (`=$!`) is the **wrapper subshell**, not
the real `npm`→`node` server that holds the port. Every watchdog/cleanup
did a bare `kill -9 $AGENT_PID`, which reaped only the subshell and
**orphaned the real server** (reparented to PID 1, still bound to the
port). The watchdog's "kill agent → container restart → boot-purge"
contract therefore never fired: the frontend kept proxying to a dead
agent → 502 forever.
## Fixes (each with local red-green on the real entrypoint in
`node:22-slim`)
1. **cleanup() EXIT trap** → routes through `_kill_agent_tree` (was
orphaning the agent on every SIGTERM/redeploy).
2. **`_kill_agent_tree`** → `/proc`-based tree-kill with a bounded
re-scan (root killed last) so mid-walk forks can't escape; refuses PID ≤
1 (fail-closed).
3. **size-watchdog** hardened against non-numeric `du` and transient
errors (no silent gate-disable, no permanent loop death).
4. **strands health-watchdog** → 180 s startup-grace window (parity with
langgraph); the now-effective kill would otherwise loop a slow cold
start.
5. **`wait -n` under `set -e`** → capture exit code so the restart
diagnostic isn't dead code on the primary (137) path.
6. **structural:** one `_require_int` validator over *every*
operator-overridable numeric knob (fail-safe to default), and **every**
wrapped-PID kill (incl. `NEXTJS_PID`) routed through the guarded
tree-kill; dangerous `${AGENT_PID:-0}` sentinel removed.
7. **`_require_int`** requires a positive integer (rejects `0` and
leading-zero/octal).
## Incident status
Staging **and** prod LGT were restored immediately via redeploy
(boot-purge cleared the oversized state) — both `/api/health` → 200.
This PR stops the recurrence.
## Review
Converged through a 5-round unbiased review-fix loop (1 + 4
confirmation), zero mandatory findings at close, all load-bearing guards
independently re-verified. `bash -n` + shellcheck (`-S warning`) clean;
170/170 shell bats pass.
## Follow-up (tracked, separate PR — non-load-bearing)
`_require_int` upper-bound clamp (LOW arith-overflow, needs a 20+-digit
value); a stale `cleanup()` comment; size-guard unarmed during the
startup-grace window; SIZE_PID trap-registration micro-window;
diagnostic label on near-simultaneous exit; cosmetic log nits; startup
readiness `sleep 3`+`kill -0` probes the wrapper subshell; no dedicated
Next.js frontend watchdog.
The _require_int validator in the langgraph-typescript and strands-typescript
entrypoints accepted '0' and leading-zero/octal forms like '010'/'08'. Operator
typos on any numeric knob then broke a guard:
- SIZE_THRESHOLD_MB=0 kills the agent on cycle 1 (instant restart loop)
- HEALTH_STRIKE_LIMIT=0 kills on first probe miss
- SIZE_CHECK_INTERVAL=0 / HEALTH_CHECK_INTERVAL=0 busy-spin on 'while sleep 0'
- '010' is read as OCTAL (8) in arithmetic; '08'/'09' abort under set -e
Tighten the predicate to accept only a positive integer with no leading zero
([1-9][0-9]*). Invalid values keep the existing fail-safe behavior: WARN and
fall back to the documented default. Helper stays byte-identical across both
files.
CLASS 1 (guard silently disabled by a bad numeric override): add a reusable
_require_int validator and run it at startup over EVERY operator-overridable
numeric knob in both entrypoints (size threshold/interval, startup grace,
health-probe interval, strike limit). A non-integer/empty override now WARNs
and falls back to the documented default instead of breaking a sleep/loop/
arithmetic test. Closes instance #3 (LANGGRAPH_SIZE_CHECK_INTERVAL='60s'
killing the size-monitor loop on its first iteration).
CLASS 2 (wrapped-PID orphan + kill-0 footgun): route the cleanup() NEXTJS_PID
kill through _kill_agent_tree (it is process-sub-wrapped like the agent, so a
bare kill orphaned the real Next.js node server holding $PORT across redeploy).
Harden _kill_agent_tree and _agent_descendants to refuse a PID that is empty,
non-numeric, 0, or 1 (fail closed), making kill -9 0 / kill -9 1 structurally
impossible. Remove the ${AGENT_PID:-0} sentinel in the --check-size-once seam;
skip with a warning when AGENT_PID is unset instead of defaulting to 0.
Shared helper code kept byte-identical between the two entrypoints.
Both entrypoints run under set -e. The tail `wait -n $AGENT_PID $NEXTJS_PID`
returns non-zero on the PRIMARY designed exit path (137 = size-gate/watchdog
SIGKILL of the agent tree, or an agent crash), so set -e aborted the script AT
that line — making EXIT_CODE=$?, the entire 'which process exited with code N'
diagnostic, and the final `exit $EXIT_CODE` dead code on exactly the
interesting exits. Capture the code with `EXIT_CODE=0; wait -n ... || EXIT_CODE=$?`
so the diagnostic and explicit exit run and preserve the exact code (incl. 137);
the container-restart path is unchanged.
Same class: langgraph's LANGGRAPH_SIZE_THRESHOLD_MB was used in
`[ "$DIR_SIZE_MB" -ge "$threshold" ]` with no numericity guard, so a
non-integer operator override made the test error and silently no-op the size
gate every cycle. Validate the threshold the same way DIR_SIZE_MB already is
(numeric case guard + 'size guard inactive' WARNING, then skip safely).
The health-watchdog armed its 3-strike/~90s kill counter immediately with no
startup-grace window. langgraph-typescript has a 180s grace precisely to keep a
slow cold start from being killed mid-boot into a restart loop. Now that the
tree-kill makes the strands kill effective (the orphan bug previously made it
cosmetic), a slow tsx cold start (>90s) would be genuinely killed and loop.
Port langgraph's grace mechanism verbatim (GRACE=180, no env override, poll
every 5s, exit 0 on agent death during startup, arm anyway if grace elapses),
adapted to strands' :8000/health probe.
Two tightly-related defects in the size-gated restart machinery in
entrypoint.sh:
1. _watchdog_check_size_once validated the du/awk result only for
emptiness, not numericity. A non-integer value (junk du output, a
transient read error, a test-seam stub) reached the
`[ "$DIR_SIZE_MB" -ge ... ]` comparison and threw "integer expression
expected"; sitting inside an `if`, set -e was suppressed so the test
evaluated false and the size gate was SILENTLY skipped with no
warning (unlike the empty-string branch). Now match ^[0-9]+$ via a
case and emit the same "size guard inactive" WARNING, so the gate can
never silently disappear.
2. The size sub-loop used `_watchdog_check_size_once || break`, treating
ANY non-zero (including a transient check error) as a kill and
permanently ending the monitor for the container's lifetime. Now
break ONLY on the real-kill signal (rc==1) — preserving the
kill -> wait -n -> container-restart -> boot-purge contract — while a
transient non-zero keeps the monitor live and re-checks next cycle.
Verified RED->GREEN against the real entrypoint in node:22-slim with a
stubbed du seam: non-numeric du now warns and keeps the gate active; a
transient error no longer permanently disables the loop.
The tree-kill enumerated agent descendants in a single /proc snapshot then
killed. A child that forks a new child (or reparents) between the scan and the
kill escaped the walk, reparented to PID 1, and kept the agent port bound —
defeating the tree-kill's whole purpose of freeing the port before the
container restart.
Replace the single snapshot with a BOUNDED re-scan loop: keep the root alive as
the walk anchor, re-enumerate and SIGKILL live descendants deepest-first each
pass (up to 5 passes, 0.2s apart) until a scan comes back empty, then kill the
root last. Killing the root FIRST would immediately reparent every descendant to
PID 1 and make them unreachable by the root-anchored PPID walk, so root-last is
required for the re-scan to reap late/mid-walk descendants. A descendant that
fully daemonizes (double-fork to PID 1) before we reach it remains out of reach
— documented as an inherent limit of PPID-based reaping without job control; the
agent's npm->node tree does not daemonize.
Also document why the ${stat##*) } PPID parse is safe against a comm containing
") " (longest-prefix to the last ") " always lands on the true terminator).
Applied identically to langgraph-typescript (:8123) and strands-typescript
(:8000). Proven via local RED-GREEN in node:22-slim: pre-fix leaks a mid-walk
escapee (port stays bound), post-fix reaps it (0 orphans, port freed).
The cleanup() EXIT/SIGTERM trap in both langgraph-typescript and
strands-typescript entrypoints did a bare `kill $AGENT_PID`. Because
$AGENT_PID is the outer process-substitution subshell (not the real
npm->node server), this reaped only the subshell and orphaned the node
server (reparented to PID 1, still holding :8123 / :8000) on every
graceful/SIGTERM shutdown -- e.g. every Railway redeploy/rollover.
Route cleanup() through the existing _kill_agent_tree helper (as the
size-watchdog and health-strike kill sites already do), and move the
_agent_descendants/_kill_agent_tree helpers above cleanup()/the trap so
they are defined whenever the trap can first fire.
Brings the 499-commit-stale foundations branch up to date with main so #5761
has a clean diff and no stale reverts (e.g. forwardHeaders). Conflicts:
- CopilotThreadsDrawer.tsx: took main's (main renamed CopilotDrawer -> ThreadsDrawer
+ added the collapse feature; the branch's edit was a no-op import-type split).
- pnpm-lock.yaml: regenerated with the pinned pnpm 10.33.4 (adds @copilotkit/bot-intelligence).
strands-typescript/entrypoint.sh carries the identical latent trap fixed in
langgraph-typescript by this branch. The agent is launched through a process
substitution — `cd /app/src/agent && npm start &> >(awk …) &` — so
$AGENT_PID (=$!) is the outer subshell wrapping that pipeline, NOT the npm→node
tree it forks (`npm start` runs `node --import tsx server.ts`, which stays a
child of npm, not an exec-replacement). The health-strike `kill -9 $AGENT_PID`
therefore reaps only the subshell; npm and node reparent to PID 1 and KEEP
RUNNING, still bound to :8000. `wait -n` never observes the real server die →
container never restarts → the frontend proxies to a dead-but-not-restarted
agent forever (edge 502s). strands-typescript has no size-gate, so this only
fires after the 3-strike (~90s) health counter exhausts, but it is a real
latent footgun with the same root cause.
Fix: reuse the /proc-based `_kill_agent_tree` helper (node:22-slim ships
neither ps nor pgrep, and job control is off so a group kill would take out
the whole entrypoint) at the single health-strike kill site. The whole
npm→node tree now dies, :8000 is freed, `wait -n` returns and the container
restarts.
Red-green proven in a real node:22-slim container against the exact
process-sub → npm → node structure: RED (bare kill -9 $AGENT_PID) leaves node
orphaned and :8000 still LISTENing; GREEN (_kill_agent_tree) reaps the tree,
0 orphans, port freed.
The size-gated watchdog added in ef103f5f5 does `kill -9 $AGENT_PID`
expecting the container to exit and Railway to restart (re-running the
boot-purge). But $AGENT_PID is the process-substitution subshell wrapping
`cd /app/src/agent && npm start &> >(awk …)`, NOT the npm→node tree it
forks. A single-PID kill reaps only that subshell; npm and node reparent
to PID 1 and keep running, still bound to :8123 with the bloated in-memory
state resident. The frontend then proxies to a dead-but-not-restarted
agent and the edge 502s forever.
Fix: add a /proc-based `_kill_agent_tree` (node:22-slim ships neither ps
nor pgrep, and job control is off so a group kill would take out the whole
entrypoint) and use it at both watchdog kill sites (size gate + health
strikes). The whole npm→node tree now dies, :8123 is freed, `wait -n`
returns, the container restarts and re-runs boot-purge — preserving the
original RangeError-prevention intent.
Two bugs fixed:
1. Tool result event path: the `a2ui_operations` container was emitted
inside a `TextMessageContentEvent` block. The A2UI middleware only scans
`TOOL_CALL_RESULT` events for the container, so the card never mounted
and the raw JSON appeared as plain text in the chat. Fixed by emitting a
`ToolCallResultEvent` (matching the claude-sdk-python peer).
2. Operation shape: the ops used the legacy flat form
(`{"type": "create_surface", ...}`) which the renderer silently ignores.
Updated to the v0.9 nested form (`{"version": "v0.9", "createSurface":
{...}}`) used by every other working peer (claude-sdk-python, strands,
google-adk).
Also adds the missing langroid aimock D6 fixture for `gen-ui-a2ui-fixed`
(`display_flight` → tool result → confirmation text) so the D6 probe has
a mock response to drive the full surface-render assertion.
D6 cell: d6:langroid/gen-ui-a2ui-fixed red → green
## Summary
Spring-ai's `DisplayFlightTool` was emitting legacy flat A2UI operations
(`{"type":"create_surface",...}`) but the A2UI middleware expects v0.9
nested operations (`{"version":"v0.9","createSurface":{...}}`). This is
the same flat→nested migration done for Python/TS in #5832 and langroid
in #5839. The flat shape was silently ignored by the middleware, so the
flight card never mounted and the `a2ui-fixed-schema` D6 cell was
permanently red.
**Root cause** (confirmed by prior local red-green on the disproven TS
fix):
`DisplayFlightTool.apply()` emitted `a2ui_operations` in the legacy flat
format. The middleware's `tryParseA2UIOperations` parses the container
correctly, but the op dispatchers inside require the v0.9 shape. No
surface
was ever created → card never mounted.
## Fix
Updated the three operations in `DisplayFlightTool.java` to v0.9 nested
format:
| Before (flat, ignored) | After (v0.9 nested, works) |
|---|---|
| `{"type":"create_surface","surfaceId":...,"catalogId":...}` |
`{"version":"v0.9","createSurface":{"surfaceId":...,"catalogId":...}}` |
| `{"type":"update_components","surfaceId":...,"components":...}` |
`{"version":"v0.9","updateComponents":{"surfaceId":...,"components":...}}`
|
| `{"type":"update_data_model","surfaceId":...,"data":{...}}` |
`{"version":"v0.9","updateDataModel":{"surfaceId":...,"path":"/","value":{...}}}`
|
Shape matches `sdk-python/copilotkit/a2ui.py` and the shared Python
tools.
## Local Red-Green Proof (real control-plane probe, `--rebuild` both
runs)
**RED — original flat ops (`{"type":"create_surface",...}`):**
```
$ showcase test spring-ai:a2ui-fixed-schema --d6 --rebuild --keep
✗ d6:spring-ai/gen-ui-a2ui-fixed red (0.0s)
state=red
⚠ Tests failed for spring-ai:a2ui-fixed-schema (exit 1)
```
**GREEN — v0.9 nested ops
(`{"version":"v0.9","createSurface":{...}}`):**
```
$ showcase test spring-ai:a2ui-fixed-schema --d6 --rebuild --keep
✓ d6:spring-ai/gen-ui-a2ui-fixed green (0.0s)
1 passed
✓ Tests passed for spring-ai:a2ui-fixed-schema
```
## Java Build & Tests
All 64 existing spring-ai JUnit tests pass after the change (`mvn test`:
64 run, 0 failures, 0 errors). Code compiles cleanly (`mvn compile -q`).
## Files Changed
-
`showcase/integrations/spring-ai/src/main/java/com/copilotkit/showcase/springai/tools/DisplayFlightTool.java`
— v0.9 nested op format
The `route.ts` file is unchanged from main (the prior no-op `a2ui: {
injectA2UITool: true }` addition has been reverted — it was disproven as
a fix by a real local red-green).
- Remove unused imports: Iterable, ConversableAgent (from autogen),
AGStreamInput (from autogen.ag_ui.adapter) — none appear in executable
code, only in docstring prose.
- Fix raw_msgs possibly-unbound at dispatch guard: initialize to None
before the try block so the identity check at line 302 is always
safe even if model_dump raises before raw_msgs is assigned. Also
tighten the guard to `raw_msgs is not None` to make the no-normalization
fallback explicit.
autogen.ag_ui import unresolved and LLMConfig(dict) "Expected 0 positional
arguments" are ENVIRONMENT findings — autogen.ag_ui ships only in the
ag2[ag-ui] extra (present in the container, not in local Pyright's venv),
and LLMConfig({...}) is the codebase-wide pattern that works at runtime
with ag2>=0.9 as installed in the container.
AG2's ConversableAgent runs every user message through
``autogen.code_utils.content_str``, which only accepts content-part
types in {"text", "input_text", "image_url", "input_image", "function",
"tool_call", "tool_calls"}. CopilotChat / the AG-UI runtime emits image
and document attachments as the modern shape
{"type": "image" | "document", "source": {...}}
and the demo page's legacy-converter-shim.tsx ALSO appends a legacy
{"type": "binary", mimeType, data | url}
mirror alongside it (to keep the @ag-ui/langgraph converter happy on
LangChain-based integrations — it rides through on the ag2 path too).
Both shapes trip autogen's allowed-types gate with
ValueError("Wrong content format: unknown type image within the
content")
…BEFORE the request reaches the vision model — observed live in the
D6 multimodal probe (commit d8a0a25db, which originally quarantined
the feature as NSF).
Fix
---
Add ``agents/_multimodal_normalize.py``: a ``NormalizingAGUIStream``
subclass of ``AGUIStream`` that overrides ``dispatch()`` to normalize
AG-UI image/document/binary content parts to OpenAI Chat Completions
``image_url`` parts AFTER ``RunAgentInput`` Pydantic parsing and BEFORE
``AgentService`` serialises the messages for autogen.
This is the only correct interception point:
- Too early (ASGI body rewrite before Pydantic): ``RunAgentInput``
rejects ``image_url`` because it is not an AG-UI standard type —
the discriminated union only accepts image/document/binary/text.
- Too late (inside ConversableAgent): requires patching autogen
internals.
The override works by calling ``normalize_messages_for_autogen()`` on
the dict-serialised messages (same form as ``run_stream`` produces via
``model_dump()``) and re-injecting them via a ``_PatchedRunAgentInput``
wrapper that overrides only ``.messages``, delegating all other
attribute access to the original ``RunAgentInput``.
Conversions:
- {"type": "image", "source": {"type": "data", value, mime_type}} →
{"type": "image_url", "image_url": {"url": "data:<mime>;base64,<value>"}}
- {"type": "image", "source": {"type": "url", value}} →
{"type": "image_url", "image_url": {"url": value}}
- {"type": "document", "source": ...} → image_url with the document's
mime preserved (data:application/pdf;base64,...). The vision model
still can't natively read PDFs, but the request reaches the model
instead of being rejected upstream, which is the failure mode this
fix targets.
- {"type": "binary", mimeType, data | url} → image_url (the
legacy-shim parts ride through cleanly).
- {"type": "text", ...} and already-normalised image_url parts pass
through unchanged (identity-preserved on no-op turns).
Failure path: any normalization error is logged at WARNING and the
original messages are forwarded unchanged — autogen's own ValueError
fires verbatim with its error surface intact.
Manifest + fixture
------------------
- showcase/integrations/ag2/manifest.yaml: remove multimodal from
not_supported_features (with its now-stale comment) and add it back
to the features list next to voice.
- showcase/aimock/d6/ag2/multimodal.json: add the D6 fixture pair
using the actual autoPrompt strings from sample-attachment-buttons.tsx
("can you tell me what is in this demo image I just attached" /
"can you tell me what is in this demo pdf I just attached").
TDD evidence (red-green)
------------------------
showcase/integrations/ag2/tests/python/test_multimodal_normalize.py
contains 14 unit tests, pinned at three layers:
1. RED/GREEN against autogen's actual content gate:
* test_autogen_rejects_raw_agui_image_part — confirms
content_str([{type: image, source: ...}]) raises the verbatim
ValueError the D6 probe surfaced. This is the regression pin: if
autogen ever relaxes the gate, this test fails and we know to
revisit the normalizer.
* test_normalized_content_is_accepted_by_autogen — after
normalize_messages_for_autogen(...), content_str accepts every
part and renders "<image>" for the image_url part.
2. Shape coverage: modern image data/url, modern document, legacy
binary data/url, mimeType camelCase alias, plain-text passthrough,
plain-string content, assistant/tool messages untouched,
unrecognised source → text placeholder, idempotency.
3. NormalizingAGUIStream class surface tripwire.
Control-plane D6 RED→GREEN:
RED (no normalizer, pre-fix container): d6:ag2/multimodal → red
(HTTP 500 agent_run_error_event from content_str ValueError)
GREEN (NormalizingAGUIStream applied): d6:ag2/multimodal → green
The DisplayFlightTool was emitting legacy flat A2UI operations
({"type":"create_surface",...}) but the A2UI middleware expects v0.9
nested operations ({"version":"v0.9","createSurface":{...}}).
This is the same flat→nested migration done for Python/TS in #5832 and
langroid in #5839. The flat shape was silently ignored by the middleware,
so the flight card never mounted and the a2ui-fixed-schema D6 cell was
permanently red.
Fix: update the three ops to the v0.9 nested format:
- createSurface (was type:create_surface)
- updateComponents (was type:update_components)
- updateDataModel with value: (was type:update_data_model with data:)
Local red-green proof (real control-plane probe, --rebuild both times):
RED: d6:spring-ai/gen-ui-a2ui-fixed red (flat ops, card never mounts)
GREEN: d6:spring-ai/gen-ui-a2ui-fixed green (v0.9 nested ops, card mounts)
## Summary
- Productizes the Claude SDK Python and TypeScript showcase demos with
LangGraph-parity frontends.
- Wires the Claude demo backends through the official Claude Agent
SDK/AG-UI adapter paths using `claude-sonnet-4.6`.
- Keeps Claude integration docs hidden for this PR and excludes
generated/authored docs artifacts from scope.
## Why
The goal is to bring the productized LangGraph demo surface to Claude
Agents SDKs without publishing integration docs in this pass. This keeps
the PR focused on local showcase demos, runtime behavior, fixtures, and
validation support.
## How
- Ported the demo frontend surfaces and local shell-dojo support for
Claude SDK Python/TypeScript.
- Added official Claude SDK adapter/backend wiring plus real-Claude
local compose support.
- Updated Claude aimock fixtures and validation ratchets for the
expanded demo set.
- Set both Claude manifests to `docs_mode: hidden` and removed docs
setup/snippet artifacts from the PR scope.
## Root Cause
`agno 2.6.20` removed `agno.os.interfaces.agui.utils`. The floating
`agno>=2.5.17` pin in `requirements.txt` caused staging to pull the
breaking version on the next build, causing a startup failure.
## Red-Green Proof
**RED** — with `agno>=2.6.20` installed:
```
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'agno.os.interfaces.agui.utils'
```
**GREEN** — with `agno==2.6.19` installed:
```
GREEN: all 3 symbols OK
```
(Symbols confirmed: `async_stream_agno_response_as_agui_events`,
`extract_agui_user_input`, `validate_agui_state`)
## Changes
- `showcase/integrations/agno/requirements.txt`: pinned `agno>=2.5.17` →
`agno==2.6.19` (exact pin, last version with `agui.utils`)
- `showcase/integrations/agno/src/agent_server.py`: added TODO comment
at line 73 import site noting migration to agno 2.6.20+ API is a
follow-up; no structural changes to imports
## Follow-up
Migration of `agent_server.py` imports to the agno 2.6.20+ API (once the
replacement for `agui.utils` is identified) is tracked in the TODO
comment at line 73.
## Note on CI
The `validate-pins` CI check will likely flag pre-existing non-exact
pins across ~15 other integrations (`openai ^5.9.0`, `crewai` ranges,
etc.). This is pre-existing debt not introduced by this PR.
## What & why
Resolves [OSS-132](https://linear.app/copilotkit/issue/OSS-132).
Investigated with systematic-debugging; every conclusion verified
against the **real** OpenAI Responses API.
**Net change: a TanStack version bump only.** No showcase schema change.
- `@tanstack/ai` `0.18.0` → `0.35.0`
- `@tanstack/ai-openai` `0.9.1` → `0.15.6`
- `package-lock.json` regenerated (Dockerfile uses `npm ci
--legacy-peer-deps`)
## The bug
The built-in-agent showcase 400s on every prompt against real OpenAI.
The state tools (`AGUISendStateSnapshot` / `AGUISendStateDelta` /
`set_steps`) declare arbitrary payloads as `z.any()`, which serializes
to a **typeless** JSON-Schema property (`{ "description": ... }`, no
`"type"`).
The old `@tanstack/openai-base`'s `isStrictModeCompatible()` only
screened for `oneOf/allOf/not/$ref/$defs`, so it missed the missing
`type`, sent the tool with `strict: true`, and OpenAI rejected it:
```
400 Invalid schema for function 'AGUISendStateSnapshot':
In context=('properties','snapshot'), schema must have a 'type' key.
```
This was **masked in production** because the deployed showcase runs
against aimock, which replays fixtures without validating the request
schema — a raw `curl` to prod returns a clean `RUN_FINISHED`, green for
the wrong reason.
The ticket's original framing (zod3/zod4 drift → typeless *root*, `got
"None"`) was already fixed by the zod-4 migration; this is the same
symptom one layer down (typeless *property*).
## The fix is upstream
`@tanstack/ai-openai@0.15.6` (via `@tanstack/openai-base@0.9.2`) fixes
`isStrictModeCompatible`: it now detects typeless / `z.any()` properties
and sends `strict: false`. OpenAI accepts typeless properties under
`strict: false` — so `z.any()` works again with no schema change on our
side.
(`@tanstack/ai-openai@0.15.5` also dropped `@tanstack/ai-client` from
its peerDependencies, so no `ai-client` dep is added.)
## Verification (real OpenAI, gpt-4o)
| Probe | Result |
|---|---|
| Typeless property, `strict: true` (raw OpenAI) | **400** — `schema
must have a 'type' key` |
| Typeless property, `strict: false` (raw OpenAI) | **ACCEPTED** —
confirms it was the strict flag, not the schema |
| `z.any()` tool on old adapter (0.9.1/0.15.4) | adapter sends `strict:
true` → **400** |
| `z.any()` tool on new adapter (0.15.6) | adapter sends **`strict:
false`** → **ACCEPTED**, model calls the tool |
| All 3 `z.any()` state tools attached, new adapter | **ACCEPTED**, no
400 |
## Not covered here
The showcase's aimock + Playwright e2e suite was **not** run locally
(this worktree has no installed toolchain). CI runs it on this PR;
please confirm the gen-ui / shared-state demos still pass before merge.
---
_Branch history shows an interim `z.string()` workaround that was
reverted once the upstream fix shipped; the net diff is the version bump
only. Squash-merge recommended._
agno 2.6.20 removed agno.os.interfaces.agui.utils; the floating
agno>=2.5.17 pin in requirements.txt caused staging to pull the
breaking version. Pinned to 2.6.19 (last version with the module).
Added TODO comment at the import site for future migration.
Gates per-request POST + 2xx Response-status + GET health-probe logs behind SHOWCASE_ROUTE_DEBUG across 19 integrations to stay under Railway's 500-logs/sec cap, while logging non-2xx responses unconditionally so production errors stay visible.
Jordan Ritter