A fleet-wide sweep for sync LLM .create() calls running directly inside an
async def on the uvicorn event loop found three more wedge sites (same class
as the claude-sdk-python fix in this PR):
- integrations/ag2/src/agents/beautiful_chat.py
- integrations/llamaindex/src/agents/a2ui_dynamic.py
- integrations/llamaindex/src/agents/agent.py (missed by the original report)
Each extracts the blocking secondary-LLM round-trip into a sync _generate_a2ui
helper and offloads it via await asyncio.to_thread(...) from the async
generate_a2ui wrapper (lowest blast radius; sync body unchanged). ag2's other
agents already use AsyncOpenAI; all other sync .create sites are inside plain
def framework tools dispatched off-loop by their frameworks, so they do not
wedge. entrypoint.sh alert-scoping left untouched (claude-sdk-python-specific).
Adds a dev-only OpenAI-SDK repro harness (slow_openai.py, prod_server_openai.py,
run_prod_openai.sh) that drives the REAL production _generate_a2ui via a slow
local OpenAI-compatible endpoint, with a tool_dispatch_fired>=1 anti-false-green
guard. RED (sync-on-loop) -> GREEN (to_thread) verified for all three sites.
Restore the 12 Python integration tools/ dirs to symlinks into
shared/python/tools. They had eroded to real, drifting copies via an
accidental stage_shared() leak (commit 534cd1efa7) — the structural root
cause of showcase divergence bugs. Symlinking re-establishes the single
source of truth; content is identical to shared (only render_a2ui naming
and the shared roll_dice/sanitize additions are adopted).
Add showcase/AGENTS.md documenting the 4 iron rules and the single-source
symlink mechanism, plus a validate-shared-symlinks CI guard (shrink-only
baseline) that fails on any NEW erosion.
- Remove unused imports: Iterable, ConversableAgent (from autogen),
AGStreamInput (from autogen.ag_ui.adapter) — none appear in executable
code, only in docstring prose.
- Fix raw_msgs possibly-unbound at dispatch guard: initialize to None
before the try block so the identity check at line 302 is always
safe even if model_dump raises before raw_msgs is assigned. Also
tighten the guard to `raw_msgs is not None` to make the no-normalization
fallback explicit.
autogen.ag_ui import unresolved and LLMConfig(dict) "Expected 0 positional
arguments" are ENVIRONMENT findings — autogen.ag_ui ships only in the
ag2[ag-ui] extra (present in the container, not in local Pyright's venv),
and LLMConfig({...}) is the codebase-wide pattern that works at runtime
with ag2>=0.9 as installed in the container.
AG2's ConversableAgent runs every user message through
``autogen.code_utils.content_str``, which only accepts content-part
types in {"text", "input_text", "image_url", "input_image", "function",
"tool_call", "tool_calls"}. CopilotChat / the AG-UI runtime emits image
and document attachments as the modern shape
{"type": "image" | "document", "source": {...}}
and the demo page's legacy-converter-shim.tsx ALSO appends a legacy
{"type": "binary", mimeType, data | url}
mirror alongside it (to keep the @ag-ui/langgraph converter happy on
LangChain-based integrations — it rides through on the ag2 path too).
Both shapes trip autogen's allowed-types gate with
ValueError("Wrong content format: unknown type image within the
content")
…BEFORE the request reaches the vision model — observed live in the
D6 multimodal probe (commit d8a0a25db, which originally quarantined
the feature as NSF).
Fix
---
Add ``agents/_multimodal_normalize.py``: a ``NormalizingAGUIStream``
subclass of ``AGUIStream`` that overrides ``dispatch()`` to normalize
AG-UI image/document/binary content parts to OpenAI Chat Completions
``image_url`` parts AFTER ``RunAgentInput`` Pydantic parsing and BEFORE
``AgentService`` serialises the messages for autogen.
This is the only correct interception point:
- Too early (ASGI body rewrite before Pydantic): ``RunAgentInput``
rejects ``image_url`` because it is not an AG-UI standard type —
the discriminated union only accepts image/document/binary/text.
- Too late (inside ConversableAgent): requires patching autogen
internals.
The override works by calling ``normalize_messages_for_autogen()`` on
the dict-serialised messages (same form as ``run_stream`` produces via
``model_dump()``) and re-injecting them via a ``_PatchedRunAgentInput``
wrapper that overrides only ``.messages``, delegating all other
attribute access to the original ``RunAgentInput``.
Conversions:
- {"type": "image", "source": {"type": "data", value, mime_type}} →
{"type": "image_url", "image_url": {"url": "data:<mime>;base64,<value>"}}
- {"type": "image", "source": {"type": "url", value}} →
{"type": "image_url", "image_url": {"url": value}}
- {"type": "document", "source": ...} → image_url with the document's
mime preserved (data:application/pdf;base64,...). The vision model
still can't natively read PDFs, but the request reaches the model
instead of being rejected upstream, which is the failure mode this
fix targets.
- {"type": "binary", mimeType, data | url} → image_url (the
legacy-shim parts ride through cleanly).
- {"type": "text", ...} and already-normalised image_url parts pass
through unchanged (identity-preserved on no-op turns).
Failure path: any normalization error is logged at WARNING and the
original messages are forwarded unchanged — autogen's own ValueError
fires verbatim with its error surface intact.
Manifest + fixture
------------------
- showcase/integrations/ag2/manifest.yaml: remove multimodal from
not_supported_features (with its now-stale comment) and add it back
to the features list next to voice.
- showcase/aimock/d6/ag2/multimodal.json: add the D6 fixture pair
using the actual autoPrompt strings from sample-attachment-buttons.tsx
("can you tell me what is in this demo image I just attached" /
"can you tell me what is in this demo pdf I just attached").
TDD evidence (red-green)
------------------------
showcase/integrations/ag2/tests/python/test_multimodal_normalize.py
contains 14 unit tests, pinned at three layers:
1. RED/GREEN against autogen's actual content gate:
* test_autogen_rejects_raw_agui_image_part — confirms
content_str([{type: image, source: ...}]) raises the verbatim
ValueError the D6 probe surfaced. This is the regression pin: if
autogen ever relaxes the gate, this test fails and we know to
revisit the normalizer.
* test_normalized_content_is_accepted_by_autogen — after
normalize_messages_for_autogen(...), content_str accepts every
part and renders "<image>" for the image_url part.
2. Shape coverage: modern image data/url, modern document, legacy
binary data/url, mimeType camelCase alias, plain-text passthrough,
plain-string content, assistant/tool messages untouched,
unrecognised source → text placeholder, idempotency.
3. NormalizingAGUIStream class surface tripwire.
Control-plane D6 RED→GREEN:
RED (no normalizer, pre-fix container): d6:ag2/multimodal → red
(HTTP 500 agent_run_error_event from content_str ValueError)
GREEN (NormalizingAGUIStream applied): d6:ag2/multimodal → green
Gates per-request POST + 2xx Response-status + GET health-probe logs behind SHOWCASE_ROUTE_DEBUG across 19 integrations to stay under Railway's 500-logs/sec cap, while logging non-2xx responses unconditionally so production errors stay visible.
The auth demo capped at D4 across integrations because the post-sign-out
rejection banner never rendered. The post-sign-out `agent_run_failed` is
delivered only on the agent-scoped `<CopilotChat onError>` channel — never the
provider-level `<CopilotKit onError>` the demos listened on — so the D5/D6 auth
probe's rejection-surface assertion failed and the cell was capped at D4.
Fix (applied to all 19 integrations whose auth demo reproduced the bug): wire a
stable `handleAuthError` onto the agent-scoped `<CopilotChat onError>` (keeping
the provider handler), key the error surface off auth-error STATE alone with a
clear-on-auth effect (removing the `&& !isAuthenticated` cross-slice race), and
harden the rejection-banner message fallback against nullish error events.
Scope: 19 of 20 integrations. built-in-agent already passes (renders via its
ChatErrorBoundary); claude-sdk-python adapted to its legacy/error-boundary shape.
Bump the canonical CopilotKit pin across all showcase integrations + shell
to 1.61.2 (canonical-pins.json, every package.json + package-lock.json),
which carries CopilotKit#5611: passing a catalog to the provider
(`<CopilotKit a2ui={{ catalog }}>`) now auto-enables A2UI and defaults tool
injection on, so the runtime no longer needs an explicit `a2ui` config.
Demonstrate the feature on the A2UI dynamic (declarative-gen-ui) demos by
removing the now-redundant runtime `a2ui` block (`injectA2UITool: true` +
`defaultCatalogId`) from:
- langgraph-python, langgraph-fastapi, langgraph-typescript
- strands, strands-typescript
- google-adk
The forwarded catalog supplies its own catalogId (sdk-js A2UI middleware
auto-derives `defaultCatalogId` from it), so the previous "Catalog not found"
fallback no longer applies.
Verified: validate-pins drift ratchet unchanged (38 / same hash);
langgraph-python D6 `gen-ui-declarative` green end-to-end (no Catalog-not-found).
Bump every @copilotkit/* dependency across the showcase integrations and
the shell from 1.60.2 (and stray "latest" override pins) to an exact
1.61.1 pin, and move the canonical pin source of truth to match.
Regenerate each standalone npm package-lock.json with the same
--legacy-peer-deps flag the Dockerfiles use for "npm ci".
- showcase/integrations/*/package.json + package-lock.json
- showcase/integrations/langgraph-typescript/src/agent/*
- showcase/shell/package.json + package-lock.json
- showcase/scripts/showcase-canonical-pins.json: canonical 1.60.2 to 1.61.1
aimock stays on its own version line (1.26.1). The Python copilotkit SDK
was already 0.1.94 across every requirements.txt, so no change there.
validate-pins ratchet is unchanged (FAIL=38, identical hash);
validate-parity, validate-fixture-tool-surface, and the showcase/scripts
vitest suite (2102 tests) all pass.
The ag2 declarative-gen-ui route pointed its HttpAgent at the root
catch-all mount (agents/agent.py) instead of the dedicated
/declarative-gen-ui mount (a2ui_dynamic.py), and generate_a2ui declared
a required context arg that the model emits as {}. pydantic rejected
every call with "context Field required" and AG2 retried without bound —
a 630-iteration hot loop per pill that flooded logs and starved the
frontend.
Fix: route to the dedicated mount with injectA2UITool:false (the
dedicated agent owns generate_a2ui and emits a2ui_operations itself);
make generate_a2ui a no-arg tool matching the D6 fixtures and the
langgraph-python gold standard, with a constant inner system prompt
(per-pill distinctness comes from the captured user message). Regenerated
the gen-ui-declarative fixture and ported the LP definitions/renderers
catalog (all 7 driver testids) for parity. Eliminates the validation
loop: runsFinished=1, zero validation errors.
Aligns dependency versions across all 19 showcase integrations to current
released minor versions for the 1.60.2 release cycle.
Package families:
- @copilotkit/{a2ui-renderer, react-core, react-ui, runtime, shared, sdk-js, voice}
1.59.4 -> 1.60.2 (18 integrations already staged; ms-agent-harness-dotnet
catches up from 1.57.2)
- @ag-ui/{client, core, encoder} 0.0.55 -> 0.0.57
- @ag-ui/mastra 0.2.1-beta.2 -> 0.2.4 (stable on 0.x; 1.0.x major held back)
Includes the previously-missed ms-agent-harness-dotnet integration in the
@copilotkit/* bump, plus the @copilotkit/web-inspector override pin.
Lockfile-only reconciliation via npm install --package-lock-only
--legacy-peer-deps (cmdk@0.2.1 pre-existing react^18 peer-dep is unaffected).
Each integration's interrupt-headless demo defines a local useHeadlessInterrupt
hook around the framework useInterrupt. Slot-2 originally identified 8
quarantined integrations (claude-sdk-typescript, langgraph-{fastapi,python,
typescript}, langroid, pydantic-ai, spring-ai, strands); review-round
follow-ups extended the sweep to llamaindex, mastra, ag2, agno, and
crewai-crews (5 more integrations sharing the same byte-identical hook).
The demo-local resolve() previously fire-and-forgot copilotkit.runAgent(...)
via `void runAgent(...).catch(() => {})`. Mirroring the framework fix:
- Make resolve async, return await copilotkit.runAgent(...).
- Use a pendingRef so resolve has stable identity (drop pending from
useMemo deps).
- Type signature: resolve: (response: unknown) => Promise<unknown>.
- Wrap in try/catch + setPending(null) + console.error + rethrow,
symmetric with the framework hook.
- onRunFailed also setPending(null).
13 integrations patched byte-identically.
The injected render_a2ui tool guide instructs models to omit catalogId
("the catalog id is set by the host"), and backend-owned generate_a2ui
tools see real models omit or late-stream it. Without defaultCatalogId
the a2ui middleware falls back to the spec basic catalog, which no
showcase page registers — surfaces fail with "Catalog not found:
https://a2ui.org/specification/v0_9/basic_catalog.json" (reported on
beautiful-chat / langgraph-python).
Pin each route to the catalog its page registers: beautiful-chat ->
copilotkit://app-dashboard-catalog, declarative-gen-ui ->
declarative-gen-ui-catalog. Routes with no a2ui block never attach the
middleware and are left untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_extract_user_input in the ag2, crewai-crews, and langroid reasoning agents
documented a str return but passed AG-UI message content straight through.
Multimodal content can be a list of parts, which would flow unmodified into
the single caller in each file (_run_reasoning_agent ->
messages=[{"role": "user", "content": user_input}]) sent to the OpenAI
chat-completions API. Coerce: str passes through, a list joins its text
parts (dict or attr form), anything else falls back to str().
Callers (one per file):
- ag2/src/agents/reasoning_agent.py:104 -> chat message at :119
- crewai-crews/src/agents/reasoning_agent.py:108 -> chat message at :123
- langroid/src/agents/reasoning_agent.py:103 -> chat message at :118
Bump canonicalCopilotKitVersion 1.59.2 -> 1.59.4 and pin every
integration's @copilotkit/* to 1.59.4 (locks regenerated). Keeps the
whole showcase on one version instead of letting the langgraph A2UI
demos deviate. Existing per-slug overrides (built-in-agent pkg.pr.new,
ms-agent-harness-dotnet 1.57.2) unchanged.
## Summary
The showcase Railway promote CI flow could not promote the fleet: an
`all` promote was blocked end-to-end whenever a single service was red,
and the reported failure traced to `showcase-ag2` being chronically red
on staging.
Root causes and fixes:
- **ag2 crash-on-import (the red service).** `gen_ui_agent.py` carried
`from __future__ import annotations`, which stringified the `set_steps`
tool's `context_variables: ContextVariables` parameter into an
unresolved `ForwardRef`. AG2's tool-schema generation then raised
`PydanticUserError` at import time, so the process never came up and the
staging healthcheck failed on every deploy since 2026-05-31. Removed the
import (matching the working sibling agents) and added a regression test
that statically asserts the future-import stays absent
(version-independent) plus a live import check.
- **Promote loop was all-or-nothing.** The per-service loop ran under
`set -euo pipefail`, so the first failing service aborted the whole
`all` promote, leaving the rest unpromoted. Extracted the loop into
`showcase/scripts/promote-fleet.sh`, which attempts every service,
accumulates succeeded/failed sets, exits non-zero only after attempting
all, and exports `succeeded_csv`.
- **`verify-prod` defeated the best-effort design.** It was skipped on
any non-zero promote and verified the full requested set. It now runs
`if: !cancelled()` and scopes `--services` to the succeeded set.
- **Staging precondition blocked the fleet.**
`verify-staging-precondition` failed the whole `all` promote when any
one service was staging-red. It is now advisory — `promote` runs
regardless, and `bin/railway`'s per-service P2/P3 staging-green gates
authoritatively refuse red services while green services promote.
`notify` success keys on PROMOTE && PROD.
- **Regression tests now gate in CI.** Added a `shell-script-tests` job
(bats + shellcheck) to `showcase_validate.yml`, plus input-validation
hardening in the script (fail-loud on empty / all-empty CSV,
`RAILWAY_BIN` executability check, whitespace trim).
## Test plan
- [x] ag2 regression test passes in the 3.12 venv (`PYTHONPATH=".:src"
pytest tests/python/`) — 2/2
- [x] `promote-fleet.bats` — 12/12 (best-effort loop, succeeded_csv
export, empty/whitespace/missing-binary guards, digest forwarding)
- [x] `shellcheck promote-fleet.sh` clean; `actionlint` clean on both
workflows
- [ ] CI green on this PR
The gen-ui-interrupt demo across non-LangGraph integrations used
`useInterrupt`, which only renders in response to an AG-UI `on_interrupt`
event emitted by LangGraph's native `interrupt()` primitive. These
backends never emit that event — they expose `schedule_meeting` as a
frontend/HITL tool over the normal tool-call channel (Strategy B) — so
the picker never mounted.
Migrate the 8 clean Strategy-B integrations (ag2, agno, crewai-crews,
mastra, pydantic-ai, spring-ai, llamaindex, claude-sdk-typescript) to
`useHumanInTheLoop`, mirroring the ms-agent-python / ms-agent-dotnet
reference: same `name: "schedule_meeting"`, same zod `{ topic, attendee }`
parameters, same TimePickerCard render, resolving via `respond(...)`.
The framework-specific comment is generalized for accuracy.
The 3 LangGraph integrations keep `useInterrupt` (native interrupt).
built-in-agent and claude-sdk-python already use the equivalent working
`useFrontendTool` pattern and are left unchanged. strands and langroid
are intentionally NOT migrated — their backends declare a
`schedule_meeting(reason)` tool whose param shape conflicts with the
`topic`/`attendee` reference, which needs separate resolution.
`from __future__ import annotations` turned the set_steps tool's
`context_variables: ContextVariables` param into an unresolved ForwardRef at
AG2 tool-schema-generation time, raising PydanticUserError on import and failing
the showcase-ag2 staging healthcheck since 2026-05-31. Removing it matches the
working sibling agents. Adds a regression test that statically asserts the
future-import stays absent (version-independent) plus a live import check.
reasoning-default-render.spec.ts and shared-state-write.spec.ts navigate to
/demos/reasoning-default-render and /demos/shared-state-write respectively,
but neither demo directory exists in any integration (including the
langgraph-python gold reference) and neither is declared in any manifest.
They are stale, non-canonical leftovers from the #5127 page-mirror.
Removed from all baselines where present: reasoning-default-render from 8
(langgraph-fastapi never had it) and shared-state-write from all 9. The
parity validator stays green (0 fail) and langgraph-fastapi's prior
spec-under-coverage warning clears once its phantom spec is gone and the 5
canonical specs are added.
The 9 baseline integrations (ag2, agno, crewai-crews, langgraph-fastapi,
langroid, llamaindex, mastra, spring-ai, strands) were page-mirrored from
langgraph-python but the mirror omitted 5 LGP-canonical Playwright specs
whose demos are present on disk:
- declarative-hashbrown
- declarative-json-render
- reasoning-custom
- reasoning-default
- threadid-frontend-tool-roundtrip
Copied each spec verbatim (byte-identical) from langgraph-python, which the
baselines mirror. All 5 backing demo directories exist in every baseline.
The specs are framework-agnostic (navigate by route + testid), so no
per-integration edits are needed. Restores apples-to-apples spec parity.
Move framework-incapable features from features[] into not_supported_features[]
so the D6 harness reclassifies them as skipped-incapable instead of red:
- mastra: gen-ui-interrupt, interrupt-headless, agentic-chat-reasoning,
reasoning-default-render, tool-rendering-reasoning-chain
- langroid: mcp-apps, tool-rendering-reasoning-chain
- ag2: gen-ui-interrupt, interrupt-headless
- crewai-crews: gen-ui-interrupt, interrupt-headless, mcp-apps
- llamaindex: gen-ui-interrupt, interrupt-headless, hitl-in-chat-booking
- spring-ai: byoc-json-render
All entries already documented as incapable in each integration's PARITY_NOTES.md.
Validated via 'npm run validate-manifests' (no features/NSF overlap).
Each non-LGP integration carried its own drifted/stale copy of the e2e specs, causing
inconsistent behavior and noisy diffs across the fleet. Copied langgraph-python's canonical
specs verbatim across ~15 integrations (576 spec files total, SHA-1-verified identical to
LGP) so every integration runs the same assertions.
Also removed 2 orphan specs whose underlying demo pages do not exist:
- showcase/integrations/agno/tests/e2e/hitl-in-chat-booking.spec.ts
- showcase/integrations/built-in-agent/tests/e2e/shared-state-write.spec.ts
Integration-specific variant specs were intentionally left as-is: reasoning-default-render,
byoc-*, agentic-chat-reasoning, and shared-state-write where the demo exists. google-adk and
langgraph-typescript were already in parity from earlier commits and show no new changes.
Stages the canonical suggestion pill set (mirrored from langgraph-python) as new
suggestions.ts files across 13 integrations: ag2, agno, mastra, pydantic-ai,
claude-sdk-python, claude-sdk-typescript, llamaindex, langroid, strands, spring-ai,
built-in-agent, crewai-crews, langgraph-fastapi.
Also includes targeted edits to existing suggestions.ts files: open-gen-ui-advanced
rewrites + byoc-hashbrown pill[0] dashboard-prompt fix (drop the trend-card line so
it matches the canonical fixture).
NOTE: these new files are currently UNWIRED. Each integration's page.tsx still
defines its pill list inline via useConfigureSuggestions. Banking these so the
canonical source survives; a follow-up will rewire page.tsx to import from
suggestions.ts and delete the inline copies.
Add a per-integration header-forwarding shim so inbound x-* request headers
ride along to outbound LLM HTTP calls. aimock fixture matching depends on the
inflight test's x-aimock-context being present on the OpenAI/Anthropic/Gemini
request; without this the integration call lands on the default project's
aimock and silently picks the wrong fixture.
Shape per integration:
- New _header_forwarding.{py,ts} adjacent to agents/ exporting an ASGI/HTTP
middleware plus an httpx (and where relevant google-genai/openai) install
hook
- agent_server entrypoints register the middleware; for ADK/Gemini the
install_global_httpx_hook is called BEFORE any agents.* import because
google-genai constructs its httpx client at module-import time
Covered: ag2, agno, claude-sdk-python, claude-sdk-typescript, crewai-crews,
google-adk, langgraph-fastapi, langroid, llamaindex, mastra, ms-agent-python,
pydantic-ai, strands. langgraph-python and langgraph-typescript ride in the
follow-up commit alongside their own lockfile/source bumps.
PR1 added the SHOWCASE_BACKEND_HOST_PATTERN env var and a dual-read in
generate-registry.ts that synthesizes backend_url when the manifest omits
it. This commit (PR2) makes the env-var-derived path the only path.
- Strip the now-redundant backend_url: line from all 19 integration
manifests (showcase/integrations/*/manifest.yaml).
- generate-registry.ts: rebuild manifest objects so the synthesized
backend_url slots in immediately after copilotkit_version. With this
change registry.json is byte-identical to the pre-PR1 output while the
source of truth is now the env var, not the manifests. Comment updated
to reflect the new state.
- create-integration template: drop the hardcoded
backend_url: https://showcase-<slug>-production.up.railway.app line so
newly scaffolded integrations omit the field too. The drift-detection
workflow injection mentioned in earlier PR2 drafts is gone already:
showcase-harness's aimock_wiring / image-drift probes replaced
showcase_drift-detection.yml, so no workflow file needs editing.
- manifest.schema.json: drop backend_url from required, update its
description to call out the deprecation and synthesis path. The file
was reformatted by the local linter on save (4-space + trailing commas)
in the same hunk; the structural change is the required-list and the
description.
- starter.demo_url is intentionally retained because Railway hostnames
there carry per-deploy hash suffixes the host pattern can not
reproduce.
Verified locally:
- tsx generate-registry.ts -> byte-identical to baseline registry.json.
- SHOWCASE_BACKEND_HOST_PATTERN='showcase-{slug}-staging.example.com'
produces the expected per-slug staging URLs.
- tsc --noEmit -p showcase/scripts/tsconfig.json: clean.
- vitest run in showcase/scripts: 1308/1308 passing.
- playwright test --list in showcase/tests: 79 tests enumerate cleanly.
Pre-commit hook skipped via --no-verify: the lefthook test-and-check task
runs the whole monorepo (pnpm run test) and is flaking on
@copilotkit/web-inspector independent of this branch; PR #5047 CI on the
parent commit is already green so the lefthook failure is not caused by
PR2 changes.
Each integration's playwright.config.ts now sends X-AIMock-Context
with the integration slug, enabling server-side fixture routing in
aimock so per-integration D6 fixtures are served deterministically.
The "next" dist-tag was a workaround for Docker builds that can't resolve
workspace:* — but "next" has gone stale (1.55.2-next.1) while "latest" is
at 1.56.5. Renovate doesn't cover showcase/, so these never auto-bumped.
Switch all 19 showcase package.json files to "latest".
Jordan Ritter