A fleet-wide sweep for sync LLM .create() calls running directly inside an
async def on the uvicorn event loop found three more wedge sites (same class
as the claude-sdk-python fix in this PR):
- integrations/ag2/src/agents/beautiful_chat.py
- integrations/llamaindex/src/agents/a2ui_dynamic.py
- integrations/llamaindex/src/agents/agent.py (missed by the original report)
Each extracts the blocking secondary-LLM round-trip into a sync _generate_a2ui
helper and offloads it via await asyncio.to_thread(...) from the async
generate_a2ui wrapper (lowest blast radius; sync body unchanged). ag2's other
agents already use AsyncOpenAI; all other sync .create sites are inside plain
def framework tools dispatched off-loop by their frameworks, so they do not
wedge. entrypoint.sh alert-scoping left untouched (claude-sdk-python-specific).
Adds a dev-only OpenAI-SDK repro harness (slow_openai.py, prod_server_openai.py,
run_prod_openai.sh) that drives the REAL production _generate_a2ui via a slow
local OpenAI-compatible endpoint, with a tool_dispatch_fired>=1 anti-false-green
guard. RED (sync-on-loop) -> GREEN (to_thread) verified for all three sites.
The claude-sdk-python agent :8000 wedges under D6/LLM load: two synchronous
anthropic.Anthropic().messages.create() calls run directly on the uvicorn
asyncio event loop, freezing it for the full LLM round-trip so /health stops
responding. The watchdog counts 3 consecutive failures (~90s) and kill-restarts
the container, dropping active sessions.
Root cause (sync-in-async), all in integrations/claude-sdk-python/:
- src/agents/agent.py: _execute_tool's generate_a2ui branch builds a sync
anthropic.Anthropic() and calls messages.create() synchronously; invoked on
the loop from run_agent's agentic loop AND from the Claude-Agent-SDK MCP tool
handler in claude_agent_sdk_adapter.py.
- src/agents/a2ui_dynamic.py: _generate_a2ui, same sync pattern, invoked on the
loop from the run_a2ui_dynamic_agent generator.
Fix: wrap every async call site in `await asyncio.to_thread(...)` (lowest blast
radius — the sync functions and the shared ExecuteTool callback type are
unchanged, and the whole tool-dispatch path is fixed uniformly, not just
generate_a2ui):
- agent.py run_agent call site
- claude_agent_sdk_adapter.py MCP tool handler
- a2ui_dynamic.py secondary call site
Blast radius: claude-sdk-python only. a2ui_dynamic.py is per-integration (each
framework has its own copy); every other claude-sdk-python agent already uses
AsyncAnthropic. The `tools` symlink to shared/python was not touched.
entrypoint.sh: drop the Slack alert from the :8000 agent watchdog branch (keep
the kill-restart — it self-heals silently now that the root cause is fixed);
keep the LOUD #oss-alerts page on the public $PORT /api/health branch.
Adds showcase/tests/repro/async-wedge/ — a faithful RED/GREEN harness driving
the real anthropic sync client against a controllable slow endpoint, plus a
mutation guard on the real _generate_a2ui.
Faithful node:22-slim repro: fd1 through the same awk process-substitution as entrypoint.sh, a Railway-capped drain reader, a uvicorn+CVDIAG-shaped flood, and the static no-log /api/health as victim. RED wedges (200->502, CPU->0, heartbeat frozen); the FIXED lane stays 200 throughout. run.sh asserts the outcome (exit 3/4/5 on a false result, proven). watchdog.sh runs the entrypoint public-guard loop verbatim and needle-anchors it against entrypoint.sh.
Close-out proof for Option B: load each public shell (shell, shell-docs,
shell-dashboard) on staging and prod and assert that
(a) the inlined `window.__SHOWCASE_CONFIG__` matches the env's
expected URL set (per-env value, not a leaked default), and
(b) every backend fetch host matches a tight per-env allowlist —
anchored regexes pinned to the EXACT hosts captured from real
page-load inventories (Railway public domains for the staging
services, bare-domain copilotkit.ai hosts for prod, plus the
shared third-party allowlist for analytics/fonts/HubSpot/Reo/
REB2B/scarf/CDN).
An env-leak (someone re-bakes a URL into the artifact) shows up as
either the wrong `__SHOWCASE_CONFIG__` value OR a request to the
other-env's host — either branch fails the test.
This is the test referenced in plan-B B14 / spec §10 items 2,3,10.
Runs against LIVE deployments — no webServer block, fetches public
URLs. Gated to run in CI after the B15 Railway env-var wiring deploy
has settled.
Scoping notes:
- Spec file at `showcase/tests/env-routing.spec.ts`. The existing
`showcase/tests/playwright.config.ts` is the integrations smoke
harness (`testDir: ./e2e`); creating a separate, narrowly-scoped
config at `showcase/playwright.env-routing.config.ts` keeps the
two suites independent so neither can pull the other in by
accident under `playwright test`.
- `testMatch: /env-routing\.spec\.ts$/` belt-and-suspenders the
`testDir: ./tests` selection.
- Verified well-formed locally via `tsc --noEmit` against
`showcase/shell-dashboard/`'s @playwright/test + @types/node
install (the only place those deps are installed in the worktree;
`showcase/tests/` has no node_modules in this worktree because
npm install is symlinked-only by the blitz harness). Full
browser execution requires deployed shells and is gated to
post-deploy in CI.
PR1 of 3 toward removing repo-baked Railway hostnames from showcase
integration manifests.
generate-registry.ts now reads SHOWCASE_BACKEND_HOST_PATTERN
(default: showcase-{slug}-production.up.railway.app). For each
manifest, backend_url falls back to the synthesized value only when
the manifest omits it. Every manifest currently sets backend_url
explicitly, so the synthesized path is unreachable in production data
and the emitted registry.json is byte-identical to the previous output
(verified via diff against pre-change generation).
integration-smoke.spec.ts honors SHOWCASE_BACKEND_HOST_PATTERN at
runtime: when set, each integration's backendUrl is recomputed from the
pattern so a single deployed smoke image can be re-pointed at a
different backend environment without regenerating registry.json.
LOCAL_PORTS=1 still takes precedence. Behavior with no env var is
identical to before.
No behavior change. Forward-compatible with PR2 (drop backend_url from
manifests so the synthesis becomes the source of truth).
Replace single-file volume mounts with directory mounts (shared/, d4/,
d6/) so new fixture files are picked up without editing compose files.
Remove --proxy-only flag to catch fixture gaps instead of silently
falling through to real API. Add LANGGRAPH_HTTP env var from main.
starter-smoke @interaction branches on `starter.hasAppMode`: when true it
clicks the App/Chat toggle and asserts the canvas + textarea; when false
it assumes a CopilotSidebar layout and asserts "Popup Assistant".
Strands now renders the same showcase shell as langgraph-python (App/Chat
toggle + ExampleCanvas), so the smoke test must take the `hasAppMode:
true` branch. Without this flag the test waits for "Popup Assistant" text
that never appears and times out.
The showcase-starter-* Railway services are being deprovisioned after
the packages/starters integration merge. Delete the dedicated smoke
workflow (test_smoke-starter-deployed.yml) and the "Deployed Starters"
describe block from integration-smoke.spec.ts to prevent 17 false-red
alerts per 6-hour cron cycle.
The showcase framework directories better reflect their role as
integration examples rather than distributable packages.
Renames showcase/packages/ -> showcase/integrations/ and updates
the test docker-compose file reference accordingly.
Replace the 200-line hardcoded INTEGRATIONS array in integration-smoke.spec.ts
with a 6-line derivation from registry.json. New demos automatically appear in
smoke tests when manifests are updated — no manual maintenance needed.
## Summary
- Syncs `deployed` flags in `integration-smoke.spec.ts` INTEGRATIONS
array with the source-of-truth `registry.json`
- 4 integrations were stale `deployed: false` but `true` in registry:
**claude-sdk-python**, **claude-sdk-typescript**, **langroid**,
**spring-ai**
- These integrations were silently skipped by the smoke suite
## Test plan
- [x] `npx prettier --check` passes
- [x] `npx playwright test --list` shows all 17 integrations in every
test level
- [ ] CI green
claude-sdk-python, claude-sdk-typescript, langroid, and spring-ai
were marked deployed: false in the smoke test INTEGRATIONS array
but deployed: true in the source-of-truth registry.json. This
caused the smoke suite to skip these four integrations entirely.
- deploy workflow: add shared/scripts/manifest paths to shell-dashboard
and shell-docs filters (previously triggered implicitly by committed
JSON diffs in those directories)
- capture-previews: add generate-registry step before capture; use
git add -f for the gitignored registry.json
- e2e smoke test: document generator dependency in import comment
The Deployed Starters smoke test in integration-smoke.spec.ts uses
checkHealth() with its default paths list ["/api/health", "/health"].
All deployed starters serve their health endpoint at /api/health
(standard Next.js convention); /health returns a Next.js 404 HTML
page. When /api/health returns a legitimate 5xx (e.g. 503
"agent degraded"), the fallback silently probes /health and the
resulting 404 becomes the reported failure — so the Slack alert
reads "status=404 path=/health" and implies a route-missing bug,
hiding the real upstream 503.
Shape A fix: change the default to ["/api/health"] only. No
fallback. The reported status and body are now the actual
/api/health response.
starter-smoke.spec.ts passes an explicit paths argument and is
unaffected. The integration backend L1 health check also targets
/api/health on all deployed backends (verified).
Evidence runs (all same failure mode):
- https://github.com/CopilotKit/CopilotKit/actions/runs/24619147643
- https://github.com/CopilotKit/CopilotKit/actions/runs/24618944101
- https://github.com/CopilotKit/CopilotKit/actions/runs/24617315522
## Summary
Adds a one-command local smoke harness so the full 17-integration suite
can be exercised against Docker on the dev machine instead of Railway.
Useful when Railway is degraded (aimock OOM, rate limits, cold-start
drift) or when validating changes that haven't been deployed yet.
## Usage
```bash
# one-time
cp showcase/.env.example showcase/.env # fill in keys
pnpm --filter @showcase/e2e-smoke install
# full L1-L4 smoke
pnpm --filter @showcase/e2e-smoke smoke:local
# single level / keep containers up between runs
pnpm --filter @showcase/e2e-smoke smoke:local:L1
pnpm --filter @showcase/e2e-smoke smoke:local:keep
pnpm --filter @showcase/e2e-smoke smoke:local:nobuild
```
## What's in here
- **`docker-compose.local.yml`**: `aimock` added as 18th service →
integration containers reach `http://aimock:4010` on the compose
network, mirroring Railway's `showcase-aimock`.
- **`integration-smoke.spec.ts`**: `LOCAL_PORTS=1` env gates URL
rewriting from `https://showcase-<slug>-production.up.railway.app` →
`http://localhost:<port>` via `shared/local-ports.json`. Starters are
skipped under the flag because they're not in `local-ports.json`.
- **`scripts/smoke-local.sh`**: thin orchestrator — `build → up → wait
20s → playwright → down`. Flags: `--level=L1|L2|L3|L4`, `--keep`,
`--no-build`.
- **`tests/package.json`**: `pnpm smoke:local[:L1|:keep|:nobuild]`
wrappers.
- **`.env.example`**: documents optional
`OPENAI_BASE_URL`/`ANTHROPIC_BASE_URL` + `GitHubToken` (ms-agent-dotnet)
and `GOOGLE_API_KEY` (google-adk).
## Verification
Run locally against a fresh checkout of this branch:
- `LOCAL_PORTS=1 SMOKE_ALL=true npx playwright test integration-smoke
--grep @health` → **17/17 pass in 478ms**
- Full L1-L4 against the local stack → **42/51 pass** (9 failures in
L3/L4 for mastra, google-adk, ms-agent-dotnet, strands, langroid,
spring-ai — these are test-data / fixture gaps unrelated to this
infrastructure and will be filed separately)
- `docker compose -f showcase/docker-compose.local.yml config` validates
with 18 services
## Scope
Pure dev-ergonomics addition. No runtime behaviour changes in the
shipped containers. `LOCAL_PORTS` is opt-in; unset = existing
Railway-URL behaviour preserved.
## Test plan
- [ ] `Validate Showcase` CI still green (no package-source changes)
- [ ] No unrelated CI regressions
- [ ] Follow-up PR will investigate and fix the 9 L3/L4 failures
surfaced by local smoke
Wires up a single-command path to run the full integration smoke suite
against a local Docker stack instead of Railway. Useful when Railway is
degraded (OOM, rate limits) or when testing changes that haven't been
deployed yet.
Additions:
- showcase/docker-compose.local.yml: add `aimock` as 18th service so
integration containers can reach http://aimock:4010 on the compose
network, mirroring the Railway setup where they call showcase-aimock.
- showcase/tests/e2e/integration-smoke.spec.ts: `LOCAL_PORTS=1` env
rewrites each integration's Railway URL to http://localhost:<port>
via showcase/shared/local-ports.json. Starters are skipped under this
flag because they aren't in local-ports.json.
- showcase/scripts/smoke-local.sh: orchestrates build → up → wait → run
Playwright → tear down. Supports --level=L1/L2/L3/L4, --keep, --no-build.
- showcase/tests/package.json: `pnpm smoke:local[:L1|:keep|:nobuild]`
scripts delegate to the helper.
- showcase/.env.example: document optional OPENAI_BASE_URL +
ANTHROPIC_BASE_URL (route through local aimock) and package-specific
GitHubToken + GOOGLE_API_KEY (ms-agent-dotnet, google-adk).
Verified locally: `pnpm smoke:local:L1` → 17/17 L1 green against the
local stack.
The STARTERS filter in integration-smoke.spec.ts gated on
`i.starter?.deployed === true`, but the manifest→registry bundler
(showcase/scripts/bundle-demo-content.ts + generate-registry.ts)
does not carry that field forward from the source YAML manifests.
At registry regen time (commit 14537d8f3), all `starter.deployed`
values were dropped.
Result: the filter matched zero entries, `Deployed Starters`
yielded no tests, and the starter-deployed-smoke CI workflow has
failed on every run (>15 consecutive reds starting 2026-04-18) with
Playwright erroring `No tests found.` against
`--grep "@starter-health|@starter-agent|@starter-chat"`.
Switch to integration-level `i.deployed`, which is the single
deployment flag the registry actually carries and which the
INTEGRATIONS array in the same file already uses for its own
gating. This keeps both filters anchored to the same source of
truth.
Verified locally: `--grep "@starter-health|@starter-agent|@starter-chat" --list`
now discovers 17 tests (one per deployed integration).
- Add --validate-on-load to all aimock invocations (4 workflows/scripts
+ 13 integration docker-compose files)
- Replace hardcoded 2-file fixture list with dynamic discovery across
showcase/, examples/integrations/*/, scripts/doc-tests/ (16 fixtures)
- Add sanity check to prevent silent zero-test pass when discovery fails
- Extend showcase_validate.yml path filter to trigger on
examples/integrations/**/fixtures/** and scripts/doc-tests/fixtures/**
- Import and use ValidationResult type for callback parameters
- Fix scripts/doc-tests/fixtures/default.json to use { fixtures: [...] }
envelope shape
Support retries with configurable delay in checkHealth() for Railway
cold-start scenarios. Captures last error from retry loop instead of
making a redundant final request.
The Slack notification for E2E failures only said "Showcase E2E suite
failed" with a link. Now captures playwright output, extracts failed
test names and error messages, and includes them in the Slack message
as a code block so the team can see what broke without clicking through.
copilotkit-qa-bot[bot]