- Summarize count of stale services in the normal case instead of
listing every image by name
- Only expand to the detailed per-service list when rebuild triggers
actually fail, including each service's error reason
- Track triggered_count and failed_count separately so the alert
accurately reflects what happened
- Exit 1 on rebuild-trigger failure so the workflow run shows red in
the Actions UI, and guard the generic failure() notifier with
has_stale != 'true' to prevent double-posting to Slack (the detailed
drift-alert step already covers the drift case)
Surface jq parse failures instead of silently emitting an empty list,
wrap capture() in try/catch so a non-matching title no longer poisons
the whole extraction pipeline, and iterate every test per spec so
multi-project configs don't drop failures.
Replace byte-level cut -c1-200 with head -c 200 | iconv UTF-8//IGNORE
so truncated summaries never emit partial UTF-8 sequences as mojibake.
Broaden the ANSI stripper to cover SGR, OSC, and G0/G1 charset
designator escapes in both sed and jq.
Use mktemp for slack message/payload files with an always() cleanup
step so self-hosted runners stay clean, route matrix.starter through
env for consistency with the existing pattern, and fix two misleading
comments (reporter behavior, cap-at-5 placement). Drop the unused
walk_suites helper.
Add a sentence to the warning callout explaining that if agentId doesn't
match a runtime-registered agent, the interrupt will silently never fire.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
CrewAI's ChatWithCrewFlow.__init__ (called by ag_ui_crewai's
add_crewai_crew_fastapi_endpoint at module import) makes blocking
synchronous LLM calls via generate_input_description_with_ai and
generate_crew_description_with_ai in crewai/cli/crew_chat.py. Any LLM
hiccup (aimock regression, OpenAI outage, network blip) crashes the
Python process before uvicorn can bind, causing Railway healthcheck
failure and deploy rollback.
Patch both functions to return static strings before ag_ui_crewai is
imported. The AI-generated descriptions are only used by the CrewAI
chat UI (not the CopilotKit runtime), so static defaults are
functionally equivalent for our showcase.
Verified via docker build: unhardened image crashes on import with
APIError at crew_chat.py:481; hardened image starts cleanly with an
invalid OPENAI_BASE_URL and responds on /api/health.
Upstream fix (deferred construction) landed on ag-ui main but is not
yet released in ag-ui-crewai > 0.1.5. Remove shim when released.
Upstream issue: https://github.com/crewAIInc/crewAI/issues/5510
Follow-up to #3971. aimock supports fixture schema validation at startup via
--validate-on-load, but it's opt-in. The showcase Dockerfile did not pass
the flag, so fixtures with unrecognized response keys (e.g. "text" instead
of "content") loaded silently and only failed at request time with HTTP 500.
That's what crashed crewai-crews on startup.
Changes:
- showcase/aimock/Dockerfile: pass --validate-on-load so broken fixtures
fail the container boot, not individual requests.
- showcase/scripts/__tests__/aimock-fixtures.test.ts: new vitest spec that
loads feature-parity.json and smoke.json via @copilotkit/aimock's
loadFixtureFile + validateFixtures and asserts zero errors. Runs as part
of the existing showcase-validate CI workflow.
- showcase/scripts/package.json: add @copilotkit/aimock dependency for the
validator import.
Verified red-green: with the pre-#3971 broken "text" fixtures, validateFixtures
flags 5 errors; post-#3971 it returns zero. Docker red-green: container with
an intentionally broken fixture fails to start with "Validation failed: 1
error(s)" and non-zero exit.
The `jq -n --arg text "...\n..."` pattern passed the two literal characters
`\n` to jq, which preserved them as-is in the JSON string. Slack then
rendered the literal backslash-n instead of a line break, producing
messages like `*Starter Deployed Smoke Test Failed*\nView run`.
Switch the three affected alert builders to `printf` into a file with real
LF bytes and load via `jq -n --rawfile` so escaping is handled correctly.
This matches the pattern already used in `showcase_smoke-monitor.yml`.
Also enrich the Starter Deployed Smoke alert with:
- failed starter slug(s) in the header
- direct link to the failed job (not just the workflow run)
- up to 5 failure entries each showing: slug, test level tags
(@starter-health/@starter-agent/@starter-chat), first line of error
- "…and N more" footer when more than 5 failed
Enrichment is driven by a new JSON reporter output from the Playwright run;
if the report is missing (e.g. pre-test step failed) the step falls back to
the minimal header so alerts still fire.
Fixes the literal `\n` rendering seen in Slack for:
- starter_deployed_smoke.yml (Starter Deployed Smoke Test Failed)
- starter-smoke.yml (Starter smoke test failing: <starter>)
- showcase_drift-detection.yml (Showcase E2E suite failed)
## Summary
- Five fixtures in `showcase/aimock/feature-parity.json` used an
unrecognized `"text"` field instead of `"content"`. aimock's
`/v1/chat/completions` handler returns 500 `"Fixture response did not
match any known type"` when the response shape doesn't match its
discriminators.
- CrewAI's `ChatWithCrewFlow.__init__` calls
`generate_input_description_with_ai` at module import time. That prompt
contains `"Reporting Analyst"` / `"reports"` text (derived from the
`LatestAiDevelopment` crew yaml), which substring-matched the broken
`"report"` fixture and produced a 500. CrewAI treated that as
`InternalServerError`, crashing the crewai-crews container before
uvicorn could bind a port. Railway healthcheck failed → deployment
rolled back to the previous ACTIVE build.
- Fix: convert the five broken fixtures (`plan`, `steps`, `mars`,
`dashboard`, `report`) from `"text"` to `"content"`, and add a targeted
fixture that matches CrewAI's exact startup prompt prefix (`"Based on
the following context, write a concise"`) so it wins over the generic
fixtures.
## Root cause
aimock discriminators in `helpers.ts` only recognize `content`,
`toolCalls`, `error`, etc. — never `text`. Any fixture using `"text"`
silently fell through to the 500 branch. See
[server.ts:563-663](https://github.com/CopilotKit/aimock/blob/main/src/server.ts#L563)
and
[helpers.ts:51-80](https://github.com/CopilotKit/aimock/blob/main/src/helpers.ts#L51).
The regression window matches — fixtures landed 2026-04-14 12:52 PDT
(5ac07d4d9); Railway aimock picked them up at 2026-04-14T21:57 UTC.
## Verification
Red-green reproduced locally against `@copilotkit/aimock@1.10.0`:
```
PRE-FIX status: 500
PRE-FIX body: {"error":{"message":"Fixture response did not match any known type","type":"server_error"}}
POST-FIX status: 200
POST-FIX body: {"id":"chatcmpl-...","object":"chat.completion",...,"content":"A short input value used to parameterize the crew's tasks and agents."}
```
Also verified: the five previously-broken user messages (`report`,
`plan`, `dashboard`, `mars ... steps`, etc.) all return 200 with valid
content after the fix.
## Follow-up (not in this PR)
CrewAI's blocking LLM call at module import is fragile — any aimock
hiccup still crashes the container before it can bind a port. That is
upstream behavior (`crewai/cli/crew_chat.py:481
generate_input_description_with_ai`) and should be tracked as a separate
issue.
## Test plan
- [x] Local: red-green confirmed against `@copilotkit/aimock@1.10.0` —
500 pre-fix, 200 post-fix for CrewAI-style startup prompt
- [x] Local: the five previously-broken fixtures (`plan`, `steps`,
`mars`, `dashboard`, `report`) now return 200 + valid content
- [ ] Railway: aimock image rebuilds and picks up new fixtures;
crewai-crews next deploy reaches ACTIVE
## Summary
- Removes the flaky, unnecessary `--with-deps` from `npx playwright
install chromium` in all 13 starter `docker-compose.test.yml` files
- `mcr.microsoft.com/playwright:v1.52.0-noble` already ships with both
Chromium browsers and all required system libs pre-installed;
`--with-deps` only forces a redundant `apt-get update && apt-get
install`
- That apt-get call is the sole root cause of the recurring
scheduled-run failures (Slack alerts every few hours):
> `tests-1 | Failed to install browsers`
> `tests-1 | Error: Installation process exited with code: 100`
preceded by Ubuntu mirror errors like `File has unexpected size (… !=
…). Mirror sync in progress?` or `Hash Sum mismatch` against
`archive.ubuntu.com`. Each matrix job races apt independently, so the
failing subset rotates every run (confirmed across runs 24526747926 /
24509673514 / 24495760568 — different 3–5 starters fail each time,
always with the identical apt signature). This is flake, not regression.
## Root cause
`npx playwright install chromium --with-deps` executes `apt-get update
&& apt-get install -y libnss3 libatk1.0-0 libatk-bridge2.0-0 …` inside
the tests container. When Ubuntu's archive mirrors are mid-sync, apt-get
aborts with exit code 100, Playwright surfaces it as "Installation
process exited with code: 100", and the `tests` service exits 1 —
failing the smoke job.
Verified locally that `npx playwright install chromium` (no
`--with-deps`) runs clean in
`mcr.microsoft.com/playwright:v1.52.0-noble` and that
`/ms-playwright/chromium-1169` is already present in the image — so no
deps step is needed.
## Caveat — what happens on a future Playwright client bump
If `@playwright/test` in `showcase/tests/package.json` is bumped to a
version newer than what the pinned image tag bundles, `npx playwright
install chromium` (still in the command) will download the **matching
browser binary** into `/ms-playwright`. That covers the browser itself,
but **not system libs** — if the newer browser needs libs that aren't in
`v1.52.0-noble`, launch will fail and we would need to either (a) re-add
`--with-deps` for that run, or (b) bump the image tag to match. The
mastra compose file now carries a code comment calling out this coupling
so future bumps keep image tag and client version aligned.
## Test plan
- [ ] Starter smoke workflow run on this PR passes for all 12 matrix
entries (exercises the exact code change)
- [ ] Next scheduled run after merge shows zero "exit code: 100"
failures in Slack alerts
- [ ] If the Playwright client drifts past the image's bundled browser
build, `npx playwright install chromium` downloads the matching browser
binary (but missing system libs would need `--with-deps` back, or an
image tag bump)
## Follow-up commit — keep playwright install stderr
Addresses CR: stderr from `npx playwright install chromium` is
diagnostic signal (network errors, version drift, missing libs). Only
the noisy `npm install` step retains `2>/dev/null`; Playwright install
stderr is now visible in logs. Image/version coupling is documented
inline on `examples/integrations/mastra/docker-compose.test.yml`.
Playwright install stderr is diagnostic signal — version/network errors,
missing system libs, or browser download failures all surface here. Only
suppress stderr on the noisy npm install step. Also pin the image-tag /
client-version relationship with a code comment on one compose file so
future bumps know to keep them aligned.
The showcase aimock feature-parity fixture had five entries that used the
unrecognized field "text" instead of "content" in the response body.
aimock's /v1/chat/completions handler validates the response shape through
discriminators (isTextResponse, isToolCallResponse, etc.) and when none
match, returns 500 "Fixture response did not match any known type".
When CrewAI's ChatWithCrewFlow.__init__ calls generate_input_description_with_ai,
it sends a prompt containing "Reporting Analyst" / "reports" text (from the
LatestAiDevelopment crew definition). aimock substring-matches this against
the "report" fixture, which had the broken "text" shape, and returned 500.
CrewAI treats that as an InternalServerError, the crewai-crews container
crashes during module import (before uvicorn binds), Railway healthchecks
fail, and the deployment rolls back to the previous ACTIVE build.
Fix:
- Convert the 5 fixtures with "text" (plan, steps, mars, dashboard, report)
to the valid "content" field so they return 200 OK.
- Add a targeted fixture that matches CrewAI's exact startup prompt prefix
("Based on the following context, write a concise") and returns a clean
text description. This catches generate_input_description_with_ai and
generate_crew_description_with_ai before they can fall through to the
more generic fixtures, providing a more predictable response for CrewAI
startup.
Verified locally against @copilotkit/aimock@1.10.0:
- Before: the CrewAI-style prompt returns 500 with the exact error message
seen in Railway logs.
- After: returns 200 with a valid chat.completion envelope.
Note: CrewAI's blocking LLM call at module import remains fragile (any
aimock hiccup crashes the container before it can bind a port). That is
an upstream issue and will be tracked separately.
The Mastra showcase package registers a single local agent
(`weatherAgent`), but the demo pages request names like `agentic_chat`,
`human_in_the_loop`, `tool-rendering`, etc. `MastraAgent.getLocalAgents`
returns the agents under their registration keys, so the runtime could
not resolve the demo-requested names and the L3 mastra chat test
failed.
Mirror the crewai-crews pattern: wrap the local agents map and expose
`weatherAgent` under every demo-expected name. `weatherAgent` itself is
preserved so the internal smoke route keeps working.
The Playwright Docker image (mcr.microsoft.com/playwright:v1.52.0-noble)
ships with all Chromium system libraries and browsers pre-installed, so
`--with-deps` adds no runtime value — it only forces a redundant
`apt-get update && apt-get install` inside the tests container.
That apt-get call is the sole source of the intermittent
"Installation process exited with code: 100" / "Failed to install
browsers" failures that rotate across the 12 smoke matrix entries. The
underlying cause is transient Ubuntu archive mirror hash/size mismatches
("File has unexpected size ... Mirror sync in progress?", "Hash Sum
mismatch"), which cause apt to abort with exit 100. Because each matrix
job races apt against archive.ubuntu.com independently, the failing
subset rotates per run (run 24526747926 hit 5 starters; run 24509673514
hit a different 5; run 24495760568 hit 3) — classic flake, not a per-
starter regression.
Dropping --with-deps eliminates the apt-get call entirely. The browsers
themselves are already present at /ms-playwright/chromium-* in the base
image, and `npx playwright install chromium` remains as a cheap no-op
that self-heals if the pinned Playwright version ever drifts from the
image's bundled browser build.
## Summary
- **Problem 1**: Health check constructed URLs like
`showcase-X-production.up.railway.app` which never matched actual
Railway domains (many have hash suffixes like `-3f57`). Crashed services
silently passed health checks.
- **Problem 2**: No Railway deploy status check — only HTTP health was
checked, so CRASHED deployments were never caught.
- **Fix**: Replaced the URL-guessing health check with Railway API
polling that queries actual deployment status and real service domain.
Fails immediately on CRASHED, validates both Railway SUCCESS status and
HTTP 200 on the real domain.
## Test plan
- [ ] Trigger a `workflow_dispatch` deploy for a single service and
verify the health check step queries Railway API and logs status/domain
- [ ] Verify a healthy service shows `Railway status=SUCCESS` and `HTTP
check: ... → 200`
- [ ] Verify a crashed service (e.g. bad image) fails the job with
`::error::Service X CRASHED on Railway`
The previous health check constructed URLs as `${IMAGE}-production.up.railway.app`
which never matched Railway's hash-suffixed domains, so it silently
passed on crashed services (the issue that made claude-sdk-typescript
and mastra appear "deployed" while crashing at runtime).
New verification:
- Capture prior deployment ID before redeploying so verify step can
distinguish fresh deployment from stale (avoids false-positive where
first poll sees previous SUCCESS deployment and exits 0 immediately)
- Poll Railway API for actual deployment status, fail fast on terminal
failures (CRASHED, FAILED, REMOVED, SKIPPED)
- Use real staticUrl from Railway API instead of guessing URL pattern
- Hit /api/health instead of root (backends 404 at /)
- Require 2 consecutive 200 responses before declaring healthy (catches
SUCCESS-then-crash from JVM lazy init failures, Python OOM on first
request)
- Check GraphQL response body for errors (HTTP 200 + {errors:[...]}
is how Railway signals auth/query failures)
- Validate RAILWAY_TOKEN is set before polling
- 360s total budget (24 × 15s) to accommodate slow-boot services
(spring-ai, mastra)
## Summary
- Shared A2UI snippet (`docs/snippets/shared/generative-ui/a2ui.mdx`)
now recommends `a2ui: { injectA2UITool: true }` so framework
integrations that don't ship their own rendering tool get `render_a2ui`
injected into the agent's tool list, with usage guidelines.
- LangGraph fixed-schema guide
(`docs/content/docs/integrations/langgraph/generative-ui/a2ui/fixed-schema.mdx`)
drops `injectA2UITool: true` — that guide's Python `search_flights` tool
returns A2UI operations directly, which the middleware auto-detects, so
tool injection is unnecessary there.
## Why
Previously the shared snippet recommended `a2ui: true` with no tool
injection — which only works when the agent already has a path to
produce A2UI output. For most integrations where the agent doesn't bring
its own rendering tool, that led to surfaces not rendering. Meanwhile
the LangGraph fixed-schema page overspecified `injectA2UITool: true`
even though its example doesn't need it.
## Test plan
- [ ] Visual check of rendered snippet on the affected integration pages
(agno, mastra, llamaindex, pydantic-ai, crewai-flows,
microsoft-agent-framework, agent-spec, ag2, built-in-agent, aws-strands,
adk, plus `learn/generative-ui/specs/a2ui`)
- [ ] Verify LangGraph fixed-schema example still works with `a2ui: {}`
Most framework integrations don't ship their own A2UI rendering tool, so the
shared snippet now guides users to inject `render_a2ui` via
`a2ui: { injectA2UITool: true }`. The LangGraph fixed-schema guide returns A2UI
operations directly from a Python tool, so the middleware auto-detects them and
tool injection is unnecessary — switched to `a2ui: {}` there with a note.
## Release monorepo v1.56.2
**Scope:** `monorepo` | **Bump:** `patch`
---
### How this release process works
1. **This PR was created automatically** by the "release / create-pr"
workflow.
It bumped the `monorepo` packages to `1.56.2`
and generated AI-enhanced release notes.
2. **CI runs on this PR** — the full test suite (unit tests, lint, type
checks, build)
must pass before merging. This is the review gate.
3. **Review the release notes** in `release-notes.md` in this PR.
If a Notion draft was created, you can edit the release notes there
before merging.
4. **When this PR is merged**, the `release / publish` workflow
automatically:
- Builds all packages
- Publishes the `monorepo` packages to npm at version `1.56.2`
- Creates git tag `monorepo/v1.56.2`
- Creates a GitHub Release with the final release notes
### Before merging
- [ ] CI is green (tests, lint, types, build)
- [ ] Version bumps look correct
- [ ] Release notes are accurate (edit in Notion if a draft was created)
---
> **Do not merge until CI is fully green.** The full test suite runs
automatically on this PR.
## Summary
- Remove `@langchain/community` from `@copilotkit/sdk-js`
peerDependencies — it's declared as `^0.3.58` but never imported
anywhere in the package
- This was the only constraint blocking `@langchain/core@1.x` resolution
for consumers using the LangGraph JS starter
## Context
`@copilotkit/sdk-js` has zero imports from `@langchain/community`. The
`^0.3.58` peer dep forced npm to resolve `@langchain/community@0.3.x`,
which transitively pulled in `@getzep/zep-cloud` → `langchain@0.3.x`,
conflicting with `sdk-js`'s own `langchain@>=1.0.0` peer requirement.
Removing the unused peer dep eliminates this resolution conflict
entirely.
## Test plan
- [ ] `npm install` in a fresh project using `@copilotkit/sdk-js`
resolves without peer dep warnings for `@langchain/community`
- [ ] Existing LangGraph JS starter installs cleanly
- [ ] No runtime behavior change (nothing imported from the removed dep)
## Release monorepo v1.56.1
**Scope:** `monorepo` | **Bump:** `patch`
---
### How this release process works
1. **This PR was created automatically** by the "release / create-pr"
workflow.
It bumped the `monorepo` packages to `1.56.1`
and generated AI-enhanced release notes.
2. **CI runs on this PR** — the full test suite (unit tests, lint, type
checks, build)
must pass before merging. This is the review gate.
3. **Review the release notes** in `release-notes.md` in this PR.
If a Notion draft was created, you can edit the release notes there
before merging.
4. **When this PR is merged**, the `release / publish` workflow
automatically:
- Builds all packages
- Publishes the `monorepo` packages to npm at version `1.56.1`
- Creates git tag `monorepo/v1.56.1`
- Creates a GitHub Release with the final release notes
### Before merging
- [ ] CI is green (tests, lint, types, build)
- [ ] Version bumps look correct
- [ ] Release notes are accurate (edit in Notion if a draft was created)
---
> **Do not merge until CI is fully green.** The full test suite runs
automatically on this PR.
@copilotkit/sdk-js declares @langchain/community as a peer dependency
but never imports from it. The ^0.3.58 constraint was the only thing
blocking @langchain/core@1.x resolution for consumers. Remove it
entirely since it serves no purpose.
- Remove @langchain/community from sdk-js peerDependencies (unused,
was blocking @langchain/core@1.x resolution)
- Add langchain@^1.0.0 to langgraph-js agent deps to prevent
transitive resolution to 0.3.x
- Add postcss.config.mjs to mcp-apps threejs-server
Jordan Ritter