- Summarize count of stale services in the normal case instead of
listing every image by name
- Only expand to the detailed per-service list when rebuild triggers
actually fail, including each service's error reason
- Track triggered_count and failed_count separately so the alert
accurately reflects what happened
- Exit 1 on rebuild-trigger failure so the workflow run shows red in
the Actions UI, and guard the generic failure() notifier with
has_stale != 'true' to prevent double-posting to Slack (the detailed
drift-alert step already covers the drift case)
Surface jq parse failures instead of silently emitting an empty list,
wrap capture() in try/catch so a non-matching title no longer poisons
the whole extraction pipeline, and iterate every test per spec so
multi-project configs don't drop failures.
Replace byte-level cut -c1-200 with head -c 200 | iconv UTF-8//IGNORE
so truncated summaries never emit partial UTF-8 sequences as mojibake.
Broaden the ANSI stripper to cover SGR, OSC, and G0/G1 charset
designator escapes in both sed and jq.
Use mktemp for slack message/payload files with an always() cleanup
step so self-hosted runners stay clean, route matrix.starter through
env for consistency with the existing pattern, and fix two misleading
comments (reporter behavior, cap-at-5 placement). Drop the unused
walk_suites helper.
The `jq -n --arg text "...\n..."` pattern passed the two literal characters
`\n` to jq, which preserved them as-is in the JSON string. Slack then
rendered the literal backslash-n instead of a line break, producing
messages like `*Starter Deployed Smoke Test Failed*\nView run`.
Switch the three affected alert builders to `printf` into a file with real
LF bytes and load via `jq -n --rawfile` so escaping is handled correctly.
This matches the pattern already used in `showcase_smoke-monitor.yml`.
Also enrich the Starter Deployed Smoke alert with:
- failed starter slug(s) in the header
- direct link to the failed job (not just the workflow run)
- up to 5 failure entries each showing: slug, test level tags
(@starter-health/@starter-agent/@starter-chat), first line of error
- "…and N more" footer when more than 5 failed
Enrichment is driven by a new JSON reporter output from the Playwright run;
if the report is missing (e.g. pre-test step failed) the step falls back to
the minimal header so alerts still fire.
Fixes the literal `\n` rendering seen in Slack for:
- starter_deployed_smoke.yml (Starter Deployed Smoke Test Failed)
- starter-smoke.yml (Starter smoke test failing: <starter>)
- showcase_drift-detection.yml (Showcase E2E suite failed)
## Summary
- **Problem 1**: Health check constructed URLs like
`showcase-X-production.up.railway.app` which never matched actual
Railway domains (many have hash suffixes like `-3f57`). Crashed services
silently passed health checks.
- **Problem 2**: No Railway deploy status check — only HTTP health was
checked, so CRASHED deployments were never caught.
- **Fix**: Replaced the URL-guessing health check with Railway API
polling that queries actual deployment status and real service domain.
Fails immediately on CRASHED, validates both Railway SUCCESS status and
HTTP 200 on the real domain.
## Test plan
- [ ] Trigger a `workflow_dispatch` deploy for a single service and
verify the health check step queries Railway API and logs status/domain
- [ ] Verify a healthy service shows `Railway status=SUCCESS` and `HTTP
check: ... → 200`
- [ ] Verify a crashed service (e.g. bad image) fails the job with
`::error::Service X CRASHED on Railway`
The previous health check constructed URLs as `${IMAGE}-production.up.railway.app`
which never matched Railway's hash-suffixed domains, so it silently
passed on crashed services (the issue that made claude-sdk-typescript
and mastra appear "deployed" while crashing at runtime).
New verification:
- Capture prior deployment ID before redeploying so verify step can
distinguish fresh deployment from stale (avoids false-positive where
first poll sees previous SUCCESS deployment and exits 0 immediately)
- Poll Railway API for actual deployment status, fail fast on terminal
failures (CRASHED, FAILED, REMOVED, SKIPPED)
- Use real staticUrl from Railway API instead of guessing URL pattern
- Hit /api/health instead of root (backends 404 at /)
- Require 2 consecutive 200 responses before declaring healthy (catches
SUCCESS-then-crash from JVM lazy init failures, Python OOM on first
request)
- Check GraphQL response body for errors (HTTP 200 + {errors:[...]}
is how Railway signals auth/query failures)
- Validate RAILWAY_TOKEN is set before polling
- 360s total budget (24 × 15s) to accommodate slow-boot services
(spring-ai, mastra)
## Summary
Ports 3 proven patterns from ag-ui's release system to CopilotKit.
### C1: Registry error vs 404 distinction
`getPublishedVersion` in `publish-release.ts` now distinguishes between
npm E404 (package genuinely not published — proceed) and real errors
(network timeout, auth failure, rate limit — stop). Previously all
errors returned `null`, silently bypassing the version guard.
### C2: Pre-existing tag check
Added a check in `publish-release.yml` that verifies the tag doesn't
already exist before attempting to create it. Prevents the "published
but no tag" state on retries.
### C3: Pre-publish tests in prerelease
Added `pnpm run test` between build and publish in the canary workflow.
A broken canary erodes trust in the prerelease channel.
**Cross-pollination context:** [Notion
page](https://www.notion.so/3413aa38185281828aa1dfa014808ddc)
The ag-ui side (strict version ordering, AI release notes, atomic PR
creation) ships via ag-ui PR #1487.
## Test plan
- [ ] Verify `getPublishedVersion` returns null on E404 but throws on
network errors
- [ ] Verify pre-existing tag check fails fast before publish
- [ ] Verify prerelease workflow runs tests before canary publish
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- The docs-sync warning notification was sending "see workflow run for
details" with no actionable information
- Now reads `review-items.txt` and includes the file list directly in
the Slack message
- Recipients can see which files need attention without digging through
CI logs
## Test plan
- [ ] Trigger docs-sync with a file that has showcase-local
modifications (exit code 3 path)
- [ ] Verify Slack notification includes the file list in a code block
- [ ] Verify auto-push-only path (exit code 0) still does NOT send the
warning notification
The warning notification for files needing manual review was sending
'see workflow run for details' with no actionable information.
- Read review-items.txt and include file list in the Slack message
- Use jq for proper JSON escaping (handles newlines, quotes, special chars)
- Guard against missing review-items.txt with fallback and ::warning::
- Review-needed notification fires independently of push/merge outcome
New workflow running starter health/agent/chat tests against Railway:
- Triggers: 6h cron, after showcase deploy, manual dispatch
- Alerts on schedule + workflow_run failures (Slack + GitHub issue)
- Issue dedup by title match, continue-on-error on Slack
- Proper working-directory for npm ci and Playwright install
## Summary
- **drift-detection**: Split inline payload into `jq`-built file +
`payload-file-path`; sanitize playwright output (strip ANSI, head -3,
cap 200 chars)
- **starter-smoke**: Replace `toJSON(format(...))` double-encoding with
`jq` payload builder
- **showcase_deploy**: Replace 300-char inline ternary with readable
shell conditional + `jq`
All three workflows now use the same pattern: build a sanitized JSON
file with `jq -n`, then reference it via `payload-file-path`. This
eliminates raw `%0A` in Slack messages, unformatted stack traces, and
double-encoded JSON.
## Test plan
- [ ] Trigger `showcase_drift-detection.yml` manually — verify Slack
alert formats correctly on failure
- [ ] Trigger `starter-smoke.yml` manually — verify Slack alert on a
known-failing starter
- [ ] Trigger `showcase_deploy.yml` with `service: shell` — verify
deploy notification renders cleanly
- [ ] Confirm no `%0A` or raw escape sequences appear in any Slack
message
Use jq to build JSON payloads safely and payload-file-path to avoid
inline multiline content. Limits error context to 3 lines, strips ANSI
codes, and caps field length at 200 chars.
- drift-detection: split payload build from post, sanitize playwright output
- starter-smoke: replace toJSON(format(...)) double-encoding with jq
- showcase_deploy: replace 300-char inline ternary with readable shell conditional
The RAILWAY_TOKEN in GitHub secrets can't call serviceInstanceUpdate
(403 Forbidden). Services are now all configured to pull :latest, so
serviceInstanceRedeploy will pull the latest image automatically.
Railway was pinned to old SHA tags — serviceInstanceRedeploy just
restarts the existing image. Now the deploy step calls
serviceInstanceUpdate to set the image to the exact SHA just pushed,
then triggers the redeploy. Also re-enables Docker cache.
The GHA Docker layer cache was serving stale builds — renderer adapter
code wasn't in the deployed images despite successful builds. Disabling
cache-from forces a full rebuild. Will re-enable after cache is fresh.
- Remove test-integration-tmp from workflow (package was deleted)
- starter-langgraph-python: disable Turbopack for Next.js build
(serverExternalPackages incompatible with Turbopack)
- starter-crewai-crews: pin crewai-tools~=0.47.1 to avoid version
conflict with crewai==0.130.0
- shell-dojolike: add missing zod dependency (required by shared
frontend modules)
The paths-filter YAML had test_integration_tmp defined twice (lines 71
and 90), causing a "duplicated mapping key" parse error that blocked all
deploy runs.
The CI workflow's shared module copy step used trailing slashes on both
source and destination (cp -r src/ dest/src/), which on Linux copies the
*contents* into an already-created dest/src/ — resulting in
shared_frontend/src/src/ instead of shared_frontend/src/. Same issue
for shared_typescript/tools/.
Root cause confirmed via diagnostic instrumentation: index.ts existed at
the wrong depth, leaving the webpack alias target empty.
Fix: mkdir only the parent, cp without trailing slashes so the directory
itself is placed correctly. Also removes the diagnostic debug line from
pydantic-ai Dockerfile.
- Node 20 → 22 in all Dockerfiles + CI workflows
- CI copies shared_python, shared_frontend/src, shared_typescript/tools
- tsconfig paths for @copilotkit/showcase-shared
- .gitignore for CI artifacts
The v1 action uses SubtleCrypto.importKey() which fails with 'Invalid keyData'
on certain PEM key formats. v2 handles this more robustly.
Also adds step-level failure info to the Slack notification so we know
WHICH step failed instead of just 'workflow failed'.
## Summary
Two fixes:
### 1. AG2 showcase agent blocks on human feedback
The `ConversableAgent` defaults to `human_input_mode="TERMINATE"`, which
prompts for user feedback when the conversation terminates. In the AG-UI
streaming context via `AGUIStream`, this blocks with:
> "Please give feedback to the sender. Press enter to skip and use
auto-reply, or type 'exit' to stop the conversation:"
This was broken from day one (showcase deployed April 8, first L4 test
ran April 9 and failed). The ag2 image was never rebuilt since — the
unpinned `ag2>=0.9.0` dep resolved to 0.11.5 at build time.
Fix: set `human_input_mode="NEVER"`.
### 2. Showcase E2E Slack alerts lack detail
The Slack notification only said "Showcase E2E suite failed" with a
link. Now captures playwright output and includes failed test names +
error messages:
```
❌ Showcase E2E suite failed
View run
1) [L4: tools] ag2 renders tool results @tools
Error: ag2 response doesn't contain weather info: "Please give feedback..."
```
Test names now carry their level prefix (`[L1: health]`, `[L2: agent]`,
`[L3: chat]`, `[L4: tools]`) so you can tell what broke from the Slack
message alone.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The Slack notification for E2E failures only said "Showcase E2E suite
failed" with a link. Now captures playwright output, extracts failed
test names and error messages, and includes them in the Slack message
as a code block so the team can see what broke without clicking through.
## Summary
The drift detection workflow runs `npx playwright test --grep
"@health|@agent|@chat"` which matches BOTH `integration-smoke.spec.ts`
(deployed Railway backends) and `starter-smoke.spec.ts` (Docker-built
starters). The starter-smoke tests try to connect to `localhost:3000`
which doesn't exist in CI, causing all runs to fail.
Fix: scope to `npx playwright test integration-smoke --grep ...` so only
the showcase integration tests run.
## Test plan
- [x] Verified locally: 13 integration-smoke tests pass, 0 starter-smoke
tests included
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Jordan Ritter