showcase_aimock-e2e.yml:
- workflow_dispatch slug is now a choice-type enum restricted to Python
packages that ship aimock_toggle.py; a TS (mastra) or Java (spring-ai)
slug would have skipped the Python agent start step and then failed with
a misleading Playwright timeout. For comment-trigger paths, the pkg-type
step short-circuits with a clear ::error:: when the slug does not ship
aimock_toggle.py.
- Slug parsing replaced PCRE grep -oP '\\K\\S+' with a POSIX-safe
grep -oE + sed pipeline so BSD/Alpine grep works too (future-proof
against runner image changes).
- aimock pinned to @copilotkit/aimock@^1.14.3 (was @latest) with
--ignore-scripts; unpinned @latest let a bad aimock publish silently
poison CI for everyone.
- pnpm install now runs with --ignore-scripts — trusted commenter triggers
/test-aimock on untrusted PR content, so postinstall scripts must not
run on the runner.
- OPENAI_BASE_URL is no longer pre-set for Python agent start on packages
that ship aimock_toggle.py — forcing the toggle itself to do the
redirection proves the toggle works rather than masking it with env
already set.
- Dead AIMOCK_URL export removed from the Next.js dev-server step (the
runtime only reads OPENAI_BASE_URL).
- Python agent health-check extended 60s -> 90s (45 iter * 2s) for CrewAI
cold imports.
- actions/setup-python caches pip keyed on requirements.txt.
- github-script Post-result step passes slug + job status via env rather
than ${{ }} interpolation — even though the slug is already whitelisted,
the env pattern is the defensive default for dynamic values.
showcase_validate.yml:
- python-unit-tests is now a matrix on Python 3.10 + 3.12 (fail-fast
disabled) so the typing_extensions fallback branch gets CI coverage;
previously only 3.12 ran.
- pip install per package is a HARD FAIL (was ::warning:: + continue);
hiding broken requirements.txt behind a warning let a package ship green
with unresolvable runtime deps. A missing requirements.txt is still
handled gracefully via the -f guard.
- pip caching added via actions/setup-python.
The showcase_deploy.yml notify job posted a green success message on
every run. Bulk drift rebuilds fan out one showcase_deploy.yml run per
stale service (up to ~18), so a single drift cycle produced ~18
"Showcase deploy: 1 service(s) deployed to Railway" messages in
#oss-alerts — pure noise.
Align with the channel policy: surface only actionable state
(failures + state transitions). The bulk-rebuild aggregate is already
posted by showcase_smoke-monitor.yml as "📦 Image drift detected
— N rebuilds triggered". Ad-hoc single-service pushes/dispatches stay
quiet on success — the Actions UI is the source of truth.
Kept as-is: failure messages (pre-build and build), mid-matrix
cancellation info (state transition worth humans seeing). Also added an
empty-webhook guard on the Post step so a missing SLACK_WEBHOOK_OSS_ALERTS
secret fails closed instead of erroring out the step.
The `secrets.*` context is not a valid named-value inside step-level
`if:` expressions on push events — GitHub Actions rejects it at
workflow parse time with "Unrecognized named-value: 'secrets'", which
caused both showcase_validate.yml and showcase_drift-report.yml to
fail at startup with zero jobs spawned after #4018 + #4060 merged.
Hoist the webhook into a job-level `env: SLACK_WEBHOOK` and reference
`env.SLACK_WEBHOOK` in every step-level `if:`. The `with: webhook:`
keys still use `secrets.*` directly (valid in that context).
PR CI didn't catch this because pull_request events parse if:
expressions less strictly than push events.
showcase_validate.yml and showcase_drift-report.yml were flipped to
runs-on: depot-ubuntu-24.04-4 in #4018 but inherited the repo's
default least-privilege permissions (contents: read only). Depot
runner provisioning uses OIDC and requires id-token: write, so the
jobs failed to spawn on the first main-branch push — matching the
pattern already used by showcase_deploy.yml's Depot job.
## Showcase validation tooling (Bundle 3)
Ships three CLI validators, a shared parsing lib, and two CI workflows
that enforce consistency across the 17 showcase packages and detect
drift before it lands on main. Consolidates four earlier tooling PRs
(#3985, #3987, #3995, #3996).
## What's in the box
### `showcase/scripts/` — three validators
| Tool | Purpose | Exit codes |
|------|---------|-----------|
| `audit.ts` | Cross-checks manifest-declared demos against
`tests/e2e/*.spec.ts` and `qa/*.md`, plus `examples/integrations/`
provenance via `SLUG_TO_EXAMPLES` / `FALLBACK_MAP` | 0 ok, 1 anomalies,
2 invalid-input, 3 unreadable, 4 internal, 5 strict-warnings |
| `validate-pins.ts` | Framework-dep pin-drift between
`showcase/packages/*/` and their dojo `examples/integrations/*/`
counterparts. Parses package.json, requirements.txt, pyproject.toml
(Poetry + PEP 621) | 0 ok, 1 drift, 2 internal, 3 unreadable |
| `validate-parity.ts` | Enforces demo ↔ spec ↔ qa coverage per package
with a monotonic demo-count floor | 0 ok, 1 warnings, 2 invalid-input, 3
unreadable, 4 internal, 5 must-failure |
### `showcase/scripts/lib/` — shared primitives
- **`slug-map.ts`** — single-source-of-truth `ENTRIES` for the showcase
slug taxonomy;
`BORN_IN_SHOWCASE`/`SLUG_MAP`/`SLUG_TO_EXAMPLES`/`FALLBACK_MAP` derived
and frozen at module load. `SlugEntry` is a discriminated union that
makes illegal states (born-in-showcase with non-empty examples)
unrepresentable. `freezeSet`/`freezeMap` helpers install throwing
replacements via `Object.defineProperty({writable:false,
configurable:false})` so `Set.add` / `Map.set` truly fail at runtime.
- **`manifest.ts`** — `parseManifest` returns a tagged `ParsedManifest`
union (`ok` | `missing` | `malformed{subkind: "syntax"|"shape"}` |
`unreadable`) with a never-throws content contract. Uses `statSync` +
errno inspection (not `existsSync`, which conflates ENOENT with EACCES).
`DemoId` is a branded string minted only via `createDemoId`.
### `.github/workflows/` — CI enforcement
- **`showcase_validate.yml`** — runs on PR and push-to-main. Enforces
the e2e-spec floor, runs the validators, and drives the pin-drift
ratchet.
- **`showcase_drift-report.yml`** — weekly Monday 10:00 UTC +
workflow_dispatch. Computes `set_status` (OK / SET DRIFTED / COUNT
DRIFTED) and posts to Slack.
Both workflows run on `depot-ubuntu-24.04-4` (Startup plan, unlimited)
for persistent pnpm/npm cache across runs.
## The pin-drift ratchet
`validate-pins.ts` currently finds **111 existing pin-drift failures**
across 12 showcase packages. Rather than block the PR on those, we
baseline them in `showcase/scripts/fail-baseline.json` and ratchet:
- `validatePinsFailCount` must not increase; CI tells you to ratchet
down when it decreases.
- `validatePinsFailHash` is SHA-256 of the sorted-uniqued `[FAIL]` set.
When the count is equal but the hash differs, a fail healed AND a new
one regressed — CI prints the diff and fails.
- `baselineDemoCount` (9) is the single source of truth for the e2e-spec
floor; consumed by both the workflow and `validate-parity.ts` with sync
enforced by a dedicated regression test.
Tracked in #4047. The 111 failures are mostly showcase packages pinning
`@copilotkit/*` to the `next` dist-tag while dojo pins concrete
versions; direction of fix (align showcase → dojo vs. bump dojo →
showcase) is a separate versioning decision outside this PR.
## Correctness posture
- **977 tests**, 13 files, covering every `Anomaly` / `PackageIssue` /
`ParsedManifest` variant in-process and via subprocess CLI for every
exit code. EACCES/ENOTDIR/TOCTOU paths are exercised via chmod probes
(with `it.skipIf` fallback when CI runs as root) and path-filtered
`vi.spyOn` fall-throughs.
- **`fs.statSync` + errno everywhere** — `fs.existsSync` silently
collapses ENOENT with EACCES and is a known anti-pattern in validation
tooling; the codebase uses structured errno discrimination throughout.
- **Tagged discriminated unions with exhaustive `switch` + `never`
guards** — `bucketFor` in `audit.ts`, `deriveMessage` in
`validate-parity.ts`. Adding a new variant without wiring every site is
a compile error.
- **Partial-report preservation** — when an infra error hits
mid-slug-loop, `UnreadableInputError.partialReport` carries
already-collected drift findings so the top-level catch prints them
before exiting 3. One bad package never orphans signal for the rest.
- **Per-slug isolation** — in `validate-parity.ts runParityImpl`, each
slug's audit is wrapped; a crash surfaces as a `crashed` `PackageIssue`
and forces `EXIT_INTERNAL` without aborting siblings.
- **Pipefail + scoped `|| true`** — every workflow step uses `set -euo
pipefail` with grep's no-match tolerance wrapped in `{ grep || true; }`
so producer failures (sort, shasum, cut) still surface.
## Diff
+16,455 / −18 across 34 files (26 source + 5 fixture trees + 2 workflows
+ 1 baseline).
Commits grouped by purpose:
1. `chore(showcase/scripts)`: vitest config + test deps
2. `feat(showcase/scripts)`: shared slug-map and manifest parsing lib
3. `feat(showcase/scripts)`: audit.ts coverage auditor
4. `feat(showcase/scripts)`: validate-pins.ts pin-drift validator
5. `feat(showcase/scripts)`: validate-parity.ts demo/spec/qa parity
validator
6. `ci(showcase)`: validation + weekly drift-report workflows (Depot
runners)
## Test plan
- [x] `pnpm vitest run` in `showcase/scripts/` — 977/977 green
- [x] Exit-code taxonomy verified end-to-end via subprocess tests for
every documented code
- [x] EACCES/ENOENT/ENOTDIR routing verified in all three validators
- [x] Partial-report preservation verified in both in-process and
subprocess paths
- [x] Per-slug crash isolation verified (one broken slug does not orphan
siblings)
- [x] Baseline sync contract (`BASELINE_DEMO_COUNT` ↔
`fail-baseline.json.baselineDemoCount`) pinned by test
- [ ] First CI run on Depot to confirm cold-cache timing (expected 5–8m
vs. 18–20m on ubuntu-latest)
Refs: [Full Action
Inventory](https://www.notion.so/3443aa38185281b5a1dfc6e0890264e1),
#4047
Two GitHub Actions workflows that consume the three validators in
showcase/scripts/ and surface drift to CI and Slack.
showcase_validate.yml — runs on pull_request and push-to-main:
- Enforces per-package e2e spec count against
fail-baseline.json.baselineDemoCount, with find failures
aggregated (not exit-on-first) so multiple package issues surface
in one run. Preserves find's exit status by capturing via command
substitution rather than process substitution (mapfile does not
propagate exit codes through < <(...)).
- Runs validate-pins.ts with both count + content-hash ratchet:
the sorted-uniqued [FAIL] set is SHA-256 hashed so a
count-preserving set change ('one fail healed, another regressed')
is still flagged.
- Separates stdout from stderr before hashing so progress chatter
cannot corrupt the ratchet hash.
- Preserves validator exit codes distinctly (1 drift, 2 internal,
3 unreadable, 4+ future) so downstream consumers can distinguish
crashes from legitimate drift.
- set -euo pipefail throughout, with { grep || true; } scoped to
tolerate grep no-match without masking producer failures.
- Slack notifications gated on push events and secret presence;
payload values wrapped via toJSON(format(...)) for injection
safety.
showcase_drift-report.yml — weekly Monday 10:00 UTC + dispatch:
- Same ratchet + hash logic applied in report mode: computes
set_status (OK / SET DRIFTED / COUNT DRIFTED) and posts to Slack.
- Mirrors validate.yml's pipefail + grep-scope + toJSON discipline.
pull_request runs in the fork context with restricted permissions, which
would block the updateBranch API call. pull_request_target runs in the
base repo with full permissions — the label trigger is safe since no
checkout of PR code happens here.
Also treat 422 responses (already up to date, nothing to merge) as
informational rather than job failures.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Triggers on the qa:update-branch label being added to a PR, calls the
GitHub update-branch API to merge the base into the PR, and removes the
label so it can be re-applied. Enables the QA bot to keep PRs current
without human intervention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Internal job id was still 'starter-smoke:' and artifact pattern was
'starter-smoke-${{ matrix.starter }}'. Rename to 'smoke-starter' /
'smoke-starter-${{ matrix.starter }}' to match the new
test_<layer>-<target> / smoke-<layer> naming convention. Cosmetic only —
no other workflow references this job id or artifact name.
Asymmetric path filter — .changeset was listed in PR trigger but not push.
The dorny/paths-filter step already includes .changeset in the ts: filter,
so this matches the intent on both triggers.
Showcase-only and sdk-python-only PRs were triggering the full TS unit matrix
(3 Node versions x full monorepo). Neither directory affects TS unit tests.
Now paths-ignore matches the spirit of the existing 'examples/**' exclusion.
## Problem
All 18 stale-service rebuild triggers in [run
24534637524](https://github.com/CopilotKit/CopilotKit/actions/runs/24534637524)
failed with:
```
failed to run git: fatal: not a git repository (or any of the parent directories): .git
```
The image-drift workflow is completely broken — when it detects stale
showcase images, the follow-up `gh workflow run showcase_deploy.yml`
step 500-fails for every service, so no rebuilds actually happen.
## Root cause
The `smoke` job in `.github/workflows/showcase_smoke-monitor.yml` has
**no `actions/checkout@v4` step**. `gh workflow run` (unlike `gh api` or
`gh run list`) resolves the target repository by shelling out to `git`
in the current directory to read the remote. With no checkout present,
`git` aborts and the entire rebuild loop fails.
PR #3968 didn't introduce the missing-checkout problem — it was latent —
but it was the first run to reach the rebuild path with a real stale
list, exposing it.
## Fix
Pass `--repo "${{ github.repository }}"` to `gh workflow run` so the CLI
skips the git-based repo lookup entirely. Minimal, one-line change; no
checkout needed.
```diff
-ERR_OUTPUT=$(gh workflow run showcase_deploy.yml -f service="${SVC}" 2>&1) || RC=$?
+ERR_OUTPUT=$(gh workflow run showcase_deploy.yml --repo "${{ github.repository }}" -f service="${SVC}" 2>&1) || RC=$?
```
## Test plan
- [x] YAML parses
- [x] Local pre-commit (lint + test + check:packages) green
- [ ] Next scheduled smoke-monitor run (or manual `workflow_dispatch`
with a stale service) triggers rebuilds without the `not a git
repository` error
## Summary
The "files needing manual review" Slack warning in
`showcase_docs-sync.yml` listed flagged files but did not link the
auto-opened PR, forcing reviewers to hunt for it manually. This PR
captures the PR URL (already exposed as `steps.push.outputs.pr_url`) and
threads it into the alert payload so reviewers can click through
directly.
## Changes
- Review-needed alert now includes a `Review: <PR URL>` line
- Split the alert into two variants:
- PR opened (normal case): includes the PR link
- No PR opened (edge case where the clean-transform portion was empty):
posts review items without a link
- Auto-sync and merge-failed alerts already linked the PR — this brings
the review-needed alert to parity
## Test plan
- [ ] YAML validated locally (`python3 -c "import yaml;
yaml.safe_load(...)"`)
- [ ] Next docs sync that produces review items should include the PR
link in Slack
Addresses three CR findings on PR #3980 — rebuilt from origin/main to
drop stale test-integration-tmp plumbing that was reintroduced in the
prior iteration.
Finding 1 — Scoped cancel-skip for concurrency group cancellations:
- Pre-build cancellation (detect-changes/check-lockfile cancelled):
stay silent, newer run will redo all work from scratch.
- Mid-matrix build cancellation: post a distinct muted info message
so humans can spot anomalies (fail-fast:false matrix legs rolling
up to `cancelled` when some legs silently passed via warn-not-fail
health-probe timeout).
Finding 2 — Per-service `health_path` with no silent fallback:
- Every service in ALL_SERVICES carries an explicit `health_path`
field (`/api/health` for Next.js, `/health` for aimock).
- Verify step fails loud with `::error::` + exit 1 when health_path
is unset, rather than silently defaulting to /api/health. An
unscoped fallback could mask a broken endpoint when an unrelated
catch-all / CDN / actuator happens to 200 at a different path.
Finding 3 — Comment accuracy:
- health_path comment now cites the policy ("no fallback — misconfig
is a config bug, not runtime behavior to hide").
- Cancel-skip comment enumerates all three cancellation causes
(concurrency supersede, manual cancel, upstream failure cascade)
and explains the pre-build vs mid-build split.
The smoke-monitor job has no actions/checkout step, so `gh workflow
run` cannot resolve the repo from a git remote and fails with:
failed to run git: fatal: not a git repository
Pass --repo ${{ github.repository }} so gh CLI skips the git lookup.
All 18 rebuild triggers in run 24534637524 failed for this reason.
Fixes 3 HIGH findings from R1 review on #3988:
1. pr_url empty was used as a proxy for "no PR opened because clean-transform
was empty", but it's also empty on every error path (bot-token failure, gh
pr create failure, push failure). Replace with an explicit pr_opened=true/
false output from the push step — true only after the PR URL is captured,
false only on the deliberate CHANGED=0 path. Error paths leave it unset so
alerts fall through to the failure() handler.
2. review_items_json was string-interpolated raw into a JSON payload inside
triple-backticks. Any filename containing ", \\, or a control character
would break the payload. Moved to a jq-based payload-file-path pattern:
a dedicated Build Slack payloads step writes each payload to disk with jq
--arg, so all values are safely JSON-escaped regardless of content. Slack
steps consume the tmpfiles via payload-file-path.
3. If a notify-* step itself fails (webhook 5xx, rate limit, malformed JSON),
the review-needed alert was silently lost — the existing failure() alert
was gated on pr_url == '' and would not fire. Added an unconditional
fallback step that posts a plain-text "alert machinery failed" message via
curl when any notify-* step's outcome is failure, so we never lose a
review-needed or failure notification.
The "files needing manual review" Slack warning listed files but
didn't link the auto-opened PR, forcing reviewers to hunt for it
in GitHub. Capture the PR URL from the create/merge step output
(already exposed as steps.push.outputs.pr_url) and include it as
a "Review:" line in the payload.
Split the alert into two variants:
- PR opened (normal case): includes the PR link
- No PR opened (edge case where clean-transform portion was empty):
posts review items without a link
Auto-sync and merge-failed alerts already linked the PR — this
brings the review-needed alert to parity.
## Summary
Follow-up hardening to #3971. aimock supports fixture schema validation
at startup via `--validate-on-load`, but the flag is **opt-in** and the
showcase Dockerfile was not passing it. That meant fixtures with
unrecognized response keys (e.g. `"text"` instead of `"content"`) loaded
silently and only surfaced as HTTP 500s at request time — which is
exactly what crashed crewai-crews and triggered #3971.
This PR wires up two independent safety nets so a broken fixture can't
ship again:
1. **Dockerfile (fail-fast at container boot)** —
`showcase/aimock/Dockerfile` now passes `--validate-on-load`. If any
fixture fails the aimock schema, the container exits non-zero instead of
starting and serving 500s. Railway will not promote a bad build.
2. **CI test (fail-fast in PR review)** — new vitest spec at
`showcase/scripts/__tests__/aimock-fixtures.test.ts` imports
`loadFixtureFile` + `validateFixtures` from `@copilotkit/aimock` and
asserts zero errors against both `feature-parity.json` and `smoke.json`.
Runs inside the existing ` Showcase: Validate` workflow
(`showcase/scripts` vitest suite) on every PR that touches
`showcase/**`.
## Verification
**Red-green on the vitest spec:**
- Rebased onto the tip of main *before* #3971 merged: the spec fails
with 5 errors — exactly the 5 broken `"text"` fixtures (`plan`, `steps`,
`mars`, `dashboard`, `report`) that #3971 repaired.
- Rebased forward onto main *after* #3971: spec passes with 0 errors,
all 549 showcase/scripts tests green.
**Red-green on the Dockerfile:**
- Current fixtures + `--validate-on-load`: container boots cleanly, logs
`Loaded 39 fixture(s) from /fixtures`.
- Injecting an intentionally broken fixture (`response: { "text": "..."
}`): container fails to start with `[aimock] Fixture 0: response is not
a recognized type (must have content, toolCalls, error, or embedding)` /
`Validation failed: 1 error(s), 0 warning(s)` and non-zero exit.
## Test plan
- [x] Local: full `showcase/scripts` vitest suite passes (549/549)
- [x] Local: `pnpm run test` (monorepo) passes
- [x] Docker: image builds and starts with `--validate-on-load` against
current fixtures
- [x] Docker red-green: broken fixture fails container start with
non-zero exit
- [ ] CI: ` Showcase: Validate` job runs the new test file on PR
## Summary
Fixes Slack alerts that rendered literal `\n` (backslash + n) instead of
actual newlines — messages looked like `*Starter Deployed Smoke Test
Failed*\nView run` in Slack.
Root cause: `jq -n --arg text "...\n..."` passes the two literal
characters `\` and `n` to jq (bash doesn't interpret `\n` inside double
quotes). `--arg` stores them verbatim; jq then JSON-escapes the
backslash, producing `"\\n"` in the payload, which Slack parses back to
the two-character string `\n` and renders as-is.
## Changes
Switched three alert builders from `jq --arg` with embedded `\n` to the
safer pattern already used in `showcase_smoke-monitor.yml`: write a
message file with real LF bytes via `printf`, then `jq -n --rawfile text
…` for correct JSON escaping.
Affected workflows:
- `starter_deployed_smoke.yml` — Starter Deployed Smoke Test Failed (the
alert from the screenshot)
- `starter-smoke.yml` — Starter smoke test failing: <starter>
- `showcase_drift-detection.yml` — Showcase E2E suite failed
## Enrichment (starter_deployed_smoke.yml)
The deployed-smoke failure alert was just `*Starter Deployed Smoke Test
Failed* | View run`. Now emits a Playwright JSON report, extracts
failures, and builds a richer payload:
- Failed starter slugs listed in the header (parsed from spec titles)
- Direct link to the failed job in addition to the workflow run
- Up to 5 failure entries, each with:
- starter slug
- test-level tags (`@starter-health` / `@starter-agent` /
`@starter-chat` / `@starter-tools`)
- first line of the error message (ANSI-stripped, 240-char cap)
- "…and N more failure(s)" footer when the count exceeds 5
Falls back to the minimal header when no JSON report exists (e.g.
pre-test setup failed) so alerts still fire.
The other two alerts already had a summary but now also include a "View
job" link for direct navigation.
## Test plan
- [ ] Trigger `starter_deployed_smoke.yml` via `workflow_dispatch`
against a starter known to fail (or simulate) and confirm Slack renders
real newlines plus the enriched payload
- [ ] Trigger `starter-smoke.yml` via `workflow_dispatch` (PR run skips
the Slack step) and confirm alerting format when a starter is forced to
fail
- [ ] Trigger `showcase_drift-detection.yml` via `workflow_dispatch` and
confirm Slack renders real newlines and the fenced code block
- [ ] Grep `.github/workflows/` for `jq -n --arg text ".*\\n"` — should
return zero matches
Supersedes #3912 (closed).
- Add --validate-on-load to all aimock invocations (4 workflows/scripts
+ 13 integration docker-compose files)
- Replace hardcoded 2-file fixture list with dynamic discovery across
showcase/, examples/integrations/*/, scripts/doc-tests/ (16 fixtures)
- Add sanity check to prevent silent zero-test pass when discovery fails
- Extend showcase_validate.yml path filter to trigger on
examples/integrations/**/fixtures/** and scripts/doc-tests/fixtures/**
- Import and use ValidationResult type for callback parameters
- Fix scripts/doc-tests/fixtures/default.json to use { fixtures: [...] }
envelope shape
- Summarize count of stale services in the normal case instead of
listing every image by name
- Only expand to the detailed per-service list when rebuild triggers
actually fail, including each service's error reason
- Track triggered_count and failed_count separately so the alert
accurately reflects what happened
- Exit 1 on rebuild-trigger failure so the workflow run shows red in
the Actions UI, and guard the generic failure() notifier with
has_stale != 'true' to prevent double-posting to Slack (the detailed
drift-alert step already covers the drift case)
Surface jq parse failures instead of silently emitting an empty list,
wrap capture() in try/catch so a non-matching title no longer poisons
the whole extraction pipeline, and iterate every test per spec so
multi-project configs don't drop failures.
Replace byte-level cut -c1-200 with head -c 200 | iconv UTF-8//IGNORE
so truncated summaries never emit partial UTF-8 sequences as mojibake.
Broaden the ANSI stripper to cover SGR, OSC, and G0/G1 charset
designator escapes in both sed and jq.
Use mktemp for slack message/payload files with an always() cleanup
step so self-hosted runners stay clean, route matrix.starter through
env for consistency with the existing pattern, and fix two misleading
comments (reporter behavior, cap-at-5 placement). Drop the unused
walk_suites helper.
The `jq -n --arg text "...\n..."` pattern passed the two literal characters
`\n` to jq, which preserved them as-is in the JSON string. Slack then
rendered the literal backslash-n instead of a line break, producing
messages like `*Starter Deployed Smoke Test Failed*\nView run`.
Switch the three affected alert builders to `printf` into a file with real
LF bytes and load via `jq -n --rawfile` so escaping is handled correctly.
This matches the pattern already used in `showcase_smoke-monitor.yml`.
Also enrich the Starter Deployed Smoke alert with:
- failed starter slug(s) in the header
- direct link to the failed job (not just the workflow run)
- up to 5 failure entries each showing: slug, test level tags
(@starter-health/@starter-agent/@starter-chat), first line of error
- "…and N more" footer when more than 5 failed
Enrichment is driven by a new JSON reporter output from the Playwright run;
if the report is missing (e.g. pre-test step failed) the step falls back to
the minimal header so alerts still fire.
Fixes the literal `\n` rendering seen in Slack for:
- starter_deployed_smoke.yml (Starter Deployed Smoke Test Failed)
- starter-smoke.yml (Starter smoke test failing: <starter>)
- showcase_drift-detection.yml (Showcase E2E suite failed)
## Summary
- **Problem 1**: Health check constructed URLs like
`showcase-X-production.up.railway.app` which never matched actual
Railway domains (many have hash suffixes like `-3f57`). Crashed services
silently passed health checks.
- **Problem 2**: No Railway deploy status check — only HTTP health was
checked, so CRASHED deployments were never caught.
- **Fix**: Replaced the URL-guessing health check with Railway API
polling that queries actual deployment status and real service domain.
Fails immediately on CRASHED, validates both Railway SUCCESS status and
HTTP 200 on the real domain.
## Test plan
- [ ] Trigger a `workflow_dispatch` deploy for a single service and
verify the health check step queries Railway API and logs status/domain
- [ ] Verify a healthy service shows `Railway status=SUCCESS` and `HTTP
check: ... → 200`
- [ ] Verify a crashed service (e.g. bad image) fails the job with
`::error::Service X CRASHED on Railway`
The previous health check constructed URLs as `${IMAGE}-production.up.railway.app`
which never matched Railway's hash-suffixed domains, so it silently
passed on crashed services (the issue that made claude-sdk-typescript
and mastra appear "deployed" while crashing at runtime).
New verification:
- Capture prior deployment ID before redeploying so verify step can
distinguish fresh deployment from stale (avoids false-positive where
first poll sees previous SUCCESS deployment and exits 0 immediately)
- Poll Railway API for actual deployment status, fail fast on terminal
failures (CRASHED, FAILED, REMOVED, SKIPPED)
- Use real staticUrl from Railway API instead of guessing URL pattern
- Hit /api/health instead of root (backends 404 at /)
- Require 2 consecutive 200 responses before declaring healthy (catches
SUCCESS-then-crash from JVM lazy init failures, Python OOM on first
request)
- Check GraphQL response body for errors (HTTP 200 + {errors:[...]}
is how Railway signals auth/query failures)
- Validate RAILWAY_TOKEN is set before polling
- 360s total budget (24 × 15s) to accommodate slow-boot services
(spring-ai, mastra)
## Summary
Ports 3 proven patterns from ag-ui's release system to CopilotKit.
### C1: Registry error vs 404 distinction
`getPublishedVersion` in `publish-release.ts` now distinguishes between
npm E404 (package genuinely not published — proceed) and real errors
(network timeout, auth failure, rate limit — stop). Previously all
errors returned `null`, silently bypassing the version guard.
### C2: Pre-existing tag check
Added a check in `publish-release.yml` that verifies the tag doesn't
already exist before attempting to create it. Prevents the "published
but no tag" state on retries.
### C3: Pre-publish tests in prerelease
Added `pnpm run test` between build and publish in the canary workflow.
A broken canary erodes trust in the prerelease channel.
**Cross-pollination context:** [Notion
page](https://www.notion.so/3413aa38185281828aa1dfa014808ddc)
The ag-ui side (strict version ordering, AI release notes, atomic PR
creation) ships via ag-ui PR #1487.
## Test plan
- [ ] Verify `getPublishedVersion` returns null on E404 but throws on
network errors
- [ ] Verify pre-existing tag check fails fast before publish
- [ ] Verify prerelease workflow runs tests before canary publish
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- The docs-sync warning notification was sending "see workflow run for
details" with no actionable information
- Now reads `review-items.txt` and includes the file list directly in
the Slack message
- Recipients can see which files need attention without digging through
CI logs
## Test plan
- [ ] Trigger docs-sync with a file that has showcase-local
modifications (exit code 3 path)
- [ ] Verify Slack notification includes the file list in a code block
- [ ] Verify auto-push-only path (exit code 0) still does NOT send the
warning notification
The warning notification for files needing manual review was sending
'see workflow run for details' with no actionable information.
- Read review-items.txt and include file list in the Slack message
- Use jq for proper JSON escaping (handles newlines, quotes, special chars)
- Guard against missing review-items.txt with fallback and ::warning::
- Review-needed notification fires independently of push/merge outcome
New workflow running starter health/agent/chat tests against Railway:
- Triggers: 6h cron, after showcase deploy, manual dispatch
- Alerts on schedule + workflow_run failures (Slack + GitHub issue)
- Issue dedup by title match, continue-on-error on Slack
- Proper working-directory for npm ci and Playwright install
Jordan Ritter