Revalidation against current main (the branch was 1345 commits behind):
- docs workflow ran `pnpm reconcile` but `add-paths` listed only the
fragment, so its PR would land a fresh fragment beside a stale
telemetry-events.json and fail the registry's telemetry-reconcile
staleness gate — the exact failure the runtime workflow was already
fixed for. Verified against the registry's shipped emitters: every
automated fragment PR there (website.corp, Intelligence surfaces)
carries telemetry-events.json alongside its fragment.
- Refresh the action pins to the SHAs main now uses everywhere
(checkout v7, setup-node v7.0.0, pnpm/action-setup v6.0.10).
- Narrow the docs trigger to code under shell-docs/src, excluding
src/content (1000+ MDX/JSON prose files that cannot hold a
posthog.capture call site) so prose edits stop firing a full install.
- Note in the zizmor justification why setup-node v7's new
package-manager-cache auto-path still leaves this workflow cacheless
(it engages only for npm-declared repos; this one declares pnpm).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A fragment-only PR fails oss-path-to-production's telemetry-reconcile gate (it
recomputes telemetry-events.json and fails on staleness). After emitting each
fragment, install the registry's deps and run pnpm reconcile, then include
telemetry-events.json in the PR alongside the fragment — matching the Intelligence
CLI + surface-emitter pattern.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The cross-repo fragment PRs must be authored by the dedicated telemetry-registry
GitHub App that's installed on oss-path-to-production (the same App the
Intelligence CLI release workflow uses), not CopilotKit's DEVOPS_BOT release bot.
Switch both workflows to app-id/private-key from secrets.TELEMETRY_REGISTRY_APP_ID
/ TELEMETRY_REGISTRY_APP_PRIVATE_KEY and gate the mint on the App ID env var.
These secrets must be added to the CopilotKit repo (they currently live only on
Intelligence).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds scripts/telemetry/ (emit-fragment.ts + extract.ts) and two CI workflows
that generate CopilotKit's telemetry-registry fragments and open path-limited
PRs into CopilotKit/oss-path-to-production:
- runtime (bespoke catalog): reads the AnalyticsEvents type map for event names
+ properties, scans capture() sites for call_sites, fails loud if the v1/v2
catalogs diverge. Triggered on stable monorepo release.
- docs (callee mode): extracts inline posthog.capture literals from
showcase/shell-docs (drops $-reserved events). Triggered on push to main
touching showcase/shell-docs/**.
Both are content-gated: the fragment is left untouched (and no PR opened) when
the event set is unchanged, so releases/edits don't churn the registry. Cross-
repo token follows the least-privilege recipe (no owner, bare repositories,
contents+PR write); mint gated on a job-level env var. zizmor clean (one
justified cache-poisoning suppression). 13 unit tests; tsc + oxlint clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
Adds a `react-version` matrix axis (**18**, **19**) to the unit-test
workflow so react-core, react-ui, and a2ui-renderer are exercised across
the full **supported peer range** (`^18 || ^19`), not just the
repo-default React 19.
This is a **reconstruction of the durable parts of #4221**
(@tylerslaton) onto current `main`. That PR went stale (~5,000 commits
behind, conflicting) and never landed. Rather than rebase it, this
rebuilds its design fresh — and deliberately **scopes to the supported
React range**: React 17 is dropped, because it is no longer a supported
peer version and carried ~80% of the original PR's complexity
(polyfills, `use-sync-external-store` source shims, `jsx-runtime`
aliases, a legacy `renderHook` fallback).
## What surfaced
Dropping R17 and validating R18 revealed a **latent React 18
incompatibility on current `main`**: the `window = {}` test pattern
crashes React 18's concurrent renderer with `"Should not already be
working."` mid-commit, which then corrupts the scheduler for the rest of
the file — **22 failures across 5 files** under React 18. (React 19
happens to tolerate the empty-window swap, so it was invisible until
now.)
The original PR fixed this but mislabeled it R17-only; it's actually
needed for R18, a *supported* version. So the matrix earned its keep on
day one.
Replacing `window = {}` with `stubWindowLocation()` is the load-bearing
fix — it resolves the crash cascade. Separately, **two** tests differ
under R18 purely in *render scheduling*, and are handled by narrow
version gates:
| Test | React 18 behavior | Why it's not a bug |
|---|---|---|
| `renderCustomMessages` → "executes multiple renderers in order" |
`executionOrder` is `["first", "first"]` | Renderer double-invoke.
`second` still never runs, which is the actual contract. |
| `use-human-in-the-loop` → `statusHistory` | `inProgress → executing →
inProgress → complete` | Transient backwards transition from extra
effect runs. Start, end, and the set of observed statuses are all still
correct. |
**No assertion tolerates a different state value.** An earlier revision
of this PR also relaxed the three-turn state-snapshot assertion to
accept `Turn: 2` on R18; @tylerslaton correctly flagged that as an
observable-behavior difference rather than a scheduling artifact.
Re-verified against a real 18.3.1 install — the strict `Turn: 3`
assertion passes **25/25** consecutive runs — so that gate was
unnecessary and has been removed (`a227f46a8`). The two gates above were
re-tested the same way and both genuinely reproduce.
## Changes
| File | What |
|---|---|
| `.github/workflows/test_unit.yml` | `react-version: ["18","19"]` axis.
R19 installs frozen; R18 overrides the root `pnpm.overrides` React
version and installs unfrozen. Adds a guard verifying the installed
React matches the matrix leg, and suffixes `NX_CI_EXECUTION_ID` with the
React version. Layered on top of the existing nx-affected selection
logic. |
| `test-helpers/stub-window-location.ts` *(new)* | Clears
`window.location` (so the localhost auto-open-inspector heuristic skips)
while keeping the real jsdom window — the safe replacement for `window =
{}`. |
| `use-agent-error-state`, `CopilotKitProvider.onError`,
`CopilotKitProvider.test` | Swap `window = {}` for
`stubWindowLocation()`. |
| `use-human-in-the-loop.e2e`, `renderCustomMessages.e2e` | Two
React-version-gated assertions, both **render-scheduling only** (see
table above). State assertions stay strict on every leg. |
No dependency or lockfile changes. None of the R17-only machinery from
#4221.
## CI cost
Full runs go from 3 legs (node 20/22/24) to **6** (node × react). On
PRs, nx-affected still scopes what actually builds/tests; the full 6×
only hits `workflow_dispatch` or when `test_unit.yml` itself changes (so
this PR runs all 6). This is the honest price of adding R18 coverage.
## Testing
Run locally in a worktree via the exact install-override logic the
workflow uses — `react`/`react-dom` → 18.3.1,
`@types/react`/`@types/react-dom` → `^18`, `@testing-library/react` →
`^14.3.1`, `streamdown>react` → 18.3.1, then `pnpm install
--no-frozen-lockfile`. Installed versions confirmed by resolving from
`packages/react-core` (18.3.1 / 19.2.3, `@testing-library/react` 14.3.1
on the R18 leg).
| Check | Result |
|---|---|
| react-core full suite @ React 18.3.1 | **117 files, 1433/1433
passing** ✓ |
| react-core full suite @ React 19.2.3 | **117 files, 1433/1433
passing** ✓ |
| Strict `Turn: 3` state-snapshot assertion @ R18, ×25 runs | **25 pass
/ 0 fail** — gate removed as unnecessary |
| `executionOrder` gate reverted to strict @ R18 | **fails**
(`['first','first']`) — gate justified |
| HITL `statusHistory` gate reverted to strict @ R18 | **fails** (extra
`inProgress`) — gate justified |
| `oxlint` (project-aware) | **0 warnings, 0 errors** — unchanged from
`main` |
| `oxfmt --check` | clean |
| Workflow YAML parse + lefthook commit hooks (lint-fix, package tests,
commitlint) | green |
Before the `window` fix, the R18 leg was **22 failing across 5 files**;
it is now fully green.
Credit to @tylerslaton for the original design in #4221, and for
catching the over-relaxed state assertion in review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
Brings the `setup-slack-channel` skill into the set that `copilotkit
skills install` distributes, and narrows both it and
`copilotkit-channels` so they stop matching the same request.
## Why
The skill was merged into
[CopilotKit/channels-sdk#7](https://github.com/CopilotKit/channels-sdk/pull/7),
which makes it reachable from a channels-sdk checkout and nowhere else.
`copilotkit skills install` reads `CopilotKit/CopilotKit/skills`, so
until the skill lives here, no CLI user can get it.
No CLI change is needed to pick it up: skill names are free-form
(`--skill` is validated by regex only, with no allowlist) and the
default install is `--skill *`.
## How
**The skill is copied from the merged channels-sdk source**, with one
addition: a `version: 1.0.0` frontmatter field, so it matches the other
standalone skills in this repo (every one of them carries a version;
only the package-mirrored skills omit it). The body, references, and
manifest asset are otherwise unchanged. It stays browser-first and makes
no reference to the `copilotkit channels` commands, per the call to
defer those to the web app until they have more real-world usage.
**Both descriptions are narrowed** so the trigger overlap is gone:
| Skill | Owns | Hands off |
| --- | --- | --- |
| `copilotkit-channels` | The code half — declaring, wiring, and
customising a Channel. Assumes the provider app exists | First-time
Slack setup → `setup-slack-channel` |
| `setup-slack-channel` | The provider half — the Slack app, its tokens,
attaching it to a Channel | Code questions → `copilotkit-channels` |
Before this, both matched "connect my agent to Slack" and the agent
picked whichever it read first. That collision was invisible in
channels-sdk, where `setup-slack-channel` is the only skill present; it
becomes live the moment both ship in the installed set.
## Notes for reviewers
- **Scope is Slack only.** A Teams sibling is deliberately a separate
pass.
- **No existing skill content changes** — the `copilotkit-channels` diff
is its frontmatter description and nothing else.
- Two known follow-ups are tracked and intentionally not addressed here:
[channels-sdk#9](https://github.com/CopilotKit/channels-sdk/issues/9)
(CLI-capability claims, deferred by decision) and
[channels-sdk#2](https://github.com/CopilotKit/channels-sdk/issues/2)
(`build-channels-bot` staleness, unrelated).
Refs CopilotKit/channels-sdk#10
## Added during the `main` merge
Resolving the conflict surfaced a second, unrelated problem that had to
be fixed for this PR to be safe to land.
`skills/setup-slack-channel` is a **standalone** skill — it has no
`packages/*/skills/` source. `scripts/sync-plugin-skills.ts` treats any
such directory as an orphan unless it is listed in
`RESERVED_LIFECYCLE_SLUGS`, which this one was not. Verified against the
pre-fix script:
- `pnpm check:plugin-skills` → exit 1, `orphan file(s) in mirror:
skills/setup-slack-channel`
- `pnpm sync:plugin-skills` (write mode) → **recursively deleted all 8
files of the new skill**
The `plugin-skills-check` workflow's path filter does not match
`skills/setup-slack-channel/**`, so this PR would not have caught it —
it would have gone red on the next unrelated PR that touched the script
or a package skill, or silently eaten the skill on the next sync run.
Fix is two lines: add the slug to `RESERVED_LIFECYCLE_SLUGS`, and move
the paired `size` assertion in
`scripts/__tests__/sync-plugin-skills.test.ts` from 9 to 10. The test
file already documents this exact hazard in a comment.
### Conflict resolution
The only conflict was the `copilotkit-channels` frontmatter description,
which #6320 rewrote in parallel. The two sides disagreed about Teams:
this branch said the skill "assumes the provider app already exists",
while #6320 established that Teams provider setup **is** this skill's
job because the CLI or dashboard wizard performs it. The resolution
keeps this branch's code-half framing and the `setup-slack-channel`
handoff, but scopes that handoff to *first-time Slack app creation* only
— so it contradicts neither #6320's Teams sections nor the Slack
provider troubleshooting that stays in this file. Took #6320's `version:
1.1.0`.
Addresses review on #6340.
Interactivity is no longer disabled on the managed path. The shared generator
emits `interactivity.is_enabled: true` with an Intelligence-hosted request URL
(Intelligence `libs/channels-setup/src/slack.ts:209-211`) and the ingress
handles `block_actions` (`apps/app-api/src/routes/channels-routes.ts:868`), so
HITL buttons and selects do fire. The skill's own bundled manifest asset already
said `is_enabled: true`, so the prose contradicted the file shipped beside it.
What is still undelivered is `slash_commands` (absent from the generator) and
`view_submission` (`apps/app-api/src/channels/slack-ingress.ts:1050`), so the
`onCommand` / `onModalSubmit` warnings stay. Corrected in all three places that
claimed otherwise, and the troubleshooting entry now tells the reader a dead
button is a real failure rather than a capability limit.
Browser-only framing is now a routing instruction rather than an architectural
claim, since `copilotkit channels add` does create the Channel and attach the
adapter. Same outcome, but it no longer contradicts `--help` for an agent that
was told to check it.
Trigger scope is narrowed in the frontmatter description instead of rewriting
Phase 0. The phases assume OpenTag conventions (`app/channel.tsx`, `app/env.ts`,
`INTELLIGENCE_CHANNEL_NAME`, an agent on port 8123), which are not what
`copilotkit init` scaffolds — naming that in the description keeps the skill from
firing on any "connect my agent to Slack" once it installs into customer repos.
Also widens the plugin-skills-check path filter to `skills/**` rather than
adding the one new slug. The orphan scan reads the whole mirror, so enumerating
individual directories is what let this PR's own blocker go untested and would
have armed it again for the next standalone skill.
`bun-version: latest` let a Bun release change module-resolution behaviour
between runs. 1.3.14 is the version the recent passing and failing runs both
resolved, so pin it and make the job reproducible.
The publish job reinstalled all 4608 projects' dependencies (~41s) to run
two things: `pnpm tsx` and `pnpm pack`.
A root-only install is NOT sufficient — verified locally that pnpm pack
then dies with ERR_PNPM_CANNOT_RESOLVE_WORKSPACE_PROTOCOL, because pack
resolves each package's `workspace:` deps into real version ranges and
needs its workspace siblings installed to do it. That resolution is what
makes cross-scope canary deps pin the same-run version, so it is
load-bearing.
The true minimum is root + packages/**: measured 17s vs 46s locally, and
the packed tarball still carries resolved ranges (^0.4.0, ^1.64.1) with no
leftover workspace: refs. Everything trimmed is examples/ and showcase/ app
dependencies that no publish step touches.
Safe because no publishable package declares prepack or prepare, so pnpm
pack runs no lifecycle scripts; voice's prepublishOnly fires for neither
pnpm pack nor `npm publish <tarball>`.
Stable keeps the full install — its dependency surface is wider
(lib/notion.js, the umbrella verifier's root script) and it is the
highest-stakes path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
canary.yml mirrors the source branch to a unique
canary/<slug>-<run_id>-<attempt> ref, and Actions scopes cache entries to
the writing branch (plus the default branch). A cache saved from a canary
ref is therefore unreachable by every later canary — measured at 874 MiB
of orphan per run, competing for the repo's shared 10 GB budget and
evicting entries other workflows depend on.
It also cost more than it bought: 20s + 26s of post-job saves against a
25s faster install.
Canaries now restore read-only (free win when main has an entry, zero
cost and zero pollution when it doesn't); stable releases, which run on
main where a write is durable and reused, do the writing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The canary flow took ~11.5 min steady-state (and 20 min in an observed
run). Measured from run 30473499191, the time went to five avoidable
places rather than to real work.
1. `npx --yes npm@11.15.0 publish` ran per package, and npx re-resolves
the spec against the registry on EVERY invocation: ~16s of each
package's ~21s. A 9-package channels canary paid ~2.4 min of pure npx
overhead; a 16-package monorepo release paid over 4 min. Hoist the
pinned npm into lib/npm-cli.ts, install it once into a throwaway
prefix, and reuse the binary.
2. publish-release.yml was the only workflow in the repo with no pnpm
store cache, so all three jobs installed 4608 packages cold every
time. Usually ~45s each, but registry-bandwidth bound and heavy
tailed: the observed run spent 9m08s here on tarballs arriving at
2-49 KiB/s. Add the same node-version-keyed cache the rest of CI uses.
3. The notify job ran for canaries only to compute "post nothing" — the
builder already returns should_post=false for mode=prerelease and the
self-watchdog is already gated off. ~85s of dead work on the critical
path, since canary.yml waits for the whole run. Skip the job, keeping
it reachable for a python_publish dispatch.
4. The build job fetched full history for canaries, which need none (no
tag, no GH Release, no release-note commit range, and `nx run-many`
resolves no merge base). That rode along in the 837 MiB workspace
artifact too. Shallow-fetch prereleases; stable keeps depth 0 because
its publish job pushes tags out of that artifact's .git.
5. Two smaller ones: the artifact was gzipped and then re-deflated into
the artifact zip (compression-level: 0), and the orchestrator's
run-discovery loop slept 6s before its first poll.
Verified: 143 release-script tests pass (6 new for the npm-cli helper),
actionlint + shellcheck + the scope-dropdown guard are clean, the
prerelease dry-run path still enumerates all 9 channels packages, and a
live probe confirms the helper installs npm 11.15.0 once (3.2s) and
memoizes thereafter (0ms).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
## The race, with evidence
Three PRs merged within 34 seconds on 2026-07-26. `Showcase: Build &
Push` has **no concurrency group** (deliberately), so all three ran
simultaneously and raced to push the same `:latest` tags.
| run | commit | PR | start → end |
|---|---|---|---|
| `30190815370` | `7b28934387` | #6162 | 06:18:05 → 06:29:44 |
| `30190823203` | `59f275eedc` | #6161 | 06:18:21 → 06:29:44 |
| `30190831480` | `db75a04837` | #6158 | 06:18:39 → **06:29:13** ←
newest, finished FIRST |
The newest commit finished first, so the older builds overwrote its
`:latest`. Per-service job completion times — older beating newer on
every shared slot:
| service | newer (`db75a04837`) | older (`59f275eedc`) | older won by |
|---|---|---|---|
| `shell-dashboard` | 06:25:33 | 06:25:34 | +1s |
| `showcase-harness` | 06:27:51 | 06:27:55 | +4s |
| `shell` | 06:27:17 | 06:27:28 | +11s |
| `shell-dojo` | 06:25:10 | 06:25:26 | +16s |
**All three runs reported `success`.** Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
Same failure class as #6171: a success that doesn't mean what it says.
## What I verified in YAML vs took on trust
Verified by reading the files / querying the API:
- **Tagging** — `showcase_build.yml` pushed `:latest` **and** `:${{
github.sha }}` in one `depot/build-push-action` step, in both the
`build` and `build-starters` matrices. **A per-commit sha tag already
existed**; confirmed in GHCR (`showcase-shell-dashboard` has digests
tagged `db75a04837…`, `59f275eedc…`, `d28384a2eb…`).
- **Nothing serialized the pushes.** No concurrency group; confirmed the
header comment states this is intentional.
- **Deploy consumes `:latest`** — `verify-railway-image-refs.ts` is the
SSOT assertion: staging is `ghcr.io/copilotkit/<repo>:latest` (mutable),
**prod is `ghcr.io/copilotkit/<repo>@sha256:<digest>` (already immutably
pinned)**. So this race is **staging-only**; prod was never exposed.
- **`Showcase: Verify Deploy` structurally cannot catch it.** It is a
health probe; it asserts no digest or commit provenance anywhere. Its
#6171 per-commit concurrency key is about *which run verifies*, not
*what image is running*. A stale-but-healthy service passes.
- **The racing runs build DISJOINT service sets** (see below) — I pulled
the actual job lists.
Taken on trust: nothing material. The issue description matched the API
on every point I checked.
## Why NOT a concurrency group
`detect-changes` builds a **per-push, path-filtered** matrix, so
concurrent runs build overlapping but **non-identical** service sets:
- `7b28934387` → ag2, agno, built-in-agent, claude-sdk-python,
claude-sdk-typescript, crewai-crews, langgraph-fastapi,
langgraph-python, langroid, llamaindex, mastra, pydantic-ai, shell-docs,
spring-ai, strands (**15**)
- `db75a04837` → crewai-crews, llamaindex, shell, shell-dashboard,
shell-docs, shell-dojo, showcase-harness (**7**)
`cancel-in-progress: true` would have cancelled the `7b28934387` run and
the ~10 services **only it builds would never have shipped** — trading a
stale-image bug for a never-shipped bug. Per [GitHub's
docs](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#concurrency),
`cancel-in-progress: false` is no better: *"Any previously pending job
or workflow in the concurrency group will be canceled"* — with three
rapid merges the **middle** commit's build is dropped outright. GitHub
also does not guarantee FIFO ordering of queued runs.
Concurrent runs here are **not redundant**, so they must not be
cancelled.
**Re the #6171 interaction:** this design does **not** use
`cancel-in-progress`, so no build is ever superseded-and-cancelled and
the cancelled-slot notifier is never tripped by this change. That
interaction stays theoretical — deliberately.
## The design
Make the one shared mutable resource monotonic instead of serializing
the fleet.
1. The build step pushes **only** the immutable `:<sha>` tag, plus an
`org.opencontainers.image.revision` label.
2. A guard resolves the commit behind the current `:latest`, asks GitHub
`compare/<theirs>...<ours>`, and advances `:latest` (registry-side
retag, no pull) **unless ours is `behind`** — i.e. `:latest` already
holds a descendant and moving it would roll staging back.
**Fails open by design.** No `:latest` yet, unlabelled legacy image,
unreachable API, diverged history → advance. A stuck `:latest` is the
very failure being fixed, so it declines only on *positive proof* of
regression.
**Placement:** in `redeploy-staging` / `redeploy-staging-starters`,
immediately before the Railway pull that consumes `:latest` — not per
build slot. Those jobs already have Node (build slots do **not**, so
per-slot would mean an unpinned `npx tsx` fetch on ~50 parallel
runners), and deciding right before the pull makes the window as narrow
as possible. The image list is the **same matrix ∩ build-success
intersection** that decides what gets redeployed, so a failed build can
never move a tag.
Also adds `showcase_build.yml` to `showcase_validate.yml`'s trigger
paths — the new test asserts against that file's live text, and without
the path a PR re-adding `:latest` would never run the test that catches
it.
## Tradeoffs / what stays open
- **Residual sub-second TOCTOU.** GHCR has no compare-and-swap on tags,
so two runs reading `:latest` simultaneously could still both advance.
This narrows the window from the whole build (~10 min) to inspect→retag.
Fully closing it means retiring the mutable staging tag and pinning
staging to digests the way prod already is — a change to the Railway
image-ref SSOT contract, not a workflow change. **Recommended
follow-up.**
- **The guard is inert for one build per image.** Today's `:latest`
images carry no labels (verified: `showcase-shell-dashboard:latest` has
no `Labels` at all), so the first post-merge build fails open and
advances unconditionally — same as today. Protection starts from the
second build of each image.
- Failure to retag exits non-zero, redding the redeploy job and stopping
the deploy. That is intended: redeploying against a tag that did not
move is exactly the silent false green being fixed.
## Proof
**Red/green on the live YAML.** `advance-latest-tag.test.ts` parses the
real `showcase_build.yml` (extending #6171's `redeploy-guard.test.ts`
pattern). Reverting the workflow to its pre-fix state: **14 failed / 23
passed**. With the fix: **37 passed**. Full `showcase/scripts` suite:
**2382 passed, 73 files**.
**The load-bearing predicate, verified live against the real incident
commits:**
```
compare/db75a04837...59f275eedc => behind (older run arriving late → DECLINE)
compare/59f275eedc...db75a04837 => ahead (newer run → advance)
compare/db75a04837...db75a04837 => identical
```
**Label reading, verified against a real multi-platform registry image**
— `docker buildx imagetools inspect ghcr.io/astral-sh/uv:latest` piped
through `extractRevisionLabel()` returns
`3010295ae7ff572de459987ad70db315a62ecd61`, matching `jq` exactly. The
platform-keyed shape is handled.
**Shell/jq transforms** exercised directly, including the empty-CSV edge
case (empty → empty, step skipped by its `if:`).
**Lint:** `actionlint` finding counts byte-identical to the pre-change
baseline (no new findings; the 11 pre-existing are unrelated). `zizmor
--min-severity low` with the repo config: **no findings**.
**Typecheck:** both new files are in `showcase/scripts/tsconfig.json`'s
include set and produce **zero** errors. Worth stating plainly: `nx
run-many -t check-types` **does not reach `showcase/scripts`** — the
project isn't in the nx graph and has no `check-types` target (there are
9 pre-existing type errors in sibling files, which is how I confirmed
it). So the typecheck above is mine, not CI's. The *tests* are gated:
`showcase_validate.yml` runs bare `pnpm exec vitest run` in
`showcase/scripts`, which auto-discovers the new file.
### What I could NOT prove
**I did not construct a real concurrent race on scratch branches.**
Doing it faithfully needs two builds pushing the same GHCR repo with
controlled finish ordering, which means merging to `main` — the only
branch the build workflow triggers on. No run IDs for a live race
demonstration; I am not implying one.
Unproven until this runs on main: that `docker buildx imagetools create`
retags cleanly under the runner's GHCR credentials, and that `npx tsx`
behaves in the redeploy jobs (it is already the established invocation
there — `redeploy-env.ts` — so this is low risk, not zero).
## Normal single-merge builds are unaffected
No concurrency group is added, so nothing queues or cancels. A lone
merge finds `:latest` at its own parent → `ahead` → advances, exactly as
before. Cost is one `imagetools inspect` + one `gh api` + one
registry-side retag per built service, in a job that already exists — no
extra job, no extra checkout, no change to build parallelism.
---
Branched from `db75a04837`; #6156/#6159 landed after, so this will need
main merged in before it goes green.
Probable conflict with the concurrent `git lfs pull` work in
`showcase_validate.yml` — my edit there is only the top-level `on:
paths:` list, so it should merge cleanly, but flagging it.
The previous suite was 37/37 green against a workflow that could not have
run at all: the guard step had no `packages: write` and no GHCR login, so
every `imagetools create` would have 401'd. Asserting that a step EXISTS
proves nothing about whether it can succeed. Each change below was checked
by reintroducing the defect and confirming the test goes red.
Registry-auth precondition (the miss that let the above ship)
- Both redeploy jobs must declare `packages: write` AND a `docker/login-action`
step for ghcr.io, ordered BEFORE the guard. Dropping either reds the suite.
The intersection, executed rather than restated
- The old test only checked that the string `images=` appears in the compute
step; swapping its jq for the full matrix — so a FAILED build moves
`:latest` — kept it green. The real `changed` shell now RUNS, against the
real ALL_SERVICES matrix read out of the workflow, and the emitted
$GITHUB_OUTPUT is asserted: a failed build is in neither set, and the
`skip_build` slot is in `services` but NOT in `images` (handing it to the
guard fails "manifest unknown" and blocks the redeploy for the whole fleet).
Same treatment for the starter lane.
Fixtures joined to real registry output
- `extractRevisionLabel` was pinned to hand-written payloads that were never
compared with reality. Since `readLatestRevision` maps every failure to
null, and null ADVANCES, a parser that silently never matches yields a
permanently-blind guard with a fully green suite. Both fixtures are now
verbatim `docker buildx imagetools inspect --format '{{json .Image}}'`
output (buildx v0.35.0), unformatted so key order survives: a platform-keyed
multi-arch image carrying the label, and our own `:latest`, which turns out
to be a bare config object with NO labels at all. The label key used in the
injection test is read from the workflow's own `labels:` input, so parser
and producer cannot drift apart silently.
Closed vacuities
- The `:latest` ban read one step's `tags` and was vacuously green on an empty
list. It now covers every tagged step plus hand-rolled `docker push`/`docker
tag`/`imagetools create`, and fails on an empty `tags`.
- The `cancel-in-progress` ban read only top-level config; a job-level
`concurrency` on `build` reproduced the exact harm and stayed green. All
jobs are checked now.
- `already-current` is keyed on the digest, matching the script.
- The unreachable `(null, "ahead")` row is labelled as the defensive
input-space case it is, and a new test pins that the real flow never calls
compare() with an unknown revision.
New coverage: classifyProbeFailure (incl. a REAL execFileSync timeout, and
the two precedence traps — gh's rate-limited 403 is throttling, not auth; a
registry's 404-with-denied is auth, not absent), readFlag, escapeAnnotationData
(a forged `\n::error::` stays inert), isDirectInvocation through a symlink,
the `::error` annotation on fleet failure, and the digest-mismatch advance.
Also: GITHUB_SHA / GITHUB_REPOSITORY are no longer shadowed in the workflow.
The runner exports both and the script reads process.env, so the old
assertions pinned a redundancy rather than a capability — removed together,
as the comment there required. Workflow-reading scaffolding duplicated with
redeploy-guard.test.ts is extracted and the YAML parse memoized.
Reverts only the cosmetic third item from the previous commit. Dropping the
redundant GITHUB_SHA / GITHUB_REPOSITORY shadowing is functionally correct --
the runner exports both and advance-latest-tag.ts reads them off process.env
-- but showcase/scripts/advance-latest-tag.test.ts asserts those exact keys
are declared on the guard step, so removing them red "Validate Showcase".
RED (run 30238078822, "Validate Showcase" > Run build pipeline tests):
Failed Tests 2
FAIL advance-latest-tag.test.ts > showcase_build.yml - redeploy-staging job
> runs the guard with the image list, sha, repo and a token
FAIL advance-latest-tag.test.ts > showcase_build.yml - redeploy-staging-starters
job > runs the guard with the image list, sha, repo and a token
AssertionError: expected undefined to be '${{ github.sha }}'
- Expected: "${{ github.sha }}"
+ Received: undefined
advance-latest-tag.test.ts:424
GREEN (same six contract assertions per job, replicated against the YAML --
the test is pure yaml-parse, no runtime deps):
redeploy-staging / redeploy-staging-starters
PASS IMAGES PASS GITHUB_SHA PASS GITHUB_REPOSITORY
PASS GH_TOKEN truthy PASS guard `if` PASS guard BEFORE redeploy
The test file is owned by a concurrent agent on this PR, so the assertions
cannot be relaxed here. If the redundancy is worth removing, the env keys and
the assertions must go together in one change.
Everything else from the previous commit is untouched and was already
validated by that CI run -- only these two assertions failed, so the GHCR
login, `packages: write`, the skip_build/images filter, the timeout bump and
the tsx pin all passed.
Lint unchanged: actionlint 10 findings before and after (0 new); zizmor
CI-equivalent reports "No findings" before and after.
Two Critical defects in the new "Advance :latest" guard, both of which would
have broken staging redeploys on this PR's own merge commit.
1. Registry auth missing. `redeploy-staging` and `redeploy-staging-starters`
run `docker buildx imagetools create` (an authenticated GHCR WRITE) but
declared only `permissions: contents: read` with no `docker/login-action`
step. Every retag would 401, the guard would exit 1, the redeploy step
would be skipped -- and since the build step no longer pushes `:latest`
itself, `:latest` would freeze permanently and staging would never
redeploy again. The same missing auth also 401s `imagetools inspect` on
these private packages, which the guard treats as "unknown revision" and
advances anyway: it would have failed OPEN and never actually guarded.
Fixed by adding `packages: write` plus a GHCR login mirroring the `build`
job's existing "Login to GHCR" step. Restores both the read and write path.
2. skip_build slots entered the retag list. The `images` CSV projection did
not exclude `skip_build` slots. `webhooks` is built out-of-band, never
pushes a `:<sha>` tag, yet reports job.status: success -- so it reached the
guard, `imagetools create` failed with "manifest unknown", and the whole
fleet's redeploy was blocked. Fires on `service=all` dispatch and via the
`workflow_config` path filter.
Fixed by carrying `skip_build` through the jq projection and filtering it
out of `images` ONLY. It stays in `services`: bouncing webhooks so Railway
re-pulls its out-of-band `:latest` is the documented intent (see the
`webhooks` entry in showcase/scripts/railway-envs.ts).
Also, same file:
- timeout-minutes 5 -> 20 on both jobs. The guard adds up to 3 serial
round-trips per slot (~84 at full fleet width). An overrun concludes
`cancelled`, which is invisible to both `if: failure()` and
`if: cancelled()` downstream -- the alerting-suppression class this repo
has been fixing.
- `npx tsx` -> `npx --yes tsx@4.21.0` (the pnpm-lock-resolved version),
matching the no-drift convention showcase_validate.yml enforces via
`pnpm exec tsx`. `pnpm exec` is unavailable here: no workspace install.
- Dropped the redundant GITHUB_SHA / GITHUB_REPOSITORY shadowing; the runner
exports both and advance-latest-tag.ts reads them off process.env.
Deliberately NOT added: a `concurrency` group. Its absence is load-bearing --
detect-changes builds a path-filtered per-push matrix, so concurrent runs
build disjoint service sets and cancelling the older run would leave services
never shipped rather than merely stale.
RED/GREEN
---------
Workflow auth cannot be fully proven without merging, so the proof is scoped
to what is actually demonstrable locally. Both probes lift the code under test
VERBATIM out of the workflow at runtime rather than paraphrasing it.
Probe A -- the real jq projection, fed the real ALL_SERVICES array with a
`service=all` / `workflow_config` dispatch (every slot selected, every slot
status:success, which is exactly what the skip_build slot reports):
RED images : ...,ghcr.io/copilotkit/showcase-eval-webhook (28 entries)
services: ...,webhooks
-> webhooks IS in the retag set. imagetools create -> manifest unknown.
GREEN images : ends ...,ghcr.io/copilotkit/showcase-pocketbase (27 entries)
services: ...,webhooks (unchanged, still 28)
-> webhooks OUT of images, STILL IN services. Exactly the split wanted.
Probe B -- structural parse of the workflow, over every job that runs
advance-latest-tag.ts:
RED redeploy-staging permissions={'contents':'read'} login=NO
redeploy-staging-starters permissions={'contents':'read'} login=NO
VERDICT: 2 retag job(s) lacking GHCR auth
GREEN redeploy-staging permissions={'contents':'read',
'packages':'write'} login=YES
redeploy-staging-starters permissions={'contents':'read',
'packages':'write'} login=YES
VERDICT: 0 retag job(s) lacking GHCR auth
Lint (delta vs the pre-fix file, not raw counts)
------------------------------------------------
actionlint : 10 findings before, 10 after; 0 new, 0 removed (all pre-existing
depot runner-label + SC2086 noise).
zizmor : CI-equivalent (--min-severity low, default persona, repo config)
reports "No findings" both before and after. Auditor persona at
all severities is also identical before/after (1 concurrency-limits,
1 template-injection, 3 undocumented-permissions, 6
anonymous-definition, 9 secrets-outside-env) -- the two new
`packages: write` entries carry trailing justification comments so
they add no undocumented-permissions findings.
Three merges landed on main within 34 seconds on 2026-07-26. The build
workflow has no concurrency group by design, so all three ran at once and
raced to push the same `:latest` tags. The NEWEST commit's build finished
FIRST, so the two older builds overwrote it:
run 30190815370 7b28934387 (#6162) 06:18:05 -> 06:29:44
run 30190823203 59f275eedc (#6161) 06:18:21 -> 06:29:44
run 30190831480 db75a04837 (#6158) 06:18:39 -> 06:29:13 <- NEWEST
Per-service, the older commit beat the newer one on every shared slot:
shell-dashboard (+1s), showcase-harness (+4s), shell (+11s), shell-dojo
(+16s). All three runs reported `success`. Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
A concurrency group is the wrong fix. `detect-changes` builds a per-push,
path-filtered matrix, so concurrent runs build overlapping but NON-IDENTICAL
service sets -- 7b28934387 was the only run building ag2, agno, langroid,
spring-ai, strands and 10 others. `cancel-in-progress: true` would have
dropped those entirely, trading a stale-image bug for a never-shipped bug.
`cancel-in-progress: false` is no better: GitHub keeps at most one pending
run per group and cancels any previously-pending one.
Instead make the mutable pointer monotonic. The build step now pushes only
the immutable `:<sha>` tag plus an `org.opencontainers.image.revision`
label. A new guard then advances `:latest` unless the tag already points at
a DESCENDANT of the commit being built, which is exactly the regression
case. Every ambiguous state (no tag, unlabelled legacy image, unreachable
API, diverged history) advances -- a stuck `:latest` is the failure mode
being fixed, so the guard declines only on positive proof of regression.
The guard runs in redeploy-staging / redeploy-staging-starters, immediately
before the Railway pull that consumes `:latest`, over the same
matrix-intersect-build-success set that decides what gets redeployed. That
placement keeps the read->retag window minimal and avoids an unpinned
`npx tsx` fetch on ~50 parallel build slots that have no Node.
Also adds showcase_build.yml to showcase_validate.yml's trigger paths: the
new test asserts against that file's live text, and without the path a PR
re-adding `:latest` to the push step would never run the test that catches it.
Prod is unaffected -- verify-railway-image-refs.ts already pins prod to
`@sha256:<digest>`; only staging consumes the mutable tag.
PR #6163 converted showcase/integrations/*/public/demo-files/* to Git LFS.
showcase_validate.yml's python-unit-tests job checks out without LFS, so the
working tree holds 129-byte pointer text instead of the real assets. The new
ms-agent-python multimodal test is the first Python test to read those bytes:
pypdf reads the pointer, extracts nothing, and the test fails with
"real pypdf text extraction produced nothing".
Fetch only the demo-file assets (40 objects, ~320 KB) rather than setting
lfs: true, which would pull all ~248 tracked objects (~475 MB, including
13-26 MB README gifs) into a 2-4 minute job under a 10 minute cap. A job that
hits timeout-minutes reports 'cancelled' -- neither success nor failure -- and
silently suppresses alerting, so the timeout margin is worth protecting.
The repo is public, so LFS downloads resolve anonymously and the pull works
with persist-credentials: false. A post-pull %PDF- header check fails the step
immediately if the pull ever no-ops, instead of surfacing minutes later as a
misleading pytest assertion.
Root cause of the cancellations, from the Depot logs of the killed slots:
they were not a transient flake. Every one died at EXACTLY its
`timeout-minutes` value (shell/shell-docs/shell-dashboard/shell-dojo at
10m04s of a 10-minute budget; showcase-aimock at 5 minutes), and the log
ends with `failed to solve: Canceled: context canceled` followed by
Depot's `Step canceled by GitHub`. GitHub reports a `timeout-minutes`
kill as job conclusion `cancelled`, which is what fed the whole silent
chain.
The builds were making real but slow progress the entire time — repeated
docker.io base-image pull stalls, and the final `generate-registry` layer
alone took 54s — and shell was still 2 steps from done when the budget
killed it. The `context: "."` slots build the whole monorepo root and are
the heaviest in the fleet, yet they carried the SMALLEST budgets (10 min
vs 15 for each small per-integration build). Measured, same slots:
warm / uncontended ......... 2-5 min
contended, killed at ....... 10 min (runs 30162773601, 30162770765)
showcase-pocketbase ........ 8.8 min of a 10 min budget (near-miss)
So a retry is the wrong lever twice over: a cancelled job cannot run
further steps, and the build was not erroring. Give the four shell slots
20, showcase-aimock 12, and showcase-pocketbase 15. `webhooks` stays at 5
(it is skip_build, measured at 0.6 min).
This is mitigation, not the root fix — the underlying cause is Depot
builder contention from three concurrent full-fleet rebuilds (~84
simultaneous amd64 builds). Noted inline as a follow-up, since the
durable lever is a non-cancelling concurrency queue and this workflow
deliberately has no concurrency group.
Two independent ways verification was silently lost on 2026-07-25.
1. `resolve-matrix` required `github.event.workflow_run.conclusion ==
'success'`. But `redeploy-staging` gates on the artifact-derived
`any_success`, not on the matrix rollup, so a build with some slots
cancelled (rollup `cancelled`) or failed (rollup `failure`) still
pushes real images and still redeploys the slots that built. Run
30162773601 redeployed 23 services to staging and this workflow never
started. An unverified real deploy is worse than a verified partial
one, so the trigger is now "the build reached a terminal conclusion"
and WHAT to verify stays decided by evidence: no redeploy-summary
artifact still means has_services=false and verify is skipped, and the
redeploy gate still narrows to the per-service success set, so a
cancelled slot can never be probed against a stale `:latest`.
2. The `showcase-verify-deploy` concurrency group was global, so any
later-finishing build preempted an earlier run's verification even
though they verify DIFFERENT commits. Verify run 30163309977 for the
one genuinely successful build of the day (#6168, 7282ecddf) started
at 15:17:13 and was cancelled 9 seconds later by run 30163312882 —
which was triggered by a different, partially-cancelled build and then
skipped every job anyway. Key the group per triggering commit.
Build run 30162773601 (merge of #6160) forced a full-fleet rebuild; 5 of
28 slots were killed by their `timeout-minutes` budget, the other 23
built and WERE redeployed to staging, and the run emitted no signal at
all: `notify` was skipped, so no Slack alert and no PR comment, and the
run rolled up to conclusion `cancelled`.
No existing guard could catch it. Measured on purpose-built probe run
30166429073 (matrix leg killed by `timeout-minutes`, sibling leg green):
killed leg `job.status` ........ cancelled
matrix rollup `needs.*.result` . cancelled
`if: cancelled()` .............. SKIPPED (evaluated FALSE)
`if: failure()` ................ SKIPPED (evaluated FALSE)
pre-fix `notify` condition ..... SKIPPED <- the bug
post-fix `notify` condition .... RAN <- the fix
run conclusion ................. cancelled
So `failure() || cancelled()` would NOT have fixed this. The signal has
to come from the per-slot build results.
- stop laundering `cancelled` into `skipped` in the per-slot writer
- expose `any_cancelled` / `cancelled_services` from the aggregator job
- add `notify-cancelled-builds`: exits non-zero so the run concludes
`failure` rather than `cancelled` (a slot killed by its timeout budget
is a failure, and `cancelled` is what suppressed everything), and
Slacks the affected service names
- add the `any_cancelled` clause to `notify` so the merge author gets the
PR comment, with wording that distinguishes incomplete from failed
`!cancelled()` is retained on both jobs as the intentional-vs-flake
discriminator: a human cancelling the whole RUN makes `cancelled()` true
and stays silent, while a leg-level timeout leaves it false and alerts.
Extends redeploy-guard.test.ts, which evaluates the LIVE `if:` strings
from the workflow, with the exact production scenario. It pins the
pre-fix guard string as a literal so the test proves the difference the
fix makes, not merely the current behaviour.
## The gap
**No CI job ran the showcase unit suites.** Not "ran them partially" —
no job invoked them at all. Verified per suite against `91f0321397` (the
tip of `main` this branched from), by reading every workflow's actual
`run:` lines rather than trusting job names:
| Suite | Tests | Ran in CI before? | Where |
|---|---|---|---|
| `showcase/harness` | 3646 (3625 pass / 3 fail / 18 skip) | **NO** |
nowhere |
| `showcase/shell-dashboard` | 1331 (1330 pass / 1 skip) | **NO** |
nowhere |
| `showcase/shell` | ~57 (8 files) | **NO** | nowhere |
| `showcase/scripts` | — | yes | `showcase_validate.yml` → "Run build
pipeline tests" |
Why each candidate workflow misses them:
- **`test_unit.yml`** — the one whose name implies it covers this —
excludes them *twice over*: `paths-ignore` lists `showcase/**`, **and**
its nx selection is hard-scoped to `--projects='packages/**'`.
- **`showcase_validate.yml`** runs `pnpm exec vitest run` only in
`showcase/scripts`. It has two `working-directory: showcase/harness`
steps, but they are a CVDIAG perf bench (`vitest bench
src/cvdiag/emit-perf.bench.ts`) and an ESM boot-smoke — neither runs the
unit suite.
- **`static_quality.yml`** does run `nx run-many -t check-types`. This
resolves the reported name/invocation divergence: the *job* is called
`check-types` and the harness's script is called `typecheck`, so no
`check-types` target exists on the harness and **the harness is not
typechecked either**.
Net effect: ~5000 showcase unit tests gated nothing. A PR could carry
real defects in `showcase/harness/src/**` and still show an all-green
check list, because no job structurally could have caught them.
## What this wires
New workflow **`test / unit-showcase`**
(`.github/workflows/test_unit-showcase.yml`), two parallel jobs on PR +
push-to-main filtered to `showcase/**`:
- **`harness unit suite`** — `nx run
@copilotkit/showcase-harness:test:ci`, then `nx run
@copilotkit/showcase-harness:test:quarantine-ratchet`. Driven through nx
per the root `CLAUDE.md` task convention.
- **`shell-dashboard unit suite`** — `vitest run` directly; the
dashboard is deliberately outside the pnpm workspace (see the note in
`pnpm-workspace.yaml`) so it is not an nx project and has no target to
run.
New nx targets are declared in **`showcase/harness/package.json`'s
`nx.targets` block**, not in `nx.json` — they are harness-specific, and
a `nx.json` `targetDefaults` entry would silently apply to any future
project that happens to share the script name. Both are `cache: false`
on purpose: the nx `test` named-input covers `src/**` and `*.test.*` but
**not** `vitest.ci.config.ts` or `vitest.quarantine.json`, so a cached
result could survive an edit to the quarantine list.
### Why a separate workflow rather than steps in `showcase_validate.yml`
That file is a single ~900-line job already budgeted at 25 minutes and
is the busiest merge-path workflow in the repo. Splitting the unit
suites out gives them their own name in the check list, their own
concurrency group, and — since the two suites need different package
managers — lets them run in parallel instead of lengthening the critical
job. Naming follows the existing convention (`test_unit.yml`,
`test_unit-python-sdk.yml`, `test_unit-spring-ai.yml`).
**No `continue-on-error` and no `|| true` anywhere in the file.** A gate
that cannot fail is not a gate.
## Hazards, and how each is handled
### 1. The hanging spike test
*Correction to the reported location:* it is
`showcase/shell-dashboard/tests/runtime-env-switch.spike.test.ts`, not
under `showcase/harness/`. It reaches the dashboard suite via that
package's `include: ["src/**/*.test.{ts,tsx}",
"tests/**/*.spike.test.ts"]`.
**Reproduced locally.** vitest sat at **0.0% CPU with zero output** and
never even spawned the `next build` its `beforeAll` calls — no worker
fork, nothing. Killed at ~2 minutes; consistent with the reported
~25-minute stall.
**Handled by excluding it from the unit gate**, not by a hard timeout.
Justification: it is an integration spike by construction — one `next
build` plus two `next start` boots on fixed ports 3801/3802 — and this
package's own config comment already says it is *"too heavy for the
per-file unit suite"* even though the `include` glob pulls it in anyway.
A hard timeout would convert a 25-minute stall into a red gate with
nothing actionable in it; this suite has to stay a fast, trustworthy
unit signal. The spike needs its own job with a real server budget,
which is out of scope for wiring up the unit gate.
Note the `--exclude` values restate the package's existing config
excludes (`tests/visual/**`, `node_modules/**`) because vitest's CLI
`--exclude` **replaces** the config value rather than extending it —
dropping them would silently re-enable them.
### 2. Gitignored generated artifacts
`showcase/.gitignore` ignores `shell/src/data/*.json` and
`shell-dashboard/src/data/*.json`, but the suites consume them:
- `harness/src/probes/frontend-matrix.test.ts` **statically imports**
`shell/src/data/frontend-catalog.json` — a missing file is a module
*load error*, not a test failure.
- `harness/src/fleet/control-plane/d0-gone-predicate.test.ts` reads
`shell/src/data/registry.json` at runtime.
- `shell-dashboard/src/lib/docs-status.ts` **statically imports**
`@/data/docs-status.json`.
Generated as an explicit job step (`generate-registry.ts`, plus
`probe-docs.ts` for the dashboard), mirroring how
`showcase_validate.yml` drives the same generators. The dashboard job
invokes vitest directly rather than `npm test` specifically so the
`pretest` hook does not re-run `probe-docs.ts` and double its ~50
outbound HEAD requests.
Also load-bearing: `npm ci --ignore-scripts` in the dashboard job. That
package's `postinstall` is `cd ../scripts && npm install`, which would
lay an npm-resolved `showcase/scripts/node_modules` over the
pnpm-managed one.
### 3. The 3 pre-existing failures — quarantined behind a ratchet, none
deleted or weakened
Confirmed byte-identical to `origin/main` and pre-existing. A job that
is red on arrival gets ignored or disabled, so the three files are
excluded via `showcase/harness/vitest.quarantine.json` +
`vitest.ci.config.ts` — **and `scripts/quarantine-ratchet.ts` re-runs
each one and requires it to STILL FAIL.** The moment somebody fixes a
quarantined test, the ratchet goes red and names the entry to delete. An
entry therefore cannot outlive the failure it excuses. The ratchet also
fails on an entry pointing at a missing file, on an entry whose filter
matches more than one file, and on a malformed or under-documented
manifest (every entry must carry `file` / `since` / `reason` /
`unquarantineWhen`).
Local `pnpm test` still uses the unfiltered config, so a developer sees
the quarantined failures.
| File | Disposition | Why |
|---|---|---|
| `src/probes/frontend-matrix.test.ts` | quarantine | Asserts hard-coded
per-frontend counts (`react=660`, `angular=636`) against the
**generated** `frontend-catalog.json`, which has grown to 1302 runnable
cells (`react=664`). Stale literals, not a product defect — the two
assertions that express the test's stated intent ("without loss or
duplication": `length === metadata.runnable`, ids unique) both pass.
Exit criterion: derive the counts from the catalog; re-hard-coding fresh
numbers just resets the clock. |
| `src/probes/helpers/d5-mapping-drift.test.ts` | quarantine |
**Stranded by a completed refactor.** It regex-scrapes
`CATALOG_TO_D5_KEY` out of `shell-dashboard/src/lib/live-status.ts` to
prove the dashboard copy hasn't drifted from the harness's
`REGISTRY_TO_D5`. That file is now a one-line `export *` barrel over the
canonical `harness/src/shared/cell-model/live-status.ts` — there is no
second copy to drift and nothing for the regex to find, so the test
throws its own "could not locate" guard. Exit criterion: cell-model
owner deletes it (single-source is a strictly stronger guarantee) or
rewrites it to assert the barrel re-export. |
| `src/probes/helpers/d5-representatives.test.ts` | quarantine |
**Genuine product gap, deliberately not fixed here.**
`d5-browser-use-smoke` registers `browser-use-smoke` in `D5_REGISTRY`
but `D5_REPRESENTATIVES` has no entry, so
`getD5Representative('browser-use-smoke')` returns `undefined` and the
D5 driver has no representative fixture. The test is correct and
reporting a real hole; the fix belongs to the D5 owner, and this PR
touches no product source. |
### 4. New finding: a load-sensitive flaky test
Not one of the 3, and worth fixing separately:
`src/probes/loader/probe-invoker.test.ts` → *"times out at the invoker
level when enumerate() ignores abortSignal"* asserts a **wall-clock**
`elapsed < 150` against a 100ms invoker timeout racing a 200ms
enumerate. 50ms of headroom does not survive CPU contention — it passed
**5/5 in isolation** but failed twice at ~217ms while the machine was
running other vitest suites, and a 4-vCPU runner executing 173 test
files in parallel is exactly that condition.
Mitigated with `retry: 2` **in the CI config only** (base config
unchanged, so local runs still surface non-determinism). This is not
suppression: a genuinely broken test fails all three attempts and the
gate stays red, and vitest prints every retried test. The real fix is
for the probe-loader owner to drive that assertion off fake timers or
widen the bound, after which the `retry` should come out. Flagging the
tradeoff explicitly rather than burying it.
## Deliberate-breakage proof
A job that has never failed is not yet a gate. Scratch commit
`e100df27a7` broke two product-source files, then `4f092eccb1` reverted
it. The final branch diff is **578 insertions, 0 deletions, and zero
product source touched.**
| Run | SHA | `harness unit suite` | `shell-dashboard unit suite` |
Quarantine ratchet |
|---|---|---|---|---|
| baseline
[`30140087665`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140087665)
| `452a26115e` | **success** (3m33s) | **success** (1m35s) | success |
| **breakage**
[`30140218159`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140218159)
| `e100df27a7` | **FAILURE** (3m31s) | **FAILURE** (1m28s) | success |
The breakage — both reproduced locally before pushing:
- `showcase/harness/src/shared/cell-model/staleness.ts`:
`D4_STALE_AFTER_MS` 1h → 10h (a frozen-green D4 row would credit D4 for
ten hours instead of one)
- `showcase/shell-dashboard/src/lib/sort-order.ts`: `sortOrder` `mastra:
8` → `88`
What the RED run actually caught:
```
harness unit suite — Test Files 3 failed | 168 passed | 2 skipped (173)
Tests 4 failed | 3612 passed | 18 skipped (3634)
FAIL src/http/matrix.test.ts > stale green ladder: raw state='green' but /api/matrix folds to gray (U8)
FAIL src/shared/cell-model/cell-model.contribution.test.ts > family fold: one stale-green sibling forces STALE_DEGRADED
AssertionError: expected 'GREEN_FRESH' to be 'STALE_DEGRADED'
FAIL src/shared/cell-model/cell-model.equivalence.test.ts > golden-master identity > fixture: pos-d4-stale-green
FAIL src/shared/cell-model/cell-model.equivalence.test.ts > golden-master identity > fixture: stale-column
shell-dashboard unit suite — Test Files 3 failed | 64 passed (67)
Tests 4 failed | 1326 passed | 1 skipped (1331)
FAIL src/components/__tests__/depth-utils.test.ts > does not credit D4 when its green chat row is stale
AssertionError: expected 4 to be 3
FAIL src/lib/__tests__/cell-model.test.ts > comm-error staleness window > scopes the window PER ROW FAMILY (G3e)
AssertionError: expected 36060000 to be less than 21600000
```
Three things this establishes beyond "the job can go red":
1. **It catches exactly the class of defect that went undetected on
#6156** — a change to `showcase/harness/src/shared/cell-model/**` landed
4 failures in the harness gate, including two golden-master equivalence
fixtures.
2. **The dashboard job also caught the harness defect**, through the
`live-status.ts` re-export barrel. The shared cell-model now has two
independent gates over it.
3. **`retry: 2` is not suppression.** Every `FAIL` line above appears
three times in the log — the test was retried the full budget and failed
all three attempts. A genuinely broken test cannot pass through it. And
the **quarantine ratchet stayed green on the broken run**, confirming
the quarantine is scoped to its three files and does not swallow
unrelated regressions.
## Runtime
Measured on CI (Depot 4-vCPU), baseline run
[`30140087665`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140087665)
— **both jobs green**:
| Job | CI wall-clock (whole job) | Suite only (local) | Files |
|---|---|---|---|
| `harness unit suite` | **3m33s** | 63.5s | 173 (171 pass / 2 skip,
3616 tests) |
| `shell-dashboard unit suite` | **1m35s** | 7.2s | 67 (1330 pass / 1
skip) |
They run in parallel, so the added critical-path cost is ~3m33s —
comfortably under `showcase_validate.yml`'s existing 25-minute budget
and nowhere near irritating-enough-to-disable. In both jobs the budget
is dominated by install and (for the harness) nx `^build`, not by the
tests.
**No sharding or `nx affected` needed.** `nx affected` would in fact be
actively wrong here: the entire point is that a change anywhere under
`showcase/**` should run the suite covering it, and the historical
failure mode was scope-narrowing, not slowness. If install time later
dominates, the lever is pnpm/npm store caching (already configured) —
not cutting test scope.
## Not covered
`showcase/shell`'s 8 test files (~57 tests) are **also ungated** and I
left them out to keep this change to one concern. They need a third
install path (`showcase/shell` is likewise outside the pnpm workspace)
and its `vitest.config.ts` already has a `globalSetup` that generates
its own `registry.json`. Cheap follow-up.
## Coordination
Concurrent work is adding a vacuity/drift text gate under
`showcase/scripts` and compile-time completeness exports. This PR owns
only the **suite invocation**. The one plausible cherry-pick conflict is
`showcase/harness/package.json` (I added two `scripts` entries and an
`nx.targets` block); `.github/workflows/test_unit-showcase.yml` is a new
file and `nx.json` is deliberately untouched.
No CI job ran the showcase unit suites. Verified per suite on 91f0321397:
showcase/harness 3646 tests NOT RUN
showcase/shell-dashboard 1331 tests NOT RUN
showcase/shell 57 tests NOT RUN
showcase/scripts - run by showcase_validate.yml
`test_unit.yml` excludes them twice over: `paths-ignore` lists `showcase/**`
AND its nx selection is scoped to `--projects='packages/**'`.
`showcase_validate.yml` runs vitest only in `showcase/scripts`; its two
`working-directory: showcase/harness` steps are a CVDIAG perf bench and an ESM
boot-smoke. `static_quality.yml` runs `nx run-many -t check-types` and the
harness's script is named `typecheck`, so it is not typechecked there either.
So ~5000 showcase unit tests gated nothing — a PR could carry real defects in
`showcase/harness/src/**` and still show an all-green check list.
This adds `test / unit-showcase` with two parallel jobs (harness via nx,
dashboard via npm since it is deliberately outside the pnpm workspace). Both
hazards that make a bare `vitest run` useless here are handled as job steps:
- generated artifacts: `showcase/.gitignore` ignores `shell/src/data/*.json`
and `shell-dashboard/src/data/*.json`, but tests statically import
`frontend-catalog.json` and `docs-status.json` — a missing file is a module
load error, not a test failure. Generated before the suite runs.
- `runtime-env-switch.spike.test.ts` HANGS: vitest sits at 0.0% CPU with zero
output and never spawns the `next build` its beforeAll calls. It is an
integration spike (one `next build` plus two `next start` boots on fixed
ports) that this package's own config comment already calls "too heavy for
the per-file unit suite". Excluded from the unit gate; it needs its own job
with a real server budget.
A separate workflow rather than steps in showcase_validate.yml: that file is a
single ~900-line job already budgeted at 25 minutes, and the two suites need
different package managers, so they parallelize instead of lengthening the
critical job. Naming matches test_unit.yml / test_unit-python-sdk.yml.
No `continue-on-error` and no `|| true` anywhere in the file.
Demo assets stay uniformly LFS-tracked. The build is the single place this
gets resolved, so every workflow that checks out and then serves or bakes
integration `public/` assets now fetches LFS objects.
`showcase_build.yml` (the workflow that builds and pushes the deployed
images) has hardcoded `lfs: true` since 7bde1eef3a, so the deploy path was
already correct. Two paths were not:
- `showcase_build_check.yml` read `lfs: ${{ matrix.service.lfs }}`, which was
`false` for every integration, so the pre-merge build check baked pointer
stubs into the images it built. Build-only, so nothing deployed, but it
meant PR CI could never catch a broken demo asset.
- `test_e2e-showcase-on-demand.yml` passed no `lfs` input at all. That job
runs `next dev` on the integration package and drives it with Playwright,
serving `public/` straight from the working tree, so `multimodal.spec.ts`
would click "Try with sample image", receive a ~130-byte pointer stub as
`image/png`, and fail on the magic-byte guard — with nothing wrong in the
code under test. Nine integrations ship that spec.
Both now use a uniform `lfs: true`.
Also deletes the per-slot `"lfs"` field from both `ALL_SERVICES` definitions.
After the above there were no consumers left: `showcase_build.yml` ignored it
in favour of a hardcoded `true`, and `showcase_build_check.yml` was its only
reader. Leaving a dead flag that reads as authoritative is how this was
mis-set in the first place, and deleting it means a new integration cannot be
added with LFS fetching silently off — there is no flag to forget.
No asset or .gitattributes changes: LFS is used as intended and a committed
pointer remains the correct on-disk state for an LFS-tracked file.
Two release-tooling defects turned a pair of canary publishes into a broken
combination for consumers (a canary runtime resolving the last STABLE
channels-intelligence, which still called the removed `channel.addAdapter`).
1. Canary versions were prereleases of an ALREADY-PUBLISHED version. A stable
release leaves the working tree on the version it just published, and
computePrereleaseVersion appended `-canary.<id>` to exactly that, so the
canary sorted BELOW its own release (`0.2.1-canary.x < 0.2.1`): the `canary`
dist-tag pointed behind `latest`, and no dependent range could ever resolve
it. Base the canary on the next unreleased version instead (patch bump,
reusing computeNextStableVersion's prerelease rule).
2. A canary published one scope at a time, but the scopes are only independent
on the version axis. `@copilotkit/runtime` carries
`"@copilotkit/channels-intelligence": "workspace:*"`, and `pnpm pack`
resolves that against the working tree — so a `monorepo` canary pinned the
channels family to its last stable release even when the commit changed both
sides of the contract. Add a prerelease-only `all` selector that bumps and
publishes every scope from one commit under one shared canary id, and warn
loudly when a single-scope canary leaves a cross-scope pin behind.
`all` is a selector, never a scope: stable releases stay single-scope (their
tag, release branch, and npm/Slack links all derive from one scope name), which
publish-release.yml enforces in both jobs and the dropdown guard enforces per
workflow.
dispatch-gate hardcoded --scope affected, so a manual dispatch always
ran the affected set. On a large PR that touches shared files, affected
resolves to every integration, and the eval's fleet bring-up then tries
to build 20+ images on one runner and dies (docker compose up -d exit
255). The comment path already supports targeting one slug
(/eval d5 mastra -> --slug mastra); this gives workflow_dispatch the same
lever via an optional `slug` input, validated with the same ^[a-z0-9-]+$
rule and read as untrusted env. Empty slug keeps the affected default.
This also makes the /eval fix validatable pre-merge: dispatch with
`-f slug=<one>` reproduces the real targeted comment path with a bounded
bring-up instead of the pathological affected=all case.
Dropping --ci moved the /eval job onto the CLI's Docker lifecycle, but
that immediately fails on a bare runner: docker-compose.local.yml declares
`env_file: .env` on every service and showcase/.env is gitignored, so
`docker compose` errors with "env file .../showcase/.env not found" before
any container starts. Validated: dispatch runs 29846236558 and 29845135213
both died ~6s into the eval step on exactly this.
Add a step that writes showcase/.env with dummy values before the eval.
aimock serves the recorded fixtures and never validates tokens, and the
aimock base URLs are already hardcoded in the compose environment block,
so the keys/URLs here just satisfy env_file and document intent.
The /eval job runs on a bare depot runner with no step that starts the
showcase fleet, but passed --ci to the eval CLI. --ci makes the CLI skip
the Docker lifecycle and assume services are already running, so it found
no healthy container and failed in ~1s with zero test results.
Drop --ci so the CLI's non-ci path builds + starts the in-scope slug(s) +
aimock and health-checks them before running. Safe with --json retained:
compose() uses piped (captured) stdio and all progress logs are
!opts.json-guarded, so stdout stays clean JSON for the post-result job.
The notify-all-builds-failed and notify jobs keyed off a 'failure' rollup /
bare failure(), so a build where every real service failed but one leg was
cancelled (contention) rolled up to 'cancelled' and sent no alert — the same
blind spot as the redeploy guard. Fire on any_success == 'false' (guarded by a
status function so a user-cancelled run stays silent). Extends the guard test.
Benjamin Taylor