mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
codex/fix-docs-staging-500
904 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Alem Tuzlak
|
4880afc846 |
test(skills): guard public skill API contracts (#6457)
## Summary - extend the existing public-skill drift suite to validate maintained setup assets against the generated public API manifest - fail when a skill imports an unpublished CopilotKit package or entrypoint, or a manifest-deprecated API - run the guard in the existing plugin-skills workflow when skills or the manifest change ## Why this matters Coding agents copy these skill assets directly into user projects. Mirror-sync tests prove that our duplicated skill files match, but they do not prove that the examples still reference packages and APIs we actually publish. A stale import can make CopilotKit fail at the first install or build step, which is exactly the kind of failure that prevents agents from choosing and successfully adopting us. This PR adds the smallest deterministic guard for that risk. It reuses our existing Vitest suite and canonical public API manifest; it does not introduce an eval harness, run agents, score behavior, collect metrics, add a provider, or add dependencies. ## Scope This is package-contract validation, not behavioral evaluation. Broader questions such as whether an agent follows a skill well, how many attempts it needs, and whether the generated application behaves correctly remain separate work and should start with a concrete decision the deterministic checks cannot answer. ## Verification - `pnpm exec vitest run scripts/__tests__/public-skill-drift.test.ts scripts/__tests__/sync-plugin-skills.test.ts` (17 tests) - `pnpm check:plugin-skills` - `pnpm check:public-api-manifest` - targeted TypeScript, oxfmt, and oxlint checks - mutation check: replacing `BuiltInAgent` with deprecated `BasicAgent` fails with the manifest-provided replacement Linear: PDX-320 |
||
|
Alem Tuzlak
|
47ad5e34a3 |
refactor(react-native)!: converge tool-call rendering onto CopilotKit's shared registry (#6438)
## What does this PR do?
`@copilotkit/react-native` maintained a **private tool-call render
registry** (`hooks/RenderToolContext.tsx`) alongside the canonical one
that `CopilotKitCoreReact` already provides — and which every React
Native app already ships, unused. This PR deletes the fork and points
React Native at the shared registry.
That fork caused three bugs:
| Bug | Symptom | Cause |
|---|---|---|
| **Tool renders never streamed** | A component registered with
`useRenderTool` / `useComponent` painted nothing until the tool call
completed | `CopilotChat` used `JSON.parse` on the argument buffer.
While a model writes a tool call that buffer is *invalid JSON by design*
— AG-UI delivers `TOOL_CALL_ARGS` deltas that are concatenated
client-side — so the parse threw on every delta, warned, and fell back
to `{}` |
| **`useComponent` rendered nowhere** | Silently, with no error | It
writes to core's registry; React Native's chat read React Native's
private `Map` |
| **Chat history degraded** | Navigating away from the registering
screen turned earlier tool calls into a `Called: <name>` placeholder |
The private `Map` deleted renderers on unmount; core deliberately keeps
them |
`@copilotkit/react-core` has used `partialJSONParse` on this path since
v2 shipped. React Native diverged because `useRenderToolCall` was
excluded from its re-exports on the stated grounds that it "depends on
DOM elements via `DefaultToolCallRenderer`" — a claim that was never
true of the hook itself. It was only ever reachable through the fat
`/v2` entry, whose weight is the real hazard (#4893). #5883 moved it
into `/v2/headless` on 2026-07-23; the exclusion comment was rewritten
the next day without revisiting the reason.
### What changed
- **One registry.** `useRenderTool` registers through `useFrontendTool`
into `CopilotKitCoreReact.renderToolCalls`. `CopilotChat` and any custom
surface consume react-core's `useRenderToolCall`.
- **Types are derived, not declared.** `RenderToolProps` is now
`React.ComponentProps<ReactToolCallRenderer<T>["render"]>`, so React
Native cannot drift from `ReactToolCallRenderer` — the contract every
registered renderer is actually invoked against. Change that contract
and `check-types` names every React Native renderer the change breaks.
React Native narrows only the *return* type to `ReactElement | null`,
which `FlatList`'s `renderItem` genuinely requires.
_Scope of that guarantee (corrected during review):_ it does **not**
extend to the type react-core publicly exports under the same name.
Web's `RenderToolProps<S>`
(`react-core/src/v2/hooks/use-render-tool.tsx`) is a separate
hand-declared union, generic over a schema, carrying arguments under
`parameters` (not `args`) and declaring `status` as string literals
rather than `ToolCallStatus` members. Both divergences are live today
and nothing type-checks them shut — the one place the shapes meet,
react-core's own bridge, compiles because a string-enum member is
assignable to its own literal type but not the reverse. Aligning web's
alias is a breaking web API change, filed separately.
- **`RenderToolContext.tsx` deleted** (−150 lines), along with 15 tests
that described the removed subsystem. One of them — `unregisters the
render function on unmount` — asserted the chat-history bug as a
requirement.
- **Two structural CI guards for #4893**, in opposite directions: a test
failing if any React Native source imports the fat `/v2` entry, and a
script failing if react-core's `/v2/headless` or `/v2/context` chunks
ever link shiki/mermaid/cytoscape/katex/streamdown. Both were verified
able to fail by deliberately introducing the regression. These are
*structural* assertions, not size budgets — `dev-docs/bundle-size.md`
freezes `limit` fields until OSS-122.
- **`react-native` added to the bundle-size glob**, which it had never
been in, plus a `size:headless` measurement.
React Native also gains capabilities it lacked: render props inferred
from your schema, `name`/`toolCallId` on render props, and `result` on
completed calls.
**Corrected during review — two capabilities this originally claimed are
not delivered:**
- **Wildcard (`"*"`) renderers do not work on React Native.** Because
`useRenderTool` routes through `useFrontendTool` (which calls
`addTool`), `name: "*"` registers a frontend tool literally named `*` —
advertised to the model, and colliding with core's separate
wildcard-executable-tool path. react-core's `useRenderTool` is
renderer-only and special-cases the wildcard; React Native's is not. The
guide now advises against it.
- **`followUp` (and `available`) are not forwarded**, and the handler's
`context` argument is dropped, so `stopAgent()`'s abort signal is
unreachable from an RN handler.
Both are tracked in § Known limitations for the follow-up that converges
React Native onto react-core's hooks — deleting RN's `useRenderTool` in
favour of re-exporting `useFrontendTool` (tool + renderer) and
react-core's `useRenderTool` (renderer-only, wildcard-capable). That is
an API change with its own migration note, so it is not in this PR.
### ⚠️ Breaking (in a minor)
`useRenderToolRegistry` and `RenderToolProvider` are **removed**. Both
are documented on the docs site, so this is a real break — see the
`BREAKING CHANGE:` footer on `db67ccf`, which is what the release notes
derive from, plus the rewritten reference pages.
```diff
- const registry = useRenderToolRegistry();
- const renderer = registry.get(toolCall.function.name);
- return renderer ? renderer({ args, status }) : null;
+ const renderToolCall = useRenderToolCall();
+ return renderToolCall({ toolCall });
```
Also note two semantic changes: `args` is `Partial<T>` **only** while
`status` is `"inProgress"`, and a render function is now captured at
registration — if it closes over changing values you must declare them
in `deps` (React Native previously refreshed the closure on every
render).
**Known limitation:** agent-scoped renderer resolution does not take
effect on React Native. `CopilotChatConfigurationProvider` is not in
RN's provider tree, so `agentId` always resolves to the default.
Renderers still resolve by name; two agents registering the same tool
name resolve arbitrarily. Filed separately.
### A data point worth recording
Adding `useRenderToolCall` to the measured headless entry moved the
bundle **92.8 kB → 92.7 kB**. Flat. The hook React Native spent months
not using was already inside the chunk every RN app resolves whole —
Metro doesn't tree-shake, so the fork never saved a byte. It cost them.
### Testing
- `@copilotkit/react-native`: **253 passing / 22 files** ·
`@copilotkit/react-core`: **1480 passing / 123 files** · `check-types`
and `build` green for both.
- Each of the three bugs has a deterministic test driving a real
`CopilotKitCoreReact` — no mocking of the code under test.
- Both #4893 guards carry mutation evidence: introduce the regression,
watch them fail, revert, watch them pass.
### Follow-up
`useRenderTool`'s JSDoc is split across two blocks, which orphans the
primary description from IDE hover (the `@param deps` warning still
surfaces). One-line fix, deliberately left out of the final fix wave.
## Related PRs and Issues
- **Supersedes #6346** (@davidmckayv) — its diagnoses were correct and
its test assertions are ported here, re-driven through the real registry
rather than a mocked local one. Credited via `Co-Authored-By` on
`4104bd1`.
- Addresses the React Native half of **#4893**.
- Builds on **#5883**, which created the lean `/v2/headless` entry this
PR consumes.
## Checklist
- [x] I have read the Contribution Guide
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation
- [x] "Allow edits by maintainers" is checked
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Tyler Slaton
|
f13fcb09e9 |
ci: add manual AEO production checks (#6459)
## Summary - add an on-demand production check for the website and docs discovery surfaces defined by #6458 - derive the ten in-scope routes and media types from the public contract instead of maintaining a second monitoring manifest - exercise those routes as four documented crawler user agents with a global concurrency cap of four - validate status, content type, canonical host, robots/sitemaps, one sampled sitemap link, LLM index links, and soft-404 behavior - retain failure evidence and provide a deliberate `exercise_alert` input for proving the `#oss-alerts` path ## Why this matters AEO is a production property, not a one-time content change. A correct repository can still deploy a broken canonical, HTML fallback, stale sitemap, or inaccessible LLM index. Those failures happen at the top of the agent-led growth funnel: if agents cannot reliably discover and verify CopilotKit, downstream recommendation and activation work never gets a chance to perform. This PR adds the smallest useful operating check for that risk. It is deliberately limited to PDX-340's website/docs scope. It does not monitor MCP, the CopilotKit capability document, raw Markdown, Open Graph, or JSON-LD. Existing deploy-parser utilities are reused where practical, requests run with bounded concurrency, and failures include the exact URL, crawler identity, observed status/type, and a bounded response excerpt. The workflow is intentionally manual at first. We should not create a scheduled noisy alarm while the website LLM endpoints are known red, and we should not claim Slack ownership until a deliberate failure proves the secret and alert path. A small follow-up can add the schedule after one normal run is green and one `exercise_alert` run reaches `#oss-alerts`. ## Stacked dependency - Depends on #6458; this PR is intentionally based on `codex/pdx-317-aeo-surface-contract`. ## Validation - `pnpm nx run @copilotkit/showcase-scripts:validate-aeo-contract --skip-nx-cache` - `pnpm nx run @copilotkit/showcase-scripts:test --skip-nx-cache -- __tests__/check-aeo-synthetics.test.ts __tests__/aeo-synthetics-wiring.test.ts __tests__/verify-deploy.drivers.test.ts` (102 tests) - targeted `oxfmt` and `oxlint` checks - `git diff --check` and commit hooks ## Live baseline (2026-08-12) The narrowed production command fails with eight records: four crawler identities × two website gaps. - `https://www.copilotkit.ai/llms.txt` returns HTTP 200 `text/html` with a noindex soft-404 instead of plain text - `https://www.copilotkit.ai/llms-full.txt` returns the same soft-404 The remaining website/docs targets pass: both home canonicals, both robots files, both sitemaps and sampled links, and both docs LLM indexes. The current failures are why this PR ships manual-first rather than enabling a schedule. ## Status PDX-340 remains In Progress until the website endpoints are fixed, a normal workflow run is green, the deliberate Slack alert reaches `#oss-alerts`, and a follow-up enables the agreed schedule. |
||
|
Tyler Slaton
|
d21aebc6e2 |
docs: define public AEO surface contract (#6458)
## Summary - publish a single shared, versioned technical contract for website, docs, and docs MCP AEO surfaces - publish the human policy through the existing shell-docs MDX pipeline at `/aeo` - expose the machine-readable contract at `/.well-known/copilotkit-capabilities/v1.json` - validate the contract with JSON Schema/Ajv plus narrow repository and CI cross-reference checks - run the actual shell-doc behavior tests in CI and assign external website and Pathfinder gaps to named owners ## Why this matters Answer engines and coding agents decide which source to trust from machine signals such as canonical hosts, stable URLs, response types, and consistent capability claims. When those signals disagree, CopilotKit can be classified incorrectly, cited from the wrong hostname, or skipped even when it is the right product. This PR gives those public surfaces a versioned source of truth. It separates standards, community conventions, and CopilotKit-specific guarantees; records real endpoint paths and media types; and makes ownership explicit when behavior lives in another repository or service. That gives us a reliable base for improving agent discovery without pretending one repository can enforce every public surface. The implementation deliberately uses the current docs architecture: `/aeo` is ordinary shell-docs MDX under `showcase/shell-docs/src/content/docs/`, not a bespoke page or the retired docs tree. Schema shape lives in JSON Schema, while the small TypeScript layer only checks relationships JSON Schema cannot express, such as whether referenced files and CI commands exist. ## Validation - `pnpm nx run @copilotkit/showcase-scripts:validate-aeo-contract --skip-nx-cache` - `pnpm nx run @copilotkit/showcase-scripts:test --skip-nx-cache -- __tests__/validate-aeo-contract.test.ts` (6 tests) - `npm --prefix showcase/shell-docs test -- src/app/sitemap.test.ts src/app/llms.txt/route.test.ts src/app/llms-full.txt/route.test.ts 'src/app/llms-mdx/[[...slug]]/route.test.ts' src/app/well-known/copilotkit-capabilities/v1.json/route.test.ts src/lib/runtime-config.test.ts src/lib/__tests__/next-config-redirects.test.ts` (43 tests) - `npm run typecheck` in `showcase/shell-docs` - `npm run build` in `showcase/shell-docs` - targeted `oxfmt`, `oxlint`, TypeScript, diff, and commit-hook checks ## External follow-ups - CopilotKit/website must link the same policy and fix `/llms.txt` plus `/llms-full.txt`, which returned 200 `text/html` soft-404 pages during the production audit - Pathfinder/docs MCP owners must define a machine-readable discovery surface; the current contract records `/sse` as the known transport without presenting transport availability as discovery ## Related - PDX-317 |
||
|
Jerel John Velarde
|
fe0e7cf28f |
feat(examples): add grok-generative-ui showcase
grok-4.6 runs xAI's X Search server-side, then composes the answer out of real React components through five CopilotKit frontend tools. Every post rendered is a real post the model found. Registers next.config.ts in the build-config allowlist and adds the row to the examples index. |
||
|
Sam Julien
|
be3485d8f3 | fix(ci): narrow AEO synthetic rollout | ||
|
Sam Julien
|
a01b9f1034 | ci: monitor public AEO surfaces | ||
|
Sam Julien
|
25ce46ba5f | fix(docs): simplify AEO surface contract | ||
|
Sam Julien
|
9f2fb8e43b | docs: define public AEO surface contract | ||
|
Sam Julien
|
e9759825b6 | test(skills): simplify public contract guard | ||
|
Sam Julien
|
7bba4446ff | test(skills): narrow public API contract checks | ||
|
Sam Julien
|
4f258e409a | test(skills): add manifest-backed public evals | ||
|
renovate[bot]
|
b2ab37469a | chore(deps): update astral-sh/setup-uv action to v10 | ||
|
Maxim
|
f34b7aa23c |
ci: measure @copilotkit/react-native bundle size (was absent from the glob)
react-native was missing from static_bundle_size.yml's package glob, so its dist/ has never been measured despite being the consumer most exposed to the #4893 regression (Metro does not tree-shake). This adds coverage: - Extend the compressed-size-action glob to include react-native. - New scripts/measure-headless.mjs: an esbuild-driven gzip signal for the @copilotkit/react-native/headless entry, mirroring react-core's measure-copilotchat.mjs (stdin + resolveDir, gzip sum, job-summary output). - Wire a build + measure step into the copilotchat-import-size CI job. First baseline: @copilotkit/react-native/headless = 92.8 kB gzip (esbuild regression signal, not a Metro figure). No limit fields (Phase 1 policy — see dev-docs/bundle-size.md). esbuild added as a react-native devDependency (^0.27.0, matching react-core); the root ">=0.25.4" override keeps the monorepo on a single esbuild (0.27.3). Two corrections to the drafted script, verified by running it: - Fed the entry via esbuild stdin with resolveDir=pkgRoot; a temp-dir entry cannot resolve @copilotkit/react-native/headless through workspace node_modules. - Dropped useRenderToolCall from the import list — the RN headless surface deliberately does not export it (DOM-dependent; see src/index.ts). Co-Authored-By: Claude <noreply@anthropic.com> |
||
|
Maxim
|
68a30c2535 | test(react-core): hard-fail if the /v2/headless chunk links the render stack (#4893) | ||
|
renovate[bot]
|
f4c959e3fe | chore(deps): update reviewdog/action-actionlint action to v1.73.1 | ||
|
Ben Taylor
|
291cd32832 |
chore: stop changeset files from reappearing in PRs (#6406)
## What does this PR do? Community PRs keep arriving with `.changeset/*.md` files even though the repo migrated off Changesets to conventional-commit-driven releases. `.changeset/` has now been deleted from `main` twice (`5afa55f067` on 2026-06-16, `1e5ba689e0` on 2026-07-29) and **five open PRs carry changeset files today** (#6287, #6289, #6290, #6292, #6346). Three mechanisms keep feeding it: 1. **Stale forks.** `rodboev/CopilotKit`'s default branch still contains 10 of the pre-cleanup `.changeset/*.md` debris files. Three of the five open PRs come from that fork — the contributor's agent opens the repo, sees a directory full of changesets, and adds one more. (No `config.json`, and `@changesets/cli` isn't installed anywhere, so these are hand-written by agents, not CLI output.) 2. **Merging stale PRs re-seeds `main`.** The two files Tyler removed in `1e5ba689e0` arrived via 2026-06-10-authored branches (#2910, #5360) merged on 2026-07-25 — they sat on `main` for four days, and anyone who forked in that window inherited the directory. His hunch in that commit message was right. 3. **Convention inference, uncontradicted.** #6346 is from a branch in this repo, where `.changeset/` does *not* exist, and it still has one. The repo reads as a Changesets repo: pnpm workspace monorepo, per-package `CHANGELOG.md` in Changesets' exact `### Patch Changes` output format, `chore: release monorepo vX.Y.Z` release PRs. Nothing in `CONTRIBUTING.md`, the PR template, `AGENTS.md`, `CLAUDE.md`, or `.claude/docs/` said otherwise, so the guess was well-supported. This PR closes all three off: - **`CONTRIBUTING.md`** — new "Changelogs and releases — do not add a changeset" section: we did use Changesets, `scripts/release/` now builds changelogs from commit subjects, `.changeset/*.md` is inert, write a good conventional commit subject instead, and leave versions/changelogs to maintainers. Includes a note to rebase old forks. - **`AGENTS.md` / `CLAUDE.md`** — the same rule as an Essentials bullet. This is the highest-leverage change: the contributors doing this are coding agents, and agents load these files automatically while mostly not reading `CONTRIBUTING.md`. - **`static / check binaries`** — fail the PR on added `.changeset/*` files, so this stops depending on review catching it (which is what failed in July and restarted the loop). Added to the existing forbidden-files gate rather than a new workflow: it already runs on every PR to `main`, is fork-safe (`contents: read`, no secrets), and has exactly this `git diff --name-only origin/BASE...HEAD` + `VIOLATIONS` shape. Filters on `--diff-filter=AM` so a PR that *deletes* stale changesets still passes. - **`.oxfmtrc.json`** — drop the ignore entry for `.github/actions/changesets-action/src/run.ts`, a path that hasn't existed for a long time. It was the last grep-visible "we use changesets" signal in a root config file. ## Related PRs and Issues - Follows up `1e5ba689e0` ("fix: remove all changesets"), whose commit message asked for exactly this: a durable record of the decision that future agents can find. - Open PRs that would be caught by the new gate: #6287, #6289, #6290, #6292, #6346. ## Testing Docs + CI-config change, so verification focused on the guard. `actionlint` was run on the workflow, then the step body was extracted with `yq` and executed against real branches. **Lint / parse:** ``` $ actionlint .github/workflows/static_check-binaries.yml actionlint: clean $ python3 -c "import json; json.load(open('.oxfmtrc.json'))" # oxfmtrc still valid JSON oxfmtrc JSON OK ``` **True positive** — real head of #6292, via `yq '.jobs.check-binaries.steps[1].run'` piped to bash with `BASE_REF=main`: ``` ::error::Changeset files detected in PR: .changeset/enable-mcp-apps-tool-filters.md This repo no longer uses Changesets — releases are driven by conventional commit subjects (see scripts/release/). Nothing reads .changeset/*.md. Delete these files and describe the change in your commit subject instead. See the 'Changelogs and releases' section of CONTRIBUTING.md. This PR contains files that should not be committed (see the errors above). Please remove them and update your .gitignore if needed. exit=1 ``` **True negative** — same script on this branch, which has five changed files and no changesets: ``` $ git diff --name-only origin/main...HEAD .github/workflows/static_check-binaries.yml .oxfmtrc.json AGENTS.md CLAUDE.md CONTRIBUTING.md $ BASE_REF=main bash step.sh No binary artifacts or oversized files detected. exit=0 ``` **Delete-safety** — a commit that *removes* changesets must not be punished. Using the real cleanup commit (`8806f668d1...1e5ba689e0`): ``` unfiltered: with --diff-filter=AM (what the gate uses): .changeset/coalesce-...md (empty) .changeset/fix-parallel-...md ``` Not verified locally: the gate firing in real GitHub Actions — that needs this PR's own CI run (the `static / check binaries` check on this PR exercises the true-negative path). 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Ben Taylor
|
c6d59c529f |
feat(telemetry): emit telemetry-registry fragments for runtime + docs surfaces (#5891)
## What Adds the CopilotKit side of the [telemetry event registry](https://github.com/CopilotKit/oss-path-to-production/blob/main/docs/telemetry-registry-publish-roadmap.md): tooling + CI that generate this repo's registry **fragments** and open path-limited PRs into `CopilotKit/oss-path-to-production`, where the reconciler folds them into `telemetry-events.json`. Two surfaces, two mechanisms (per the surface-owns-its-extractor design): | Surface | Events | Extraction | Trigger | |---|---|---|---| | **runtime** | 5 `oss.runtime.*` | **bespoke catalog** — reads the `AnalyticsEvents` type map (names + properties), scans `capture()` sites for `call_sites`; **fails loud if the v1/v2 catalogs diverge** | stable **monorepo** release (`on: release`, tag `vX.Y.Z`) | | **docs** (`showcase/shell-docs`) | 11 | **callee mode** — inline `posthog.capture("name", {…})` literals; drops `$`-reserved events | push to `main` touching `showcase/shell-docs/src/**` (excluding `src/content`) | ## Key properties - **Content-gated.** The emitter leaves the target fragment byte-for-byte untouched when the extracted event set is unchanged, so a PR opens **only when telemetry actually changes** — no per-release / per-commit churn. - **Reconciled canonical in every PR.** Both workflows run the registry's `pnpm reconcile` and commit `telemetry-events.json` alongside the fragment, matching the registry's shipped emitters — a fragment-only PR fails its `telemetry-reconcile` staleness gate. - **Least-privilege cross-repo token.** No explicit `owner` (defaults to the app installation's org) + bare `repositories: oss-path-to-production` + `contents`/`pull-requests` write only; mint gated on a job-level env var (GitHub rejects `secrets.*` in `if:`). - **zizmor clean** at CI's `--min-severity low` (one `cache-poisoning` suppression, justified in `.github/zizmor.yml`: the workflow configures no cache and publishes a PR, not build artifacts). ## Files - `scripts/telemetry/extract.ts` — pure extraction (callee scan + catalog reader), deterministic output. - `scripts/telemetry/emit-fragment.ts` — CLI: `--surface runtime|docs --out <path>`, assembles + content-gates the fragment. - `scripts/__tests__/telemetry-fragment.test.ts` — 13 unit tests (fixtures) + a loose real-catalog drift smoke test. - `.github/workflows/telemetry-{runtime,docs}-fragment.yml` — the two CI jobs. ## Testing Rebased onto `main` (`55aaad21a6`) and revalidated end-to-end on 2026-08-05 — the branch had fallen 1345 commits behind. **Unit / static** - `vitest run scripts/__tests__/telemetry-fragment.test.ts` → **13/13 passed**. - `tsc --noEmit --strict --esModuleInterop` over both scripts → **clean** (`scripts/` has no tsconfig, so this is the ad-hoc invocation). - `oxlint scripts/telemetry` → **0 warnings, 0 errors**; `oxfmt --check` → **all files correctly formatted**. - `zizmor --min-severity low --config .github/zizmor.yml .github/workflows` (CI's exact invocation) → **No findings to report** (32 ignored, 233 suppressed). **Runtime surface — mechanism proven against the live rebased tree** ``` $ tsx scripts/telemetry/emit-fragment.ts --surface runtime --out /tmp/CopilotKit.runtime.json runtime: wrote 5 events → /tmp/CopilotKit.runtime.json (released_in runtime@1.66.2) ``` Diffed event-for-event against the registry's committed `CopilotKit.runtime.json`: **semantically identical** (same 5 events, same `call_sites`, same `properties_seen`) — the only difference is ordering, since the emitter sorts alphabetically and the hand-seeded fragment is in catalog-declaration order. Confirmed the reorder is a no-op at the canonical level (see below), so the first automated run opens one reordering PR with an empty `telemetry-events.json` diff and is quiet thereafter. Also confirmed the catalog is still complete on current `main`: the only `oss.*` event literals anywhere under `packages/runtime/src` + `packages/shared/src` are the 5 catalog entries (43/22/12/9/9 occurrences), so no untyped event is being silently dropped. Both v1 and v2 catalogs remain byte-identical, so the divergence guard passes. **Docs surface** ``` $ tsx scripts/telemetry/emit-fragment.ts --surface docs --out /tmp/CopilotKit.docs.json docs: wrote 11 events → /tmp/CopilotKit.docs.json (released_in shell-docs@5855496103) ``` 11 events (up from 7 when this PR was authored — the docs site grew): `cli_command_copied`, `docs_conversion_clicked`, `docs_conversion_copied`, `docs.framework_selected`, `docs.frontend_selected`, `docs.journey_continued`, `hero_command_copied`, `markdown_copied`, `open_in_llm_clicked`, `talk_to_us_clicked`, `try_for_free_clicked`. `$pageview` correctly dropped. **End-to-end against the real registry** Dropped both emitted fragments into a clean `oss-path-to-production@main` worktree and ran its own `pnpm reconcile`: - Both fragments **validate against `fragment.schema.json`** (ajv, via the reconciler's loader). - Reconcile succeeded; `telemetry-events.json` grew by 216 lines with 11 new `"surface": "docs"` observations. - **Zero `oss.runtime.*` entries changed** — confirming the runtime fragment's reordering has no canonical effect. ## Fixed during revalidation - **`add-paths` bug in the docs workflow (would have failed on first run).** It ran `pnpm reconcile` but listed only the fragment in `add-paths`, so its PR would have landed a fresh fragment beside a stale `telemetry-events.json` and tripped the registry's `telemetry-reconcile` staleness gate — the exact failure the runtime workflow was already fixed for. Verified against the registry's shipped emitters: every automated fragment PR there (`website.corp` #232/#220, Intelligence surfaces #228) carries `telemetry-events.json` alongside its fragment. - **Stale action pins.** Refreshed to the SHAs `main` now uses everywhere: `actions/checkout` v7, `actions/setup-node` v7.0.0, `pnpm/action-setup` v6.0.10. - **Over-broad docs trigger.** Narrowed from `showcase/shell-docs/**` to the code under `src/**`, excluding `src/content/**` — 1012 MDX + 140 JSON prose files with zero `.ts`/`.tsx`, none of which can hold a `posthog.capture` call site. Prose edits no longer fire a full monorepo install. - **zizmor justification accuracy.** `setup-node` v7 adds a `package-manager-cache` input defaulting to `true`; per its `action.yml` it engages only when `package.json` declares **npm**, and this repo declares pnpm — so the workflow is still cacheless and the suppression still holds. Noted inline. ## Prerequisite — now satisfied The registry App secrets (`TELEMETRY_REGISTRY_APP_ID`, `TELEMETRY_REGISTRY_APP_PRIVATE_KEY`) are configured on this repo (added 2026-07-09), and `app/copilotkit-telemetry-bot` is demonstrably installed on `oss-path-to-production` — it has been opening fragment PRs there from other surfaces (#232, #228, #220). No further setup needed. ## Not in this PR - The registry-side seed of the **docs** surface. The docs fragment first appears via this workflow's initial run, which now also carries the reconciled canonical, so it lands green. - The **web-inspector** surface, hand-seeded in the registry since this PR was authored, remains manual. Automating it is a follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
renovate[bot]
|
732987da9f | chore(deps): update dorny/paths-filter action to v4.0.3 | ||
|
Benjamin Taylor
|
b77ebb435f |
chore: stop changeset files from reappearing in PRs
The repo migrated off @changesets/* to conventional-commit-driven releases (scripts/release/ reads commit subjects from git log <lastTag>..HEAD), but .changeset/ has been removed twice already ( |
||
|
Benjamin Taylor
|
b3c7d57406 |
fix(telemetry): commit the reconciled canonical in docs fragment PRs too
Revalidation against current main (the branch was 1345 commits behind): - docs workflow ran `pnpm reconcile` but `add-paths` listed only the fragment, so its PR would land a fresh fragment beside a stale telemetry-events.json and fail the registry's telemetry-reconcile staleness gate — the exact failure the runtime workflow was already fixed for. Verified against the registry's shipped emitters: every automated fragment PR there (website.corp, Intelligence surfaces) carries telemetry-events.json alongside its fragment. - Refresh the action pins to the SHAs main now uses everywhere (checkout v7, setup-node v7.0.0, pnpm/action-setup v6.0.10). - Narrow the docs trigger to code under shell-docs/src, excluding src/content (1000+ MDX/JSON prose files that cannot hold a posthog.capture call site) so prose edits stop firing a full install. - Note in the zizmor justification why setup-node v7's new package-manager-cache auto-path still leaves this workflow cacheless (it engages only for npm-declared repos; this one declares pnpm). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Benjamin Taylor
|
5855496103 |
fix(telemetry): reconcile + commit canonical in fragment PRs
A fragment-only PR fails oss-path-to-production's telemetry-reconcile gate (it recomputes telemetry-events.json and fails on staleness). After emitting each fragment, install the registry's deps and run pnpm reconcile, then include telemetry-events.json in the PR alongside the fragment — matching the Intelligence CLI + surface-emitter pattern. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Benjamin Taylor
|
c785dcea64 |
fix(telemetry): use TELEMETRY_REGISTRY_APP_* (dedicated registry App), not DEVOPS_BOT
The cross-repo fragment PRs must be authored by the dedicated telemetry-registry GitHub App that's installed on oss-path-to-production (the same App the Intelligence CLI release workflow uses), not CopilotKit's DEVOPS_BOT release bot. Switch both workflows to app-id/private-key from secrets.TELEMETRY_REGISTRY_APP_ID / TELEMETRY_REGISTRY_APP_PRIVATE_KEY and gate the mint on the App ID env var. These secrets must be added to the CopilotKit repo (they currently live only on Intelligence). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Benjamin Taylor
|
6e702c906b |
feat(telemetry): emit telemetry-registry fragments for runtime + docs surfaces
Adds scripts/telemetry/ (emit-fragment.ts + extract.ts) and two CI workflows that generate CopilotKit's telemetry-registry fragments and open path-limited PRs into CopilotKit/oss-path-to-production: - runtime (bespoke catalog): reads the AnalyticsEvents type map for event names + properties, scans capture() sites for call_sites, fails loud if the v1/v2 catalogs diverge. Triggered on stable monorepo release. - docs (callee mode): extracts inline posthog.capture literals from showcase/shell-docs (drops $-reserved events). Triggered on push to main touching showcase/shell-docs/**. Both are content-gated: the fragment is left untouched (and no PR opened) when the event set is unchanged, so releases/edits don't churn the registry. Cross- repo token follows the least-privilege recipe (no owner, bare repositories, contents+PR write); mint gated on a job-level env var. zizmor clean (one justified cache-poisoning suppression). 13 unit tests; tsc + oxlint clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Tyler Slaton
|
55aaad21a6 |
test(react-core): add React 18 + 19 unit-test CI matrix (#6060)
## Summary
Adds a `react-version` matrix axis (**18**, **19**) to the unit-test
workflow so react-core, react-ui, and a2ui-renderer are exercised across
the full **supported peer range** (`^18 || ^19`), not just the
repo-default React 19.
This is a **reconstruction of the durable parts of #4221**
(@tylerslaton) onto current `main`. That PR went stale (~5,000 commits
behind, conflicting) and never landed. Rather than rebase it, this
rebuilds its design fresh — and deliberately **scopes to the supported
React range**: React 17 is dropped, because it is no longer a supported
peer version and carried ~80% of the original PR's complexity
(polyfills, `use-sync-external-store` source shims, `jsx-runtime`
aliases, a legacy `renderHook` fallback).
## What surfaced
Dropping R17 and validating R18 revealed a **latent React 18
incompatibility on current `main`**: the `window = {}` test pattern
crashes React 18's concurrent renderer with `"Should not already be
working."` mid-commit, which then corrupts the scheduler for the rest of
the file — **22 failures across 5 files** under React 18. (React 19
happens to tolerate the empty-window swap, so it was invisible until
now.)
The original PR fixed this but mislabeled it R17-only; it's actually
needed for R18, a *supported* version. So the matrix earned its keep on
day one.
Replacing `window = {}` with `stubWindowLocation()` is the load-bearing
fix — it resolves the crash cascade. Separately, **two** tests differ
under R18 purely in *render scheduling*, and are handled by narrow
version gates:
| Test | React 18 behavior | Why it's not a bug |
|---|---|---|
| `renderCustomMessages` → "executes multiple renderers in order" |
`executionOrder` is `["first", "first"]` | Renderer double-invoke.
`second` still never runs, which is the actual contract. |
| `use-human-in-the-loop` → `statusHistory` | `inProgress → executing →
inProgress → complete` | Transient backwards transition from extra
effect runs. Start, end, and the set of observed statuses are all still
correct. |
**No assertion tolerates a different state value.** An earlier revision
of this PR also relaxed the three-turn state-snapshot assertion to
accept `Turn: 2` on R18; @tylerslaton correctly flagged that as an
observable-behavior difference rather than a scheduling artifact.
Re-verified against a real 18.3.1 install — the strict `Turn: 3`
assertion passes **25/25** consecutive runs — so that gate was
unnecessary and has been removed (`a227f46a8`). The two gates above were
re-tested the same way and both genuinely reproduce.
## Changes
| File | What |
|---|---|
| `.github/workflows/test_unit.yml` | `react-version: ["18","19"]` axis.
R19 installs frozen; R18 overrides the root `pnpm.overrides` React
version and installs unfrozen. Adds a guard verifying the installed
React matches the matrix leg, and suffixes `NX_CI_EXECUTION_ID` with the
React version. Layered on top of the existing nx-affected selection
logic. |
| `test-helpers/stub-window-location.ts` *(new)* | Clears
`window.location` (so the localhost auto-open-inspector heuristic skips)
while keeping the real jsdom window — the safe replacement for `window =
{}`. |
| `use-agent-error-state`, `CopilotKitProvider.onError`,
`CopilotKitProvider.test` | Swap `window = {}` for
`stubWindowLocation()`. |
| `use-human-in-the-loop.e2e`, `renderCustomMessages.e2e` | Two
React-version-gated assertions, both **render-scheduling only** (see
table above). State assertions stay strict on every leg. |
No dependency or lockfile changes. None of the R17-only machinery from
#4221.
## CI cost
Full runs go from 3 legs (node 20/22/24) to **6** (node × react). On
PRs, nx-affected still scopes what actually builds/tests; the full 6×
only hits `workflow_dispatch` or when `test_unit.yml` itself changes (so
this PR runs all 6). This is the honest price of adding R18 coverage.
## Testing
Run locally in a worktree via the exact install-override logic the
workflow uses — `react`/`react-dom` → 18.3.1,
`@types/react`/`@types/react-dom` → `^18`, `@testing-library/react` →
`^14.3.1`, `streamdown>react` → 18.3.1, then `pnpm install
--no-frozen-lockfile`. Installed versions confirmed by resolving from
`packages/react-core` (18.3.1 / 19.2.3, `@testing-library/react` 14.3.1
on the R18 leg).
| Check | Result |
|---|---|
| react-core full suite @ React 18.3.1 | **117 files, 1433/1433
passing** ✓ |
| react-core full suite @ React 19.2.3 | **117 files, 1433/1433
passing** ✓ |
| Strict `Turn: 3` state-snapshot assertion @ R18, ×25 runs | **25 pass
/ 0 fail** — gate removed as unnecessary |
| `executionOrder` gate reverted to strict @ R18 | **fails**
(`['first','first']`) — gate justified |
| HITL `statusHistory` gate reverted to strict @ R18 | **fails** (extra
`inProgress`) — gate justified |
| `oxlint` (project-aware) | **0 warnings, 0 errors** — unchanged from
`main` |
| `oxfmt --check` | clean |
| Workflow YAML parse + lefthook commit hooks (lint-fix, package tests,
commitlint) | green |
Before the `window` fix, the R18 leg was **22 failing across 5 files**;
it is now fully green.
Credit to @tylerslaton for the original design in #4221, and for
catching the over-relaxed state assertion in review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Sam Julien
|
5a142af768 |
feat(skills): add setup-slack-channel and split it from copilotkit-channels (#6340)
## Summary Brings the `setup-slack-channel` skill into the set that `copilotkit skills install` distributes, and narrows both it and `copilotkit-channels` so they stop matching the same request. ## Why The skill was merged into [CopilotKit/channels-sdk#7](https://github.com/CopilotKit/channels-sdk/pull/7), which makes it reachable from a channels-sdk checkout and nowhere else. `copilotkit skills install` reads `CopilotKit/CopilotKit/skills`, so until the skill lives here, no CLI user can get it. No CLI change is needed to pick it up: skill names are free-form (`--skill` is validated by regex only, with no allowlist) and the default install is `--skill *`. ## How **The skill is copied from the merged channels-sdk source**, with one addition: a `version: 1.0.0` frontmatter field, so it matches the other standalone skills in this repo (every one of them carries a version; only the package-mirrored skills omit it). The body, references, and manifest asset are otherwise unchanged. It stays browser-first and makes no reference to the `copilotkit channels` commands, per the call to defer those to the web app until they have more real-world usage. **Both descriptions are narrowed** so the trigger overlap is gone: | Skill | Owns | Hands off | | --- | --- | --- | | `copilotkit-channels` | The code half — declaring, wiring, and customising a Channel. Assumes the provider app exists | First-time Slack setup → `setup-slack-channel` | | `setup-slack-channel` | The provider half — the Slack app, its tokens, attaching it to a Channel | Code questions → `copilotkit-channels` | Before this, both matched "connect my agent to Slack" and the agent picked whichever it read first. That collision was invisible in channels-sdk, where `setup-slack-channel` is the only skill present; it becomes live the moment both ship in the installed set. ## Notes for reviewers - **Scope is Slack only.** A Teams sibling is deliberately a separate pass. - **No existing skill content changes** — the `copilotkit-channels` diff is its frontmatter description and nothing else. - Two known follow-ups are tracked and intentionally not addressed here: [channels-sdk#9](https://github.com/CopilotKit/channels-sdk/issues/9) (CLI-capability claims, deferred by decision) and [channels-sdk#2](https://github.com/CopilotKit/channels-sdk/issues/2) (`build-channels-bot` staleness, unrelated). Refs CopilotKit/channels-sdk#10 ## Added during the `main` merge Resolving the conflict surfaced a second, unrelated problem that had to be fixed for this PR to be safe to land. `skills/setup-slack-channel` is a **standalone** skill — it has no `packages/*/skills/` source. `scripts/sync-plugin-skills.ts` treats any such directory as an orphan unless it is listed in `RESERVED_LIFECYCLE_SLUGS`, which this one was not. Verified against the pre-fix script: - `pnpm check:plugin-skills` → exit 1, `orphan file(s) in mirror: skills/setup-slack-channel` - `pnpm sync:plugin-skills` (write mode) → **recursively deleted all 8 files of the new skill** The `plugin-skills-check` workflow's path filter does not match `skills/setup-slack-channel/**`, so this PR would not have caught it — it would have gone red on the next unrelated PR that touched the script or a package skill, or silently eaten the skill on the next sync run. Fix is two lines: add the slug to `RESERVED_LIFECYCLE_SLUGS`, and move the paired `size` assertion in `scripts/__tests__/sync-plugin-skills.test.ts` from 9 to 10. The test file already documents this exact hazard in a comment. ### Conflict resolution The only conflict was the `copilotkit-channels` frontmatter description, which #6320 rewrote in parallel. The two sides disagreed about Teams: this branch said the skill "assumes the provider app already exists", while #6320 established that Teams provider setup **is** this skill's job because the CLI or dashboard wizard performs it. The resolution keeps this branch's code-half framing and the `setup-slack-channel` handoff, but scopes that handoff to *first-time Slack app creation* only — so it contradicts neither #6320's Teams sections nor the Slack provider troubleshooting that stays in this file. Took #6320's `version: 1.1.0`. |
||
|
GeneralJerel
|
77e5415c45 |
fix(skills): correct stale Slack capability claims and narrow trigger scope
Addresses review on #6340. Interactivity is no longer disabled on the managed path. The shared generator emits `interactivity.is_enabled: true` with an Intelligence-hosted request URL (Intelligence `libs/channels-setup/src/slack.ts:209-211`) and the ingress handles `block_actions` (`apps/app-api/src/routes/channels-routes.ts:868`), so HITL buttons and selects do fire. The skill's own bundled manifest asset already said `is_enabled: true`, so the prose contradicted the file shipped beside it. What is still undelivered is `slash_commands` (absent from the generator) and `view_submission` (`apps/app-api/src/channels/slack-ingress.ts:1050`), so the `onCommand` / `onModalSubmit` warnings stay. Corrected in all three places that claimed otherwise, and the troubleshooting entry now tells the reader a dead button is a real failure rather than a capability limit. Browser-only framing is now a routing instruction rather than an architectural claim, since `copilotkit channels add` does create the Channel and attach the adapter. Same outcome, but it no longer contradicts `--help` for an agent that was told to check it. Trigger scope is narrowed in the frontmatter description instead of rewriting Phase 0. The phases assume OpenTag conventions (`app/channel.tsx`, `app/env.ts`, `INTELLIGENCE_CHANNEL_NAME`, an agent on port 8123), which are not what `copilotkit init` scaffolds — naming that in the description keeps the skill from firing on any "connect my agent to Slack" once it installs into customer repos. Also widens the plugin-skills-check path filter to `skills/**` rather than adding the one new slug. The orphan scan reads the whole mirror, so enumerating individual directories is what let this PR's own blocker go untested and would have armed it again for the next standalone skill. |
||
|
Jordan Ritter
|
a9f98314d0 |
ci(runtime): pin Bun to 1.3.14 for the integration job
`bun-version: latest` let a Bun release change module-resolution behaviour between runs. 1.3.14 is the version the recent passing and failing runs both resolved, so pin it and make the job reproducible. |
||
|
renovate[bot]
|
42e0df471a | chore(deps): update github actions | ||
|
renovate[bot]
|
82c8751b21 | chore(deps): update github actions | ||
|
Maxim
|
25748ffdc0 |
chore: allowlist the reskinnable-demo build config
The repo gates which build-config files may exist (next.config, vite.config, webpack.config and friends) so a new bundler config cannot appear unreviewed. A new app needs its next.config.mjs registered. Kept as its own commit because the check's own failure message asks for CODEOWNERS approval when the allowlist changes — isolating it means that approval applies to exactly one reviewable line rather than being buried in the app scaffold. |
||
|
Benjamin Taylor
|
235872741b |
perf(release): install only what the canary publish path needs
The publish job reinstalled all 4608 projects' dependencies (~41s) to run two things: `pnpm tsx` and `pnpm pack`. A root-only install is NOT sufficient — verified locally that pnpm pack then dies with ERR_PNPM_CANNOT_RESOLVE_WORKSPACE_PROTOCOL, because pack resolves each package's `workspace:` deps into real version ranges and needs its workspace siblings installed to do it. That resolution is what makes cross-scope canary deps pin the same-run version, so it is load-bearing. The true minimum is root + packages/**: measured 17s vs 46s locally, and the packed tarball still carries resolved ranges (^0.4.0, ^1.64.1) with no leftover workspace: refs. Everything trimmed is examples/ and showcase/ app dependencies that no publish step touches. Safe because no publishable package declares prepack or prepare, so pnpm pack runs no lifecycle scripts; voice's prepublishOnly fires for neither pnpm pack nor `npm publish <tarball>`. Stable keeps the full install — its dependency surface is wider (lib/notion.js, the umbrella verifier's root script) and it is the highest-stakes path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Benjamin Taylor
|
0ff45fd77c |
fix(release): don't save pnpm cache from ephemeral canary refs
canary.yml mirrors the source branch to a unique canary/<slug>-<run_id>-<attempt> ref, and Actions scopes cache entries to the writing branch (plus the default branch). A cache saved from a canary ref is therefore unreachable by every later canary — measured at 874 MiB of orphan per run, competing for the repo's shared 10 GB budget and evicting entries other workflows depend on. It also cost more than it bought: 20s + 26s of post-job saves against a 25s faster install. Canaries now restore read-only (free win when main has an entry, zero cost and zero pollution when it doesn't); stable releases, which run on main where a write is durable and reused, do the writing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Benjamin Taylor
|
b8cf4dd848 |
docs(release): note the measured pnpm cache save cost
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Benjamin Taylor
|
6a35e3cdbf |
perf(release): cut canary publish wall-clock roughly in half
The canary flow took ~11.5 min steady-state (and 20 min in an observed run). Measured from run 30473499191, the time went to five avoidable places rather than to real work. 1. `npx --yes npm@11.15.0 publish` ran per package, and npx re-resolves the spec against the registry on EVERY invocation: ~16s of each package's ~21s. A 9-package channels canary paid ~2.4 min of pure npx overhead; a 16-package monorepo release paid over 4 min. Hoist the pinned npm into lib/npm-cli.ts, install it once into a throwaway prefix, and reuse the binary. 2. publish-release.yml was the only workflow in the repo with no pnpm store cache, so all three jobs installed 4608 packages cold every time. Usually ~45s each, but registry-bandwidth bound and heavy tailed: the observed run spent 9m08s here on tarballs arriving at 2-49 KiB/s. Add the same node-version-keyed cache the rest of CI uses. 3. The notify job ran for canaries only to compute "post nothing" — the builder already returns should_post=false for mode=prerelease and the self-watchdog is already gated off. ~85s of dead work on the critical path, since canary.yml waits for the whole run. Skip the job, keeping it reachable for a python_publish dispatch. 4. The build job fetched full history for canaries, which need none (no tag, no GH Release, no release-note commit range, and `nx run-many` resolves no merge base). That rode along in the 837 MiB workspace artifact too. Shallow-fetch prereleases; stable keeps depth 0 because its publish job pushes tags out of that artifact's .git. 5. Two smaller ones: the artifact was gzipped and then re-deflated into the artifact zip (compression-level: 0), and the orchestrator's run-discovery loop slept 6s before its first poll. Verified: 143 release-script tests pass (6 new for the npm-cli helper), actionlint + shellcheck + the scope-dropdown guard are clean, the prerelease dry-run path still enumerates all 9 channels packages, and a live probe confirms the helper installs npm 11.15.0 once (3.2s) and memoizes thereafter (0ms). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jordan Ritter
|
8433f5b118 |
fix(showcase/ci): make the :latest tag monotonic so an older build cannot overwrite a newer one (#6174)
## The race, with evidence Three PRs merged within 34 seconds on 2026-07-26. `Showcase: Build & Push` has **no concurrency group** (deliberately), so all three ran simultaneously and raced to push the same `:latest` tags. | run | commit | PR | start → end | |---|---|---|---| | `30190815370` | `7b28934387` | #6162 | 06:18:05 → 06:29:44 | | `30190823203` | `59f275eedc` | #6161 | 06:18:21 → 06:29:44 | | `30190831480` | `db75a04837` | #6158 | 06:18:39 → **06:29:13** ← newest, finished FIRST | The newest commit finished first, so the older builds overwrote its `:latest`. Per-service job completion times — older beating newer on every shared slot: | service | newer (`db75a04837`) | older (`59f275eedc`) | older won by | |---|---|---|---| | `shell-dashboard` | 06:25:33 | 06:25:34 | +1s | | `showcase-harness` | 06:27:51 | 06:27:55 | +4s | | `shell` | 06:27:17 | 06:27:28 | +11s | | `shell-dojo` | 06:25:10 | 06:25:26 | +16s | **All three runs reported `success`.** Staging served pre-#6158 code while CI, the redeploy gate and deploy verification all looked clean. Same failure class as #6171: a success that doesn't mean what it says. ## What I verified in YAML vs took on trust Verified by reading the files / querying the API: - **Tagging** — `showcase_build.yml` pushed `:latest` **and** `:${{ github.sha }}` in one `depot/build-push-action` step, in both the `build` and `build-starters` matrices. **A per-commit sha tag already existed**; confirmed in GHCR (`showcase-shell-dashboard` has digests tagged `db75a04837…`, `59f275eedc…`, `d28384a2eb…`). - **Nothing serialized the pushes.** No concurrency group; confirmed the header comment states this is intentional. - **Deploy consumes `:latest`** — `verify-railway-image-refs.ts` is the SSOT assertion: staging is `ghcr.io/copilotkit/<repo>:latest` (mutable), **prod is `ghcr.io/copilotkit/<repo>@sha256:<digest>` (already immutably pinned)**. So this race is **staging-only**; prod was never exposed. - **`Showcase: Verify Deploy` structurally cannot catch it.** It is a health probe; it asserts no digest or commit provenance anywhere. Its #6171 per-commit concurrency key is about *which run verifies*, not *what image is running*. A stale-but-healthy service passes. - **The racing runs build DISJOINT service sets** (see below) — I pulled the actual job lists. Taken on trust: nothing material. The issue description matched the API on every point I checked. ## Why NOT a concurrency group `detect-changes` builds a **per-push, path-filtered** matrix, so concurrent runs build overlapping but **non-identical** service sets: - `7b28934387` → ag2, agno, built-in-agent, claude-sdk-python, claude-sdk-typescript, crewai-crews, langgraph-fastapi, langgraph-python, langroid, llamaindex, mastra, pydantic-ai, shell-docs, spring-ai, strands (**15**) - `db75a04837` → crewai-crews, llamaindex, shell, shell-dashboard, shell-docs, shell-dojo, showcase-harness (**7**) `cancel-in-progress: true` would have cancelled the `7b28934387` run and the ~10 services **only it builds would never have shipped** — trading a stale-image bug for a never-shipped bug. Per [GitHub's docs](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#concurrency), `cancel-in-progress: false` is no better: *"Any previously pending job or workflow in the concurrency group will be canceled"* — with three rapid merges the **middle** commit's build is dropped outright. GitHub also does not guarantee FIFO ordering of queued runs. Concurrent runs here are **not redundant**, so they must not be cancelled. **Re the #6171 interaction:** this design does **not** use `cancel-in-progress`, so no build is ever superseded-and-cancelled and the cancelled-slot notifier is never tripped by this change. That interaction stays theoretical — deliberately. ## The design Make the one shared mutable resource monotonic instead of serializing the fleet. 1. The build step pushes **only** the immutable `:<sha>` tag, plus an `org.opencontainers.image.revision` label. 2. A guard resolves the commit behind the current `:latest`, asks GitHub `compare/<theirs>...<ours>`, and advances `:latest` (registry-side retag, no pull) **unless ours is `behind`** — i.e. `:latest` already holds a descendant and moving it would roll staging back. **Fails open by design.** No `:latest` yet, unlabelled legacy image, unreachable API, diverged history → advance. A stuck `:latest` is the very failure being fixed, so it declines only on *positive proof* of regression. **Placement:** in `redeploy-staging` / `redeploy-staging-starters`, immediately before the Railway pull that consumes `:latest` — not per build slot. Those jobs already have Node (build slots do **not**, so per-slot would mean an unpinned `npx tsx` fetch on ~50 parallel runners), and deciding right before the pull makes the window as narrow as possible. The image list is the **same matrix ∩ build-success intersection** that decides what gets redeployed, so a failed build can never move a tag. Also adds `showcase_build.yml` to `showcase_validate.yml`'s trigger paths — the new test asserts against that file's live text, and without the path a PR re-adding `:latest` would never run the test that catches it. ## Tradeoffs / what stays open - **Residual sub-second TOCTOU.** GHCR has no compare-and-swap on tags, so two runs reading `:latest` simultaneously could still both advance. This narrows the window from the whole build (~10 min) to inspect→retag. Fully closing it means retiring the mutable staging tag and pinning staging to digests the way prod already is — a change to the Railway image-ref SSOT contract, not a workflow change. **Recommended follow-up.** - **The guard is inert for one build per image.** Today's `:latest` images carry no labels (verified: `showcase-shell-dashboard:latest` has no `Labels` at all), so the first post-merge build fails open and advances unconditionally — same as today. Protection starts from the second build of each image. - Failure to retag exits non-zero, redding the redeploy job and stopping the deploy. That is intended: redeploying against a tag that did not move is exactly the silent false green being fixed. ## Proof **Red/green on the live YAML.** `advance-latest-tag.test.ts` parses the real `showcase_build.yml` (extending #6171's `redeploy-guard.test.ts` pattern). Reverting the workflow to its pre-fix state: **14 failed / 23 passed**. With the fix: **37 passed**. Full `showcase/scripts` suite: **2382 passed, 73 files**. **The load-bearing predicate, verified live against the real incident commits:** ``` compare/db75a04837...59f275eedc => behind (older run arriving late → DECLINE) compare/59f275eedc...db75a04837 => ahead (newer run → advance) compare/db75a04837...db75a04837 => identical ``` **Label reading, verified against a real multi-platform registry image** — `docker buildx imagetools inspect ghcr.io/astral-sh/uv:latest` piped through `extractRevisionLabel()` returns `3010295ae7ff572de459987ad70db315a62ecd61`, matching `jq` exactly. The platform-keyed shape is handled. **Shell/jq transforms** exercised directly, including the empty-CSV edge case (empty → empty, step skipped by its `if:`). **Lint:** `actionlint` finding counts byte-identical to the pre-change baseline (no new findings; the 11 pre-existing are unrelated). `zizmor --min-severity low` with the repo config: **no findings**. **Typecheck:** both new files are in `showcase/scripts/tsconfig.json`'s include set and produce **zero** errors. Worth stating plainly: `nx run-many -t check-types` **does not reach `showcase/scripts`** — the project isn't in the nx graph and has no `check-types` target (there are 9 pre-existing type errors in sibling files, which is how I confirmed it). So the typecheck above is mine, not CI's. The *tests* are gated: `showcase_validate.yml` runs bare `pnpm exec vitest run` in `showcase/scripts`, which auto-discovers the new file. ### What I could NOT prove **I did not construct a real concurrent race on scratch branches.** Doing it faithfully needs two builds pushing the same GHCR repo with controlled finish ordering, which means merging to `main` — the only branch the build workflow triggers on. No run IDs for a live race demonstration; I am not implying one. Unproven until this runs on main: that `docker buildx imagetools create` retags cleanly under the runner's GHCR credentials, and that `npx tsx` behaves in the redeploy jobs (it is already the established invocation there — `redeploy-env.ts` — so this is low risk, not zero). ## Normal single-merge builds are unaffected No concurrency group is added, so nothing queues or cancels. A lone merge finds `:latest` at its own parent → `ahead` → advances, exactly as before. Cost is one `imagetools inspect` + one `gh api` + one registry-side retag per built service, in a job that already exists — no extra job, no extra checkout, no change to build parallelism. --- Branched from `db75a04837`; #6156/#6159 landed after, so this will need main merged in before it goes green. Probable conflict with the concurrent `git lfs pull` work in `showcase_validate.yml` — my edit there is only the top-level `on: paths:` list, so it should merge cleanly, but flagging it. |
||
|
renovate[bot]
|
4875d6d1f2 | chore(deps): update docker/login-action action to v4.6.0 | ||
|
Alem Tuzlak
|
7e3d06f0d0 | Merge branch 'main' into ci/showcase-latest-tag-race | ||
|
renovate[bot]
|
a36772ffe5 | chore(deps): update docker/login-action action to v4.5.2 | ||
|
renovate[bot]
|
b6e0bc0cd2 | chore(deps): update jarvusinnovations/background-action action to v2 | ||
|
Jordan Ritter
|
15222d8589 |
test(showcase/ci): make the :latest guard suite assert capability, not shape
The previous suite was 37/37 green against a workflow that could not have
run at all: the guard step had no `packages: write` and no GHCR login, so
every `imagetools create` would have 401'd. Asserting that a step EXISTS
proves nothing about whether it can succeed. Each change below was checked
by reintroducing the defect and confirming the test goes red.
Registry-auth precondition (the miss that let the above ship)
- Both redeploy jobs must declare `packages: write` AND a `docker/login-action`
step for ghcr.io, ordered BEFORE the guard. Dropping either reds the suite.
The intersection, executed rather than restated
- The old test only checked that the string `images=` appears in the compute
step; swapping its jq for the full matrix — so a FAILED build moves
`:latest` — kept it green. The real `changed` shell now RUNS, against the
real ALL_SERVICES matrix read out of the workflow, and the emitted
$GITHUB_OUTPUT is asserted: a failed build is in neither set, and the
`skip_build` slot is in `services` but NOT in `images` (handing it to the
guard fails "manifest unknown" and blocks the redeploy for the whole fleet).
Same treatment for the starter lane.
Fixtures joined to real registry output
- `extractRevisionLabel` was pinned to hand-written payloads that were never
compared with reality. Since `readLatestRevision` maps every failure to
null, and null ADVANCES, a parser that silently never matches yields a
permanently-blind guard with a fully green suite. Both fixtures are now
verbatim `docker buildx imagetools inspect --format '{{json .Image}}'`
output (buildx v0.35.0), unformatted so key order survives: a platform-keyed
multi-arch image carrying the label, and our own `:latest`, which turns out
to be a bare config object with NO labels at all. The label key used in the
injection test is read from the workflow's own `labels:` input, so parser
and producer cannot drift apart silently.
Closed vacuities
- The `:latest` ban read one step's `tags` and was vacuously green on an empty
list. It now covers every tagged step plus hand-rolled `docker push`/`docker
tag`/`imagetools create`, and fails on an empty `tags`.
- The `cancel-in-progress` ban read only top-level config; a job-level
`concurrency` on `build` reproduced the exact harm and stayed green. All
jobs are checked now.
- `already-current` is keyed on the digest, matching the script.
- The unreachable `(null, "ahead")` row is labelled as the defensive
input-space case it is, and a new test pins that the real flow never calls
compare() with an unknown revision.
New coverage: classifyProbeFailure (incl. a REAL execFileSync timeout, and
the two precedence traps — gh's rate-limited 403 is throttling, not auth; a
registry's 404-with-denied is auth, not absent), readFlag, escapeAnnotationData
(a forged `\n::error::` stays inert), isDirectInvocation through a symlink,
the `::error` annotation on fleet failure, and the digest-mismatch advance.
Also: GITHUB_SHA / GITHUB_REPOSITORY are no longer shadowed in the workflow.
The runner exports both and the script reads process.env, so the old
assertions pinned a redundancy rather than a capability — removed together,
as the comment there required. Workflow-reading scaffolding duplicated with
redeploy-guard.test.ts is extracted and the YAML parse memoized.
|
||
|
Jordan Ritter
|
428c9c301f |
ci: keep the guard's GITHUB_SHA/GITHUB_REPOSITORY env, asserted by the contract test
Reverts only the cosmetic third item from the previous commit. Dropping the
redundant GITHUB_SHA / GITHUB_REPOSITORY shadowing is functionally correct --
the runner exports both and advance-latest-tag.ts reads them off process.env
-- but showcase/scripts/advance-latest-tag.test.ts asserts those exact keys
are declared on the guard step, so removing them red "Validate Showcase".
RED (run 30238078822, "Validate Showcase" > Run build pipeline tests):
Failed Tests 2
FAIL advance-latest-tag.test.ts > showcase_build.yml - redeploy-staging job
> runs the guard with the image list, sha, repo and a token
FAIL advance-latest-tag.test.ts > showcase_build.yml - redeploy-staging-starters
job > runs the guard with the image list, sha, repo and a token
AssertionError: expected undefined to be '${{ github.sha }}'
- Expected: "${{ github.sha }}"
+ Received: undefined
advance-latest-tag.test.ts:424
GREEN (same six contract assertions per job, replicated against the YAML --
the test is pure yaml-parse, no runtime deps):
redeploy-staging / redeploy-staging-starters
PASS IMAGES PASS GITHUB_SHA PASS GITHUB_REPOSITORY
PASS GH_TOKEN truthy PASS guard `if` PASS guard BEFORE redeploy
The test file is owned by a concurrent agent on this PR, so the assertions
cannot be relaxed here. If the redundancy is worth removing, the env keys and
the assertions must go together in one change.
Everything else from the previous commit is untouched and was already
validated by that CI run -- only these two assertions failed, so the GHCR
login, `packages: write`, the skip_build/images filter, the timeout bump and
the tsx pin all passed.
Lint unchanged: actionlint 10 findings before and after (0 new); zizmor
CI-equivalent reports "No findings" before and after.
|
||
|
Jordan Ritter
|
3450d7d7cd |
ci: give the :latest guard GHCR auth and keep skip_build slots out of the retag set
Two Critical defects in the new "Advance :latest" guard, both of which would have broken staging redeploys on this PR's own merge commit. 1. Registry auth missing. `redeploy-staging` and `redeploy-staging-starters` run `docker buildx imagetools create` (an authenticated GHCR WRITE) but declared only `permissions: contents: read` with no `docker/login-action` step. Every retag would 401, the guard would exit 1, the redeploy step would be skipped -- and since the build step no longer pushes `:latest` itself, `:latest` would freeze permanently and staging would never redeploy again. The same missing auth also 401s `imagetools inspect` on these private packages, which the guard treats as "unknown revision" and advances anyway: it would have failed OPEN and never actually guarded. Fixed by adding `packages: write` plus a GHCR login mirroring the `build` job's existing "Login to GHCR" step. Restores both the read and write path. 2. skip_build slots entered the retag list. The `images` CSV projection did not exclude `skip_build` slots. `webhooks` is built out-of-band, never pushes a `:<sha>` tag, yet reports job.status: success -- so it reached the guard, `imagetools create` failed with "manifest unknown", and the whole fleet's redeploy was blocked. Fires on `service=all` dispatch and via the `workflow_config` path filter. Fixed by carrying `skip_build` through the jq projection and filtering it out of `images` ONLY. It stays in `services`: bouncing webhooks so Railway re-pulls its out-of-band `:latest` is the documented intent (see the `webhooks` entry in showcase/scripts/railway-envs.ts). Also, same file: - timeout-minutes 5 -> 20 on both jobs. The guard adds up to 3 serial round-trips per slot (~84 at full fleet width). An overrun concludes `cancelled`, which is invisible to both `if: failure()` and `if: cancelled()` downstream -- the alerting-suppression class this repo has been fixing. - `npx tsx` -> `npx --yes tsx@4.21.0` (the pnpm-lock-resolved version), matching the no-drift convention showcase_validate.yml enforces via `pnpm exec tsx`. `pnpm exec` is unavailable here: no workspace install. - Dropped the redundant GITHUB_SHA / GITHUB_REPOSITORY shadowing; the runner exports both and advance-latest-tag.ts reads them off process.env. Deliberately NOT added: a `concurrency` group. Its absence is load-bearing -- detect-changes builds a path-filtered per-push matrix, so concurrent runs build disjoint service sets and cancelling the older run would leave services never shipped rather than merely stale. RED/GREEN --------- Workflow auth cannot be fully proven without merging, so the proof is scoped to what is actually demonstrable locally. Both probes lift the code under test VERBATIM out of the workflow at runtime rather than paraphrasing it. Probe A -- the real jq projection, fed the real ALL_SERVICES array with a `service=all` / `workflow_config` dispatch (every slot selected, every slot status:success, which is exactly what the skip_build slot reports): RED images : ...,ghcr.io/copilotkit/showcase-eval-webhook (28 entries) services: ...,webhooks -> webhooks IS in the retag set. imagetools create -> manifest unknown. GREEN images : ends ...,ghcr.io/copilotkit/showcase-pocketbase (27 entries) services: ...,webhooks (unchanged, still 28) -> webhooks OUT of images, STILL IN services. Exactly the split wanted. Probe B -- structural parse of the workflow, over every job that runs advance-latest-tag.ts: RED redeploy-staging permissions={'contents':'read'} login=NO redeploy-staging-starters permissions={'contents':'read'} login=NO VERDICT: 2 retag job(s) lacking GHCR auth GREEN redeploy-staging permissions={'contents':'read', 'packages':'write'} login=YES redeploy-staging-starters permissions={'contents':'read', 'packages':'write'} login=YES VERDICT: 0 retag job(s) lacking GHCR auth Lint (delta vs the pre-fix file, not raw counts) ------------------------------------------------ actionlint : 10 findings before, 10 after; 0 new, 0 removed (all pre-existing depot runner-label + SC2086 noise). zizmor : CI-equivalent (--min-severity low, default persona, repo config) reports "No findings" both before and after. Auditor persona at all severities is also identical before/after (1 concurrency-limits, 1 template-injection, 3 undocumented-permissions, 6 anonymous-definition, 9 secrets-outside-env) -- the two new `packages: write` entries carry trailing justification comments so they add no undocumented-permissions findings. |
||
|
Jordan Ritter
|
5afff788dd | Merge remote-tracking branch 'origin/main' into ci/showcase-latest-tag-race | ||
|
Jordan Ritter
|
fe197ff72d |
fix(showcase/ci): make the :latest tag monotonic so an older build cannot overwrite a newer one
Three merges landed on main within 34 seconds on 2026-07-26. The build workflow has no concurrency group by design, so all three ran at once and raced to push the same `:latest` tags. The NEWEST commit's build finished FIRST, so the two older builds overwrote it: run 30190815370 |
||
|
Jordan Ritter
|
5bf4b6f74f |
ci(showcase): pull demo-file LFS objects in the Python unit-test job
PR #6163 converted showcase/integrations/*/public/demo-files/* to Git LFS. showcase_validate.yml's python-unit-tests job checks out without LFS, so the working tree holds 129-byte pointer text instead of the real assets. The new ms-agent-python multimodal test is the first Python test to read those bytes: pypdf reads the pointer, extracts nothing, and the test fails with "real pypdf text extraction produced nothing". Fetch only the demo-file assets (40 objects, ~320 KB) rather than setting lfs: true, which would pull all ~248 tracked objects (~475 MB, including 13-26 MB README gifs) into a 2-4 minute job under a 10 minute cap. A job that hits timeout-minutes reports 'cancelled' -- neither success nor failure -- and silently suppresses alerting, so the timeout margin is worth protecting. The repo is public, so LFS downloads resolve anonymously and the pull works with persist-credentials: false. A post-pull %PDF- header check fails the step immediately if the pull ever no-ops, instead of surfacing minutes later as a misleading pytest assertion. |
||
|
Jordan Ritter
|
b065664bc9 |
fix(showcase/ci): give the monorepo-root build slots a realistic timeout budget
Root cause of the cancellations, from the Depot logs of the killed slots: they were not a transient flake. Every one died at EXACTLY its `timeout-minutes` value (shell/shell-docs/shell-dashboard/shell-dojo at 10m04s of a 10-minute budget; showcase-aimock at 5 minutes), and the log ends with `failed to solve: Canceled: context canceled` followed by Depot's `Step canceled by GitHub`. GitHub reports a `timeout-minutes` kill as job conclusion `cancelled`, which is what fed the whole silent chain. The builds were making real but slow progress the entire time — repeated docker.io base-image pull stalls, and the final `generate-registry` layer alone took 54s — and shell was still 2 steps from done when the budget killed it. The `context: "."` slots build the whole monorepo root and are the heaviest in the fleet, yet they carried the SMALLEST budgets (10 min vs 15 for each small per-integration build). Measured, same slots: warm / uncontended ......... 2-5 min contended, killed at ....... 10 min (runs 30162773601, 30162770765) showcase-pocketbase ........ 8.8 min of a 10 min budget (near-miss) So a retry is the wrong lever twice over: a cancelled job cannot run further steps, and the build was not erroring. Give the four shell slots 20, showcase-aimock 12, and showcase-pocketbase 15. `webhooks` stays at 5 (it is skip_build, measured at 0.6 min). This is mitigation, not the root fix — the underlying cause is Depot builder contention from three concurrent full-fleet rebuilds (~84 simultaneous amd64 builds). Noted inline as a follow-up, since the durable lever is a non-cancelling concurrency queue and this workflow deliberately has no concurrency group. |
||
|
Jordan Ritter
|
2e85bfab08 |
fix(showcase/ci): verify partial deploys and stop cross-commit verify preemption
Two independent ways verification was silently lost on 2026-07-25.
1. `resolve-matrix` required `github.event.workflow_run.conclusion ==
'success'`. But `redeploy-staging` gates on the artifact-derived
`any_success`, not on the matrix rollup, so a build with some slots
cancelled (rollup `cancelled`) or failed (rollup `failure`) still
pushes real images and still redeploys the slots that built. Run
30162773601 redeployed 23 services to staging and this workflow never
started. An unverified real deploy is worse than a verified partial
one, so the trigger is now "the build reached a terminal conclusion"
and WHAT to verify stays decided by evidence: no redeploy-summary
artifact still means has_services=false and verify is skipped, and the
redeploy gate still narrows to the per-service success set, so a
cancelled slot can never be probed against a stale `:latest`.
2. The `showcase-verify-deploy` concurrency group was global, so any
later-finishing build preempted an earlier run's verification even
though they verify DIFFERENT commits. Verify run 30163309977 for the
one genuinely successful build of the day (#6168,
|
||
|
Jordan Ritter
|
12de577952 |
fix(showcase/ci): alert and red the run when build slots are cancelled
Build run 30162773601 (merge of #6160) forced a full-fleet rebuild; 5 of 28 slots were killed by their `timeout-minutes` budget, the other 23 built and WERE redeployed to staging, and the run emitted no signal at all: `notify` was skipped, so no Slack alert and no PR comment, and the run rolled up to conclusion `cancelled`. No existing guard could catch it. Measured on purpose-built probe run 30166429073 (matrix leg killed by `timeout-minutes`, sibling leg green): killed leg `job.status` ........ cancelled matrix rollup `needs.*.result` . cancelled `if: cancelled()` .............. SKIPPED (evaluated FALSE) `if: failure()` ................ SKIPPED (evaluated FALSE) pre-fix `notify` condition ..... SKIPPED <- the bug post-fix `notify` condition .... RAN <- the fix run conclusion ................. cancelled So `failure() || cancelled()` would NOT have fixed this. The signal has to come from the per-slot build results. - stop laundering `cancelled` into `skipped` in the per-slot writer - expose `any_cancelled` / `cancelled_services` from the aggregator job - add `notify-cancelled-builds`: exits non-zero so the run concludes `failure` rather than `cancelled` (a slot killed by its timeout budget is a failure, and `cancelled` is what suppressed everything), and Slacks the affected service names - add the `any_cancelled` clause to `notify` so the merge author gets the PR comment, with wording that distinguishes incomplete from failed `!cancelled()` is retained on both jobs as the intentional-vs-flake discriminator: a human cancelling the whole RUN makes `cancelled()` true and stays silent, while a leg-level timeout leaves it false and alerts. Extends redeploy-guard.test.ts, which evaluates the LIVE `if:` strings from the workflow, with the exact production scenario. It pins the pre-fix guard string as a literal so the test proves the difference the fix makes, not merely the current behaviour. |
||
|
Jordan Ritter
|
7282ecddfb |
ci(showcase): run the harness and shell-dashboard unit suites in CI (#6168)
## The gap
**No CI job ran the showcase unit suites.** Not "ran them partially" —
no job invoked them at all. Verified per suite against `91f0321397` (the
tip of `main` this branched from), by reading every workflow's actual
`run:` lines rather than trusting job names:
| Suite | Tests | Ran in CI before? | Where |
|---|---|---|---|
| `showcase/harness` | 3646 (3625 pass / 3 fail / 18 skip) | **NO** |
nowhere |
| `showcase/shell-dashboard` | 1331 (1330 pass / 1 skip) | **NO** |
nowhere |
| `showcase/shell` | ~57 (8 files) | **NO** | nowhere |
| `showcase/scripts` | — | yes | `showcase_validate.yml` → "Run build
pipeline tests" |
Why each candidate workflow misses them:
- **`test_unit.yml`** — the one whose name implies it covers this —
excludes them *twice over*: `paths-ignore` lists `showcase/**`, **and**
its nx selection is hard-scoped to `--projects='packages/**'`.
- **`showcase_validate.yml`** runs `pnpm exec vitest run` only in
`showcase/scripts`. It has two `working-directory: showcase/harness`
steps, but they are a CVDIAG perf bench (`vitest bench
src/cvdiag/emit-perf.bench.ts`) and an ESM boot-smoke — neither runs the
unit suite.
- **`static_quality.yml`** does run `nx run-many -t check-types`. This
resolves the reported name/invocation divergence: the *job* is called
`check-types` and the harness's script is called `typecheck`, so no
`check-types` target exists on the harness and **the harness is not
typechecked either**.
Net effect: ~5000 showcase unit tests gated nothing. A PR could carry
real defects in `showcase/harness/src/**` and still show an all-green
check list, because no job structurally could have caught them.
## What this wires
New workflow **`test / unit-showcase`**
(`.github/workflows/test_unit-showcase.yml`), two parallel jobs on PR +
push-to-main filtered to `showcase/**`:
- **`harness unit suite`** — `nx run
@copilotkit/showcase-harness:test:ci`, then `nx run
@copilotkit/showcase-harness:test:quarantine-ratchet`. Driven through nx
per the root `CLAUDE.md` task convention.
- **`shell-dashboard unit suite`** — `vitest run` directly; the
dashboard is deliberately outside the pnpm workspace (see the note in
`pnpm-workspace.yaml`) so it is not an nx project and has no target to
run.
New nx targets are declared in **`showcase/harness/package.json`'s
`nx.targets` block**, not in `nx.json` — they are harness-specific, and
a `nx.json` `targetDefaults` entry would silently apply to any future
project that happens to share the script name. Both are `cache: false`
on purpose: the nx `test` named-input covers `src/**` and `*.test.*` but
**not** `vitest.ci.config.ts` or `vitest.quarantine.json`, so a cached
result could survive an edit to the quarantine list.
### Why a separate workflow rather than steps in `showcase_validate.yml`
That file is a single ~900-line job already budgeted at 25 minutes and
is the busiest merge-path workflow in the repo. Splitting the unit
suites out gives them their own name in the check list, their own
concurrency group, and — since the two suites need different package
managers — lets them run in parallel instead of lengthening the critical
job. Naming follows the existing convention (`test_unit.yml`,
`test_unit-python-sdk.yml`, `test_unit-spring-ai.yml`).
**No `continue-on-error` and no `|| true` anywhere in the file.** A gate
that cannot fail is not a gate.
## Hazards, and how each is handled
### 1. The hanging spike test
*Correction to the reported location:* it is
`showcase/shell-dashboard/tests/runtime-env-switch.spike.test.ts`, not
under `showcase/harness/`. It reaches the dashboard suite via that
package's `include: ["src/**/*.test.{ts,tsx}",
"tests/**/*.spike.test.ts"]`.
**Reproduced locally.** vitest sat at **0.0% CPU with zero output** and
never even spawned the `next build` its `beforeAll` calls — no worker
fork, nothing. Killed at ~2 minutes; consistent with the reported
~25-minute stall.
**Handled by excluding it from the unit gate**, not by a hard timeout.
Justification: it is an integration spike by construction — one `next
build` plus two `next start` boots on fixed ports 3801/3802 — and this
package's own config comment already says it is *"too heavy for the
per-file unit suite"* even though the `include` glob pulls it in anyway.
A hard timeout would convert a 25-minute stall into a red gate with
nothing actionable in it; this suite has to stay a fast, trustworthy
unit signal. The spike needs its own job with a real server budget,
which is out of scope for wiring up the unit gate.
Note the `--exclude` values restate the package's existing config
excludes (`tests/visual/**`, `node_modules/**`) because vitest's CLI
`--exclude` **replaces** the config value rather than extending it —
dropping them would silently re-enable them.
### 2. Gitignored generated artifacts
`showcase/.gitignore` ignores `shell/src/data/*.json` and
`shell-dashboard/src/data/*.json`, but the suites consume them:
- `harness/src/probes/frontend-matrix.test.ts` **statically imports**
`shell/src/data/frontend-catalog.json` — a missing file is a module
*load error*, not a test failure.
- `harness/src/fleet/control-plane/d0-gone-predicate.test.ts` reads
`shell/src/data/registry.json` at runtime.
- `shell-dashboard/src/lib/docs-status.ts` **statically imports**
`@/data/docs-status.json`.
Generated as an explicit job step (`generate-registry.ts`, plus
`probe-docs.ts` for the dashboard), mirroring how
`showcase_validate.yml` drives the same generators. The dashboard job
invokes vitest directly rather than `npm test` specifically so the
`pretest` hook does not re-run `probe-docs.ts` and double its ~50
outbound HEAD requests.
Also load-bearing: `npm ci --ignore-scripts` in the dashboard job. That
package's `postinstall` is `cd ../scripts && npm install`, which would
lay an npm-resolved `showcase/scripts/node_modules` over the
pnpm-managed one.
### 3. The 3 pre-existing failures — quarantined behind a ratchet, none
deleted or weakened
Confirmed byte-identical to `origin/main` and pre-existing. A job that
is red on arrival gets ignored or disabled, so the three files are
excluded via `showcase/harness/vitest.quarantine.json` +
`vitest.ci.config.ts` — **and `scripts/quarantine-ratchet.ts` re-runs
each one and requires it to STILL FAIL.** The moment somebody fixes a
quarantined test, the ratchet goes red and names the entry to delete. An
entry therefore cannot outlive the failure it excuses. The ratchet also
fails on an entry pointing at a missing file, on an entry whose filter
matches more than one file, and on a malformed or under-documented
manifest (every entry must carry `file` / `since` / `reason` /
`unquarantineWhen`).
Local `pnpm test` still uses the unfiltered config, so a developer sees
the quarantined failures.
| File | Disposition | Why |
|---|---|---|
| `src/probes/frontend-matrix.test.ts` | quarantine | Asserts hard-coded
per-frontend counts (`react=660`, `angular=636`) against the
**generated** `frontend-catalog.json`, which has grown to 1302 runnable
cells (`react=664`). Stale literals, not a product defect — the two
assertions that express the test's stated intent ("without loss or
duplication": `length === metadata.runnable`, ids unique) both pass.
Exit criterion: derive the counts from the catalog; re-hard-coding fresh
numbers just resets the clock. |
| `src/probes/helpers/d5-mapping-drift.test.ts` | quarantine |
**Stranded by a completed refactor.** It regex-scrapes
`CATALOG_TO_D5_KEY` out of `shell-dashboard/src/lib/live-status.ts` to
prove the dashboard copy hasn't drifted from the harness's
`REGISTRY_TO_D5`. That file is now a one-line `export *` barrel over the
canonical `harness/src/shared/cell-model/live-status.ts` — there is no
second copy to drift and nothing for the regex to find, so the test
throws its own "could not locate" guard. Exit criterion: cell-model
owner deletes it (single-source is a strictly stronger guarantee) or
rewrites it to assert the barrel re-export. |
| `src/probes/helpers/d5-representatives.test.ts` | quarantine |
**Genuine product gap, deliberately not fixed here.**
`d5-browser-use-smoke` registers `browser-use-smoke` in `D5_REGISTRY`
but `D5_REPRESENTATIVES` has no entry, so
`getD5Representative('browser-use-smoke')` returns `undefined` and the
D5 driver has no representative fixture. The test is correct and
reporting a real hole; the fix belongs to the D5 owner, and this PR
touches no product source. |
### 4. New finding: a load-sensitive flaky test
Not one of the 3, and worth fixing separately:
`src/probes/loader/probe-invoker.test.ts` → *"times out at the invoker
level when enumerate() ignores abortSignal"* asserts a **wall-clock**
`elapsed < 150` against a 100ms invoker timeout racing a 200ms
enumerate. 50ms of headroom does not survive CPU contention — it passed
**5/5 in isolation** but failed twice at ~217ms while the machine was
running other vitest suites, and a 4-vCPU runner executing 173 test
files in parallel is exactly that condition.
Mitigated with `retry: 2` **in the CI config only** (base config
unchanged, so local runs still surface non-determinism). This is not
suppression: a genuinely broken test fails all three attempts and the
gate stays red, and vitest prints every retried test. The real fix is
for the probe-loader owner to drive that assertion off fake timers or
widen the bound, after which the `retry` should come out. Flagging the
tradeoff explicitly rather than burying it.
## Deliberate-breakage proof
A job that has never failed is not yet a gate. Scratch commit
`e100df27a7` broke two product-source files, then `4f092eccb1` reverted
it. The final branch diff is **578 insertions, 0 deletions, and zero
product source touched.**
| Run | SHA | `harness unit suite` | `shell-dashboard unit suite` |
Quarantine ratchet |
|---|---|---|---|---|
| baseline
[`30140087665`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140087665)
| `452a26115e` | **success** (3m33s) | **success** (1m35s) | success |
| **breakage**
[`30140218159`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140218159)
| `e100df27a7` | **FAILURE** (3m31s) | **FAILURE** (1m28s) | success |
The breakage — both reproduced locally before pushing:
- `showcase/harness/src/shared/cell-model/staleness.ts`:
`D4_STALE_AFTER_MS` 1h → 10h (a frozen-green D4 row would credit D4 for
ten hours instead of one)
- `showcase/shell-dashboard/src/lib/sort-order.ts`: `sortOrder` `mastra:
8` → `88`
What the RED run actually caught:
```
harness unit suite — Test Files 3 failed | 168 passed | 2 skipped (173)
Tests 4 failed | 3612 passed | 18 skipped (3634)
FAIL src/http/matrix.test.ts > stale green ladder: raw state='green' but /api/matrix folds to gray (U8)
FAIL src/shared/cell-model/cell-model.contribution.test.ts > family fold: one stale-green sibling forces STALE_DEGRADED
AssertionError: expected 'GREEN_FRESH' to be 'STALE_DEGRADED'
FAIL src/shared/cell-model/cell-model.equivalence.test.ts > golden-master identity > fixture: pos-d4-stale-green
FAIL src/shared/cell-model/cell-model.equivalence.test.ts > golden-master identity > fixture: stale-column
shell-dashboard unit suite — Test Files 3 failed | 64 passed (67)
Tests 4 failed | 1326 passed | 1 skipped (1331)
FAIL src/components/__tests__/depth-utils.test.ts > does not credit D4 when its green chat row is stale
AssertionError: expected 4 to be 3
FAIL src/lib/__tests__/cell-model.test.ts > comm-error staleness window > scopes the window PER ROW FAMILY (G3e)
AssertionError: expected 36060000 to be less than 21600000
```
Three things this establishes beyond "the job can go red":
1. **It catches exactly the class of defect that went undetected on
#6156** — a change to `showcase/harness/src/shared/cell-model/**` landed
4 failures in the harness gate, including two golden-master equivalence
fixtures.
2. **The dashboard job also caught the harness defect**, through the
`live-status.ts` re-export barrel. The shared cell-model now has two
independent gates over it.
3. **`retry: 2` is not suppression.** Every `FAIL` line above appears
three times in the log — the test was retried the full budget and failed
all three attempts. A genuinely broken test cannot pass through it. And
the **quarantine ratchet stayed green on the broken run**, confirming
the quarantine is scoped to its three files and does not swallow
unrelated regressions.
## Runtime
Measured on CI (Depot 4-vCPU), baseline run
[`30140087665`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140087665)
— **both jobs green**:
| Job | CI wall-clock (whole job) | Suite only (local) | Files |
|---|---|---|---|
| `harness unit suite` | **3m33s** | 63.5s | 173 (171 pass / 2 skip,
3616 tests) |
| `shell-dashboard unit suite` | **1m35s** | 7.2s | 67 (1330 pass / 1
skip) |
They run in parallel, so the added critical-path cost is ~3m33s —
comfortably under `showcase_validate.yml`'s existing 25-minute budget
and nowhere near irritating-enough-to-disable. In both jobs the budget
is dominated by install and (for the harness) nx `^build`, not by the
tests.
**No sharding or `nx affected` needed.** `nx affected` would in fact be
actively wrong here: the entire point is that a change anywhere under
`showcase/**` should run the suite covering it, and the historical
failure mode was scope-narrowing, not slowness. If install time later
dominates, the lever is pnpm/npm store caching (already configured) —
not cutting test scope.
## Not covered
`showcase/shell`'s 8 test files (~57 tests) are **also ungated** and I
left them out to keep this change to one concern. They need a third
install path (`showcase/shell` is likewise outside the pnpm workspace)
and its `vitest.config.ts` already has a `globalSetup` that generates
its own `registry.json`. Cheap follow-up.
## Coordination
Concurrent work is adding a vacuity/drift text gate under
`showcase/scripts` and compile-time completeness exports. This PR owns
only the **suite invocation**. The one plausible cherry-pick conflict is
`showcase/harness/package.json` (I added two `scripts` entries and an
`nx.targets` block); `.github/workflows/test_unit-showcase.yml` is a new
file and `nx.json` is deliberately untouched.
|