mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
codex/channel-components-v2
14751 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Mike Ryan
|
5f82fc19af | feat(channels): add streaming channel components | ||
|
Maxim
|
65742fbea1 |
feat(reskinnable-demo): bring every skin to demo-beat parity, and hoist teach mode, PDFs and attachments into the shell (#6455)
Takes `airline` and `keel` from ~1 demo beat each to full parity, and hoists three per-skin mechanisms into the shell on the way. Merges `main`, so `bookstore` is included. **Verified live** by the author: Aeronova's new opening chart and Rowan's beat-3c fix both behave correctly against a running app. ## What changed **1. Three mechanisms hoisted out of the skins and into the shell** `teach-mode recording`, `PDF generation` and `attachment staging` had each been copied into three skins, and the copies had diverged. Every failure mode of a diverged copy is silent — `useRecording` returns inert no-ops outside a provider, `logStep` early-returns while idle — so a broken copy still compiles and renders and is discovered on stage. They now live in `src/shell/teach`, `src/shell/documents` and `src/shell/attach`, with a "DO NOT IMPLEMENT THE CHAIN" guard in `templates.md` so a fourth copy cannot grow. **2. `logistics`, `airline` and `keel` brought to beat parity** `logistics` gained beats 2 and 3a–3d, then 4, 5 and 6. `airline` and `keel` were converted from in-memory `useData` stores to REST substrates and taken through every beat. Airline stays a PASSENGER concierge on purpose: its beat-6 gate is ENTITLEMENT (a fare's own conditions), not organizational authority — a rejected first attempt reframed it as an ops-control desk, and the passenger framing turned out to make the gate stronger, since no choice of option can evade a fare rule. **3. `main` merged, including `bookstore`** Seven skins now. The merge conflicted in seven files because both sides hand-maintained the same roster; resolved by taking the union and replacing counts with the commands that derive them. ## Current state, derived rather than asserted ``` ls src/skins/ -> 7 skins ls -d src/app/api/*/v1 -> 7 REST substrates ls src/skins/*/intelligence/seed-memories.ts -> 7/7 grep -rln useAgentContext src/skins/*/layout.tsx -> 7/7 route readables grep -l offerWorkflowRecording src/skins/*/tools.tsx -> 6/7 teach loops grep -l 'useData:' src/skins/*/skin.tsx -> bookstore only ``` `bookstore` is the one skin not demo-complete — it marks beats 3d and 6 `SKIPPED` with a reason in its own beat map, which is a scope decision rather than a gap. It is also the only remaining `useData` implementor, so both substrates are live. ## Bugs found and fixed that were not in scope - **`resolvePage` returned `Object.prototype` members.** `/banking/constructor` answered 500 where it owed 404, on three shipped skins: an object literal inherits the prototype, so `PAGES["constructor"]` is a truthy Function and `?? null` never fires. Fixed, plus a shell guard walking every registered skin, mutation-verified. - **A real `TS2352` in a test file** that three green gates missed, because nothing in this repo type-checks tests. Now `pnpm typecheck`. - **A genuinely flaky test** in `shell/attach`, quantified at 39ms against a 40ms budget under load. Its old assertion also passed under a mutated implementation; the replacement drives the encode instead of timing it. - **Rowan's beat-3c pill described the levers instead of firing the HITL card** — the tool said "confirm the levers with them first" without saying the card IS the confirmation, and the prompt never named the tool. ## Verification `pnpm lint` · `pnpm typecheck` · `pnpm test:unit` (214 files / 2448 tests) · `pnpm build` — all clean. ⚠️ **What tests cannot cover.** Beats 2, 4, 5 and 6 are runtime-conditional and need a live Intelligence stack. The suites prove the code and the prompts are right, not that the model obeys them. Airline's chart and Rowan's fix were confirmed by hand; the memory and teach-mode beats on the other skins have not been re-walked. ⚠️ **Several commits used `--no-verify`**, each recording why in its body: the pre-commit hook fails on a pre-existing `@copilotkit/vue` SSR test that times out at 5s on this machine and fails standalone with no merge in progress. This branch's diff is entirely inside `examples/showcases/reskinnable-demo`. Also fixed along the way: `packages/runtime`'s `better-sqlite3` was compiled against Node 24 while `.nvmrc` pins Node 22, so every `SqliteAgentRunner` test threw on load. ## Reskin skill impact Answered per the standing rule in `CLAUDE.md`. The skill was updated in the same PR: the `Skin` contract's `useData` row, the beat matrix, the demo-completeness routing table, the memory-scope guidance (`user`, not banking's `project` — `forget-memories` skips project rows, so a project-scoped learned procedure survives every presenter reset), the beat-3c two-readings failure, and the `resolvePage` prototype hazard. Historical narration was stripped throughout: the docs now record current state and forward instruction only. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Maxim
|
6e07637083 | Merge branch 'main' into feat/reskinnable-demo-beat-parity | ||
|
Tyler Slaton
|
be40072891 |
revert: remove public AEO surface contract (#6483)
## Summary - revert #6458 and remove the public AEO contract, validator, docs page, capability endpoint, CI enforcement, and related tests - preserve the later AEO production synthetics from #6459 by giving them a self-contained host and endpoint configuration - update the synthetic workflow and runbook so they no longer refer to the reverted contract ## Why PR #6458 needs to be rolled back. A literal revert left #6459 importing the removed validator and reading the removed contract, so this PR also decouples that follow-on while retaining its production checks. ## Impact The `/aeo` page and `/.well-known/copilotkit-capabilities/v1.json` endpoint are removed, along with the contract validation CI. Existing website/docs crawler synthetics remain available as a manual workflow. ## Validation - `pnpm nx run @copilotkit/showcase-scripts:test --skip-nx-cache -- __tests__/check-aeo-synthetics.test.ts __tests__/aeo-synthetics-wiring.test.ts` (6 tests) - `npm --prefix showcase/shell-docs test -- src/app/sitemap.test.ts src/lib/__tests__/next-config-redirects.test.ts` (12 tests) - `git diff --check origin/main...HEAD` Reverts #6458. |
||
|
Maxim
|
335209b39a | Merge branch 'main' into feat/reskinnable-demo-beat-parity | ||
|
Tyler Slaton
|
cf59bc51ba | fix(showcase): decouple AEO synthetics from reverted contract | ||
|
Maxim
|
3bf6e30e9a |
feat(reskinnable-demo): open Aeronova's demo on a flight-cadence chart
Beat 1 is the demo's first move, and it was answering "how do my trips look?"
with a trip wall. It now answers "How often do I fly?" with a picture: every
trip on the account laid out on a day scale, a today divider, the disrupted
ones called out, and the average gap between trips.
WHY A STRIP AND NOT BARS. The account holds seven trips across about ten weeks.
Monthly bars collapse that to three columns, hide which trips are disrupted, and
read as a stub on a projector. The strip uses all seven, and the GAPS are the
actual answer to "how often" -- which is why the summary quotes the average gap
rather than a count.
MEASURED against the shipped seed and the app's own clock, pinned in
data/flight-cadence.test.ts:
7 markers - 0 flown - 7 ahead - 2 disrupted - average gap 11 days
Note the clock. This app runs on a FIXED demo clock (`store.ts` publishes
`now: SEED_NOW`, 2026-07-14), not the wall clock, so every seeded trip is AHEAD
and the strip is forward-looking. "About every 11 days" is therefore the honest
answer, and it is a better one than any count of flights behind us.
Structure:
- `data/flight-cadence.ts` -- pure, no React, no Date. Takes `now` as an
argument and reads days out of the ISO string by civil-day arithmetic.
Both rules are load-bearing here: a `Date.now()` would put the divider in
one place on the server and another in the browser (the hydration class
this branch already chased once), and `new Date(iso)` on a string carrying
an airport's UTC offset re-expresses a 23:00 Lima departure as the next
day. `components/local-clock.ts` makes the same argument for display; this
is its data-side counterpart.
- `components/flight-cadence-chart.tsx` -- paints only. Receives `position`
already normalised to 0..1, so there is no date maths in a component where
nothing could unit-test it.
- `showFlightCadence` registered with `useComponent`, NOT `useFrontendTool`:
only a component replays out of thread history, which is what beat 2 asks
the audience to reload and see.
Three details worth keeping:
- Only flights someone HOLDS a booking on are drawn. The ledger's `flights`
also carries the rebooking candidates, and counting offers would inflate
the answer to the question being asked.
- An unreadable departure is DROPPED and counted, never placed at day 0. A
marker at the wrong point asserts a cadence that is false while still
looking like data.
- The helper takes a structural `{ id, flightId }` rather than `Booking`, so
it accepts the client's `BookingDto` without a cast -- and therefore cannot
see `waiverGround`, beat 6's sixth leak channel.
Tests: 12 on the helper (including the offset case, the drop-don't-relocate
case, and the seed figures), 7 on the component (every marker by flight number,
the cancelled trip named in WORDS and not only as a coloured dot, summary and
picture derived from one object), and `beat-1.test.ts` pinning the contract --
pill wording, registration via useComponent rather than useFrontendTool, the
prompt naming the tool and demanding prose alongside the chart, and no `Date`
in either new file.
Also uses airline's existing amber/negative tones from `trip-list.tsx` rather
than inventing a `warn` design token -- there isn't one; the vocabulary is
brand / positive / negative.
Gates: lint clean, tsc 0 errors, 214 files / 2448 tests, build exit 0.
--no-verify for the reason recorded in
|
||
|
Tyler Slaton
|
83bd1f9088 |
Revert "docs: define public AEO surface contract (#6458)"
This reverts commit |
||
|
Maxim
|
e3d9c911a1 |
chore(reskinnable-demo): add a typecheck script and point the docs at it
`tsc --noEmit` is the only command in this tree that type-checks the 211 test
files -- `next build` visits only what the app's module graph reaches, and
vitest does not type-check at all. The docs already said so and told readers to
run `pnpm exec tsc --noEmit`; this makes it a script, so the command people are
told to run is one word and shows up in `package.json` beside the others.
Note this is a NEW convention here, not a missing piece being restored: no
package in this monorepo defines a typecheck script, so build-time checking is
the house norm and test files fall outside it everywhere, not just in this app.
This closes the DISCOVERABILITY half of that gap for this app only.
It does NOT make the check enforced. Nothing runs it unless a person or an
agent chooses to. Wiring it into CI is a repo-wide decision with real CI cost
across 45 packages and is deliberately not taken here.
Earned: a slot reported three green gates (lint, test:unit, build) and still
shipped a TS2352 in a test file, because none of those three look at test
files.
8 doc references updated from `pnpm exec tsc --noEmit` to `pnpm typecheck`
across README.md, CLAUDE.md, SKILL.md and demo-beats.md. Verified the script
runs clean under the new name.
--no-verify for the reason recorded in
|
||
|
Maxim
|
b7c144d94a |
fix(reskinnable-demo): make Rowan's queue pill move the user, not describe the move
Reported from the running demo: clicking "Oldest pending requests" often got a
prose reply --
Confirm the levers and I'll take you there: **pending** only, sorted by
**oldest first**, top **10**.
-- and nothing else. No tool call, no confirm card, no navigation. Beat 3c
failing while looking like it worked: the answer is correct and well formatted,
and "that was a maneuver, not a link" goes unproven.
ROOT CAUSE, and why the model was not disobeying. It was obeying a sentence
that reads two ways. `showRequestQueue`'s description said "Confirm the levers
with them first" without saying WHERE that happens. The HITL card IS the
confirmation -- it lists the levers and waits -- but nothing said so, so
confirming in chat satisfied the instruction as written. Two other things left
it with no reason to prefer the tool:
- `people/agent.ts` never mentioned `showRequestQueue`, or navigation at all.
Nothing connected "show me the oldest requests" to a tool call.
- `top` was `.optional()`, and an optional lever invites the model to go and
ask for the missing value first.
`logistics` hit this and was fixed; `people` never was, because nothing pinned
the fix. This applies logistics' shape:
- the description now says the card confirms, and says not to confirm in prose;
- the prompt gains MOVE THEM, DON'T DESCRIBE THE MOVE, naming the tool and the
"in front of ... rather than describe one" framing;
- every lever is REQUIRED, with 0 as the "no limit" sentinel. That needs no
page change: the render sets the `top` query param only `if (args?.top)`,
which is falsy at 0, so the page applies no limit.
`beat-3c.test.ts` pins all three. It is source-level on purpose -- what went
wrong is what the MODEL was told, which lives in `description` and the prompt,
and nothing else in this app checks either. Mutation-verified: reverting `top`
to `.optional()` turns it red.
NOT changed: commerce. Its `top` is `.int().positive().optional()` with a stated
reason -- omitting it is exactly what its `parseTopLever` honours -- so that is a
different, documented design rather than the same defect. Its prompt already
names its nav tool.
Reskin skill impact: YES, fixed here. demo-beats.md ss 3c now records the
two-readings failure, the quoted prose it produces, both halves of the close
(description AND prompt), and the note that commerce's optional `top` is
deliberate so nobody copies the wrong shape.
Gates: lint clean, 211 files / 2420 tests passing. Committed with --no-verify
for the reason recorded in
|
||
|
Mark
|
8a6d14b29a |
fix(showcase): port pydantic-ai integration to v2 and restore live system prompts (#6379)
Ports `showcase/integrations/pydantic-ai` — the last pydantic-ai surface still on v1 — to Pydantic AI v2. Refs #6364. Three commits plus a bot formatting fix, best reviewed separately. ## 1. `chore(showcase): port pydantic-ai integration to Pydantic AI v2` - **`requirements.txt`** → `pydantic-ai-slim[ag-ui,openai]==2.22.0`, `ag-ui-protocol==0.1.19`. Drops the `opentelemetry-api<1.44` ceiling from #6374; v2 resolves cleanly against otel 1.44.0, so the workaround is no longer needed. `starlette<1.0.0` is unchanged and satisfies v2's `>=0.46.2`. - **9 `StateDeps` imports** move from `pydantic_ai.ag_ui` (removed in v2) to `pydantic_ai.ui`. - **`agent_server.py`** — `Agent.to_ag_ui()` was removed in 2.0.0, so a `mount_agent()` helper builds the equivalent Starlette sub-app and mounts it. The shape is deliberately identical to what v1's `AGUIApp` produced — a Starlette app whose only route is `POST /`, named `run_agent` — so **all 19 mount paths behave exactly as before, trailing slashes included, and no TypeScript route file changes**. `deps` is constructed **per request**. v1's `run_ag_ui` did `deps = replace(deps, state=state)`, handing each run its own object; v2's adapter does `deps.state = state`, mutating what it is given. A single shared instance under v2 therefore lets concurrent runs overwrite each other's state mid-run. ## 2. `fix(showcase): apply the multimodal provider gate to v2 native content` v2's `AGUIAdapter.load_messages` converts AG-UI attachments to native content types *before* the model boundary; v1 delivered the raw AG-UI part dicts. `_NATIVE_CONTENT` listed `BinaryContent` as a flatten fixpoint, so under v2 inline attachments were waved straight through and the entire provider gate was skipped: - inline PDFs were no longer text-extracted, so raw bytes went to OpenAI - unsupported image subtypes (HEIC/SVG/TIFF) were no longer degraded and reached the provider as images, which fails the turn - missing-mime magic-byte sniffing never ran - `AudioUrl`/`VideoUrl` were neither fixpoints nor classifiable, so they hit the fail-loud raise `BinaryContent` is no longer a fixpoint. `_classify_native_content` maps native content onto the same `(kind, scheme, mime, value)` tuple the AG-UI classifier already produces, so **every existing gate applies unchanged** — no gate logic was rewritten. `audio/*` and `video/*` are named explicitly because `_kind_for` routes them to `"other"`, and a missing mime defaults to `"image"` so the sniffer runs. Net behaviour matches v1: a supported inline image still flattens to an `ImageUrl` data URI, which is why most of the suite went green without touching assertions. Five assertions did change. They checked that state-backing content was still AG-UI `InputContent`, which encoded v1's bridging. They now assert the flatten's output (`ImageUrl`) never appears in state — the leak they were written to guard. The adjacent identity and snapshot checks that prove non-mutation are untouched. ## 3. `fix(showcase): gate url-source content and correct the v1-parity claim` Adversarial review of the first two commits found the gate was only half fixed. `_NATIVE_CONTENT` still short-circuited `ImageUrl` and `DocumentUrl`, which v2 builds from unvetted client input, so url-source attachments bypassed the gate where v1 routed them through it: - an `image/heic` or `image/svg+xml` url reached the provider as `input_image`, which the Responses API rejects — failing the turn - an `audio/mpeg` document url reached it as `input_file` - a blank-mime inline PDF went to the image sniffer instead of text extraction, because `load_messages` collapses `ImageInputContent` and `DocumentInputContent` to the same bare `BinaryContent` and erases the modality v1 defaulted on Native content is now gated **before** the fixpoint check rather than instead of it. `_classify_native_content` returns a tuple only when the gate must act; `None` means provider-safe and falls through to the fixpoint, preserving object identity. `ImageUrl` is gated rather than rerouted so a provider-safe one keeps its identity and any explicit `_media_type`. It also corrected a false claim. The `mount_agent` docstring said routing *and* behaviour were unchanged. Routing is; model input is not. v2 defaults `manage_system_prompt='server'`, so each agent's `system_prompt=` now reaches the model. On v1 it never did — `_agent_graph` emitted system parts only `if not messages` and the AG-UI bridge always supplied history — so **18 of 19 agents had silently dead system prompts on main**. A/B on both versions with the same agent and request: v1 sends 0 system-prompt parts, v2 sends 1. The new behaviour is correct and kept; the docstring now says so. ## Verification Against pydantic-ai 2.22.0, in a venv built from this branch's `requirements.txt`: - **52/52 Python tests pass**, up from 42/52. `test_multimodal_content_mapping.py`'s `importorskip` pointed at the removed `pydantic_ai.ag_ui`, which would have skipped all 43 of its tests **green** under v2; it now targets `pydantic_ai.ui.ag_ui` and uses the public `AGUIAdapter.load_messages` in place of the v1 private helper. - **16/19 mounts** return `200 text/event-stream` with `RUN_STARTED … RUN_FINISHED` and no `RUN_ERROR`, driven through the real app with `TestClient` using trailing-slash URLs as the TS routes do. The other three (`/a2ui_dynamic`, `/beautiful_chat`, `/`) reach tool execution and then fail on a raw `OpenAI()` client constructed inside a tool, which the harness cannot intercept and aimock handles in CI. - **Per-request deps isolation** confirmed on `/shared_state_read_write`: state sent by one request does not appear in the next. `build-check (pydantic-ai)` is green on this branch, and because `requirements.txt` changed, the cached pip layer was invalidated — so that was a **genuine fresh resolve of pydantic-ai 2.22.0 inside the real Dockerfile**, not a cached pass. It also confirms dropping the `opentelemetry-api<1.44` ceiling is safe. ### D6 harness probes — run, with a baseline The behavioural gate is the shared harness D6 probes. No CI job runs them for showcase paths, so they were run locally on both this branch and `main`: | | main (v1) | this branch (v2) | |---|---|---| | passed | **33** / 36 | **34** / 36 | | `reasoning-display` | ✗ `no reasoning-role message rendered within 5000ms` | ✅ **passes** | | `gen-ui-agent` | ✗ `waitForTurnComplete … runStartCount=2, done-signal-missing` | ✗ identical error | | `shared-state-read` | ✗ `Strict mode: 1 candidate fixture(s) skipped by sequence/turn state` | ✗ identical error | ```bash cd showcase AIMOCK_URL_LOCAL=http://localhost:4010 bin/showcase test pydantic-ai --d6 --direct --rebuild --cycle --verbose ``` **The port takes D6 from 33/36 to 34/36.** The two remaining failures are pre-existing on `main` with byte-identical error strings — this branch neither causes nor fixes them, and both are tracked in #6381 rather than blocking here. `gen-ui-agent` is root-caused and is not fixture drift: that demo was never ported to pydantic-ai. `src/agents/gen_ui_agent.py` exists in llamaindex with a real `set_steps` tool but has no counterpart here, the route points at `/gen_ui_tool_based/` (the chart-viz agent), and `set_steps` is declared nowhere in the package. The fixture fabricates `set_steps` calls the backend cannot honour, so pydantic-ai rejects the unknown tool and exhausts its single retry. Confirmed live against real OpenAI: the cell returns plain text, which is correct for the code as written. `reasoning-display` going green is the notable behavioural gain, and it retires a documented v1 limitation. `PARITY_NOTES.md:91-97` justifies omitting the reasoning-message branch of `use-rendered-messages.tsx` on the grounds that "PydanticAI's AG-UI adapter does not emit reasoning content today" — true on v1, false on v2. (That block is stale on two further counts: it cites `@ag-ui/core@0.0.43` where `package.json` pins 0.0.57, and claims `ReasoningMessage` is not exported where it is imported at `reasoning-block.tsx:4`.) Correcting it is tracked on #6364. To be precise about what that proves: **v2 forwards reasoning content where v1 dropped it.** The probe supplies the reasoning channel via its fixture, so what is verified is the forwarding path — adapter → AG-UI stream → frontend renderer — end to end. Whether a given model actually emits a reasoning summary live is a separate matter and outside this port's control: it requires a native reasoning model (`reasoning_agent.py` defaults to `gpt-5`, overridable via `REASONING_MODEL`) and, for summary text, a verified OpenAI organisation. A live run here returned prose with no reasoning block, consistent with the org-verification gate rather than anything in the port. Also verified: the image builds from scratch on v2. Because `requirements.txt` changed, the cached pip layer was invalidated, so `build-check (pydantic-ai)` in CI was a genuine fresh resolve of pydantic-ai 2.22.0 inside the real Dockerfile — which also confirms dropping the `opentelemetry-api<1.44` ceiling is safe. ### CI gate coverage, for the record No CI job exercises this package's runtime behaviour on a PR, on this branch or on `main`: - `test / e2e / dojo` runs from the upstream `ag-ui` checkout (`ref: main`) against upstream example agents, and filters on `packages/**` / `sdk-python/**` - `test_showcase-frontend-matrix.yml` is dispatch-only and builds the integration from `base/` — a frozen-backend React baseline - `showcase_validate.yml` asserts `tests/e2e/` exists with a minimum spec count; it does not run it - the package's own `tests/e2e/` (37 files) is invoked by nothing — per `AGENTS.md` rule 1 the measuring test is the shared harness probe, so that layer is legacy ## Remaining for #6364 Two acceptance criteria are outstanding, which is why this says Refs rather than Closes: - the harness D6 value-test (`bin/showcase test pydantic-ai --d6 --rebuild`), which no CI gate runs for showcase paths - `PARITY_NOTES.md` has 6 version-dependent blocks, 4 of which were already inaccurate against the tree before this PR; left alone deliberately to keep this diff scoped ## Possible follow-up `multimodal_agent.py` still reaches into three private APIs (`pydantic_ai._run_context`, `pydantic_ai.models.wrapper`, `pydantic_ai.models.{ModelRequestParameters,StreamedResponse}`) and subclasses `WrapperModel`, overriding `request`/`count_tokens`/`request_stream`. v2 adds a supported alternative: `AbstractCapability.before_model_request`, which receives a `ModelRequestContext` carrying `messages` and `streaming`. Migrating would delete those private imports and ~85 lines. Deliberately not in this PR — it fixes nothing and would obscure the review. |
||
|
Alem Tuzlak
|
4e9eee3094 |
feat(runtime): add MiniMax built-in models (#6464)
Reason: Add the current MiniMax text models to BuiltInAgent model resolution. - Register MiniMax-M3 and MiniMax-M2.7 as built-in model identifiers. - Resolve MiniMax model strings through the global endpoint with API key and regional base URL configuration. - Document both model specifiers and cover global and China endpoint selection. Checks: - `node_modules/.bin/nx run @copilotkit/runtime:test -- src/agent/__tests__/resolve-model-baseurl.test.ts` - `node_modules/.bin/nx run @copilotkit/runtime:check-types` - `pnpm validate:model-names` - `node_modules/.bin/nx format:check --files=packages/runtime/src/agent/index.ts,packages/runtime/src/agent/__tests__/resolve-model-baseurl.test.ts` - `git diff --check` |
||
|
Ben Taylor
|
7187a0aa19 |
fix(channels-slack): three defects that silently broke Slack Block Kit (#6462)
Closes OSS-819. Part of OSS-794, which stays open for the OpenTag demonstration (OSS-820). *Reopened from #6454 — the branch was renamed so Linear links the right sub-issue, and GitHub closed the original rather than retargeting it. Same three commits, unchanged.* Three defects in the Slack Block Kit catalog, each verified against a real workspace. **99 lines changed across three files.** The reason these sat undetected matters more than their size: **a payload Slack refuses produces no error anywhere.** No log line, no exception, no failing test — the message simply never arrives, which is indistinguishable from a bot that had nothing to say. The renderer compounds it by design, dropping unknown nodes silently so one bad node cannot fail a whole message. ## 1. `container` was refused on every send Its children serialized into `blocks`; Slack reads `child_blocks`. ## 2. Every menu, checkbox, radio group, overflow and confirm dialog was refused The codec stamped `type` onto every catalog entry, including composition objects whose schema has none — Slack's option object is `{text, value}`, and the same holds for `confirm`, `option_group`, `conversation_filter`, `dispatch_action_config`, `slack_file`, `trigger` and `workflow`. An unknown field makes Slack reject the entire message, so the whole interactive surface was unusable through `Slack.Object.*`. Measured against a live workspace: **1 of 26 block elements delivered before this fix, 23 after.** Note the existing `native-catalog.test.ts` asserted the very assumption that was wrong — that every entry serializes its discriminator. It was green while the product was broken. It now asserts the corrected rule. ## 3. An image could not use a file already in the workspace The required-field check demanded `image_url` unconditionally; Slack accepts `image_url` *or* `slack_file`. An image needs alt text plus either source now, and passing neither is still an error. ## Two catalog corrections `file` leaves the authorable manifest. Slack: *"You can't add this block to app surfaces directly, but it will show up when retrieving messages that contain remote files."* The same sentence appears verbatim in `@slack/types`' own doc comment. It is an inbound shape; offering it as a component meant offering something that can never succeed. `alert` stays out with its citation — *"Alert blocks are currently only supported in modals."* Verified rather than assumed: Slack's own example payload posted verbatim into a message is refused, while a plain section in the same delivery seconds later arrives. ## How these were found A fixture per catalog entry — 19 authorable blocks, 26 elements, 15 composition objects — with the expected payload **transcribed from `docs.slack.dev`, not captured from our serializer**, delivered through a managed Channel into a real workspace. **55 of 60 deliver.** That corpus is a working instrument, not a deliverable, so it is deliberately not part of this PR — ~1700 lines of fixtures to maintain against a 99-line change is a bad trade for reviewers. It lives with the team and gets re-run when the catalog moves. One methodological note, because it changed what we count as proof: the first live run passed entries that demonstrated nothing. A rich-text block with one unstyled run renders exactly like a plain section; a carousel with one card renders like a card. Both were accepted and worthless as evidence — caught by a human looking at the output, not by the harness. Fixtures had to *exercise* each entry, and that is what surfaced defect 2. ## Found in the same pass, tracked separately - **OSS-817** — the managed path dropped every picker's value (9 of 26 elements). Fixed and confirmed live. - **OSS-818** — handler ids collide across structurally identical messages. ## Verification `test`, `check-types` and `build` green across `channels-slack`, `channels`, `channels-intelligence` and `runtime`, both with and without the fixture corpus present. Every block, element and object was delivered into a live Slack workspace and reviewed by eye. |
||
|
Alem Tuzlak
|
14f90410ff |
docs(examples): fix stale clone paths in v1 example READMEs (#6471)
<!-- Thank you for sending the PR! We appreciate you spending the time to work on these changes. Help us understand your motivation by explaining why you decided to make this change. **Please PLEASE reach out to us first before starting any significant work on new or existing features.** By the time you've gotten here, you're looking at creating a pull request so hopefully we're not too late. We love community contributions! That said, we want to make sure we're all on the same page before you start. Investing a lot of time and effort just to find out it doesn't align with the upstream project feels awful, and we don't want that to happen. It also helps to make sure the work you're planning isn't already in progress. As described in our contributing guide, please file an issue first: https://github.com/ag-ui-protocol/ag-ui/issues Or, reach out to us on Discord: https://discord.com/invite/6dffbvGU3D You can learn more about contributing to copilotkit here: https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md Happy contributing! --> ## What does this PR do? Fixes three `examples/v1/*` README files whose "Clone the repository" step `cd`s into a directory that no longer exists (leftover from when examples were reorganized under `examples/v1/`). Following the README as written fails at the first step with `cd: no such file or directory`. - `examples/v1/chat-with-your-data/README.md`: `cd CopilotKit/examples/copilot-chat-with-your-data` → `cd CopilotKit/examples/v1/chat-with-your-data` - `examples/v1/form-filling/README.md`: `cd CopilotKit/examples/copilot-form-filling` → `cd CopilotKit/examples/v1/form-filling` - `examples/v1/state-machine/README.md`: `cd CopilotKit/examples/copilot-state-machine` → `cd CopilotKit/examples/v1/state-machine` This matches the already-correct format in `examples/v1/travel/README.md`. Docs-only change, no code/behavior affected. ## Related PRs and Issues - N/A ## Checklist - [X] I have read the [Contribution Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md) - [ ] If the PR changes or adds functionality, I have updated the relevant documentation - [X] "Allow edits by maintainers" is checked (lets us help iterate on your PR directly — faster turnaround for everyone) |
||
|
Alem Tuzlak
|
4880afc846 |
test(skills): guard public skill API contracts (#6457)
## Summary - extend the existing public-skill drift suite to validate maintained setup assets against the generated public API manifest - fail when a skill imports an unpublished CopilotKit package or entrypoint, or a manifest-deprecated API - run the guard in the existing plugin-skills workflow when skills or the manifest change ## Why this matters Coding agents copy these skill assets directly into user projects. Mirror-sync tests prove that our duplicated skill files match, but they do not prove that the examples still reference packages and APIs we actually publish. A stale import can make CopilotKit fail at the first install or build step, which is exactly the kind of failure that prevents agents from choosing and successfully adopting us. This PR adds the smallest deterministic guard for that risk. It reuses our existing Vitest suite and canonical public API manifest; it does not introduce an eval harness, run agents, score behavior, collect metrics, add a provider, or add dependencies. ## Scope This is package-contract validation, not behavioral evaluation. Broader questions such as whether an agent follows a skill well, how many attempts it needs, and whether the generated application behaves correctly remain separate work and should start with a concrete decision the deterministic checks cannot answer. ## Verification - `pnpm exec vitest run scripts/__tests__/public-skill-drift.test.ts scripts/__tests__/sync-plugin-skills.test.ts` (17 tests) - `pnpm check:plugin-skills` - `pnpm check:public-api-manifest` - targeted TypeScript, oxfmt, and oxlint checks - mutation check: replacing `BuiltInAgent` with deprecated `BasicAgent` fails with the manifest-provided replacement Linear: PDX-320 |
||
|
Alem Tuzlak
|
44d54c65d6 |
fix(react-core): repair useCopilotReadable effect deps, convert args, and dependencies (#6409)
Fixes #6383. Fixes #6243.
Both issues land in the same 35 lines of `useCopilotReadable`, so they
are fixed together. This PR also covers a third defect neither issue
reports.
All of it traces to a single commit:
|
||
|
Alem Tuzlak
|
19fb1329b7 |
fix(shell-docs): repair 15 reader-visible doc defects (#6425)
Fifteen defects in the shell-docs tree, each verified against the running site or the source of truth rather than pattern-matched. Found while root-causing [PDX-313](https://linear.app/copilotkit/issue/PDX-313). Scoped deliberately: this is content only. The checker changes that surfaced these follow separately. ## Snippet components used with props but never imported (7) The subtlest item here, and invisible to anyone skimming the source. `<FrontendTools components={…} framework="pydantic-ai" />` without an import falls through to `stubWithPartial` in the global mdx-registry, which drops props "on the floor" by design. So `framework` never reached the partial and the shared snippet rendered **untailored** — the reader got generic content on a framework-specific page. The `mastra` and `ag2` siblings were already correct. All seven broken ones are in authored trees, matching the template-residue pattern from OSS-777. ## Tutorial cross-links that land on the homepage (4) `/tutorials/ai-todo-app` and `/tutorials/ai-powered-textarea` have no `index.mdx`, so they `307 -> /`. A reader clicking "next: the todo app tutorial" gets the docs homepage. The pages are at `/overview`. ## Dead `YouTubeVideo` imports (2) The component is provided globally by `mdx-registry.tsx`, and four other pages render it with no import at all. These two imported a module that has never existed in the repo. ## Stale `byoc-*` demo ids (2) Renamed to `declarative-*` in 70e2fb31 (2026-05-10, *"rename byoc-\* slugs to declarative-\*"*); the docs were never updated, so the ids resolve against nothing in the registry. Only the three registry ID references per page change — `snippet_cell`, `InlineDemo`, `IntegrationGrid`. ## What was cut, and why An earlier revision of this PR also rewrote nine `/integrations/<fw>/*` links to their canonical URLs. Checking production, those were never broken: ``` /integrations/adk/quickstart -> 301 /google-adk/quickstart ``` `seo-redirects.ts` keeps that retired surface alive for inbound SEO traffic, so readers always landed correctly. Canonicalizing them is still worth doing — a 301 costs a round trip and couples internal navigation to a legacy surface — but it is cosmetic, and it was padding a diff whose value is the defects above. Dropped; tracked separately. ## Left alone deliberately The `runtimeUrl` / `agent` code samples on `generative-ui/hashbrown.mdx` and `generative-ui/json-render.mdx`. The API routes were renamed to `copilotkit-declarative-*`, but the agent ids were **not** renamed consistently: | demo | agent id | | --- | --- | | `declarative-hashbrown` | `agent="declarative-hashbrown-demo"` (renamed) | | `declarative-json-render` | `AGENT_ID = "byoc_json_render"` (not renamed) | A blind find-and-replace over `byoc-` would have shipped a broken copy-paste sample. Needs an owner's call. ## Review notes 15 files, +17/-12. The seven import additions are the only changes that affect what renders; the rest are identifier strings and link targets. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Alem Tuzlak
|
47ad5e34a3 |
refactor(react-native)!: converge tool-call rendering onto CopilotKit's shared registry (#6438)
## What does this PR do?
`@copilotkit/react-native` maintained a **private tool-call render
registry** (`hooks/RenderToolContext.tsx`) alongside the canonical one
that `CopilotKitCoreReact` already provides — and which every React
Native app already ships, unused. This PR deletes the fork and points
React Native at the shared registry.
That fork caused three bugs:
| Bug | Symptom | Cause |
|---|---|---|
| **Tool renders never streamed** | A component registered with
`useRenderTool` / `useComponent` painted nothing until the tool call
completed | `CopilotChat` used `JSON.parse` on the argument buffer.
While a model writes a tool call that buffer is *invalid JSON by design*
— AG-UI delivers `TOOL_CALL_ARGS` deltas that are concatenated
client-side — so the parse threw on every delta, warned, and fell back
to `{}` |
| **`useComponent` rendered nowhere** | Silently, with no error | It
writes to core's registry; React Native's chat read React Native's
private `Map` |
| **Chat history degraded** | Navigating away from the registering
screen turned earlier tool calls into a `Called: <name>` placeholder |
The private `Map` deleted renderers on unmount; core deliberately keeps
them |
`@copilotkit/react-core` has used `partialJSONParse` on this path since
v2 shipped. React Native diverged because `useRenderToolCall` was
excluded from its re-exports on the stated grounds that it "depends on
DOM elements via `DefaultToolCallRenderer`" — a claim that was never
true of the hook itself. It was only ever reachable through the fat
`/v2` entry, whose weight is the real hazard (#4893). #5883 moved it
into `/v2/headless` on 2026-07-23; the exclusion comment was rewritten
the next day without revisiting the reason.
### What changed
- **One registry.** `useRenderTool` registers through `useFrontendTool`
into `CopilotKitCoreReact.renderToolCalls`. `CopilotChat` and any custom
surface consume react-core's `useRenderToolCall`.
- **Types are derived, not declared.** `RenderToolProps` is now
`React.ComponentProps<ReactToolCallRenderer<T>["render"]>`, so React
Native cannot drift from `ReactToolCallRenderer` — the contract every
registered renderer is actually invoked against. Change that contract
and `check-types` names every React Native renderer the change breaks.
React Native narrows only the *return* type to `ReactElement | null`,
which `FlatList`'s `renderItem` genuinely requires.
_Scope of that guarantee (corrected during review):_ it does **not**
extend to the type react-core publicly exports under the same name.
Web's `RenderToolProps<S>`
(`react-core/src/v2/hooks/use-render-tool.tsx`) is a separate
hand-declared union, generic over a schema, carrying arguments under
`parameters` (not `args`) and declaring `status` as string literals
rather than `ToolCallStatus` members. Both divergences are live today
and nothing type-checks them shut — the one place the shapes meet,
react-core's own bridge, compiles because a string-enum member is
assignable to its own literal type but not the reverse. Aligning web's
alias is a breaking web API change, filed separately.
- **`RenderToolContext.tsx` deleted** (−150 lines), along with 15 tests
that described the removed subsystem. One of them — `unregisters the
render function on unmount` — asserted the chat-history bug as a
requirement.
- **Two structural CI guards for #4893**, in opposite directions: a test
failing if any React Native source imports the fat `/v2` entry, and a
script failing if react-core's `/v2/headless` or `/v2/context` chunks
ever link shiki/mermaid/cytoscape/katex/streamdown. Both were verified
able to fail by deliberately introducing the regression. These are
*structural* assertions, not size budgets — `dev-docs/bundle-size.md`
freezes `limit` fields until OSS-122.
- **`react-native` added to the bundle-size glob**, which it had never
been in, plus a `size:headless` measurement.
React Native also gains capabilities it lacked: render props inferred
from your schema, `name`/`toolCallId` on render props, and `result` on
completed calls.
**Corrected during review — two capabilities this originally claimed are
not delivered:**
- **Wildcard (`"*"`) renderers do not work on React Native.** Because
`useRenderTool` routes through `useFrontendTool` (which calls
`addTool`), `name: "*"` registers a frontend tool literally named `*` —
advertised to the model, and colliding with core's separate
wildcard-executable-tool path. react-core's `useRenderTool` is
renderer-only and special-cases the wildcard; React Native's is not. The
guide now advises against it.
- **`followUp` (and `available`) are not forwarded**, and the handler's
`context` argument is dropped, so `stopAgent()`'s abort signal is
unreachable from an RN handler.
Both are tracked in § Known limitations for the follow-up that converges
React Native onto react-core's hooks — deleting RN's `useRenderTool` in
favour of re-exporting `useFrontendTool` (tool + renderer) and
react-core's `useRenderTool` (renderer-only, wildcard-capable). That is
an API change with its own migration note, so it is not in this PR.
### ⚠️ Breaking (in a minor)
`useRenderToolRegistry` and `RenderToolProvider` are **removed**. Both
are documented on the docs site, so this is a real break — see the
`BREAKING CHANGE:` footer on `db67ccf`, which is what the release notes
derive from, plus the rewritten reference pages.
```diff
- const registry = useRenderToolRegistry();
- const renderer = registry.get(toolCall.function.name);
- return renderer ? renderer({ args, status }) : null;
+ const renderToolCall = useRenderToolCall();
+ return renderToolCall({ toolCall });
```
Also note two semantic changes: `args` is `Partial<T>` **only** while
`status` is `"inProgress"`, and a render function is now captured at
registration — if it closes over changing values you must declare them
in `deps` (React Native previously refreshed the closure on every
render).
**Known limitation:** agent-scoped renderer resolution does not take
effect on React Native. `CopilotChatConfigurationProvider` is not in
RN's provider tree, so `agentId` always resolves to the default.
Renderers still resolve by name; two agents registering the same tool
name resolve arbitrarily. Filed separately.
### A data point worth recording
Adding `useRenderToolCall` to the measured headless entry moved the
bundle **92.8 kB → 92.7 kB**. Flat. The hook React Native spent months
not using was already inside the chunk every RN app resolves whole —
Metro doesn't tree-shake, so the fork never saved a byte. It cost them.
### Testing
- `@copilotkit/react-native`: **253 passing / 22 files** ·
`@copilotkit/react-core`: **1480 passing / 123 files** · `check-types`
and `build` green for both.
- Each of the three bugs has a deterministic test driving a real
`CopilotKitCoreReact` — no mocking of the code under test.
- Both #4893 guards carry mutation evidence: introduce the regression,
watch them fail, revert, watch them pass.
### Follow-up
`useRenderTool`'s JSDoc is split across two blocks, which orphans the
primary description from IDE hover (the `@param deps` warning still
surfaces). One-line fix, deliberately left out of the final fix wave.
## Related PRs and Issues
- **Supersedes #6346** (@davidmckayv) — its diagnoses were correct and
its test assertions are ported here, re-driven through the real registry
rather than a mocked local one. Credited via `Co-Authored-By` on
`4104bd1`.
- Addresses the React Native half of **#4893**.
- Builds on **#5883**, which created the lean `/v2/headless` entry this
PR consumes.
## Checklist
- [x] I have read the Contribution Guide
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation
- [x] "Allow edits by maintainers" is checked
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Alem Tuzlak
|
c16476b960 |
chore: release python sdk 0.1.95 (#6444)
Closes #6231. ## Why `copilotkit` on PyPI is stuck at **0.1.94 (2026-06-04)**. The newest upload of any kind is the **0.1.95a4** prerelease from **2026-06-19**, and four merged sdk-python fixes postdate it — so none of them exist in any installable artifact. Consuming them today requires a VCS pin. The version in `sdk-python/pyproject.toml` was never bumped, which is why nothing published: the Python lane in `publish-release.yml` fires on a merged PR that changes that version and no-ops otherwise. `sdk-python` is not one of the `release / create-pr` scopes (`monorepo | angular | channels`), so it never gets swept along with the JS releases. This PR is the bump. ## What ships Eleven commits since 0.1.94, including the two fixes the issue is blocked on: | commit | landed on main | | |---|---|---| | `44c43e477` | Jul 3 | fold app context into the system prompt — fixes `langchain-anthropic` rejecting a second, non-consecutive system message | | `bb32138e1` | Jul 24 | read copilotkit context from config when state is empty | | `fee7ec237` | Jul 24 | bridge copilotkit context into LangGraph subgraphs | | `a76d59ae0` | Jul 26 | capture subgraph context from run input (#3886) | Plus `ag-ui-langgraph >=0.0.42`, the ag-ui state-channel declaration with the `a2ui_params` host override, and the A2UI single-arg `A2UIToolParams` work. ## Testing - **Confirmed the fixes are genuinely unpublished.** Downloaded the `0.1.95a4` sdist from PyPI and grepped it: `_get_copilotkit_context` and the config-fallback docstring introduced by `bb32138e1` are absent. The reporter's containment analysis is correct. - **Reconciled the one date that looked wrong.** `bb32138e1` carries an author date of Jun 10, before the Jun 19 prerelease, which would suggest it should have been included. Its committer date is Jul 24 — it landed on main after the prerelease was cut. All four fixes genuinely postdate every published artifact. - **Verified all four commits are ancestors of `origin/main`** and touch `sdk-python/`. - **Python unit CI green on main** — `test_unit-python-sdk.yml` succeeded on Jul 27 at `e9148b305`, which is after the last sdk-python change (`a76d59ae0`, Jul 26). - **Matched the precedent.** The previous release, `2b5d2e0113` ("chore: release python sdk 0.1.94"), was a one-line change to the same file. `sdk-python/uv.lock` has no root `copilotkit` entry and `poetry.lock` records only dependency versions, so neither needs to move; there is no `sdk-python/CHANGELOG.md` and no `__version__` in `__init__.py`. `pyproject.toml` is the single source. - `0.1.95` sorts above the existing `0.1.95a4` prerelease, so the publish lane's version-delta detection will fire. ## Follow-up, deliberately not in this PR Seven files pin the old version and should move once 0.1.95 is actually on PyPI — pinning ahead of the publish would break them: - `examples/integrations/{claude-sdk-python,langgraph-fastapi,langgraph-python,strands-python}/agent/pyproject.toml` - `showcase/integrations/{langgraph-fastapi,langgraph-python,strands}/requirements.txt` Two showcase files (`_header_forwarding_middleware.py` in langgraph-fastapi and langgraph-python) also carry comments describing a workaround vendored against "copilotkit 0.1.94's copilotkit_lg_middleware module" — worth rechecking whether the subgraph fixes make that vendoring unnecessary. Keeping the bump minimal so the publish lane cannot be held up by an unrelated example failure. 🤖 Generated with [Claude Code](https://claude.com/claude-code) |
||
|
Ben Taylor
|
3a80c2696e |
feat(vue): mirror React's useAgent thread scoping, remove thread cloning (#6234)
## Problem
Vue's `useAgent` implemented per-thread agent **cloning** — a mechanism
React never had. Passing a `threadId` silently handed you a copy of the
agent:
```ts
useAgent({ agentId: "assistant", threadId: "thread-1" }) // → a clone, keyed (agent, threadId)
```
The clones lived in a module-level `WeakMap` (`globalThreadCloneMap`),
so:
- Nothing tied a clone's lifetime to the scope that created it — they
were never released.
- Components had to *look up* which copy was live.
`CopilotChatMessageView` called `getThreadClone(registryAgent,
config.threadId) ?? registryAgent` just to find the agent actually being
rendered.
- `getThreadClone` / `globalThreadCloneMap` were exported from the
module purely so components could do that lookup.
Meanwhile React grew an explicit contract for the same use case in
#6141: a private *proxied* agent, registered under a local `agentId` and
routed to a `runtimeAgentId`.
## Change
Deletes cloning entirely and ports React #6141's contract to Vue.
`cloneForThread`, `getOrCreateThreadClone`, `getThreadClone` and
`globalThreadCloneMap` are gone — zero references remain, including in
prose.
`UseAgentProps` becomes a base plus a two-branch union, with the same
all-or-nothing rule React now enforces:
```ts
useAgent() // shared registry agent
useAgent({ agentId }) // shared registry agent
useAgent({ agentId, runtimeAgentId, threadId }) // private proxied agent
```
Every partial set — `{ agentId, threadId }`, `{ agentId, runtimeAgentId
}`, `{ runtimeAgentId, threadId }` — is a compile error, backed by the
same three runtime guards with the same messages for callers TypeScript
doesn't reach.
### Parity with #6141
| | React (#6141) | Vue (this PR) |
|---|---|---|
| scoped branch | `agentId` / `threadId` / `runtimeAgentId`, all
required `string` | same, as `MaybeRefOrGetter<string>` |
| unscoped branch | `agentId?: string`, `threadId?: undefined`,
`runtimeAgentId?: undefined` | identical |
| runtime guards | 3 | same 3, same messages |
| thread resolution | prop → chat config, gated on `hasExplicitThreadId`
| identical |
| proxy registration | balanced effect on core + both ids | same deps |
## Two Vue-specific details
Both are load-bearing and were found by tests failing, not by
inspection:
**The pin watcher's first source is `() => agent.value`, not `agent`.**
Vue sets `forceTrigger` when any array watch source is a shallow ref, so
passing the ref directly re-ran the pin on *every* `triggerRef(agent)` —
i.e. every streamed message — re-pinning the inherited thread over one
`CopilotChat` had deliberately set for the chat it renders. Two existing
suites cover this (`uses the explicit agentId and threadId over
inherited configuration`). React has no equivalent hazard because effect
deps compare by identity.
**`CopilotChat` assigns `agent.threadId` inside its `/connect`
watcher**, not a separate one. `CopilotKitCore.connectAgent` reads that
field *synchronously* (`run-handler.ts`) to decide whether a restore is
fresh, so a later assignment lets `/connect` address the previous thread
— skipping the messages/state reset and re-stamping its restore key with
the stale id. Same placement as React's `CopilotChat`.
`CopilotChatMessageView` now resolves the registry agent directly
instead of consulting the clone map, and reads `copilotkit.agents` so it
recomputes when the registry changes.
## What callers see
**One agent per `agentId`** — the model React has always had. Thread
isolation is now explicit instead of implicit: ask for it and you get a
real, separately-registered agent rather than a copy that appears out of
nowhere.
```ts
// before — silently produced a copy of the "assistant" agent
useAgent({ agentId: "assistant", threadId: "thread-1" })
// now — an explicit private agent of your own, routed to "assistant"
useAgent({ agentId: "chat-1", runtimeAgentId: "assistant", threadId: "thread-1" })
```
Nothing in this repo needed updating: `CopilotChat`, `use-capabilities`,
`use-interrupt` and all six example apps already used `{ agentId }`.
`<CopilotChat agentId threadId>` is unchanged for consumers.
## Tests
`use-agent-thread-isolation.test.ts` (433 lines) covered clone semantics
that no longer exist; it's replaced by
`use-agent-thread-pinning.test.ts`, which pins the new invariants — one
instance per `agentId` never a copy, config-thread pinning gated on
explicitness, and all three all-or-nothing guards.
Four component suites used `getThreadClone` purely as a lookup to find
the agent under test and now read from the registry.
`MockMCPProxyAgent` recorded `addMessage` **only inside its `clone()`
override**, so those assertions were passing only because cloning
existed. The recording moves onto the class. `clone()` itself is left
intact everywhere — `CopilotKitCore`'s `SuggestionEngine` still clones
agents (`packages/core/src/core/suggestion-engine.ts`), so removing
those overrides would have planted a latent trap.
## Deliberately not included
Found while reviewing this area, real, but out of scope — each wants its
own change:
- `useAgent`'s header watcher **replaces** `agent.headers` instead of
calling `copilotkit.applyHeadersToAgent()`, dropping per-agent
construction-time headers. Regresses #5635 in Vue; React does this
correctly.
- `credentials` never reach a provisional agent.
- No `onAgentsChanged` subscription anywhere in `packages/vue`, so `()
=> copilotkit.value.agents` as a watch source never re-evaluates on
registry change.
- `/connect` is skipped for a plain `HttpAgent` — the `hasCustomConnect`
prototype comparison matches every real agent. Vue-only, no React
equivalent.
- `CopilotThreadsDrawer.ssr.test.ts` is a latent flake (5s timeout on a
dynamic import; passes in isolation).
|
||
|
Tyler Slaton
|
f13fcb09e9 |
ci: add manual AEO production checks (#6459)
## Summary - add an on-demand production check for the website and docs discovery surfaces defined by #6458 - derive the ten in-scope routes and media types from the public contract instead of maintaining a second monitoring manifest - exercise those routes as four documented crawler user agents with a global concurrency cap of four - validate status, content type, canonical host, robots/sitemaps, one sampled sitemap link, LLM index links, and soft-404 behavior - retain failure evidence and provide a deliberate `exercise_alert` input for proving the `#oss-alerts` path ## Why this matters AEO is a production property, not a one-time content change. A correct repository can still deploy a broken canonical, HTML fallback, stale sitemap, or inaccessible LLM index. Those failures happen at the top of the agent-led growth funnel: if agents cannot reliably discover and verify CopilotKit, downstream recommendation and activation work never gets a chance to perform. This PR adds the smallest useful operating check for that risk. It is deliberately limited to PDX-340's website/docs scope. It does not monitor MCP, the CopilotKit capability document, raw Markdown, Open Graph, or JSON-LD. Existing deploy-parser utilities are reused where practical, requests run with bounded concurrency, and failures include the exact URL, crawler identity, observed status/type, and a bounded response excerpt. The workflow is intentionally manual at first. We should not create a scheduled noisy alarm while the website LLM endpoints are known red, and we should not claim Slack ownership until a deliberate failure proves the secret and alert path. A small follow-up can add the schedule after one normal run is green and one `exercise_alert` run reaches `#oss-alerts`. ## Stacked dependency - Depends on #6458; this PR is intentionally based on `codex/pdx-317-aeo-surface-contract`. ## Validation - `pnpm nx run @copilotkit/showcase-scripts:validate-aeo-contract --skip-nx-cache` - `pnpm nx run @copilotkit/showcase-scripts:test --skip-nx-cache -- __tests__/check-aeo-synthetics.test.ts __tests__/aeo-synthetics-wiring.test.ts __tests__/verify-deploy.drivers.test.ts` (102 tests) - targeted `oxfmt` and `oxlint` checks - `git diff --check` and commit hooks ## Live baseline (2026-08-12) The narrowed production command fails with eight records: four crawler identities × two website gaps. - `https://www.copilotkit.ai/llms.txt` returns HTTP 200 `text/html` with a noindex soft-404 instead of plain text - `https://www.copilotkit.ai/llms-full.txt` returns the same soft-404 The remaining website/docs targets pass: both home canonicals, both robots files, both sitemaps and sampled links, and both docs LLM indexes. The current failures are why this PR ships manual-first rather than enabling a schedule. ## Status PDX-340 remains In Progress until the website endpoints are fixed, a normal workflow run is green, the deliberate Slack alert reaches `#oss-alerts`, and a follow-up enables the agreed schedule. |
||
|
Tyler Slaton
|
d21aebc6e2 |
docs: define public AEO surface contract (#6458)
## Summary - publish a single shared, versioned technical contract for website, docs, and docs MCP AEO surfaces - publish the human policy through the existing shell-docs MDX pipeline at `/aeo` - expose the machine-readable contract at `/.well-known/copilotkit-capabilities/v1.json` - validate the contract with JSON Schema/Ajv plus narrow repository and CI cross-reference checks - run the actual shell-doc behavior tests in CI and assign external website and Pathfinder gaps to named owners ## Why this matters Answer engines and coding agents decide which source to trust from machine signals such as canonical hosts, stable URLs, response types, and consistent capability claims. When those signals disagree, CopilotKit can be classified incorrectly, cited from the wrong hostname, or skipped even when it is the right product. This PR gives those public surfaces a versioned source of truth. It separates standards, community conventions, and CopilotKit-specific guarantees; records real endpoint paths and media types; and makes ownership explicit when behavior lives in another repository or service. That gives us a reliable base for improving agent discovery without pretending one repository can enforce every public surface. The implementation deliberately uses the current docs architecture: `/aeo` is ordinary shell-docs MDX under `showcase/shell-docs/src/content/docs/`, not a bespoke page or the retired docs tree. Schema shape lives in JSON Schema, while the small TypeScript layer only checks relationships JSON Schema cannot express, such as whether referenced files and CI commands exist. ## Validation - `pnpm nx run @copilotkit/showcase-scripts:validate-aeo-contract --skip-nx-cache` - `pnpm nx run @copilotkit/showcase-scripts:test --skip-nx-cache -- __tests__/validate-aeo-contract.test.ts` (6 tests) - `npm --prefix showcase/shell-docs test -- src/app/sitemap.test.ts src/app/llms.txt/route.test.ts src/app/llms-full.txt/route.test.ts 'src/app/llms-mdx/[[...slug]]/route.test.ts' src/app/well-known/copilotkit-capabilities/v1.json/route.test.ts src/lib/runtime-config.test.ts src/lib/__tests__/next-config-redirects.test.ts` (43 tests) - `npm run typecheck` in `showcase/shell-docs` - `npm run build` in `showcase/shell-docs` - targeted `oxfmt`, `oxlint`, TypeScript, diff, and commit-hook checks ## External follow-ups - CopilotKit/website must link the same policy and fix `/llms.txt` plus `/llms-full.txt`, which returned 200 `text/html` soft-404 pages during the production audit - Pathfinder/docs MCP owners must define a machine-readable discovery surface; the current contract records `/sse` as the known transport without presenting transport availability as discovery ## Related - PDX-317 |
||
|
Maxim
|
4b17ea7d35 |
fix(scripts): tokenize before hunting loader calls in the purity gate
The #4893 hard-fail gate's loader-call detector gave WRONG VERDICTS IN BOTH DIRECTIONS. It layered two regexes — a comment/string/template alternation that blanked only the comment branch, and `\b(?:import|require(?:\.resolve)?)\s*\(` over the result — then classified an argument as static from the FIRST CHARACTER after the paren. All nine shapes below were reproduced against the real gate before the rewrite: false FAIL throw new Error("use require(path) instead") false FAIL `import(${x})` inside a template false FAIL o.import(y) / mod.require(x) (member calls, not loaders) false PASS /https:\/\//; …import(n) (the regex's `//` blanked the rest of the line, hiding a real dynamic call) false PASS import(`stream${n}`) (merely STARTS with a quote) false PASS import("zo" + n) (same) false PASS import(`${base}/v2/index.mjs`) (same — the fat entry) false PASS __require(name) (no \b inside `__require`) Replaced with `scanSource`, a single-pass tokenizer that classifies every character as code / comment / string / template / regex and returns a length-preserving masked view plus a literal-span list. The one surviving regex now only ever sees code, so import-shaped TEXT cannot reach it at all; an argument counts as static only when it is one COMPLETE literal with no concatenation or interpolation; `__require` is matched; and a member call is rejected both by lookbehind and by a whitespace-skipping back-scan (so `m\n .import(x)` is not a loader either). Proven in both directions: nine innocent/violation pairs run through the real `assertEntryPurity`, each innocent form CLEAN and each matching real violation FAIL. Re-proved end-to-end by prepending `import "streamdown"` to the real dist/v2/headless.mjs — exit 1 naming all five families — then restoring it byte-identically. On the untouched dist the scan sees 66 loader calls in the `.cjs` graph and classifies all 66 static, so it passes because it LOOKED. Also adds the first `.cjs` fixtures: every existing fixture was `.mjs`, leaving the script's `format: "cjs"` branch and the `require()` shape asserted by nothing. Tests 24 → 47. `stripComments` is renamed `maskNonCode`, since it now blanks literals and regexes too; it had no caller outside this script and its test. The RN guard keeps its own copy, untouched. dev-docs/bundle-size.md: the four holes a sibling agent documented as known limitations this round are closed and removed from that list; what genuinely remains (regex-vs-division heuristic, no JSX/TS, indirect loaders) replaces them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Maxim
|
4c17a8fe8c |
fix(react-native): key the messages fingerprint on object content
`messagesFingerprint`'s content key collapsed every object to 0, so an
in-place content replacement that kept the same message id was invisible to
every memo derived from it. Its comment claimed to mirror react-core's
`messagesMemoKey`, which stopped being true when react-core #6325
(
|
||
|
Maxim
|
1b39c12e36 |
fix(scripts): stop the headless CLI gates skipping themselves on odd paths
Both scripts decided "am I the entrypoint?" by comparing `import.meta.url` to a `file://`-concatenated `process.argv[1]`. `import.meta.url` is percent-encoded and symlink-resolved; raw argv[1] is neither. So the comparison was false for any checkout path containing a space, for any invocation through a symlink (macOS /tmp is one), and on Windows — and a false guard skipped the whole CLI block. Reproduced before fixing: the #4893 purity gate and the bundle-size measurement both exited 0 having printed nothing and asserted nothing, which is worse than a gate with holes because it manufactures confidence. The guard was added by this PR so the modules could export internals to their new negative tests; making the gates testable introduced a way for them not to run. Both now compare real filesystem paths through an exported `isEntrypoint`: `fileURLToPath` defeats the encoding and Windows forms, `fs.realpathSync` on both sides defeats symlinks, and a `path.resolve` fallback keeps a nonexistent argv[1] from throwing. Each `node --test` suite gains five entry-guard tests, including an end-to-end spawn of the real script through a symlinked package-root alias whose name contains a space — the only case that catches the call site regressing back to a string comparison (verified: it fails against the old expression). The unit cases assert the naive comparison really would have failed, so none of them can pass vacuously. Both negative gates were re-proven to still bite: a doctored dist entry pulling streamdown fails the purity gate, and a stubbed dist entry trips the measurement's plausibility floor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Maxim
|
c7d176f264 |
test(react-native): make the #4893 entry guard fail on violations, not on growth
The headless import-graph guard pinned the resolved graph EXACTLY — the
11-module list and the 8 bare specifiers, both `toEqual`. That catch-all was
deliberate (a heavy dependency nobody enumerated still had to be looked at),
but it also went red on innocent growth: adding any first-party `src/` module
to the headless graph failed it, on someone else's unrelated PR. A guard that
fails on innocent changes gets deleted by the third person who hits it, and
then it guards nothing.
Express the catch-all over PACKAGES instead of MODULES: the graph may only
reach packages a headless consumer is guaranteed to be able to resolve — this
package's `dependencies` plus its NON-optional `peerDependencies`, read from
package.json rather than hand-copied. That is precisely the promise the
headless entry sells ("bundles with nothing stubbed in metro.config.js"), so
it still fails on any new third-party edge, on every optional peer, on a
devDependency, and on a Node builtin — while a new first-party module or
another import of an already-sanctioned package is free.
The two other things the pin bought are kept explicitly:
- Comment stripping. The eight phantom specifiers JSDoc examples used to
harvest were all self-references, and this package's own name is not in the
guaranteed set, so a `stripComments` regression still fails here.
- Non-vacuity. Every remaining graph assertion is a deny-list, and a deny-list
over a truncated graph passes for the wrong reason, so a subset floor
asserts the walk still reaches the provider, the polyfills and the
react-core headless edge.
Not changed: comment stripping itself, the import()/require()/require.resolve
extraction, non-literal loader flagging, emitted-extension resolution, the
loud failure on unresolvable edges, the entry-presence tests, the #4893
fat-entry ban (still the assertion that catches `@copilotkit/react-core/v2`)
or the heavy-dependency ban. The runtime-export `beforeAll` is untouched.
Proven both directions: a new first-party module passes the loosened guard and
fails the old pin; `@copilotkit/react-core/v2`, `shiki`, an unenumerated
devDependency edge and a truncated walk each fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Maxim
|
9eacbfdec3 |
docs(react-native): name the TS18048 args break in useRenderTool
The PR's reviewer asked that the `RenderToolProps` shape change be covered where upgraders
actually read it. A `BREAKING CHANGE:` footer on
|
||
|
Maxim
|
4cf8640d97 |
docs(react-native): retract the wildcard-renderer claim for RN
The guide told readers that a `"*"` entry resolves as a renderer-only wildcard
on React Native "exactly as it does on the web", and framed the one difference
as an ergonomic tax (a `"*"` entry "still needs `parameters`"). That is false,
and the real consequence is not ergonomic.
react-core's `useRenderTool` is renderer-only: its body's sole registration is
`addHookRenderToolCall` (use-render-tool.tsx:190), and it special-cases
`name === "*" && !parameters` into a schema-less fallback renderer
(use-render-tool.tsx:166) which `useRenderToolCall` resolves last
(use-render-tool-call.tsx:151). React Native's hook instead delegates wholesale
to `useFrontendTool` (useRenderTool.ts:53), which calls `addTool`
unconditionally (use-frontend-tool.tsx:23). So on RN `name: "*"` registers a
frontend tool literally named `*`. `buildFrontendTools` has no wildcard
exclusion (run-handler.ts:1236), so that tool is advertised to the model in the
run's tool list, and it occupies core's separate wildcard-executable-tool slot
(run-handler.ts:610, 626) whose handler is invoked for every unmatched tool call
with args wrapped as `{ toolName, args }` (run-handler.ts:988) rather than in
the caller's declared shape.
- Rewrote the bullet to advise against `"*"` on React Native and state the
mechanism. Also completed its requirements list: `description` is as
non-optional as `parameters` (useRenderTool.ts:13), and react-core's hook
takes no `description` at all.
- Fixed the example's `status !== "complete"`, which was a web-shaped parity
assumption that does not compile: RN's props derive from
`ReactToolCallRenderer`, whose `status` is the `ToolCallStatus` enum, not
web's string literals. Branches on `ToolCallStatus.Complete` now, and notes
the `args`-vs-`parameters` difference alongside it.
- Added a Known limitations entry, in the voice of the `threadId` one, covering
the wildcard gap plus two more the audit turned up: `followUp` and
`available` are accepted by `useFrontendTool` and not forwarded, and the
`handler` type drops the `context` argument core does pass at runtime
(run-handler.ts:821). All tracked for the convergence follow-up.
No `followUp` claim was present on the page to correct — the gap is real, so it
is documented as a limitation rather than a retraction. No behaviour change:
`packages/react-native/src/hooks/useRenderTool.ts` is untouched, the
convergence is its own PR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Maxim
|
ed087edbf4 |
docs(bundle-size): describe the purity gate that actually exists
|
||
|
Maxim
|
00caf5fa7b |
docs(react-native): fix the chat memo comment's identity rationale
`messagesFingerprint`'s header JSDoc and the matching inline comment near the
`listItems` memo justified keying on message CONTENT with a claim that is false:
that `agent.messages` is mutated in place throughout, and that "the AG-UI apply
pipeline reuses one array for a whole run".
It does not. `@ag-ui/client`'s `AbstractAgent.processApplyEvents` REASSIGNS
`this.messages = applied.messages` for every applied event, so a streaming run
hands down a new array — and new message, `toolCall` and `function` objects — per
delta. Verified against a real AG-UI run by the PR reviewer, and confirmed here in
@ag-ui/client 0.0.57's `AbstractAgent`. The old grep behind the claim ("assigning
`.messages` in packages/core/src hits test files only") is accurate but proves
nothing: `@ag-ui/client` is a dependency, outside that tree.
The fix itself stands. Identity is unreliable in BOTH directions, which is the
actual rationale: it changes on the apply path, and it does NOT change on the
paths these memos exist to serve — core splices tool results in place
(`agent.messages.splice(insertAt, 0, toolMessage)`,
packages/core/src/core/run-handler.ts:931, :1080), `AbstractAgent.addMessage` is a
`this.messages.push(...)`, and `useAgent` re-renders with a bare `forceUpdate()`
(packages/react-core/src/v2/hooks/use-agent.tsx:382-396). A signal that both
misses changes and fires without them cannot be a dependency, so the derivations
must key on content.
Comments only: three sites reworded (the JSDoc, the "cannot be used" pointer at
the `messagesKey` call, and the inline note on the `listItems` memo). `git diff`
touches no behaviour, type or dependency array — every changed line is a comment.
The `contentKey` length-vs-value paragraph is left alone; another change owns it.
Note: the same false claim is in commit 77ed31c437's body, which cannot be
rewritten, and in a GitHub review comment.
Not run in this worktree: it has no node_modules, and the change is comment-only,
so it cannot affect types, lint or tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Maxim
|
21da05c278 |
docs(react-native): retract the useRenderTool status-compare claim
An earlier commit rewrote the useRenderTool reference to say that comparing a render prop's `status` against a raw string no longer type-checks, and called that a breaking change. That was wrong, and it made upgraders believe working code was broken. `ToolCallStatus` is a string enum, and TypeScript relates an enum literal type to a same-valued plain string literal (not the reverse). Equality tests comparability both ways, so `status === "complete"` compiles AND narrows. Only two forms fail: comparing against a string that matches no member (TS2367), and assigning a raw string to a `status`-typed variable (TS2322) — assignment, never comparison. Corrected all three sites that claimed otherwise (the RenderToolProps narrative, the `status` PropertyReference, and the migration section's "both halves are breaking"). The enum-based examples stay, now framed as recommended style — self-documenting, and loud if a member's value changes — rather than a compilation requirement. The migration section now names the added `InProgress` arm as the one genuinely breaking half. Also fixed a separate false claim in the same file: the out-of-chat rendering example said that without a `toolMessage` the status "stays InProgress ... forever". `useRenderToolCall` reads `executingToolCallIds` from the provider and branches toolMessage -> Complete, else isExecuting -> Executing, else InProgress, so the Executing arm is reachable with no tool message at all. Docs only; no source behaviour changes. |
||
|
Jerel Velarde
|
b075704c77 |
feat(examples): add grok-generative-ui showcase (#6475)
## What does this PR do? Adds a new showcase: **`examples/showcases/grok-generative-ui`**. Ask what X thinks about anything. `grok-4.6` runs xAI's **X Search** (`x_search`) server-side, then composes the answer out of real React components through CopilotKit frontend tools — the model picks which components appear and what goes in them. There is no fixed dashboard being filled in. Every post rendered is a real post the model found; nothing on screen is authored by hand. https://github.com/user-attachments/assets/9a818bc7-07bb-442a-82f6-c61e554a983d **What it demonstrates** - 5 frontend tools via `useFrontendTool` — one search, four renderers (`renderSummary`, `renderSentimentSplit`, `renderArgumentMap`, `renderReceipts`) - `BuiltInAgent` pointed at a `LanguageModel` instance (`xai.responses("grok-4.6")`) rather than a model string — which is what makes xAI's server-side tools reachable - Headless `CopilotChatView` with its `scrollView` / `input` slots composed into the page layout, so one mounted chat serves both the centered hero state and the docked rail - Paint-in reveal: each panel walks skeleton → wireframe → rendered as its tool call lands **Gotchas documented in the README** (each cost real debugging time and isn't obvious from the docs) - Registering a backend `ToolDefinition` on `BuiltInAgent` alongside `useFrontendTool` tools **silently stops the frontend tools from reaching the model** — it reports them as unavailable and narrates fabricated results instead of rendering. All five tools are frontend tools here for that reason. - Runs must go through `copilotkit.runAgent({ agent })`; the raw `agent.runAgent()` runs without the registered frontend tools. - `CopilotChatView` returns its own welcome layout **before** reading the `children` render prop, so a composed layout is silently discarded until the first message unless you pass `welcomeScreen={false}`. - `maxSteps` defaults to `1`, so the agent searches and stops before rendering. **Repo-level changes (2 files, both required)** - `.github/config-allowlist.txt` — registers `examples/showcases/grok-generative-ui/next.config.ts`. Required; `static / check binaries` fails otherwise. This is CODEOWNERS-gated, so this PR needs core-dev review and will not qualify for showcase auto-merge. - `examples/README.md` — adds the index row and bumps the counts (24 → 25, 50 → 51). **Notes** - Standalone npm project, pinned deps, own `package-lock.json`. Not added to `pnpm-workspace.yaml`, matching every showcase except the three that are explicitly enumerated there. - `npm install && npm run build` verified clean in place. - No changeset, per CONTRIBUTING. - `demo.png` committed through git-lfs (387 KB, under the 1 MB gate). ## Related PRs and Issues None. ## Checklist - [x] I have read the [Contribution Guide](https://github.com/CopilotKit/CopilotKit/blob/main/CONTRIBUTING.md) - [x] If the PR changes or adds functionality, I have updated the relevant documentation - [x] "Allow edits by maintainers" is checked |
||
|
Ben Taylor
|
a3b814a041 |
fix(core): refresh Intelligence delegate headers before every join (#6469)
## Summary
`ProxiedCopilotRuntimeAgent` builds its `IntelligenceAgent` delegate
**once** and caches it for the proxy's lifetime, copying `headers` into
the delegate's constructor config. Nothing ever refreshed that copy, so
**a header that changed after the delegate was created never reached
`/connect` or `/run`** — for the life of the agent.
For a multi-tenant app carrying the active tenant in a header, the join
was attempted under the *previous* tenant's identity with the *new*
tenant's thread id, and the platform correctly answered
`THREAD_NOT_FOUND`. Only a full page reload cleared it, because that
rebuilds the delegate. A rotated or refreshed `Authorization` bearer has
the same exposure.
Reported by Sameday against 1.67.1 with a deterministic staging repro:
```
19:45:23.554 | /copilotkit/runtime/threads | hdr=<tenant B> | 200
19:45:23.886 | /copilotkit/runtime/threads/subscribe | hdr=<tenant B> | 200
19:45:23.893 | /copilotkit/runtime/agent/<id>/connect | hdr=<tenant A> | body.companyId=<tenant B> | 404
```
`/threads` carries **B** while `/connect` carries **A**, ~340ms apart in
the same switch. Not a race — a stale copy with no refresh path.
## Root cause
`setHeaders` / `applyHeadersToAgent` could not fix this: they write an
agent's `.headers`, and `IntelligenceAgent` exposed only `private
config`. `syncDelegate` *looks* like the refresh path, but its
`hasHeaders` probe is `"headers" in agent` — false for the delegate,
since `headers` is declared on `HttpAgent`, not on `AbstractAgent`. So
`config.headers` was the sole header source for Intelligence REST calls,
with no refresh path at all.
## The fix
Expose `headers` as a public accessor pair backed by `config`, and read
it in `requestJoinCredentials$`.
**The accessor is the entire fix**: it makes `hasHeaders` true, so
`syncDelegate` — which already runs on every `resolveDelegate()`, and is
preceded by `applyHeadersToAgent` in `RunHandler.connectAgent` — starts
actually refreshing the delegate before each join. No new plumbing.
Two things worth flagging for reviewers:
1. **The originally-suggested fix ("make `requestJoinCredentials$` read
live headers") does not work on its own** — and is actively harmful.
There was no live header source on the class to read: without the
accessor, `this.headers` is `undefined` and **every header is dropped**
(verified: only `Content-Type` survives). The read here goes through the
accessor for a single source of truth, not because that read carries the
fix.
2. **The setter replaces the config object rather than mutating it**,
because `clone()` shares the config reference. The join path alone would
mask an in-place write (`syncDelegate` rewrites headers just before
every join), but the credential re-acquisition inside a running pipeline
(`intelligence-agent.ts:563`) does not re-sync — so a clone's tenant
could ride out on the original's socket-error refresh. That's the same
cross-tenant leak this accessor exists to prevent.
`credentials` had the identical defect via `config.credentials`
(`hasCredentials` was false too) and gets the same treatment.
## Testing
**Unit tests (5 new, each written first and watched fail).** The pre-fix
failure is the staging symptom reproduced:
```
FAIL > sends a header changed after the delegate was created
AssertionError: expected { …(2) } to match object { 'X-Tenant': 'tenant-b' }
- "X-Tenant": "tenant-b",
+ "X-Tenant": "tenant-a",
```
Coverage: a header changed post-construction reaches `/connect`; the
same on the `/run` path (which was independently verified broken
pre-fix, sending tenant A where B was expected); credentials likewise; a
clone's header update must not reach the original
(`IntelligenceAgent.clone()` invariant — this one fails under in-place
config mutation); and a per-thread clone and its original each send
their own tenant.
**Verified beyond the unit tests.** Because the mocked-harness result
alone doesn't prove the production wiring, I drove the real chain —
`CopilotKitCore.setHeaders` → registry → proxy → delegate → outbound
POST — in a plain Node process with no vitest and no `vi.mock`, stubbing
only `fetch` at the network boundary. Same script against the unfixed
file, then the fix:
```
BEFORE (origin/main) AFTER (this PR)
"headers" in delegate: false "headers" in delegate: true
delegate.headers: undefined delegate.headers: { X-Tenant: tenant-b }
proxy.headers after setHeaders(B): proxy.headers after setHeaders(B):
{ X-Tenant: tenant-b } { X-Tenant: tenant-b }
0: POST /connect X-Tenant=tenant-a 0: POST /connect X-Tenant=tenant-a
1: POST /connect X-Tenant=tenant-a <-- 1: POST /connect X-Tenant=tenant-b credentials=include
FAIL (stale headers) PASS (live headers reach /connect)
```
The "before" column reproduces the report's tell exactly:
`proxy.headers` correct at tenant B while `/connect` still sends tenant
A, through the very API the report found ineffective.
**Gates** (run in a worktree with a freshly built `@copilotkit/shared`,
since a stale dist otherwise produces 20 unrelated
`core-inspector-metadata` failures and 4 `tsc` errors):
| Gate | Result |
| --- | --- |
| `@copilotkit/core` vitest | **654 passed / 654**, 59/59 files |
| `tsc --noEmit` | clean |
| `oxlint` | 0 errors (2 warnings, both pre-existing test helpers) |
| `oxfmt` | no reformatting needed |
**Not covered:** `fetch` is stubbed, so this does not exercise a live
Intelligence gateway or a browser tenant switch — it proves the outbound
header is correct, not the platform's response to it.
## Note for whoever merges
#6450 and #6468 also touch `intelligence-agent.ts` (thread-restore work)
but neither goes near the header path, so conflicts should be textual at
worst.
## Follow-up left out of scope
Two separate pre-existing defects surfaced while verifying this one.
Neither is touched here.
**1. `credentials` passed to a `ProxiedCopilotRuntimeAgent` constructor
are dropped at registration.** `applyCredentialsToAgent` overwrites
`agent.credentials` from core unconditionally, with no per-agent
baseline — unlike `applyHeadersToAgent`, which merges over the
`agentOwnHeaders` baseline captured for exactly this reason (#5635).
Probed in a real process: an agent constructed with `credentials:
"include"` in a core with none configured reports `undefined`
immediately after registration, and every join goes out without
credentials. Identical before and after this PR, so it is not a
regression from this change — but the headers/credentials asymmetry
looks unintended, given #5433 was specifically about preserving proxied
runtime credentials.
**2. `buildRuntimeUrl` reads `config.agentId`
(`intelligence-agent.ts:770`), (`intelligence-agent.ts:770`), so
`syncDelegate`'s `delegate.agentId = routedAgentId()` is cosmetic for
the REST URL. Same root-cause class as this bug, but latent rather than
live (routing is fixed per proxy instance).
Happy to file both separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Benjamin Taylor
|
70545f072a |
test(core): correct an overclaiming comment, tighten the credentials assertion
The per-thread-clone test's comment claimed it guards the copy-on-write setter. It does not: syncDelegate rewrites headers before every join, so it passes even with an in-place write (verified). Say what it actually pins — each proxy's joins carry its own tenant — and point at the clone-invariant test that does guard the setter. Also assert the pre-change join carried no credentials, so the credentials test shows a transition rather than a single end state. |
||
|
Murat Sari
|
9785e135ea |
fix(angular): prevent duplicate OpenGenerativeUI sandboxes (#6477)
Prevents stale async loaders from creating duplicate OpenGenerativeUI sandbox iframes by tying final sandbox creation to `afterRenderEffect` cleanup. Adds regression tests for identical content and A → B → A races. ### Testing - Angular tests - Type checking - Angular package build |
||
|
Murat Sari
|
dde84a5804 | fix(angular): prevent duplicate OpenGenerativeUI sandboxes | ||
|
Jerel John Velarde
|
bd09c3d790 |
chore(examples): drop grok showcase lockfile
A 13.9k-line new file trips the fork-PR supply-chain heuristic (security_fork-pr-alert flags any added file over 5000 lines), and the job cannot post its explanation because fork tokens are read-only. Several showcases ship no lockfile; this one is not a pnpm workspace member, so nothing depends on it. |
||
|
Jerel John Velarde
|
fe0e7cf28f |
feat(examples): add grok-generative-ui showcase
grok-4.6 runs xAI's X Search server-side, then composes the answer out of real React components through five CopilotKit frontend tools. Every post rendered is a real post the model found. Registers next.config.ts in the build-config allowlist and adds the row to the examples index. |
||
|
Murat Sari
|
6de1b96da2 |
fix(core): prevent duplicate interrupt tool results (#6201) (#6470)
## Summary Only create client-side tool results for `tool_call` interrupts. Backend-owned interrupts now resume without synthetic tool messages, preventing duplicate results. Includes Core, Angular, and React test coverage. ## References - Fixes #6201 - Related: #6270 - [AG-UI interrupts](https://docs.ag-ui.com/concepts/interrupts) ## Validation - [x] Core, Angular, and React tests - [x] Full workspace test suite - [x] Type checks |
||
|
KNChiu
|
d7dd1bcfbe | docs(examples): fix stale clone paths in v1 example READMEs | ||
|
Murat Sari
|
01c7283210 | fix(core): prevent duplicate interrupt tool results (#6201) | ||
|
Benjamin Taylor
|
7d1cdc15df |
test(core): pin the run path against stale Intelligence headers
The report names both /connect and /run. The run path reaches the delegate through #runViaDelegate, which shares resolveDelegate with the connect path, so the accessor fixes both — but that was inferred from the shared call site rather than pinned. Verified failing against the pre-fix file (sent tenant-a where tenant-b was expected). |
||
|
Benjamin Taylor
|
a3562c20a6 |
fix(core): refresh Intelligence delegate headers before every join
`ProxiedCopilotRuntimeAgent` builds its `IntelligenceAgent` delegate once and caches it for the proxy's lifetime, copying `headers` into the delegate's constructor config. Nothing ever refreshed that copy, so a header that changed later never reached `/connect` or `/run` — for the life of the agent. `setHeaders`/`applyHeadersToAgent` could not fix it: they write an agent's `.headers`, and `IntelligenceAgent` exposed only `private config`. `syncDelegate` looked like the refresh path but its `hasHeaders` probe is `"headers" in agent`, which was false for the delegate. Multi-tenant apps that carry the active tenant in a header saw the join attempted under the previous tenant's identity with the new tenant's thread id, answered THREAD_NOT_FOUND. A rotated `Authorization` bearer has the same exposure. Only a full reload cleared it. Expose `headers` as a public accessor pair backed by `config`. The accessor is the entire fix: it makes `hasHeaders` true, so `syncDelegate` — which already runs on every `resolveDelegate()` — starts actually refreshing the delegate before each join. Note that changing `requestJoinCredentials$` to read live headers, as the report suggested, does nothing on its own: there was no live source on the class to read, and without the accessor `this.headers` is `undefined`, which drops every header. It reads through the accessor here for a single source of truth, not because that read carries the fix. The setter replaces the config object rather than mutating it, because `clone()` shares the config reference. The join path alone would mask an in-place write (syncDelegate rewrites headers just before every join), but the credential re-acquisition inside a running pipeline does not re-sync, so a clone's tenant could ride out on the original's socket-error refresh. `credentials` had the identical defect via `config.credentials` (`hasCredentials` was false too) and gets the same treatment. Verified beyond the unit tests by driving the real chain (`CopilotKitCore.setHeaders` -> registry -> proxy -> delegate -> outbound POST) in a plain Node process with only `fetch` stubbed: before, `"headers" in delegate` was false and the join after a tenant switch still sent tenant A; after, it sends tenant B. Reported by Sameday against 1.67.1 with a deterministic staging repro. |
||
|
Maxim
|
784f2e7529 |
docs(reskinnable-demo): retire the last "all six skins" claims after bookstore
The merge of main brought a seventh skin. These are the surviving count claims
outside the conflicted files, all of which the seventh skin falsified:
- `airline` was described as "the one PASSENGER-FACING skin"; `bookstore` is
also customer-facing, so it now names the pair.
- demo-beats.md still told a skin author "every registered skin is
demo-complete, so there is no partial precedent to copy". Bookstore IS a
partial precedent, deliberately, so the sentence now says so.
- Eight in-skin comments said "all six skins" while describing something that
is true of the WHOLE roster (the shared PDF primitive's coverage, the dark
treatment, and — load-bearing — the project-scope warning in three
seed-memories.ts files, where undercounting understates the blast radius of
a project-scoped sweep). All now say "every skin", which cannot rot.
Reskin-skill staleness check (CLAUDE.md standing rule): yes, demo-beats.md is
part of the skill and is corrected here.
Verified from examples/showcases/reskinnable-demo: `pnpm lint` clean,
`pnpm exec tsc --noEmit` 0 errors, and the roster/config drift guards plus the
touched skin tests pass (110 tests).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Maxim
|
6473cdcf9d |
feat(reskinnable-demo): merge main, and reconcile the docs with a seventh skin
Brings in `bookstore` and 31 other commits from main.
WHY THIS MERGE CONFLICTED IN SEVEN FILES. Both sides hand-maintained the
same roster. This branch had just rewritten the docs around four
conclusions that were true when written:
- `useData` has zero implementors
- no in-memory skin remains
- every registered skin is demo-complete
- there are six skins
`bookstore` falsifies all four: it sets `useData: useBookstoreData`, so the
optional hook has a live implementor and an in-memory skin exists again; it
ships intelligence/{seed,forget}-memories.ts but no teach loop, so it is not
demo-complete; and it is the seventh.
Neither side was wrong. The resolution is the union, and where a list or a
count was load-bearing it is now the command that derives it -- which is the
convention this branch adopted precisely because two hand-maintained copies
of one roster is what produced these conflicts.
The de-narration this branch applied is preserved: main's older phrasings
carried retrospective prose that was deliberately removed, and it has not
been reintroduced.
Registration verified rather than assumed -- bookstore is present in
LINTED_SKIN_IDS, skinIds, skinIdentities, SkinRegistry and agentRegistry.
That last one has no drift guard at all, so a missing key there fails only
when someone sends a chat message.
Gates on the merged tree: lint clean, `tsc --noEmit` 0 errors, 210 test files
/ 2414 tests passing (up from 197/2227 -- bookstore's own, nothing dropped),
build exit 0.
COMMITTED WITH --no-verify, DELIBERATELY, WITH THE USER'S APPROVAL.
The pre-commit hook was bypassed. That is normally forbidden here, so the
reason is recorded rather than left to be guessed:
- This branch's ENTIRE diff against main is inside
examples/showcases/reskinnable-demo. `git diff --name-only origin/main...HEAD`
lists nothing outside it.
- The hook fails on `@copilotkit/vue` -> CopilotThreadsDrawer.ssr.test.ts,
"does not eagerly evaluate the Lit element module when the package entry is
imported". That test fails STANDALONE on this machine
(`npx nx test @copilotkit/vue` -> 1 failed | 1073 passed, exit 1), with no
merge in progress and nothing of ours involved. It asserts a lazy-import
property but enforces it with a 5000ms wall-clock timeout, so it fails
whenever module resolution is slow rather than when Lit is actually
eagerly evaluated.
- This is simply the first commit on the branch to touch packages/*, so it is
the first to make `nx affected` run that suite. Ninety earlier commits
touched only the demo app and never triggered it.
What WAS verified on the merged tree, by hand, before committing:
pnpm lint clean
pnpm exec tsc --noEmit 0 errors
pnpm test:unit 210 files / 2414 tests passing
pnpm build exit 0
npx nx test @copilotkit/runtime 138 files passing
That last one only passes because of a second pre-existing breakage fixed
along the way: packages/runtime's better-sqlite3 binary was compiled against
NODE_MODULE_VERSION 137 (Node 24) while .nvmrc pins Node 22 (127), so every
SqliteAgentRunner test threw on load. `pnpm rebuild -r better-sqlite3` fixed
it. That fix is environmental and is not part of this commit.
Two follow-ups worth someone's time, neither blocking:
1. The vue SSR test should assert the property (module not evaluated) rather
than time the import.
2. Nothing in the repo pins the Node version for native rebuilds, so a
contributor who once ran a task under Node 24 silently poisons
better-sqlite3 for every later Node 22 run.
|
||
|
Jordan Ritter
|
7fd4c5ee78 |
fix(showcase/harness): reap chromium zombies, alarm on PID saturation, stop saturated workers claiming (#6333)
> **This PR carries three changes to the same prod incident.** Change 1
fixes
> the PID leak itself; changes 2 and 3 fix the two reasons the leak was
able to
> starve the fleet unannounced. Each has its own RED → GREEN below.
>
> | # | Commit | What it fixes |
> |---|---|---|
> | 1 | `e66fb1af13` | **The leak.** No PID-1 reaper, so crashed
chromium grandchildren become permanent zombies holding cgroup PID
slots. |
> | 2 | `7f28e9bd39` | **Nothing alarmed.** The worker heartbeats its
cgroup PID gauges and no non-test code read them. |
> | 3 | `63ef4081c2` | **Saturated workers kept claiming.** The claim
gate only checked free browser-context slots, so a worker at 1000/1000
PIDs won jobs it could not run and burned each to a 600000ms abort. |
---
# CHANGE 1 — the leak (tini as PID 1)
## The bug
`harness-workers` has **no PID-1 zombie reaper**, so it leaks cgroup PID
slots until it cannot launch a browser.
libuv's `SIGCHLD` handler only `waitpid()`s the pids **Node itself
spawned**. Playwright spawns the chromium *browser* process (reaped
fine), but that browser's **zygote/renderer/GPU children are
grandchildren**. When a browser process dies — prod's `Target crashed`
path, or a pool self-heal relaunch — those grandchildren are re-parented
onto PID 1 and exit. Node-as-PID-1 never waits on them, so each becomes
a permanent `<defunct>` still holding a cgroup PID slot.
Prod, on a single container up 6d22h: `zombieCount` 70 → **757**,
`cgroupPidsCurrent` 286 → **1000/1000**. Once saturated, probes could
not launch browsers (`Target crashed`, `browserContext.newPage: … has
been closed`, `feature exceeded 300000ms wall-clock`) and **295 of 414
red prod d6 cells went `abort`, fleet-wide across ~20 slugs.**
This Dockerfile already documented the outage mode in a 40-line `FIX #3`
block — but **both of its stated mitigations are demand-side**
(`BROWSER_POOL_MAX_CONTEXTS` 40→24, resource-gauge warning logs). Those
cap the *peak* of `pids.current`. They cannot stop a *monotonic* leak,
and prod died of a monotonic leak. `resource-gauges.ts` only ever
**measured** zombies (`if (s.state === "Z") zombieCount += 1`); nothing
reaped them.
## The change
One file, +59/-3. `tini` as PID 1.
```dockerfile
RUN node ./node_modules/playwright/cli.js install --with-deps chromium \
&& apt-get install -y --no-install-recommends tini \
&& /usr/bin/tini --version \
&& rm -rf /var/lib/apt/lists/*
…
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["/bin/bash", "-c", "ulimit -u $(ulimit -Hu) 2>/dev/null || true; exec node dist/orchestrator.js"]
```
Why this form:
- **`tini` baked into the image, not Docker `--init`** — `--init` is a
*runtime* flag and Railway does not expose it, the same reason
`pids.max` isn't settable from here (already documented in this file).
- **Not an in-process reaper** — Node has no `waitpid` binding, so an
application `SIGCHLD` handler would need a native addon. This is the
platform's job.
- **Rides the existing apt layer** — playwright's `--with-deps` just ran
`apt-get update`, so this costs one ~267KB package download instead of a
second index refresh. The `tini --version` call is a build-time
assertion on the path baked into `ENTRYPOINT`: if a future Debian moves
the binary, the **build** fails rather than the container failing to
start in prod.
- **No `-g`** — tini's process-group broadcast would signal the live
chromium pool alongside node and pre-empt `orchestrator.ts`'s
`process.once("SIGTERM", drainAndExit)`. Without it, delivery is
byte-identical to today: exactly one process gets the signal.
- **Exec form, `exec` kept in CMD** — tini really is PID 1 and really
receives Railway's SIGTERM; node is tini's direct child with no bash
lingering in the tree.
No existing convention to match: `grep` for `tini`/`dumb-init`/`--init`
across every Dockerfile in the repo returns zero hits. Nothing in the
repo asserts on this image's `CMD`/`ENTRYPOINT`, and nothing outside the
CI build step `docker run`s it, so adding an `ENTRYPOINT` breaks no
consumer.
---
# RED → GREEN
Real container, real chromium, real zombies. **Identical argv for
both**; only the image differs, and the image supplies PID 1:
```
docker run --rm --pids-limit 1000 -e REPRO_CYCLES=230 \
-v <repro>:/repro:ro <image> /bin/bash -c "exec node /repro/zombie-repro.mjs"
```
`--pids-limit 1000` mirrors Railway's platform-fixed ceiling. Each cycle
launches chromium with the exact args `browser-pool.ts` uses
(`headless`, `--no-sandbox`, `--disable-dev-shm-usage`), opens a
context+page, then **SIGKILLs the browser process** — prod's `Target
crashed` path, orphaning its children onto PID 1.
**Measurement is container-wide on purpose.** The harness's own
`sampleResourceGauges()` walks the tree from `process.pid`; under a real
init the orphans re-parent to the *init*, not to node, so a node-rooted
walk would report zero zombies **whether or not they were reaped** — a
vacuous GREEN. Counting every state-`Z` process in `/proc` plus the
cgroup counter is blind to which process is PID 1, so it is honest in
both directions.
## RED (unmodified `main`, `pid1=node`)
20 cycles — exactly **+5 permanent zombies per crashed browser**:
```
baseline zombieCount=0 procCount=1 threadCount=7 cgroupPidsCurrent=7 cgroupPidsMax=1000 selfPid=1 pid1=node
PID PPID S THR COMMAND
1 0 R 7 node
cycle=1 pre-kill zombieCount=0 procCount=7 threadCount=76 cgroupPidsCurrent=76 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=1 zombieCount=5 procCount=6 threadCount=16 cgroupPidsCurrent=16 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=2 zombieCount=10 procCount=11 threadCount=21 cgroupPidsCurrent=21 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=3 zombieCount=15 procCount=16 threadCount=26 cgroupPidsCurrent=26 cgroupPidsMax=1000 selfPid=1 pid1=node
…
cycle=20 zombieCount=100 procCount=101 threadCount=111 cgroupPidsCurrent=111 cgroupPidsMax=1000 selfPid=1 pid1=node
final zombieCount=100 procCount=101 threadCount=111 cgroupPidsCurrent=111 cgroupPidsMax=1000 selfPid=1 pid1=node
SUMMARY cycles=20 zombiesBefore=0 zombiesAfter=100 pidsBefore=7 pidsAfter=111 launchFailure=none
```
The leaked processes, verbatim from the final `/proc` table — all
`ppid=1`, all `Z`:
```
21 1 Z 1 headless_shell <defunct>
22 1 Z 1 headless_shell <defunct>
36 1 Z 1 headless_shell <defunct>
61 1 Z 1 headless_shell <defunct>
77 1 Z 1 headless_shell <defunct>
87 1 Z 1 headless_shell <defunct>
88 1 Z 1 headless_shell <defunct>
104 1 Z 1 headless_shell <defunct>
```
`cgroupPidsCurrent` after each cycle never returns to 7 — it is `7 +
5×cycles`. A monotonic ratchet, the same curve prod walked over 7 days.
### RED, driven to saturation (started before the Dockerfile was
touched)
```
baseline zombieCount=0 cgroupPidsCurrent=7 /1000
cycle=25 zombieCount=125 cgroupPidsCurrent=136 /1000
cycle=50 zombieCount=250 cgroupPidsCurrent=261 /1000
cycle=75 zombieCount=375 cgroupPidsCurrent=386 /1000
cycle=100 zombieCount=500 cgroupPidsCurrent=511 /1000
cycle=125 zombieCount=625 cgroupPidsCurrent=636 /1000
cycle=150 zombieCount=750 cgroupPidsCurrent=761 /1000 ← prod's ~757 reproduced
cycle=175 zombieCount=875 cgroupPidsCurrent=886 /1000
cycle=184 pre-kill zombieCount=915 cgroupPidsCurrent=993 /1000
cycle=185 pre-kill zombieCount=920 cgroupPidsCurrent=996 /1000
cycle=185 zombieCount=925 cgroupPidsCurrent=936 /1000
← no further output. Cycle 186 never completed.
```
The run **wedged** at cycle 185 with no output for >12 minutes:
`chromium.launch()` could not get PIDs and never returned — it did not
even surface Playwright's 30s launch timeout, which is exactly why prod
shows 300s feature timeouts and 1200s run aborts instead of a clean
launch error. `docker ps`: `Up 17 minutes (unhealthy)`.
```
$ docker exec red-sat cat /sys/fs/cgroup/pids.current /sys/fs/cgroup/pids.max
1000
1000
```
And the errno-11 signature this Dockerfile's own comment names — the
container could no longer `fork()` at all:
```
$ docker exec red-sat /bin/bash -c 'for i in 1 2 3; do cat /proc/uptime; done'
/bin/bash: fork: retry: Resource temporarily unavailable
/bin/bash: fork: retry: Resource temporarily unavailable
/bin/bash: fork: retry: Resource temporarily unavailable
/bin/bash: fork: retry: Resource temporarily unavailable
/bin/bash: fork: Resource temporarily unavailable
```
That is the prod outage, end to end, in a local container.
## GREEN (this PR's image, `pid1=tini`) — same repro, same argv, 230
cycles
```
baseline zombieCount=0 procCount=2 threadCount=8 cgroupPidsCurrent=8 cgroupPidsMax=1000 selfPid=7 pid1=tini
PID PPID S THR COMMAND
1 0 S 1 tini
7 1 R 7 node
cycle=1 pre-kill zombieCount=0 procCount=8 threadCount=76 cgroupPidsCurrent=76 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=1 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=50 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=100 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=150 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=200 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=230 pre-kill zombieCount=0 procCount=8 threadCount=77 cgroupPidsCurrent=77 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=230 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
final zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
SUMMARY cycles=230 zombiesBefore=0 zombiesAfter=0 pidsBefore=8 pidsAfter=12 launchFailure=none
```
`procCount` returning to **2** (tini + node) after every single cycle is
the reap actually happening: the browser's 6 processes appear pre-kill
(`procCount=8`) and are fully gone post-kill.
| | RED (`pid1=node`) | GREEN (`pid1=tini`) |
| --- | --- | --- |
| zombies after 20 cycles | 100 | **0** |
| zombies at cycle 185 | 925 | **0** |
| `pids.current` floor at cycle 185 | 936, peaking 996/1000 | **12** |
| reached cycle 230? | **no — wedged at 185** | yes |
| `chromium.launch()` | hangs; `fork: Resource temporarily unavailable`
| fine, `launchFailure=none` |
| leak per crashed browser | **+5** | **0** |
---
## Signals and non-zero exit still work
The prober's container must still self-restart on crash, so this was
proven, not assumed. `signal-child.mjs` installs the same
`process.once("SIGTERM", …)` shape `orchestrator.ts` uses; run on
**both** images:
```
--- GREEN (ENTRYPOINT tini, pid1=tini) ---
[showcase-harness:green MODE=term-graceful] exitCode=0
CHILD mode=term-graceful pid=7 ppid=1
CHILD ready — idling until signalled
CHILD received SIGTERM at pid=7 — draining ← tini FORWARDED it to node
[showcase-harness:green MODE=exit-42] exitCode=42 ← non-zero preserved
[showcase-harness:green MODE=abort] exitCode=134 ← 128+6 (SIGABRT), non-zero
--- RED (no ENTRYPOINT, pid1=node) — baseline ---
[showcase-harness:red MODE=term-graceful] exitCode=0
CHILD received SIGTERM at pid=1 — draining
[showcase-harness:red MODE=exit-42] exitCode=42
[showcase-harness:red MODE=abort] exitCode=133
```
- **SIGTERM reaches node, not just tini** — the handler fires at
`pid=7`, so `docker stop`'s SIGTERM was forwarded into the process that
owns the graceful drain. `orchestrator.ts`'s
`drainControlPlaneAndExit("SIGTERM")` path is intact.
- **Non-zero exits survive** — `exit(42)` → container `42`; a crash
still exits non-zero, so Railway still restarts. The HEALTHCHECK
restart-on-sustained-503 behaviour this Dockerfile calls "THE INTENDED
OUTCOME" is untouched.
- **tini is strictly *more* correct here**: note RED's abort → **133**
vs GREEN's → **134**. A process running as PID 1 in a namespace has
default signal actions *ignored* by the kernel, so node-as-PID-1 was
distorting its own fatal-signal exit code (128+SIGTRAP instead of
128+SIGABRT). Both non-zero, so no regression — but the fixed image
reports crashes accurately.
## Cross-arch check
Local images are arm64; prod builds amd64. `tini` availability was
verified on amd64 explicitly rather than assumed — same path the
build-time assertion checks:
```
Package: tini
Architecture: amd64
Version: 0.19.0-1+b3
-rwxr-xr-x 1 root root 27792 Jun 6 2025 /usr/bin/tini
tini version 0.19.0
```
## Pre-push checks
| Check | Result |
| --- | --- |
| `docker buildx build -f showcase/harness/Dockerfile` (== CI's build
check) | **PASS**, both images |
| `nx run @copilotkit/showcase-harness:typecheck` | **PASS** |
| `nx run @copilotkit/showcase-harness:test:ci` | 3677 passed, 18
skipped, 1 failed — **environmental**, see below |
| commitlint | **PASS** |
| lefthook `pre-commit` + `commit-msg` | **PASS** |
| `git status` | clean |
The single failure is `d0-gone-predicate.test.ts > resolves the real
generated registry.json…`, failing on `ENOENT …
showcase/shell/src/data/registry.json`. That file is **gitignored and
generated at build time** — this Dockerfile's own comment says so, which
is why the image generates it via `generate-registry.ts`. Proven
environmental rather than assumed: generated the file, re-ran that exact
spec, **11/11 passed**. A Dockerfile-only diff cannot affect a vitest
run.
No formatter/linter covers this file: lefthook's `lint-fix` glob is
`*.{js,jsx,ts,tsx,mjs,cjs,md,css,yml,yaml,html,vue,py}`, and there is no
hadolint anywhere in CI.
## Not in this PR
- **This does not fix prod by itself.** Prod is digest-pinned and still
running the leaking container; it needs the live mitigation restart
(owned by a separate worker — **prod was not touched here**) and then a
promote to pick this image up.
- **The ≥3-cell value test is not done and cannot be done from this PR**
— confirming red cells flip requires the prod restart plus a fleet
sweep. Note the ~20 `conversation-error` and ~16 `goto-error` cells are
almost certainly unrelated real bugs this will **not** fix; only the
`abort`/`feature-timeout`/`driver-error` population is in scope.
- **Alerting / scheduled recycle is deliberately out of scope** (one
concern per change). Still worth doing: the documented
`pool-unrecoverable` alarm demonstrably never fired at 1000/1000, and
`cgroupPidsCurrent/cgroupPidsMax > 0.75` would have given ~36h of
warning before the flip.
- Leak rate on the **graceful** `browser.close()` path was not measured
— irrelevant to the fix (tini reaps either way), but it means prod's
exact 757 can't be attributed to crashes alone.
---
# RE-VERIFICATION 2026-08-12 — rebased onto `origin/main`, red-green
re-run from scratch
The proof above was produced 2026-08-03 against a base that predates
`ced993447f` ("copy patches/ into harness build context"), which touches
THIS
Dockerfile. The branch has been rebased onto current `origin/main` and
the
whole red-green was re-run on freshly built images so the numbers
describe the
actual merge candidate.
**Both images built locally from this repo, same machine, same context,
~3 min apart:**
```
showcase-harness:red-main sha256:b86686e9a06d51f884d5e59e9d15d667fd298284efe7a15622ff3bf2575a3ffd created=2026-08-12T20:15:50Z (origin/main Dockerfile, /usr/bin/tini ABSENT, Entrypoint=["docker-entrypoint.sh"])
showcase-harness:green-tini sha256:d666b823f16bda07bf62d35f1d72ef099d13406a61e2f0cd51c99bf51eb14df4 created=2026-08-12T20:18:54Z (this branch, /usr/bin/tini present, Entrypoint=["/usr/bin/tini","--"])
```
Identical driver, identical argv, 40 cycles each, runs 18 seconds apart:
```
docker run --rm --pids-limit 1000 -e REPRO_CYCLES=40 -v <repro>:/repro:ro <image> \
/bin/bash -c "exec node /repro/zombie-repro.mjs"
```
## RED — `showcase-harness:red-main`, `pid1=node` (2026-08-12T20:19:28Z)
```
baseline zombieCount=0 procCount=1 threadCount=7 cgroupPidsCurrent=7 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=1 pre-kill zombieCount=0 procCount=7 threadCount=76 cgroupPidsCurrent=76 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=1 zombieCount=5 procCount=6 threadCount=16 cgroupPidsCurrent=16 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=2 pre-kill zombieCount=5 procCount=12 threadCount=83 cgroupPidsCurrent=83 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=2 zombieCount=10 procCount=11 threadCount=21 cgroupPidsCurrent=21 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=10 pre-kill zombieCount=45 procCount=52 threadCount=120 cgroupPidsCurrent=120 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=10 zombieCount=50 procCount=51 threadCount=61 cgroupPidsCurrent=61 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=20 pre-kill zombieCount=95 procCount=102 threadCount=174 cgroupPidsCurrent=174 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=20 zombieCount=100 procCount=101 threadCount=111 cgroupPidsCurrent=111 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=30 pre-kill zombieCount=145 procCount=152 threadCount=219 cgroupPidsCurrent=219 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=30 zombieCount=150 procCount=151 threadCount=161 cgroupPidsCurrent=161 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=40 pre-kill zombieCount=195 procCount=202 threadCount=271 cgroupPidsCurrent=271 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=40 zombieCount=200 procCount=201 threadCount=211 cgroupPidsCurrent=211 cgroupPidsMax=1000 selfPid=1 pid1=node
final zombieCount=200 procCount=201 threadCount=211 cgroupPidsCurrent=211 cgroupPidsMax=1000 selfPid=1 pid1=node
SUMMARY cycles=40 zombiesBefore=0 zombiesAfter=200 pidsBefore=7 pidsAfter=211 launchFailure=none
```
All 200 leaked processes are `ppid=1`, state `Z` (tail of the final
`/proc` table):
```
2610 1 Z 1 headless_shell <defunct>
2612 1 Z 1 headless_shell <defunct>
2646 1 Z 1 headless_shell <defunct>
2661 1 Z 1 headless_shell <defunct>
2662 1 Z 1 headless_shell <defunct>
2678 1 Z 1 headless_shell <defunct>
2680 1 Z 1 headless_shell <defunct>
2717 1 Z 1 headless_shell <defunct>
```
## GREEN — `showcase-harness:green-tini`, `pid1=tini`
(2026-08-12T20:19:46Z)
```
baseline zombieCount=0 procCount=2 threadCount=8 cgroupPidsCurrent=8 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=1 pre-kill zombieCount=0 procCount=8 threadCount=76 cgroupPidsCurrent=76 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=1 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=2 pre-kill zombieCount=0 procCount=8 threadCount=78 cgroupPidsCurrent=78 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=2 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=10 pre-kill zombieCount=0 procCount=8 threadCount=79 cgroupPidsCurrent=79 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=10 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=20 pre-kill zombieCount=0 procCount=8 threadCount=76 cgroupPidsCurrent=76 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=20 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=30 pre-kill zombieCount=0 procCount=8 threadCount=77 cgroupPidsCurrent=77 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=30 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=40 pre-kill zombieCount=0 procCount=8 threadCount=79 cgroupPidsCurrent=79 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=40 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
final zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
SUMMARY cycles=40 zombiesBefore=0 zombiesAfter=0 pidsBefore=8 pidsAfter=12 launchFailure=none
```
`grep -c defunct` over the GREEN run: **0**. `procCount` returns to
**2**
(tini + node) after every cycle — the browser's 6 processes appear
pre-kill
(`procCount=8`, `threadCount` 76–79) and are fully gone post-kill.
| | RED (`pid1=node`) | GREEN (`pid1=tini`) |
| --- | --- | --- |
| image id | `b86686e9a06d` | `d666b823f16b` |
| cycles | 40 | 40 |
| zombies after | **200** | **0** |
| zombies per crashed browser | **+5** | **0** |
| `pids.current` before → after | 7 → **211** | 8 → **12** |
| `<defunct>` rows in final table | **200** | **0** |
| `launchFailure` | none | none |
## Signals / non-zero exit, both images, this run
```
[showcase-harness:RED MODE=term-graceful] exitCode=0
CHILD mode=term-graceful pid=1 ppid=0
CHILD received SIGTERM at pid=1 — draining
[showcase-harness:RED MODE=exit-42] exitCode=42
[showcase-harness:RED MODE=abort] exitCode=133
RED /proc/1/comm = bash
[showcase-harness:GREEN MODE=term-graceful] exitCode=0
CHILD mode=term-graceful pid=7 ppid=1
CHILD received SIGTERM at pid=7 — draining <- tini FORWARDED it to node
[showcase-harness:GREEN MODE=exit-42] exitCode=42
[showcase-harness:GREEN MODE=abort] exitCode=134
GREEN /proc/1/comm = tini, child self=7
```
SIGTERM reaches node (handler fires at `pid=7`), non-zero exits survive
(42 → 42; abort → 134 = 128+6), so Railway's restart-on-crash is intact.
## The prod leak, measured live the same day
`resource_snapshots` in prod PocketBase, immediately before the
mitigation
restart on the unfixed digest `sha256:80a0c5c2…`:
```
2026-08-12 20:10:00.674Z worker-7abf6 heartbeat pids=1000/1000 zombies=758 procs=778 threads=1000
2026-08-12 20:09:59.783Z worker-5a213 heartbeat pids=1000/1000 zombies=761 procs=781 threads=1000
2026-08-12 20:09:54.868Z worker-de14c heartbeat pids=837/1000 zombies=744 procs=753 threads=837
```
and immediately after it (`init` rows): `pids=164–169/1000 zombies=0
procs=16`.
A new zombie was already recorded at `2026-08-12 20:12:27.182Z
worker-de14c … zombies=1`.
## Cross-arch
Local images are arm64 (`/usr/bin/tini` 68592 bytes); prod builds amd64.
Measured on amd64 directly: `Package: tini / Architecture: amd64 /
Version: 0.19.0-1+b3`, `/usr/bin/tini` 27792 bytes, `tini version
0.19.0`;
package download 267,204 bytes, 749 kB on disk.
## Deploy note
`computePromoteClosure(["showcase-harness"])` = `pocketbase(0),
harness(1),
dashboard(1), harness-workers(1)` — the workers are in the promote
scope.
Prod `startCommand` is `null` and no repo consumer overrides the
entrypoint,
so Railway uses the image's ENTRYPOINT+CMD. **PR #5750 is CLOSED
unmerged**, so
promoting `harness-workers` resets
`multiRegionConfig.us-west2.numReplicas` to
1 and it must be re-asserted to 6 and read back after the promote.
---
# CHANGE 2 — nothing alarmed (worker PID-saturation alarm)
Commit `7f28e9bd39`.
## The gap
The measured prod condition on 2026-08-12 20:09–20:10Z was
`pids=1000/1000`,
`1000/1000`, `837/1000` across three workers, 320 `timeout after
600000ms`
abort rows, an `e2e-smoke` run 42 minutes into a 15-minute cadence with
3 jobs
still pending, prod d6 534/784 vs staging 744/788. **No alarm fired**,
and this
was the second occurrence (2026-08-03, 2026-08-10).
The documented `pool-unrecoverable` alarm did not fire *by
construction*: it is
raised only when `BrowserPool`'s self-heal circuit-breaker exhausts
`selfHealMaxHardRecoveries`. A worker that is PID-starved but still
heartbeating
never reaches that state — it launches browsers fine, it just cannot
fork the
process tree a probe needs.
The signal was already on the wire and read by nobody:
- `worker/registration.ts:192` writes `capacity_pids_current` /
`capacity_pids_max` onto the worker's `workers` row on its ~75s
heartbeat
(`DEFAULT_WORKER_HEARTBEAT_MS = 75_000`).
- `control-plane/fleet-health.ts` lists that entire roster every 15s
(`DEFAULT_FLEET_HEALTH_INTERVAL_MS`) — and its own comment on the row
type
said *"the capacity gauges are ignored here."*
- Nothing in non-test source compared `pidsCurrent` to `pidsMax`.
## Where the alarm goes, and why
**Control-plane, not worker.** `writers/status-writer.ts` documents
`fleet-cp`
as *"the only authoritative fleet writer; workers never write status
directly"*,
so a worker cannot raise a `system:` health key. `fleet-health` already
holds
the roster read this needs, on a 15s cadence, so the alarm costs zero
extra PB
traffic.
**Routed to `#oss-alerts`, not to the browser-pool webhook.**
`SLACK_WEBHOOK_BROWSER_POOL_UNRECOVERABLE` appears **nowhere in this
repo
outside its own definition** — `grep` finds it only in `orchestrator.ts`
— so it
is unset everywhere and that alarm has no receiver even when it does
fire.
`SLACK_WEBHOOK_OSS_ALERTS` is a real wired secret that the
family-silence
monitor and the D0-gone monitor already post through. Routing a new
alarm to a
known-unwired webhook would reproduce the exact gap this closes.
## Thresholds
- **Alarm at `pids.current / pids.max >= 0.75`.** At the observed leak
rate
(~1000 PIDs over ~7 days) that is roughly 36 hours of lead time — more
than a
working day. A busy worker's context budget is nowhere near 750 PIDs.
- **Latched per worker, clearing at 0.65.** The monitor ticks every 15s;
an
unlatched alarm would page ~8600 times across that 36-hour window. An
alarm
that fires constantly is worse than none, so `createFleetHealthMonitor`
now
**fails loud at construction** on a ratio pair with no hysteresis gap.
- **Evaluated before the online/stale branch**, because the failing
worker is
fully `online` — it heartbeats on time throughout.
- **Unmeasured gauges never alarm.** `registration.ts` maps the pool's
`-1`
sentinel to `null`; `pidUsageRatio` returns `null` for
null/absent/negative
current or non-positive max, and null is never read as zero.
---
## RED → GREEN (change 2)
Driver exercises the **real chain end-to-end**, faking only the OS
cgroup read
(`BrowserPoolOptions.cgroupPidsReader`, an existing injectable seam) and
the PB
transport. The column names are whatever the real registration writer
emits and
whatever the real fleet-health reader consumes, so a column drift fails
this
driver:
```
BrowserPool.budget() [real, probes/helpers/browser-pool.ts]
-> workerCapacityFromBudget [real, fleet/contracts.ts]
-> registerWorker() [real, fleet/worker/registration.ts]
-> workers row
-> createFleetHealthMonitor().checkOnce()
[real, fleet/control-plane/fleet-health.ts]
-> alarm sink
```
Identical command both sides; only the source differs:
```
./node_modules/.bin/tsx <scratch>/drive-alarm.mts
```
### RED — unmodified source
```
=== A1 SATURATED 900/1000 (ratio 0.90) ===
pool.budget() pids : 900/1000
workers row gauges : capacity_pids_current=900 capacity_pids_max=1000
fleet-health cycles run : 1
ALARM HOOK CALLS : 0
result.pidSaturated : <field does not exist>
error-level log events : <none>
=== A2 CEILING 1000/1000 (ratio 1.00) ===
pool.budget() pids : 1000/1000
workers row gauges : capacity_pids_current=1000 capacity_pids_max=1000
fleet-health cycles run : 1
ALARM HOOK CALLS : 0
result.pidSaturated : <field does not exist>
error-level log events : <none>
=== A6 LATCH 900/1000 over 5 cycles ===
fleet-health cycles run : 5
ALARM HOOK CALLS : 0
result.pidSaturated : <field does not exist>
error-level log events : <none>
```
The exact prod condition — a worker at the ceiling — produces **zero**
alarms.
### GREEN — with this change
```
=== A1 SATURATED 900/1000 (ratio 0.90) ===
pool.budget() pids : 900/1000
workers row gauges : capacity_pids_current=900 capacity_pids_max=1000
fleet-health cycles run : 1
ALARM HOOK CALLS : 1
result.pidSaturated : 1
error-level log events : ["fleet.health.worker-pid-saturated"]
ALARM PAYLOAD : {"workerId":"worker-drv","pidsCurrent":900,"pidsMax":1000,"ratio":0.9,"threshold":0.75,"lastHeartbeatAt":"2026-08-12T20:44:54.244Z","observedAt":"2026-08-12T20:44:54.244Z"}
=== A2 CEILING 1000/1000 (ratio 1.00) ===
ALARM HOOK CALLS : 1
result.pidSaturated : 1
error-level log events : ["fleet.health.worker-pid-saturated"]
ALARM PAYLOAD : {"workerId":"worker-drv","pidsCurrent":1000,"pidsMax":1000,"ratio":1,"threshold":0.75,"lastHeartbeatAt":"2026-08-12T20:44:54.253Z","observedAt":"2026-08-12T20:44:54.253Z"}
```
### The alarm must also stay QUIET — same run, same command
```
=== A3 QUIET 500/1000 (ratio 0.50) ===
ALARM HOOK CALLS : 0
result.pidSaturated : 0
error-level log events : <none>
=== A4 QUIET 740/1000 (ratio 0.74, just under) ===
ALARM HOOK CALLS : 0
result.pidSaturated : 0
error-level log events : <none>
=== A5 QUIET -1/-1 (cgroup unreadable) ===
pool.budget() pids : -1/-1
workers row gauges : capacity_pids_current=null capacity_pids_max=null
ALARM HOOK CALLS : 0
result.pidSaturated : 0
error-level log events : <none>
=== A6 LATCH 900/1000 over 5 cycles ===
fleet-health cycles run : 5
ALARM HOOK CALLS : 1 <- once, not five times
```
`A4` pins the boundary at 0.74. `A5` pins the off-Linux case: the `-1`
sentinel
lands as `null` on the row and stays silent rather than reading as `0`.
`A6`
pins the latch — 5 saturated cycles, 1 alarm.
---
# CHANGE 3 — saturated workers kept claiming (PID headroom claim gate)
Commit `63ef4081c2`.
## The defect
`fleet/worker/worker-loop.ts` had exactly one conditional in the claim
loop:
```ts
if (budget.available <= 0) { … idle … }
```
and `available` is free Playwright **browser-context** slots and nothing
else —
`browser-pool.ts`: `available: Math.max(0, this.maxContexts -
this.liveContextCount)`.
Context slots have no relationship to cgroup PIDs. A worker whose
container has
leaked to `pids=1000/1000` therefore still advertises **full** capacity,
keeps
winning claims, cannot fork the browser tree, and burns each job's
entire
600000ms lease before aborting. That is the mechanism behind the 320
abort rows.
## Blast radius — what happens when ALL workers are saturated
The gate deliberately reuses the **same decline-and-idle path** as the
existing
no-budget branch, which is what makes total saturation safe:
- the job is **never claimed**, so it is never dropped and never
requeued — it
stays `pending` for a healthy worker, or for this one after a redeploy;
- the loop then sleeps a full `pollIntervalMs`, so a fully-saturated
fleet
**idles at the poll cadence** instead of spinning;
- the queue **stalls visibly**: `probe_jobs` rows pile up pending, a
rising-edge
`fleet.worker.pid-headroom-exhausted` warn says why (warn → stderr →
Sentry;
steady state drops to debug so a multi-hour stall does not flood), and
change 2's `system:worker-pid-saturation` alarm fired at 0.75 **before**
this
gate engaged at 0.90.
A stalled, alarmed queue is recoverable by redeploy. Claim-fail-repeat
silently
burned the whole cadence.
## Why the gate is 0.90 and the alarm is 0.75
They do different jobs and must not be equal. 0.75 pages an operator
while the
worker is still perfectly able to run jobs; gating dispatch that early
would
convert a warning into ~36 hours of withheld fleet capacity. 0.90 is
where we
stop trusting the worker to fork at all — 100 free PIDs at the prod
ceiling.
**That reserve is chosen, not derived: the per-job PID cost of a
chromium
context tree is not measured**, hence the `WORKER_PID_CLAIM_GATE_RATIO`
override. The invariant that matters is ordering — the alarm always
fires before
capacity is withdrawn.
The gate is **inert** when the cgroup gauges are unreadable
(`pidUsageRatio` →
`null`: off-Linux, every macOS dev box), so local workers claim exactly
as
before.
---
## RED → GREEN (change 3)
Driver runs the **real `startWorkerLoop`** against the **real
`BrowserPool`**
budget path, faking only the cgroup read and the queue transport. The
queue fake
holds a real `pending` list, so *"was the job dropped / does it stay
pending"* is
directly observed, not inferred. `budget()` calls are counted as a
loop-iteration proxy for the spin check.
Identical command both sides:
```
./node_modules/.bin/tsx <scratch>/drive-gate.mts
```
### RED — unmodified source
```
=== B1 SATURATED 1 worker @ 1000/1000, 3 jobs pending ===
workers=1 cgroup pids=1000/1000 contexts available=24
window : 600ms @ poll 50ms
queue.claimNext CALLS : 15
jobs CLAIMED : 3 ["job-1","job-2","job-3"]
jobs STILL PENDING : 0 []
jobs REPORTED : 3
loop iterations : 15
=== B2 ALL-SATURATED 3 workers @ 1000/1000, 3 jobs pending ===
workers=3 cgroup pids=1000/1000 contexts available=24
queue.claimNext CALLS : 39
jobs CLAIMED : 3 ["job-1","job-2","job-3"]
jobs STILL PENDING : 0 []
jobs REPORTED : 3
loop iterations : 39
```
A worker at the PID ceiling claims **all three jobs**. In prod each of
those
becomes a 600000ms abort.
### GREEN — with this change
```
=== B1 SATURATED 1 worker @ 1000/1000, 3 jobs pending ===
workers=1 cgroup pids=1000/1000 contexts available=24
window : 600ms @ poll 50ms
queue.claimNext CALLS : 0
jobs CLAIMED : 0 []
jobs STILL PENDING : 3 ["job-1","job-2","job-3"]
jobs REPORTED : 0
loop iterations : 12 (spin check: bounded by window/poll)
```
`claimNext` is never called; all three jobs are **still pending** — not
dropped,
not requeued.
### ALL workers saturated — the blast-radius case
```
=== B2 ALL-SATURATED 3 workers @ 1000/1000, 3 jobs pending ===
workers=3 cgroup pids=1000/1000 contexts available=24
window : 600ms @ poll 50ms
queue.claimNext CALLS : 0
jobs CLAIMED : 0 []
jobs STILL PENDING : 3 ["job-1","job-2","job-3"]
jobs REPORTED : 0
loop iterations : 36 (spin check: bounded by window/poll)
```
Zero claims, **3 of 3 jobs still pending**, and 36 iterations across 3
workers
over a 600ms window at a 50ms poll — i.e. 12 per worker, the poll
cadence. No
spin, no loss.
### The gate must NOT engage below threshold — same run, same command
```
=== B3 HEALTHY 1 worker @ 100/1000, 3 jobs pending ===
queue.claimNext CALLS : 15
jobs CLAIMED : 3 jobs STILL PENDING : 0 jobs REPORTED : 3
=== B4 UNDER-GATE 1 worker @ 890/1000, 3 jobs pending ===
queue.claimNext CALLS : 15
jobs CLAIMED : 3 jobs STILL PENDING : 0 jobs REPORTED : 3
=== B5 UNMEASURED 1 worker @ -1/-1, 3 jobs pending ===
queue.claimNext CALLS : 15
jobs CLAIMED : 3 jobs STILL PENDING : 0 jobs REPORTED : 3
```
`B4` is the load-bearing one: 0.89 is **above** change 2's 0.75 alarm
and below
this gate. The operator has been paged and the worker keeps working —
which is
the whole reason the two thresholds differ. `B5` pins the off-Linux
case.
---
# Tests and gates (changes 2 and 3)
`showcase/harness`, on the pushed head `63ef4081c2`:
| Gate | Command | Result |
|---|---|---|
| Typecheck | `npx tsc --noEmit -p tsconfig.json` | exit 0 |
| Lint | `npx oxlint src/fleet src/orchestrator.ts` | 0 errors; warning
count unchanged vs base (2 on `fleet-health.ts` before and after) |
| Format | `npx oxfmt --write` | clean |
| Unit suite | `vitest run` | **3696 passed, 18 skipped, 1 failed** —
175/178 files pass |
The single failure is `src/probes/frontend-matrix.test.ts` —
**pre-existing**.
Verified by stashing this branch's changes and re-running it on the
unmodified
tree, where it fails identically (`1 failed | 5 passed`).
14 tests are new (`3682 → 3696`).
**Mutation-tested for vacuity.** With the four source files reverted to
`e66fb1af13` and the new test files kept, **11 of the new tests fail**:
```
Test Files 2 failed (2)
Tests 11 failed | 80 passed (91)
FAIL fleet-health.test.ts > PID saturation > alarms on an ONLINE worker whose PID gauges cross the threshold
FAIL fleet-health.test.ts > PID saturation > stays silent below the threshold, including just under it
FAIL fleet-health.test.ts > PID saturation > stays silent when the gauges are UNMEASURED (null / unbounded max)
FAIL fleet-health.test.ts > PID saturation > is EDGE-triggered: a sustained saturation alarms once, not every cycle
FAIL fleet-health.test.ts > PID saturation > re-arms only after the worker drops below the CLEAR ratio
FAIL fleet-health.test.ts > PID saturation > never lets a throwing alarm hook abort the cycle
FAIL fleet-health.test.ts > PID saturation > fails loud on a ratio config that would alarm every cycle
FAIL worker-loop.test.ts > PID headroom gate > declines to claim at the PID ceiling — the job is never claimed, so it stays pending
FAIL worker-loop.test.ts > PID headroom gate > declines at the gate ratio boundary (0.90)
FAIL worker-loop.test.ts > PID headroom gate > honours an injected gate ratio
FAIL fleet-health.test.ts > checkOnce > never throws when the roster read fails — returns an empty cycle
```
The three that pass on both sides are the sub-threshold controls (`still
claims
just BELOW the gate`, `claims normally on a healthy worker`, `stays
INERT when
the cgroup gauges are unreadable`) — they are supposed to be unchanged
by the
fix, and they are.
# New env knobs (all optional, all defaulted)
| Var | Default | Effect |
|---|---|---|
| `WORKER_PID_SATURATION_RATIO` | `0.75` | Control-plane alarm
threshold. |
| `WORKER_PID_SATURATION_CLEAR_RATIO` | `0.65` | Alarm latch hysteresis
clear. |
| `WORKER_PID_CLAIM_GATE_RATIO` | `0.90` | Worker claim-gate threshold.
|
An out-of-range override on the two alarm ratios falls back to the
default
rather than tripping the fail-loud guard and taking the control-plane
down over
a typo'd env var.
# Not verified
- Neither change 2 nor change 3 has been exercised against **live prod
or
staging PocketBase**; both red-greens are local, on the real modules
with the
cgroup read and PB/queue transport injected.
- **No Slack post was sent.** `SLACK_WEBHOOK_OSS_ALERTS` was not set in
the
driver, so the send leg was not exercised end-to-end — only that its
failure
is swallowed without costing the durable status row.
- The per-job PID cost of a chromium context tree is **not measured**,
so the
0.90 gate's 100-PID reserve is a chosen value.
|
||
|
Mike Ryan
|
528dea6483 |
fix(react-core): ship a single v2 context instance (#6440)
## Problem
`@copilotkit/react-core` ships **two independent copies** of the v2
context module, so `useLicenseContext` imported from
`@copilotkit/react-core/v2/context` returns the default forever —
`status: null` even when `/info` reports `licenseStatus: "valid"`.
Reported downstream as a chat-history sidebar that never loads, because
`useThreads` is gated on license status.
`src/v2/context.ts` is compiled by two separate tsdown builds:
| Build | Output | Contains |
|---|---|---|
| `entry: ["src/index.tsx", "src/v2/index.ts"]` | `dist/` shared chunk |
inlined copy **A** |
| `entry: {context: "src/v2/context.ts"}` | `dist/v2/context.*` |
standalone copy **B** |
There is no import edge between them, so `createContext()` runs twice.
`CopilotKitProvider` lives in the shared chunk and publishes to **A**;
`@copilotkit/react-core/v2/context` exports **B**, which nothing ever
provides.
Verified against the published 1.66.4 artifact:
```
$ grep -n "createContext" dist/v2/context.mjs
104:const CopilotKitContext = createContext(null);
124:const LicenseContext = createContext({
$ grep -n "createContext" dist/copilotkit-nRjRp2_5.mjs # inside //#region src/v2/context.ts
1522:const CopilotKitContext = createContext(null);
1544:const LicenseContext = createContext({
$ grep -E '^import .*from "[^"]*context[^"]*"' dist/copilotkit-nRjRp2_5.mjs
# (empty — no import edge)
```
`CopilotKitContext` is duplicated identically, so `useCopilotKit`
imported from that subpath throws `"useCopilotKit must be used within
CopilotKitProvider"`. The subpath was effectively unusable for web
consumers; license was just the *silent* failure mode.
**Compounding defect:** `src/v2/providers/index.ts` enumerates its
exports by name and omits `useLicenseContext` (even though
`CopilotKitProvider.tsx:19` re-exports it). So the live copy had **no
public import path at all**, leaving consumers with no correct
alternative.
Not a 1.66.x regression — broken since
|
||
|
Jordan Ritter
|
63ef4081c2 |
fix(showcase/harness): stop a PID-saturated worker from claiming jobs
worker-loop's claim gate had exactly one conditional -- `if (budget.available <= 0)` -- and `available` is free Playwright browser-CONTEXT slots (maxContexts - liveContextCount), nothing else. A worker whose container has leaked PIDs to the cgroup ceiling therefore still advertises full capacity and keeps winning claims it cannot possibly run: the driver cannot fork, so each job burns its entire 600000ms lease and lands as an abort. Measured 2026-08-10: workers at pids=1000/1000 with 758-761 zombies, 320 `timeout after 600000ms` rows. Decline the claim above 0.90 of pids.max. Deliberately far above the control-plane's 0.75 saturation ALARM: the two thresholds do different jobs. 0.75 pages an operator while the worker is still perfectly able to run jobs; gating dispatch that early would convert a warning into ~36h of withheld fleet capacity. 0.90 leaves 100 free PIDs at the prod ceiling -- a chosen reserve, not a derived one (the per-job PID cost of a chromium context tree is not measured), hence the env override. The ordering that matters is that the alarm always fires before capacity is withdrawn. BLAST RADIUS -- reuses the SAME decline-and-idle path as the existing no-budget branch, which is what makes an all-workers-saturated fleet safe: the job is never claimed, so it is never dropped and never requeued; it simply stays `pending`. The loop then sleeps a full pollIntervalMs, so a fully-saturated fleet idles at the poll cadence instead of spinning. The queue stalls VISIBLY -- pending rows pile up, the rising-edge warn says why, and the control-plane's system:worker-pid-saturation alarm fired at 0.75 before this gate engaged at 0.90. A stalled, alarmed queue is recoverable by redeploy; claim-fail-repeat silently burned the whole cadence. The gate is INERT when the cgroup gauges are unreadable (pidUsageRatio -> null: off-Linux, every macOS dev box), so local workers claim exactly as before. |
||
|
Jordan Ritter
|
7f28e9bd39 |
fix(showcase/harness): alarm when a worker's cgroup PIDs saturate
Prod harness-workers replicas reach the platform-fixed pids.max=1000
ceiling roughly every 7 days and NOTHING alarmed. Measured 2026-08-10
20:09-20:10Z: pids=1000/1000 zombies=758, 1000/1000 zombies=761,
837/1000 zombies=744, with 320 `timeout after 600000ms` abort rows and
an e2e-smoke run 42 minutes into a 15-minute cadence with 3 jobs still
pending. The documented pool-unrecoverable alarm did not fire: it only
trips when the self-heal breaker gives up, which a PID-starved but
still-heartbeating worker never reaches.
The signal was already on the wire and read by nobody. The worker's
~75s heartbeat writes capacity_pids_current / capacity_pids_max onto
its `workers` row (worker/registration.ts), and fleet-health lists that
whole roster every 15s — its own comment said "the capacity gauges are
ignored here". Nothing in non-test source compared them.
Alarm control-plane-side rather than worker-side, because status-writer
documents `fleet-cp` as the only authoritative fleet writer ("workers
never write status directly"), and because fleet-health already holds
the roster read this needs.
- fleet-health raises a rising-edge alarm at pids.current/pids.max >=
0.75 (~36h of lead time at the observed leak rate), evaluated BEFORE
the online/stale branch since the failing worker is fully ONLINE.
- Latched per worker with a 0.65 hysteresis clear: the monitor ticks
every 15s, so an unlatched alarm would page ~8600 times across the
lead-time window. Construction fails loud on a ratio pair with no gap.
- Unmeasured gauges (null off-Linux, unbounded pids.max) never alarm --
pidUsageRatio returns null and null is never read as zero.
- Routed to a system:worker-pid-saturation status row plus #oss-alerts
via SLACK_WEBHOOK_OSS_ALERTS, the target family-silence and the
D0-gone monitor already post to. Deliberately NOT
SLACK_WEBHOOK_BROWSER_POOL_UNRECOVERABLE, which appears nowhere in
this repo outside its own definition and is unset everywhere -- an
alarm nobody receives is the gap being closed.
|
||
|
Jordan Ritter
|
e66fb1af13 |
fix(showcase/harness): reap orphaned chromium children with tini as PID 1
harness-workers leaked one cgroup PID slot per orphaned chromium grandchild. Node as PID 1 only waitpid()s processes it spawned, so every browser crash stranded ~5 <defunct> renderers permanently. Prod climbed to pids.current=1000/1000 with zombieCount=757 over 6d22h uptime, after which no browser could launch and ~295 d6 cells went abort fleet-wide. Install tini in the existing playwright apt layer and run it as PID 1 via exec-form ENTRYPOINT. No -g, so signal delivery to node is unchanged and orchestrator.ts's SIGTERM drain still runs; tini propagates the child's exit status so a crash still exits non-zero and Railway still restarts. |