mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
v1.68.0
14810 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Maxim
|
4b17ea7d35 |
fix(scripts): tokenize before hunting loader calls in the purity gate
The #4893 hard-fail gate's loader-call detector gave WRONG VERDICTS IN BOTH DIRECTIONS. It layered two regexes — a comment/string/template alternation that blanked only the comment branch, and `\b(?:import|require(?:\.resolve)?)\s*\(` over the result — then classified an argument as static from the FIRST CHARACTER after the paren. All nine shapes below were reproduced against the real gate before the rewrite: false FAIL throw new Error("use require(path) instead") false FAIL `import(${x})` inside a template false FAIL o.import(y) / mod.require(x) (member calls, not loaders) false PASS /https:\/\//; …import(n) (the regex's `//` blanked the rest of the line, hiding a real dynamic call) false PASS import(`stream${n}`) (merely STARTS with a quote) false PASS import("zo" + n) (same) false PASS import(`${base}/v2/index.mjs`) (same — the fat entry) false PASS __require(name) (no \b inside `__require`) Replaced with `scanSource`, a single-pass tokenizer that classifies every character as code / comment / string / template / regex and returns a length-preserving masked view plus a literal-span list. The one surviving regex now only ever sees code, so import-shaped TEXT cannot reach it at all; an argument counts as static only when it is one COMPLETE literal with no concatenation or interpolation; `__require` is matched; and a member call is rejected both by lookbehind and by a whitespace-skipping back-scan (so `m\n .import(x)` is not a loader either). Proven in both directions: nine innocent/violation pairs run through the real `assertEntryPurity`, each innocent form CLEAN and each matching real violation FAIL. Re-proved end-to-end by prepending `import "streamdown"` to the real dist/v2/headless.mjs — exit 1 naming all five families — then restoring it byte-identically. On the untouched dist the scan sees 66 loader calls in the `.cjs` graph and classifies all 66 static, so it passes because it LOOKED. Also adds the first `.cjs` fixtures: every existing fixture was `.mjs`, leaving the script's `format: "cjs"` branch and the `require()` shape asserted by nothing. Tests 24 → 47. `stripComments` is renamed `maskNonCode`, since it now blanks literals and regexes too; it had no caller outside this script and its test. The RN guard keeps its own copy, untouched. dev-docs/bundle-size.md: the four holes a sibling agent documented as known limitations this round are closed and removed from that list; what genuinely remains (regex-vs-division heuristic, no JSX/TS, indirect loaders) replaces them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Maxim
|
4c17a8fe8c |
fix(react-native): key the messages fingerprint on object content
`messagesFingerprint`'s content key collapsed every object to 0, so an
in-place content replacement that kept the same message id was invisible to
every memo derived from it. Its comment claimed to mirror react-core's
`messagesMemoKey`, which stopped being true when react-core #6325
(
|
||
|
Maxim
|
1b39c12e36 |
fix(scripts): stop the headless CLI gates skipping themselves on odd paths
Both scripts decided "am I the entrypoint?" by comparing `import.meta.url` to a `file://`-concatenated `process.argv[1]`. `import.meta.url` is percent-encoded and symlink-resolved; raw argv[1] is neither. So the comparison was false for any checkout path containing a space, for any invocation through a symlink (macOS /tmp is one), and on Windows — and a false guard skipped the whole CLI block. Reproduced before fixing: the #4893 purity gate and the bundle-size measurement both exited 0 having printed nothing and asserted nothing, which is worse than a gate with holes because it manufactures confidence. The guard was added by this PR so the modules could export internals to their new negative tests; making the gates testable introduced a way for them not to run. Both now compare real filesystem paths through an exported `isEntrypoint`: `fileURLToPath` defeats the encoding and Windows forms, `fs.realpathSync` on both sides defeats symlinks, and a `path.resolve` fallback keeps a nonexistent argv[1] from throwing. Each `node --test` suite gains five entry-guard tests, including an end-to-end spawn of the real script through a symlinked package-root alias whose name contains a space — the only case that catches the call site regressing back to a string comparison (verified: it fails against the old expression). The unit cases assert the naive comparison really would have failed, so none of them can pass vacuously. Both negative gates were re-proven to still bite: a doctored dist entry pulling streamdown fails the purity gate, and a stubbed dist entry trips the measurement's plausibility floor. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Maxim
|
c7d176f264 |
test(react-native): make the #4893 entry guard fail on violations, not on growth
The headless import-graph guard pinned the resolved graph EXACTLY — the
11-module list and the 8 bare specifiers, both `toEqual`. That catch-all was
deliberate (a heavy dependency nobody enumerated still had to be looked at),
but it also went red on innocent growth: adding any first-party `src/` module
to the headless graph failed it, on someone else's unrelated PR. A guard that
fails on innocent changes gets deleted by the third person who hits it, and
then it guards nothing.
Express the catch-all over PACKAGES instead of MODULES: the graph may only
reach packages a headless consumer is guaranteed to be able to resolve — this
package's `dependencies` plus its NON-optional `peerDependencies`, read from
package.json rather than hand-copied. That is precisely the promise the
headless entry sells ("bundles with nothing stubbed in metro.config.js"), so
it still fails on any new third-party edge, on every optional peer, on a
devDependency, and on a Node builtin — while a new first-party module or
another import of an already-sanctioned package is free.
The two other things the pin bought are kept explicitly:
- Comment stripping. The eight phantom specifiers JSDoc examples used to
harvest were all self-references, and this package's own name is not in the
guaranteed set, so a `stripComments` regression still fails here.
- Non-vacuity. Every remaining graph assertion is a deny-list, and a deny-list
over a truncated graph passes for the wrong reason, so a subset floor
asserts the walk still reaches the provider, the polyfills and the
react-core headless edge.
Not changed: comment stripping itself, the import()/require()/require.resolve
extraction, non-literal loader flagging, emitted-extension resolution, the
loud failure on unresolvable edges, the entry-presence tests, the #4893
fat-entry ban (still the assertion that catches `@copilotkit/react-core/v2`)
or the heavy-dependency ban. The runtime-export `beforeAll` is untouched.
Proven both directions: a new first-party module passes the loosened guard and
fails the old pin; `@copilotkit/react-core/v2`, `shiki`, an unenumerated
devDependency edge and a truncated walk each fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Maxim
|
9eacbfdec3 |
docs(react-native): name the TS18048 args break in useRenderTool
The PR's reviewer asked that the `RenderToolProps` shape change be covered where upgraders
actually read it. A `BREAKING CHANGE:` footer on
|
||
|
Maxim
|
4cf8640d97 |
docs(react-native): retract the wildcard-renderer claim for RN
The guide told readers that a `"*"` entry resolves as a renderer-only wildcard
on React Native "exactly as it does on the web", and framed the one difference
as an ergonomic tax (a `"*"` entry "still needs `parameters`"). That is false,
and the real consequence is not ergonomic.
react-core's `useRenderTool` is renderer-only: its body's sole registration is
`addHookRenderToolCall` (use-render-tool.tsx:190), and it special-cases
`name === "*" && !parameters` into a schema-less fallback renderer
(use-render-tool.tsx:166) which `useRenderToolCall` resolves last
(use-render-tool-call.tsx:151). React Native's hook instead delegates wholesale
to `useFrontendTool` (useRenderTool.ts:53), which calls `addTool`
unconditionally (use-frontend-tool.tsx:23). So on RN `name: "*"` registers a
frontend tool literally named `*`. `buildFrontendTools` has no wildcard
exclusion (run-handler.ts:1236), so that tool is advertised to the model in the
run's tool list, and it occupies core's separate wildcard-executable-tool slot
(run-handler.ts:610, 626) whose handler is invoked for every unmatched tool call
with args wrapped as `{ toolName, args }` (run-handler.ts:988) rather than in
the caller's declared shape.
- Rewrote the bullet to advise against `"*"` on React Native and state the
mechanism. Also completed its requirements list: `description` is as
non-optional as `parameters` (useRenderTool.ts:13), and react-core's hook
takes no `description` at all.
- Fixed the example's `status !== "complete"`, which was a web-shaped parity
assumption that does not compile: RN's props derive from
`ReactToolCallRenderer`, whose `status` is the `ToolCallStatus` enum, not
web's string literals. Branches on `ToolCallStatus.Complete` now, and notes
the `args`-vs-`parameters` difference alongside it.
- Added a Known limitations entry, in the voice of the `threadId` one, covering
the wildcard gap plus two more the audit turned up: `followUp` and
`available` are accepted by `useFrontendTool` and not forwarded, and the
`handler` type drops the `context` argument core does pass at runtime
(run-handler.ts:821). All tracked for the convergence follow-up.
No `followUp` claim was present on the page to correct — the gap is real, so it
is documented as a limitation rather than a retraction. No behaviour change:
`packages/react-native/src/hooks/useRenderTool.ts` is untouched, the
convergence is its own PR.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Maxim
|
ed087edbf4 |
docs(bundle-size): describe the purity gate that actually exists
|
||
|
Maxim
|
00caf5fa7b |
docs(react-native): fix the chat memo comment's identity rationale
`messagesFingerprint`'s header JSDoc and the matching inline comment near the
`listItems` memo justified keying on message CONTENT with a claim that is false:
that `agent.messages` is mutated in place throughout, and that "the AG-UI apply
pipeline reuses one array for a whole run".
It does not. `@ag-ui/client`'s `AbstractAgent.processApplyEvents` REASSIGNS
`this.messages = applied.messages` for every applied event, so a streaming run
hands down a new array — and new message, `toolCall` and `function` objects — per
delta. Verified against a real AG-UI run by the PR reviewer, and confirmed here in
@ag-ui/client 0.0.57's `AbstractAgent`. The old grep behind the claim ("assigning
`.messages` in packages/core/src hits test files only") is accurate but proves
nothing: `@ag-ui/client` is a dependency, outside that tree.
The fix itself stands. Identity is unreliable in BOTH directions, which is the
actual rationale: it changes on the apply path, and it does NOT change on the
paths these memos exist to serve — core splices tool results in place
(`agent.messages.splice(insertAt, 0, toolMessage)`,
packages/core/src/core/run-handler.ts:931, :1080), `AbstractAgent.addMessage` is a
`this.messages.push(...)`, and `useAgent` re-renders with a bare `forceUpdate()`
(packages/react-core/src/v2/hooks/use-agent.tsx:382-396). A signal that both
misses changes and fires without them cannot be a dependency, so the derivations
must key on content.
Comments only: three sites reworded (the JSDoc, the "cannot be used" pointer at
the `messagesKey` call, and the inline note on the `listItems` memo). `git diff`
touches no behaviour, type or dependency array — every changed line is a comment.
The `contentKey` length-vs-value paragraph is left alone; another change owns it.
Note: the same false claim is in commit 77ed31c437's body, which cannot be
rewritten, and in a GitHub review comment.
Not run in this worktree: it has no node_modules, and the change is comment-only,
so it cannot affect types, lint or tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Maxim
|
21da05c278 |
docs(react-native): retract the useRenderTool status-compare claim
An earlier commit rewrote the useRenderTool reference to say that comparing a render prop's `status` against a raw string no longer type-checks, and called that a breaking change. That was wrong, and it made upgraders believe working code was broken. `ToolCallStatus` is a string enum, and TypeScript relates an enum literal type to a same-valued plain string literal (not the reverse). Equality tests comparability both ways, so `status === "complete"` compiles AND narrows. Only two forms fail: comparing against a string that matches no member (TS2367), and assigning a raw string to a `status`-typed variable (TS2322) — assignment, never comparison. Corrected all three sites that claimed otherwise (the RenderToolProps narrative, the `status` PropertyReference, and the migration section's "both halves are breaking"). The enum-based examples stay, now framed as recommended style — self-documenting, and loud if a member's value changes — rather than a compilation requirement. The migration section now names the added `InProgress` arm as the one genuinely breaking half. Also fixed a separate false claim in the same file: the out-of-chat rendering example said that without a `toolMessage` the status "stays InProgress ... forever". `useRenderToolCall` reads `executingToolCallIds` from the provider and branches toolMessage -> Complete, else isExecuting -> Executing, else InProgress, so the Executing arm is reachable with no tool message at all. Docs only; no source behaviour changes. |
||
|
Jerel Velarde
|
b075704c77 |
feat(examples): add grok-generative-ui showcase (#6475)
## What does this PR do? Adds a new showcase: **`examples/showcases/grok-generative-ui`**. Ask what X thinks about anything. `grok-4.6` runs xAI's **X Search** (`x_search`) server-side, then composes the answer out of real React components through CopilotKit frontend tools — the model picks which components appear and what goes in them. There is no fixed dashboard being filled in. Every post rendered is a real post the model found; nothing on screen is authored by hand. https://github.com/user-attachments/assets/9a818bc7-07bb-442a-82f6-c61e554a983d **What it demonstrates** - 5 frontend tools via `useFrontendTool` — one search, four renderers (`renderSummary`, `renderSentimentSplit`, `renderArgumentMap`, `renderReceipts`) - `BuiltInAgent` pointed at a `LanguageModel` instance (`xai.responses("grok-4.6")`) rather than a model string — which is what makes xAI's server-side tools reachable - Headless `CopilotChatView` with its `scrollView` / `input` slots composed into the page layout, so one mounted chat serves both the centered hero state and the docked rail - Paint-in reveal: each panel walks skeleton → wireframe → rendered as its tool call lands **Gotchas documented in the README** (each cost real debugging time and isn't obvious from the docs) - Registering a backend `ToolDefinition` on `BuiltInAgent` alongside `useFrontendTool` tools **silently stops the frontend tools from reaching the model** — it reports them as unavailable and narrates fabricated results instead of rendering. All five tools are frontend tools here for that reason. - Runs must go through `copilotkit.runAgent({ agent })`; the raw `agent.runAgent()` runs without the registered frontend tools. - `CopilotChatView` returns its own welcome layout **before** reading the `children` render prop, so a composed layout is silently discarded until the first message unless you pass `welcomeScreen={false}`. - `maxSteps` defaults to `1`, so the agent searches and stops before rendering. **Repo-level changes (2 files, both required)** - `.github/config-allowlist.txt` — registers `examples/showcases/grok-generative-ui/next.config.ts`. Required; `static / check binaries` fails otherwise. This is CODEOWNERS-gated, so this PR needs core-dev review and will not qualify for showcase auto-merge. - `examples/README.md` — adds the index row and bumps the counts (24 → 25, 50 → 51). **Notes** - Standalone npm project, pinned deps, own `package-lock.json`. Not added to `pnpm-workspace.yaml`, matching every showcase except the three that are explicitly enumerated there. - `npm install && npm run build` verified clean in place. - No changeset, per CONTRIBUTING. - `demo.png` committed through git-lfs (387 KB, under the 1 MB gate). ## Related PRs and Issues None. ## Checklist - [x] I have read the [Contribution Guide](https://github.com/CopilotKit/CopilotKit/blob/main/CONTRIBUTING.md) - [x] If the PR changes or adds functionality, I have updated the relevant documentation - [x] "Allow edits by maintainers" is checked |
||
|
Ben Taylor
|
a3b814a041 |
fix(core): refresh Intelligence delegate headers before every join (#6469)
## Summary
`ProxiedCopilotRuntimeAgent` builds its `IntelligenceAgent` delegate
**once** and caches it for the proxy's lifetime, copying `headers` into
the delegate's constructor config. Nothing ever refreshed that copy, so
**a header that changed after the delegate was created never reached
`/connect` or `/run`** — for the life of the agent.
For a multi-tenant app carrying the active tenant in a header, the join
was attempted under the *previous* tenant's identity with the *new*
tenant's thread id, and the platform correctly answered
`THREAD_NOT_FOUND`. Only a full page reload cleared it, because that
rebuilds the delegate. A rotated or refreshed `Authorization` bearer has
the same exposure.
Reported by Sameday against 1.67.1 with a deterministic staging repro:
```
19:45:23.554 | /copilotkit/runtime/threads | hdr=<tenant B> | 200
19:45:23.886 | /copilotkit/runtime/threads/subscribe | hdr=<tenant B> | 200
19:45:23.893 | /copilotkit/runtime/agent/<id>/connect | hdr=<tenant A> | body.companyId=<tenant B> | 404
```
`/threads` carries **B** while `/connect` carries **A**, ~340ms apart in
the same switch. Not a race — a stale copy with no refresh path.
## Root cause
`setHeaders` / `applyHeadersToAgent` could not fix this: they write an
agent's `.headers`, and `IntelligenceAgent` exposed only `private
config`. `syncDelegate` *looks* like the refresh path, but its
`hasHeaders` probe is `"headers" in agent` — false for the delegate,
since `headers` is declared on `HttpAgent`, not on `AbstractAgent`. So
`config.headers` was the sole header source for Intelligence REST calls,
with no refresh path at all.
## The fix
Expose `headers` as a public accessor pair backed by `config`, and read
it in `requestJoinCredentials$`.
**The accessor is the entire fix**: it makes `hasHeaders` true, so
`syncDelegate` — which already runs on every `resolveDelegate()`, and is
preceded by `applyHeadersToAgent` in `RunHandler.connectAgent` — starts
actually refreshing the delegate before each join. No new plumbing.
Two things worth flagging for reviewers:
1. **The originally-suggested fix ("make `requestJoinCredentials$` read
live headers") does not work on its own** — and is actively harmful.
There was no live header source on the class to read: without the
accessor, `this.headers` is `undefined` and **every header is dropped**
(verified: only `Content-Type` survives). The read here goes through the
accessor for a single source of truth, not because that read carries the
fix.
2. **The setter replaces the config object rather than mutating it**,
because `clone()` shares the config reference. The join path alone would
mask an in-place write (`syncDelegate` rewrites headers just before
every join), but the credential re-acquisition inside a running pipeline
(`intelligence-agent.ts:563`) does not re-sync — so a clone's tenant
could ride out on the original's socket-error refresh. That's the same
cross-tenant leak this accessor exists to prevent.
`credentials` had the identical defect via `config.credentials`
(`hasCredentials` was false too) and gets the same treatment.
## Testing
**Unit tests (5 new, each written first and watched fail).** The pre-fix
failure is the staging symptom reproduced:
```
FAIL > sends a header changed after the delegate was created
AssertionError: expected { …(2) } to match object { 'X-Tenant': 'tenant-b' }
- "X-Tenant": "tenant-b",
+ "X-Tenant": "tenant-a",
```
Coverage: a header changed post-construction reaches `/connect`; the
same on the `/run` path (which was independently verified broken
pre-fix, sending tenant A where B was expected); credentials likewise; a
clone's header update must not reach the original
(`IntelligenceAgent.clone()` invariant — this one fails under in-place
config mutation); and a per-thread clone and its original each send
their own tenant.
**Verified beyond the unit tests.** Because the mocked-harness result
alone doesn't prove the production wiring, I drove the real chain —
`CopilotKitCore.setHeaders` → registry → proxy → delegate → outbound
POST — in a plain Node process with no vitest and no `vi.mock`, stubbing
only `fetch` at the network boundary. Same script against the unfixed
file, then the fix:
```
BEFORE (origin/main) AFTER (this PR)
"headers" in delegate: false "headers" in delegate: true
delegate.headers: undefined delegate.headers: { X-Tenant: tenant-b }
proxy.headers after setHeaders(B): proxy.headers after setHeaders(B):
{ X-Tenant: tenant-b } { X-Tenant: tenant-b }
0: POST /connect X-Tenant=tenant-a 0: POST /connect X-Tenant=tenant-a
1: POST /connect X-Tenant=tenant-a <-- 1: POST /connect X-Tenant=tenant-b credentials=include
FAIL (stale headers) PASS (live headers reach /connect)
```
The "before" column reproduces the report's tell exactly:
`proxy.headers` correct at tenant B while `/connect` still sends tenant
A, through the very API the report found ineffective.
**Gates** (run in a worktree with a freshly built `@copilotkit/shared`,
since a stale dist otherwise produces 20 unrelated
`core-inspector-metadata` failures and 4 `tsc` errors):
| Gate | Result |
| --- | --- |
| `@copilotkit/core` vitest | **654 passed / 654**, 59/59 files |
| `tsc --noEmit` | clean |
| `oxlint` | 0 errors (2 warnings, both pre-existing test helpers) |
| `oxfmt` | no reformatting needed |
**Not covered:** `fetch` is stubbed, so this does not exercise a live
Intelligence gateway or a browser tenant switch — it proves the outbound
header is correct, not the platform's response to it.
## Note for whoever merges
#6450 and #6468 also touch `intelligence-agent.ts` (thread-restore work)
but neither goes near the header path, so conflicts should be textual at
worst.
## Follow-up left out of scope
Two separate pre-existing defects surfaced while verifying this one.
Neither is touched here.
**1. `credentials` passed to a `ProxiedCopilotRuntimeAgent` constructor
are dropped at registration.** `applyCredentialsToAgent` overwrites
`agent.credentials` from core unconditionally, with no per-agent
baseline — unlike `applyHeadersToAgent`, which merges over the
`agentOwnHeaders` baseline captured for exactly this reason (#5635).
Probed in a real process: an agent constructed with `credentials:
"include"` in a core with none configured reports `undefined`
immediately after registration, and every join goes out without
credentials. Identical before and after this PR, so it is not a
regression from this change — but the headers/credentials asymmetry
looks unintended, given #5433 was specifically about preserving proxied
runtime credentials.
**2. `buildRuntimeUrl` reads `config.agentId`
(`intelligence-agent.ts:770`), (`intelligence-agent.ts:770`), so
`syncDelegate`'s `delegate.agentId = routedAgentId()` is cosmetic for
the REST URL. Same root-cause class as this bug, but latent rather than
live (routing is fixed per proxy instance).
Happy to file both separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Benjamin Taylor
|
70545f072a |
test(core): correct an overclaiming comment, tighten the credentials assertion
The per-thread-clone test's comment claimed it guards the copy-on-write setter. It does not: syncDelegate rewrites headers before every join, so it passes even with an in-place write (verified). Say what it actually pins — each proxy's joins carry its own tenant — and point at the clone-invariant test that does guard the setter. Also assert the pre-change join carried no credentials, so the credentials test shows a transition rather than a single end state. |
||
|
Murat Sari
|
9785e135ea |
fix(angular): prevent duplicate OpenGenerativeUI sandboxes (#6477)
Prevents stale async loaders from creating duplicate OpenGenerativeUI sandbox iframes by tying final sandbox creation to `afterRenderEffect` cleanup. Adds regression tests for identical content and A → B → A races. ### Testing - Angular tests - Type checking - Angular package build |
||
|
Murat Sari
|
dde84a5804 | fix(angular): prevent duplicate OpenGenerativeUI sandboxes | ||
|
Jerel John Velarde
|
bd09c3d790 |
chore(examples): drop grok showcase lockfile
A 13.9k-line new file trips the fork-PR supply-chain heuristic (security_fork-pr-alert flags any added file over 5000 lines), and the job cannot post its explanation because fork tokens are read-only. Several showcases ship no lockfile; this one is not a pnpm workspace member, so nothing depends on it. |
||
|
Jerel John Velarde
|
fe0e7cf28f |
feat(examples): add grok-generative-ui showcase
grok-4.6 runs xAI's X Search server-side, then composes the answer out of real React components through five CopilotKit frontend tools. Every post rendered is a real post the model found. Registers next.config.ts in the build-config allowlist and adds the row to the examples index. |
||
|
Murat Sari
|
6de1b96da2 |
fix(core): prevent duplicate interrupt tool results (#6201) (#6470)
## Summary Only create client-side tool results for `tool_call` interrupts. Backend-owned interrupts now resume without synthetic tool messages, preventing duplicate results. Includes Core, Angular, and React test coverage. ## References - Fixes #6201 - Related: #6270 - [AG-UI interrupts](https://docs.ag-ui.com/concepts/interrupts) ## Validation - [x] Core, Angular, and React tests - [x] Full workspace test suite - [x] Type checks |
||
|
Ran Shem Tov
|
fe21ee439e | fix(showcase): repair CrewAI CI build gates | ||
|
KNChiu
|
d7dd1bcfbe | docs(examples): fix stale clone paths in v1 example READMEs | ||
|
Murat Sari
|
01c7283210 | fix(core): prevent duplicate interrupt tool results (#6201) | ||
|
Ran Shem Tov
|
04142ef4e2 | fix(ci): allowlist CrewAI conversational config | ||
|
Ran Shem Tov
|
a3ee26b424 | Merge remote-tracking branch 'origin/main' into codex/crewai-full-d6 | ||
|
Ran Shem Tov
|
48a01b6203 | fix(showcase): harden CrewAI probe parity | ||
|
Benjamin Taylor
|
7d1cdc15df |
test(core): pin the run path against stale Intelligence headers
The report names both /connect and /run. The run path reaches the delegate through #runViaDelegate, which shares resolveDelegate with the connect path, so the accessor fixes both — but that was inferred from the shared call site rather than pinned. Verified failing against the pre-fix file (sent tenant-a where tenant-b was expected). |
||
|
Benjamin Taylor
|
a3562c20a6 |
fix(core): refresh Intelligence delegate headers before every join
`ProxiedCopilotRuntimeAgent` builds its `IntelligenceAgent` delegate once and caches it for the proxy's lifetime, copying `headers` into the delegate's constructor config. Nothing ever refreshed that copy, so a header that changed later never reached `/connect` or `/run` — for the life of the agent. `setHeaders`/`applyHeadersToAgent` could not fix it: they write an agent's `.headers`, and `IntelligenceAgent` exposed only `private config`. `syncDelegate` looked like the refresh path but its `hasHeaders` probe is `"headers" in agent`, which was false for the delegate. Multi-tenant apps that carry the active tenant in a header saw the join attempted under the previous tenant's identity with the new tenant's thread id, answered THREAD_NOT_FOUND. A rotated `Authorization` bearer has the same exposure. Only a full reload cleared it. Expose `headers` as a public accessor pair backed by `config`. The accessor is the entire fix: it makes `hasHeaders` true, so `syncDelegate` — which already runs on every `resolveDelegate()` — starts actually refreshing the delegate before each join. Note that changing `requestJoinCredentials$` to read live headers, as the report suggested, does nothing on its own: there was no live source on the class to read, and without the accessor `this.headers` is `undefined`, which drops every header. It reads through the accessor here for a single source of truth, not because that read carries the fix. The setter replaces the config object rather than mutating it, because `clone()` shares the config reference. The join path alone would mask an in-place write (syncDelegate rewrites headers just before every join), but the credential re-acquisition inside a running pipeline does not re-sync, so a clone's tenant could ride out on the original's socket-error refresh. `credentials` had the identical defect via `config.credentials` (`hasCredentials` was false too) and gets the same treatment. Verified beyond the unit tests by driving the real chain (`CopilotKitCore.setHeaders` -> registry -> proxy -> delegate -> outbound POST) in a plain Node process with only `fetch` stubbed: before, `"headers" in delegate` was false and the join after a tenant switch still sent tenant A; after, it sends tenant B. Reported by Sameday against 1.67.1 with a deterministic staging repro. |
||
|
Maxim
|
784f2e7529 |
docs(reskinnable-demo): retire the last "all six skins" claims after bookstore
The merge of main brought a seventh skin. These are the surviving count claims
outside the conflicted files, all of which the seventh skin falsified:
- `airline` was described as "the one PASSENGER-FACING skin"; `bookstore` is
also customer-facing, so it now names the pair.
- demo-beats.md still told a skin author "every registered skin is
demo-complete, so there is no partial precedent to copy". Bookstore IS a
partial precedent, deliberately, so the sentence now says so.
- Eight in-skin comments said "all six skins" while describing something that
is true of the WHOLE roster (the shared PDF primitive's coverage, the dark
treatment, and — load-bearing — the project-scope warning in three
seed-memories.ts files, where undercounting understates the blast radius of
a project-scoped sweep). All now say "every skin", which cannot rot.
Reskin-skill staleness check (CLAUDE.md standing rule): yes, demo-beats.md is
part of the skill and is corrected here.
Verified from examples/showcases/reskinnable-demo: `pnpm lint` clean,
`pnpm exec tsc --noEmit` 0 errors, and the roster/config drift guards plus the
touched skin tests pass (110 tests).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Maxim
|
6473cdcf9d |
feat(reskinnable-demo): merge main, and reconcile the docs with a seventh skin
Brings in `bookstore` and 31 other commits from main.
WHY THIS MERGE CONFLICTED IN SEVEN FILES. Both sides hand-maintained the
same roster. This branch had just rewritten the docs around four
conclusions that were true when written:
- `useData` has zero implementors
- no in-memory skin remains
- every registered skin is demo-complete
- there are six skins
`bookstore` falsifies all four: it sets `useData: useBookstoreData`, so the
optional hook has a live implementor and an in-memory skin exists again; it
ships intelligence/{seed,forget}-memories.ts but no teach loop, so it is not
demo-complete; and it is the seventh.
Neither side was wrong. The resolution is the union, and where a list or a
count was load-bearing it is now the command that derives it -- which is the
convention this branch adopted precisely because two hand-maintained copies
of one roster is what produced these conflicts.
The de-narration this branch applied is preserved: main's older phrasings
carried retrospective prose that was deliberately removed, and it has not
been reintroduced.
Registration verified rather than assumed -- bookstore is present in
LINTED_SKIN_IDS, skinIds, skinIdentities, SkinRegistry and agentRegistry.
That last one has no drift guard at all, so a missing key there fails only
when someone sends a chat message.
Gates on the merged tree: lint clean, `tsc --noEmit` 0 errors, 210 test files
/ 2414 tests passing (up from 197/2227 -- bookstore's own, nothing dropped),
build exit 0.
COMMITTED WITH --no-verify, DELIBERATELY, WITH THE USER'S APPROVAL.
The pre-commit hook was bypassed. That is normally forbidden here, so the
reason is recorded rather than left to be guessed:
- This branch's ENTIRE diff against main is inside
examples/showcases/reskinnable-demo. `git diff --name-only origin/main...HEAD`
lists nothing outside it.
- The hook fails on `@copilotkit/vue` -> CopilotThreadsDrawer.ssr.test.ts,
"does not eagerly evaluate the Lit element module when the package entry is
imported". That test fails STANDALONE on this machine
(`npx nx test @copilotkit/vue` -> 1 failed | 1073 passed, exit 1), with no
merge in progress and nothing of ours involved. It asserts a lazy-import
property but enforces it with a 5000ms wall-clock timeout, so it fails
whenever module resolution is slow rather than when Lit is actually
eagerly evaluated.
- This is simply the first commit on the branch to touch packages/*, so it is
the first to make `nx affected` run that suite. Ninety earlier commits
touched only the demo app and never triggered it.
What WAS verified on the merged tree, by hand, before committing:
pnpm lint clean
pnpm exec tsc --noEmit 0 errors
pnpm test:unit 210 files / 2414 tests passing
pnpm build exit 0
npx nx test @copilotkit/runtime 138 files passing
That last one only passes because of a second pre-existing breakage fixed
along the way: packages/runtime's better-sqlite3 binary was compiled against
NODE_MODULE_VERSION 137 (Node 24) while .nvmrc pins Node 22 (127), so every
SqliteAgentRunner test threw on load. `pnpm rebuild -r better-sqlite3` fixed
it. That fix is environmental and is not part of this commit.
Two follow-ups worth someone's time, neither blocking:
1. The vue SSR test should assert the property (module not evaluated) rather
than time the import.
2. Nothing in the repo pins the Node version for native rebuilds, so a
contributor who once ran a task under Node 24 silently poisons
better-sqlite3 for every later Node 22 run.
|
||
|
Jordan Ritter
|
7fd4c5ee78 |
fix(showcase/harness): reap chromium zombies, alarm on PID saturation, stop saturated workers claiming (#6333)
> **This PR carries three changes to the same prod incident.** Change 1
fixes
> the PID leak itself; changes 2 and 3 fix the two reasons the leak was
able to
> starve the fleet unannounced. Each has its own RED → GREEN below.
>
> | # | Commit | What it fixes |
> |---|---|---|
> | 1 | `e66fb1af13` | **The leak.** No PID-1 reaper, so crashed
chromium grandchildren become permanent zombies holding cgroup PID
slots. |
> | 2 | `7f28e9bd39` | **Nothing alarmed.** The worker heartbeats its
cgroup PID gauges and no non-test code read them. |
> | 3 | `63ef4081c2` | **Saturated workers kept claiming.** The claim
gate only checked free browser-context slots, so a worker at 1000/1000
PIDs won jobs it could not run and burned each to a 600000ms abort. |
---
# CHANGE 1 — the leak (tini as PID 1)
## The bug
`harness-workers` has **no PID-1 zombie reaper**, so it leaks cgroup PID
slots until it cannot launch a browser.
libuv's `SIGCHLD` handler only `waitpid()`s the pids **Node itself
spawned**. Playwright spawns the chromium *browser* process (reaped
fine), but that browser's **zygote/renderer/GPU children are
grandchildren**. When a browser process dies — prod's `Target crashed`
path, or a pool self-heal relaunch — those grandchildren are re-parented
onto PID 1 and exit. Node-as-PID-1 never waits on them, so each becomes
a permanent `<defunct>` still holding a cgroup PID slot.
Prod, on a single container up 6d22h: `zombieCount` 70 → **757**,
`cgroupPidsCurrent` 286 → **1000/1000**. Once saturated, probes could
not launch browsers (`Target crashed`, `browserContext.newPage: … has
been closed`, `feature exceeded 300000ms wall-clock`) and **295 of 414
red prod d6 cells went `abort`, fleet-wide across ~20 slugs.**
This Dockerfile already documented the outage mode in a 40-line `FIX #3`
block — but **both of its stated mitigations are demand-side**
(`BROWSER_POOL_MAX_CONTEXTS` 40→24, resource-gauge warning logs). Those
cap the *peak* of `pids.current`. They cannot stop a *monotonic* leak,
and prod died of a monotonic leak. `resource-gauges.ts` only ever
**measured** zombies (`if (s.state === "Z") zombieCount += 1`); nothing
reaped them.
## The change
One file, +59/-3. `tini` as PID 1.
```dockerfile
RUN node ./node_modules/playwright/cli.js install --with-deps chromium \
&& apt-get install -y --no-install-recommends tini \
&& /usr/bin/tini --version \
&& rm -rf /var/lib/apt/lists/*
…
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["/bin/bash", "-c", "ulimit -u $(ulimit -Hu) 2>/dev/null || true; exec node dist/orchestrator.js"]
```
Why this form:
- **`tini` baked into the image, not Docker `--init`** — `--init` is a
*runtime* flag and Railway does not expose it, the same reason
`pids.max` isn't settable from here (already documented in this file).
- **Not an in-process reaper** — Node has no `waitpid` binding, so an
application `SIGCHLD` handler would need a native addon. This is the
platform's job.
- **Rides the existing apt layer** — playwright's `--with-deps` just ran
`apt-get update`, so this costs one ~267KB package download instead of a
second index refresh. The `tini --version` call is a build-time
assertion on the path baked into `ENTRYPOINT`: if a future Debian moves
the binary, the **build** fails rather than the container failing to
start in prod.
- **No `-g`** — tini's process-group broadcast would signal the live
chromium pool alongside node and pre-empt `orchestrator.ts`'s
`process.once("SIGTERM", drainAndExit)`. Without it, delivery is
byte-identical to today: exactly one process gets the signal.
- **Exec form, `exec` kept in CMD** — tini really is PID 1 and really
receives Railway's SIGTERM; node is tini's direct child with no bash
lingering in the tree.
No existing convention to match: `grep` for `tini`/`dumb-init`/`--init`
across every Dockerfile in the repo returns zero hits. Nothing in the
repo asserts on this image's `CMD`/`ENTRYPOINT`, and nothing outside the
CI build step `docker run`s it, so adding an `ENTRYPOINT` breaks no
consumer.
---
# RED → GREEN
Real container, real chromium, real zombies. **Identical argv for
both**; only the image differs, and the image supplies PID 1:
```
docker run --rm --pids-limit 1000 -e REPRO_CYCLES=230 \
-v <repro>:/repro:ro <image> /bin/bash -c "exec node /repro/zombie-repro.mjs"
```
`--pids-limit 1000` mirrors Railway's platform-fixed ceiling. Each cycle
launches chromium with the exact args `browser-pool.ts` uses
(`headless`, `--no-sandbox`, `--disable-dev-shm-usage`), opens a
context+page, then **SIGKILLs the browser process** — prod's `Target
crashed` path, orphaning its children onto PID 1.
**Measurement is container-wide on purpose.** The harness's own
`sampleResourceGauges()` walks the tree from `process.pid`; under a real
init the orphans re-parent to the *init*, not to node, so a node-rooted
walk would report zero zombies **whether or not they were reaped** — a
vacuous GREEN. Counting every state-`Z` process in `/proc` plus the
cgroup counter is blind to which process is PID 1, so it is honest in
both directions.
## RED (unmodified `main`, `pid1=node`)
20 cycles — exactly **+5 permanent zombies per crashed browser**:
```
baseline zombieCount=0 procCount=1 threadCount=7 cgroupPidsCurrent=7 cgroupPidsMax=1000 selfPid=1 pid1=node
PID PPID S THR COMMAND
1 0 R 7 node
cycle=1 pre-kill zombieCount=0 procCount=7 threadCount=76 cgroupPidsCurrent=76 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=1 zombieCount=5 procCount=6 threadCount=16 cgroupPidsCurrent=16 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=2 zombieCount=10 procCount=11 threadCount=21 cgroupPidsCurrent=21 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=3 zombieCount=15 procCount=16 threadCount=26 cgroupPidsCurrent=26 cgroupPidsMax=1000 selfPid=1 pid1=node
…
cycle=20 zombieCount=100 procCount=101 threadCount=111 cgroupPidsCurrent=111 cgroupPidsMax=1000 selfPid=1 pid1=node
final zombieCount=100 procCount=101 threadCount=111 cgroupPidsCurrent=111 cgroupPidsMax=1000 selfPid=1 pid1=node
SUMMARY cycles=20 zombiesBefore=0 zombiesAfter=100 pidsBefore=7 pidsAfter=111 launchFailure=none
```
The leaked processes, verbatim from the final `/proc` table — all
`ppid=1`, all `Z`:
```
21 1 Z 1 headless_shell <defunct>
22 1 Z 1 headless_shell <defunct>
36 1 Z 1 headless_shell <defunct>
61 1 Z 1 headless_shell <defunct>
77 1 Z 1 headless_shell <defunct>
87 1 Z 1 headless_shell <defunct>
88 1 Z 1 headless_shell <defunct>
104 1 Z 1 headless_shell <defunct>
```
`cgroupPidsCurrent` after each cycle never returns to 7 — it is `7 +
5×cycles`. A monotonic ratchet, the same curve prod walked over 7 days.
### RED, driven to saturation (started before the Dockerfile was
touched)
```
baseline zombieCount=0 cgroupPidsCurrent=7 /1000
cycle=25 zombieCount=125 cgroupPidsCurrent=136 /1000
cycle=50 zombieCount=250 cgroupPidsCurrent=261 /1000
cycle=75 zombieCount=375 cgroupPidsCurrent=386 /1000
cycle=100 zombieCount=500 cgroupPidsCurrent=511 /1000
cycle=125 zombieCount=625 cgroupPidsCurrent=636 /1000
cycle=150 zombieCount=750 cgroupPidsCurrent=761 /1000 ← prod's ~757 reproduced
cycle=175 zombieCount=875 cgroupPidsCurrent=886 /1000
cycle=184 pre-kill zombieCount=915 cgroupPidsCurrent=993 /1000
cycle=185 pre-kill zombieCount=920 cgroupPidsCurrent=996 /1000
cycle=185 zombieCount=925 cgroupPidsCurrent=936 /1000
← no further output. Cycle 186 never completed.
```
The run **wedged** at cycle 185 with no output for >12 minutes:
`chromium.launch()` could not get PIDs and never returned — it did not
even surface Playwright's 30s launch timeout, which is exactly why prod
shows 300s feature timeouts and 1200s run aborts instead of a clean
launch error. `docker ps`: `Up 17 minutes (unhealthy)`.
```
$ docker exec red-sat cat /sys/fs/cgroup/pids.current /sys/fs/cgroup/pids.max
1000
1000
```
And the errno-11 signature this Dockerfile's own comment names — the
container could no longer `fork()` at all:
```
$ docker exec red-sat /bin/bash -c 'for i in 1 2 3; do cat /proc/uptime; done'
/bin/bash: fork: retry: Resource temporarily unavailable
/bin/bash: fork: retry: Resource temporarily unavailable
/bin/bash: fork: retry: Resource temporarily unavailable
/bin/bash: fork: retry: Resource temporarily unavailable
/bin/bash: fork: Resource temporarily unavailable
```
That is the prod outage, end to end, in a local container.
## GREEN (this PR's image, `pid1=tini`) — same repro, same argv, 230
cycles
```
baseline zombieCount=0 procCount=2 threadCount=8 cgroupPidsCurrent=8 cgroupPidsMax=1000 selfPid=7 pid1=tini
PID PPID S THR COMMAND
1 0 S 1 tini
7 1 R 7 node
cycle=1 pre-kill zombieCount=0 procCount=8 threadCount=76 cgroupPidsCurrent=76 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=1 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=50 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=100 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=150 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=200 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=230 pre-kill zombieCount=0 procCount=8 threadCount=77 cgroupPidsCurrent=77 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=230 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
final zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
SUMMARY cycles=230 zombiesBefore=0 zombiesAfter=0 pidsBefore=8 pidsAfter=12 launchFailure=none
```
`procCount` returning to **2** (tini + node) after every single cycle is
the reap actually happening: the browser's 6 processes appear pre-kill
(`procCount=8`) and are fully gone post-kill.
| | RED (`pid1=node`) | GREEN (`pid1=tini`) |
| --- | --- | --- |
| zombies after 20 cycles | 100 | **0** |
| zombies at cycle 185 | 925 | **0** |
| `pids.current` floor at cycle 185 | 936, peaking 996/1000 | **12** |
| reached cycle 230? | **no — wedged at 185** | yes |
| `chromium.launch()` | hangs; `fork: Resource temporarily unavailable`
| fine, `launchFailure=none` |
| leak per crashed browser | **+5** | **0** |
---
## Signals and non-zero exit still work
The prober's container must still self-restart on crash, so this was
proven, not assumed. `signal-child.mjs` installs the same
`process.once("SIGTERM", …)` shape `orchestrator.ts` uses; run on
**both** images:
```
--- GREEN (ENTRYPOINT tini, pid1=tini) ---
[showcase-harness:green MODE=term-graceful] exitCode=0
CHILD mode=term-graceful pid=7 ppid=1
CHILD ready — idling until signalled
CHILD received SIGTERM at pid=7 — draining ← tini FORWARDED it to node
[showcase-harness:green MODE=exit-42] exitCode=42 ← non-zero preserved
[showcase-harness:green MODE=abort] exitCode=134 ← 128+6 (SIGABRT), non-zero
--- RED (no ENTRYPOINT, pid1=node) — baseline ---
[showcase-harness:red MODE=term-graceful] exitCode=0
CHILD received SIGTERM at pid=1 — draining
[showcase-harness:red MODE=exit-42] exitCode=42
[showcase-harness:red MODE=abort] exitCode=133
```
- **SIGTERM reaches node, not just tini** — the handler fires at
`pid=7`, so `docker stop`'s SIGTERM was forwarded into the process that
owns the graceful drain. `orchestrator.ts`'s
`drainControlPlaneAndExit("SIGTERM")` path is intact.
- **Non-zero exits survive** — `exit(42)` → container `42`; a crash
still exits non-zero, so Railway still restarts. The HEALTHCHECK
restart-on-sustained-503 behaviour this Dockerfile calls "THE INTENDED
OUTCOME" is untouched.
- **tini is strictly *more* correct here**: note RED's abort → **133**
vs GREEN's → **134**. A process running as PID 1 in a namespace has
default signal actions *ignored* by the kernel, so node-as-PID-1 was
distorting its own fatal-signal exit code (128+SIGTRAP instead of
128+SIGABRT). Both non-zero, so no regression — but the fixed image
reports crashes accurately.
## Cross-arch check
Local images are arm64; prod builds amd64. `tini` availability was
verified on amd64 explicitly rather than assumed — same path the
build-time assertion checks:
```
Package: tini
Architecture: amd64
Version: 0.19.0-1+b3
-rwxr-xr-x 1 root root 27792 Jun 6 2025 /usr/bin/tini
tini version 0.19.0
```
## Pre-push checks
| Check | Result |
| --- | --- |
| `docker buildx build -f showcase/harness/Dockerfile` (== CI's build
check) | **PASS**, both images |
| `nx run @copilotkit/showcase-harness:typecheck` | **PASS** |
| `nx run @copilotkit/showcase-harness:test:ci` | 3677 passed, 18
skipped, 1 failed — **environmental**, see below |
| commitlint | **PASS** |
| lefthook `pre-commit` + `commit-msg` | **PASS** |
| `git status` | clean |
The single failure is `d0-gone-predicate.test.ts > resolves the real
generated registry.json…`, failing on `ENOENT …
showcase/shell/src/data/registry.json`. That file is **gitignored and
generated at build time** — this Dockerfile's own comment says so, which
is why the image generates it via `generate-registry.ts`. Proven
environmental rather than assumed: generated the file, re-ran that exact
spec, **11/11 passed**. A Dockerfile-only diff cannot affect a vitest
run.
No formatter/linter covers this file: lefthook's `lint-fix` glob is
`*.{js,jsx,ts,tsx,mjs,cjs,md,css,yml,yaml,html,vue,py}`, and there is no
hadolint anywhere in CI.
## Not in this PR
- **This does not fix prod by itself.** Prod is digest-pinned and still
running the leaking container; it needs the live mitigation restart
(owned by a separate worker — **prod was not touched here**) and then a
promote to pick this image up.
- **The ≥3-cell value test is not done and cannot be done from this PR**
— confirming red cells flip requires the prod restart plus a fleet
sweep. Note the ~20 `conversation-error` and ~16 `goto-error` cells are
almost certainly unrelated real bugs this will **not** fix; only the
`abort`/`feature-timeout`/`driver-error` population is in scope.
- **Alerting / scheduled recycle is deliberately out of scope** (one
concern per change). Still worth doing: the documented
`pool-unrecoverable` alarm demonstrably never fired at 1000/1000, and
`cgroupPidsCurrent/cgroupPidsMax > 0.75` would have given ~36h of
warning before the flip.
- Leak rate on the **graceful** `browser.close()` path was not measured
— irrelevant to the fix (tini reaps either way), but it means prod's
exact 757 can't be attributed to crashes alone.
---
# RE-VERIFICATION 2026-08-12 — rebased onto `origin/main`, red-green
re-run from scratch
The proof above was produced 2026-08-03 against a base that predates
`ced993447f` ("copy patches/ into harness build context"), which touches
THIS
Dockerfile. The branch has been rebased onto current `origin/main` and
the
whole red-green was re-run on freshly built images so the numbers
describe the
actual merge candidate.
**Both images built locally from this repo, same machine, same context,
~3 min apart:**
```
showcase-harness:red-main sha256:b86686e9a06d51f884d5e59e9d15d667fd298284efe7a15622ff3bf2575a3ffd created=2026-08-12T20:15:50Z (origin/main Dockerfile, /usr/bin/tini ABSENT, Entrypoint=["docker-entrypoint.sh"])
showcase-harness:green-tini sha256:d666b823f16bda07bf62d35f1d72ef099d13406a61e2f0cd51c99bf51eb14df4 created=2026-08-12T20:18:54Z (this branch, /usr/bin/tini present, Entrypoint=["/usr/bin/tini","--"])
```
Identical driver, identical argv, 40 cycles each, runs 18 seconds apart:
```
docker run --rm --pids-limit 1000 -e REPRO_CYCLES=40 -v <repro>:/repro:ro <image> \
/bin/bash -c "exec node /repro/zombie-repro.mjs"
```
## RED — `showcase-harness:red-main`, `pid1=node` (2026-08-12T20:19:28Z)
```
baseline zombieCount=0 procCount=1 threadCount=7 cgroupPidsCurrent=7 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=1 pre-kill zombieCount=0 procCount=7 threadCount=76 cgroupPidsCurrent=76 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=1 zombieCount=5 procCount=6 threadCount=16 cgroupPidsCurrent=16 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=2 pre-kill zombieCount=5 procCount=12 threadCount=83 cgroupPidsCurrent=83 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=2 zombieCount=10 procCount=11 threadCount=21 cgroupPidsCurrent=21 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=10 pre-kill zombieCount=45 procCount=52 threadCount=120 cgroupPidsCurrent=120 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=10 zombieCount=50 procCount=51 threadCount=61 cgroupPidsCurrent=61 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=20 pre-kill zombieCount=95 procCount=102 threadCount=174 cgroupPidsCurrent=174 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=20 zombieCount=100 procCount=101 threadCount=111 cgroupPidsCurrent=111 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=30 pre-kill zombieCount=145 procCount=152 threadCount=219 cgroupPidsCurrent=219 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=30 zombieCount=150 procCount=151 threadCount=161 cgroupPidsCurrent=161 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=40 pre-kill zombieCount=195 procCount=202 threadCount=271 cgroupPidsCurrent=271 cgroupPidsMax=1000 selfPid=1 pid1=node
cycle=40 zombieCount=200 procCount=201 threadCount=211 cgroupPidsCurrent=211 cgroupPidsMax=1000 selfPid=1 pid1=node
final zombieCount=200 procCount=201 threadCount=211 cgroupPidsCurrent=211 cgroupPidsMax=1000 selfPid=1 pid1=node
SUMMARY cycles=40 zombiesBefore=0 zombiesAfter=200 pidsBefore=7 pidsAfter=211 launchFailure=none
```
All 200 leaked processes are `ppid=1`, state `Z` (tail of the final
`/proc` table):
```
2610 1 Z 1 headless_shell <defunct>
2612 1 Z 1 headless_shell <defunct>
2646 1 Z 1 headless_shell <defunct>
2661 1 Z 1 headless_shell <defunct>
2662 1 Z 1 headless_shell <defunct>
2678 1 Z 1 headless_shell <defunct>
2680 1 Z 1 headless_shell <defunct>
2717 1 Z 1 headless_shell <defunct>
```
## GREEN — `showcase-harness:green-tini`, `pid1=tini`
(2026-08-12T20:19:46Z)
```
baseline zombieCount=0 procCount=2 threadCount=8 cgroupPidsCurrent=8 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=1 pre-kill zombieCount=0 procCount=8 threadCount=76 cgroupPidsCurrent=76 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=1 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=2 pre-kill zombieCount=0 procCount=8 threadCount=78 cgroupPidsCurrent=78 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=2 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=10 pre-kill zombieCount=0 procCount=8 threadCount=79 cgroupPidsCurrent=79 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=10 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=20 pre-kill zombieCount=0 procCount=8 threadCount=76 cgroupPidsCurrent=76 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=20 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=30 pre-kill zombieCount=0 procCount=8 threadCount=77 cgroupPidsCurrent=77 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=30 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=40 pre-kill zombieCount=0 procCount=8 threadCount=79 cgroupPidsCurrent=79 cgroupPidsMax=1000 selfPid=7 pid1=tini
cycle=40 zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
final zombieCount=0 procCount=2 threadCount=12 cgroupPidsCurrent=12 cgroupPidsMax=1000 selfPid=7 pid1=tini
SUMMARY cycles=40 zombiesBefore=0 zombiesAfter=0 pidsBefore=8 pidsAfter=12 launchFailure=none
```
`grep -c defunct` over the GREEN run: **0**. `procCount` returns to
**2**
(tini + node) after every cycle — the browser's 6 processes appear
pre-kill
(`procCount=8`, `threadCount` 76–79) and are fully gone post-kill.
| | RED (`pid1=node`) | GREEN (`pid1=tini`) |
| --- | --- | --- |
| image id | `b86686e9a06d` | `d666b823f16b` |
| cycles | 40 | 40 |
| zombies after | **200** | **0** |
| zombies per crashed browser | **+5** | **0** |
| `pids.current` before → after | 7 → **211** | 8 → **12** |
| `<defunct>` rows in final table | **200** | **0** |
| `launchFailure` | none | none |
## Signals / non-zero exit, both images, this run
```
[showcase-harness:RED MODE=term-graceful] exitCode=0
CHILD mode=term-graceful pid=1 ppid=0
CHILD received SIGTERM at pid=1 — draining
[showcase-harness:RED MODE=exit-42] exitCode=42
[showcase-harness:RED MODE=abort] exitCode=133
RED /proc/1/comm = bash
[showcase-harness:GREEN MODE=term-graceful] exitCode=0
CHILD mode=term-graceful pid=7 ppid=1
CHILD received SIGTERM at pid=7 — draining <- tini FORWARDED it to node
[showcase-harness:GREEN MODE=exit-42] exitCode=42
[showcase-harness:GREEN MODE=abort] exitCode=134
GREEN /proc/1/comm = tini, child self=7
```
SIGTERM reaches node (handler fires at `pid=7`), non-zero exits survive
(42 → 42; abort → 134 = 128+6), so Railway's restart-on-crash is intact.
## The prod leak, measured live the same day
`resource_snapshots` in prod PocketBase, immediately before the
mitigation
restart on the unfixed digest `sha256:80a0c5c2…`:
```
2026-08-12 20:10:00.674Z worker-7abf6 heartbeat pids=1000/1000 zombies=758 procs=778 threads=1000
2026-08-12 20:09:59.783Z worker-5a213 heartbeat pids=1000/1000 zombies=761 procs=781 threads=1000
2026-08-12 20:09:54.868Z worker-de14c heartbeat pids=837/1000 zombies=744 procs=753 threads=837
```
and immediately after it (`init` rows): `pids=164–169/1000 zombies=0
procs=16`.
A new zombie was already recorded at `2026-08-12 20:12:27.182Z
worker-de14c … zombies=1`.
## Cross-arch
Local images are arm64 (`/usr/bin/tini` 68592 bytes); prod builds amd64.
Measured on amd64 directly: `Package: tini / Architecture: amd64 /
Version: 0.19.0-1+b3`, `/usr/bin/tini` 27792 bytes, `tini version
0.19.0`;
package download 267,204 bytes, 749 kB on disk.
## Deploy note
`computePromoteClosure(["showcase-harness"])` = `pocketbase(0),
harness(1),
dashboard(1), harness-workers(1)` — the workers are in the promote
scope.
Prod `startCommand` is `null` and no repo consumer overrides the
entrypoint,
so Railway uses the image's ENTRYPOINT+CMD. **PR #5750 is CLOSED
unmerged**, so
promoting `harness-workers` resets
`multiRegionConfig.us-west2.numReplicas` to
1 and it must be re-asserted to 6 and read back after the promote.
---
# CHANGE 2 — nothing alarmed (worker PID-saturation alarm)
Commit `7f28e9bd39`.
## The gap
The measured prod condition on 2026-08-12 20:09–20:10Z was
`pids=1000/1000`,
`1000/1000`, `837/1000` across three workers, 320 `timeout after
600000ms`
abort rows, an `e2e-smoke` run 42 minutes into a 15-minute cadence with
3 jobs
still pending, prod d6 534/784 vs staging 744/788. **No alarm fired**,
and this
was the second occurrence (2026-08-03, 2026-08-10).
The documented `pool-unrecoverable` alarm did not fire *by
construction*: it is
raised only when `BrowserPool`'s self-heal circuit-breaker exhausts
`selfHealMaxHardRecoveries`. A worker that is PID-starved but still
heartbeating
never reaches that state — it launches browsers fine, it just cannot
fork the
process tree a probe needs.
The signal was already on the wire and read by nobody:
- `worker/registration.ts:192` writes `capacity_pids_current` /
`capacity_pids_max` onto the worker's `workers` row on its ~75s
heartbeat
(`DEFAULT_WORKER_HEARTBEAT_MS = 75_000`).
- `control-plane/fleet-health.ts` lists that entire roster every 15s
(`DEFAULT_FLEET_HEALTH_INTERVAL_MS`) — and its own comment on the row
type
said *"the capacity gauges are ignored here."*
- Nothing in non-test source compared `pidsCurrent` to `pidsMax`.
## Where the alarm goes, and why
**Control-plane, not worker.** `writers/status-writer.ts` documents
`fleet-cp`
as *"the only authoritative fleet writer; workers never write status
directly"*,
so a worker cannot raise a `system:` health key. `fleet-health` already
holds
the roster read this needs, on a 15s cadence, so the alarm costs zero
extra PB
traffic.
**Routed to `#oss-alerts`, not to the browser-pool webhook.**
`SLACK_WEBHOOK_BROWSER_POOL_UNRECOVERABLE` appears **nowhere in this
repo
outside its own definition** — `grep` finds it only in `orchestrator.ts`
— so it
is unset everywhere and that alarm has no receiver even when it does
fire.
`SLACK_WEBHOOK_OSS_ALERTS` is a real wired secret that the
family-silence
monitor and the D0-gone monitor already post through. Routing a new
alarm to a
known-unwired webhook would reproduce the exact gap this closes.
## Thresholds
- **Alarm at `pids.current / pids.max >= 0.75`.** At the observed leak
rate
(~1000 PIDs over ~7 days) that is roughly 36 hours of lead time — more
than a
working day. A busy worker's context budget is nowhere near 750 PIDs.
- **Latched per worker, clearing at 0.65.** The monitor ticks every 15s;
an
unlatched alarm would page ~8600 times across that 36-hour window. An
alarm
that fires constantly is worse than none, so `createFleetHealthMonitor`
now
**fails loud at construction** on a ratio pair with no hysteresis gap.
- **Evaluated before the online/stale branch**, because the failing
worker is
fully `online` — it heartbeats on time throughout.
- **Unmeasured gauges never alarm.** `registration.ts` maps the pool's
`-1`
sentinel to `null`; `pidUsageRatio` returns `null` for
null/absent/negative
current or non-positive max, and null is never read as zero.
---
## RED → GREEN (change 2)
Driver exercises the **real chain end-to-end**, faking only the OS
cgroup read
(`BrowserPoolOptions.cgroupPidsReader`, an existing injectable seam) and
the PB
transport. The column names are whatever the real registration writer
emits and
whatever the real fleet-health reader consumes, so a column drift fails
this
driver:
```
BrowserPool.budget() [real, probes/helpers/browser-pool.ts]
-> workerCapacityFromBudget [real, fleet/contracts.ts]
-> registerWorker() [real, fleet/worker/registration.ts]
-> workers row
-> createFleetHealthMonitor().checkOnce()
[real, fleet/control-plane/fleet-health.ts]
-> alarm sink
```
Identical command both sides; only the source differs:
```
./node_modules/.bin/tsx <scratch>/drive-alarm.mts
```
### RED — unmodified source
```
=== A1 SATURATED 900/1000 (ratio 0.90) ===
pool.budget() pids : 900/1000
workers row gauges : capacity_pids_current=900 capacity_pids_max=1000
fleet-health cycles run : 1
ALARM HOOK CALLS : 0
result.pidSaturated : <field does not exist>
error-level log events : <none>
=== A2 CEILING 1000/1000 (ratio 1.00) ===
pool.budget() pids : 1000/1000
workers row gauges : capacity_pids_current=1000 capacity_pids_max=1000
fleet-health cycles run : 1
ALARM HOOK CALLS : 0
result.pidSaturated : <field does not exist>
error-level log events : <none>
=== A6 LATCH 900/1000 over 5 cycles ===
fleet-health cycles run : 5
ALARM HOOK CALLS : 0
result.pidSaturated : <field does not exist>
error-level log events : <none>
```
The exact prod condition — a worker at the ceiling — produces **zero**
alarms.
### GREEN — with this change
```
=== A1 SATURATED 900/1000 (ratio 0.90) ===
pool.budget() pids : 900/1000
workers row gauges : capacity_pids_current=900 capacity_pids_max=1000
fleet-health cycles run : 1
ALARM HOOK CALLS : 1
result.pidSaturated : 1
error-level log events : ["fleet.health.worker-pid-saturated"]
ALARM PAYLOAD : {"workerId":"worker-drv","pidsCurrent":900,"pidsMax":1000,"ratio":0.9,"threshold":0.75,"lastHeartbeatAt":"2026-08-12T20:44:54.244Z","observedAt":"2026-08-12T20:44:54.244Z"}
=== A2 CEILING 1000/1000 (ratio 1.00) ===
ALARM HOOK CALLS : 1
result.pidSaturated : 1
error-level log events : ["fleet.health.worker-pid-saturated"]
ALARM PAYLOAD : {"workerId":"worker-drv","pidsCurrent":1000,"pidsMax":1000,"ratio":1,"threshold":0.75,"lastHeartbeatAt":"2026-08-12T20:44:54.253Z","observedAt":"2026-08-12T20:44:54.253Z"}
```
### The alarm must also stay QUIET — same run, same command
```
=== A3 QUIET 500/1000 (ratio 0.50) ===
ALARM HOOK CALLS : 0
result.pidSaturated : 0
error-level log events : <none>
=== A4 QUIET 740/1000 (ratio 0.74, just under) ===
ALARM HOOK CALLS : 0
result.pidSaturated : 0
error-level log events : <none>
=== A5 QUIET -1/-1 (cgroup unreadable) ===
pool.budget() pids : -1/-1
workers row gauges : capacity_pids_current=null capacity_pids_max=null
ALARM HOOK CALLS : 0
result.pidSaturated : 0
error-level log events : <none>
=== A6 LATCH 900/1000 over 5 cycles ===
fleet-health cycles run : 5
ALARM HOOK CALLS : 1 <- once, not five times
```
`A4` pins the boundary at 0.74. `A5` pins the off-Linux case: the `-1`
sentinel
lands as `null` on the row and stays silent rather than reading as `0`.
`A6`
pins the latch — 5 saturated cycles, 1 alarm.
---
# CHANGE 3 — saturated workers kept claiming (PID headroom claim gate)
Commit `63ef4081c2`.
## The defect
`fleet/worker/worker-loop.ts` had exactly one conditional in the claim
loop:
```ts
if (budget.available <= 0) { … idle … }
```
and `available` is free Playwright **browser-context** slots and nothing
else —
`browser-pool.ts`: `available: Math.max(0, this.maxContexts -
this.liveContextCount)`.
Context slots have no relationship to cgroup PIDs. A worker whose
container has
leaked to `pids=1000/1000` therefore still advertises **full** capacity,
keeps
winning claims, cannot fork the browser tree, and burns each job's
entire
600000ms lease before aborting. That is the mechanism behind the 320
abort rows.
## Blast radius — what happens when ALL workers are saturated
The gate deliberately reuses the **same decline-and-idle path** as the
existing
no-budget branch, which is what makes total saturation safe:
- the job is **never claimed**, so it is never dropped and never
requeued — it
stays `pending` for a healthy worker, or for this one after a redeploy;
- the loop then sleeps a full `pollIntervalMs`, so a fully-saturated
fleet
**idles at the poll cadence** instead of spinning;
- the queue **stalls visibly**: `probe_jobs` rows pile up pending, a
rising-edge
`fleet.worker.pid-headroom-exhausted` warn says why (warn → stderr →
Sentry;
steady state drops to debug so a multi-hour stall does not flood), and
change 2's `system:worker-pid-saturation` alarm fired at 0.75 **before**
this
gate engaged at 0.90.
A stalled, alarmed queue is recoverable by redeploy. Claim-fail-repeat
silently
burned the whole cadence.
## Why the gate is 0.90 and the alarm is 0.75
They do different jobs and must not be equal. 0.75 pages an operator
while the
worker is still perfectly able to run jobs; gating dispatch that early
would
convert a warning into ~36 hours of withheld fleet capacity. 0.90 is
where we
stop trusting the worker to fork at all — 100 free PIDs at the prod
ceiling.
**That reserve is chosen, not derived: the per-job PID cost of a
chromium
context tree is not measured**, hence the `WORKER_PID_CLAIM_GATE_RATIO`
override. The invariant that matters is ordering — the alarm always
fires before
capacity is withdrawn.
The gate is **inert** when the cgroup gauges are unreadable
(`pidUsageRatio` →
`null`: off-Linux, every macOS dev box), so local workers claim exactly
as
before.
---
## RED → GREEN (change 3)
Driver runs the **real `startWorkerLoop`** against the **real
`BrowserPool`**
budget path, faking only the cgroup read and the queue transport. The
queue fake
holds a real `pending` list, so *"was the job dropped / does it stay
pending"* is
directly observed, not inferred. `budget()` calls are counted as a
loop-iteration proxy for the spin check.
Identical command both sides:
```
./node_modules/.bin/tsx <scratch>/drive-gate.mts
```
### RED — unmodified source
```
=== B1 SATURATED 1 worker @ 1000/1000, 3 jobs pending ===
workers=1 cgroup pids=1000/1000 contexts available=24
window : 600ms @ poll 50ms
queue.claimNext CALLS : 15
jobs CLAIMED : 3 ["job-1","job-2","job-3"]
jobs STILL PENDING : 0 []
jobs REPORTED : 3
loop iterations : 15
=== B2 ALL-SATURATED 3 workers @ 1000/1000, 3 jobs pending ===
workers=3 cgroup pids=1000/1000 contexts available=24
queue.claimNext CALLS : 39
jobs CLAIMED : 3 ["job-1","job-2","job-3"]
jobs STILL PENDING : 0 []
jobs REPORTED : 3
loop iterations : 39
```
A worker at the PID ceiling claims **all three jobs**. In prod each of
those
becomes a 600000ms abort.
### GREEN — with this change
```
=== B1 SATURATED 1 worker @ 1000/1000, 3 jobs pending ===
workers=1 cgroup pids=1000/1000 contexts available=24
window : 600ms @ poll 50ms
queue.claimNext CALLS : 0
jobs CLAIMED : 0 []
jobs STILL PENDING : 3 ["job-1","job-2","job-3"]
jobs REPORTED : 0
loop iterations : 12 (spin check: bounded by window/poll)
```
`claimNext` is never called; all three jobs are **still pending** — not
dropped,
not requeued.
### ALL workers saturated — the blast-radius case
```
=== B2 ALL-SATURATED 3 workers @ 1000/1000, 3 jobs pending ===
workers=3 cgroup pids=1000/1000 contexts available=24
window : 600ms @ poll 50ms
queue.claimNext CALLS : 0
jobs CLAIMED : 0 []
jobs STILL PENDING : 3 ["job-1","job-2","job-3"]
jobs REPORTED : 0
loop iterations : 36 (spin check: bounded by window/poll)
```
Zero claims, **3 of 3 jobs still pending**, and 36 iterations across 3
workers
over a 600ms window at a 50ms poll — i.e. 12 per worker, the poll
cadence. No
spin, no loss.
### The gate must NOT engage below threshold — same run, same command
```
=== B3 HEALTHY 1 worker @ 100/1000, 3 jobs pending ===
queue.claimNext CALLS : 15
jobs CLAIMED : 3 jobs STILL PENDING : 0 jobs REPORTED : 3
=== B4 UNDER-GATE 1 worker @ 890/1000, 3 jobs pending ===
queue.claimNext CALLS : 15
jobs CLAIMED : 3 jobs STILL PENDING : 0 jobs REPORTED : 3
=== B5 UNMEASURED 1 worker @ -1/-1, 3 jobs pending ===
queue.claimNext CALLS : 15
jobs CLAIMED : 3 jobs STILL PENDING : 0 jobs REPORTED : 3
```
`B4` is the load-bearing one: 0.89 is **above** change 2's 0.75 alarm
and below
this gate. The operator has been paged and the worker keeps working —
which is
the whole reason the two thresholds differ. `B5` pins the off-Linux
case.
---
# Tests and gates (changes 2 and 3)
`showcase/harness`, on the pushed head `63ef4081c2`:
| Gate | Command | Result |
|---|---|---|
| Typecheck | `npx tsc --noEmit -p tsconfig.json` | exit 0 |
| Lint | `npx oxlint src/fleet src/orchestrator.ts` | 0 errors; warning
count unchanged vs base (2 on `fleet-health.ts` before and after) |
| Format | `npx oxfmt --write` | clean |
| Unit suite | `vitest run` | **3696 passed, 18 skipped, 1 failed** —
175/178 files pass |
The single failure is `src/probes/frontend-matrix.test.ts` —
**pre-existing**.
Verified by stashing this branch's changes and re-running it on the
unmodified
tree, where it fails identically (`1 failed | 5 passed`).
14 tests are new (`3682 → 3696`).
**Mutation-tested for vacuity.** With the four source files reverted to
`e66fb1af13` and the new test files kept, **11 of the new tests fail**:
```
Test Files 2 failed (2)
Tests 11 failed | 80 passed (91)
FAIL fleet-health.test.ts > PID saturation > alarms on an ONLINE worker whose PID gauges cross the threshold
FAIL fleet-health.test.ts > PID saturation > stays silent below the threshold, including just under it
FAIL fleet-health.test.ts > PID saturation > stays silent when the gauges are UNMEASURED (null / unbounded max)
FAIL fleet-health.test.ts > PID saturation > is EDGE-triggered: a sustained saturation alarms once, not every cycle
FAIL fleet-health.test.ts > PID saturation > re-arms only after the worker drops below the CLEAR ratio
FAIL fleet-health.test.ts > PID saturation > never lets a throwing alarm hook abort the cycle
FAIL fleet-health.test.ts > PID saturation > fails loud on a ratio config that would alarm every cycle
FAIL worker-loop.test.ts > PID headroom gate > declines to claim at the PID ceiling — the job is never claimed, so it stays pending
FAIL worker-loop.test.ts > PID headroom gate > declines at the gate ratio boundary (0.90)
FAIL worker-loop.test.ts > PID headroom gate > honours an injected gate ratio
FAIL fleet-health.test.ts > checkOnce > never throws when the roster read fails — returns an empty cycle
```
The three that pass on both sides are the sub-threshold controls (`still
claims
just BELOW the gate`, `claims normally on a healthy worker`, `stays
INERT when
the cgroup gauges are unreadable`) — they are supposed to be unchanged
by the
fix, and they are.
# New env knobs (all optional, all defaulted)
| Var | Default | Effect |
|---|---|---|
| `WORKER_PID_SATURATION_RATIO` | `0.75` | Control-plane alarm
threshold. |
| `WORKER_PID_SATURATION_CLEAR_RATIO` | `0.65` | Alarm latch hysteresis
clear. |
| `WORKER_PID_CLAIM_GATE_RATIO` | `0.90` | Worker claim-gate threshold.
|
An out-of-range override on the two alarm ratios falls back to the
default
rather than tripping the fail-loud guard and taking the control-plane
down over
a typo'd env var.
# Not verified
- Neither change 2 nor change 3 has been exercised against **live prod
or
staging PocketBase**; both red-greens are local, on the real modules
with the
cgroup read and PB/queue transport injected.
- **No Slack post was sent.** `SLACK_WEBHOOK_OSS_ALERTS` was not set in
the
driver, so the send leg was not exercised end-to-end — only that its
failure
is swallowed without costing the durable status row.
- The per-job PID cost of a chromium context tree is **not measured**,
so the
0.90 gate's 100-PID reserve is a chosen value.
|
||
|
Mike Ryan
|
528dea6483 |
fix(react-core): ship a single v2 context instance (#6440)
## Problem
`@copilotkit/react-core` ships **two independent copies** of the v2
context module, so `useLicenseContext` imported from
`@copilotkit/react-core/v2/context` returns the default forever —
`status: null` even when `/info` reports `licenseStatus: "valid"`.
Reported downstream as a chat-history sidebar that never loads, because
`useThreads` is gated on license status.
`src/v2/context.ts` is compiled by two separate tsdown builds:
| Build | Output | Contains |
|---|---|---|
| `entry: ["src/index.tsx", "src/v2/index.ts"]` | `dist/` shared chunk |
inlined copy **A** |
| `entry: {context: "src/v2/context.ts"}` | `dist/v2/context.*` |
standalone copy **B** |
There is no import edge between them, so `createContext()` runs twice.
`CopilotKitProvider` lives in the shared chunk and publishes to **A**;
`@copilotkit/react-core/v2/context` exports **B**, which nothing ever
provides.
Verified against the published 1.66.4 artifact:
```
$ grep -n "createContext" dist/v2/context.mjs
104:const CopilotKitContext = createContext(null);
124:const LicenseContext = createContext({
$ grep -n "createContext" dist/copilotkit-nRjRp2_5.mjs # inside //#region src/v2/context.ts
1522:const CopilotKitContext = createContext(null);
1544:const LicenseContext = createContext({
$ grep -E '^import .*from "[^"]*context[^"]*"' dist/copilotkit-nRjRp2_5.mjs
# (empty — no import edge)
```
`CopilotKitContext` is duplicated identically, so `useCopilotKit`
imported from that subpath throws `"useCopilotKit must be used within
CopilotKitProvider"`. The subpath was effectively unusable for web
consumers; license was just the *silent* failure mode.
**Compounding defect:** `src/v2/providers/index.ts` enumerates its
exports by name and omits `useLicenseContext` (even though
`CopilotKitProvider.tsx:19` re-exports it). So the live copy had **no
public import path at all**, leaving consumers with no correct
alternative.
Not a 1.66.x regression — broken since
|
||
|
Jordan Ritter
|
63ef4081c2 |
fix(showcase/harness): stop a PID-saturated worker from claiming jobs
worker-loop's claim gate had exactly one conditional -- `if (budget.available <= 0)` -- and `available` is free Playwright browser-CONTEXT slots (maxContexts - liveContextCount), nothing else. A worker whose container has leaked PIDs to the cgroup ceiling therefore still advertises full capacity and keeps winning claims it cannot possibly run: the driver cannot fork, so each job burns its entire 600000ms lease and lands as an abort. Measured 2026-08-10: workers at pids=1000/1000 with 758-761 zombies, 320 `timeout after 600000ms` rows. Decline the claim above 0.90 of pids.max. Deliberately far above the control-plane's 0.75 saturation ALARM: the two thresholds do different jobs. 0.75 pages an operator while the worker is still perfectly able to run jobs; gating dispatch that early would convert a warning into ~36h of withheld fleet capacity. 0.90 leaves 100 free PIDs at the prod ceiling -- a chosen reserve, not a derived one (the per-job PID cost of a chromium context tree is not measured), hence the env override. The ordering that matters is that the alarm always fires before capacity is withdrawn. BLAST RADIUS -- reuses the SAME decline-and-idle path as the existing no-budget branch, which is what makes an all-workers-saturated fleet safe: the job is never claimed, so it is never dropped and never requeued; it simply stays `pending`. The loop then sleeps a full pollIntervalMs, so a fully-saturated fleet idles at the poll cadence instead of spinning. The queue stalls VISIBLY -- pending rows pile up, the rising-edge warn says why, and the control-plane's system:worker-pid-saturation alarm fired at 0.75 before this gate engaged at 0.90. A stalled, alarmed queue is recoverable by redeploy; claim-fail-repeat silently burned the whole cadence. The gate is INERT when the cgroup gauges are unreadable (pidUsageRatio -> null: off-Linux, every macOS dev box), so local workers claim exactly as before. |
||
|
Jordan Ritter
|
7f28e9bd39 |
fix(showcase/harness): alarm when a worker's cgroup PIDs saturate
Prod harness-workers replicas reach the platform-fixed pids.max=1000
ceiling roughly every 7 days and NOTHING alarmed. Measured 2026-08-10
20:09-20:10Z: pids=1000/1000 zombies=758, 1000/1000 zombies=761,
837/1000 zombies=744, with 320 `timeout after 600000ms` abort rows and
an e2e-smoke run 42 minutes into a 15-minute cadence with 3 jobs still
pending. The documented pool-unrecoverable alarm did not fire: it only
trips when the self-heal breaker gives up, which a PID-starved but
still-heartbeating worker never reaches.
The signal was already on the wire and read by nobody. The worker's
~75s heartbeat writes capacity_pids_current / capacity_pids_max onto
its `workers` row (worker/registration.ts), and fleet-health lists that
whole roster every 15s — its own comment said "the capacity gauges are
ignored here". Nothing in non-test source compared them.
Alarm control-plane-side rather than worker-side, because status-writer
documents `fleet-cp` as the only authoritative fleet writer ("workers
never write status directly"), and because fleet-health already holds
the roster read this needs.
- fleet-health raises a rising-edge alarm at pids.current/pids.max >=
0.75 (~36h of lead time at the observed leak rate), evaluated BEFORE
the online/stale branch since the failing worker is fully ONLINE.
- Latched per worker with a 0.65 hysteresis clear: the monitor ticks
every 15s, so an unlatched alarm would page ~8600 times across the
lead-time window. Construction fails loud on a ratio pair with no gap.
- Unmeasured gauges (null off-Linux, unbounded pids.max) never alarm --
pidUsageRatio returns null and null is never read as zero.
- Routed to a system:worker-pid-saturation status row plus #oss-alerts
via SLACK_WEBHOOK_OSS_ALERTS, the target family-silence and the
D0-gone monitor already post to. Deliberately NOT
SLACK_WEBHOOK_BROWSER_POOL_UNRECOVERABLE, which appears nowhere in
this repo outside its own definition and is unset everywhere -- an
alarm nobody receives is the gap being closed.
|
||
|
Jordan Ritter
|
e66fb1af13 |
fix(showcase/harness): reap orphaned chromium children with tini as PID 1
harness-workers leaked one cgroup PID slot per orphaned chromium grandchild. Node as PID 1 only waitpid()s processes it spawned, so every browser crash stranded ~5 <defunct> renderers permanently. Prod climbed to pids.current=1000/1000 with zombieCount=757 over 6d22h uptime, after which no browser could launch and ~295 d6 cells went abort fleet-wide. Install tini in the existing playwright apt layer and run it as PID 1 via exec-form ENTRYPOINT. No -g, so signal delivery to node is unchanged and orchestrator.ts's SIGTERM drain still runs; tini propagates the child's exit status so a crash still exits non-zero and Railway still restarts. |
||
|
Sam Julien
|
be3485d8f3 | fix(ci): narrow AEO synthetic rollout | ||
|
Sam Julien
|
a01b9f1034 | ci: monitor public AEO surfaces | ||
|
Sam Julien
|
25ce46ba5f | fix(docs): simplify AEO surface contract | ||
|
Sam Julien
|
d771e54993 | fix(docs): include shared contract in Turbopack root | ||
|
Sam Julien
|
9f2fb8e43b | docs: define public AEO surface contract | ||
|
Maxim
|
fb2aedb0e0 |
docs(reskinnable-demo): de-narrate the reskin skill and its guard comments
Second pass of the history sweep. The first cleaned CLAUDE.md and README.md; this finishes the reskin skill and the in-code comments that still recounted who hit a defect, when it was found, and how long it survived. Every rule, gate, command and checklist item is kept. What went is the narration around them — "it named only the first four skins for two releases", "caught by `eslint --print-config`, by hand, once", "drifted out of true three review rounds running", "it shipped that way once", "one CR pass found sixteen of them live", "measured in logistics", "each raised after the fact". Where a cut would have left a rule reading as arbitrary, the mechanism is restated in one present-tense clause instead: a hand-copied list rots silently and nothing fails when it is stale; flat-config `rules` are REPLACED, not merged, so a block silently drops every selector it does not restate; a schema leak is routinely line-wrapped, so a source-text guard never matches. Two stale cross-references fixed while in there: failure-modes.md quoted a CLAUDE.md sentence that the first pass removed, and claimed the roster-docs test header lists "two" known instances outside its doc set (it lists one). Skill impact, per the standing rule in CLAUDE.md: this change IS the skill, and it is prose-only — no contract field, link builder, lint rule, gate, beat mechanism, skin identity or file path changed, so no template or verification step needed a matching edit. The two doc properties `skin-roster-docs.test.ts` depends on were preserved deliberately: templates.md keeps "the six shipped skins" ahead of its brace glob, and SKILL.md keeps its "Six are registered —" id list, since both are what arm the brace-glob and valid-id-list rules. Gates: `pnpm lint`, `pnpm exec tsc --noEmit`, `pnpm test:unit` (197 files / 2227 tests) and `pnpm build` all green. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Maxim
|
6740613fb8 |
docs(reskinnable-demo): de-narrate in-code comments that recounted build history
Same cutting rule, applied only where a comment narrated what a past slot or
agent did rather than explaining the code: "a later slot owns that file", "as of
the beat-parity work", "two parallel agents each hand-edited this paragraph",
"this very paragraph did it once", "learned the hard way in the banking skin".
Every WHY stays; several are restated as present-tense properties (do not
reintroduce a client ticker, do not add a second seed of AV1423).
Four of these were also FACTUALLY STALE and are now correct:
- airline/attach-hotel-confirmation.ts claimed no pill carries
HOTEL_CONFIRMATION_MESSAGE; suggestions.ts has carried it since beat 3d landed.
- keel/attach-bulletin.ts said the same of BULLETIN_MESSAGE.
- keel/tools-replay-safety.test.ts said keel was not yet in the
statusKeyedTerminalRender glob; it is.
- airline/data/{store,trip-types,types}.ts described use-data.ts / useAirlineData
as "still live and still driving the trip, loyalty and disruption pages"; the
hook is deleted and the ledger is the only substrate.
skin-roster-docs.test.ts: comments only. No fixture entry, exemption or rule was
touched — the "legitimate phrasings" list still pins the numeral+adjective
discriminator, and the header still documents both false-positive shapes.
Reskin skill impact: checked — no rule, path or symbol the skill references
changed, so no skill edit is required beyond the prose pass in
|
||
|
Sam Julien
|
e9759825b6 | test(skills): simplify public contract guard | ||
|
Sam Julien
|
7bba4446ff | test(skills): narrow public API contract checks | ||
|
Sam Julien
|
4f258e409a | test(skills): add manifest-backed public evals | ||
|
Guido Vizoso
|
224dd10f07 |
feat(reskinnable-demo): add bookstore b2b skin (#6467)
## What does this PR do?
Adds `bookstore` ("Bookstore"), the reskinnable-demo's seventh skin, and
implements
demo beat 5 — **the book club run** — on it.
**The skin.** A customer-facing storefront on an in-memory substrate: a
25-book
catalog, generated covers, browse/book/cart pages, per-shopper cart and
orders
mirrored to `localStorage`, and per-shopper Intelligence identity. It is
the only
skin facing an end customer rather than an internal operator, which is
what makes
it a different proof from `commerce` (a merchant ops console) rather
than a second
copy of it.
**Beat 5.** The agent is *seeded* with an operational procedure it must
recall and
replay, rather than being told the steps in its prompt:
1. `addToCart` the club pick's hardcover
2. `swapEdition` to the paperback (same `workId`, different edition)
3. `applyPromoCode` with the club's code
4. `setDeliveryBy` the club's next meeting date
Three sibling tools — `addToWishlist`, `setReminder`, `applyStoreCredit`
— are
registered and genuinely work. Declining them is the beat's actual test:
a
distractor that errors proves nothing.
The promo code appears nowhere in `tools.tsx` (0 occurrences), so the
agent can
only obtain it from the club readable. If the code gets applied, the
readable was
genuinely read.
### Details worth a reviewer's attention
- **`localCalendarDay()`** (`data/club.ts`) re-anchors the caller's
*local*
calendar day onto UTC midnight. `nextMeetingISO` is UTC-only by design;
feeding
it a wall-clock `new Date()` gives a presenter west of UTC the wrong
meeting
day — at `2026-12-31T23:00-08:00` it skips a **full week**. Both the
club
readable and `setDeliveryBy`'s past-check use it, so the date the
readable
advertises can never be refused as past (proven over 23,107 weekday×date
combinations).
- **The discount renders as two rows, not one.** `discountCents` is a
single
scalar, so one row labelled with the club would render club-plus-credit
under
the club's name and misattribute the credit — reachable exactly when the
store-credit distractor misfires. `splitCartDiscount` recomputes the
club-only
part and takes credit as the remainder; `clubPart + creditPart ===
discountCents`
holds in every case, including a credit exceeding the subtotal.
- **All four totals agree to the cent.** The agent's cart readable, the
cart page,
the checkout form and the persisted order record all compute the same
figure.
Two of those were priced *without* the discount before this PR, so the
agent
would have announced the pre-discount total at the moment the procedure
tells it
to "confirm the new total in bold".
- **Every write tool uses `[]` deps and `dataRef.current`.**
`useFrontendTool`
keys its registration effect on `JSON.stringify(deps)`, so a callback in
a dep
array stringifies to a constant and permanently pins the pre-hydration
store.
This bug shipped once in this skin already.
- **Renders key off `result`, never `status`**, so a reopened thread
replays
correctly.
- **The empty-recall branch in `agent.ts` clause 7 came from a live
failure.** With
the memory unseeded, the agent correctly reported the miss and then
offered to
*learn* the procedure — which the clause already forbade, and which is
beat 6's
moment. It now says so plainly and stops, without guessing the pick,
edition,
code or date from the catalog or cart. An invented answer that looks
right is
worse than an honest failure: on stage the two are indistinguishable.
### Does this make anything in `.claude/skills/reskin/` wrong or
misleading?
**Yes — and it's updated in this PR.** Both `templates.md` (`SAVED
PROCEDURE`) and
`demo-beats.md` (beat 5's spec list) documented only the happy path, so
a new skin
author would ship the empty-recall gap above. Both now require the
branch and cite
the worked example.
`CLAUDE.md` had also drifted in five places, each phrased differently
enough that
keyword searches kept missing one; an exhaustive audit of all 36
bookstore claims
in the file closed it. `§ Commands` now also records that `pnpm lint` is
ESLint
only while lefthook's `pre-commit` runs `oxlint --fix` + `oxfmt --write`
and
re-stages — the two disagree on type-import style, so a contributor can
satisfy
the documented gate and be silently rewritten.
### Known gaps, deliberately not addressed here
- **`banking`'s beat-5 clause has the same empty-recall gap**, and its
only such
instruction affirmatively calls `offerWorkflowRecording`. Correctly
scoped to its
teach path, but it leaves a pattern pointing the wrong way for beat 5.
Left for a
separate change, since banking is the reference demo.
- Both empty-recall docs distinguish the beats by *beat number*, which
the agent
never sees. Banking makes it runtime-detectable via `overLimit: true`.
Bookstore
has no beat 6 so nothing is broken today; the first skin shipping both
will hit it.
- `beats 2, 4 and 5` exist only in Intelligence mode. Without it the
skin still
runs, but three of its four headline claims disappear — the runtime
warning at the
top of `suggestions.ts` says so.
## Related PRs and Issues
- (none)
## Checklist
- [x] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation
- [x] "Allow edits by maintainers" is checked
|
||
|
Maxim
|
085c92e6cc |
docs(reskinnable-demo): cut historical prose from the reskin skill
Same rule as the previous commit, applied to SKILL.md, demo-beats.md, failure-modes.md and templates.md: keep the rule and the mechanism that makes it a rule, drop who hit it, when, how it was found and how long it survived. Largest removals: the "there is no longer a partial skin to warn you off" retrospective closing demo-beats.md, the CR-pass provenance header on failure-modes.md, "this paragraph has now been wrong twice" under the pill count, the three-copies-of-the-staging-chain incident report, and the count-of-selectors paragraph that recorded its own rot. Past-tense incident illustrations were restated in the present tense rather than deleted, so every worked example still names its file. Two stale claims fixed while passing through: templates.md § tools.tsx said only banking, people and commerce key renders off `result` (every skin does), and SKILL.md described `--nw-nav-inset-*` as recently retired rather than simply absent. Reskin skill impact: this IS the skill; the app docs move in the commit before this one and the two are consistent. |
||
|
Maxim
|
050814b433 |
docs(reskinnable-demo): cut historical prose from CLAUDE.md and README
Record the current state and the forward-looking instruction; drop the retrospective narration around it. Removed: the "MIGRATION not a split" substrate-history block, the three worked examples of past skill staleness under the standing rule, "was the FIRST skin"/"the second skin built"/"the retrofits" framing, "it did rot for two releases", the glass-engine replacement note, and the TS2352-in-a-green-slot anecdote. Every rule, gate, command, derivation and mechanism is kept; where a cut would have left a rule reading as arbitrary the reason is restated in the present tense (a nested thread rail compounds the assistant's floor; a client ticker is a second clock; a template teaching a removed pattern still compiles). Reskin skill impact: checked — the skill is edited in the following commit for the same reason, so the two stay in step. |
||
|
Guido Vizoso
|
2ab26d66d5 |
docs: beat-5 coverage and reskin skill
Bring the app's own documentation back in line with what the skin now does, and record in the authoring skill the failure that a live run exposed. CLAUDE.md had drifted in five separate places, each phrased differently enough that keyword searches kept missing one: the seed catalog size, the count of SKIPPED beat-map rows (twice), the beat-matrix cell for stored-procedure replay, the intro paragraph's list of skipped beats, and a claim that the seed file seeds "no procedure at all". An exhaustive audit of every bookstore claim in the file — 36 of them — is what finally closed it. Also documents a repo-level trap in § Commands: pnpm lint is ESLint only, while lefthook's pre-commit additionally runs oxlint --fix and oxfmt --write over staged files and re-stages the result. The two disagree (oxlint enforces prefer-top-level type imports; ESLint does not), so a contributor can satisfy the documented gate and still be silently rewritten at commit time. This already misled a reviewer into filing a finding asking for the exact thing the hook auto-reverts. The reskin skill gains an empty-recall requirement for the stored-procedure beat, in both templates.md and demo-beats.md, which previously documented only the happy path. A skin author following either would ship the gap this skin shipped: with the memory empty, the agent reported the miss correctly and then offered to learn the procedure — beat 6's moment, arriving as an improvised fallback. Both now require saying so and stopping, with no guessing and no teach-offer, and cite the worked example. Note the reference skin has the same gap: banking's beat-5 clause has no empty-recall branch, and its only such instruction affirmatively calls offerWorkflowRecording. Correctly scoped to its teach path, but it leaves a pattern pointing the wrong way for beat 5. Left for a separate change. |
||
|
Guido Vizoso
|
79cbc7ef84 |
feat(bookstore): seeded procedure, prompt and pill
The demo half of beat 5: a procedure the agent already knows, an instruction to recall rather than improvise it, and a pill so the presenter never types. - intelligence/seed-memories.ts: a kind "operational", scope "user" memory naming addToCart -> swapEdition -> applyPromoCode -> setDeliveryBy in order, and explicitly excluding the three distractors. The procedure is SEEDED, not taught — it is recalled. Scope is "user" and never "project", which would return the memory for every user of a shared instance. - agent.ts: clause 7 calls recall_memory FIRST, runs all four steps in order without confirmation, and states that finding the club is not running the procedure — reporting the pick, code or date and stopping is the failure mode, not a partial success. It scopes openCheckout out, so the run ends with a filled but unpaid cart, and refuses the teach-offer: this is a recall, not a teaching moment. - The empty-recall branch exists because a live run went off-script the moment the store was empty. With nothing recalled the model said so correctly and then offered to LEARN the procedure — which the clause already forbade, and which is beat 6's moment. It now says plainly that nothing was found and stops, without guessing the pick, edition, code or date from the catalog or cart: an invented answer that looks right is worse than an honest failure, because on stage the two are indistinguishable. The four tool names are frozen string literals shared by the prompt and the seed, and no test reads either, so renaming one breaks the beat with a green suite. A drift guard is the next commit's concern, not this one's. |
||
|
Guido Vizoso
|
f1913bf485 |
feat(bookstore): cart discount and delivery UI
Price the cart through the three-argument cartTotals and show what the club run actually did to it. The discount is rendered as up to TWO rows, not one. discountCents is a single scalar, so a single row labelled with the club would render club-plus-credit under the club's name and silently misattribute the credit — and that combined case is reachable exactly when the applyStoreCredit distractor misfires, the most scrutinised second of the demo. splitCartDiscount recomputes the club-only discount and takes the credit as the remainder, so both parts are attributed honestly and clubPart + creditPart === discountCents holds for every case, including a credit that exceeds the subtotal (the club keeps its full percentage; credit takes only the applied remainder). Also adds the delivery-by badge, wishlist and reminder counts, and the same figures on the page readable so "what's on my screen" agrees with what the agent says. card_last4 remains the only card datum that leaves the checkout. |
||
|
Guido Vizoso
|
4ee549a616 |
feat(bookstore): the book club mechanism
Everything the saved book-club procedure needs in order to run: the club constant and its computed next-meeting date, the edition pair the swap moves between, discount-aware pricing, the six store writes, and the twelve registered frontend tools. - data/club.ts: BOOKSTORE_CLUB (pick, promo code, 15%, meeting weekday), nextMeetingDate/nextMeetingISO (UTC-only by design) and localCalendarDay, which re-anchors the caller's LOCAL calendar day onto UTC midnight. Without it a presenter west of UTC demoing on a Thursday evening gets next Thursday: at 2026-12-31T23:00-08:00 the naive path skips a full week. - data/seed.ts: a 25th book, the club pick's paperback, sharing workId "trust" with the hardcover so swapEdition has a real work to move within. - data/query.ts: cartTotals gains an optional pricing object and returns subtotalCents/discountCents alongside totalCents, which stays the POST-discount amount charged. Inputs are sanitised so 0 <= discountCents <= subtotalCents holds for any input, including a non-finite credit or discountPercent. - data/use-data.ts: promoCode, deliverBy, storeCreditCents, wishlist and reminders persist under one extras key with a field-by-field validator; six writes returning WriteResult; placeOrder prices through cartTotals and consumes all three sticky fields. swapEdition merges into an existing target line rather than duplicating a bookId, and setDeliveryBy's past-check reads the local calendar day so the club's own date is never refused. - tools.tsx: the club readable (the only agent-reachable source of the promo code), the three procedure writes, the three distractors that genuinely work, and discount-aware pricing in both the cart readable and openCheckout's render so the total the agent speaks matches the cart page, the checkout form and the order record. Every registration uses [] deps and reaches the store through dataRef: useFrontendTool keys its effect on JSON.stringify(deps), so a callback in a dep array stringifies to a constant and pins the pre-hydration store. Reskin skill: checked, no impact — skin-internal data, store and tool wiring; no Skin contract field, registration, routing or gate changed. |
||
|
Guido Vizoso
|
1c4003d2a3 | fix(bookstore): seed the default memory bucket and stop claiming per-shopper isolation |