## Why
A managed Channel that loses its gateway link is invisible outside the
host process. The only trace is the injected `log` seam, wired to
`logger.warn` — which for a self-hosted or Railway-hosted runtime
reaches nobody who can act on it.
## What
- `oss.runtime.channel_session_dropped` — carries the cause already
computed for the log line (`reason`, and the transport `code` when the
transport named one).
- `oss.runtime.channel_session_recovered` — carries `downForMs`, so
outage duration is measurable rather than inferred from log timestamps.
- The drop cause is now replayed on every "still down" reminder. In prod
those lines read `still down after 233134s; Phoenix is retrying` with
**no cause at all**, so an operator had to scroll back to the first line
— 15 minutes earlier, or hours, given the exponential backoff — to learn
it was an HTTP 502.
## Deliberate choices
- **No Channel name in the events.** It is a customer-chosen identifier
that can carry business meaning, so it stays out of anonymous OSS
telemetry. No message content or credentials either. Per-channel
aggregate counts still work without it.
- **An `online` transition with no preceding drop emits nothing** — a
session can report online without having dropped, and that is not a
recovery.
- **Capture is fire-and-forget with failures swallowed**, the same
contract `fireInstanceCreatedTelemetry` uses. The `try` also covers a
`capture` that throws synchronously. Telemetry must never break a live
session.
- The `gave_up` line's OSS-670 wording is untouched — it deliberately
says retries continue, and that is now accurate.
## Testing
Three tests added to `channel-manager-reconnect.test.ts`, each watched
fail first: the dropped event with its cause, the recovered event with a
positive duration, and the no-bogus-recovery guard. A fourth pins the
cause on the repeat log line.
```
✓ src/v2/runtime/core/__tests__/channel-manager-reconnect.test.ts (11 tests)
Tests 11 passed (11)
```
Wider run: 106 tests pass across `core/__tests__` and `telemetry`. Two
notes, both verified pre-existing by stashing this branch's changes and
re-running:
- `channel-manager-recovery.test.ts` fails to *load* in my worktree
(`Cannot find package '@copilotkit/channels-slack/render'`) — a
subpath-export resolution artifact of a worktree with symlinked
`node_modules`, identical with these changes stashed.
- `tsc --noEmit` reports 11 errors, the same 11 before and after this
change, none in the files touched here.
`oxfmt` and `oxlint` clean on all three files. Lefthook was bypassed on
the commit because of the same worktree `node_modules` symlinking; I ran
both tools manually over exactly the staged files instead.
Refs OSS-825.
normalizeMessageContent maps array content and handles only "text" and
"binary" parts. Any other part -- the {"type": "image", ...} case from
#1748 -- maps to "" and is filtered out with no signal at all, so an
agent emitting structured content sees its output silently vanish.
Carrying those parts through needs an AssistantMessage schema change and
is tracked separately in OSS-767. This makes the current drop visible in
the meantime, once per unrecognised part type so streaming does not flood
the log.
Refs #1748
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`isStreamConsumed` checked `req.complete` and the private
`_readableState.ended`/`endEmitted` alongside `req.readableEnded`. The first
two are set by the Node HTTP parser once all network bytes reach the socket,
which happens before the route handler reads anything. Any framework that
awaits between socket read and dispatch — notably the Next.js pages router —
therefore reported an unread body as already consumed.
With `bodyParser: false` there is no `req.body` to rebuild the request from
either, so `copilotRuntimeNodeHttpEndpoint` logged "Request stream consumed
with no available body" and forwarded an empty payload upstream, and the
request failed with `400 Invalid JSON payload`.
Rely only on `readableEnded`, which flips true after the `end` event fires from
genuinely draining the stream. The `parsedBody !== undefined` check at the call
site still covers the body-parser case.
Diagnosed by @AlexNti in #3489, which patched the since-retired
`packages/v1/runtime` path; re-applied here on `packages/runtime` with the
tests ported and a live `http.IncomingMessage` regression test added.
Reason: Add the current MiniMax text models to BuiltInAgent model
resolution.
- Register MiniMax-M3 and MiniMax-M2.7 as built-in model identifiers.
- Resolve MiniMax model strings through the global endpoint with API key
and regional base URL configuration.
- Document both model specifiers and cover global and China endpoint
selection.
Checks:
- `node_modules/.bin/nx run @copilotkit/runtime:test --
src/agent/__tests__/resolve-model-baseurl.test.ts`
- `node_modules/.bin/nx run @copilotkit/runtime:check-types`
- `pnpm validate:model-names`
- `node_modules/.bin/nx format:check
--files=packages/runtime/src/agent/index.ts,packages/runtime/src/agent/__tests__/resolve-model-baseurl.test.ts`
- `git diff --check`
A managed Channel that loses its gateway link was invisible outside the
host process: the only trace was the injected `log` seam, which for a
self-hosted or Railway-hosted runtime reaches nobody who can act on it.
One 2026-08-12 outage ran 2.7 days on `kite-community` before a customer
reported it, and reconstructing it needed pod forensics.
Emit `channel_session_dropped` (carrying the cause we already compute for
the log line) and `channel_session_recovered` (carrying how long the
outage lasted). A session may report `online` without a preceding drop,
which is not a recovery and is not reported as one.
Events deliberately omit the Channel name — it is a customer-chosen
identifier — and carry no message content. Capture is fire-and-forget
with failures swallowed, the same contract fireInstanceCreatedTelemetry
uses: telemetry must never break a live session.
Also replay the drop cause on each "still down" reminder. In prod those
lines read `still down after 233134s; Phoenix is retrying` with no cause
at all, so an operator had to find the first line, 15 minutes earlier, to
learn it was an HTTP 502.
## Summary
`packages/runtime` declares its own `"uuid": "^10.0.0"` dependency, but
nothing in the package's source actually imports it directly — id
generation in `@copilotkit/runtime` goes through `randomUUID()`
re-exported from `@copilotkit/shared`, which already depends on
`uuid@^11.1.0`. The unused v10 pin just adds an npm deprecation warning
for every consumer installing `@copilotkit/runtime`:
```
npm warn deprecated uuid@10.0.0: uuid@10 and below is no longer supported. For ESM codebases, update to uuid@latest. For CommonJS codebases, use uuid@11 (but be aware this version will likely be deprecated in 2028).
```
This bumps it to `^11.1.0` to match `@copilotkit/shared` and clears the
warning.
## Test plan
- [x] `pnpm --filter "@copilotkit/runtime^..." run build` — all
workspace dependencies build cleanly
- [x] `pnpm --filter @copilotkit/runtime run check-types` — no type
errors
- [x] `pnpm --filter @copilotkit/runtime run test` — 126 test files /
1746 tests passing
- [x] Confirmed no file in `packages/runtime/src` imports `uuid`
directly (grepped for both `from "uuid"` / `from 'uuid'` and
`require("uuid")` — zero matches)
- [x] Confirmed `pnpm-lock.yaml` now resolves `uuid@11.1.0` for this
dependency, which is not on npm's deprecated-versions list
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Closes#5911
Adds an optional `agentId` field to `AgentRunnerConnectRequest` so
custom agent runners can hydrate messages when the cache is empty. Also
passes the already-available `agentId` in `handleSseConnect` through to
`runner.connect()`.
## Changes
- `packages/runtime/src/v2/runtime/runner/agent-runner.ts`: Added
`agentId?: string` to `AgentRunnerConnectRequest` interface
- `packages/runtime/src/v2/runtime/handlers/sse/connect.ts`: Pass
`agentId` to `runner.connect()`
## Verification
- Runtime package builds successfully (`pnpm --filter
@copilotkit/runtime build`)
- The change is purely additive — the field is optional and does not
affect existing runners
## Summary
`CopilotRuntime` exposes an `onError` callback, but the v1-to-v2
delegation drops it before server request processing. Runtime failures
therefore can't provide the incoming request headers that applications
use to correlate errors with a user or tenant.
## Root cause
The legacy constructor retains the callback type, while delegated
runtime options omit it. Common, SSE, and Intelligence handlers consume
failures inside their own boundaries, before a generic endpoint hook can
reconstruct the legacy event.
## Changes
- Add one internal runtime reporter that snapshots incoming Fetch
request headers.
- Attach the configured legacy callback to the delegated runtime
instance.
- Route common, SSE, and Intelligence agent-run failures, including
standard `RUN_ERROR` events, through that reporter exactly once.
- Redact sensitive request headers (authorization, proxy-authorization,
cookie, set-cookie, x-api-key, api-key, and the CopilotKit public-key
header) before they reach the `onError` event.
- Preserve responses, stream close behavior, telemetry, cleanup,
logging, endpoint hooks, and callback isolation.
- Add production-path regressions.
## Compatibility
The existing `CopilotErrorEvent` type and optional
`context.request.headers` field remain unchanged. Header names and
values come from the failing request's Fetch `Headers` object, with
sensitive credentials stripped before the event is emitted. The callback
receives a fresh record, so mutation cannot alter the request or a later
event.
## Out of scope
React provider propagation, Chat, CopilotMessages, the deprecated
runtime-client hook, redaction policy, HTTP response exposure, v2
endpoint-hook semantics, and non-agent runtime routes remain outside
this slice.
## Related PRs and Issues
Addresses #2716.
Scope follows
https://github.com/CopilotKit/CopilotKit/issues/2716#issuecomment-5086936254.
Runtime error-routing precedent:
https://github.com/CopilotKit/CopilotKit/pull/2143.
## Test plan
- [x] Runtime error regression and reporter tests, 12 + 4 tests passed.
Covers public routing, sanitized header propagation with credential
redaction, callback rejection containment, snapshots, mutation
isolation, and malformed requests.
- [x] SSE and Intelligence telemetry tests, 7 + 14 tests passed. Covers
agent-run failures, `RUN_ERROR`, setup boundaries, exact-once reporting,
telemetry, cleanup, and response preservation.
- [x] Runtime preservation suites, 57 + 129 + 56 tests passed. Existing
request, endpoint-hook, and runtime-library behavior remains intact.
- [x] Typecheck, formatting, and lint passed on changed runtime files;
lint reported four pre-existing warnings.
- [ ] CI green (`static / quality`, `test / unit` on Node 20/22/24).
## What does this PR do?
Fixes stale Copilot Runtime documentation links that still point to
`/concepts/copilot-runtime` and now route users to the existing
`/backend/copilot-runtime` page.
This updates both the source JSDoc and the generated reference MDX so
the current docs content and future regenerated reference docs stay
aligned.
## Related PRs and Issues
- Closes#2082
## Testing
- `rg -n "concepts/copilot-runtime" packages
showcase/shell-docs/src/content` returns no matches
- `rg -n "backend/copilot-runtime"
packages/runtime/src/lib/runtime/copilot-runtime.ts
packages/react-core/src/components/copilot-provider/copilotkit-props.tsx
showcase/shell-docs/src/content/reference/v1/classes/CopilotRuntime.mdx
showcase/shell-docs/src/content/reference/v1/components/CopilotKit.mdx`
- `git diff --check`
## Checklist
- [x] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation
- [x] "Allow edits by maintainers" is checked (lets us help iterate on
your PR directly — faster turnaround for everyone)
## What does this PR do?
Adds the CopilotKit consumer side of ENT-1173 across Shared, Runtime,
Core, Web Inspector, and the existing Shell Docs pages.
- Defines and parses optional trusted Inspector metadata for identity,
plan, license, action, usage, and expiry. Runtime proxies it through a
private, failure-isolated route, and Core refreshes it without changing
connection state.
- Groups Inspector navigation into Threads, Agents, and Learning.
Threads renders finite, unlimited, unknown, overage, and expiring usage
states plus matching trusted plan or license actions.
- Keeps explicit `threadEndpoints` as the only authority for Thread
requests. Locked or absent capability states make no list, subscription,
detail, message, event, or state calls.
- Keeps the zero-thread video, three example Threads, detail tabs, and
guided tour in empty and locked states. General Intelligence remains the
default onboarding path; only trusted `team_self_hosted` metadata uses
self-hosted onboarding.
- Gives an active license with missing Runtime routes a short **Finish
setting up Rich Threads** state. Users can copy a safe coding-agent
prompt or open the public Runtime setup guide. The same copy control
appears in that guide, and raw Markdown/LLM views include the full
prompt.
- Keeps finite usage green below 90%, orange from 90% to the limit, and
red at or above the limit. At 90%, a trusted plan action changes from
**Manage Your Plan** to a purple **Upgrade Your Plan** without changing
its trusted URL, action kind, or telemetry contract.
- Adds a deterministic 33-state loopback lab for CopilotKit developers.
It has no production route or export, is absent from public docs and
package metadata, and is excluded from the npm tarball.
`Expiring Soon` is display-only; this PR does not enable the thread
culler. Managed Enterprise receives no manage-plan action, and Team
Self-Hosted receives no hosted plan action. Optional metadata and the
additive expiry field remain compatible across mixed producer, Runtime,
Core, and Inspector versions.
A small Channels test-only change updates fetch mocks for current
TypeScript types. It changes no Slack or Teams docs or runtime behavior.
## Related PRs and issues
- Refs
[ENT-1173](https://linear.app/copilotkit/issue/ENT-1173/ship-plg-ready-inspector-navigation-metadata-and-locked-threads)
- Producer:
[CopilotKit/Intelligence#696](https://github.com/CopilotKit/Intelligence/pull/696)
## Validation
- `@copilotkit/web-inspector`: 20 files and 372 tests passed; typecheck
and production build passed.
- Shell Docs: 57 files and 383 tests passed; lint, typecheck, and
production build passed. The build generated all 222 static pages.
- Browser checks cover the copy-prompt flow, unchanged white **Manage
Your Plan**, purple **Upgrade Your Plan**, orange 4,500/5,000 usage, and
red 5,000/5,000 usage.
- Independent review found no Critical or Important issues.
- The broader Runtime, React Native, Channels, package-quality,
compatibility, and Node-version checks from the prior pushed head remain
green.
## Checklist
- [x] I have read the [Contribution
Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md)
- [x] I updated the relevant documentation
- [ ] "Allow edits by maintainers" is checked
**Half 2 of 2 for OSS-739.** Gateway half ships first:
CopilotKit/Intelligence#746.
## Why
`status().overall === "online"` proved only that the runtime reached the
Gateway with a valid project API key. It said nothing about whether a
Slack/Teams app was bound to the Channel, so **a Channel with no
provider at all reported `online`** — and every version of our Channels
onboarding guidance used that value to certify end-to-end success.
`setup_required` had **no producer**. The manager set it only when the
activation engine threw `SETUP_REQUIRED`, and the engine stopped doing
that at the 2026-07-29 realtime-boundary cutover (`8f166577ce`). In
published `@copilotkit/channels-intelligence@0.7.0` the string survives
in exactly one file — a shipped *test*. The 15 doc comments describing
the state outlived the mechanism, which is why nobody noticed for a
week.
## Change
- `connectRealtimeGateway` captures the control join reply (it was
**discarded**) and exposes `providerStates()`. Phoenix's `Push.resend`
preserves `recHooks`, so the hook re-fires on every auto-rejoin — a
Channel provisioned while the runtime was disconnected is picked up with
no extra plumbing.
- The launcher and the manager's handle view delegate it as a
**getter**, not a captured snapshot, for that same reason.
- `status()` gains `detail`, reporting `transport` and `provider`
separately so a caller can assert the leg it cares about:
```ts
status() → {
overall: "setup_required",
channels: { support: "setup_required" },
detail: { support: { status: "setup_required", transport: "online", provider: "not_attached" } },
}
```
`channels` keeps its shape — turning its values into objects would break
the CLI's `channels-report` and the starter channel-host — but its
values are now the fold of the two legs, which is what makes `overall`
honest.
- The stale `setup_required` doc comments are corrected, with a note not
to describe the state again without a path that can emit it.
## Back-compat: `unknown` is load-bearing
An older Gateway, a Gateway whose lookup failed, a handle without the
seam, a Channel the Gateway did not mention, an unrecognised state, and
a throwing getter **all** yield `unknown`, which keeps the
transport-derived status — exactly today's behaviour. Only a *positively
reported* absence downgrades a Channel, so no existing deployment turns
amber on upgrade.
The **41 pre-existing channel-manager tests pass unchanged**, which is
that guarantee.
## Testing
- `channel-manager-provider-leg.test.ts` — 14, incl. the regression test
that never existed ("reports setup_required for a joined Channel with no
provider attached") and one case per degradation path
- `realtime-gateway-provider-states.test.ts` — 8 parser cases
- `realtime-gateway.test.ts` — +2 proving the wiring end-to-end through
real Phoenix framing, not just the parser
- Full suites: runtime **1874/1874**, channels-intelligence **192/192**;
`check-types` and `build` clean for both packages
**I mutation-tested the fold** — neutralising it fails 5 tests including
the linchpin — so these assert behaviour rather than passing vacuously.
Worth noting: two type errors (`ChannelsHandle` in `runtime.ts`, a
session mock) were invisible to vitest, which transpiles without
typechecking. The pre-commit build gate caught them.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
`AGUISendStateDelta` can emit an array append against a state where the
target array has not been initialized. The emitted patch then fails
during event compaction with `OPERATION_PATH_CANNOT_ADD`.
This change keeps the authoritative state represented in emitted events,
initializes a missing array immediately before its first `/-` append,
and applies the same contract across the generic, AI SDK, and TanStack
state-delta paths. Existing arrays and valid deltas retain their current
contents and operation order.
Closes https://github.com/CopilotKit/CopilotKit/issues/5998
## Changes
- Emit the input state before the first delta when no earlier state
event represents it
- Initialize a missing array before normalized classic, AI SDK, and
TanStack state deltas append through `/-`; custom raw-event mode remains
outside this normalizer
- Preserve populated arrays and already-valid patch operations
- Preserve caller-owned state when structured cloning falls back
- Cover missing-array reconstruction, sibling converters, failure
guards, and existing-array preservation
- Follow the current `release.config.json` and Nx release convention; no
Changeset file is added.
## Test plan
- [x] `pnpm -C packages/runtime exec vitest run
src/agent/__tests__/state-tools.test.ts
src/agent/__tests__/converter-aisdk.test.ts
src/agent/__tests__/converter-tanstack.test.ts`
- [x] `pnpm -C packages/runtime exec vitest run`
- [x] `pnpm exec nx run @copilotkit/runtime:check-types`
- [x] `pnpm exec oxfmt --check` on changed runtime files
- [x] `pnpm exec oxlint` on changed runtime files, zero errors with two
pre-existing no-shadow warnings
- [x] Verify no `.changeset` file is added because current main uses Nx
release
- [x] Verify the final diff contains only the private helper, runtime
implementation, sibling converters, and focused tests
## Summary
`mcpApps.servers` entries that carry `includeTools` or `excludeTools`
are currently accepted even though the pinned
`@ag-ui/mcp-apps-middleware` package has no option for them. The runtime
then ignores the keys, so tools an operator intended to restrict remain
available. This change rejects that configuration instead of allowing a
silent no-op.
## What CopilotKit owns
- `mcpApps.servers` configuration and `agentId` scoping.
- Projection of selected servers into `MCPAppsMiddleware`.
- Reporting unsupported configuration before middleware construction.
Discovery, model-emitted tool execution, frontend-proxied execution,
server identity, and tool provenance belong to
`@ag-ui/mcp-apps-middleware`.
## Changes
- Extract the server projection into `resolveMcpAppsServers`, which
scans all configured entries for defined policy keys, filters by
`agentId`, strips only `agentId`, and forwards other fields unchanged.
- Return a configuration error naming the unsupported key, server,
pinned middleware version, owning package, and issue when a policy key
is supplied.
- Add tests for agent scoping, field forwarding, malformed and empty
values, undefined spread values, constructor avoidance, and the existing
HTTP error path.
- Document the ownership boundary and add a runtime changeset.
## Why the filter stays external
The pinned package is version `0.0.3`. It owns the private server maps,
UI-tool discovery, model-emitted execution, and frontend proxy
execution. A CopilotKit middleware could observe only one of those paths
and would have to duplicate private server identity and tool provenance.
The complete `includeTools` and `excludeTools` implementation belongs in
the external package, where one predicate can cover discovery and both
execution paths.
## Current behavior
Plain JavaScript or JSON configuration can supply `excludeTools:
["delete_account"]` without a TypeScript excess-property check. The
runtime currently accepts the configuration, constructs
`MCPAppsMiddleware`, and leaves the tool available. The new behavior
returns an HTTP 500 through the existing runtime error path, names the
unsupported key and dependency, and does not construct the middleware.
## Follow-up
The counterpart change in `@ag-ui/mcp-apps-middleware` should add the
fields to the per-server configuration, preserve absent versus empty
include lists, resolve server identity through its existing maps, and
apply one predicate after UI-resource discovery and before model-emitted
and proxied tool execution. Once that version is released, CopilotKit
can remove the rejection and pass the fields through unchanged.
## Related issue
Refs #5930.
The cross-repository ownership split follows the proposal in
https://github.com/CopilotKit/CopilotKit/issues/5930#issuecomment-5128722524.
This PR does not close the issue.
## Test plan
- [x] `pnpm -C packages/runtime exec vitest run
src/v2/runtime/__tests__/mcp-apps-servers.test.ts
src/v2/runtime/__tests__/mcp-apps-middleware-integration.test.ts`
passed, 2 files and 20 tests
- [x] `pnpm -C packages/runtime exec vitest run` passed, 129 files and
1,836 tests
- [x] `pnpm exec nx run @copilotkit/runtime:check-types` passed
- [x] `pnpm exec oxlint` and `pnpm exec oxfmt --check` passed on changed
TypeScript files
- [x] `pnpm check:plugin-skills` passed
- [ ] `CI green for static / quality and test / unit on Node 20, 22, and
24`
Self-review of the previous commit.
Corrects a factual error I introduced: the `SETUP_REQUIRED` note claimed such a
Channel "has no transport at all (the launcher never returned a handle)". False
for the `hasDirectAdapter` branch, which starts the developer-owned transport and
assigns a synthetic handle — so there IS a running transport there. Dropped the
wrong reasoning and shortened the note to the part that holds: the misattribution
is cosmetic on a path with no producer, and a future producer should report
through the `providerStates` seam.
Removes three tests that did not earn their place:
- "calls the seam ON the session" — redundant. `ProviderStateGateway.providerStates`
reads `this`, so the two remaining tests already fail if the launcher ever used
a detached reference. It died on the same mutation as the first test, for the
same reason.
- "omits providerStates for a session without the seam" — survived the mutation
that removes the forward, so it guarded nothing.
- the channel-level-error rejoin case — same `Push.resend` hook as the transport
drop, so it re-proved one mechanism at ~1s extra wall-clock. Kept the drop
case: it asserts a genuinely fresh socket, which is the "provisioned while the
runtime was disconnected" story the design claim is about.
- "keeps the last reported states while a rejoin has not yet succeeded" — pinned
behaviour with no observable consequence, since the transport leg dominates the
fold while offline.
Also trims the drift-guard comment: why a guard was NOT added belongs in the PR
discussion, not permanently in source.
Re-mutation-tested after trimming: removing the forward kills both remaining seam
tests; making `providerStates` a snapshot kills the rejoin test while the other
41 gateway tests pass.
Verified: channels-intelligence 195/195, runtime 1874/1874, build + oxfmt clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Addresses review on #6360.
`startChannelsWithGatewayControl` is public so callers can compose over a
session they manage themselves, but it forwarded only `onClose` and
`onStateChange` — not `providerStates`. A handle without the provider seam makes
`ChannelManager.providerLeg` fall back to `unknown`, which keeps the
transport-derived status and reports `online` for a Channel with no provider
bound: the exact false green OSS-739 removes, still reachable through a public
export. The guard is widened too, so a session exposing only `providerStates`
is no longer dropped on the fall-through path.
Tests the reconnect claim that makes `providerStates` a getter rather than a
snapshot. Nothing exercised it: the gateway tests covered only the initial join
reply, and the manager-side rejoin test proves the manager re-reads on each
`status()` call, not that the session's value ever changes. The fake socket's
join reply can now vary per join, so a drop -> rejoin carrying a different
`channels` map asserts the refresh over real Phoenix framing — via both rejoin
paths (channel-level error on a live socket, and a full transport drop onto a
fresh socket), plus the case where a rejoin has not yet succeeded and the last
known states must persist.
Mutation-tested both: removing the forward kills 3 of 4 seam tests, and making
`providerStates` a captured snapshot kills both rejoin tests while all 3
pre-existing provider-state tests still pass — which is the gap itself.
Docs corrected against their real mechanisms:
- `attached`/`unhealthy`/`not_attached` now state the gateway's actual rule
(adapter `status == "active"` is part of the predicate; a configured adapter in
`error` is `unhealthy` with no failed health check), plus the best-of adapter
fold that keeps a Slack-only Channel `attached`.
- `ready()` no longer promises it rejects on `error`. It awaits activation, so
it can resolve while `status().overall === "error"` from an `unhealthy`
provider. Says that instead.
- Notes the legacy `SETUP_REQUIRED` path reports a provider condition on the
transport leg (dead, cosmetic, left rather than guessed at), and why the
provider-state set is duplicated across the duck-typed package seam.
Verified: channels-intelligence 199/199, runtime 1874/1874, both builds clean,
oxfmt/oxlint clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the LLM emits jsFunctions/css as null (frequent on open-gen-ui-advanced),
setParam coerces the value to undefined and emitParamDelta produced a JSON
Patch `{op:"add", path}` with no `value` property. fast-json-patch rejects
that client-side with OPERATION_VALUE_REQUIRED and drops the whole activity
patch (console warning, benign but noisy and fragile).
Guard emitParamDelta to skip emitting when the value is undefined. Empty
arrays and completion markers still emit. Adds a regression test feeding a
null jsFunctions and asserting no value-less patch is produced.
Updates the runtime agent-runner skill references to match the bounded
in-memory runner: correct the InMemoryAgentRunner store as a process-global
singleton, document its bounds and onConcurrentRun concurrency handling, note
that dedup weakens past the run cap, fix rotted in-memory.ts citations onto
stable symbols, and correct the multi-instance SqliteAgentRunner scaling
guidance. The generated skills/ mirror is regenerated in lockstep so source and
mirror stay in sync.
Co-Authored-By: Claude <noreply@anthropic.com>
Adds a dedicated bounded-thread-store suite and extends the in-memory runner
suite to lock in the behaviors introduced with the bounded, teardown-isolated
runner:
- LRU thread eviction, per-thread run-cap FIFO trimming, and byte-ceiling
eviction (including that a live/running or stop-requested thread is never
evicted, and that a just-appended thread pushes OTHER threads out rather than
self-evicting).
- InMemoryLimits validation/normalization: invalid bounds clamp to defaults
instead of crashing enforceRunCap, and the 0/Infinity disable sentinels are
preserved.
- Thread-level createdAt and message-snapshot decoupling survive run-cap
eviction and interleaved empty-snapshot runs.
- stop() guards and the supersede path: an aborted run finalizes as a clean
RUN_FINISHED against its own captured intent, a superseded run cannot clobber
its replacement's state or history, and an immediate abort-throw that emitted
nothing creates no phantom historic run.
Also restores the shared store's default limits after tests that reconfigure
the process-global store so suites stay isolated, and updates a handle-run
comment for the GLOBAL_STORE -> shared store rename.
Co-Authored-By: Claude <noreply@anthropic.com>
A run's async teardown used to read the shared, mutable store.stopRequested to
decide whether to finalize as a clean RUN_FINISHED or a synthetic RUN_ERROR.
Under supersede (or a stop() immediately followed by a new run()), a later run
resets that shared flag, so the earlier run's fire-and-forget finalization could
be mislabelled — an intentionally stopped run finalized as an error, or vice
versa — and could push history under, or clobber the state of, the newer run
that now owns the thread.
Fix by capturing a per-run RunFinalizeControl when the run starts. stop() and a
superseding run() flip THAT run's captured control (not just the store flag),
and the run's teardown reads its own captured intent, so a later run resetting
store state can never change how an earlier run finalizes.
The teardown itself is unified into a single finalizeRun helper shared by the
success and error paths (they were near-identical and must stay symmetric) and
made ownership-aware:
- It only pushes history / resets shared store state when this run still owns
the thread (store.currentRunId still equals this run's id), so a superseded
run cannot corrupt the successor's history or state.
- The error path additionally requires at least one real (pre-finalize) event,
reviving a guard that had gone dead: an immediate throw that emitted nothing
must not create a phantom historic run holding only the synthetic terminal.
- On completion it releases the run's infinite ReplaySubject buffer via an
identity guard (store.subject === nextSubject), reclaiming the duplicate
buffer on the owning path while leaving a live successor's subject untouched.
The concurrency branch now also triggers on store.stopRequested, not just
isRunning: stop() flips isRunning off the instant it aborts but the run keeps
finalizing, and a run() slipping through that window went entirely unhandled.
The previous-subject bridge is removed: forwarding a dying superseded run's
subject would replay its RUN_STARTED and push its terminal into the live run's
stream, an invalid AG-UI sequence — a superseded run must stay isolated to its
own subscribers.
Co-Authored-By: Claude <noreply@anthropic.com>
Route every storage access in the runner through the shared ɵBoundedThreadStore
instead of the old unbounded GLOBAL_STORE Map: run() acquires threads via
getOrCreate (which applies LRU eviction), and connect/isRunning/stop/
listThreads/getThreadMessages/getThreadEvents/clearThreads read through the
store's touch-aware accessors so reads keep LRU order honest.
The constructor now accepts InMemoryLimits inline alongside onConcurrentRun.
Note the scope difference, called out in the JSDoc: onConcurrentRun is
per-runner, but the limits reconfigure the PROCESS-GLOBAL store shared by every
runner. A partial limits update coalesces each unspecified field against the
store's current effective bounds (not the hardcoded defaults), so tuning one
bound never silently resets its siblings; a genuine clobber of an
already-customized store warns once.
getThreadMessages now returns the thread-level snapshot (a shallow array-level
copy) rather than the last run's snapshot, so run-cap eviction and interleaved
empty-snapshot runs can never lose it. getThreadState is hardened to reject
arrays (which pass `typeof === "object"`) and to return a defensive shallow
copy so callers cannot mutate stored snapshot state.
Co-Authored-By: Claude <noreply@anthropic.com>
The in-memory runner previously kept every thread and run forever in an
unbounded process-global Map, so a long-lived process leaked memory without
limit. Introduce ɵBoundedThreadStore as the single backing store, enforcing
three independent bounds resolved from InMemoryLimits (defaults in
ɵINMEMORY_DEFAULTS):
- maxThreads: LRU eviction of whole threads.
- maxRunsPerThread: FIFO run-cap per thread.
- maxBytes: approximate cross-thread byte ceiling (via ɵestimateBytes),
enforced at run completion by evicting other LRU non-running threads.
Limit values are validated and normalized once (ɵnormalizeLimits /
ɵisValidLimit): only a non-negative integer or +Infinity is well-formed.
Invalid values (negatives, -Infinity, NaN, fractional caps) would otherwise
turn the `count > limit` enforcement guards into infinite loops or a shift()
of undefined; they are instead clamped to the documented default with a single
warning. Clamp-and-warn rather than throw matches this file's best-effort
posture (ɵestimateBytes swallows serialization failures), because constructing
a non-durable convenience runner must never abort — or later surface an
unhandled rejection — on a typo'd bound.
Thread creation time and the latest non-empty message snapshot are held at the
THREAD level (InMemoryEventStore.createdAt / messagesSnapshot), decoupled from
historicRuns so run-cap FIFO eviction can neither drift the reported creation
time forward nor drop the message history. Eviction — whole-thread LRU and
per-thread run-cap trimming alike — is logged once per store (warn-once latch)
so bounded history loss is visible rather than silent.
Also defines the per-run RunFinalizeControl shape and the store's
activeFinalize holder that the run-teardown isolation builds on.
Co-Authored-By: Claude <noreply@anthropic.com>
## What changed
- Mark initial gateway HTTP 5xx and transient transport failures as
retryable.
- Retry initial managed Channel activation with exponential backoff from
1 second to a 30-second cap until it connects or the manager stops.
- Preserve retry hints from `gateway_draining` join replies and retry
initial join timeouts.
- Keep HTTP 4xx and NXDOMAIN failures terminal.
- Back off established-session outage reminders from 30 seconds to a
15-minute cap while Phoenix continues reconnecting.
## Why
The OpenTag Railway runtime saw the gateway host return HTTP 502 during
an outage. Established Phoenix sessions keep retrying, but a runtime
that starts during the outage stops after its one initial connect
window. It cannot recover when the gateway comes back unless the process
restarts. Fixed 30-second reminder logs also flood long outages.
The gateway drain work now rejects new joins with a structured retryable
response. The client must preserve that response so the runtime can
retry instead of leaving the Channel in a terminal error state.
## Companion change
CopilotKit/OpenTag#25 keeps the Railway HTTP server alive while an
initial Channel retry is pending. OpenTag must consume a CopilotKit
release containing this PR before that companion change can recover by
itself.
## Validation
- `pnpm nx run-many -t test,check-types,build -p
@copilotkit/runtime,@copilotkit/channels-intelligence`
- `pnpm nx run-many -t publint,attw -p
@copilotkit/runtime,@copilotkit/channels-intelligence`
- pre-commit tests and package checks for all affected projects
- `pnpm exec oxfmt --check` on all five changed files
- `pnpm exec oxlint` on all five changed files
- `git diff --check`
contextablemark