## Problem
Main CI cannot compile the channels-intelligence test suite because
ReplyTarget and Sub are referenced without declarations.
## Why
The merged Slack tool-status regression coverage introduced the
references but omitted the corresponding type declarations. Every
dependent Nx workflow fails during the shared package build.
## Fix
Restore the ReplyTarget type import and the local Sub subscriber helper
type in render-events.test.ts.
Validation: pnpm exec nx run @copilotkit/channels-intelligence:build;
pnpm exec nx run @copilotkit/channels-intelligence:test; pnpm exec nx
run @copilotkit/channels-intelligence:check-types.
## Problem
Slack threads render every tool invocation as visible progress, leaving
noisy `Used …` rows ahead of the final answer.
## Why
The direct Slack renderer treated tool status as enabled when
`showToolStatus` was omitted, while the managed Intelligence renderer
always emitted tool lifecycle frames. The managed renderer's opt-in was
not reachable through the supported `createChannel()` and
`CopilotRuntime` API.
## Fix
- Hide tool-call progress by default for direct and managed Slack
routes.
- Expose `createChannel({ showToolStatus: true })` as the managed
Channels opt-in and carry it through runtime activation to the
Intelligence launcher.
- Preserve `slack({ showToolStatus: true })` as the direct Slack opt-in.
- Keep existing managed behavior for non-Slack routes.
- Continue capturing and executing tool calls while their status frames
are hidden.
- Add regression coverage for the public managed API, legacy and
native-streaming Slack, and Realtime Gateway paths.
Paired with https://github.com/CopilotKit/Intelligence/pull/632.
Three fixes from the OSS-670 investigation, where a dev gateway restart
landed mid-turn: the agent's answer was produced and then thrown away,
the end user got a generic error, and the bot went silent for 19 minutes
with nothing in its logs.
## A revoked lease is a fenced outcome, not a turn failure
`CHANNEL_DELIVERY_LEASE_INVALID` means another owner holds the delivery
— app-api fences every intent on `leased_until > now()` AND the
lease-token hash, so nothing we send can be accepted and redelivery is
app-api's job.
- The guard lives in `IntelligenceAdapter.dispatch`, the site that
actually runs. An earlier revision guarded only the realtime transport,
which is unreachable in production because `dispatch` catches every turn
error and nacks first — so a fenced delivery still put a doomed
`channel.delivery.fail.v1` on the wire, and that second 409 is what
surfaced as `nack after turn failure failed` followed by `no delivery
state`.
- The code is no longer retried through the 3-attempt render-batch
ladder.
- A `fail` intent that is itself fenced is logged and dropped rather
than rejected.
- The predicate lives in `transports.ts` because both transports surface
the same code (the HTTP path as a 409 body).
The adapter-dispatch test is a real regression test: reverting the guard
makes it fail on `channel.delivery.fail.v1` being pushed.
### Abandoning is a third outcome, not a silent one
Swallowing the coded error means the `DeliverySource` observes a
**successful** `onDelivery` — so neither its own revoked-lease branch
nor `ack` runs, and those are the only two places the per-delivery lease
record is dropped. Without more, every fence leaked a lease token,
delivery scope and accepted-seq high-water map for a delivery this
runtime can never speak for again (app-api may already have handed it to
another runtime, so the id may never be seen here again).
`DeliverySource.abandon(deliveryId, reason)` is that third outcome
alongside ack/nack: retire local state, send nothing. It is synchronous
and wire-free by design, optional on the interface so existing
implementations stay valid, and implemented on
`RealtimeGatewayTransport` (`deliveries`), `HttpDeliverySource`
(`leases`) and `InMemoryDeliverySource`. The transport's internal
revoked-lease catch routes through it too, so there is one retirement
path rather than two.
Both wiring tests fail with the `abandon` call reverted — the adapter
test on an empty `abandoned[]`, and the composed adapter+transport test
because its probe `nack` pushes a second, doomed
`channel.delivery.fail.v1` off the retained state.
## A refused reconnect is diagnosed instead of retried in silence
`socket.onError` returned early once the socket had connected, so a
refused *reconnect* recorded nothing and Phoenix retried forever with no
output. The cause is now recorded, and when the transport exposes no OS
code an HTTP probe of the endpoint runs once per outage.
The probe is unauthenticated, so it deliberately does **not** claim your
key is revoked — its 401/403 would look identical with a perfectly good
credential. It establishes reachability: 5xx reads as an
unhealthy/restarting gateway, anything else as "reachable, handshake
refused — verify the project API key".
## Honest status, loud while down
`gave_up` maps to status `error`, but Phoenix keeps retrying underneath
and a successful rejoin restores `online` — the log now says so instead
of reading as terminal. The drop cause travels with each state
transition (it was previously `{ meta: undefined }`), and a "still down"
line repeats while the session is out, including after `gave_up`.
Channel status semantics and the "manager never re-activates on a drop"
contract are unchanged.
## Not included
Delivery-lease renewal has no owner across a gateway restart — draining
stops the channel process that renews it, so a long turn loses its lease
to expiry regardless. That, and bounding an in-flight turn whose lease
is already dead, are follow-ups on the ticket. Retiring the local record
(above) stops the leak; it does not keep the lease alive.
Tests: `channels-intelligence` 13 files / 244 tests; `runtime` 127 files
/ 1799 tests; `check-types` clean on both.
`IntelligenceAdapter.dispatch` swallows a revoked-lease error so no doomed
`fail` intent goes out, which means the DeliverySource observes a SUCCESSFUL
`onDelivery` — so neither the transport's own revoked-lease branch nor `ack`
runs, and those are the only two places the per-delivery lease record is
dropped. Every fence therefore leaked a lease token, delivery scope and
accepted-seq high-water map for a delivery this runtime can never speak for
again (app-api may already have handed it to another runtime, so the id may
never be seen here again).
Adds an explicit `DeliverySource.abandon` — the third outcome alongside
ack/nack: retire local state, send nothing. Implemented on both real sources
(`RealtimeGatewayTransport.deliveries`, `HttpDeliverySource.leases`) and on
`InMemoryDeliverySource` so tests can assert it; the transport's internal
revoked-lease catch now routes through it too. Optional on the interface so
existing implementations stay valid.
Both new wiring tests fail with the `abandon` call reverted: the adapter test
sees an empty `abandoned[]`, and the composed adapter+transport test's probe
`nack` pushes a second, doomed `channel.delivery.fail.v1` because the
DeliveryState was still retained.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Review found the fenced-lease guard sat only in the realtime transport, while
IntelligenceAdapter.dispatch catches every turn error and nacks first — so a
revoked lease still put a doomed channel.delivery.fail.v1 on the wire and the
earlier test only passed because it stubbed onDelivery. The guard now lives in
dispatch too, and the predicate moved to transports.ts because both transports
surface the same code (the HTTP path as a 409 body).
Also from review:
- Clear lastTransportError/lastTransportCode on recovery, so a later outage that
reports nothing is no longer attributed to the previous outage's error.
- Fence the endpoint probe on an outage generation, so a verdict that resolves
after the link healed cannot latch onto the next, unrelated outage.
- Stop inferring a revoked credential from any HTTP status. The probe is
unauthenticated, so its 401/403 says nothing about our key; it can only
establish reachability. 5xx now reads as an unhealthy/restarting gateway.
The new adapter-dispatch test is a real regression test: with the dispatch guard
reverted it fails on channel.delivery.fail.v1 being pushed.
## Summary
Migrates the banking showcase off its bespoke "Glass Engine" inspector
onto the productized `@copilotkit/web-inspector`, and builds out the two
inspector features that migration requires: a **Capabilities tab**
(enable/disable frontend tools + A2UI catalog components, with real
enforcement) and a **Memory recall** view backed by a new `recall()` RAG
read path and a project-scoped realtime channel.
This is the CopilotKit half of a cross-repo effort; the Intelligence
half landed in Intelligence#557.
## What's included
**Capabilities (tool + catalog enable/disable, real enforcement)**
- `core`: `RunHandler` tracks per-tool and per-catalog-component
enable/disable overrides; `buildFrontendTools` filters disabled tools
out of the tool list sent to the agent; `core.ts` exposes
`setToolEnabled` / `setCatalogComponentEnabled` / `catalogComponents`.
- `react-core`: catalog toggling enforced on **both** the context and
render paths (`filteredCatalog`).
- `a2ui-renderer`: `filterCatalog` preserves `themeSchema` and
`functions` when dropping disabled components.
- `web-inspector`: new **Capabilities** nav item + view with reactive
toggles (gated on presence of tools/catalog).
**Memory recall (RAG read path)**
- `core`: `memory.recall()` with score projection; new
`project_meta:memories:{code}` realtime channel that surfaces
project-scoped memories alongside user memories, degrading to user-only
when project creds are absent.
- `runtime`: `/memories` recall handler + project-join creds threaded
through the `/memories/subscribe` proxy, gated behind the opt-in
`exposeMemoryRoutes` server flag (**default OFF**).
- `web-inspector`: Learning tab renamed **Memory**; recall section with
relevance bars and scope badges.
**Banking showcase migration**
- Enables the product web-inspector via `showDevConsole`; removes the
bespoke inspector, Glass Engine deployment gate, and dead memory routes.
Capabilities tab is **ungated** in this demo.
## Review / quality
- Full MSAL CR loop (6 modules + cross-cutting + 7-agent final gate)
converged with **zero mandatory findings**; Procedure 3 bucket-(c)
promotion audit returned zero promotions.
- 4 real bugs fixed during review: recall limit/query validation, a
broken fired-capability dot, catalog registration guard consistency, and
`filterCatalog` themeSchema preservation.
- Test coverage added across all touched packages (memory recall,
capability toggles, catalog filtering, recall UI).
## Follow-ups (tracked, not in this PR)
Project-channel observability (give-up monitor / bounded rejoin warns),
memory type-design hardening (recall field-validation parity,
`SaveMemoryResponse` discriminated union), and capabilities consistency
(`setToolEnabled` subscriber-event parity). None are load-bearing for
the shipped surface.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- restore provider-scoped Channels overview pages for Slack and
Microsoft Teams
- separate Intelligence configuration from the connect-and-run guides
and make the handoff explicit
- preserve backend selection across Channels navigation, search,
sitemap, and LLM routes
- update the Channels architecture diagrams and Intelligence setup
screenshot
- retain framework-specific connection setup for all public backends
## Why
The merged Channels docs mixed platform configuration and runtime setup,
removed the overview entry point, and exposed ambiguous or stale
navigation. This reorganizes the existing guides into a clear sequence:
**Overview → Configure in Intelligence → Connect and run**.
## Validation
- `pnpm exec oxfmt --check` on changed TS, TSX, and MDX files
- `npm run lint` in `showcase/shell-docs`
- `npm run typecheck` in `showcase/shell-docs`
- `npm test` in `showcase/shell-docs` (51 files, 351 tests)
- `npm run build` in `showcase/shell-docs` (222 static pages)
- local browser checks for Slack and Microsoft Teams with Built-in
Agent, LangGraph, Mastra, and Deep Agents
## The race, with evidence
Three PRs merged within 34 seconds on 2026-07-26. `Showcase: Build &
Push` has **no concurrency group** (deliberately), so all three ran
simultaneously and raced to push the same `:latest` tags.
| run | commit | PR | start → end |
|---|---|---|---|
| `30190815370` | `7b28934387` | #6162 | 06:18:05 → 06:29:44 |
| `30190823203` | `59f275eedc` | #6161 | 06:18:21 → 06:29:44 |
| `30190831480` | `db75a04837` | #6158 | 06:18:39 → **06:29:13** ←
newest, finished FIRST |
The newest commit finished first, so the older builds overwrote its
`:latest`. Per-service job completion times — older beating newer on
every shared slot:
| service | newer (`db75a04837`) | older (`59f275eedc`) | older won by |
|---|---|---|---|
| `shell-dashboard` | 06:25:33 | 06:25:34 | +1s |
| `showcase-harness` | 06:27:51 | 06:27:55 | +4s |
| `shell` | 06:27:17 | 06:27:28 | +11s |
| `shell-dojo` | 06:25:10 | 06:25:26 | +16s |
**All three runs reported `success`.** Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
Same failure class as #6171: a success that doesn't mean what it says.
## What I verified in YAML vs took on trust
Verified by reading the files / querying the API:
- **Tagging** — `showcase_build.yml` pushed `:latest` **and** `:${{
github.sha }}` in one `depot/build-push-action` step, in both the
`build` and `build-starters` matrices. **A per-commit sha tag already
existed**; confirmed in GHCR (`showcase-shell-dashboard` has digests
tagged `db75a04837…`, `59f275eedc…`, `d28384a2eb…`).
- **Nothing serialized the pushes.** No concurrency group; confirmed the
header comment states this is intentional.
- **Deploy consumes `:latest`** — `verify-railway-image-refs.ts` is the
SSOT assertion: staging is `ghcr.io/copilotkit/<repo>:latest` (mutable),
**prod is `ghcr.io/copilotkit/<repo>@sha256:<digest>` (already immutably
pinned)**. So this race is **staging-only**; prod was never exposed.
- **`Showcase: Verify Deploy` structurally cannot catch it.** It is a
health probe; it asserts no digest or commit provenance anywhere. Its
#6171 per-commit concurrency key is about *which run verifies*, not
*what image is running*. A stale-but-healthy service passes.
- **The racing runs build DISJOINT service sets** (see below) — I pulled
the actual job lists.
Taken on trust: nothing material. The issue description matched the API
on every point I checked.
## Why NOT a concurrency group
`detect-changes` builds a **per-push, path-filtered** matrix, so
concurrent runs build overlapping but **non-identical** service sets:
- `7b28934387` → ag2, agno, built-in-agent, claude-sdk-python,
claude-sdk-typescript, crewai-crews, langgraph-fastapi,
langgraph-python, langroid, llamaindex, mastra, pydantic-ai, shell-docs,
spring-ai, strands (**15**)
- `db75a04837` → crewai-crews, llamaindex, shell, shell-dashboard,
shell-docs, shell-dojo, showcase-harness (**7**)
`cancel-in-progress: true` would have cancelled the `7b28934387` run and
the ~10 services **only it builds would never have shipped** — trading a
stale-image bug for a never-shipped bug. Per [GitHub's
docs](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#concurrency),
`cancel-in-progress: false` is no better: *"Any previously pending job
or workflow in the concurrency group will be canceled"* — with three
rapid merges the **middle** commit's build is dropped outright. GitHub
also does not guarantee FIFO ordering of queued runs.
Concurrent runs here are **not redundant**, so they must not be
cancelled.
**Re the #6171 interaction:** this design does **not** use
`cancel-in-progress`, so no build is ever superseded-and-cancelled and
the cancelled-slot notifier is never tripped by this change. That
interaction stays theoretical — deliberately.
## The design
Make the one shared mutable resource monotonic instead of serializing
the fleet.
1. The build step pushes **only** the immutable `:<sha>` tag, plus an
`org.opencontainers.image.revision` label.
2. A guard resolves the commit behind the current `:latest`, asks GitHub
`compare/<theirs>...<ours>`, and advances `:latest` (registry-side
retag, no pull) **unless ours is `behind`** — i.e. `:latest` already
holds a descendant and moving it would roll staging back.
**Fails open by design.** No `:latest` yet, unlabelled legacy image,
unreachable API, diverged history → advance. A stuck `:latest` is the
very failure being fixed, so it declines only on *positive proof* of
regression.
**Placement:** in `redeploy-staging` / `redeploy-staging-starters`,
immediately before the Railway pull that consumes `:latest` — not per
build slot. Those jobs already have Node (build slots do **not**, so
per-slot would mean an unpinned `npx tsx` fetch on ~50 parallel
runners), and deciding right before the pull makes the window as narrow
as possible. The image list is the **same matrix ∩ build-success
intersection** that decides what gets redeployed, so a failed build can
never move a tag.
Also adds `showcase_build.yml` to `showcase_validate.yml`'s trigger
paths — the new test asserts against that file's live text, and without
the path a PR re-adding `:latest` would never run the test that catches
it.
## Tradeoffs / what stays open
- **Residual sub-second TOCTOU.** GHCR has no compare-and-swap on tags,
so two runs reading `:latest` simultaneously could still both advance.
This narrows the window from the whole build (~10 min) to inspect→retag.
Fully closing it means retiring the mutable staging tag and pinning
staging to digests the way prod already is — a change to the Railway
image-ref SSOT contract, not a workflow change. **Recommended
follow-up.**
- **The guard is inert for one build per image.** Today's `:latest`
images carry no labels (verified: `showcase-shell-dashboard:latest` has
no `Labels` at all), so the first post-merge build fails open and
advances unconditionally — same as today. Protection starts from the
second build of each image.
- Failure to retag exits non-zero, redding the redeploy job and stopping
the deploy. That is intended: redeploying against a tag that did not
move is exactly the silent false green being fixed.
## Proof
**Red/green on the live YAML.** `advance-latest-tag.test.ts` parses the
real `showcase_build.yml` (extending #6171's `redeploy-guard.test.ts`
pattern). Reverting the workflow to its pre-fix state: **14 failed / 23
passed**. With the fix: **37 passed**. Full `showcase/scripts` suite:
**2382 passed, 73 files**.
**The load-bearing predicate, verified live against the real incident
commits:**
```
compare/db75a04837...59f275eedc => behind (older run arriving late → DECLINE)
compare/59f275eedc...db75a04837 => ahead (newer run → advance)
compare/db75a04837...db75a04837 => identical
```
**Label reading, verified against a real multi-platform registry image**
— `docker buildx imagetools inspect ghcr.io/astral-sh/uv:latest` piped
through `extractRevisionLabel()` returns
`3010295ae7ff572de459987ad70db315a62ecd61`, matching `jq` exactly. The
platform-keyed shape is handled.
**Shell/jq transforms** exercised directly, including the empty-CSV edge
case (empty → empty, step skipped by its `if:`).
**Lint:** `actionlint` finding counts byte-identical to the pre-change
baseline (no new findings; the 11 pre-existing are unrelated). `zizmor
--min-severity low` with the repo config: **no findings**.
**Typecheck:** both new files are in `showcase/scripts/tsconfig.json`'s
include set and produce **zero** errors. Worth stating plainly: `nx
run-many -t check-types` **does not reach `showcase/scripts`** — the
project isn't in the nx graph and has no `check-types` target (there are
9 pre-existing type errors in sibling files, which is how I confirmed
it). So the typecheck above is mine, not CI's. The *tests* are gated:
`showcase_validate.yml` runs bare `pnpm exec vitest run` in
`showcase/scripts`, which auto-discovers the new file.
### What I could NOT prove
**I did not construct a real concurrent race on scratch branches.**
Doing it faithfully needs two builds pushing the same GHCR repo with
controlled finish ordering, which means merging to `main` — the only
branch the build workflow triggers on. No run IDs for a live race
demonstration; I am not implying one.
Unproven until this runs on main: that `docker buildx imagetools create`
retags cleanly under the runner's GHCR credentials, and that `npx tsx`
behaves in the redeploy jobs (it is already the established invocation
there — `redeploy-env.ts` — so this is low risk, not zero).
## Normal single-merge builds are unaffected
No concurrency group is added, so nothing queues or cancels. A lone
merge finds `:latest` at its own parent → `ahead` → advances, exactly as
before. Cost is one `imagetools inspect` + one `gh api` + one
registry-side retag per built service, in a job that already exists — no
extra job, no extra checkout, no change to build parallelism.
---
Branched from `db75a04837`; #6156/#6159 landed after, so this will need
main merged in before it goes green.
Probable conflict with the concurrent `git lfs pull` work in
`showcase_validate.yml` — my edit there is only the top-level `on:
paths:` list, so it should merge cleanly, but flagging it.
## What changed
- replace the managed single-frame render sink with immutable ordered
batches
- flush the first text at once and compact later text for up to 250 ms
- bound batches by frame, byte, and pending-lane limits
- add stable SHA-256 digests, batch IDs, durable high-water receipts,
and the `render_batch_v1` runtime capability
- update HTTP, Realtime Gateway, fallback, and in-memory transports to
the batch-only contract
## Why
Managed Channel latency currently grows with source delta count because
each delta takes one durable round trip. Batching makes durable work
track response time and semantic boundaries instead of token count.
Refs OSS-654. This PR follows the one-cutover contract in the companion
service plan, so it does not provide the older-server sequential
fallback described in that ticket.
Companion service PR:
https://github.com/CopilotKit/Intelligence/pull/637
## Validation
- `pnpm test` in `packages/channels-intelligence` — 223 passed
- `pnpm run check-types` in `packages/channels-intelligence`
- pre-commit Nx test, publint, and attw suite — passed
- 20-delta and 200-delta runs each produced two durable batches and
equal modeled receipt time
The service PR records the cross-repo E2E, load, fault, Kubernetes, and
image checks.
Mike Ryan