Commit Graph

14820 Commits

Author SHA1 Message Date
Benjamin Taylor b8cf4dd848 docs(release): note the measured pnpm cache save cost
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 19:34:03 -05:00
Benjamin Taylor 6a35e3cdbf perf(release): cut canary publish wall-clock roughly in half
The canary flow took ~11.5 min steady-state (and 20 min in an observed
run). Measured from run 30473499191, the time went to five avoidable
places rather than to real work.

1. `npx --yes npm@11.15.0 publish` ran per package, and npx re-resolves
   the spec against the registry on EVERY invocation: ~16s of each
   package's ~21s. A 9-package channels canary paid ~2.4 min of pure npx
   overhead; a 16-package monorepo release paid over 4 min. Hoist the
   pinned npm into lib/npm-cli.ts, install it once into a throwaway
   prefix, and reuse the binary.

2. publish-release.yml was the only workflow in the repo with no pnpm
   store cache, so all three jobs installed 4608 packages cold every
   time. Usually ~45s each, but registry-bandwidth bound and heavy
   tailed: the observed run spent 9m08s here on tarballs arriving at
   2-49 KiB/s. Add the same node-version-keyed cache the rest of CI uses.

3. The notify job ran for canaries only to compute "post nothing" — the
   builder already returns should_post=false for mode=prerelease and the
   self-watchdog is already gated off. ~85s of dead work on the critical
   path, since canary.yml waits for the whole run. Skip the job, keeping
   it reachable for a python_publish dispatch.

4. The build job fetched full history for canaries, which need none (no
   tag, no GH Release, no release-note commit range, and `nx run-many`
   resolves no merge base). That rode along in the 837 MiB workspace
   artifact too. Shallow-fetch prereleases; stable keeps depth 0 because
   its publish job pushes tags out of that artifact's .git.

5. Two smaller ones: the artifact was gzipped and then re-deflated into
   the artifact zip (compression-level: 0), and the orchestrator's
   run-discovery loop slept 6s before its first poll.

Verified: 143 release-script tests pass (6 new for the npm-cli helper),
actionlint + shellcheck + the scope-dropdown guard are clean, the
prerelease dry-run path still enumerates all 9 channels packages, and a
live probe confirms the helper installs npm 11.15.0 once (3.2s) and
memoizes thereafter (0ms).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 19:29:38 -05:00
Mike Ryan d545eba10e fix(channels): preserve live-session run semantics 2026-07-29 17:22:56 -07:00
Benjamin Taylor 6d5fbf3e3f fix(channels-slack): bound rollover and keep boundaries surrogate-safe
Two defects found by adversarially probing the rollover added in the previous
commit. Both were reachable at the default 11k cap.

1. Unbounded message creation. The continuation re-opener is synthetic text
   charged against the new message's budget, so a context whose opener is itself
   larger than the cap filled each fresh message with nothing but its own
   preamble and rolled over again — forever, never advancing `curPosted`.
   `detectOpenContext` reports everything up to the first newline after ``` as
   the fence "language", so an agent emitting a >11k whitespace-free blob
   (minified JSON, base64, a long log line) straight after a fence triggers it.
   Probed at the default cap: `posted` frozen at 11,000 while `startStream` was
   called 41+ times, bounded only by the probe's own budget. In production that
   is an unbounded stream of Slack messages — worse than the truncation it
   replaced. The opener is now dropped when it cannot leave
   MIN_MESSAGE_PROGRESS_CHARS of room, so every message carries text and the
   loop always terminates.

2. Surrogate pairs split across boundaries. Whitespace-free text reaches
   `breakPoint`'s hard-cut fallback routinely (CJK, emoji runs, base64), and
   cutting at an arbitrary UTF-16 offset left a lone high surrogate ending one
   message and an orphaned low surrogate starting the next — a broken glyph per
   boundary. Concatenating the messages still compares equal, which is why the
   existing preservation assertions passed straight through it. All boundaries
   now step back off a pair via `avoidSurrogateSplit`, including the per-call
   12k slices, which had the same latent exposure before rollover ever existed.

Both are pinned by tests that fail against the previous commit: a runaway guard
that turns a rollover loop into a failed test rather than a hung one, and a
per-message surrogate-edge assertion (the join-and-compare check cannot see it).
2026-07-29 19:13:55 -05:00
Benjamin Taylor 23b67bd4a2 fix(channels-slack): split long native-stream replies across messages
Slack enforces a cumulative text cap per streamed message, not just the
documented 12k-per-`markdown_text`-call cap. `NativeMessageStream` only honored
the per-call cap and kept appending to a single message, so once a reply passed
the cumulative ceiling every further `chat.appendStream` was rejected with
`msg_too_long` — permanently, for the rest of the run.

Observed in dev: a ~23k-char reply streamed 11,607 chars into one message and
then failed every subsequent append. The remaining text was silently dropped
(the append error is swallowed by design), the turn never reached finalize, and
the egress operation replayed from its uncheckpointed cursor — posting a fresh
truncated copy per attempt until it hit max_attempts.

`flushText` now rolls over: when a message fills and text remains, the boundary
is frozen on a line (else word) break, any open markdown construct is closed,
the message is finalized, and the reply continues in a new message that
re-opens that construct. Both append loops share one rollover-aware path, and a
continuation `startStream` failure no longer falls back to the legacy transport
— that sink replays the whole buffer, which would duplicate everything already
streamed.

The soft per-message limit is 11k, below the smallest total Slack was observed
to accept, leaving headroom for the closers a boundary appends.

The test transport did not model the cumulative cap at all, which is why this
survived the suite; it does now, and the case that asserted "keeps a long reply
in ONE message" was asserting the bug.
2026-07-29 19:04:14 -05:00
Mike Ryan d5beb8e5de fix(channels): enforce live-session delivery boundaries 2026-07-29 16:42:17 -07:00
Mike Ryan 538277ef4c fix(channels): hide reaction provider references 2026-07-29 16:42:17 -07:00
Mike Ryan c07b8312db fix(channels): close managed provider effect surface 2026-07-29 16:42:16 -07:00
Mike Ryan f1c4b6ca79 fix(channels): keep managed Slack replies atomic 2026-07-29 16:42:04 -07:00
Mike Ryan 74b38ecdfb fix(channels): await delivery thread operations 2026-07-29 16:42:04 -07:00
Mike Ryan f658564f7f fix(channels): declare gateway session protocol 2026-07-29 16:42:04 -07:00
Mike Ryan 624bf3c360 fix(channels): cancel drained canonical runs 2026-07-29 16:42:03 -07:00
Mike Ryan ccf8ac5ea4 fix(channels): preserve managed interaction references 2026-07-29 16:42:03 -07:00
Mike Ryan 66ac03f770 fix(channels): reject managed slash runs and awaitChoice 2026-07-29 16:42:03 -07:00
Mike Ryan 78747d622d fix(channels): settle effects before closing runs 2026-07-29 16:42:03 -07:00
Mike Ryan c67458b9a2 fix(channels): present delivery join code 2026-07-29 16:42:02 -07:00
Mike Ryan 243caa2ed7 fix(channels): preserve one canonical run lifecycle 2026-07-29 16:42:02 -07:00
Mike Ryan e60518ee2c fix(channels): harden live-session transport retries 2026-07-29 16:42:02 -07:00
Mike Ryan 6a222953a9 fix(channels): align umbrella with managed launcher 2026-07-29 16:42:02 -07:00
Mike Ryan cd6fb9e6ce fix(channels): isolate managed render imports 2026-07-29 16:42:02 -07:00
Mike Ryan 5ddcb14233 refactor(channels): use gateway live sessions 2026-07-29 16:42:01 -07:00
Mike Ryan af67df039e feat(channels): define live-session protocol 2026-07-29 16:40:47 -07:00
Adrien Pouligny 204d96fabe Merge branch 'main' into fix/runtime-deprecated-uuid-dependency 2026-07-29 16:11:19 -07:00
Mike Ryan 7c2a13c8a0 Fix channels intelligence CI build (#6241)
## Problem

Main CI cannot compile the channels-intelligence test suite because
ReplyTarget and Sub are referenced without declarations.

## Why

The merged Slack tool-status regression coverage introduced the
references but omitted the corresponding type declarations. Every
dependent Nx workflow fails during the shared package build.

## Fix

Restore the ReplyTarget type import and the local Sub subscriber helper
type in render-events.test.ts.

Validation: pnpm exec nx run @copilotkit/channels-intelligence:build;
pnpm exec nx run @copilotkit/channels-intelligence:test; pnpm exec nx
run @copilotkit/channels-intelligence:check-types.
2026-07-29 15:58:30 -07:00
Tyler Slaton 2bf86342b2 fix(channels): restore missing test declarations 2026-07-29 18:57:16 -04:00
Tyler Slaton 0ea0c1bd2e fix(channels): hide Slack tool status by default (#6204)
## Problem

Slack threads render every tool invocation as visible progress, leaving
noisy `Used …` rows ahead of the final answer.

## Why

The direct Slack renderer treated tool status as enabled when
`showToolStatus` was omitted, while the managed Intelligence renderer
always emitted tool lifecycle frames. The managed renderer's opt-in was
not reachable through the supported `createChannel()` and
`CopilotRuntime` API.

## Fix

- Hide tool-call progress by default for direct and managed Slack
routes.
- Expose `createChannel({ showToolStatus: true })` as the managed
Channels opt-in and carry it through runtime activation to the
Intelligence launcher.
- Preserve `slack({ showToolStatus: true })` as the direct Slack opt-in.
- Keep existing managed behavior for non-Slack routes.
- Continue capturing and executing tool calls while their status frames
are hidden.
- Add regression coverage for the public managed API, legacy and
native-streaming Slack, and Realtime Gateway paths.

Paired with https://github.com/CopilotKit/Intelligence/pull/632.
2026-07-29 15:50:15 -07:00
Ben Taylor 853ef42394 fix(channels): treat a revoked delivery lease as fenced and make gateway outages diagnosable (refs OSS-670) (#6240)
Three fixes from the OSS-670 investigation, where a dev gateway restart
landed mid-turn: the agent's answer was produced and then thrown away,
the end user got a generic error, and the bot went silent for 19 minutes
with nothing in its logs.

## A revoked lease is a fenced outcome, not a turn failure

`CHANNEL_DELIVERY_LEASE_INVALID` means another owner holds the delivery
— app-api fences every intent on `leased_until > now()` AND the
lease-token hash, so nothing we send can be accepted and redelivery is
app-api's job.

- The guard lives in `IntelligenceAdapter.dispatch`, the site that
actually runs. An earlier revision guarded only the realtime transport,
which is unreachable in production because `dispatch` catches every turn
error and nacks first — so a fenced delivery still put a doomed
`channel.delivery.fail.v1` on the wire, and that second 409 is what
surfaced as `nack after turn failure failed` followed by `no delivery
state`.
- The code is no longer retried through the 3-attempt render-batch
ladder.
- A `fail` intent that is itself fenced is logged and dropped rather
than rejected.
- The predicate lives in `transports.ts` because both transports surface
the same code (the HTTP path as a 409 body).

The adapter-dispatch test is a real regression test: reverting the guard
makes it fail on `channel.delivery.fail.v1` being pushed.

### Abandoning is a third outcome, not a silent one

Swallowing the coded error means the `DeliverySource` observes a
**successful** `onDelivery` — so neither its own revoked-lease branch
nor `ack` runs, and those are the only two places the per-delivery lease
record is dropped. Without more, every fence leaked a lease token,
delivery scope and accepted-seq high-water map for a delivery this
runtime can never speak for again (app-api may already have handed it to
another runtime, so the id may never be seen here again).

`DeliverySource.abandon(deliveryId, reason)` is that third outcome
alongside ack/nack: retire local state, send nothing. It is synchronous
and wire-free by design, optional on the interface so existing
implementations stay valid, and implemented on
`RealtimeGatewayTransport` (`deliveries`), `HttpDeliverySource`
(`leases`) and `InMemoryDeliverySource`. The transport's internal
revoked-lease catch routes through it too, so there is one retirement
path rather than two.

Both wiring tests fail with the `abandon` call reverted — the adapter
test on an empty `abandoned[]`, and the composed adapter+transport test
because its probe `nack` pushes a second, doomed
`channel.delivery.fail.v1` off the retained state.

## A refused reconnect is diagnosed instead of retried in silence

`socket.onError` returned early once the socket had connected, so a
refused *reconnect* recorded nothing and Phoenix retried forever with no
output. The cause is now recorded, and when the transport exposes no OS
code an HTTP probe of the endpoint runs once per outage.

The probe is unauthenticated, so it deliberately does **not** claim your
key is revoked — its 401/403 would look identical with a perfectly good
credential. It establishes reachability: 5xx reads as an
unhealthy/restarting gateway, anything else as "reachable, handshake
refused — verify the project API key".

## Honest status, loud while down

`gave_up` maps to status `error`, but Phoenix keeps retrying underneath
and a successful rejoin restores `online` — the log now says so instead
of reading as terminal. The drop cause travels with each state
transition (it was previously `{ meta: undefined }`), and a "still down"
line repeats while the session is out, including after `gave_up`.
Channel status semantics and the "manager never re-activates on a drop"
contract are unchanged.

## Not included

Delivery-lease renewal has no owner across a gateway restart — draining
stops the channel process that renews it, so a long turn loses its lease
to expiry regardless. That, and bounding an in-flight turn whose lease
is already dead, are follow-ups on the ticket. Retiring the local record
(above) stops the leak; it does not keep the lease alive.

Tests: `channels-intelligence` 13 files / 244 tests; `runtime` 127 files
/ 1799 tests; `check-types` clean on both.
2026-07-29 17:43:28 -05:00
Benjamin Taylor 1a72c404ba fix(channels): retire the local lease state when a delivery is fenced (refs OSS-670)
`IntelligenceAdapter.dispatch` swallows a revoked-lease error so no doomed
`fail` intent goes out, which means the DeliverySource observes a SUCCESSFUL
`onDelivery` — so neither the transport's own revoked-lease branch nor `ack`
runs, and those are the only two places the per-delivery lease record is
dropped. Every fence therefore leaked a lease token, delivery scope and
accepted-seq high-water map for a delivery this runtime can never speak for
again (app-api may already have handed it to another runtime, so the id may
never be seen here again).

Adds an explicit `DeliverySource.abandon` — the third outcome alongside
ack/nack: retire local state, send nothing. Implemented on both real sources
(`RealtimeGatewayTransport.deliveries`, `HttpDeliverySource.leases`) and on
`InMemoryDeliverySource` so tests can assert it; the transport's internal
revoked-lease catch now routes through it too. Optional on the interface so
existing implementations stay valid.

Both new wiring tests fail with the `abandon` call reverted: the adapter test
sees an empty `abandoned[]`, and the composed adapter+transport test's probe
`nack` pushes a second, doomed `channel.delivery.fail.v1` because the
DeliveryState was still retained.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 17:26:40 -05:00
Tyler Slaton cc45368329 test(channels): cover managed tool status launch 2026-07-29 18:24:14 -04:00
Tyler Slaton 2f095fd8c9 fix(channels): preserve managed Slack tool history 2026-07-29 18:15:42 -04:00
Tyler Slaton c1329f55ee chore(changesets): remove OSS-636 entry 2026-07-29 18:14:26 -04:00
Tyler Slaton 124f4e29ef fix(channels): expose managed tool status option 2026-07-29 18:14:25 -04:00
Tyler Slaton 6c7d45aa8e fix(channels): hide Slack tool status by default 2026-07-29 18:12:04 -04:00
Benjamin Taylor 4890c524d3 fix(channels): abandon a fenced delivery where the turn actually fails (refs OSS-670)
Review found the fenced-lease guard sat only in the realtime transport, while
IntelligenceAdapter.dispatch catches every turn error and nacks first — so a
revoked lease still put a doomed channel.delivery.fail.v1 on the wire and the
earlier test only passed because it stubbed onDelivery. The guard now lives in
dispatch too, and the predicate moved to transports.ts because both transports
surface the same code (the HTTP path as a 409 body).

Also from review:
- Clear lastTransportError/lastTransportCode on recovery, so a later outage that
  reports nothing is no longer attributed to the previous outage's error.
- Fence the endpoint probe on an outage generation, so a verdict that resolves
  after the link healed cannot latch onto the next, unrelated outage.
- Stop inferring a revoked credential from any HTTP status. The probe is
  unauthenticated, so its 401/403 says nothing about our key; it can only
  establish reachability. 5xx now reads as an unhealthy/restarting gateway.

The new adapter-dispatch test is a real regression test: with the dispatch guard
reverted it fails on channel.delivery.fail.v1 being pushed.
2026-07-29 16:43:04 -05:00
Benjamin Taylor af7b64271b feat(channels): log why a managed session dropped and keep logging while it is down (refs OSS-670) 2026-07-29 16:01:24 -05:00
Benjamin Taylor 9cc8d65dad feat(channels): diagnose a refused gateway reconnect instead of retrying silently (refs OSS-670) 2026-07-29 15:57:02 -05:00
Benjamin Taylor 6c0b533b7d feat(channels): carry the drop cause on managed-session state transitions (refs OSS-670) 2026-07-29 15:52:40 -05:00
Benjamin Taylor b509d7c5ad fix(channels): abandon a delivery whose lease was revoked instead of nacking it (refs OSS-670) 2026-07-29 15:47:57 -05:00
Benjamin Taylor 51cd36c4dc fix(channels): stop retrying render batches against a revoked lease (refs OSS-670) 2026-07-29 15:42:05 -05:00
Nathan 🔶 Tarbert b8d17a28d1 Revise supported platforms in README
Updated supported platforms and added new messaging services.
2026-07-29 16:33:55 -04:00
Maxim 217bb9bc5b feat(inspector): Capabilities tab + memory recall; migrate banking off Glass Engine (#5986)
## Summary

Migrates the banking showcase off its bespoke "Glass Engine" inspector
onto the productized `@copilotkit/web-inspector`, and builds out the two
inspector features that migration requires: a **Capabilities tab**
(enable/disable frontend tools + A2UI catalog components, with real
enforcement) and a **Memory recall** view backed by a new `recall()` RAG
read path and a project-scoped realtime channel.

This is the CopilotKit half of a cross-repo effort; the Intelligence
half landed in Intelligence#557.

## What's included

**Capabilities (tool + catalog enable/disable, real enforcement)**
- `core`: `RunHandler` tracks per-tool and per-catalog-component
enable/disable overrides; `buildFrontendTools` filters disabled tools
out of the tool list sent to the agent; `core.ts` exposes
`setToolEnabled` / `setCatalogComponentEnabled` / `catalogComponents`.
- `react-core`: catalog toggling enforced on **both** the context and
render paths (`filteredCatalog`).
- `a2ui-renderer`: `filterCatalog` preserves `themeSchema` and
`functions` when dropping disabled components.
- `web-inspector`: new **Capabilities** nav item + view with reactive
toggles (gated on presence of tools/catalog).

**Memory recall (RAG read path)**
- `core`: `memory.recall()` with score projection; new
`project_meta:memories:{code}` realtime channel that surfaces
project-scoped memories alongside user memories, degrading to user-only
when project creds are absent.
- `runtime`: `/memories` recall handler + project-join creds threaded
through the `/memories/subscribe` proxy, gated behind the opt-in
`exposeMemoryRoutes` server flag (**default OFF**).
- `web-inspector`: Learning tab renamed **Memory**; recall section with
relevance bars and scope badges.

**Banking showcase migration**
- Enables the product web-inspector via `showDevConsole`; removes the
bespoke inspector, Glass Engine deployment gate, and dead memory routes.
Capabilities tab is **ungated** in this demo.

## Review / quality

- Full MSAL CR loop (6 modules + cross-cutting + 7-agent final gate)
converged with **zero mandatory findings**; Procedure 3 bucket-(c)
promotion audit returned zero promotions.
- 4 real bugs fixed during review: recall limit/query validation, a
broken fired-capability dot, catalog registration guard consistency, and
`filterCatalog` themeSchema preservation.
- Test coverage added across all touched packages (memory recall,
capability toggles, catalog filtering, recall UI).

## Follow-ups (tracked, not in this PR)

Project-channel observability (give-up monitor / bounded rejoin warns),
memory type-design hardening (recall field-validation parity,
`SaveMemoryResponse` discriminated union), and capabilities consistency
(`setToolEnabled` subscriber-event parity). None are load-bearing for
the shipped surface.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-07-29 21:20:03 +02:00
David McKay 819a8bccf4 Merge branch 'main' into blitz/glass-inspector/integration 2026-07-29 12:07:58 -07:00
Tyler Slaton 028ad74b93 docs: clarify the managed Channels setup journey (#6237)
## Summary
- restore provider-scoped Channels overview pages for Slack and
Microsoft Teams
- separate Intelligence configuration from the connect-and-run guides
and make the handoff explicit
- preserve backend selection across Channels navigation, search,
sitemap, and LLM routes
- update the Channels architecture diagrams and Intelligence setup
screenshot
- retain framework-specific connection setup for all public backends

## Why
The merged Channels docs mixed platform configuration and runtime setup,
removed the overview entry point, and exposed ambiguous or stale
navigation. This reorganizes the existing guides into a clear sequence:
**Overview → Configure in Intelligence → Connect and run**.

## Validation
- `pnpm exec oxfmt --check` on changed TS, TSX, and MDX files
- `npm run lint` in `showcase/shell-docs`
- `npm run typecheck` in `showcase/shell-docs`
- `npm test` in `showcase/shell-docs` (51 files, 351 tests)
- `npm run build` in `showcase/shell-docs` (222 static pages)
- local browser checks for Slack and Microsoft Teams with Built-in
Agent, LangGraph, Mastra, and Deep Agents
2026-07-29 11:54:20 -07:00
Sam Julien 580021f856 docs(channels): clarify managed setup journey 2026-07-29 11:46:06 -07:00
Jordan Ritter 8433f5b118 fix(showcase/ci): make the :latest tag monotonic so an older build cannot overwrite a newer one (#6174)
## The race, with evidence

Three PRs merged within 34 seconds on 2026-07-26. `Showcase: Build &
Push` has **no concurrency group** (deliberately), so all three ran
simultaneously and raced to push the same `:latest` tags.

| run | commit | PR | start → end |
|---|---|---|---|
| `30190815370` | `7b28934387` | #6162 | 06:18:05 → 06:29:44 |
| `30190823203` | `59f275eedc` | #6161 | 06:18:21 → 06:29:44 |
| `30190831480` | `db75a04837` | #6158 | 06:18:39 → **06:29:13** ←
newest, finished FIRST |

The newest commit finished first, so the older builds overwrote its
`:latest`. Per-service job completion times — older beating newer on
every shared slot:

| service | newer (`db75a04837`) | older (`59f275eedc`) | older won by |
|---|---|---|---|
| `shell-dashboard` | 06:25:33 | 06:25:34 | +1s |
| `showcase-harness` | 06:27:51 | 06:27:55 | +4s |
| `shell` | 06:27:17 | 06:27:28 | +11s |
| `shell-dojo` | 06:25:10 | 06:25:26 | +16s |

**All three runs reported `success`.** Staging served pre-#6158 code
while CI, the redeploy gate and deploy verification all looked clean.
Same failure class as #6171: a success that doesn't mean what it says.

## What I verified in YAML vs took on trust

Verified by reading the files / querying the API:

- **Tagging** — `showcase_build.yml` pushed `:latest` **and** `:${{
github.sha }}` in one `depot/build-push-action` step, in both the
`build` and `build-starters` matrices. **A per-commit sha tag already
existed**; confirmed in GHCR (`showcase-shell-dashboard` has digests
tagged `db75a04837…`, `59f275eedc…`, `d28384a2eb…`).
- **Nothing serialized the pushes.** No concurrency group; confirmed the
header comment states this is intentional.
- **Deploy consumes `:latest`** — `verify-railway-image-refs.ts` is the
SSOT assertion: staging is `ghcr.io/copilotkit/<repo>:latest` (mutable),
**prod is `ghcr.io/copilotkit/<repo>@sha256:<digest>` (already immutably
pinned)**. So this race is **staging-only**; prod was never exposed.
- **`Showcase: Verify Deploy` structurally cannot catch it.** It is a
health probe; it asserts no digest or commit provenance anywhere. Its
#6171 per-commit concurrency key is about *which run verifies*, not
*what image is running*. A stale-but-healthy service passes.
- **The racing runs build DISJOINT service sets** (see below) — I pulled
the actual job lists.

Taken on trust: nothing material. The issue description matched the API
on every point I checked.

## Why NOT a concurrency group

`detect-changes` builds a **per-push, path-filtered** matrix, so
concurrent runs build overlapping but **non-identical** service sets:

- `7b28934387` → ag2, agno, built-in-agent, claude-sdk-python,
claude-sdk-typescript, crewai-crews, langgraph-fastapi,
langgraph-python, langroid, llamaindex, mastra, pydantic-ai, shell-docs,
spring-ai, strands (**15**)
- `db75a04837` → crewai-crews, llamaindex, shell, shell-dashboard,
shell-docs, shell-dojo, showcase-harness (**7**)

`cancel-in-progress: true` would have cancelled the `7b28934387` run and
the ~10 services **only it builds would never have shipped** — trading a
stale-image bug for a never-shipped bug. Per [GitHub's
docs](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax#concurrency),
`cancel-in-progress: false` is no better: *"Any previously pending job
or workflow in the concurrency group will be canceled"* — with three
rapid merges the **middle** commit's build is dropped outright. GitHub
also does not guarantee FIFO ordering of queued runs.

Concurrent runs here are **not redundant**, so they must not be
cancelled.

**Re the #6171 interaction:** this design does **not** use
`cancel-in-progress`, so no build is ever superseded-and-cancelled and
the cancelled-slot notifier is never tripped by this change. That
interaction stays theoretical — deliberately.

## The design

Make the one shared mutable resource monotonic instead of serializing
the fleet.

1. The build step pushes **only** the immutable `:<sha>` tag, plus an
`org.opencontainers.image.revision` label.
2. A guard resolves the commit behind the current `:latest`, asks GitHub
`compare/<theirs>...<ours>`, and advances `:latest` (registry-side
retag, no pull) **unless ours is `behind`** — i.e. `:latest` already
holds a descendant and moving it would roll staging back.

**Fails open by design.** No `:latest` yet, unlabelled legacy image,
unreachable API, diverged history → advance. A stuck `:latest` is the
very failure being fixed, so it declines only on *positive proof* of
regression.

**Placement:** in `redeploy-staging` / `redeploy-staging-starters`,
immediately before the Railway pull that consumes `:latest` — not per
build slot. Those jobs already have Node (build slots do **not**, so
per-slot would mean an unpinned `npx tsx` fetch on ~50 parallel
runners), and deciding right before the pull makes the window as narrow
as possible. The image list is the **same matrix ∩ build-success
intersection** that decides what gets redeployed, so a failed build can
never move a tag.

Also adds `showcase_build.yml` to `showcase_validate.yml`'s trigger
paths — the new test asserts against that file's live text, and without
the path a PR re-adding `:latest` would never run the test that catches
it.

## Tradeoffs / what stays open

- **Residual sub-second TOCTOU.** GHCR has no compare-and-swap on tags,
so two runs reading `:latest` simultaneously could still both advance.
This narrows the window from the whole build (~10 min) to inspect→retag.
Fully closing it means retiring the mutable staging tag and pinning
staging to digests the way prod already is — a change to the Railway
image-ref SSOT contract, not a workflow change. **Recommended
follow-up.**
- **The guard is inert for one build per image.** Today's `:latest`
images carry no labels (verified: `showcase-shell-dashboard:latest` has
no `Labels` at all), so the first post-merge build fails open and
advances unconditionally — same as today. Protection starts from the
second build of each image.
- Failure to retag exits non-zero, redding the redeploy job and stopping
the deploy. That is intended: redeploying against a tag that did not
move is exactly the silent false green being fixed.

## Proof

**Red/green on the live YAML.** `advance-latest-tag.test.ts` parses the
real `showcase_build.yml` (extending #6171's `redeploy-guard.test.ts`
pattern). Reverting the workflow to its pre-fix state: **14 failed / 23
passed**. With the fix: **37 passed**. Full `showcase/scripts` suite:
**2382 passed, 73 files**.

**The load-bearing predicate, verified live against the real incident
commits:**

```
compare/db75a04837...59f275eedc  => behind     (older run arriving late → DECLINE)
compare/59f275eedc...db75a04837  => ahead      (newer run → advance)
compare/db75a04837...db75a04837  => identical
```

**Label reading, verified against a real multi-platform registry image**
— `docker buildx imagetools inspect ghcr.io/astral-sh/uv:latest` piped
through `extractRevisionLabel()` returns
`3010295ae7ff572de459987ad70db315a62ecd61`, matching `jq` exactly. The
platform-keyed shape is handled.

**Shell/jq transforms** exercised directly, including the empty-CSV edge
case (empty → empty, step skipped by its `if:`).

**Lint:** `actionlint` finding counts byte-identical to the pre-change
baseline (no new findings; the 11 pre-existing are unrelated). `zizmor
--min-severity low` with the repo config: **no findings**.

**Typecheck:** both new files are in `showcase/scripts/tsconfig.json`'s
include set and produce **zero** errors. Worth stating plainly: `nx
run-many -t check-types` **does not reach `showcase/scripts`** — the
project isn't in the nx graph and has no `check-types` target (there are
9 pre-existing type errors in sibling files, which is how I confirmed
it). So the typecheck above is mine, not CI's. The *tests* are gated:
`showcase_validate.yml` runs bare `pnpm exec vitest run` in
`showcase/scripts`, which auto-discovers the new file.

### What I could NOT prove

**I did not construct a real concurrent race on scratch branches.**
Doing it faithfully needs two builds pushing the same GHCR repo with
controlled finish ordering, which means merging to `main` — the only
branch the build workflow triggers on. No run IDs for a live race
demonstration; I am not implying one.

Unproven until this runs on main: that `docker buildx imagetools create`
retags cleanly under the runner's GHCR credentials, and that `npx tsx`
behaves in the redeploy jobs (it is already the established invocation
there — `redeploy-env.ts` — so this is low risk, not zero).

## Normal single-merge builds are unaffected

No concurrency group is added, so nothing queues or cancels. A lone
merge finds `:latest` at its own parent → `ahead` → advances, exactly as
before. Cost is one `imagetools inspect` + one `gh api` + one
registry-side retag per built service, in a job that already exists — no
extra job, no extra checkout, no change to build parallelism.

---

Branched from `db75a04837`; #6156/#6159 landed after, so this will need
main merged in before it goes green.

Probable conflict with the concurrent `git lfs pull` work in
`showcase_validate.yml` — my edit there is only the top-level `on:
paths:` list, so it should merge cleanly, but flagging it.
2026-07-29 11:36:12 -07:00
Maxim 1c4676c687 Merge remote-tracking branch 'origin/main' into blitz/glass-inspector/integration 2026-07-29 20:19:10 +02:00
Mike Ryan 4687648376 feat(channels-intelligence): batch managed render output (#6221)
## What changed

- replace the managed single-frame render sink with immutable ordered
batches
- flush the first text at once and compact later text for up to 250 ms
- bound batches by frame, byte, and pending-lane limits
- add stable SHA-256 digests, batch IDs, durable high-water receipts,
and the `render_batch_v1` runtime capability
- update HTTP, Realtime Gateway, fallback, and in-memory transports to
the batch-only contract

## Why

Managed Channel latency currently grows with source delta count because
each delta takes one durable round trip. Batching makes durable work
track response time and semantic boundaries instead of token count.

Refs OSS-654. This PR follows the one-cutover contract in the companion
service plan, so it does not provide the older-server sequential
fallback described in that ticket.

Companion service PR:
https://github.com/CopilotKit/Intelligence/pull/637

## Validation

- `pnpm test` in `packages/channels-intelligence` — 223 passed
- `pnpm run check-types` in `packages/channels-intelligence`
- pre-commit Nx test, publint, and attw suite — passed
- 20-delta and 200-delta runs each produced two durable batches and
equal modeled receipt time

The service PR records the cross-repo E2E, load, fault, Kubernetes, and
image checks.
2026-07-29 09:58:00 -07:00
Mike Ryan 4c0577f996 fix(channels): release render lane flushers 2026-07-29 09:13:55 -07:00
Mike Ryan dbf4a43b89 fix(channels): bound paced render latency 2026-07-29 09:00:19 -07:00
Mike Ryan 26385feca6 fix(channels): address realtime batch review feedback 2026-07-29 09:00:19 -07:00