The gen-ui-interrupt d6 probe drives two interrupt turns in one thread.
Turn-2 failed because the langgraph-python/typescript fixtures ordered each
pill's toolCallId resume-leg BEFORE its toolName first-leg. aimock dedups on
userMessage (first-match-wins), so the text-only resume-leg shadowed the
tool-emitting first-leg on the duplicate userMessage — the agent emitted a
plain text reply with no schedule_meeting tool call, the interrupt never
fired, and the time-picker-card never mounted.
Reorder each pill so the toolName:schedule_meeting first-leg precedes its
toolCallId resume-leg, matching the passing mastra/pydantic-ai ordering. The
resume-leg still only matches when the last message is its tool result
(toolCallId guard), so confirmation still works.
PR #5226 added the "Starter" pseudo-category row-group (StarterSection,
using resolveStarterRow/buildStarterBadge) only to CellMatrix in
cell-matrix.tsx. But the live DashboardPage renders the matrix tab via
FeatureGrid (feature-grid.tsx); CellMatrix is reached only through
cells-view.tsx, which no live route imports — it is dead/legacy. So the
Starter row shipped in code but could never render (the served prod bundle
had zero StarterSection/starter-row-/"no starter" strings).
Port the StarterSection into FeatureGrid: render the four fixed sub-rows
(health/agent/chat/interaction) after the feature categories, resolving
each cell via the existing S3 helpers (resolveStarterRow + buildStarterBadge,
5-state vocab) and structurally excluded from the feature rollup/column tally
(never calls renderCell/buildCellModel). Aligned to FeatureGrid's table
structure (Feature col + optional parity ref-depth spacer + categoryColSpan).
Remove the duplicate from the dead CellMatrix and move its dedicated tests
to feature-grid.test.tsx, which exercises the live registry-backed grid.
Render proof: production next build now bakes starter-row-/starter-cell-/
"no starter for this integration" into the served page.js (client + server)
bundles. Full dashboard suite green (815 passed, 1 skipped).
The updateProverb useFrontendTool passed the live `agent` object in its
deps array. react-core's use-frontend-tool does JSON.stringify(extraDeps),
which throws "Converting circular structure to JSON" on agent.activeRunDetach$
(an RxJS Subject), crashing the React tree so no assistant bubble renders.
`agent`/`agent.setState` are stable references; only `state` needs to be a
dep. Also bump the docker ag-ui-crewai override to >=0.2.0,<0.3.0 to match
agent/requirements.txt and the showcase backend.
Swap the boilerplate "Create Next App" / "Generated by create next app"
title and description in each starter layout for a per-framework
"<Framework> + CopilotKit Starter" title and matching description.
crewai-flows hardcoded the agent URL with no AGENT_URL env override; add
the fallback to match siblings. agno and llamaindex concatenate a path
suffix onto the base URL, so strip any trailing slash first to avoid a
double slash when AGENT_URL ends in "/".
The welcome message promised theme/proverb/weather tools, but the page
registered none of them (only a debug console.log of agent.state).
Mirror the crewai-flows sibling to wire setThemeColor, the proverbs
shared-state tool, and a weather render tool, remove the leftover
console.log effect, and drop the stray w-screen overflow.
The agno and llamaindex add_proverb frontend tools closed over a stale
`state` snapshot with no dependency array, so rapid successive adds
dropped earlier proverbs. Read agent.state at call time inside the
handler instead. Also drop the stray w-screen on the llamaindex page
that caused horizontal scrollbar overflow.
The smoke build path (docker/Dockerfile.agent) runs `uv sync` against
agent/pyproject.toml + agent/uv.lock, but those still pinned
ag-ui-langgraph==0.0.34 (and the lock still pinned copilotkit==0.1.87).
copilotkit 0.1.93 requires ag-ui-langgraph[fastapi]>=0.0.35, so uv sync
failed with "No solution found" and the langgraph-fastapi smoke job went
red. The prior fix only bumped the production Dockerfile's
`uv pip install`, which the smoke path does not use.
Bump pyproject to ag-ui-langgraph[fastapi]==0.0.37 and regenerate uv.lock
(copilotkit 0.1.87->0.1.93, ag-ui-langgraph 0.0.34->0.0.37). Verified by
building docker/Dockerfile.agent locally: uv sync now resolves and the
agent boots ("Uvicorn running on 0.0.0.0:8123" / "Application startup
complete") serving HTTP 200 on /health.
Also reconcile parity: strands-python's entrypoint.sh legitimately diverges
from the langgraph-python north-star (it boots the strands agent via
`uv run python main.py` and serves Next via `next start`, not serve.py +
standalone server.js). Add entrypoint.sh to strands-python's
allowedDivergence, mirroring how langgraph-js already declares its own
custom entrypoint. parity:check now reports 0 errors.
The entrypoint was copy-pasted from langgraph-python: it logged
'langgraph-python starter', ran 'python serve.py' (no serve.py exists in this
starter), and started Next via standalone 'node server.js' (this image is a
non-standalone .next + node_modules build, so server.js does not exist either).
The container crash-looped on boot with
"python: can't open file '/app/serve.py'".
Run the Strands agent's self-serving agent/main.py via its uv venv
(AGENT_PORT=8123) and serve the frontend with 'next start'. Verified by booting
the image: agent reaches 'Application startup complete' on :8123 and the
container serves HTTP 200 on /.
copilotkit 0.1.93 re-exports StateStreamingMiddleware/StateItem FROM
ag_ui_langgraph.middlewares.state_streaming, which only exists in
ag-ui-langgraph>=0.0.35. The Dockerfiles installed copilotkit with --no-deps
and pinned ag-ui-langgraph[fastapi]==0.0.22, so the middlewares submodule was
missing and the agent crash-looped at boot with
'ModuleNotFoundError: No module named ag_ui_langgraph.middlewares'.
Bump the pin to 0.0.37 (the version copilotkit 0.1.93 resolves with deps).
Verified by booting the langgraph-python image: the agent now reaches
'Application startup complete' / 'Uvicorn running on 0.0.0.0:8123' with no
ImportError, and the container serves HTTP 200 on /.
The strands-python frontend imports `showcaseConfig from "../../showcase.json"`
(via src/hooks/use-example-suggestions.tsx) but the Dockerfile frontend stage
never copied showcase.json into the build context, so `next build` fails with
"Module not found: Can't resolve '../../showcase.json'". Add the COPY step,
mirroring the langgraph-js/langgraph-python Dockerfiles that already copy it.
The @copilotkit/react-core v2 pin (1.59.3) was already correct and matches all
working v2 starters; no package.json change was needed.
The build-starters Docker images hardcoded copilotkit==0.1.78 in the
langgraph-python and langgraph-fastapi Dockerfiles, but their agents import
StateStreamingMiddleware and StateItem, which 0.1.78 does not export. This
crash-loops the Python agent on boot with an ImportError. Bump the hardcoded
pin to 0.1.93 (latest stable, re-exports both symbols, matches local
sdk-python). Align all three langgraph-family agent pyproject pins
(0.1.87 -> 0.1.93) for consistency and regenerate the strands uv.lock.
The d6-all-pills-e2e probe header was sized for a legacy "8 services x 4
features" single-round fleet (timeout_ms: 600000 / 10 min). Discovery now
enumerates ~18 showcase demo services, so at max_concurrency: 8 the fan-out
runs ceil(18/8) = 3 serialized rounds. Rounds 2-3 start late and execute under
CPU contention, stretching each service toward ~200s and blowing the 10-min
per-service budget — all 18 services went red with `driver timeout after
600000ms`.
Raise the outer cap to 1200000 (20 min), matching the sibling 18-service
e2e-demos probe (same fleet, same budget). 3 rounds x ~200s fits comfortably.
Timeout-only change: max_concurrency stays at 8, so D6 peak stays at 32
contexts (under BROWSER_POOL_MAX_CONTEXTS=40, leaving 8 for the offset D5
tick). Header comment rewritten to document the real fleet size and the
fan-out math.
showcase_build.yml's existing detect-changes job shared the workflow_dispatch
`service` input with the new detect-starter-changes job, so dispatching
`service=starter-<slug>` tripped detect-changes's fail-loud `exit 1` ("did
not match any entry in ALL_SERVICES") and reddened the run even though
build-starters published fine. Scope the showcase fail-loud to non-`starter-*`
inputs via a case statement (mirroring how the starter job scopes its own
fail-loud) so a `starter-*` dispatch resolves to an empty showcase matrix and
SKIPS; typo'd showcase service names still fail loud.
Restore the 6h `schedule` cron in test_smoke-starter.yml. The PR had removed
it, leaving NO post-merge floating-dependency breakage detector for starters
— and its harness-probe replacement depends on S5 Railway services that don't
exist yet. Keep the cron until S5 starter-service probing is confirmed live.
The unmapped-starter aggregate emitted `total: 0` while `failed` listed all
four levels — a broken count invariant. Set `total: STARTER_LEVELS.length`
to match. Add the two missing red-green unit tests: the `aborted` errorClass
branch (already-aborted ctx.abortSignal → levels classified `aborted`, not
`transport-error`) and the makeSideEmit writer-throw resilience (a writer
whose write() rejects is swallowed at error-level and must not fail the
aggregate tick).
Only counts (12) were asserted on each side, so a future slug rename on
one side would silently flip a column to grey "not-supported" while both
counts stayed 12. Add a dashboard-side test that fs-reads the harness
STARTER_TO_COLUMN value set (the producer-side remap lives in a separate
pnpm workspace, so it cannot be imported) and asserts SET-EQUALITY with the
dashboard STARTER_COLUMNS. Mirrors the harness starter-mapping-drift fs-read
pattern; reds if either side renames a column slug without the other.
The starter_smoke probe runs hourly (`schedule: "40 * * * *"`), but
STARTER_STALE_AFTER_MS was set to 13h with a comment claiming a 6h cadence
— ~13 missed hourly ticks before amber, defeating the intended two-miss
flip. Re-derive the window from the 1h probe period to 2.5h: strictly
> 2 periods (so two consecutive misses, last row ~3h old, flip amber) yet
< 3h (so a single missed/slow-wake tick, last row ~2h old, stays green,
absorbing a scale-to-zero cold-start). Reconcile both the staleness.ts and
starter_smoke.yml comments to the same hourly basis + 2.5h window in
lockstep (no more "6h" in the starter context). Extend the staleness tests
with explicit hourly-tick boundaries (1 miss → green, 2 misses → amber).
Also drop the inaccurate "hollow" from the not-supported ✗ comments — there
is no hollow render variant; the state renders as grey ✗ text.
showcase_build.yml: add detect-starter-changes + build-starters jobs building
each of the 12 starters from examples/integrations/<slug>/Dockerfile via Depot
(--platform linux/amd64) and pushing ghcr.io/copilotkit/starter-<slug>:latest +
:<sha>. The starter- prefix is disjoint from showcase-* so harness discovery
stays clean.
test_smoke-starter.yml: drop the 6h schedule cron (live signal now comes from
the harness probing deployed Railway services + the harness alert path); keep
the examples/integrations/** PR build-sanity gate + offline aimock smoke run.
Workflows slot (S4) of the starter-row-group spec (model B). Railway
provisioning (S5) is intentionally out of scope, gated on cost approval.
Builds the dashboard UI for the starter-smoke dimension: resolveStarterRow
(flat starter:<col>/<level> lookup) + buildStarterBadge implementing the full
5-state cell vocabulary (green ✓, red ✗ smoke-failed, amber ~ stale, gray ?
not-yet-run, grey ✗ not-supported keyed off the S1 mapping), the StarterSection
row-group, rollup-exclusion so starter rows don't skew column aggregates, and a
13h staleness window (> 2× the probe cadence).
UI slot (S3) of the starter-row-group spec (model B).
Adds the `starter_smoke` probe driver + config/probes/starter_smoke.yml that
fans out per-starter HTTP health/agent/chat/interaction levels, registers it
in the orchestrator, and exempts it from probe-config parity (its matrix
shape differs from the depth-dimension probes). Includes 13 driver unit tests.
Probe slot (S2) of the starter-row-group spec (model B): harness HTTP-probes
the deployed (sleepable) Railway starter services.
Introduces the `starter` probe dimension in harness types and the single
source of truth for the starter-slug→dashboard-column-slug remap
(STARTER_TO_COLUMN, starterToColumnSlug, STARTER_LEVELS) plus a drift-lint
test asserting every smoke-matrix starter is mapped or explicitly excluded
and every mapped column slug resolves to a real manifest directory.
Foundation slot (S1) of the starter-row-group spec (model B).
Commit 7ac3e59a5 ("D6 all-pills probe driver") replaced the
`e2eParityDriver` registration (kind `e2e_parity`) with `e2eFullDriver`
(kind `e2e_d6`) in BOTH branches of `registerAllProbeDrivers`. After that
commit NO driver registers kind `e2e_parity`, yet
`config/probes/e2e-parity.yml` still shipped.
The probe-loader hard-rejects any YAML whose `kind` has no registered
driver (`no driver registered for kind 'e2e_parity'`), so `e2e-parity.yml`
failed to load on every boot and was never scheduled — silently dropping
that probe family. The D6/parity dashboard dimension is now produced by the
`e2e_d6` (all-pills) driver, which emits the identical
`d6:<slug>/<featureType>` rows the dashboard reads by key prefix (plus a
`d6:<slug>` aggregate), so `e2e_parity` is genuinely superseded, not just
temporarily unregistered.
Delete the orphaned config so the loader no longer rejects it and the full
probe-config set loads clean. Add a loader test that exercises the REAL
shipped `config/probes` set against the REAL driver registry and asserts
every YAML loads with zero `probes.reload.failed` errors — the existing
orchestrator guard only checks a hardcoded kind list and never read the
on-disk YAMLs, so it missed this orphan.
RACE3b's post-shutdown `liveContextCount === 0` assertion is tautological —
shutdown() sets liveContextCount to 0 unconditionally — so it proves nothing.
Drop it; the load-bearing assertions remain (acquire rejects with the shutdown
sentinel, contextToBrowser.size === 0, the orphan context was closed).
Correct the RACE3a/RACE3b comments that claimed shutdown() awaits the tracked
orphan close (reverted): the straddled orphan is closed by the orphan guard's
fire-and-forget close and never lands in contextToBrowser.
The empty-drain shutdown case (inFlightRecycles + pendingLaunches both
empty) exits the drain loop and resolves BEFORE a parked open settles and
registers its close via closeContextTracked(), so the "shutdown resolved =>
everything closed" guarantee was never actually delivered and had no
discriminating coverage. Revert the openContextOn orphan guard's close back
to main's accepted fire-and-forget `void this.closeContext(...)` and remove
the closeContextTracked() helper; inFlightRecycles is touched only by the
recycle / self-heal paths again.
Also unify the shutdown-condition error string: shutdown()'s queued-waiter
rejection now says "BrowserPool is shut down" to match the at-entry guard and
the straddle-reject legs (launch/relaunch-failure strings are a different
condition and unchanged).
Item 1 (red-green for the REACHABLE new shutdown branches): RACE1 only
covered the openContextOn orphan-guard `isShutdown` term. RACE3 adds
coverage for the two branches it did not:
- RACE3a: a shifted waiter whose serve open straddles shutdown must be
REJECTED by serveNextWaiter's post-open `if (this.isShutdown)` leg
(not re-enqueued onto the cleared queue, not hung to timeout).
RED with that guard reverted (waiter never settles), GREEN with it.
- RACE3b: an acquire whose open straddles shutdown must REJECT via
acquire()'s OUTER-catch `if (this.isShutdown) throw` leg. RED with
that guard reverted (the acquire enqueues on the cleared queue and
the test times out), GREEN with it. Both also assert shutdown()
awaited the orphan close (item 2) — the straddled context is closed
and the pool maps are empty post-shutdown.
Item 5: relabel RACE2's comment + title so it is framed as a regression
lock for the PRE-EXISTING generation-guard exactly-once-rollback
invariant (passes against main unchanged), NOT a shutdown-race proof.
Also extends the `internals()` test accessor with `waiters` so the new
tests can assert the queue is empty (no stranded caller).
Item 2 (reviewer-7 MEDIUM): the openContextOn orphan guard's
`void this.closeContext(...)` under shutdown was fire-and-forget and
tracked in NO set that shutdown() drains, so shutdown() could resolve
while a context close was still in flight — violating the implied
"shutdown resolved => everything closed" contract upheld elsewhere via
the pendingLaunches / inFlightRecycles drains. Route the orphan close
through a new closeContextTracked() that registers the close promise in
inFlightRecycles (the SAME set shutdown's drain loop re-snapshots until
empty) and self-removes on settle. Reuses the existing tracked-set
mechanism — no new machinery; outside shutdown the add/delete is a
no-op since only shutdown's drain loop reads the set.
Item 1 decision (retry-leg guard): KEEP the acquire() transient-retry
leg's `if (this.isShutdown) throw`. Contrary to a reviewer note that it
is unreachable, it guards a DISTINCT, genuinely-reachable straddle
window: the outer catch already observed isShutdown===false, we
re-reserved and re-opened, and the pool can shut down WHILE the retry's
open is in flight (the orphan guard then throws into this catch).
Without it the acquire enqueues onto the already-cleared waiter queue
and hangs until timeout. Comment updated to assert that reachability
rather than imply it merely "mirrors" the outer guard.
Items 3 + 4 (serveNextWaiter straddle): correct the inaccurate comment
that claimed shutdown() "already rejected this waiter" — it cannot, the
waiter was shift()ed off this.waiters BEFORE shutdown's reject-loop ran,
so THIS branch is the sole settler of a shifted-then-straddled waiter.
Unify the shutdown error string on the at-entry guard's existing
"BrowserPool is shut down" (was "shutting down" only here).
A serveNextWaiter()/openContextOn() that shifted a waiter + reserved a
slot BEFORE shutdown could settle its newContext() AFTER shutdown()'s
close-pass. shutdown() drains inFlightRecycles + pendingLaunches, but a
fire-and-forget serve open is tracked by neither, so the freshly-opened
context landed in contextToBrowser/liveContexts on a torn-down pool — a
leaked context that is never closed. openContextOn's orphan guard
checked generation/recycling/isConnected but NOT isShutdown.
Fix: add an isShutdown term to openContextOn's post-await orphan guard
(treat shutdown like a recycle — close the just-opened context, roll
back the reservation, throw); bail serveNextWaiter before openContextOn
when isShutdown; and reject (not re-queue) a shifted waiter / straddling
acquire when the open throws under shutdown, so it cannot strand on the
already-cleared waiter queue. Preserves all #5185/#5221 behaviors.
Adds red-green coverage for the leak (RACE1) and a regression guard for
the acquire transient-retry -> concurrent-recycle -> orphan-guard
straddle (RACE2), which was verified to keep its reservation accounting
balanced via the existing generation guard (no overshoot).
The integration-demo parity-check CI job was red on main: langgraph-js,
strands-python, and langgraph-fastapi diverged from the langgraph-python
north-star on two tracked verbatim files.
Root cause is the partial revert in 3721e7b36 (Revert of #5151), NOT #5222.
The v2 API migration (23af69041) moved north-star and all instances to the
v2 surface. The revert rolled the *instances'* example-layout/index.tsx and
docker-route-override.ts back to the v1 API (@copilotkit/react-core,
@copilotkit/runtime) while leaving the north-star on v2
(@copilotkit/react-core/v2, @copilotkit/runtime/v2). That left the demos
internally inconsistent: each instance's real src/app/api/copilotkit route
already uses runtime/v2 on copilotkit 1.59.3, but its Docker route override
and example layout were stuck on v1.
Re-baseline forward by syncing the three instances to the v2 north-star via
`pnpm parity:sync --all`. Only the 5 drifting verbatim files change; no
package.json keys move (all instances already pin 1.59.3) and no
agent-surface or allowed-divergence files are touched. `pnpm parity:check`
is now green (0 errors across all instances).
## Summary
The showcase Railway promote CI flow could not promote the fleet: an
`all` promote was blocked end-to-end whenever a single service was red,
and the reported failure traced to `showcase-ag2` being chronically red
on staging.
Root causes and fixes:
- **ag2 crash-on-import (the red service).** `gen_ui_agent.py` carried
`from __future__ import annotations`, which stringified the `set_steps`
tool's `context_variables: ContextVariables` parameter into an
unresolved `ForwardRef`. AG2's tool-schema generation then raised
`PydanticUserError` at import time, so the process never came up and the
staging healthcheck failed on every deploy since 2026-05-31. Removed the
import (matching the working sibling agents) and added a regression test
that statically asserts the future-import stays absent
(version-independent) plus a live import check.
- **Promote loop was all-or-nothing.** The per-service loop ran under
`set -euo pipefail`, so the first failing service aborted the whole
`all` promote, leaving the rest unpromoted. Extracted the loop into
`showcase/scripts/promote-fleet.sh`, which attempts every service,
accumulates succeeded/failed sets, exits non-zero only after attempting
all, and exports `succeeded_csv`.
- **`verify-prod` defeated the best-effort design.** It was skipped on
any non-zero promote and verified the full requested set. It now runs
`if: !cancelled()` and scopes `--services` to the succeeded set.
- **Staging precondition blocked the fleet.**
`verify-staging-precondition` failed the whole `all` promote when any
one service was staging-red. It is now advisory — `promote` runs
regardless, and `bin/railway`'s per-service P2/P3 staging-green gates
authoritatively refuse red services while green services promote.
`notify` success keys on PROMOTE && PROD.
- **Regression tests now gate in CI.** Added a `shell-script-tests` job
(bats + shellcheck) to `showcase_validate.yml`, plus input-validation
hardening in the script (fail-loud on empty / all-empty CSV,
`RAILWAY_BIN` executability check, whitespace trim).
## Test plan
- [x] ag2 regression test passes in the 3.12 venv (`PYTHONPATH=".:src"
pytest tests/python/`) — 2/2
- [x] `promote-fleet.bats` — 12/12 (best-effort loop, succeeded_csv
export, empty/whitespace/missing-binary guards, digest forwarding)
- [x] `shellcheck promote-fleet.sh` clean; `actionlint` clean on both
workflows
- [ ] CI green on this PR
## Summary
Adds an automated, **concise** `#engr` Slack notification when a
CopilotKit release publishes (or fails) — modeled on the
internal-skills→#engr release pings, but **one message per release**,
not one per package. The monorepo publishes ~15 npm packages + angular +
the Python SDK; naive per-package posts would be spam, so a single post
summarizes the whole release.
- **New `notify` job** in `publish-release.yml` — runs after both lanes
(`always()`), gated to real release contexts (a `workflow_dispatch` or a
*merged* PR). Posts exactly one message via the existing org-wide
`SLACK_WEBHOOK_ENGR` secret, if-guarded so an unset webhook never breaks
the step. Suppressed entirely for canaries (`mode=prerelease`) and
dry-runs.
- **Release intent is computed in the notify job from the `github.event`
payload** (dispatch inputs, merged `release/publish/*` head ref, and the
PR changed-files API for the Python SDK) — *independent* of the build
jobs. The build jobs emit their intent signals after failure-prone
steps, so a build that dies early on a genuine release would otherwise
leave the alert empty. Computing intent independently closes that
silent-swallow class. Failure arms **page on uncertainty** (fail toward
paging, never toward silence).
- **Pure, unit-tested message builder**
(`scripts/release/lib/build-release-notification.ts`) with a CLI wrapper
that serializes to `$GITHUB_OUTPUT` (random-delimiter heredoc, fail-loud
when it can't emit). Exhaustive truth-table coverage: npm/PyPI success +
failure arms, canary/dry-run suppression, cancelled-is-neutral,
empty-version/empty-scope graceful rendering, package-count
pluralization.
- **Self-watchdog:** if the notify job's own machinery fails, a
best-effort Slack post fires so the failure isn't invisible. The intent
step runs **first** (before checkout/install) so its outputs survive an
infra failure and the watchdog can still fire.
Publish/PyPI logic (the build/publish/build-python/publish-python jobs)
is untouched.
## Review
Converged through a 2-round, 7-agent unbiased code review (zero
mandatory in-scope findings remaining; the one in-scope finding — the
self-watchdog gating on a later step's output — was fixed by relocating
the intent step). Pre-existing publish-lane issues surfaced by reviewers
(prerelease canary verify-step, stable-tag retry idempotency,
canary→PyPI gating) are **outside this diff** and documented separately.
## Test plan
- [x] `vitest run` on the builder + wrapper suites — **85 tests green**
(builder 40, wrapper 27, versions 18)
- [x] `actionlint .github/workflows/publish-release.yml` — zero new
findings (4 pre-existing style findings in untouched jobs)
- [x] Both secret-using Slack steps if-guarded on `env.SLACK_WEBHOOK !=
''`
- [ ] CI green on the PR
- [ ] Observe a real `#engr` post on the next stable release
The starter had two App Router dirs: a stale top-level app/ containing only
an /api/copilotkit route (with a wrong graphId "starterAgent") and the real
src/app/ with page.tsx + layout.tsx. At copilotkit 1.59.3 the build silently
dropped the / route from src/app (masked by next.config ignoreBuildErrors),
producing only /api/copilotkit/[[...slug]]. The app served GET / as 404, so the
docker healthcheck (GET / === 200) never passed and the smoke leg failed with
the app container unhealthy before tests ran.
Removing the duplicate top-level app/ lets Next build src/app unambiguously,
restoring the / route. Local docker-compose.test.yml smoke: app healthy in ~5s,
4 passed (@health, @agent, @chat, @interaction).
The v2 migration renamed the Mastra agent key to `default` and updated
page.tsx/index.ts, but layout.tsx was missed: it still pinned
`agent="weatherAgent"` (now non-existent) and lacked
`useSingleEndpoint={false}`. The single-endpoint auto-detect races the
lazily-compiled API route, producing a 404 console error on page load
that fails the @interaction smoke leg's zero-console-error assertion.
Match the passing sibling starters (pydantic-ai/adk/ms-agent-framework-python).
Migrate adk, agno, crewai-crews, crewai-flows, mastra, pydantic-ai, ms-agent-framework-dotnet, ms-agent-framework-python, and llamaindex starters to the CopilotKit v2 `@chat` API and float them to 1.59.3 (package.json, runtime route, layout/page source, lockfiles).
The runtime CMD raised the soft nproc limit via
`/bin/sh -c "ulimit -u $(ulimit -Hu) ..."`, but on node:22-bookworm-slim
`/bin/sh` is dash, whose builtin `ulimit` does NOT support the `-u`
(max-user-processes) flag — it errors `ulimit: Illegal option -u`, which the
`2>/dev/null || true` then silently swallows. So #5185's intended PID
protection never applied; the soft limit stayed at the inherited default.
Run the CMD under `/bin/bash` (present at /usr/bin/bash) instead, whose
`ulimit -u` works. Verified against the exact base image under
`--ulimit nproc=512:4096`: bash raises the soft limit 512 -> 4096 (the hard
ceiling) while dash leaves it untouched and errors. Minimal change — only
the interpreter; the command, exec-as-PID-1, and `|| true` fallback are
unchanged.
A browser that is mid-`launch()` is not yet in `this.browsers` and has no
disconnect handler attached, so it is invisible to every teardown path
(shutdown's close pass, recycle, the disconnect handler). Under the
self-heal/relaunch storm introduced by the browser-pool hardening cluster
(#5174/#5185), a concurrent teardown therefore raced an in-flight launch:
the launch escaped teardown accounting entirely — either leaked, or (in
production) the browser was closed underneath the still-resolving
`chromium.launch()`, which then rejected with `browserType.launch: Target
page, context or browser has been closed` (SIGTRAP). The self-heal loop
relaunched forever, each relaunch hitting the same race (336
self-heal-launch-failed events in 4 min), wedging the pool.
Make launches atomic w.r.t. teardown: `launchBrowser()` registers its
launch promise in a new `pendingLaunches` set BEFORE awaiting
`rawLaunchBrowser()` and removes it on settle. `shutdown()` drains
`pendingLaunches` (alongside `inFlightRecycles`) before its close pass, so
it WAITS for every in-flight launch to settle instead of closing the
browser mid-launch. The launch seam re-checks `isShutdown` the instant the
launch settles: if a shutdown intervened it closes the freshly-launched
browser cleanly THEN, exactly once, and throws a shutdown sentinel so the
caller (init / recycle relaunch / self-heal) does not register a browser
into a pool that is going away. The launch-stagger gate still chains off the
raw result, so serialization semantics are unchanged. All #5185 behaviors
(backoff, self-heal, degraded/recovered alarms, waiter draining) preserved.
Adds a red-green test (FIX#13) reproducing an in-flight launch racing a
concurrent shutdown: pre-fix the launching browser escapes teardown
(closeCount 0, leaked); post-fix it is drained + closed cleanly exactly once
with no close-during-launch rejection.
Add a notify job that runs after both publish lanes via always() and
computes release intent directly from the github.event payload,
independent of the build jobs, so an infra failure in a lane can't
swallow the alert. The job posts the builder's rendered message to
#oss-alerts and includes a best-effort self-watchdog Slack post.
Add a pure builder that renders one concise #oss-alerts Slack message
per release from a unit-tested truth table: suppresses canary/dry-run
runs, gates npm/PyPI failure arms on event-derived release intent, and
pages on uncertain state. The CLI wrapper serializes env input to the
builder and writes the result to GITHUB_OUTPUT, failing loud on any
malformed or missing input.
The per-target partial-rollup `runWriter.update` (which advances the durable
`probe_runs.summary.{passed,failed}` counters) ran BEFORE `writer.write`
committed the corresponding `status`/`status_history` detail row, and a
`writer.write` failure is intentionally swallowed so one hiccup can't tank
sibling targets. That ordering opened a run-row-orphan window: the run row
could report `failed: N` for a target whose detail row was never durably
written — the stale-red ingestion artifact.
Reorder so the detail/status row is committed first and the run-row counter
is stamped only after, so the persisted counter can never outrun its backing
detail row. Pure statement-order swap — no schema/migration, dashboard read
path untouched, genuine failures still counted.
CopilotChatInput intentionally diverges while a run is in flight: Enter with
sendable text routes to SEND (the consecutive-interrupt unblock from #5195),
while the send/stop button always routes to STOP. Only a comment bound these
two behaviors together. Add a regression test asserting BOTH at once so a
future refactor cannot silently re-converge them (verified red against both a
button-sends-while-running and an Enter-also-stops mutation).
Replace the brittle `"activeRunCompletionPromise" in agent` probe + double
`as unknown as` casts in CopilotChat with a typed `RunCompletionAware`
contract plus an `isRunCompletionAware` type guard, both exported from core.
IntelligenceAgent now declares the property and implements the interface, so
the in-flight-run await is reachable without a cast and non-Intelligence
agents still degrade safely. Cast sites in the attachments/e2e tests and the
MockStepwiseAgent helper are updated to the typed accessor.
Factor the await-then-send logic into a shared `waitForActiveRunToSettle`
helper and call it from BOTH onSubmitInput and handleSelectSuggestion. This
closes the suggestion-path in-flight gap: selecting a suggestion mid-run no
longer pre-empts/aborts the active run (e.g. an interrupt RESUME) — the same
regression PR #5195 fixed for the typed-Enter path. Adds a red-green
regression test that parks the suggestion send on the in-flight promise.
## Summary
Reverts the four remaining Intelligence/threads merges, completing the
back-out started by #5215 (langgraph-fastapi) and #5216 (agent-spec):
- Revert #5211 — mcp-apps
- Revert #5205 — a2a-middleware
- Revert #5196 — pydantic-ai
- Revert #5151 — north-star foundation + batch 1 (incl. crewai-flows
#5189 and llamaindex #5199, which merged into it)
## End-state verification
- `git diff be20a389cf HEAD -- examples/integrations` is **empty** —
every integration example is byte-identical to pre-#5151 main
(`be20a389cf`, the merge's first parent).
- `scripts/__tests__/integration-intelligence-migration.test.ts`
(introduced by the rollout) is deleted.
- Total scope vs main: examples/integrations + that test file. Nothing
else touched.
## Heads-up: collateral on two follow-up commits
@jpr5 — your agent-key standardization commits (4bf50c7c41, 2103d0c875)
were written against the post-#5151 trees (they update `agentId` refs on
ThreadsDrawer/CopilotChatConfigurationProvider etc.), so this revert
necessarily takes those files back to their pre-#5151 content, undoing
the rename. Keeping the rename only where it auto-applied would have
left broken half-states (e.g. adk route registering `default` while the
restored layout pins `agent="my_agent"`). If you still want the rename
on the pre-threads files, it needs a fresh pass on top of this.
Also note: #5151 carried the starter-smoke heals (v2 label keys, 1.59.1
bumps, `useSingleEndpoint`, parity sync). Reverting it returns main's
starter smoke to its pre-#5151 state (9 starters red).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Jordan Ritter