Cross-session review fixes for the --isolate machinery (one concern:
source + test + docs).
1) Reaper reserved-name guard (critical): _reap_isolate_slot trusted
slot records — a record naming 'showcase' (corrupt, or written by an
older CLI version before apply_isolation reserved the name) passes
the charset regex, so the reap ran `docker compose -p showcase down
--remove-orphans --volumes` against the LIVE default stack,
destroying the PocketBase named volume. The reserved name now gets
the same treatment as the path-traversal guard: warn (naming the
record and why it is dangerous) and leave the slot intact for manual
inspection — no compose-down, no state removal.
Call-site enumeration: _reap_isolate_slot's sole caller is
_sweep_isolate_slots, at 3 sites (dead-PID reap, project-recorded/
no-owner reap, age-fallback reap), all passing
"$slot_entry" "$slot_proj" — all three flow through the new guard
identically.
Red-green: the new bats test ("a slot whose project record reads the
RESERVED 'showcase' is left intact...") was run against the UNFIXED
code first and FAILED — the sweep logged "Attempting to reclaim
stale slot 0 (project showcase has no live containers and no
recorded owner)" and reaped the slot. It passes with the guard.
2) .iso-bak restore race: two concurrent runs can both see a stale
backup; the loser's mv is the FINAL command of its `[ -f ] && mv`
AND-list, so its failure trips set -e and kills the CLI pre-claim
with a raw error. Both mv's now carry `2>/dev/null || true` — the
survivor's restore wins, the loser proceeds with restored originals.
3) Keep-test absence regexes greped only the `--project-name <name>
down` spelling; the reaper's own downs use `-p <name> down`, so a
keep-branch regression via the -p form passed undetected. Both keep
absence assertions now match `(--project-name|-p) <name> down`.
Mutation-verified: a temporary -p-form compose-down added to the
keep branch made BOTH broadened tests FAIL; reverted, suite green.
(All other absence assertions use the word-matched generic
`compose ... down` regex, which already covers both spellings.)
4) RUNBOOK.md/DEBUGGING.md contradicted shipped code: the manual
teardown was quoted without --volumes plus notes claiming
`down --remove-orphans` leaves named volumes (the shipped survival
notice and every teardown path include --volumes), and the name rule
was documented as `[a-z0-9_-]+` (actual: starts with [a-z0-9], then
[a-z0-9_-], uppercase normalized with a warn, 'showcase' reserved).
Both updated to the shipped semantics; the now-redundant separate
`down --volumes` snippets removed.
Verification: full `bats showcase/scripts/__tests__/` green (60 tests);
shellcheck on _common.sh shows no new warnings vs baseline
(pre-existing SC2034/SC2115 only, line-shifted).
Confirmation-CR bucket-(a) fixes for PR #5353 — one coherent concern:
env-name resolution has exactly ONE authority (the ENV_IDS /
ENV_ID_BY_NAME registries) and SSOT accessors fail loud instead of
silently returning wrong values.
- runRedeploy: resolve envId via ENV_ID_BY_NAME with an Object.hasOwn
guard + fail-loud throw listing the registered envs. Removes the
hardcoded `prod`/`staging` pair check and PRODUCTION/STAGING ternary
that contradicted the SSOT's documented open-env contract ("a new env
needs only a registry entry"); the registry lookup subsumes it.
- resolveEnv: derive resolution entirely from the registries (ENV_IDS
spellings -> env-id -> canonical ENV_ID_BY_NAME name) instead of its
own hardcoded synonym chain. Behavior identical for
prod/production/staging; still throws on unknowns, and now also
throws on a mis-wired registry (a spelling whose env-id has no
canonical name).
- serviceForDispatchName: fix the docstring's false "CI-built service"
claim — it does no ciBuilt filtering and tests pin the unfiltered
behavior (the non-CI-built webhooks resolves).
- repoNameFor: fail loud (consistent with instanceIdFor/domainFor)
instead of silently echoing the service name — the exact
silently-wrong-GHCR-name class this PR's hardening targets. Throws on
unknown service, on an env not registered in ENV_ID_BY_NAME
(unnormalized synonyms like "production"), and on a registered env
the service does not declare. Keeps the documented default (the
service name) for declared envs without an override.
Call-site enumeration confirming nothing relies on the old fallback:
- verify-railway-image-refs.ts:523 — iterates the entry's DECLARED
environments keys, registry-filtered, SSOT-matched service names
- __tests__/railway-envs.golden.test.ts:81 — iterates envsFor(name)
(declared envs only) over real SSOT keys
- railway-envs.test.ts repoNameFor cases — dual-env services,
prod/staging only
- __tests__/verify-railway-image-refs.test.ts:264 — FIVE_NEW keys,
all dual-env
- resolveTargetServices: throw when an explicitly-provided services
list resolves to zero entries (whitespace-only programmatic input)
instead of letting runRedeploy exit 0 having redeployed nothing; the
default undefined -> full CI-built scope is unchanged.
- makeLiveRedeploy: add signal: AbortSignal.timeout(30s) so a hung
Railway API records a per-service FAIL instead of stalling CI, and
pass GraphQL errors[].message through sanitizeErrorBody for
consistency with the HTTP-error path. Exported for direct unit tests.
Red-green: 11 new tests (open-env registry resolution incl. a
runtime-registered hypothetical env, repoNameFor negatives,
empty-resolution throw, abort-signal presence, GraphQL error
sanitization) all failed against the old code; full showcase/scripts
suite green (50 files, 1807 tests) + tsc --noEmit clean.
## Summary
Combined landing of three browser-verified showcase work waves (37
commits, per-integration grouping preserved), plus a full multi-round
code review with all mandatory fixes folded in.
**Wave 1 — @ag-ui currency (9/9 integrations).** Bump `@ag-ui/*`
frontend deps to exact `0.0.55` for: llamaindex, agno,
claude-sdk-python, pydantic-ai, strands, claude-sdk-typescript,
ms-agent-python, ms-agent-dotnet, google-adk. Exact pins land across 13
`package.json` files.
**Wave 2 — reasoning emission ports (4/4, ref #76).** Port
reasoning-message emission to ag2, crewai-crews, langroid, and spring-ai
(each also bumped to `@ag-ui` 0.0.55). Adds per-integration
`reasoning_agent` (Python) / `ReasoningController` (spring-ai Java) and
wires the copilotkit route.
**Wave 3 — google-adk demo parity (3/4).** Port `hitl`,
`threadid-frontend-tool-roundtrip`, and `gen-ui-interrupt` demos to
google-adk for parity with the gold reference, plus d6/aimock fixtures.
The `interrupt-headless` demo is intentionally kept `not_supported`
(needs aimock `customEvents` support + an ADK interrupt route — tracked
as a known upstream gap).
## Code review
A 5-round unbiased multi-agent CR loop ran against the combined diff and
converged at zero mandatory findings; the bucket-(c) promotion audit
came back clean. Key fixes folded into the branch:
- **Client error-leak hardening** in the crewai, langroid, and spring-ai
reasoning routes — internal exception details no longer leak to the
client, with `errorId` correlation between client response and server
logs (also applied to google-adk).
- **Protocol-correct reasoning error paths** — on failure the emitters
now close any open frames, emit a generic `RUN_ERROR`, and never emit
`RUN_FINISHED` after `RUN_ERROR`; verified against `@ag-ui/client`
`verifyEvents` semantics, with red-green tests.
- **`x-aimock-context` propagation** across spring-ai's async hop, so
fixture replay stays correct through the thread boundary.
- **Bounded reasoning executor** (no unbounded thread growth) and
**parse-failure observability** (failures surface in logs instead of
being swallowed).
- **Full multi-turn history threading** in the reasoning agents — prior
turns are now forwarded to the LLM, with single-turn byte-equality
preserved so existing aimock fixtures stay valid; covered by tests.
- **Phantom deps declared** — `@copilotkit/shared`/`@ag-ui` deps that
were imported but undeclared in claude-sdk-python and
ms-agent-dotnet/python are now in their `package.json`.
- **`@copilotkit/core` `"latest"` overrides pinned** to `1.59.4`.
- **Reasoning parity token-identity test** across the 3 ported Python
integrations (ag2/crewai-crews/langroid) so their reasoning streams
can't silently drift.
- **Ratchets tightened/updated**: shadow-collision ceiling ratcheted
down 151 → 123 (actual count), duplicate-fixture ceiling 288 → 290 (new
google-adk fixtures reuse standard prebuilt-probe pills,
runtime-disambiguated by demo route), and the `validate-pins` baseline
drops 57 → 48 (currency bumps retire 9 exact-pin FAIL lines).
The final commit is the `style: auto-fix formatting` bot commit
(formatting on the new Python test files).
## Test plan
- [x] Per-integration browser verification (each touched integration's
demos exercised end-to-end) with aimock fixtures added/updated.
- [x] crewai pytest — 113/113 pass.
- [x] `showcase/scripts` vitest — 1780/1780 pass.
- [x] `validate-parity` MUST checks — 19/19 pass, 0 fail.
- [x] `validate-pins` ratchet — exits 0 against the updated baseline
(FAIL-set hash matched).
- [x] D6 `gen-ui-custom` cells verified live (google-adk + agno).
- [x] CI fully green — per-integration Depot Docker `build-check`,
Python unit tests (3.10 + 3.12), commitlint, format, oxlint, Validate
Showcase, production-pinning lint.
## Follow-ups
Consolidated follow-up ledger (bucket c/d items):
https://www.notion.so/copilotkit/37b3aa38185281e5b871d0b907aaef71
_extract_user_input in the ag2, crewai-crews, and langroid reasoning agents
documented a str return but passed AG-UI message content straight through.
Multimodal content can be a list of parts, which would flow unmodified into
the single caller in each file (_run_reasoning_agent ->
messages=[{"role": "user", "content": user_input}]) sent to the OpenAI
chat-completions API. Coerce: str passes through, a list joins its text
parts (dict or attr form), anything else falls back to str().
Callers (one per file):
- ag2/src/agents/reasoning_agent.py:104 -> chat message at :119
- crewai-crews/src/agents/reasoning_agent.py:108 -> chat message at :123
- langroid/src/agents/reasoning_agent.py:103 -> chat message at :118
Add a dedicated Spring/Java ReasoningController (/reasoning/) that
reimplements the ag2 reasoning_agent.py BEHAVIOR: it makes a direct
streaming chat-completions call, reads the native delta.reasoning_content
channel (with a <reasoning>...</reasoning> regex fallback), and emits
RUN_STARTED -> REASONING_MESSAGE_START/CONTENT/END -> TEXT_MESSAGE_*
-> RUN_FINISHED so the CopilotKit reasoning slot mounts
[data-testid="reasoning-block"].
Spring AI's ChatClient drops delta.reasoning_content and the AG-UI Java
SDK has no REASONING_MESSAGE_* event types (only THINKING_*, which
@ag-ui/client drops), so the controller manages its own SseEmitter and
writes the reasoning frames as raw JSON matching the @ag-ui/client 0.0.55
wire schema. Header forwarding (x-aimock-context) rides the existing
WebClientConfig exchange filter. Wire route reasoning-custom/-default (plus
legacy aliases) to /reasoning/, mirroring ag2's reasoningAgentNames, and
bump @ag-ui/client ^0.0.43 -> 0.0.55 for REASONING_MESSAGE_* decode support.
(cherry picked from commit 4d183371c489013f0a7bcce1a447078164974aef)
## What
Adopts the new `ag-ui-langgraph` single-arg tool-factory API in
CopilotKit's LangGraph middlewares, and upgrades the `@ag-ui/*`
dependencies to the latest published releases.
Counterpart to ag-ui-protocol/ag-ui#1894 (OSS-248: re-enable A2UI
generation & design guidelines via the shared `A2UIToolParams` bag).
## Commits
1. **`feat(a2ui): use new ag-ui single-arg A2UIToolParams API`**
- sdk-python + sdk-js middleware: `get_a2ui_tools`/`getA2UITools` now
take a single `A2UIToolParams` object (model inside);
`composition_guide` folds into the `guidelines` bag.
- Guarded `A2UIToolParams` import (py); typed import (ts). a2ui test
skip-reason wording.
- Pins: `@ag-ui/langgraph 0.0.40` (js, the release carrying the new
API), `ag-ui-langgraph >=0.0.41` (py).
2. **`chore(deps): use latest @ag-ui packages and langgraph
integration`**
- `@ag-ui/core`, `@ag-ui/client`: `0.0.53 → 0.0.56` across all packages
+ root pnpm override.
- `@ag-ui/langgraph` (runtime): `0.0.39 → 0.0.40`. `ag-ui-protocol
>=0.1.19`.
- `.npmrc`: exclude first-party `@ag-ui/{core,client,encoder,proto}`
from the minimum-release-age gate so the freshly-published `0.0.56` set
installs.
- Regenerated `pnpm-lock.yaml` + `sdk-python/poetry.lock`.
3. **`chore(showcase): align a2ui to single-arg get_a2ui_tools API`**
- `showcase/.../graph.ts`: `getA2UITools(model, opts)` → single-arg.
## Notes
- All `@ag-ui` deps are live on npm/PyPI, so the branch installs
cleanly.
- Showcase **docs** alignment (TS backend tabs for dynamic-schema) is
still being sorted out — the per-integration docs architecture needs
untangling first; deferred to a follow-up.
- ISOLATE_KEEP promoted to a global so --keep survives cmd_test return
into the trap scope
- early-die and default-stack protection: failed --isolate setup no
longer tears down the default stack; half-initialized state is
cleaned up on the way out
- liveness/PID/age reaping signals with a sweep lock: heartbeat
updates, own-pid lock release, tombstones, and a claim-then-verify
duplicate-name guard close slot-registry races (TOCTOU, lock
takeover, reap order)
- teardown robustness: --volumes on every compose down, failed-down
runs preserve state for diagnosis, reap remnants get a compose-down,
path-traversal guard, uniform rm guards under set -e
- name validation: --isolate names must start with a lowercase letter
or digit; reserved name 'showcase' rejected (it aliases the default
stack)
- fail-loud warning before pre-down of an existing stack; help text
updated
Result of an 8-round, 7-agent code-review loop with red-green
verified fixes.
harness-workers runs the SAME showcase-harness GHCR image as the harness
scheduler but has ciBuilt:false (it has no build slot of its own), and the
CI staging redeploy scope was derived purely from ciBuilt — so a main-merge
rebuild of showcase-harness:latest only bounced the scheduler while the
workers silently kept running the stale image (PR #5352's worker-side fixes
never reached staging).
Model image consumption explicitly in the SSOT instead:
- railway-envs.ts: new optional `imageOf` field on ServiceEntry — the SSOT
key of the ciBuilt service whose image this entry runs. Set
`imageOf: "harness"` on harness-workers. New module-load invariant
`assertImageConsumersValid` (fail-loud, same style as
assertDispatchNamesUnique): imageOf must name an existing SSOT key, the
target must be ciBuilt, and the consumer itself must not be ciBuilt.
- redeploy-env.ts: new `expandImageConsumers(names, env)` applied inside
runRedeploy — the redeploy set becomes the resolved scope PLUS any
service whose imageOf points at a service already in scope. Env-aware:
a consumer only joins envs it declares, so the staging-only worker
never enters a prod redeploy (prod behavior unchanged).
No workflow change needed: showcase_build.yml keeps passing the
built-and-successful dispatch_names; the script expands them. Gate
behavior (gateIgnore / image-ref gate), the generated JSON
(emit --check passes byte-identical), and the promote dropdown are all
untouched. harness-legacy deliberately gets no imageOf (pinned pre-fleet
digest; must not follow rebuilds).
## Summary
- fix shell-docs heading anchors so docs headings remain block-level
rows
- prevent adjacent headings, like `## What the runtime provides`
followed by `### Authentication & security`, from rendering on the same
line
- add a `globals.css` regression guard against bringing back
`inline-flex` on `.docs-heading`
## Verification
- `npm run test -- src/app/__tests__/globals-css.test.ts` in
`showcase/shell-docs`
- `npm run lint` in `showcase/shell-docs`
- `npm run test` in `showcase/shell-docs`
- `npm run typecheck` in `showcase/shell-docs`
- `npm run build` in `showcase/shell-docs`
- local visual check at `http://localhost:3003/backend/copilot-runtime`
with Playwright: the H2 and H3 render as separate full-width rows
- lefthook pre-commit passed `check-binaries`, `lint-fix`,
`test-and-check-packages`, and commitlint
## Notes
- `npm run lint` still reports existing warnings unrelated to this
change.
- `npm run build` still emits the existing Turbopack/NFT warning from
`next.config.ts` / `llms-mdx`, but completes successfully.
## Summary
Root-caused and fixed the staging dashboard's red↔green flapping.
Empirical investigation separated it into three mechanisms:
1. **PRIMARY (recurring flap): aimock fixture-sequence desync.** Harness
drivers sent a stable `d6-<slug>` `X-Test-Id` shared across runs.
aimock's stateful strict-mode matching (sequenced fixtures match only
when per-test-id matchCount === sequenceIndex) drifts via replay
exhaustion and 500-test-id FIFO eviction, so whole integrations 503'd
together (~2,200/hr) and recovered — the flap.
2. **SECONDARY (one-off reds): worker restart mid-job.** In-flight
`driver.run` was un-cancellable; redeploys SIGKILL'd workers mid-run,
leaving partial results that fleet-health painted red
(`worker-crashed-mid-job`).
3. **COSMETIC (red floor): dead roster rows.** 22 dead-container
registry rows were never GC'd; the restart hook is a no-op stub, so
`restartsAttempted` over-counted on rows nothing would ever restart.
## Fixes
- **Per-run-unique X-Test-Id** — fold the existing per-run `mintRunId()`
into the test id via exported `buildE2eTestId(slug, runId)` (injectable
`idFactory` for tests) in the d6/d5 and d4 drivers, so every run gets a
fresh aimock sequence cursor.
- **Graceful worker drain** — on SIGTERM the worker aborts in-flight
runs, suppresses red side-emits (`drainReason === "shutdown"` live
getter + abort conjunction), **abandons** partial results (skips
`queue.report`; the 300s lease sweeper synthesizes neutral
`worker-reclaimed-pending`), and deregisters via a latched,
`lastWrite`-chained roster delete so late heartbeats cannot resurrect
the row. Stop bounded by `WORKER_DRAIN_GRACE_MS` (25s default); order
`worker.stop → registration.stop → deregister → pool.shutdown`.
- **Fleet-health GC** — roster rows dead longer than
`DEFAULT_WORKER_GC_AFTER_MS` (24h) are deleted (`gcDeleted` counter),
malformed rows are GC'd by age before the worker_id guard,
`restartsAttempted` is demoted when the restart hook is a no-op, and
`createFleetHealthMonitor` fails loud when `gcAfterMs <= staleAfterMs`.
Companion PR: CopilotKit/aimock#259 (strict-mode no-match diagnostics,
1.30.0) makes the PRIMARY mechanism observable instead of a bare 503.
## Review
- 3 unbiased CR rounds (8 reviewer agents each, verbatim non-leading
prompt) converged to **zero actionable findings**; round-1 criticals
(drain-suppression of genuine timeouts, deregister/heartbeat
resurrection race, vacuous tests, GC-skipped malformed rows) were fixed
with red-green proofs and confirmed in byte-identical follow-up rounds.
- Bucket-(c) promotion audit: 0 promotions to actionable; 3 trivial
consistency items deferred to the follow-up backlog with ~25
pre-existing harness findings.
## Test plan
- [x] Full harness suite: 122 files / 2146 tests green
- [x] `tsc --noEmit` on both tsconfigs; build green
- [x] oxfmt clean on all 14 changed files
- [x] Red-green discipline on every behavioral fix (failures observed
before fixes)
- [x] CI green on this PR
- [ ] Staging promotion (explicitly out of scope here; separate step)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
The global CopyTracker already emits the generic cli_command_copied on
any clipboard copy, but it cannot distinguish which hero card was used.
Each card now also captures hero_command_copied with command_id
(create | onboard), the full command string, and clipboard_blocked, so
create-vs-onboard funnels are queryable per landing page.
Validated locally against a live PostHog client: each click POSTs one
$autocapture, one cli_command_copied, and one hero_command_copied to
/ingest/e (HTTP 200) with the expected properties.
Jordan Ritter