mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
codex/cloudplot-showcase-migration
15424 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jordan Ritter
|
e8c1380e8e |
fix(showcase/ci): harden advance-latest-tag against silent failure and digest drift
Targeted repairs around the decision logic, which is unchanged: the compare direction, the `["behind"]`-only decline, and the fail-open invariant are all preserved. Digest/identity - `already-current` keyed on the revision LABEL treated a rebuild of the same commit (CVE refresh, cache-miss, retry) as a no-op, stranding `:latest` on the older digest while digest-pinned prod moved. It now compares manifest DIGESTS, via a new `GuardIo.readDigest`, and only when the label claims we are already current — so the common path pays nothing. - Retag passes `--prefer-index=false`. buildx carbon-copies a single index source (digest preserved) but WRAPS a bare manifest in a new index (digest changes) because `--prefer-index` defaults true. Our images are indexes only because provenance is enabled elsewhere; this pins the property regardless. Observability — fail-open stays, silence does not - `readLatestRevision` / `compare` no longer swallow the error. Failures are classified (absent / unauthorized / throttled / timeout / unavailable / unknown) and everything but a genuinely absent tag emits a `::warning` saying the guard has stopped guarding. Throttling is the likeliest failure under the concurrency this exists to handle and was previously indistinguishable from a clean first build. - Annotations are properly escaped. Multi-line registry stderr was folding the real reason out of the annotation meant to report it; encoding newlines also makes an injected `::` inert. - Subprocess timeout (60s, `ADVANCE_LATEST_TAG_TIMEOUT_MS`). A hung registry previously burned the job budget and concluded `cancelled`. - The fleet result now names which images advanced, were unchanged, or declined — not just which failed. Correctness / hygiene - `extractRevisionLabel` collects every occurrence and answers only when they agree, so a multi-config payload no longer returns whichever label key ordering happened to surface last. Dropped the dead cycle guard. - Flag parser no longer binds a following flag as a value (`--sha --repo x` yielded the literal `"--repo"` as the sha); `--flag=value` accepted; an empty `--images` is the documented no-op instead of exit 2. - Entry guard canonicalises both sides, so a symlinked invocation runs instead of exiting 0 having done nothing. - Docs: corrected the never-rethrown claim, the unreachable `identical` example, the self-contradicting `npx tsx` rationale, and the preconditions (a retag is a registry WRITE and needs `packages: write` plus a login, which the redeploy jobs do not currently have). |
||
|
Jordan Ritter
|
428c9c301f |
ci: keep the guard's GITHUB_SHA/GITHUB_REPOSITORY env, asserted by the contract test
Reverts only the cosmetic third item from the previous commit. Dropping the
redundant GITHUB_SHA / GITHUB_REPOSITORY shadowing is functionally correct --
the runner exports both and advance-latest-tag.ts reads them off process.env
-- but showcase/scripts/advance-latest-tag.test.ts asserts those exact keys
are declared on the guard step, so removing them red "Validate Showcase".
RED (run 30238078822, "Validate Showcase" > Run build pipeline tests):
Failed Tests 2
FAIL advance-latest-tag.test.ts > showcase_build.yml - redeploy-staging job
> runs the guard with the image list, sha, repo and a token
FAIL advance-latest-tag.test.ts > showcase_build.yml - redeploy-staging-starters
job > runs the guard with the image list, sha, repo and a token
AssertionError: expected undefined to be '${{ github.sha }}'
- Expected: "${{ github.sha }}"
+ Received: undefined
advance-latest-tag.test.ts:424
GREEN (same six contract assertions per job, replicated against the YAML --
the test is pure yaml-parse, no runtime deps):
redeploy-staging / redeploy-staging-starters
PASS IMAGES PASS GITHUB_SHA PASS GITHUB_REPOSITORY
PASS GH_TOKEN truthy PASS guard `if` PASS guard BEFORE redeploy
The test file is owned by a concurrent agent on this PR, so the assertions
cannot be relaxed here. If the redundancy is worth removing, the env keys and
the assertions must go together in one change.
Everything else from the previous commit is untouched and was already
validated by that CI run -- only these two assertions failed, so the GHCR
login, `packages: write`, the skip_build/images filter, the timeout bump and
the tsx pin all passed.
Lint unchanged: actionlint 10 findings before and after (0 new); zizmor
CI-equivalent reports "No findings" before and after.
|
||
|
Jordan Ritter
|
3450d7d7cd |
ci: give the :latest guard GHCR auth and keep skip_build slots out of the retag set
Two Critical defects in the new "Advance :latest" guard, both of which would have broken staging redeploys on this PR's own merge commit. 1. Registry auth missing. `redeploy-staging` and `redeploy-staging-starters` run `docker buildx imagetools create` (an authenticated GHCR WRITE) but declared only `permissions: contents: read` with no `docker/login-action` step. Every retag would 401, the guard would exit 1, the redeploy step would be skipped -- and since the build step no longer pushes `:latest` itself, `:latest` would freeze permanently and staging would never redeploy again. The same missing auth also 401s `imagetools inspect` on these private packages, which the guard treats as "unknown revision" and advances anyway: it would have failed OPEN and never actually guarded. Fixed by adding `packages: write` plus a GHCR login mirroring the `build` job's existing "Login to GHCR" step. Restores both the read and write path. 2. skip_build slots entered the retag list. The `images` CSV projection did not exclude `skip_build` slots. `webhooks` is built out-of-band, never pushes a `:<sha>` tag, yet reports job.status: success -- so it reached the guard, `imagetools create` failed with "manifest unknown", and the whole fleet's redeploy was blocked. Fires on `service=all` dispatch and via the `workflow_config` path filter. Fixed by carrying `skip_build` through the jq projection and filtering it out of `images` ONLY. It stays in `services`: bouncing webhooks so Railway re-pulls its out-of-band `:latest` is the documented intent (see the `webhooks` entry in showcase/scripts/railway-envs.ts). Also, same file: - timeout-minutes 5 -> 20 on both jobs. The guard adds up to 3 serial round-trips per slot (~84 at full fleet width). An overrun concludes `cancelled`, which is invisible to both `if: failure()` and `if: cancelled()` downstream -- the alerting-suppression class this repo has been fixing. - `npx tsx` -> `npx --yes tsx@4.21.0` (the pnpm-lock-resolved version), matching the no-drift convention showcase_validate.yml enforces via `pnpm exec tsx`. `pnpm exec` is unavailable here: no workspace install. - Dropped the redundant GITHUB_SHA / GITHUB_REPOSITORY shadowing; the runner exports both and advance-latest-tag.ts reads them off process.env. Deliberately NOT added: a `concurrency` group. Its absence is load-bearing -- detect-changes builds a path-filtered per-push matrix, so concurrent runs build disjoint service sets and cancelling the older run would leave services never shipped rather than merely stale. RED/GREEN --------- Workflow auth cannot be fully proven without merging, so the proof is scoped to what is actually demonstrable locally. Both probes lift the code under test VERBATIM out of the workflow at runtime rather than paraphrasing it. Probe A -- the real jq projection, fed the real ALL_SERVICES array with a `service=all` / `workflow_config` dispatch (every slot selected, every slot status:success, which is exactly what the skip_build slot reports): RED images : ...,ghcr.io/copilotkit/showcase-eval-webhook (28 entries) services: ...,webhooks -> webhooks IS in the retag set. imagetools create -> manifest unknown. GREEN images : ends ...,ghcr.io/copilotkit/showcase-pocketbase (27 entries) services: ...,webhooks (unchanged, still 28) -> webhooks OUT of images, STILL IN services. Exactly the split wanted. Probe B -- structural parse of the workflow, over every job that runs advance-latest-tag.ts: RED redeploy-staging permissions={'contents':'read'} login=NO redeploy-staging-starters permissions={'contents':'read'} login=NO VERDICT: 2 retag job(s) lacking GHCR auth GREEN redeploy-staging permissions={'contents':'read', 'packages':'write'} login=YES redeploy-staging-starters permissions={'contents':'read', 'packages':'write'} login=YES VERDICT: 0 retag job(s) lacking GHCR auth Lint (delta vs the pre-fix file, not raw counts) ------------------------------------------------ actionlint : 10 findings before, 10 after; 0 new, 0 removed (all pre-existing depot runner-label + SC2086 noise). zizmor : CI-equivalent (--min-severity low, default persona, repo config) reports "No findings" both before and after. Auditor persona at all severities is also identical before/after (1 concurrency-limits, 1 template-injection, 3 undocumented-permissions, 6 anonymous-definition, 9 secrets-outside-env) -- the two new `packages: write` entries carry trailing justification comments so they add no undocumented-permissions findings. |
||
|
Mark
|
a3ad8960c9 | Merge branch 'main' into mark/ms-agent-dotnet-apikeyresolver | ||
|
Jordan Ritter
|
cacfaca70d | Merge remote-tracking branch 'origin/main' into ci/showcase-latest-tag-race | ||
|
Ben Taylor
|
e9148b3050 |
fix(sdk-python): bridge copilotkit context into LangGraph subgraphs (#5373)
## Summary `CopilotKitMiddleware` loses frontend actions and app context when a middleware-wrapped agent runs as a LangGraph subgraph. Parent graphs propagate run context to child nodes, but `state["copilotkit"]` stays empty there, so the child misses both frontend-tool injection and the `App Context:` note. ## Root cause `LangGraphAGUIAgent.langgraph_default_merge_state(...)` materializes CopilotKit data into state for top-level runs, but child LangGraph nodes inherit runtime context rather than arbitrary parent state. The missing piece was earlier in the write path: by the time the base agent calls `get_stream_kwargs(...)`, it has already replaced the original AG-UI request with merged graph state, so serializing from that argument drops the frontend tools and app context on the real subgraph path. The fix moves ownership to the run entry: `LangGraphAGUIAgent.run(...)` now captures the frontend tools and app context from the real `RunAgentInput`, and `get_stream_kwargs(...)` forwards that captured payload into LangGraph using `context["copilotkit"]` when the graph supports runtime context, or `config["configurable"]["copilotkit"]` on older graph shapes. `CopilotKitMiddleware` still uses one shared lookup order everywhere it reads CopilotKit data: `state["copilotkit"]`, then `runtime.context["copilotkit"]`, then LangGraph config fallbacks. That same carrier now drives the app-context-note helper path too. ## Changes - `sdk-python/copilotkit/langgraph_agui_agent.py`: capture frontend actions and app context from the original `RunAgentInput` in `run(...)`, then write the captured payload into the LangGraph runtime carrier that the current graph shape supports before the subgraph starts. - `sdk-python/copilotkit/copilotkit_lg_middleware.py`: unify frontend-tool, app-context, and interception lookups around the same carrier order, and cover the current-`main` app-context helper path. - `sdk-python/tests/test_agui_agent.py`: add focused coverage for both the runtime-context path and the configurable fallback path on graphs without `context` support. - `sdk-python/tests/test_copilotkit_lg_middleware.py`: replace the manually seeded subgraph proof with a real `LangGraphAGUIAgent.run(...)` regression that reaches the child middleware through the production path. ## What is NOT changed - Top-level agents still prefer `state["copilotkit"]`; the runtime bridge only fills the subgraph gap. - Header propagation behavior is unchanged. - No JS LangGraph executor changes are part of this PR. ## Related PRs and Issues Closes #3886. ## Test plan - [x] `cd sdk-python && pytest tests/test_agui_agent.py tests/test_copilotkit_lg_middleware.py -q` - 108/108 pass. Covers the AGUI run-entry bridge, the configurable fallback for graphs without `context` support, frontend-tool injection, app-context note injection, and a real `create_agent` subgraph invocation through `LangGraphAGUIAgent.run(...)`. - [x] `ruff check sdk-python/copilotkit/copilotkit_lg_middleware.py sdk-python/copilotkit/langgraph_agui_agent.py sdk-python/tests/test_agui_agent.py sdk-python/tests/test_copilotkit_lg_middleware.py` - pass. Covers Python lint on every touched file. - [x] `ruff format --check sdk-python/copilotkit/copilotkit_lg_middleware.py sdk-python/copilotkit/langgraph_agui_agent.py sdk-python/tests/test_agui_agent.py sdk-python/tests/test_copilotkit_lg_middleware.py` - pass. Covers repo formatting on every touched file. - [ ] CI green (`static / quality`, `test / unit` on Node 20/22/24) |
||
|
Jordan Ritter
|
3ff7a51606 |
fix(showcase): pin ag2<1.0.0 to unblock CI after upstream module rename (#6176)
## What's broken
`ag2` **1.0.0 was published to PyPI at 2026-07-27T00:58:37Z**. It
renames the top-level module from `autogen` to `ag2` and ships **no
`autogen/` package at all**.
`showcase/integrations/ag2/requirements.txt` floated on
`ag2[openai,ag-ui]>=0.9.0`, so every *fresh* CI dependency resolution
now pulls 1.0.0 and pytest collection dies on import.
This breaks `Python unit tests (3.10)` and `Python unit tests (3.12)` in
`showcase_validate.yml` **on `main` and on every open PR**. It's
time-based, not commit-based — `main` currently reads green only because
it hasn't re-run since the release landed.
### Release timing evidence
From the PyPI JSON API:
| version | upload_time_iso_8601 |
|---|---|
| 0.14.0 | `2026-06-26T04:48:39.784002Z` |
| 1.0.0 | `2026-07-27T00:58:37.996353Z` |
### Wheel layout proof
Top-level directories in each wheel:
```
=== ag2 0.14.0 === === ag2 1.0.0 ===
ag2-0.14.0.dist-info ag2
autogen ag2-1.0.0.dist-info
templates
```
`autogen/` is simply gone in 1.0.0.
## Why pin instead of migrating the imports
**Because 1.0.0 is not a rename — it's an API change.** Diffing the
wheels:
1. `autogen/ag_ui/adapter.py` (the stable, `AgentService`-based
`AGUIStream`) is **deleted**. `ag2/ag_ui/` in 1.0.0 is the *promoted
former `autogen.beta.ag_ui`* — a different implementation with
`events.py`/`stream.py`, matching old `autogen/beta/ag_ui/`, not old
`autogen/ag_ui/`.
2. The constructor lost a parameter:
```python
# 0.14.0 autogen/ag_ui/adapter.py
def __init__(self, agent, *, event_interceptors=()) -> None:
# 1.0.0 ag2/ag_ui/stream.py
def __init__(self, agent: Agent) -> None:
```
3. `dispatch()` changed shape — `context=` is **gone**, replaced by
`variables=` plus new
`prompt`/`dependencies`/`config`/`tools`/`middleware`/`observers`/`hitl_hook`
kwargs.
`showcase/integrations/ag2/src/agents/_multimodal_normalize.py:330`
subclasses `AGUIStream` and calls:
```python
async for chunk in super().dispatch(incoming, context=context, accept=accept):
```
So a naive import-path rewrite would trade a loud `ModuleNotFoundError`
at import time for a **silent-until-hit `TypeError` at request time** —
strictly worse while CI is on fire. A genuine 1.0 migration has to port
~20 `autogen` import sites (including
`autogen.agentchat.ContextVariables`, `autogen.tools.tool`,
`autogen.code_utils.content_str`) and re-verify every ag2 demo cell
behaviorally. That's real work, not an unblock.
## Coverage — every `ag2` dependency site
Swept all `requirements*.txt`, `*.lock`, `uv.lock`, `poetry.lock`, and
`pyproject.toml` in the repo:
```
./showcase/integrations/ag2/requirements.txt:1:ag2[openai,ag-ui]>=0.9.0
```
**Exactly one** declaration — no lockfile pins it, no other integration
depends on it, transitively or otherwise. All `autogen` imports live
under `showcase/integrations/ag2/`. This one-line bound closes the whole
surface; nothing else is left floating.
## Red-green proof
Both runs use a clean `python3.12 -m venv` + `pip install -r
requirements.txt`, i.e. a fresh resolution exactly like CI's.
### RED — before the fix (resolves ag2 1.0.0)
```
$ pip show ag2 | head -2
Name: ag2
Version: 1.0.0
$ python -m pytest tests/python/ -q
==================== ERRORS ====================
________ ERROR collecting tests/python/test_multimodal_normalize.py ________
ImportError while importing test module '.../tests/python/test_multimodal_normalize.py'.
Traceback:
tests/python/test_multimodal_normalize.py:77: in <module>
from agents._multimodal_normalize import ( # noqa: E402
src/agents/_multimodal_normalize.py:75: in <module>
from autogen.ag_ui import AGUIStream, RunAgentInput
E ModuleNotFoundError: No module named 'autogen'
=============== short test summary info ===============
ERROR tests/python/test_multimodal_normalize.py
!!!!!! Interrupted: 1 error during collection !!!!!!
1 warning, 1 error in 0.30s
```
### GREEN — after the fix (resolves ag2 0.14.0)
```
$ pip show ag2 | head -2
Name: ag2
Version: 0.14.0
$ python -m pytest tests/python/ -q
...................... [100%]
=============== warnings summary ===============
tests/python/test_cvdiag_boundaries.py:34
StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated
22 passed, 1 warning in 1.52s
```
## Follow-up left undone
Pinning defers the migration; it doesn't cancel it. What remains:
- **Port to ag2 1.x.** Rewrite ~20 `autogen.*` imports to `ag2.*`,
replace the deleted adapter-based `AGUIStream` with the new stream API,
drop/replace `event_interceptors` in `_multimodal_normalize.py`, and
convert `dispatch(context=...)` to `dispatch(variables=...)`. Then
behaviorally re-verify every ag2 demo cell — the beta stream promoted
into `ag2.ag_ui` is a different implementation, so cell-level parity
cannot be assumed from a green unit suite.
- **Renovate** will eventually propose lifting the `<1.0.0` bound. That
PR must not be merged as a routine bump; it's the migration above.
- The inline comment in `requirements.txt` records all of this at the
point of change so the bound isn't lifted casually.
## Second commit: `validate-pins` ratchet hash
The `Validate Showcase` job's `validate-pins (ratchet)` step failed on
the first push. Cause is benign and expected: `ag2` matches
`FRAMEWORK_PATTERNS`, which demands an exact `==` pin, so the dep was
**already** in the drift baseline as a `[FAIL]`. Adding `,<1.0.0`
changes that line's text, which changes the SHA-256 the ratchet takes
over the sorted FAIL set.
Verified the FAIL *set* is otherwise untouched — I reproduced the
committed baseline hash on a pristine `origin/main` tree, then diffed:
```
$ diff <(pristine FAIL lines) <(fixed FAIL lines)
[FAIL] ag2: ag-ui-protocol is not an exact pin (>=0.1.10)
-[FAIL] ag2: ag2 is not an exact pin (>=0.9.0)
+[FAIL] ag2: ag2 is not an exact pin (>=0.9.0,<1.0.0)
[FAIL] ag2: openai is not an exact pin (>=1.50.0)
```
Exactly one line differs. `Summary: OK=5 SKIP=0 WARN=3 FAIL=31` both
before and after — **count unchanged at 31**, nothing healed, nothing
regressed. So this is a hash-only baseline update and the "never raise
the count without sign-off" invariant is not engaged.
Pristine run reproduced `b47ca987…` (identical to the committed
baseline), confirming the validator is deterministic and this diff is
the sole delta.
### Why a range bound rather than an exact `==0.14.0`
An exact pin would satisfy `validate-pins` outright and ratchet the
count 31 → 30, but I kept the range deliberately:
- `0.14.0` is the highest final pre-1.0 release and the 0.x line is now
legacy, so the range resolves to exactly `0.14.0` today — proven by the
GREEN run above.
- `1.0.0b0` sorts below `1.0.0` under PEP 440, but pip excludes
pre-releases when a final release satisfies the specifier, so the bound
cannot pull the beta. Empirically confirmed: the GREEN venv resolved
`0.14.0`.
- It leaves room for a future `0.14.x` patch or security release without
another PR.
- It matches the sibling deps in the same file (`ag-ui-protocol>=`,
`openai>=`) and keeps this urgent fix to a pure hash bump rather than a
count ratchet.
Tightening to an exact pin (and ratcheting the count down) is reasonable
follow-up if the team wants to drive `validate-pins` drift toward zero,
but it's a pin-discipline cleanup, not part of this unblock.
|
||
|
Jordan Ritter
|
4947c4a785 |
fix(showcase): update validate-pins hash for the ag2 bound
The `<1.0.0` upper bound changes the text of an existing `[FAIL]` line
from `ag2 is not an exact pin (>=0.9.0)` to `(>=0.9.0,<1.0.0)`, which
shifts the ratchet's SHA-256 over the sorted FAIL set.
The FAIL *set* is otherwise identical -- count stays 31, nothing healed
and nothing regressed. Verified by reproducing the committed baseline
hash b47ca987 on a pristine origin/main tree, then diffing the FAIL
lines against the fixed tree; exactly one line differs:
-[FAIL] ag2: ag2 is not an exact pin (>=0.9.0)
+[FAIL] ag2: ag2 is not an exact pin (>=0.9.0,<1.0.0)
`ag2` matches FRAMEWORK_PATTERNS, which demands an exact `==` pin, so
the dep was already in the baseline's drift set before this change and
remains in it after. Hash-only update; the count is untouched, so the
"never raise the count without sign-off" invariant is not engaged.
|
||
|
Jordan Ritter
|
5a09e0e787 |
fix(showcase): pin ag2<1.0.0 to unblock CI after upstream module rename
ag2 1.0.0 (published 2026-07-27T00:58:37Z) renames its top-level module
from `autogen` to `ag2` and ships no `autogen/` package at all. The ag2
integration floats on `ag2[openai,ag-ui]>=0.9.0`, so every fresh CI
resolution now picks up 1.0.0 and collection fails at:
src/agents/_multimodal_normalize.py:75
from autogen.ag_ui import AGUIStream, RunAgentInput
E ModuleNotFoundError: No module named 'autogen'
This breaks `Python unit tests (3.10)` and `(3.12)` in showcase_validate
on main and on every open PR. It is time-based rather than commit-based:
main only looks green because it has not re-run since the release.
Pin rather than migrate. 1.0.0 is not a rename -- it deletes the stable
`autogen/ag_ui/adapter.py` and promotes the former `autogen.beta.ag_ui`
stream into its place, dropping `AGUIStream(event_interceptors=...)` and
replacing `dispatch(context=...)` with `dispatch(variables=...)`. Since
`_multimodal_normalize.py` overrides `dispatch` and forwards `context=`,
rewriting the import path alone would swap an import error for a runtime
TypeError. A real migration needs to port ~20 `autogen` import sites and
re-verify every ag2 demo cell, which is not an urgent-unblock change.
|
||
|
Mark
|
a9ce37b75c |
fix(showcase/ms-agent-dotnet): resolve OPENAI_API_KEY-first via ApiKeyResolver
Port ms-agent-harness-dotnet's ApiKeyResolver into ms-agent-dotnet so the 15 agent factories resolve the OpenAI credential as OPENAI_API_KEY (env) -> config[OPENAI_API_KEY] -> GitHubToken, and the endpoint via OPENAI_BASE_URL -> default, instead of hardcoding configuration["GitHubToken"] per agent. Previously ms-agent-dotnet's main chat clients authenticated only with the GitHub token. On a fixture-miss fall-through, aimock proxies to real api.openai.com, which rejects the GitHub token (invalid_api_key, surfaced as 502). ms-agent-harness-dotnet already works because it resolves OPENAI_API_KEY first; this brings ms-agent-dotnet to parity so its fall-through returns 200. Interim showcase-side mitigation while the aimock cross-provider guard (PNI-108, CopilotKit/aimock#340) is deferred. - Copy agent/ApiKeyResolver.cs verbatim from ms-agent-harness-dotnet (code copy, no new dependency) - Rewire 15 factories + A2uiSecondaryToolCaller to ResolveApiKey/ResolveEndpoint; drop the dead per-file DefaultOpenAiEndpoint const - Add tests/ApiKeyResolverTests.cs (precedence, mock-endpoint fallback, non-mock fail-fast) Verified: dotnet build 0/0, dotnet test 76/76, whitespace format clean, and a live OpenAI call through the resolver returned 200. |
||
|
Ben Taylor
|
063750a19e |
fix(react-ui): reduce sidebar CSS specificity (#5805)
## Summary - lowers CopilotSidebar-scoped CSS selector specificity with `:where(.copilotKitSidebar)` - keeps the existing sidebar styles and layout behavior unchanged - adds a regression test to keep sidebar selectors easy for app CSS to override ## Root cause Sidebar-specific selectors such as `.copilotKitSidebar .copilotKitWindow` were more specific than ordinary user overrides like `.copilotKitWindow`, so user CSS could lose precedence in Chrome. Fixes #263. ## Validation - `pnpm -C packages/react-ui test` - `pnpm exec oxfmt --check packages/react-ui/src/css/sidebar-specificity.test.ts packages/react-ui/src/css/window.css packages/react-ui/src/css/header.css packages/react-ui/src/css/input.css` - `pnpm exec oxlint packages/react-ui/src/css/sidebar-specificity.test.ts` |
||
|
Jordan Ritter
|
5afff788dd | Merge remote-tracking branch 'origin/main' into ci/showcase-latest-tag-race | ||
|
Ben Taylor
|
21cd27ce50 |
fix(react-core): stop CopilotPopup remounting chat on resize (#6173)
## What does this PR do? Fixes a bug where resizing a `CopilotPopup` (v2) rescrolls the message list from top to bottom on every resize, regardless of the user's scroll position. **Root cause.** `CopilotPopup` built its `chatView` override component inside a `useMemo` keyed on `width`/`height`. When those props change — which happens continuously for consumers driving them from a drag-to-resize handle that commits new dimensions on `mouseup` — `useMemo` returns a **new component function**. That function is passed to `CopilotChat` as the `chatView` slot and rendered via `React.createElement(newType, …)`. Because React sees a new element **type** at that position, it unmounts and remounts the entire chat subtree rather than re-rendering it. The remount resets the scroll container's `scrollTop` to `0`, after which the default `initial="smooth"` auto-scroll animates the list from the top back down to the bottom — the visible "rescroll" on every resize. **Fix.** Give the override a stable module-scope identity so its type never changes, and pass the popup shell props (`header`, `toggleButton`, `width`, `height`, `clickOutsideToClose`, `defaultOpen`) through React context instead of baking them into the override via `useMemo` deps. Resizing is now a plain style update on `CopilotPopupView` — no remount — so scroll position is preserved. `Object.assign` still copies the `CopilotChatView` static slot members onto the override so callers reaching them through the namespace continue to work. **Test.** Adds a regression test (`CopilotPopup.resizeRemount.test.tsx`) that injects a mount-counting component into the chat subtree and asserts its mount count stays at `1` across several `width`/`height` changes. It fails on the pre-fix code (one extra mount per resize — observed `4` for 3 resizes) and passes with the fix. Scope is limited to `packages/react-core`: ``` packages/react-core/src/v2/components/chat/CopilotPopup.tsx | 124 +++++++++------ packages/react-core/src/v2/components/chat/__tests__/CopilotPopup.resizeRemount.test.tsx | 98 +++++++++++ 2 files changed, 185 insertions(+), 37 deletions(-) ``` Verification: full `@copilotkit/react-core` unit suite green (1443 tests), `check-types` green. ## Related PRs and Issues - Closes #6172 ## Checklist - [x] I have read the [Contribution Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md) - [ ] If the PR changes or adds functionality, I have updated the relevant documentation — _N/A: internal behavior fix, no public API or documented behavior change_ - [x] "Allow edits by maintainers" is checked |
||
|
Rod Boev
|
a76d59ae0b | fix(sdk-python): capture subgraph context from run input (#3886) | ||
|
Ben Taylor
|
73fe697b15 |
fix(web-inspector): handle empty tool result content (#5603)
## Summary
Fixes the web inspector crash when historical tool result messages have
empty `content`.
Some stored thread tool-result messages can have `content: ""`. The
inspector previously parsed tool result content with
`JSON.parse(msg.content ?? "{}")`, which still passes the empty string
into `JSON.parse` and throws `Unexpected end of JSON input`.
## Changes
-> Treat empty and whitespace-only tool result content as an empty
result object
-> Keep malformed non-empty result content flowing through the existing
parse-error fallback
-> Add regression coverage for empty and whitespace-only tool result
messages in the thread conversation mapper
## Testing
-> `pnpm exec oxlint packages/web-inspector/src/index.ts
packages/web-inspector/src/__tests__/web-inspector.spec.ts`
-> `pnpm -C packages/web-inspector exec vitest run
src/__tests__/web-inspector.spec.ts --pool=threads --testNamePattern
"maps empty tool result content" --reporter=verbose`
## Notes
-> `pnpm nx show project @copilotkit/web-inspector --json` timed out
locally
-> `pnpm -C packages/web-inspector run check-types` is currently blocked
by existing package-level TypeScript errors around explicit `.js` import
extensions and one unrelated existing error
## Related Issue
Closes #5598
|
||
|
Jordan Ritter
|
bde2d99a5b |
fix(showcase/ms-agent-python): keep the user's prompt on the multimodal PDF turn (#6159)
`d6:ms-agent-python/multimodal` has been red in staging and prod since
2026-05-30. Turn 1 (image) passes; turn 2 (PDF) fails. This fixes it —
**without touching the fixture**, because the fixture was never the
problem.
## The verbatim turn-2 error
Backend (`showcase-ms-agent-python`), and reproduced locally:
```
[/multimodal] Streaming failed
openai.InternalServerError: Error code: 503 - {'error': {'message': 'Strict mode: no fixture matched',
'type': 'invalid_request_error', 'param': None, 'code': 'no_fixture_match'}}
The above exception was the direct cause of the following exception:
agent_framework.exceptions.ChatClientException: ("<class
'agent_framework_openai._chat_completion_client.OpenAIChatCompletionClient'> service failed to
complete the prompt: Error code: 503 - {'error': {'message': 'Strict mode: no fixture matched', …
```
Surfaced in the browser as `An internal error has occurred while
streaming events.`, with the probe reporting `failure_turn: 2`,
`turns_completed: 1`.
## Request-shape diagnosis
This reads like a fixture gap and is not one. I pulled the **actual
outbound request** off the local aimock's `GET /__aimock/journal` during
a failing run. Turn 2, verbatim (bodies elided):
```
[0] role=system "You are a helpful assistant. The user may attach images or documents…"
[1] role=user "can you tell me what is in this demo image I just attached"
[2] role=user [image_url <data:image/png;base64,iVBORw0K…>]
[3] role=user [image_url <data:image/png;base64,iVBORw0K…>]
[4] role=assistant "The attached image is the CopilotKit logo — a clean, geometric mark…"
[5] role=user "can you tell me what is in this demo pdf I just attached"
[6] role=user "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to your React…"
[7] role=user "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to your React…"
```
One logical user turn arrived as **three separate user messages**, and
the *last* one carries only the flattened document — the question is
nowhere in it. That is why aimock's strict mode refused it:
`userMessage` is a substring match against the last user turn, and the
last user turn was a PDF dump.
**Root cause:** `agent_framework_openai` emits **one OpenAI message per
`Content`**. `_chat_completion_client._prepare_message_for_openai`
builds a fresh `args` dict on every iteration of its content loop, so a
user `Message` carrying `[prompt_text, flattened_doc_text]` serialises
to two consecutive user messages — prompt-only, then document-only.
`_PdfFlattenChatMiddleware` was appending the flattened `[Attached
document]` text as a *second* text `Content` beside the prompt, which is
exactly the shape that gets split.
Two corroborating details that make the mechanism airtight:
- **Why turn 1 (image) passes.** aimock already skips *text-less*
trailing user messages (`getLastUserText` in `router.ts`, whose comment
documents this exact MS Agent Framework behavior). The image turn's
split-off trailing message has no text at all, so aimock falls back to
the prompt message and matches. The PDF turn's trailing message *does*
have text — the document — so there is nothing to skip past.
- **Why `langgraph-python` is green** doing the identical `[Attached
document]` flattening: LangChain keeps multiple text parts *inside one
message* rather than splitting them into separate messages.
This is a product bug, not a mock artefact. Against a real LLM it would
not 503 — the model would just answer the wrong thing, because the
question is buried behind a document dump instead of being the current
turn.
## The fix
`showcase/integrations/ms-agent-python/src/agents/multimodal_agent.py`
1. **Merge** the flattened document *into* the message's existing prompt
text content instead of appending it as a second content. The turn stays
a single text content and serialises to a single user message:
`"<prompt>\n[Attached document]\n<body>"`.
2. The merge **copies** the prompt `Content` rather than mutating it.
This is load-bearing: the middleware restores the original `contents`
list after `call_next`, and that restore only undoes the *list* swap —
an in-place mutation would leak the raw PDF body into the AG-UI
`MESSAGES_SNAPSHOT` and render a wall of PDF text in the user's chat
bubble. There is a test for this.
3. **Attachment-only turns** (a PDF with no question) still work: with
no text content to merge into, the flattened document stands alone as
the message body.
4. **Dedupe identical flattened blocks.** The page's
`LegacyConverterShim` appends a legacy `binary` mirror alongside every
modern attachment part, so the same PDF reached the middleware twice and
its body was being sent to the model twice (visible as the duplicated
`[6]`/`[7]` above). Now emitted once.
Post-fix outbound turn 2, same journal endpoint:
```
[5] role=user "can you tell me what is in this demo pdf I just attached\n[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to your React application with CopilotKit…"
matched fixture userMessage: "can you tell me what is in this demo pdf I just attached"
```
One user message, prompt intact, document intact, emitted once.
## The fixture is untouched
```
$ git diff --stat origin/main -- showcase/aimock/
(empty)
```
The existing `userMessage` match key was always correct; the corrected
request shape is what satisfies it. Relaxing or re-recording the fixture
to match the broken request was an explicit non-goal — it would have
made the cell actively certify a model that never sees the user's
question.
## Same-pattern audit
- `_PdfFlattenChatMiddleware` is the **only** `ChatMiddleware` in
`ms-agent-python`, and the only place in the integration that constructs
`Content` or reassigns `message.contents` (`grep` for `ChatMiddleware` /
`Content.from_text` / `.contents =` across `src/` returns hits in this
one file only). No second instance of the pattern to fix.
- `ms-agent-python` is the only MS-Agent-Framework Python integration
doing PDF flattening — `ms-agent-dotnet` has a multimodal e2e spec but
no Python agent. The other `[Attached document]` implementations
(`langgraph-python`, `langgraph-fastapi`, `agno`, `claude-sdk-python`,
`langroid`, `pydantic-ai`, `langgraph-typescript`, `built-in-agent`) run
on frameworks that do not split a message's contents into separate wire
messages, so they are not exposed to this. The upstream
one-message-per-`Content` behavior is pinned by a dedicated test, so if
it ever changes we find out by that test failing rather than by a silent
regression.
- The file is a regular per-integration file, not a `shared/` symlink
(`git ls-files -s` → `100644`). No shared code touched;
`validate-shared-symlinks.ts` confirms no new erosion.
## Red / green / control
All three on the real probe surface, from a clean worktree at
`origin/main` `38613623f4`.
### RED — before the change
```
$ bin/showcase test ms-agent-python:multimodal --d6 --direct --verbose --cycle --isolate
[conversation-runner] turn 1/2 — assistant settled { bubbleIndex: 0, textLength: 100, hasAssertions: true }
[conversation-runner] turn 1/2 — assertions passed
[conversation-runner] turn 2/2 — sending message { inputLength: 29, timeoutMs: 60000 }
[conversation-runner] turn 2/2 — FAILED {
errorCategory: 'assertion-failed',
turnsCompleted: 1,
elapsedMs: 1577,
bodyTextLength: 421,
hasTextarea: true,
hasErrorBoundary: false
}
[warn] CVDIAG component=harness-d6 boundary=fixture-match … status=miss … error=chat errored: copilot-error-banner visible — An internal error has occurred while streaming events.
[info] probe.e2e-full.service-complete {"slug":"ms-agent-python","passed":0,"failed":1,"skipped":0,"incapable":0,"total":1,"state":"red","durationMs":9384}
✗ d6:ms-agent-python red (9.5s)
multimodal: chat errored: copilot-error-banner visible — An internal error has occurred while streaming events.
0 passed, 1 failed (9.5s)
⚠ Tests failed for ms-agent-python:multimodal (exit 1)
```
Evidence the outbound request lacked the prompt — aimock journal from
that run, 8 entries, `200,503,503,503,200,503,503,503` (2 attempts × 3
retries on turn 2):
```
[5] role=user STRING "can you tell me what is in this demo pdf I just attached"
[6] role=user STRING "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to…"
[7] role=user STRING "[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to…"
status: 503
```
### GREEN — after the change, fixture unchanged
```
$ bin/showcase test ms-agent-python:multimodal --d6 --direct --verbose --rebuild --keep --isolate
[conversation-runner] turn 1/2 — assistant settled { bubbleIndex: 0, textLength: 100, hasAssertions: true }
[conversation-runner] turn 1/2 — assertions passed
[conversation-runner] turn 2/2 — assistant settled { bubbleIndex: 1, textLength: 233, hasAssertions: true }
[conversation-runner] turn 2/2 — assertions passed
[conversation-runner] conversation completed successfully { turnsCompleted: 2, totalDurationMs: 8279 }
[info] probe.e2e-full.feature-complete {"slug":"ms-agent-python","featureType":"multimodal","pass":true,"durationMs":8788}
[info] probe.e2e-full.service-complete {"slug":"ms-agent-python","passed":1,"failed":0,"skipped":0,"incapable":0,"total":1,"state":"green","durationMs":10187}
✓ d6:ms-agent-python green (10.5s)
1 passed (10.5s)
✓ Tests passed for ms-agent-python:multimodal
```
Both turns pass. aimock journal for that run: **2 entries, statuses
`200,200`** (down from 8 entries with six 503s — no retries needed).
**The fixture was not modified**; `git diff origin/main --
showcase/aimock/` is empty and the diff is two files, both under
`showcase/integrations/ms-agent-python/`.
### CONTROL — an already-green integration, same command, same stack
```
$ bin/showcase test langgraph-python:multimodal --d6 --direct --isolate
[conversation-runner] turn 2/2 — assistant settled { bubbleIndex: 1, textLength: 233, hasAssertions: true }
[conversation-runner] turn 2/2 — assertions passed
[conversation-runner] conversation completed successfully { turnsCompleted: 2, totalDurationMs: 8395 }
✓ d6:langgraph-python green (9.1s)
1 passed (9.1s)
✓ Tests passed for langgraph-python:multimodal
```
Local harness, shared probe, shared frontend and fixtures are all sound
— the red was specific to this integration.
## Covering test
`showcase/integrations/ms-agent-python/tests/python/test_multimodal_pdf_prompt.py`
— 7 tests. Not fakes: each one drives the real
`_PdfFlattenChatMiddleware` and then the real
`OpenAIChatCompletionClient._prepare_message_for_openai`, and asserts
against the actual OpenAI wire payload. The PDF is the bundled
`public/demo-files/sample.pdf` through real `pypdf`, and the prompt
asserted on is **read out of the real aimock fixture** rather than
hardcoded, so the test fails if either side drifts.
Test-level red→green (stash the source change, keep the tests):
```
# pre-fix
FAILED test_multimodal_pdf_prompt.py::test_pdf_turn_last_user_message_contains_the_prompt
FAILED test_multimodal_pdf_prompt.py::test_pdf_turn_serialises_to_a_single_user_message
FAILED test_multimodal_pdf_prompt.py::test_duplicate_pdf_parts_are_flattened_once
3 failed, 4 passed in 2.37s
```
with the primary failure reading:
```
AssertionError: expected the PDF turn to serialise to 1 user message, got 2:
['can you tell me what is in this demo pdf I just attached',
'[Attached document]\nCopilotKit Quickstart\nAdd AI copilots to']
```
```
# post-fix — full integration suite (6 pre-existing CVDIAG + 7 new), CI's exact invocation
$ PYTHONPATH=".:src" python -m pytest tests/python/ -q
13 passed in 2.40s
```
Coverage: prompt survives to the final user turn; the turn stays one
user message; the upstream one-message-per-`Content` split is pinned;
original `contents` restored and the prompt `Content` not mutated;
duplicate mirror parts flattened once; attachment-only turn still
flattens; image turn left byte-identical.
## Pre-push
`validate-parity.ts` 20/20 pass · `validate-shared-symlinks.ts` no new
erosion · `aimock-fixtures.test.ts` 842 pass · full `tests/python/`
suite 13 pass · lefthook `lint-fix` + `commitlint` clean · Python lines
≤88 cols matching the file's existing style · no lockfile churn, two
files in the diff.
## Scope
One cell, one middleware, one integration. The other five red
`multimodal` cells from the same sweep have five different root causes
and are not addressed here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01PYdjeveT8Xof9TyHWMLoJr
|
||
|
Jordan Ritter
|
fe197ff72d |
fix(showcase/ci): make the :latest tag monotonic so an older build cannot overwrite a newer one
Three merges landed on main within 34 seconds on 2026-07-26. The build workflow has no concurrency group by design, so all three ran at once and raced to push the same `:latest` tags. The NEWEST commit's build finished FIRST, so the two older builds overwrote it: run 30190815370 |
||
|
Jordan Ritter
|
edc59c476c |
fix(showcase): cold-load gray chip was masking genuinely-red cells (#6156)
> **Scope, stated up front.** This PR fixes **one** of the ways a
genuinely-red
> showcase cell can render as something other than red: the cold-load
`signal`
> polarity. It does **not** eliminate the symptom. **Nine further
masking
> mechanisms remain** and are enumerated, with measured counts, under
> [What this PR does NOT fix](#what-this-pr-does-not-fix). Please read
that
> section before treating a green dashboard as evidence of a green
fleet.
## The bug
Absence of evidence about *why* a cell failed was erasing the fact that
it failed.
`classifyRung` opened its red branch with:
```ts
if (!raw.signalKnown) {
return { ...base, contribution: "NO_DATA", rawStatus: null };
}
```
`signalKnown` is `false` when the dashboard's bulk fetch projected the
`signal`
blob away. So a **genuinely RED rung whose infra-vs-product attribution
was merely
PENDING got classified `NO_DATA` and painted gray** — the muted "no live
probe
data" treatment — while the depth strip beside it, built from the raw
fold,
correctly rendered `1P ✗`. One model object simultaneously said
`d5.status ===
"red"` and `chipColor === "gray"`.
Two things make that polarity wrong:
1. **`signalKnown === false` never meant "known to have no signal."**
PocketBase
omits the `signal` key *only* under a `fields=` projection; a row that
genuinely has no signal arrives as `null`, and `null !== undefined`. The
flag
is pure provenance: "this row came from a projected fetch." It was never
evidence the failure was infra.
2. **The guess was wrong on the overwhelming majority.** Of the
non-green rows in
production, **94.2% are product-class and 5.8% infra-class** (measured
today
against prod PocketBase using the classifier's own imported
`signalHasInfraErrorClass` predicate: 3082 rows, 359 red, 21 infra-class
—
`driver-error` 11, `abort` 10 — 338 product-class). Graying every
unattributed
red lost 338 real failures to suppress 21 false alarms. A masked red is
never
investigated; a false alarm is looked at once and closed.
This was **not** a first-paint race. The gray state is a fully-rendered,
confident-looking wrong answer that persists until the probe's next
sweep rewrites
that specific row and the SSE delta redelivers it with `signal`, and it
recurs on
**every** reload, because the projection strips `signal`
unconditionally.
**Worst case for the routine per-cell dimensions is ~60 min**, from the
hourly
`e2e-demos` / `starter_smoke` / `d6-all-pills-e2e` crons in
`harness/config/probes/*.yml`; drift dimensions are effectively
unbounded. *(An
earlier revision of this body claimed "~29 min observed, ~14 min mean".
Both were
withdrawn during review: "~29 min" was derived from the `*/30` tier and
missed the
hourly tier entirely, and "~14 min mean" had no derivable basis. Do not
cite them.)*
## Measured effect
PR #6169's exhaustive sweep — **279,936 rung combinations** (7 status
rows × 6 row
variants), the same map fed to the pre-fix and post-fix engines and
diffed
field-by-field — puts the direction of this change beyond argument:
- **74,256 cells change**, and **every one moves toward *more* red.**
- **Zero** flips toward less red: no `red→gray`, no `red→green`, no
`amber→green`
anywhere in the diff.
- INV2–INV6 hold across the full sweep on the branch engine (0
violations).
Red-green on **live production data** (local `next dev` wired to prod
PocketBase,
driven with Playwright, screenshots visually inspected — DOM
measurements alone are
not trusted for visual verification; both captures back-to-back on the
same
underlying rows, only the code changing):
| Band | before | after |
|---|---|---|
| green | 632 | **632 — unchanged, no green regression** |
| amber | 0 | 0 |
| **red** | **0** | **51** |
| no data | 76 | **25** |
`0 red → 51 red`, `76 → 25 no data` — 51 recovered, arithmetic closes.
Red on
**first paint**, no gray plateau.
## The fix — two parts
### (a) Flip the fail-safe polarity — `cell-model.contribution.ts`
A red rung with unknown infra-ness now classifies `FAIL_FRESH`. Graying
a red
requires **positive infra evidence**: `hasNonInfraRed` is false only
when every
contributing red row's blob actually carries an `INFRA_ERROR_CLASSES`
attribution,
and signal-provenance is retained as an explicit second precondition so
a missing
blob can never be read as an infra attribution even if
`signalHasInfraErrorClass`
is loosened later.
```ts
if (fold.redSignalKnown && !fold.hasNonInfraRed) {
return { ...base, contribution: "INFRA_RED_FRESH", rawStatus };
}
```
`redSignalKnown` is derived inside `foldFamily`'s **red-row**
bookkeeping, not
family-wide, so a projected-away GREEN sibling cannot suppress a
legitimate infra
gray. (The original family-wide flag made `INFRA_RED_FRESH` effectively
**unreachable in the browser for every multi-row family** — D4, any
multi-pill
D5/D6; only single-key D3 was unaffected, which is why every
pre-existing fixture
missed it. Fixed by round-1 unit G1.)
This is the durable fail-safe: any future projection or fetch regression
now
degrades toward **over-reporting** failure rather than silently hiding
it.
### (b) Make first paint correct — `useLiveStatus.ts`
The supplemental `signal` fetch is widened from the comm-error
aggregates to a
**union**: the existing clause OR `state != "green"`.
```
before: filter=(dimension = "d6" || … || dimension = "d5-single-pill-e2e") && key !~ "%/%"
after: filter=((dimension = "d6" || … || dimension = "d5-single-pill-e2e") && key !~ "%/%") || (state != "green")
```
The existing fetch was not a viable vehicle as-is: it excluded these
rows twice
over — dimension `d5` is absent from `FLEET_COMM_AGGREGATE_DIMENSIONS`
entirely,
*and* its `key !~ "%/%"` predicate excludes every per-cell ladder-rung
key.
Relaxing only the `%/%` predicate would still have missed them.
**Why widen the existing fetch rather than add a second one:** it reuses
the
freshness guard (`supplementalRowIsOlder`) and the chimera-avoidance
merge
verbatim — that merge already reasons carefully about a newer bulk row
vs an older
signal-bearing twin, and duplicating it would be the risky part. It also
keeps
first paint at one extra request instead of two.
**Why scope by STATE, not by key:** signal-provenance is consulted
**only** in
`classifyRung`'s red branch — green and degraded paths never touch it —
so only
non-green rows can change a chip verdict. `!= "green"` rather than an
explicit
red/degraded list is deliberate: it also catches an out-of-vocabulary
state, which
`rankOfState` ranks **worst** — so the rows that matter most can never
fall outside
the fetch. The comm-error clause is kept as an independent disjunct, not
replaced:
a **green** aggregate row can still carry an active comm-error blob.
The fetch is page-capped at `MAX_INITIAL_FETCH_PAGES = 20`, projected to
`STATUS_LIST_FIELDS,signal`, and non-fatal at the fetch site —
truncation is handled
**asymmetrically**: supplemental truncation degrades (warn, use what
arrived), bulk
truncation throws. (Round-1 unit FETCH; the fetch was originally
unbounded, and its
rejection propagated to `setRows([])`, so an *enrichment* failure
blanked the whole
matrix.)
**Rejected: suppressing the chip while pending.** A blank or spinning
chip for up
to an hour on a large slice of the matrix hides reds exactly as
effectively as a
gray one, while additionally looking broken.
### What each part buys
Real engine over all 3082 production rows, three wires built from the
identical
row set *(green / amber / red / gray over the 708 catalog cells the band
counts)*:
| Wire | pre-fix engine | post-fix engine |
|---|---|---|
| FULL (server / `/api/matrix`) | 596 / 37 / 51 / 24 | 596 / 37 / 51 /
24 |
| COLD, pre-fix projection | **596 / 0 / 0 / 112** | 596 / 37 / 51 / 24
|
| COLD, post-fix projection | 596 / 0 / 49 / 63 | 596 / 37 / 51 / 24 |
- The pre-fix browser wire collapsed **88 cells** (51 red + 37 amber)
into gray.
The entire red *and* amber population was zeroed, because the
`!signalKnown`
early return preempted `FAIL_FRESH`, `FIRST_STRIKE_FRESH` **and**
`INFRA_RED_FRESH` alike.
- **(b) alone is not sufficient**: it recovers 49 of 51 reds and
**zero** of the
37 ambers, leaving 39 cells still mis-painted.
- **(a) is what delivers correctness** — with it, all three wires equal
server
truth exactly.
- **(b) is what keeps (a) from having to guess.** Today (a)-alone
happens to cost 0
false-positive cells, because no cell's verdict-determining rung is
currently a
pure infra red. That is luck, not a guarantee: prod holds 21
infra-classed red
rows. (b) supplies the actual evidence so those stay correctly gray, and
it also
feeds `decodeCellCommError` for non-green per-cell rows.
## Value test — per cell, all 6 (≥3 required)
Every affected cell verified individually, not inferred from a shared
symptom.
Live DOM on the real surface, plus the real engine driven offline over
all 3082
production rows through faithful replicas of the pre- and post-fix
browser wire:
| Cell (`d5:<slug>/multimodal`) | PB row | Server truth | Cold PRE |
Cold POST | Live chip PRE | Live chip POST | Flipped |
|---|---|---|---|---|---|---|---|
| `agno` | red, fc 1817 | red | **gray** | red | gray `D4` + `1P ✗` |
**red `D4`** | ✅ |
| `crewai-crews` | red, fc 1716 | red | **gray** | red | gray `D4` + `1P
✗` | **red `D4`** | ✅ |
| `llamaindex` | red, fc 2005 | red | **gray** | red | gray `D4` + `1P
✗` | **red `D4`** | ✅ |
| `mastra` | red, fc 1806 | red | **gray** | red | gray `D4` + `1P ✗` |
**red `D4`** | ✅ |
| `ms-agent-python` | red, fc 1710 | red | **gray** | red | gray `D4` +
`1P ✗` | **red `D4`** | ✅ |
| `built-in-agent` | red, fc 1803 | red | **gray** | red | gray `D4` +
`1P ✗` | **red `D4`** | ✅ |
6/6 flip. All six are `errorClass: conversation-error` — a real probe
conversation
failure, not in `INFRA_ERROR_CLASSES`, so `FAIL_FRESH` is the correct
classification. (Prod figures; `built-in-agent` is green on staging, so
staging
shows 5.)
**Depth distribution is byte-identical across the pair** (`D6:632 D5:0
D4:71 D3:0
D0:5` both times). The fix changes only the *colour*, never the depth —
the `D4`
label on these chips was always correct, so this must not be
mis-validated by
looking for a depth change.
## Tally-site audit
Every tally is downstream of `chipColor`, so all self-correct — none
needed
refactoring. All verified on the live surface post-fix:
| Site | Mechanism | Pre-fix | Post-fix | Verified |
|---|---|---|---|---|
| `feature-grid.tsx` `computeColumnTally` | gray-skip | `✗ 0` | `✗ 8`
(Agno) | ✅ header DOM |
| `feature-grid.tsx` `computeColumnTallyDetail` | gray-skip | empty list
| 8 features incl. **Attachments** | ✅ panel opened via
`tally-trigger-red` |
| `page-stats.ts` `computeHealthStats` | `case "gray": noData++` | `0
red / 76 no data` | `51 red / 25 no data` | ✅ band DOM + screenshot |
| `page-stats.ts` `computeDepthDistribution` | — (no gray gating at all)
| `D4:71` | `D4:71` | ✅ unaffected |
| `page-stats.ts` `computeD6Stats` | `d6Effective` | `null` → gray |
`"red"` → red | ✅ golden fixture |
**No other `chipColor` consumer changes behaviour.**
`d0-gone-predicate.ts` gates
the outage alerter on `chipColor === "red"`, but `d0-gone-monitor` reads
*full rows
with `signal`*, so provenance is always known there — as it is for
`/api/matrix`.
The polarity flip is reachable **only** where rows are projected, i.e.
exclusively
the browser cold-load path.
## A test asserted the buggy behaviour
`api-matrix-equivalence.test.ts` asserted:
```ts
expect(serverChip).toBe("red");
expect(coldLoadChip).toBe("gray");
expect(serverChip).not.toBe(coldLoadChip);
```
commented as "the state the browser converges to after its supplemental
`signal`
fetch … the more accurate answer." That comment is factually wrong — the
supplemental fetch resolved in ~1 s and structurally excluded those rows
twice over
— and the assertion pinned the defect as intended behaviour. **This is
why CI never
caught it.** Read in full, it is asserting the defect, not an adjacent
legitimate
case: the fixture is a genuine product red (`errorDesc: "assertion
failed: wrong
answer"`, no infra class) with `signal: undefined` on the red rung.
Inverted to assert **red on both paths**, with the fail-safe direction
explained
inline. Same for the harness's `§7 I5` case (`cell-model-v2.test.ts`)
and
`classifyRung`'s unit case. Four golden-master fixtures
(`pos-d{3,4,5,6}-red-signal-unknown`) re-frozen.
Negative controls added so the flip can't over-reach: an infra-classed
red **still**
grays (positive evidence present), and `§7 I4` (infra-only red → gray,
`isRegression` false) still passes untouched.
## The structural lever: coherence invariant INV7
`cell-model.coherence.test.ts` asserted INV1–INV6, all of which relate
`chipColor`
only to the other **chip-side** outputs. **None inspected the
`d3`/`d4`/`d5`/`d6`
depth pills.** So the engine could return one object saying both
`d5.status ===
"red"` and `chipColor === "gray"` with every invariant intact — which is
exactly
what happened, on every cold load.
INV7: **a gray chip above a red depth pill requires positive infra
evidence on
every contributing red row.** Red-greened against the old polarity — it
fails on 3
fixtures with a legible message:
```
pos-d5-red-signal-unknown: INV7 gray chip over a red pill requires POSITIVE infra
evidence on every red row — d5:acme/agentic-chat has none (signal STRIPPED (pending)).
A red whose cause is merely unknown must render red, not gray.
```
It closes the gap in the **only** safe direction: by fixing the chip
upward, never
by muting the strip. Deriving the pills from the classified
contributions — the
superficially cleaner unification — would render `1P ✗` as `1P ?` on
cold load and
destroy the only on-page element that was telling the truth, converting
a visible
inconsistency into an invisible failure.
INV7's own limitations are disclosed under
[Known limitations of the guards
themselves](#known-limitations-of-the-guards-themselves).
---
# Review-round fixes folded in
Three full 12-reviewer review rounds were run against this branch. Round
1
contributed five units plus an integration fix; round 2 contributed four
more. The
loop was **deliberately stopped** after round 3 by explicit decision
rather than
running a fourth; round 3's findings are catalogued (132 items in
`DEFERRED-r3.md`) and routed to the follow-up epic, not silently
dropped.
## Round 1 — five units
| # | Commit | Concern | Surface |
|---|---|---|---|
| G1 | `2a68282d` | Signal-provenance was derived family-wide but gated
a red-row-scoped predicate. Precondition moved into `foldFamily` as
`redSignalKnown`; `RawRung.signalKnown` **removed entirely** with all
its plumbing. | production + tests |
| A5 | `ebd7ac68` | The autocancel fake servers ignored
`filter`/`fields`, so the supplemental fetch's leftover-append silently
repaired a dropped bulk page — three pagination guards passed under
every mutation. Servers rebuilt to evaluate the query. | test-only |
| FETCH | `f5178765` | The supplemental fetch was unbounded (`for(;;)`),
unprojected, and its rejection propagated to `setRows([])`. Now capped,
projected, and non-fatal at the fetch site. | production + tests |
| INV7 | `3dea9f64` | INV7 was quantified over the whole live map rather
than the cell's contributing keys, and had no `isStaleCell` carve-out —
it failed two classes of legitimate rendering. Scoped, exempted, plus an
anti-vacuity guard. | test-only |
| DOCS | `2797618c` | Stale/unanchored numeric claims (row counts, byte
estimates, re-delivery window) corrected against production and given
their source queries. | comments only |
| — | `938be2ac` | **Integration fix.** A5 and FETCH disagreed about how
a supplemental request is *identified*; all three fake servers
misclassified it as a bulk page and the whole autocancel suite failed
3/3. Both dashboard suites now key on `signal` being IN the projection.
| test-only |
## Round 2 — the four integrated units
Integrated in order Q2 → Q1 → Q3 → Q4. Every guard was re-mutated
**after**
integration and reproduced its pre-integration result; no guard was
disarmed by
the merge.
| # | Commit(s) | Concern | What it proved | Surface |
|---|---|---|---|---|
| **Q2** | `b8b75b7d` | The dashboard's `useLiveStatus` test doubles
carried **three divergent implementations** of the same PocketBase query
evaluator, and the nominally-shared one was the weakest. Collapsed to
exactly one, every semantic a measured observation from a real
PocketBase v0.22.21 server. Deleted autocancel's 380-line private copy.
`evaluatePbList` **cannot throw**, so a parse error can no longer send
NO response and hang the hook to a 20 s `waitFor` death. | Six mutations
(short page dropped in wave merge, boundary logic removed, merge order
reversed, supplemental filter broken, supplemental filter widened to
`""`, bulk filter selects nothing) each reproduced their pre-integration
RED count exactly — 3/1/2/6/5/6 failures. The consolidated evaluator is
strictly stronger, not merely shorter. | test-only (−488 lines net in
the doubles) |
| **Q1** | `75fdfb3a` + `ed10b61f` | `truncated` meant "cap exhausted",
so a **complete** read that happened to end exactly on the cap page was
reported as truncated — the bulk caller threw, burned its reconnects and
landed on a blank dashboard behind an offline banner, on every retry and
every reload. `truncated` now means exactly "there are rows we did not
read", proven by ONE lookahead read of page cap+1 whose rows are
discarded and only whose emptiness is load-bearing. Algorithm hoisted to
an exported `paginateStatusPages(readPage)`. | Lowering the cap 20→10
and 20→5 each produces **10 failed / 6 passed**, where the pre-fix code
was **6/6 GREEN** — i.e. the guard was previously disarmed. A complete
10,000-row dataset renders `live`, `truncated === false`, seeded red
cell still `chipColor: "red"`, page set exactly `[1..21]`; 10,001 rows
still throws with `status: "error"` and "exceeded" in the message. |
production + tests |
| **Q3** | `49dea099` | INV7 false-failed on the ordinary I1 ladder-gap
shape ("no `e2e:` row + red `chat:<slug>`"). Added a carve-out keyed on
`ladderGapDepth`, **one-directional** (a gap *above* the red excuses
nothing) and never firing without a witness rung — plus a two-sided
canary that fails loudly the moment the underlying gap-break is fixed. |
Deleting the `isStaleCell` exemption fails exactly the U8 force-gray
case (the exemption is load-bearing, not decorative). Changing the
engine's gap-break `break` → `continue` fires the **two-sided canary**
with its intended message (`expected 'red' to be 'gray'`) — so the
allowance cannot go on silently excusing a shape the engine no longer
produces. Gapless and gap-above-red incoherence still FAIL; the
legitimate gap shape still PASSES. | test-only |
| **Q4** | `b04330ab` | Three explicitly load-bearing family folds were
invisible at golden-master level — `maxNonInfraRedFailCount`'s MAX,
`allRedSoftClass`'s EVERY quantifier, and `redSignalKnown`'s RED-ROW
scope — because no fixture anywhere carried two *differing* red rows. 5
new fixtures, baseline 56→61, 140 insertions / 0 deletions. | Each of
three mutations now fails via **exactly one** fixture:
`Math.max`→`Math.min` ⇒ `hetero-d4-red-failcount-max`; `every`→`any` ⇒
`hetero-starter-mixed-soft-hard`; `redSignalKnown` red-row→family scope
⇒ `hetero-d4-green-stripped-infra-red`. Precise attribution, not a
shotgun. | test-only |
One merge conflict, in `useLiveStatus.supplemental-bounds.test.tsx`: Q2
deleted the
file-local `classifyRequest` helper on the same lines where Q1 added its
new
`PAGES_THROUGH_LOOKAHEAD` constant. Resolved by keeping **both** intents
rather than
taking a side.
Baseline golden-master key count is **61**, and the completeness check
(`has exactly
one baseline entry per fixture (no stale/missing)`) runs, so a fixture
added without
a baseline entry fails rather than being skipped.
---
# What this PR does NOT fix
**This PR does not fully eliminate the original symptom.** It fixes the
`signal`-polarity masking mechanism and makes the cold load correct. A
genuinely-red
cell can still render **gray, green, or amber** by nine other routes.
All nine were
found by review of this branch and are catalogued, not speculative.
### Quantified by the #6169 sweep (279,936 combinations)
| Mechanism | Rung combinations |
|---|---|
| **M1** — ladder gap-break: `scanWorst` folds the FIRST `ABSENT`/`STUB`
rung as `ABSENT` and **`break`s**, so a fresh non-infra RED on a rung
*above* the gap is never folded and paints gray | **17,088** |
| **M2** — D6 outside the `scanWorst` walk: a red D6 never reaches the
chip | **2,048** |
| **M3** — gap not below the red: the remaining gray-over-red shapes |
**192** |
| | **19,328 total, 0 unexplained** |
M1 is highly reachable in practice: `chat:<slug>` / `tools:<slug>` are
integration-scoped, so one red `chat:` row plus an unemitted
`e2e:<slug>/<featureId>`
row (new feature, rotation slot, D3 driver never ran) is a gray chip
over a red D4
pill. `combine.ts` is deliberately untouched here — fixing any of M1–M3
re-verdicts
cells across the matrix and therefore needs its own PR with a
golden-master
**baseline re-freeze** and a **live value test**. Q3's INV7 carve-out is
silent about
M1 **on purpose and only about that**, and its two-sided canary is the
tripwire that
forces the carve-out to be deleted the moment M1 is actually fixed.
### Found in round 3, not quantified by the sweep
4. **Prototype-chain property lookups invert the anti-masking guard.**
`STATE_RANK` / `rankOfState`, `WORST_STATE_RANK` / `worstStateRank`,
`firstStrikeConfig[kind]` and `CATALOG_TO_D5_KEY` are plain object
literals read
without an own-property guard, so a runtime value colliding with an
`Object.prototype` member (`constructor`, `toString`, `valueOf`, …)
resolves to an
inherited function that the `??` fallback then never replaces. For
`rankOfState`
the out-of-vocabulary guard **inverts** — an unknown state, which the
code
deliberately ranks worst, instead classifies as an infra red and grays;
`worstStateRank` silently swallows the row entirely. `CATALOG_TO_D5_KEY`
is
unguarded at four further sites in `cell-model.ts`, where the
consequence is a
**whole-matrix render crash**. Fix is
`Object.prototype.hasOwnProperty.call` (or
a prototype-free table) at each site.
5. **Wave truncation bypasses the fail-loud throw entirely.** In the
fan-out merge,
a short page inside a wave causes every page issued *after* it in the
same wave
to be discarded — sound for a stable collection, but the pages resolve
concurrently against a table the harness writes continuously. The
discard
produces no log, no warning, no counter, no `degraded` flip, and
**`truncated`
stays `false`** because it is only ever set by the cap lookahead. So the
bulk
caller's deliberate "a truncated BULK read cannot degrade gracefully …
throw"
posture fires only for cap exhaustion, never for this class of loss. The
missing
rows render as *absent*, which `buildCellModel` resolves to `ABSENT` /
`NO_DATA`
and paints gray — the exact polarity this PR exists to remove.
`INITIAL_FANOUT_BATCH`
is documented as "purely a WIRE-EFFICIENCY knob" but is load-bearing for
correctness: at a larger batch the loss is `INITIAL_FANOUT_BATCH - 1`
pages.
6. **`INITIAL_PAGE_SIZE` above the server clamp grays ~6/7 of the
matrix.**
Termination is the client-side comparison `first.length ===
INITIAL_PAGE_SIZE`,
correct only because PocketBase clamps `perPage` to 500 server-side.
Raise the
constant — the obvious "fewer round-trips" tuning, which the surrounding
comment
invites — and every page returns 500, i.e. *short*, so the loop stops
after page
1 and returns ~500 of ~3100 rows with `truncated: false`. The bulk
caller sees a
complete read, never throws, and ~6/7 of the matrix renders no-data gray
with no
warning and no banner. The `truncated` contract cannot catch it because
the read
genuinely never hit the cap.
7. **An empty bulk page 1 publishes `live` with 0 rows.**
`paginateStatusPages`
returns `{rows: [], truncated: false}` for an empty collection and
`connect()`
then does `setRows([])` + `setStatus("live")` + `setError(null)` with no
empty-collection guard. Every cell renders no-data gray behind a healthy
"live"
indicator. Untested — the smallest fixture anywhere in these files is 3
rows.
The server-side analogue is already a known deferred item.
8. **First-strike de-amplification fires at `fail_count === 0`, painting
a genuine
red AMBER.** The gate is `maxNonInfraRedFailCount < threshold` with
`threshold`
2, so `fail_count` of **0** de-amplifies exactly like 1 →
`FIRST_STRIKE_FRESH` →
amber, i.e. "transient, not yet actionable". The threshold's semantics
are valid
only if a red row's `fail_count` is always ≥ 1, which is asserted as a
*producer*
property and enforced nowhere on the read side; out-of-vocabulary states
(the
harness can persist `"error"` as a no-data marker, which `rankOfState`
ranks above
red) and back-filled or externally-written rows both reach the
classifier at 0. A
`fail_count` of 0 on a red row is not positive evidence of a first
strike. The
golden master cannot see it: every red-bearing fixture uses
`fail_count: opts.failCount ?? (isRed ? 1 : 0)`, so no fixture puts a
red row at 0.
9. **An infra-red or `degraded` D1/D2 over a green ladder paints
chip=green,
achieved=6.** `combine.ts`'s §F gate fires only on `contribution ===
"FAIL_FRESH"`
in both the achieved-depth walk and `computeChip` step 1;
`INFRA_RED_FRESH` and
`DEGRADED` fall through as non-gating, so the cell keeps its green
ladder verdict
with `d6Effective: "green"`. Measured on the branch engine: `D1
infra-red
(driver-error) + green D2 → chip=green ach=6 reg=false d6Eff=green`, and
the same
for `D1 degraded`. **Mutation-proven invisible to the golden master:**
patching
*both* §F gate sites to also fire on `INFRA_RED_FRESH` and `DEGRADED` —
a
materially different verdict for every column whose health probe is
failing —
leaves **all 61 baseline entries byte-identical**. `health:<slug>` /
`agent:<slug>`
are integration-scoped, so this is whole-column. The polarity flip
closed the
`signal: undefined` hole here and left the `driver-error` / `abort` hole
wide open
— and `driver-error` / `abort` on a health probe is precisely what an
unreachable
container or a cancelled sweep produces.
# Known limitations of the guards themselves
Disclosed rather than omitted. These are properties of this PR's own
tests.
- **INV7's Q3 carve-out does not cover the M2 shape.** It is keyed on
`ladderGapDepth` and excuses only a gap strictly *below* the shallowest
red pill,
which is M1. The D6-outside-the-walk shape (M2, 2,048 combos) is a
different
mechanism and is not addressed by the carve-out.
- **The starter-axis INV7 canary short-circuits and proves nothing.**
`assertChipStripCoherent` returns immediately on `m.chipColor !== "gray"
||
redDepth === null`. The hard-red starter cell the test builds has
`chipColor ===
"red"`, so the function returns on the **first** disjunct — not on the
strip condition the comments claim. The call would pass identically if a
starter
cell *did* expose a red depth strip. The canary role is actually carried
by the
four `lvl.exists === false` / `lvl.status === null` assertions beside
it; the
`assertChipStripCoherent` call adds nothing. Compounding it,
`stripReadsRed` — the
identifier the comments reason about — does not exist anywhere in the
repo.
- **The five INV7 "teeth" assert `/INV7/`, which the drift-guard message
also
matches.** `assertChipStripCoherent` raises two structurally different
failures
and both messages contain the literal `INV7`: the keyspace-drift guard,
and the
real coherence failure. Every teeth test asserts only
`.toThrow(/INV7/)`. So if
`contributingKeys` ever drifts from the engine keyspace —
`collectAgentLadder`
gains a rung, `CATALOG_TO_D5_KEY` fan-out changes shape, a `keyFor`
segment changes
— `redRows` goes empty, the drift guard fires, and **all five teeth
still pass**
while INV7 has gone vacuous over the entire fixture matrix. That is
precisely the
state the drift guard was written to make impossible; it is silenced
inside the
only tests that reach it. The fix is to assert the specific message
(`/requires POSITIVE infra evidence/`) and pin the drift message in its
own case.
- **The test double cannot express absent-vs-null projection
provenance.**
`applyPbFields` silently drops projection tokens it does not model, so
the fake
cannot distinguish a key *omitted by a `fields=` projection* from a key
present
with value `null` — which is the single distinction `redSignalKnown`
rests on.
Fixtures therefore cannot validate the provenance semantics the fix is
built on;
that validation lives only in the live red-green capture above.
# Follow-ups
- **Epic: "A red must render red" — a single enforcement point for the
showcase
cell-model read model**, on Notion under CopilotKit *Plans / Proposals*:
https://app.notion.com/p/3a83aa38185281b99f7bfdd1d0f46bb9 — carries
M1–M3 and the
round-3 mechanisms above toward one enforcement point rather than nine
point-fixes.
- **`DEFERRED-r3.md` — 132 catalogued items** from review round 3
(comment accuracy,
test-double conformance, coverage gaps, altitude). Nothing in round 3
was
discarded; it was catalogued and routed.
# Quality gate
Re-run at head `b04330ab` in a clean detached worktree.
- **`oxfmt@0.36.0 --check`**: clean — "All matched files use the correct
format" (16
changed TS/TSX files)
- **`oxlint@1.51.0`** (repo `.oxlintrc.json`): **5 warnings, 0 errors.**
4 of the 5
reproduce verbatim on the `origin/main` versions of the same files
(`unicorn(consistent-function-scoping)` on `c`, `eslint(no-shadow)` on
`rows`,
`unicorn(no-array-reverse)`,
`typescript-eslint(consistent-type-imports)`). The one
net-new against `origin/main` is a second
`unicorn(consistent-function-scoping)` warning, on the test-local helper
`lvl` in
`cell-model.coherence.test.ts`. Zero errors; the repo's `oxlint` CI job
is green.
- **`tsc --noEmit`**: **0 errors** in `shell-dashboard`; `harness` clean
under
`tsconfig.build.json`. `harness`'s `tsc --noEmit` reports one
pre-existing error,
`frontend-matrix.test.ts` resolving
`../../../shell/src/data/frontend-catalog.json`
— a stale path in a file byte-unchanged from `origin/main`.
- **`shell-dashboard` tests** (measured locally at `b04330ab`): **71
files / 1407
passed / 1 skipped / 0 failed.** Scoped to `src/`;
`tests/runtime-env-switch.spike.test.ts` was deliberately not run (it
spawns
`next build` plus two `next start` invocations and is not a unit test).
- **`harness` tests** (measured locally at `b04330ab`): **3659 passed /
18 skipped /
3 failed** — the same three pre-existing D5 fixture-registry-drift
failures, in
files **byte-unchanged from `origin/main`** and driven entirely by
inputs none of
this PR's 17 files touch:
- `frontend showcase matrix > plans every runnable catalog cell without
loss or duplication` — `expected 664 to have a length of 660`
- `d5-mapping-drift > dashboard CATALOG_TO_D5_KEY structurally mirrors
harness REGISTRY_TO_D5` — the drift regex cannot locate
`CATALOG_TO_D5_KEY` in `shell-dashboard/src/lib/live-status.ts` (a
675-byte re-export shim on `main`, not modified here)
- `D5_REPRESENTATIVES > covers every D5FeatureType that has a registered
script` — missing `browser-use-smoke`
Those three are now **explicitly quarantined** by the `test /
unit-showcase`
workflow that landed on `main` in `7282ecddfb`
(`showcase/harness/vitest.quarantine.json`), and that workflow's
**quarantine
ratchet** re-runs each one to confirm it *still* fails — so they neither
hide nor
block. At this PR's head the ratchet passes: `Quarantine ratchet OK: all
3
quarantined file(s) still fail.`
- Diff hygiene: no lockfiles, no generated `src/data/*.json`.
## CI provenance — this is the PR's first honest CI
`main` has been **merged in** at `7282ecddfb` (merge commit `12ea621a`,
textually
clean, zero conflicts; the 13 commits above are unrewritten). This
matters for more
than being up to date: the `harness unit suite` and `shell-dashboard
unit suite`
jobs **did not exist** at `b04330ab`, because the workflow that enables
them landed
on `main` afterwards and GitHub runs `pull_request` workflows from the
PR head. The
earlier "17 pass" therefore ran **none** of the ~5,000 tests covering
this PR's own
subject matter (`cell-model/*` and `useLiveStatus*`).
- CI at this head (`12ea621a`): **21 pass / 0 pending / 0 fail** across
24 registered
rows.
- Both new jobs run and both pass — `shell-dashboard unit suite` **70
files / 1404
passed / 1 skipped**, `harness unit suite` green, quarantine ratchet OK.
- The **Q1–Q4 guard battery (16 re-proofs)** was re-run at the merge
head and
reproduced its pre-merge numbers exactly, so nothing was disarmed by the
merge:
Q1 cap→10 and cap→5 each **10 failed / 6 passed** (unmutated **16/16
green**);
Q2 M1–M6 = **3 / 1 / 2 / 6 / 5 / 6** (M4 flips the supplemental cap
test, M6 the
bulk cap test, M5 red in both worlds by design); Q3 `isStaleCell`
deletion = **1
failure** (the U8 force-gray case) and the engine `break`→`continue`
fires the
two-sided canary with `expected 'red' to be 'gray'`; Q4 each of the
three fold
mutations = **1 failed / 61 passed** via exactly one fixture
(`hetero-d4-red-failcount-max`, `hetero-starter-mixed-soft-hard`,
`hetero-d4-green-stripped-infra-red`), baseline key count **61**,
completeness
check present.
# Review guidance
**Surface: 17 files, +4880 / −382.** Most of that volume is tests,
fixtures, and the
regenerated golden-master baseline. Only three files carry production
behaviour:
`cell-model.contribution.ts`, `cell-model.ts`, and `useLiveStatus.ts`.
**The four round-2 units are separable commits** — `b8b75b7d` (Q2),
`75fdfb3a` +
`ed10b61f` (Q1), `49dea099` (Q3), `b04330ab` (Q4) — and round 1's are
too, so this
can be read unit by unit rather than as one diff.
**Highest-value review targets, in order:**
1. **`cell-model.contribution.ts` — the gate.** `if (fold.redSignalKnown
&&
!fold.hasNonInfraRed)`. This is the whole polarity decision. Worth
checking: that
`redSignalKnown` really is red-row-scoped inside `foldFamily` (a
family-wide
derivation is what made `INFRA_RED_FRESH` unreachable in the browser),
and that
the two conjuncts are genuinely independent evidence rather than one
restated.
2. **`useLiveStatus.ts` — `truncated` semantics.** Q1 redefined
`truncated` from
"cap exhausted" to "there are rows we did not read", proven by one
lookahead read
of page cap+1 whose rows are discarded and only whose emptiness is
load-bearing.
Worth checking: the asymmetry (bulk throws, supplemental warns) is
applied at the
right call sites, and that the lookahead genuinely cannot be confused
with a data
page. Note limitation 5 above — this contract does **not** cover
wave-level loss.
Everything else is fixtures, doubles, and the coherence suite; the
disclosed
weaknesses in those are listed under *Known limitations of the guards
themselves*
rather than left for a reviewer to rediscover.
|
||
|
Jordan Ritter
|
5bf4b6f74f |
ci(showcase): pull demo-file LFS objects in the Python unit-test job
PR #6163 converted showcase/integrations/*/public/demo-files/* to Git LFS. showcase_validate.yml's python-unit-tests job checks out without LFS, so the working tree holds 129-byte pointer text instead of the real assets. The new ms-agent-python multimodal test is the first Python test to read those bytes: pypdf reads the pointer, extracts nothing, and the test fails with "real pypdf text extraction produced nothing". Fetch only the demo-file assets (40 objects, ~320 KB) rather than setting lfs: true, which would pull all ~248 tracked objects (~475 MB, including 13-26 MB README gifs) into a 2-4 minute job under a 10 minute cap. A job that hits timeout-minutes reports 'cancelled' -- neither success nor failure -- and silently suppresses alerting, so the timeout margin is worth protecting. The repo is public, so LFS downloads resolve anonymously and the pull works with persist-credentials: false. A post-pull %PDF- header check fails the step immediately if the pull ever no-ops, instead of surfacing minutes later as a misleading pytest assertion. |
||
|
Jordan Ritter
|
db75a04837 |
chore(showcase): mark multimodal unsupported for llamaindex and crewai-crews (#6158)
## What this is
`multimodal` (Attachments) has never worked on **`llamaindex`** or
**`crewai-crews`**, but both manifests
listed it under `features`, so the fleet probed it and reported **red**.
A red chip says "this regressed".
The truth is "this was never built". This PR marks both cells
**unsupported** instead. It does **not**
implement the feature, and it does **not** suppress the cell — the cell
still exists, the demo stays wired,
and the chip renders the 🚫 unsupported glyph.
## Mechanism used (existing, not invented)
The repo already has exactly one way to declare a feature unsupported
for an integration:
**`not_supported_features` in the integration's `manifest.yaml`**. The
full derivation chain:
| Step | Location |
|---|---|
| Declaration | `showcase/integrations/<slug>/manifest.yaml` ->
`not_supported_features:` |
| Schema | `showcase/shared/manifest.schema.json` — *"feature IDs that
this integration's framework cannot architecturally support … excluded
from parity computation"* |
| Status fold |
`showcase/harness/src/shared/catalog/catalog-flatten.ts:239` —
`determineCellStatus()` checks `not_supported_features` **first**,
returns `status: "unsupported"` |
| Input mapping |
`showcase/harness/src/shared/cell-model/catalog-input.ts:53` —
`isSupported: cell.status !== "unsupported"` |
| Model | `showcase/harness/src/shared/cell-model/cell-model.ts:847` —
`if (!isSupported) return UNSUPPORTED;` (the frozen singleton at `:551`:
`supported: false`, `chipColor: "gray"`, `isRegression: false`) |
| `/api/matrix` | `showcase/harness/src/http/matrix.ts:206` ->
`matrix-compute.ts:53` — projects that same model, so the API value
**is** the rendered chip by construction |
| Render |
`showcase/shell-dashboard/src/components/unified-cell.tsx:308` — `if
(!model.supported)` renders `data-testid="unified-cell-unsupported"`
with 🚫 and `title="Not supported by this framework"` |
The mechanical guard at `catalog-flatten.ts:169` rejects a feature that
appears in **both** `features` and
`not_supported_features`, so each entry was **moved**, not duplicated.
Note this mechanism is strictly stronger than a probe-side skip:
`buildCellModel` returns `UNSUPPORTED`
regardless of what the live PocketBase row says. Verified against the
existing not-supported cells on these
same two integrations, which carry **green** PB rows and still render 🚫:
```
llamaindex/gen-ui-interrupt matrix: chip=gray supported=False | PB rows: d5=green d6=green e2e=green
llamaindex/shared-state-streaming matrix: chip=gray supported=False | PB rows: d5=green d6=green e2e=green
crewai-crews/mcp-apps matrix: chip=gray supported=False | PB rows: d5=green d6=green e2e=green
```
So the cell can never read green *or* red once declared here — which is
the property we want.
## Per-integration reason (recorded inline in each manifest)
**`llamaindex` — upstream gap.** The pinned
`llama-index-protocols-ag-ui==0.2.2`
(`llama_index/protocols/ag_ui/utils.py:82-85`) passes an AG-UI
`UserMessage`'s `content` straight into
`ChatMessage(...)`. A text-only turn passes a plain string (fine — every
other llamaindex cell is green);
an attachment turn passes a **list** of AG-UI content-part models, which
pydantic routes into
`ChatMessage.blocks`, a union discriminated on `block_type` — a field
AG-UI's `TextInputContent` /
`ImageInputContent` / `BinaryInputContent` do not carry. Live backend
error:
```
pydantic_core._pydantic_core.ValidationError: 3 validation errors for ChatMessage
blocks.0
Unable to extract tag using discriminator 'block_type' [type=union_tag_not_found,
input_value=TextInputContent(type='te... image I just attached'), input_type=TextInputContent]
```
Fixing it needs a content-part ->
`TextBlock`/`ImageBlock`/`DocumentBlock` conversion, upstream or on our
side of `get_ag_ui_workflow_router`. This PR also corrects that module's
docstring, which asserted the
router *"normalizes them via the OpenAI `input_file` path"* — it does
not.
**`crewai-crews` — never implemented, ours.** `src/agent_server.py`
registers a dedicated AG-UI endpoint for
every other demo but **no `/multimodal` route** (the block ends at the
catch-all
`add_crewai_crew_fastapi_endpoint(app, LatestAiDevelopment(), "/")`),
and there is no
`src/agents/multimodal_agent.py`. So
`src/app/api/copilotkit-multimodal/route.ts` aliases the generic shared
crew, which has no vision handling and dies on a content-part message:
```
[CopilotKit] Error (agent_run_error_event): Error: thread=… run=…: CrewAI flow failed; see server logs
```
That route file's own header comment already concedes the gap ("A
dedicated per-demo crew with vision-tuned
agent prompts is tracked as follow-up work").
## Proof
Method: the **real** `GET /api/matrix` handler (`registerMatrixRoute`)
driven over the **real** production
PocketBase `status` collection (all 3082 rows, fetched verbatim from
`showcase-pocketbase-production.up.railway.app`) and the **real**
on-disk manifests (default `loadCells` =
`buildCatalogCells`, the single flattening authority). Same rows, same
fixed clock
(`now = max(observed_at) + 60s = 1784932566438`) for both runs — the
only variable is the manifest diff.
### BEFORE — real `GET /api/matrix`, real prod PocketBase rows,
manifests at `origin/main` (
|
||
|
Jordan Ritter
|
59f275eedc |
fix(showcase/mastra): restore shared-tools symlink, shrink erosion ratchet (#6161)
## What
`showcase/integrations/mastra/shared-tools` was a **real committed
directory** where a **symlink into the single source of truth** belongs.
This restores the symlink (`-> ../../shared/typescript/tools`) and
shrinks the erosion ratchet accordingly.
This is the iron-rule violation described in `showcase/AGENTS.md` → "The
single-source symlink mechanism": `*/shared-tools`, `*/tools`, and
`*/_shared` are meant to be symlinks into `showcase/shared/...`, so
edits to the shared source silently do NOT reach an integration whose
symlink has been clobbered by a real directory.
## Root cause
| | |
|---|---|
| Symlink originally added | `93d5815cdb` — *"refactor: add shared tools
symlink to mastra, update path alias"* (target
`../../shared/typescript/tools`) |
| Symlink clobbered | `534cd1efa7` — *"fix(showcase): D5 integration
fixes across 12 frameworks"* — deleted the symlink (`shared-tools \| 1
-`) and committed **14 real files** in its place |
That commit clobbered `shared-tools` for **three** TS integrations at
once: `mastra`, `langgraph-typescript`, and `claude-sdk-typescript` —
all three grandfathered in
`showcase/scripts/validate-shared-symlinks.baseline.json`.
mastra's Dockerfile already documented the symlink as the expected
state, so the tree had drifted from its own documented contract:
```
# shared-tools/ is a symlink to ../../shared/typescript/tools — resolved at
# build time by CI which copies the target into the build context.
```
## Divergence inventory — **NONE**
Diffed the real directory against `showcase/shared/typescript/tools`
exhaustively **before** changing anything. **Zero divergence**:
identical file set, identical git blob hashes, identical sha256s across
all 14 files.
| File | git blob (main) | classification |
|---|---|---|
| `__tests__/generate-a2ui.test.ts` | `b3c8223e` | identical copy |
| `__tests__/get-weather.test.ts` | `57dd5b94` | identical copy |
| `__tests__/query-data.test.ts` | `ef284b0b` | identical copy |
| `__tests__/sales-todos.test.ts` | `8c5c0d6e` | identical copy |
| `__tests__/schedule-meeting.test.ts` | `ab470b3b` | identical copy |
| `__tests__/search-flights.test.ts` | `f64b8782` | identical copy |
| `generate-a2ui.ts` | `7d76c00c` | identical copy |
| `get-weather.ts` | `0b444f7a` | identical copy |
| `index.ts` | `3b0e2a45` | identical copy |
| `query-data.ts` | `48a2cbdc` | identical copy |
| `sales-todos.ts` | `96b1874f` | identical copy |
| `schedule-meeting.ts` | `d5e626e8` | identical copy |
| `search-flights.ts` | `7a943023` | identical copy |
| `types.ts` | `f87f26d1` | identical copy |
Classification totals:
- **stale** (shared moved on, mastra behind): none
- **mastra-specific** (intentional or accidental local edit): **none**
- **additive** (file only in mastra's copy): none
- files present in one and not the other: **none**
```
$ git diff --no-index --stat integrations/mastra/shared-tools shared/typescript/tools
(no output — identical)
```
Because the divergence set is empty, restoring the symlink is a **pure
structural fix with zero content change** — nothing mastra-specific
existed in the copy, so nothing can be silently lost. The restored link
blob is `ddc634b6` — **the same git object** the pre-erosion symlink
had.
## AFTER — structural proof
```
$ ls -la showcase/integrations/mastra/shared-tools
lrwxr-xr-x shared-tools -> ../../shared/typescript/tools
$ git ls-files -s showcase/integrations/mastra/shared-tools
120000
|
||
|
Jordan Ritter
|
7b28934387 |
fix(showcase): use each integration's own name in demo page titles (#6162)
## What `showcase/integrations/*/src/app/demos/layout.tsx` was cloned from `langgraph-python` and 14 copies kept its hardcoded `"LangChain - Python"` title. Every `/demos/*` page in those integrations rendered someone else's name in the browser tab. Each file now uses the display name from its own `manifest.yaml` `name:` field. Two string literals per file, nothing else. ## Convention matched The already-correct integrations hardcode their own display name, which equals their `manifest.yaml` `name:`: - `langgraph-typescript` → `LangGraph (TypeScript)` - `strands-typescript` → `AWS Strands (TypeScript)` Same shape used here: bare name when no demo slug is resolvable, `"<Name> - <Demo>"` otherwise. ## Full list of mismatched titles (all fixed) | Integration | Before | After | | --- | --- | --- | | `ag2` | LangChain - Python | AG2 | | `agno` | LangChain - Python | Agno | | `built-in-agent` | LangChain - Python | CopilotKit's Built-in Agent | | `claude-sdk-python` | LangChain - Python | Claude Agent SDK (Python) | | `claude-sdk-typescript` | LangChain - Python | Claude Agent SDK (TypeScript) | | `crewai-crews` | LangChain - Python | CrewAI (Crews) | | `langgraph-fastapi` | LangChain - Python | LangGraph (FastAPI) | | `langgraph-python` | LangChain - Python | LangGraph (Python) | | `langroid` | LangChain - Python | Langroid | | `llamaindex` | LangChain - Python | LlamaIndex | | `mastra` | LangChain - Python | Mastra | | `pydantic-ai` | LangChain - Python | PydanticAI | | `spring-ai` | LangChain - Python | Spring AI | | `strands` | LangChain - Python | AWS Strands (Python) | `langgraph-python` is included because `"LangChain - Python"` is the legacy Notion partner-column label, not the integration's name — its own `manifest.yaml` and root layout both say `LangGraph (Python)`. Untouched (already name themselves correctly): `langgraph-typescript`, `strands-typescript`, `ms-agent-dotnet`, `ms-agent-harness-dotnet`, `ms-agent-python`. ## Proof — rendered titles from locally served pages Not a source diff: `next dev` per app, real `document.title` read via Playwright / served HTML. Red → green on the same running server (`mastra`, `/demos/multimodal`, port 3411): ``` RED mastra DOM document.title = "LangChain - Python" <- HEAD version of the file GREEN mastra DOM document.title = "Mastra" <- with this change ``` `langgraph-python` (`/demos/multimodal`, port 3415) — the only integration whose middleware sets `x-pathname`, so it is the one that exercises the `"<Name> - <Demo>"` branch: ``` BEFORE <title>LangChain - Python - Attachments</title> AFTER <title>LangGraph (Python) - Attachments</title> ``` Second sample, `strands` (`/demos/multimodal`, port 3414): ``` BEFORE <title>LangChain - Python</title> AFTER <title>AWS Strands (Python)</title> ``` Control — `langgraph-typescript` (`/demos/multimodal`, port 3413), file not touched: ``` BEFORE <title>LangGraph (TypeScript)</title> AFTER <title>LangGraph (TypeScript)</title> (unchanged) ``` ## Why the title is not read from `manifest.yaml` at runtime The obvious "right level" fix is to have `generateMetadata` read `manifest.name` instead of a literal. I built that, then backed it out: **15 of 20 runner images never copy `manifest.yaml`**, so a `manifest.yaml` read on the no-slug path (which is the path every integration except `langgraph-python` actually takes, since only that one has middleware setting `x-pathname`) would throw ENOENT and 500 **every** demo page in production. `langgraph-python`'s Dockerfile documents exactly this failure mode at the line where it copies the manifest in. Making the config the runtime source would mean touching 15 Dockerfiles and re-verifying 15 image builds — out of proportion for a title string, and a much larger blast radius. ## Adjacent issues found, deliberately not fixed here 1. **`x-pathname` is only set by `langgraph-python`** (`src/middleware.ts`). Everywhere else `generateMetadata` never resolves a slug, so demo pages show just the integration name and never `"<Name> - <Demo>"`. The per-demo title needs middleware per app (or a different pathname source) — separate change. 2. **`google-adk` has no `src/app/demos/layout.tsx`** at all, so its demo pages fall back to the root layout's `"CopilotKit Showcase"`. That is a missing file, not a wrong string. 3. **Root `src/app/layout.tsx` titles are inconsistent** — 16 integrations use a bare `"CopilotKit Showcase"` while `mastra`, `langgraph-python`, `langgraph-typescript` and `built-in-agent` append their own name. No integration names a *different* integration there, so it is untidy rather than wrong. ## Checks - `prettier --check` on all 14 changed files: clean - `tsc --noEmit` in `mastra` (107 pre-existing errors, unrelated files) — count identical with the HEAD version of the file vs this change; no errors in `demos/layout.tsx`. Same for `strands` and `langgraph-typescript`. - No test references these titles (`shell-dashboard`'s `"LangChain - Python"` strings are the Notion partner label and are untouched), so there are no affected tests. |
||
|
Jordan Ritter
|
c2e9264dde | Merge branch 'main' into fix/ms-agent-python-multimodal-prompt | ||
|
Jordan Ritter
|
4a303f8bed | Merge branch 'main' into chore/multimodal-unsupported-llamaindex-crewai | ||
|
Jordan Ritter
|
2825b10ba1 | Merge branch 'main' into fix/mastra-shared-tools-symlink | ||
|
Jordan Ritter
|
b907a2660a | Merge branch 'main' into fix/showcase-integration-page-titles | ||
|
Jordan Ritter
|
d28384a2eb |
fix(showcase/ci): make a partially-cancelled build loud, and verify what it actually shipped (#6171)
## The hole A partially-cancelled showcase build was **indistinguishable from a clean one in both directions**: it emitted no alert, and it suppressed post-deploy verification — while having actually redeployed real images to staging. Three merges landed on `main` on 2026-07-25 with **zero post-deploy verification**. Reconstructed from the API, with run IDs: | # | build run | conclusion | what happened | |---|---|---|---| | #5483 | `30162754730` | cancelled | `shell-docs` killed at 10m04s. Alert **did** fire (all-failed path). No verify. | | — | `30162770765` | cancelled | `shell` killed at 10m04s. **Silent.** No verify. | | #6160 | `30162773601` | cancelled | 5 slots killed, **23 redeployed to staging**. **Silent.** No verify. | | #6168 | `30162784491` | success | clean — and its verify was then cancelled by a sibling. | ### The chain, for run `30162773601` 1. #6160's workflow edit forced a full-fleet rebuild. Five slots died — `shell`, `shell-docs`, `shell-dashboard`, `shell-dojo`, `showcase-aimock`. 2. The intersection guard correctly excluded them. `redeploy-staging` **succeeded** and uploaded `redeploy-summary`, redeploying the 23 that built. 3. The run concluded **`cancelled`** (5 legs cancelled, 0 failed → GitHub rolls the run up to `cancelled`). 4. `notify` was **skipped** → no Slack, no PR comment. 5. `showcase_deploy.yml` requires `conclusion == 'success'` → never started. 23 services redeployed, unverified. 6. The one legitimate verify — run `30163309977`, from #6168's successful build — started 15:17:13 and was **cancelled at 15:17:22** by run `30163312882`, which was triggered by a *different* partially-cancelled build and then skipped every job anyway. ## Which links I verified in the YAML, and where the brief was wrong | Link | Verdict | |---|---| | Intersection guard excludes cancelled slots | ✅ Confirmed — `showcase_build.yml` `redeploy-staging` intersects matrix ∩ `status == "success"`. | | Run concluded `cancelled`, not failure/success | ✅ Confirmed via API on all three runs. | | `showcase_deploy.yml` requires `conclusion == 'success'` | ✅ Confirmed verbatim on `resolve-matrix`. | | Global `showcase-verify-deploy` concurrency group preempted a sibling | ✅ Confirmed — group had no commit key, `cancel-in-progress: true`; timestamps above show the 9-second preemption. | | **`notify`'s condition is `if: failure()`** | ❌ **Wrong.** It is already `!cancelled() && (failure() \|\| any_success == 'false')`. | | **The suppressor is `failure()` not matching `cancelled`** | ❌ **Wrong, and the proposed fix would not have worked.** See below. | | **Depot `Step canceled by GitHub` is a flake** | ❌ **Wrong.** It is a `timeout-minutes` kill. See "Root cause". | ### Why `if: failure() || cancelled()` would NOT have fixed this I built a throwaway probe workflow on a scratch branch — a matrix with one leg killed by `timeout-minutes` and one green leg, plus dependent jobs guarded by each candidate expression. **Probe run [`30166429073`](https://github.com/CopilotKit/CopilotKit/actions/runs/30166429073)**: | probe | result | conclusion | |---|---|---| | killed leg's `job.status` | `cancelled` | a timeout kill reports as *cancelled* | | matrix rollup `needs.legs.result` | `cancelled` | | | `if: cancelled()` | **SKIPPED** | `cancelled()` evaluated **FALSE** | | `if: failure()` | **SKIPPED** | `failure()` evaluated **FALSE** | | pre-fix `notify` condition | **SKIPPED** | ← the bug, reproduced | | post-fix `notify` condition | **RAN** | ← the fix, proven | | workflow run conclusion | `cancelled` | matches production | `cancelled()` is **workflow-scoped**: ["Returns `true` if the workflow was cancelled."](https://docs.github.com/en/actions/reference/evaluate-expressions-in-workflows-and-actions) Individual legs dying does not cancel the *run*, so it is false. And `failure()` ["returns `true` if any ancestor job fails"](https://docs.github.com/en/actions/reference/evaluate-expressions-in-workflows-and-actions) — a **cancelled** ancestor did not *fail*, so it is false too. With 23 slots green, `any_success` is `'true'`. Every clause the old guard had went the wrong way. Independently corroborated in production: in run `30162773601`, both `!cancelled()`-guarded jobs (`aggregate-build-results`, `redeploy-staging`) **ran**, which is only possible if `cancelled()` was false. ## Root cause of the cancellations: a timeout, not a flake From the Depot log of the killed `shell` leg: ``` 15:04:03 depot build --file showcase/shell/Dockerfile ... --push 15:13:29 #25 [builder 15/15] RUN cd scripts && ... generate-registry.ts DONE 54.3s 15:13:32 failed to solve: Canceled: context canceled 15:13:32 ##[error]Step canceled by GitHub 15:13:32 BUILD_STATUS: cancelled ``` Started 15:03:28, killed 15:13:34 — **exactly `timeout-minutes: 10`**. Same for the other three shell slots; `showcase-aimock` died at exactly its 5. The build was making real but slow progress the whole time (repeated docker.io base-image pull stalls, `#9 ...` for seven minutes) and was 2 steps from done. So **a retry is the wrong lever twice over**: a cancelled job cannot run further steps, and the build was not erroring. The underlying cause is Depot builder contention — three full-fleet rebuilds ran concurrently (~84 simultaneous amd64 builds). Measured budgets: | slot | warm | contended | budget | |---|---|---|---| | shell / shell-docs / shell-dashboard / shell-dojo | 2–5 min | killed at 10 | **10 → 20** | | showcase-aimock | 2 min | killed at 5 | **5 → 12** | | showcase-pocketbase | — | **8.8 min** (near-miss) | **10 → 15** | | webhooks | 0.6 min | — | 5 (unchanged, `skip_build`) | The `context: "."` slots build the whole monorepo root and are the heaviest in the fleet, yet carried the *smallest* budgets — inverted. ## The design **1. Stop laundering the signal** (`build-outputs.ts`, per-slot writer). `cancelled` becomes a first-class `BuildOutcome` instead of being mapped to `skipped`. The per-slot result is the *only* place the cancellation survives, because the status functions can't see it. `successSet` still excludes it, so the redeploy intersection is unchanged — a slot that pushed no image still cannot enter the redeploy CSV. **2. Alert on it** — aggregator publishes `any_cancelled` + `cancelled_services`; `notify` gains an `any_cancelled == 'true'` clause (the exact expression proven by the probe), and its PR comment now distinguishes *incomplete* from *failed*. **3. Red the run, don't leave it `cancelled`** — new `notify-cancelled-builds` job exits non-zero, so the conclusion is `failure`. A slot killed by its timeout budget *is* a failure, and `cancelled` is precisely what suppressed everything downstream. **4. Verify the partial deploy anyway** — the `conclusion == 'success'` gate becomes a terminal-conclusion allowlist (`success|failure|cancelled|timed_out`). > **Tradeoff, stated plainly.** Relaxing the gate rather than only failing loudly is deliberate: a build that redeployed 23 services *needs* verifying, and gating on `success` guaranteed it never would. It is safe because the decision of *what* to verify never depended on the rollup — no `redeploy-summary` artifact still means `has_services=false` and verify is skipped, and the redeploy gate still narrows to the per-service success set, so a cancelled slot can never be probed against a stale `:latest` and reported healthy. The cost is one extra no-op deploy run for builds that die before redeploying. Note this *also* fixes the same hole for `conclusion == 'failure'`, which was equally unverified. **5. Key verification per commit** — `showcase-verify-deploy-${{ github.event.workflow_run.head_sha || github.sha }}`, so a later cancelled run can no longer destroy an earlier successful run's verification. `cancel-in-progress: true` is kept, so re-verifying the *same* commit still supersedes. (`github` context is [documented as available](https://docs.github.com/en/actions/reference/workflows-and-actions/contexts) in `concurrency`.) **6. Realistic timeout budgets** — separate commit, droppable independently. Mitigation, not the root fix; the durable lever is a non-cancelling concurrency queue on the build workflow, noted inline as a follow-up since this workflow deliberately has *no* concurrency group. ### On alerting for a genuine manual cancel It stays **silent**, and that is correct. `!cancelled()` is retained on both notify jobs as the flake-vs-intentional discriminator, and the probe proves it discriminates: a human cancelling the whole RUN makes `cancelled()` true → silent; a leg-level timeout leaves it false → alerts. Belt and braces — on a run-level cancel, `aggregate-build-results` is itself skipped by its own `!cancelled()` guard, so `any_cancelled` resolves to `''` rather than `'true'`. Two independent reasons for silence, zero manual-cancel noise. ## What I proved - **Real end-to-end red-green on GitHub Actions** — probe run [`30166429073`](https://github.com/CopilotKit/CopilotKit/actions/runs/30166429073), table above. Old condition skipped, new condition ran, on a genuinely timeout-killed matrix leg. Scratch branch deleted; the run is permanent. - **Local red-green on the TS layer** — 8 tests failing before the fix (`cancelledSet is not a function`, `invalid "status" (must be success|failure|skipped)`), 46 passing after. - **Local red-green on the live workflow expression.** `redeploy-guard.test.ts` parses the **actual `if:` strings out of the YAML** and evaluates them. Pointed at `origin/main`'s YAML, the new `GREEN: the live guard FIRES on the production partial-cancel` test fails (`expected false to be true`) and is the *only* failure; pointed at this branch, all 31 pass. The pre-fix guard is pinned as a literal so the test permanently encodes the difference rather than just the current behaviour. - **No new alert noise** — explicit tests that `notify` stays silent on a clean build, on a human run-cancel, and on a no-changes push. That last one exposed a latent inaccuracy in the test harness's own model (a skipped aggregator's outputs are `''`, not `'false'`), now fixed. - **Lint** — `actionlint` finding count unchanged at 10 vs the `origin/main` baseline (all pre-existing: the unknown `depot-ubuntu-24.04-4` label and shellcheck `info` notes on untouched scripts). `zizmor` with the repo's own config at `min-severity: low`: *No findings to report*. - **Suite** — 77 tests green across the three touched files. Full `showcase/scripts` suite has 5 pre-existing failing files (`emit-railway-envs-json`, `generate-catalog`, `generate-registry`, `integration-smoke-registry`); I reproduced the identical 5 files / 8 tests on a pristine `origin/main` worktree, so they are unrelated to this change. ## What remains unproven until it runs on `main` - The **production** partial-cancel path end-to-end. The probe reproduced the expression semantics on a synthetic matrix, not the real 28-slot fleet with real Depot builds; the Slack message and PR-comment rendering have not been exercised against a live webhook. - The relaxed deploy gate firing on a real `cancelled`/`failure` build — including the "no artifact → skipped, no noise" path. - The per-commit concurrency group under real rapid-fire merges. - Whether 20 minutes is actually enough for `shell` under peak Depot contention. It was ~3 steps from done at 10; 20 is inference from that, not measurement. `\n` handling in the new Slack payload follows the existing `fromJSON('"\n"')` convention in this file, but is not verified against a live webhook. |
||
|
Vishal Sharma
|
0c38124799 |
fix(react-core): stop CopilotPopup remounting chat on resize
CopilotPopup built its `chatView` override inside a `useMemo` keyed on `width`/`height`. Consumers driving those props from a drag-to-resize handle (committing new dimensions on mouseup) minted a new component function per resize; rendering a new element type at that slot makes React unmount and remount the whole chat subtree. The remount resets scrollTop to 0, then `initial="smooth"` re-animates the message list top-to-bottom on every resize, from any scroll position. Give the override a stable module-scope identity and pass the popup shell props (header, toggle, width, height, clickOutsideToClose, defaultOpen) through React context. Resizing is now a plain style update on CopilotPopupView with no remount, so scroll position holds. Add a regression test asserting the chat subtree stays mounted (mount count stays at 1) across width/height changes: it fails on the prior code (one extra mount per resize) and passes with the fix. |
||
|
Matt Van Horn
|
9f847a2e47 |
fix(sdk-python): keep the frontend tool call orphaned in after_agent
Addresses both review concerns on the after_agent half. after_agent no longer synthesizes a forwarded_to_frontend ToolMessage. That synthetic result did not match _INTERRUPTED_PAT, so _fix_messages_for_bedrock and langchain's patch_orphan_tool_calls treated it as a real second result and skipped the reposition path, which on the Bedrock path could leave the real frontend result non-adjacent and then stripped as unanswered. The restore helper now takes synthesize_frontend_tool_results, which after_agent sets to False so the call stays an orphan for the real result to fill, and any synthetic result already in history is dropped by id. The sync wrap_model_call now also runs _fix_messages_for_bedrock, matching awrap_model_call, so the messages it injects go through the same dedup instead of risking a duplicate tool_call id on the sync path. |
||
|
Jordan Ritter
|
b065664bc9 |
fix(showcase/ci): give the monorepo-root build slots a realistic timeout budget
Root cause of the cancellations, from the Depot logs of the killed slots: they were not a transient flake. Every one died at EXACTLY its `timeout-minutes` value (shell/shell-docs/shell-dashboard/shell-dojo at 10m04s of a 10-minute budget; showcase-aimock at 5 minutes), and the log ends with `failed to solve: Canceled: context canceled` followed by Depot's `Step canceled by GitHub`. GitHub reports a `timeout-minutes` kill as job conclusion `cancelled`, which is what fed the whole silent chain. The builds were making real but slow progress the entire time — repeated docker.io base-image pull stalls, and the final `generate-registry` layer alone took 54s — and shell was still 2 steps from done when the budget killed it. The `context: "."` slots build the whole monorepo root and are the heaviest in the fleet, yet they carried the SMALLEST budgets (10 min vs 15 for each small per-integration build). Measured, same slots: warm / uncontended ......... 2-5 min contended, killed at ....... 10 min (runs 30162773601, 30162770765) showcase-pocketbase ........ 8.8 min of a 10 min budget (near-miss) So a retry is the wrong lever twice over: a cancelled job cannot run further steps, and the build was not erroring. Give the four shell slots 20, showcase-aimock 12, and showcase-pocketbase 15. `webhooks` stays at 5 (it is skip_build, measured at 0.6 min). This is mitigation, not the root fix — the underlying cause is Depot builder contention from three concurrent full-fleet rebuilds (~84 simultaneous amd64 builds). Noted inline as a follow-up, since the durable lever is a non-cancelling concurrency queue and this workflow deliberately has no concurrency group. |
||
|
Jordan Ritter
|
2e85bfab08 |
fix(showcase/ci): verify partial deploys and stop cross-commit verify preemption
Two independent ways verification was silently lost on 2026-07-25.
1. `resolve-matrix` required `github.event.workflow_run.conclusion ==
'success'`. But `redeploy-staging` gates on the artifact-derived
`any_success`, not on the matrix rollup, so a build with some slots
cancelled (rollup `cancelled`) or failed (rollup `failure`) still
pushes real images and still redeploys the slots that built. Run
30162773601 redeployed 23 services to staging and this workflow never
started. An unverified real deploy is worse than a verified partial
one, so the trigger is now "the build reached a terminal conclusion"
and WHAT to verify stays decided by evidence: no redeploy-summary
artifact still means has_services=false and verify is skipped, and the
redeploy gate still narrows to the per-service success set, so a
cancelled slot can never be probed against a stale `:latest`.
2. The `showcase-verify-deploy` concurrency group was global, so any
later-finishing build preempted an earlier run's verification even
though they verify DIFFERENT commits. Verify run 30163309977 for the
one genuinely successful build of the day (#6168,
|
||
|
Jordan Ritter
|
12de577952 |
fix(showcase/ci): alert and red the run when build slots are cancelled
Build run 30162773601 (merge of #6160) forced a full-fleet rebuild; 5 of 28 slots were killed by their `timeout-minutes` budget, the other 23 built and WERE redeployed to staging, and the run emitted no signal at all: `notify` was skipped, so no Slack alert and no PR comment, and the run rolled up to conclusion `cancelled`. No existing guard could catch it. Measured on purpose-built probe run 30166429073 (matrix leg killed by `timeout-minutes`, sibling leg green): killed leg `job.status` ........ cancelled matrix rollup `needs.*.result` . cancelled `if: cancelled()` .............. SKIPPED (evaluated FALSE) `if: failure()` ................ SKIPPED (evaluated FALSE) pre-fix `notify` condition ..... SKIPPED <- the bug post-fix `notify` condition .... RAN <- the fix run conclusion ................. cancelled So `failure() || cancelled()` would NOT have fixed this. The signal has to come from the per-slot build results. - stop laundering `cancelled` into `skipped` in the per-slot writer - expose `any_cancelled` / `cancelled_services` from the aggregator job - add `notify-cancelled-builds`: exits non-zero so the run concludes `failure` rather than `cancelled` (a slot killed by its timeout budget is a failure, and `cancelled` is what suppressed everything), and Slacks the affected service names - add the `any_cancelled` clause to `notify` so the merge author gets the PR comment, with wording that distinguishes incomplete from failed `!cancelled()` is retained on both jobs as the intentional-vs-flake discriminator: a human cancelling the whole RUN makes `cancelled()` true and stays silent, while a leg-level timeout leaves it false and alerts. Extends redeploy-guard.test.ts, which evaluates the LIVE `if:` strings from the workflow, with the exact production scenario. It pins the pre-fix guard string as a literal so the test proves the difference the fix makes, not merely the current behaviour. |
||
|
Jordan Ritter
|
62474be1dd |
fix(showcase/ci): make a cancelled build slot a first-class outcome
`job.status` for a matrix slot is success|failure|cancelled, but the per-slot writer laundered cancelled into `skipped` before publishing its result, so a slot killed by `timeout-minutes` became indistinguishable from one that legitimately never built. That erased the only signal that could tell a partially-cancelled fleet build from a clean one. GitHub's status functions cannot recover it: `cancelled()` is documented as "returns true if the workflow was canceled" (workflow-scoped, and FALSE for a leg-only cancel), and a cancelled ancestor is not a FAILED ancestor so `failure()` is false too. Add `cancelled` to the BuildOutcome contract, add `cancelledSet()`, and have the aggregator publish `any_cancelled` + `cancelled_services` alongside `any_success`. `successSet` still excludes cancelled slots, so the redeploy intersection is unchanged — a slot that pushed no image still cannot enter the redeploy CSV. |
||
|
Jordan Ritter
|
1c24e0644b |
chore(showcase): store demo assets uniformly as Git LFS pointers (#6163)
## What
Demo assets under
`showcase/integrations/*/public/{demo-files,demo-audio}/` were stored
two different ways. This makes all of them LFS pointers — the root
`.gitattributes` convention that the majority already follow.
**Before this PR, on `main`:**
| form | integrations |
|---|---|
| **LFS pointers** (10) | `ag2`, `built-in-agent`, `claude-sdk-python`,
`claude-sdk-typescript`, `crewai-crews`, `langroid`, `llamaindex`,
`ms-agent-harness-dotnet`, `pydantic-ai`, `spring-ai` |
| **Raw blobs + per-integration `.gitattributes` override** (8) |
`google-adk`, `langgraph-fastapi`, `langgraph-python`,
`langgraph-typescript`, `ms-agent-dotnet`, `ms-agent-python`, `strands`,
`strands-typescript` |
| **Missing entirely** (2) | `agno`, `mastra` — being added as pointers
on #6157 |
Committed sizes: pointer-mode `sample.png` 130 B / `sample.pdf` 129 B /
`sample.wav` 130 B; raw-mode `sample.png` 10083 B / `sample.pdf` 2486 B
/ `sample.wav` 87078 B.
## Why the overrides existed, and why they no longer need to
Each override re-declared the same paths `-filter -diff -merge` so the
files would be committed as real binaries. The stated reason (in their
own comments) was that the image build didn't run `git lfs pull`, so a
pointer stub shipped into the image and the multimodal sample-attachment
magic-bytes guard rejected it.
That premise no longer holds. The deploy build's Checkout step in
`showcase_build.yml` hardcodes `lfs: true` (added in `7bde1eef3aa`), so
every integration image gets real binaries regardless of storage form.
The 10 pointer-mode integrations already ship through that exact
pipeline today. The overrides are now dead weight that only buys
divergence.
## What changed
- Deleted all 8 override `.gitattributes`. Each contained **nothing
but** demo-asset exemptions (the `langgraph-*`/`strands*` ones also
exempted `public/demo-audio/sample.wav`), so each is removed in full.
- Renormalized the 21 affected assets through the LFS clean filter.
- Root `.gitattributes` untouched. `.github/` untouched. `agno`/`mastra`
untouched.
Deleting an override necessarily un-exempts that integration's
`demo-audio/sample.wav` too, so the 5 raw `.wav` files are converted in
the same pass — leaving them raw under an inherited `filter=lfs` would
produce a permanently dirty checkout.
## Byte-identity
Storage form changes, content does not. Every asset's sha256 **already
equals** the LFS OID the pointer-mode integrations reference, so each
renormalized blob is bit-for-bit the pointer blob already committed on
`main`:
```
sample.png 10083 B oid 01aa5681de99461247543e9215c1e4da3242e26b2bee11593fcdbe209672d973
sample.pdf 2486 B oid 3da2afae36a1a81fd2c02f15e54bfc38b6c22e41655c31a5b54ff1e0e3daab41
sample.wav 87078 B oid bd4aa7b049f1c3e324dfd15af4068d7f8fbf2eae1dd044df270dddc5f38a5c57
```
Verified per file (21/21): worktree sha256 before == worktree sha256
after == the pointer's OID, and each staged blob hash equals the
canonical pointer blob (`a780acbd` png / `6446acbb` pdf / `f156ab54`
wav) already on `main`.
**No new LFS objects are introduced at all**, so a dangling pointer is
impossible by construction. Confirmed anyway:
- LFS batch API returns `download` actions for all three OIDs.
- All three objects downloaded and confirmed to hash to their OID.
- `git lfs push --dry-run origin <branch>` reports nothing to upload.
- `git lfs fsck --pointers HEAD` reports no new violations (only the two
pre-existing `examples/teams/appPackage/*.png` ones, also present on
`main`).
## On the CI signal, and the real proof
Being precise about what CI here does and does not prove:
- **`showcase_build.yml`** (deploy, `lfs: true`) is the job that
determines whether the deployed image gets real assets. It runs **only
on push to `main`**, not on PRs. It is unchanged by this PR.
- **`showcase_build_check.yml`** (the PR job) checks out with `lfs: ${{
matrix.service.lfs }}`, which is `false` for all 8 converted
integrations. Its images therefore contain pointer stubs. It is
build-only (`push: false`, no smoke test, no asset assertion), so it
will pass — but **a green `build_check` here does not prove real assets
ship.** I am not resting the case on it.
I did **not** rely on a local Docker build either — it smudges LFS from
the local checkout and would prove nothing.
### The real proof: prod already serves pointer-mode assets correctly
Instead of a weak build signal, here is what the **live deployed prod
images** serve today. Pointer-mode integrations are unchanged by this PR
and demonstrate that the pointer storage form already ships real bytes
through the `lfs: true` deploy checkout — for all three asset types:
```
--- .png : POINTER-mode on main (unchanged) ---
ag2 200 10083 01aa5681de REAL-exact-match
spring-ai 200 10083 01aa5681de REAL-exact-match
llamaindex 200 10083 01aa5681de REAL-exact-match
pydantic-ai 200 10083 01aa5681de REAL-exact-match
langroid 200 10083 01aa5681de REAL-exact-match
--- .png : RAW-mode on main (converted by this PR) ---
langgraph-python 200 10083 01aa5681de REAL-exact-match
strands 200 10083 01aa5681de REAL-exact-match
google-adk 200 10083 01aa5681de REAL-exact-match
ms-agent-python 200 10083 01aa5681de REAL-exact-match
langgraph-typescript 200 10083 01aa5681de REAL-exact-match
--- .wav : ALREADY pointer-mode on main (precedent for the 5 wav conversions) ---
google-adk 200 87078 bd4aa7b049 REAL-exact-match
ms-agent-python 200 87078 bd4aa7b049 REAL-exact-match
ms-agent-dotnet 200 87078 bd4aa7b049 REAL-exact-match
ag2 200 87078 bd4aa7b049 REAL-exact-match
spring-ai 200 87078 bd4aa7b049 REAL-exact-match
--- .wav : RAW on main (converted by this PR) ---
langgraph-python 200 87078 bd4aa7b049 REAL-exact-match
strands 200 87078 bd4aa7b049 REAL-exact-match
langgraph-typescript 200 87078 bd4aa7b049 REAL-exact-match
langgraph-fastapi 200 87078 bd4aa7b049 REAL-exact-match
strands-typescript 200 87078 bd4aa7b049 REAL-exact-match
--- .pdf : pointer-mode vs raw-mode ---
ag2 (ptr) 200 2486 3da2afae36 REAL-exact-match
spring-ai (ptr) 200 2486 3da2afae36 REAL-exact-match
langgraph-python(raw) 200 2486 3da2afae36 REAL-exact-match
strands (raw) 200 2486 3da2afae36 REAL-exact-match
```
Both storage forms already serve **byte-identical** content in prod
(sha256 equals the LFS OID in every case), including `.wav` where 3
integrations are already pointer-mode. Converting raw to pointer
therefore cannot change what prod serves: both forms resolve to the same
bytes through the same deploy checkout. That is the safety argument —
verified, not asserted.
## Sequencing
- **#6160** (`ci(showcase): fetch Git LFS in every workflow that serves
integration assets`) should merge **first**. It makes
`showcase_build_check.yml` fetch LFS and removes the dead per-slot `lfs`
flag; after it lands, `build_check` on this kind of change becomes a
real signal instead of a weak one. No file overlap with this PR (it
touches only `.github/workflows/`), so no merge conflict either way.
- **#6157** (agno/mastra assets) is independent — disjoint paths, any
order, no conflict.
This PR assumes the deploy build's hardcoded `lfs: true` already on
`main` and complements #6160's non-deploy workflow coverage.
## Cell impact
**Changed-cell set: empty (expected and predicted).** This changes
storage form only. The prod probes above show both storage forms already
serve byte-identical content, so the multimodal cells for the 8
converted integrations have no mechanism by which to change state. The
cells cannot be re-measured against this branch pre-merge (the deploy
build runs only on push to `main`), so this is a prediction grounded in
the byte-identity and prod-parity evidence rather than a post-deploy
dashboard reading — worth a dashboard confirmation after merge.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01PYdjeveT8Xof9TyHWMLoJr
|
||
|
Jordan Ritter
|
5a1812e49c |
fix(showcase): bundle multimodal sample assets for agno and mastra (#6157)
> ### ⚠️ DEPENDENCY — this PR does **not** fix `agno`/`mastra` on its own > > The assets here are committed as **plain Git LFS pointers**, governed by the repo-root `*.png` / `*.pdf` LFS rules, with **no per-integration `.gitattributes` override**. LFS tracking stays uniform across the fleet — demo assets are not carved out of LFS per integration. > > Integration image builds currently do **not** fetch LFS: every entry in `ALL_SERVICES` in `.github/workflows/showcase_build.yml` is `"lfs": false`. Until that is fixed, the built `agno`/`mastra` images will ship 130-byte pointer stubs, the component's magic-byte guard will reject them, and the D5 multimodal cells stay **red**. > > **This PR requires the build-side LFS-fetch fix to land first, or to be merged together with it.** Merging this alone changes nothing for `agno`/`mastra`. > > The earlier RED → GREEN proof in this PR was obtained with *real binaries* in the image (the `.gitattributes` override, since removed) and **no longer describes what this branch produces**. It is retained below, clearly marked **SUPERSEDED**. No new green proof is claimed, and none can be produced until the build-side fix exists. Re-verification happens once both changes are combined. --- ## What `agno` and `mastra` were the only **2 of 20** showcase integrations with no `public/demo-files/` directory ever committed. This adds the two bundled sample assets (plus the provenance README) to both, LFS-tracked under the repo-root rules. ## The failure The Attachments demo fetches its bundled sample and validates the magic bytes *before* building the message (`sample-attachment-buttons.tsx` → `fetchUrl: "/demo-files/sample.png"`). With the asset absent, it never gets that far. Verbatim console output from the D5 probe, identical on both slugs: ``` [error] Failed to load resource: the server responded with a status of 404 (Not Found) [error] [multimodal-demo] sample-attachment send failed Error: Could not fetch sample "sample.png" — HTTP 404. Is the file bundled under public/demo-files/sample.png? ``` Page body at failure: ``` Bundled samples: Try with sample image Try with sample PDF Could not fetch sample "sample.png" — HTTP 404. Is the file bundled under public/demo-files/sample.png? How can I help you today? ``` `userMsgCount: 0` — nothing was ever sent, so `waitForTurnComplete` timed out on turn 1 and the cell surfaced the generic `conversation-error` bucket. **The backends are fine**: both integrations have real vision-capable multimodal agents (`agno/src/agents/multimodal_agent.py`, `mastra/src/mastra/agents/index.ts`) and correct aimock fixtures whose `userMessage` keys match the component's `autoPrompt` strings verbatim. This is purely a missing-asset / packaging bug — a different bug from the LFS-pointer problem, and the reason this PR is still needed. ## Why only these 2 of 20 `public/demo-files/` is **not** part of the `shared/` symlink mechanism — unlike `_shared/`, `tools/`, and `shared-tools/`, it is a per-integration duplicated directory. That is exactly how two integrations came to be missing it. Confirmed against `origin/main`: the directory has never existed in either integration's history, while all 18 others resolve to a real commit. ## What was added, and from where `sample.png` (10083 B), `sample.pdf` (2486 B), `README.md` (1174 B) copied **byte-for-byte** from `showcase/integrations/langgraph-python/public/demo-files/`, the canonical source. Real files, not symlinks — matching the other 18 integrations. sha256 verified identical. ## Committed blob form Both binaries are committed as LFS pointers. No `.gitattributes` is added by this PR: ``` $ git show --stat HEAD showcase/integrations/agno/public/demo-files/README.md | 22 ++++++++++++++++++++++ showcase/integrations/agno/public/demo-files/sample.pdf | 3 +++ showcase/integrations/agno/public/demo-files/sample.png | 3 +++ showcase/integrations/mastra/public/demo-files/README.md | 22 ++++++++++++++++++++++ showcase/integrations/mastra/public/demo-files/sample.pdf | 3 +++ showcase/integrations/mastra/public/demo-files/sample.png | 3 +++ 6 files changed, 56 insertions(+) $ git cat-file -p HEAD:showcase/integrations/agno/public/demo-files/sample.png version https://git-lfs.github.com/spec/v1 oid sha256:01aa5681de99461247543e9215c1e4da3242e26b2bee11593fcdbe209672d973 size 10083 $ git cat-file -p HEAD:showcase/integrations/agno/public/demo-files/sample.pdf version https://git-lfs.github.com/spec/v1 oid sha256:3da2afae36a1a81fd2c02f15e54bfc38b6c22e41655c31a5b54ff1e0e3daab41 size 2486 ``` `mastra` is byte-identical to `agno` for both files. Git blob sizes are 130 / 129 bytes (pointer text), matching the pointer-mode integrations already on `main` (`llamaindex`, `ag2`, `pydantic-ai`, …), which reference these **same two OIDs**. ## LFS objects are real, not dangling A pointer with no backing object is worse than either alternative, so this was verified against the server rather than the local cache. The OIDs are the ones already stored for the identical assets in ten other integrations, so no new upload was required (`git lfs push --dry-run origin HEAD` reports nothing to transfer). Server-side existence, and then the actual bytes, were confirmed through the LFS batch API: ``` POST https://github.com/copilotkit/copilotkit.git/info/lfs/objects/batch (operation=download) 01aa5681de994612 size=10083 download action present 3da2afae36a1a81f size=2486 download action present $ curl "<download href for 01aa5681…>" -o /tmp/lfs-dl.png $ shasum -a 256 /tmp/lfs-dl.png 01aa5681de99461247543e9215c1e4da3242e26b2bee11593fcdbe209672d973 $ file /tmp/lfs-dl.png PNG image data, 393 x 90, 8-bit/color RGBA, non-interlaced ``` The object downloaded from GitHub's LFS store is a real PNG whose hash matches the pointer. ## Scope Static assets only. No source, no fixtures, no manifests, no runtime code, no lockfile churn, no workflow changes. --- # SUPERSEDED — verification below no longer applies Everything from here down was produced when this branch carried per-integration `.gitattributes` overrides that committed the assets as **real binaries**. Those overrides have been removed. The image-level and RED → GREEN evidence therefore describes a state this branch no longer produces. It is kept for the record of what was tested and why the missing-asset diagnosis is sound — **not** as a claim that this branch is green. Specifically, still valid from the run below: the RED evidence (both slugs failed with a 404 on the missing asset, and the langgraph-python CONTROL was green), and the finding that the attachment pipeline past the fetch works once the asset is genuinely present in the image. **No longer valid**: the GREEN result, because it depended on real binaries reaching the runtime layer, which now requires the build-side LFS fetch. <details> <summary>Superseded image-level and RED → GREEN evidence (click to expand)</summary> ## Image-level verification (SUPERSEDED) A file in git that Docker never copies fixes nothing, so this was checked in the built images, not just the source tree. Both Dockerfiles do `COPY --from=frontend /app/public ./public`, and neither `.dockerignore` excludes the path. **Before** (`showcase build agno mastra` from this branch's base): ``` === showcase-agno:local === ls: cannot access '/app/public/demo-files': No such file or directory === showcase-mastra:local === ls: cannot access '/app/public/demo-files': No such file or directory ``` **After** (same command, post-change — with the since-removed `.gitattributes` overrides): ``` === showcase-agno:local (post-change) === -rw-r--r-- 1 app app 1174 Jul 24 22:43 README.md -rw-r--r-- 1 app app 2486 Jul 24 22:43 sample.pdf -rw-r--r-- 1 app app 10083 Jul 24 22:43 sample.png png magic: 0000000 211 P N G pdf magic: 0000000 % P D F === showcase-mastra:local (post-change) === -rw-r--r-- 1 app app 1174 Jul 24 22:43 README.md -rw-r--r-- 1 app app 2486 Jul 24 22:43 sample.pdf -rw-r--r-- 1 app app 10083 Jul 24 22:43 sample.png png magic: 0000000 211 P N G pdf magic: 0000000 % P D F ``` Served over HTTP from the recreated containers: ``` agno /demos/multimodal 200 agno /demo-files/sample.png 200 agno /demo-files/sample.pdf 200 mastra /demos/multimodal 200 mastra /demo-files/sample.png 200 mastra /demo-files/sample.pdf 200 ``` (pre-change, same endpoints: page `200`, both assets `404`) ## RED → GREEN → CONTROL (GREEN half SUPERSEDED) All three exercised the real failure surface — the actual Playwright D5 probe driving the live demo page against the built image, not tests against fakes. ### RED — before the change (still valid) `bin/showcase test mastra:multimodal --d5 --direct --verbose` ``` [conversation-runner] turn 1/2 — FAILED { errorCategory: 'assertion-failed', ... } probe.e2e-full.runFeature.conversation-failed {"featureType":"multimodal","slug":"mastra","failureTurn":1,"turnsCompleted":0,"totalTurns":2,"error":"waitForTurnComplete: turn 1 did not complete within 60000ms (reason=dom-missing, runsFinished=0, count=0, attrPresent=true, runningNow=false, runStartCount=0)"} ... "bodyTextSnippet":"Bundled samples:\nTry with sample image\nTry with sample PDF\nCould not fetch sample \"sample.png\" — HTTP 404. Is the file bundled under public/demo-files/sample.png?\nHow can I help you today?..." "consoleLogs":["[error] Failed to load resource: the server responded with a status of 404 (Not Found)","[error] [multimodal-demo] sample-attachment send failed Error: Could not fetch sample \"sample.png\" — HTTP 404. Is the file bundled under public/demo-files/sample.png?"] probe.e2e-full.service-complete {"slug":"mastra","passed":0,"failed":1,"skipped":0,"incapable":0,"total":1,"state":"red","durationMs":121319} ✗ d5-single-pill-e2e:mastra red (121.5s) 0 passed, 1 failed (121.5s) ⚠ Tests failed for mastra:multimodal (exit 1) ``` `bin/showcase test agno:multimodal --d5 --direct --verbose` ``` probe.e2e-full.runFeature.flap-diagnostics {"slug":"agno","featureType":"multimodal","error":"waitForTurnComplete: turn 1 did not complete within 60000ms (reason=dom-missing, runsFinished=0, count=0, attrPresent=true, runningNow=false, runStartCount=0)"} "bodyTextSnippet":"Bundled samples:\nTry with sample image\nTry with sample PDF\nCould not fetch sample \"sample.png\" — HTTP 404. Is the file bundled under public/demo-files/sample.png?\nHow can I help you today?..." "consoleLogs":["[error] Failed to load resource: the server responded with a status of 404 (Not Found)","[error] [multimodal-demo] sample-attachment send failed Error: Could not fetch sample \"sample.png\" — HTTP 404. Is the file bundled under public/demo-files/sample.png?"] probe.e2e-full.service-complete {"slug":"agno","passed":0,"failed":1,"skipped":0,"incapable":0,"total":1,"state":"red","durationMs":120976} ⚠ Tests failed for agno:multimodal (exit 1) ``` ### GREEN — SUPERSEDED, does not describe this branch Obtained with the since-removed `.gitattributes` overrides, i.e. with real binaries baked into the image. Same commands, after `showcase build agno mastra` + `showcase recreate agno mastra`. `agno`: ``` [conversation-runner] turn 1/2 — assertions passed [conversation-runner] turn 2/2 — assertions passed probe.e2e-full.service-complete {"slug":"agno","passed":1,"failed":0,"skipped":0,"incapable":0,"total":1,"state":"green","durationMs":8492} ✓ Tests passed for agno:multimodal ``` `mastra`: ``` [conversation-runner] turn 1/2 — assertions passed [conversation-runner] turn 2/2 — assertions passed probe.e2e-full.feature-complete {"slug":"mastra","featureType":"multimodal","pass":true,"durationMs":7462} probe.e2e-full.service-complete {"slug":"mastra","passed":1,"failed":0,"skipped":0,"incapable":0,"total":1,"state":"green","durationMs":7828} ✓ Tests passed for mastra:multimodal ``` Both turns passed on both slugs — the image turn *and* the PDF turn. What this settled and still stands: the attachment pipeline past the fetch had never once executed on these two services, and it works when the real asset is present. What it does **not** establish: that this branch as it now stands is green. ### CONTROL — an already-green integration, unchanged (still valid) `bin/showcase test langgraph-python:multimodal --d5 --direct --verbose` (no code touched): ``` [conversation-runner] turn 1/2 — assertions passed [conversation-runner] turn 2/2 — assistant settled { bubbleIndex: 1, textLength: 233, hasAssertions: true } [conversation-runner] turn 2/2 — assertions passed probe.e2e-full.feature-complete {"slug":"langgraph-python","featureType":"multimodal","pass":true,"durationMs":8783} probe.e2e-full.service-complete {"slug":"langgraph-python","passed":1,"failed":0,"skipped":0,"incapable":0,"total":1,"state":"green","durationMs":9547} ``` Confirms the local setup, the shared probe, the shared frontend and the shared fixtures were all healthy — the two REDs above were real per-integration failures, not a setup artifact. </details> --- ## Fleet state, for whoever lands the build fix Measured on `origin/main`, `demo-files/sample.png` splits three ways across the 20 integrations: - **10 pointer-mode** (130-byte LFS pointers, no override): `ag2`, `built-in-agent`, `claude-sdk-python`, `claude-sdk-typescript`, `crewai-crews`, `langroid`, `llamaindex`, `ms-agent-harness-dotnet`, `pydantic-ai`, `spring-ai` - **8 raw-blob mode** (10083-byte real binaries, carved out of LFS by a per-integration `.gitattributes`): `google-adk`, `langgraph-fastapi`, `langgraph-python`, `langgraph-typescript`, `ms-agent-dotnet`, `ms-agent-python`, `strands-typescript`, `strands` - **2 missing entirely**: `agno`, `mastra` — what this PR adds, in pointer mode So `main` is currently inconsistent in both directions. This PR puts its two integrations on the uniform-LFS side. The remaining 8 raw-blob overrides should be removed by the build-side fix so the whole fleet is pointer-mode plus an LFS-fetching build. ## Noted, not fixed (out of scope) - `showcase/integrations/mastra/shared-tools` is a **real directory** on `origin/main` (mode `040000`), not the symlink the iron rules require. Pre-existing erosion, unrelated to this change, worth a separate PR. - The `mastra` multimodal page's `<title>` renders as "LangChain - Python" — a frontend-parity oddity visible in the probe diagnostics. Unrelated. |
||
|
Jordan Ritter
|
8a726e701a | Merge branch 'main' into chore/showcase-demo-assets-uniform-lfs | ||
|
Jordan Ritter
|
fb28e93473 | Merge branch 'main' into fix/multimodal-missing-demo-files | ||
|
Jordan Ritter
|
12ea621a77 | Merge branch 'main' into fix/cold-load-gray-masks-red | ||
|
Jordan Ritter
|
7282ecddfb |
ci(showcase): run the harness and shell-dashboard unit suites in CI (#6168)
## The gap
**No CI job ran the showcase unit suites.** Not "ran them partially" —
no job invoked them at all. Verified per suite against `91f0321397` (the
tip of `main` this branched from), by reading every workflow's actual
`run:` lines rather than trusting job names:
| Suite | Tests | Ran in CI before? | Where |
|---|---|---|---|
| `showcase/harness` | 3646 (3625 pass / 3 fail / 18 skip) | **NO** |
nowhere |
| `showcase/shell-dashboard` | 1331 (1330 pass / 1 skip) | **NO** |
nowhere |
| `showcase/shell` | ~57 (8 files) | **NO** | nowhere |
| `showcase/scripts` | — | yes | `showcase_validate.yml` → "Run build
pipeline tests" |
Why each candidate workflow misses them:
- **`test_unit.yml`** — the one whose name implies it covers this —
excludes them *twice over*: `paths-ignore` lists `showcase/**`, **and**
its nx selection is hard-scoped to `--projects='packages/**'`.
- **`showcase_validate.yml`** runs `pnpm exec vitest run` only in
`showcase/scripts`. It has two `working-directory: showcase/harness`
steps, but they are a CVDIAG perf bench (`vitest bench
src/cvdiag/emit-perf.bench.ts`) and an ESM boot-smoke — neither runs the
unit suite.
- **`static_quality.yml`** does run `nx run-many -t check-types`. This
resolves the reported name/invocation divergence: the *job* is called
`check-types` and the harness's script is called `typecheck`, so no
`check-types` target exists on the harness and **the harness is not
typechecked either**.
Net effect: ~5000 showcase unit tests gated nothing. A PR could carry
real defects in `showcase/harness/src/**` and still show an all-green
check list, because no job structurally could have caught them.
## What this wires
New workflow **`test / unit-showcase`**
(`.github/workflows/test_unit-showcase.yml`), two parallel jobs on PR +
push-to-main filtered to `showcase/**`:
- **`harness unit suite`** — `nx run
@copilotkit/showcase-harness:test:ci`, then `nx run
@copilotkit/showcase-harness:test:quarantine-ratchet`. Driven through nx
per the root `CLAUDE.md` task convention.
- **`shell-dashboard unit suite`** — `vitest run` directly; the
dashboard is deliberately outside the pnpm workspace (see the note in
`pnpm-workspace.yaml`) so it is not an nx project and has no target to
run.
New nx targets are declared in **`showcase/harness/package.json`'s
`nx.targets` block**, not in `nx.json` — they are harness-specific, and
a `nx.json` `targetDefaults` entry would silently apply to any future
project that happens to share the script name. Both are `cache: false`
on purpose: the nx `test` named-input covers `src/**` and `*.test.*` but
**not** `vitest.ci.config.ts` or `vitest.quarantine.json`, so a cached
result could survive an edit to the quarantine list.
### Why a separate workflow rather than steps in `showcase_validate.yml`
That file is a single ~900-line job already budgeted at 25 minutes and
is the busiest merge-path workflow in the repo. Splitting the unit
suites out gives them their own name in the check list, their own
concurrency group, and — since the two suites need different package
managers — lets them run in parallel instead of lengthening the critical
job. Naming follows the existing convention (`test_unit.yml`,
`test_unit-python-sdk.yml`, `test_unit-spring-ai.yml`).
**No `continue-on-error` and no `|| true` anywhere in the file.** A gate
that cannot fail is not a gate.
## Hazards, and how each is handled
### 1. The hanging spike test
*Correction to the reported location:* it is
`showcase/shell-dashboard/tests/runtime-env-switch.spike.test.ts`, not
under `showcase/harness/`. It reaches the dashboard suite via that
package's `include: ["src/**/*.test.{ts,tsx}",
"tests/**/*.spike.test.ts"]`.
**Reproduced locally.** vitest sat at **0.0% CPU with zero output** and
never even spawned the `next build` its `beforeAll` calls — no worker
fork, nothing. Killed at ~2 minutes; consistent with the reported
~25-minute stall.
**Handled by excluding it from the unit gate**, not by a hard timeout.
Justification: it is an integration spike by construction — one `next
build` plus two `next start` boots on fixed ports 3801/3802 — and this
package's own config comment already says it is *"too heavy for the
per-file unit suite"* even though the `include` glob pulls it in anyway.
A hard timeout would convert a 25-minute stall into a red gate with
nothing actionable in it; this suite has to stay a fast, trustworthy
unit signal. The spike needs its own job with a real server budget,
which is out of scope for wiring up the unit gate.
Note the `--exclude` values restate the package's existing config
excludes (`tests/visual/**`, `node_modules/**`) because vitest's CLI
`--exclude` **replaces** the config value rather than extending it —
dropping them would silently re-enable them.
### 2. Gitignored generated artifacts
`showcase/.gitignore` ignores `shell/src/data/*.json` and
`shell-dashboard/src/data/*.json`, but the suites consume them:
- `harness/src/probes/frontend-matrix.test.ts` **statically imports**
`shell/src/data/frontend-catalog.json` — a missing file is a module
*load error*, not a test failure.
- `harness/src/fleet/control-plane/d0-gone-predicate.test.ts` reads
`shell/src/data/registry.json` at runtime.
- `shell-dashboard/src/lib/docs-status.ts` **statically imports**
`@/data/docs-status.json`.
Generated as an explicit job step (`generate-registry.ts`, plus
`probe-docs.ts` for the dashboard), mirroring how
`showcase_validate.yml` drives the same generators. The dashboard job
invokes vitest directly rather than `npm test` specifically so the
`pretest` hook does not re-run `probe-docs.ts` and double its ~50
outbound HEAD requests.
Also load-bearing: `npm ci --ignore-scripts` in the dashboard job. That
package's `postinstall` is `cd ../scripts && npm install`, which would
lay an npm-resolved `showcase/scripts/node_modules` over the
pnpm-managed one.
### 3. The 3 pre-existing failures — quarantined behind a ratchet, none
deleted or weakened
Confirmed byte-identical to `origin/main` and pre-existing. A job that
is red on arrival gets ignored or disabled, so the three files are
excluded via `showcase/harness/vitest.quarantine.json` +
`vitest.ci.config.ts` — **and `scripts/quarantine-ratchet.ts` re-runs
each one and requires it to STILL FAIL.** The moment somebody fixes a
quarantined test, the ratchet goes red and names the entry to delete. An
entry therefore cannot outlive the failure it excuses. The ratchet also
fails on an entry pointing at a missing file, on an entry whose filter
matches more than one file, and on a malformed or under-documented
manifest (every entry must carry `file` / `since` / `reason` /
`unquarantineWhen`).
Local `pnpm test` still uses the unfiltered config, so a developer sees
the quarantined failures.
| File | Disposition | Why |
|---|---|---|
| `src/probes/frontend-matrix.test.ts` | quarantine | Asserts hard-coded
per-frontend counts (`react=660`, `angular=636`) against the
**generated** `frontend-catalog.json`, which has grown to 1302 runnable
cells (`react=664`). Stale literals, not a product defect — the two
assertions that express the test's stated intent ("without loss or
duplication": `length === metadata.runnable`, ids unique) both pass.
Exit criterion: derive the counts from the catalog; re-hard-coding fresh
numbers just resets the clock. |
| `src/probes/helpers/d5-mapping-drift.test.ts` | quarantine |
**Stranded by a completed refactor.** It regex-scrapes
`CATALOG_TO_D5_KEY` out of `shell-dashboard/src/lib/live-status.ts` to
prove the dashboard copy hasn't drifted from the harness's
`REGISTRY_TO_D5`. That file is now a one-line `export *` barrel over the
canonical `harness/src/shared/cell-model/live-status.ts` — there is no
second copy to drift and nothing for the regex to find, so the test
throws its own "could not locate" guard. Exit criterion: cell-model
owner deletes it (single-source is a strictly stronger guarantee) or
rewrites it to assert the barrel re-export. |
| `src/probes/helpers/d5-representatives.test.ts` | quarantine |
**Genuine product gap, deliberately not fixed here.**
`d5-browser-use-smoke` registers `browser-use-smoke` in `D5_REGISTRY`
but `D5_REPRESENTATIVES` has no entry, so
`getD5Representative('browser-use-smoke')` returns `undefined` and the
D5 driver has no representative fixture. The test is correct and
reporting a real hole; the fix belongs to the D5 owner, and this PR
touches no product source. |
### 4. New finding: a load-sensitive flaky test
Not one of the 3, and worth fixing separately:
`src/probes/loader/probe-invoker.test.ts` → *"times out at the invoker
level when enumerate() ignores abortSignal"* asserts a **wall-clock**
`elapsed < 150` against a 100ms invoker timeout racing a 200ms
enumerate. 50ms of headroom does not survive CPU contention — it passed
**5/5 in isolation** but failed twice at ~217ms while the machine was
running other vitest suites, and a 4-vCPU runner executing 173 test
files in parallel is exactly that condition.
Mitigated with `retry: 2` **in the CI config only** (base config
unchanged, so local runs still surface non-determinism). This is not
suppression: a genuinely broken test fails all three attempts and the
gate stays red, and vitest prints every retried test. The real fix is
for the probe-loader owner to drive that assertion off fake timers or
widen the bound, after which the `retry` should come out. Flagging the
tradeoff explicitly rather than burying it.
## Deliberate-breakage proof
A job that has never failed is not yet a gate. Scratch commit
`e100df27a7` broke two product-source files, then `4f092eccb1` reverted
it. The final branch diff is **578 insertions, 0 deletions, and zero
product source touched.**
| Run | SHA | `harness unit suite` | `shell-dashboard unit suite` |
Quarantine ratchet |
|---|---|---|---|---|
| baseline
[`30140087665`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140087665)
| `452a26115e` | **success** (3m33s) | **success** (1m35s) | success |
| **breakage**
[`30140218159`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140218159)
| `e100df27a7` | **FAILURE** (3m31s) | **FAILURE** (1m28s) | success |
The breakage — both reproduced locally before pushing:
- `showcase/harness/src/shared/cell-model/staleness.ts`:
`D4_STALE_AFTER_MS` 1h → 10h (a frozen-green D4 row would credit D4 for
ten hours instead of one)
- `showcase/shell-dashboard/src/lib/sort-order.ts`: `sortOrder` `mastra:
8` → `88`
What the RED run actually caught:
```
harness unit suite — Test Files 3 failed | 168 passed | 2 skipped (173)
Tests 4 failed | 3612 passed | 18 skipped (3634)
FAIL src/http/matrix.test.ts > stale green ladder: raw state='green' but /api/matrix folds to gray (U8)
FAIL src/shared/cell-model/cell-model.contribution.test.ts > family fold: one stale-green sibling forces STALE_DEGRADED
AssertionError: expected 'GREEN_FRESH' to be 'STALE_DEGRADED'
FAIL src/shared/cell-model/cell-model.equivalence.test.ts > golden-master identity > fixture: pos-d4-stale-green
FAIL src/shared/cell-model/cell-model.equivalence.test.ts > golden-master identity > fixture: stale-column
shell-dashboard unit suite — Test Files 3 failed | 64 passed (67)
Tests 4 failed | 1326 passed | 1 skipped (1331)
FAIL src/components/__tests__/depth-utils.test.ts > does not credit D4 when its green chat row is stale
AssertionError: expected 4 to be 3
FAIL src/lib/__tests__/cell-model.test.ts > comm-error staleness window > scopes the window PER ROW FAMILY (G3e)
AssertionError: expected 36060000 to be less than 21600000
```
Three things this establishes beyond "the job can go red":
1. **It catches exactly the class of defect that went undetected on
#6156** — a change to `showcase/harness/src/shared/cell-model/**` landed
4 failures in the harness gate, including two golden-master equivalence
fixtures.
2. **The dashboard job also caught the harness defect**, through the
`live-status.ts` re-export barrel. The shared cell-model now has two
independent gates over it.
3. **`retry: 2` is not suppression.** Every `FAIL` line above appears
three times in the log — the test was retried the full budget and failed
all three attempts. A genuinely broken test cannot pass through it. And
the **quarantine ratchet stayed green on the broken run**, confirming
the quarantine is scoped to its three files and does not swallow
unrelated regressions.
## Runtime
Measured on CI (Depot 4-vCPU), baseline run
[`30140087665`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140087665)
— **both jobs green**:
| Job | CI wall-clock (whole job) | Suite only (local) | Files |
|---|---|---|---|
| `harness unit suite` | **3m33s** | 63.5s | 173 (171 pass / 2 skip,
3616 tests) |
| `shell-dashboard unit suite` | **1m35s** | 7.2s | 67 (1330 pass / 1
skip) |
They run in parallel, so the added critical-path cost is ~3m33s —
comfortably under `showcase_validate.yml`'s existing 25-minute budget
and nowhere near irritating-enough-to-disable. In both jobs the budget
is dominated by install and (for the harness) nx `^build`, not by the
tests.
**No sharding or `nx affected` needed.** `nx affected` would in fact be
actively wrong here: the entire point is that a change anywhere under
`showcase/**` should run the suite covering it, and the historical
failure mode was scope-narrowing, not slowness. If install time later
dominates, the lever is pnpm/npm store caching (already configured) —
not cutting test scope.
## Not covered
`showcase/shell`'s 8 test files (~57 tests) are **also ungated** and I
left them out to keep this change to one concern. They need a third
install path (`showcase/shell` is likewise outside the pnpm workspace)
and its `vitest.config.ts` already has a `globalSetup` that generates
its own `registry.json`. Cheap follow-up.
## Coordination
Concurrent work is adding a vacuity/drift text gate under
`showcase/scripts` and compile-time completeness exports. This PR owns
only the **suite invocation**. The one plausible cherry-pick conflict is
`showcase/harness/package.json` (I added two `scripts` entries and an
`nx.targets` block); `.github/workflows/test_unit-showcase.yml` is a new
file and `nx.json` is deliberately untouched.
|
||
|
Jordan Ritter
|
368058d99b |
ci(showcase): fetch Git LFS in every workflow that serves integration assets (#6160)
## Correction first: the deploy path was already fixed
This PR was opened to migrate ~12 integrations' demo assets out of LFS.
That approach was dropped — the assets stay uniformly LFS-tracked and
the build is the single place this gets resolved.
While establishing scope, one premise turned out to be false and it
matters for how this gets prioritised:
**`showcase_build.yml` does not read the per-slot `"lfs"` flag.**
`ALL_SERVICES` does carry `"lfs":false` for every integration, which
reads as though integration images ship pointer stubs. They do not. The
image-building `build` job (lines 418–604, containing the
`depot/build-push-action` that pushes the deployed image) **hardcodes
`lfs: true`**, added 2026-06-07 in `7bde1eef3aa` — *"ci(showcase): pull
Git LFS assets during build so demo images ship real binaries"* — for
exactly this bug. The only reader of the flag was
`showcase_build_check.yml`.
Evidence, from a real CI run on `main` (`30116731237`, job
`89559665519`, `build (built-in-agent, …)`, 2026-07-24) — note
`built-in-agent` is `"lfs":false` in `ALL_SERVICES`:
```
lfs: true
[command]/usr/bin/git lfs version
git-lfs/3.7.1 (GitHub; linux amd64; go 1.24.4)
[command]/usr/bin/git lfs install --local
##[group]Fetching LFS objects
[command]/usr/bin/git lfs fetch origin refs/remotes/origin/main
```
And confirmed end-to-end against what prod and staging actually serve —
all 10 pointer-committed integrations, both assets:
```
PROD ag2 built-in-agent claude-sdk-python claude-sdk-typescript crewai-crews
langroid llamaindex ms-agent-harness-dotnet pydantic-ai spring-ai
sample.png 200 10083 B ‰PNG real asset
sample.pdf 200 2486 B %PDF real asset
STAGING ag2, llamaindex, spring-ai, google-adk
sample.png 200 10083 B ‰PNG real asset
PROD/STG agno, mastra sample.png/pdf 404 absent <- separate bug, PR #6157
```
**So: zero integration images ship stubs, and there are no broken
multimodal cells from this cause.** The catalogued blast radius is not
expanded. The `agno`/`mastra` failures are `HTTP 404` from files being
**absent**, a different root cause.
Two consequences worth flagging:
- **#6157 is not blocked on this PR.** Once it adds `agno`/`mastra`'s
assets — as plain LFS-tracked files, no overrides — `showcase_build.yml`
already fetches them, so their images get the real bytes on the next
build.
- This PR is **hardening**, not an incident fix.
## What this PR actually changes
Two paths genuinely did not fetch LFS:
| workflow | before | after | impact |
|---|---|---|---|
| `showcase_build.yml` | `lfs: true` (hardcoded) | unchanged | deploy
images already correct |
| `showcase_build_check.yml` | `lfs: ${{ matrix.service.lfs }}` →
**false** | `lfs: true` | pre-merge check baked pointer stubs;
build-only so nothing deployed, but PR CI could never catch a broken
asset |
| `test_e2e-showcase-on-demand.yml` | **no `lfs` input** | `lfs: true` |
**real spec failures** — see below |
| `showcase_capture-previews.yml` | `lfs: true` | unchanged | already
correct |
| `test_showcase-frontend-matrix.yml` | `lfs: true` | unchanged |
already correct |
| `showcase_validate.yml` | none | unchanged | static validation, serves
no assets |
`test_e2e-showcase-on-demand.yml` was the meaningful gap. It runs `npx
next dev` on the integration package and drives it with Playwright,
serving `public/` **straight from the working tree** — so with no LFS
fetch, `multimodal.spec.ts` clicks "Try with sample image", gets a
~130-byte pointer stub served as `image/png`, and fails the magic-byte
guard in `sample-attachment-buttons.tsx` with nothing wrong in the code
under test. Nine of the ten affected integrations ship that spec (all
but `ms-agent-harness-dotnet`).
## Making it not recur
The per-slot `"lfs"` field is **deleted** from both `ALL_SERVICES`
definitions. After the change above it had no consumers —
`showcase_build.yml` ignored it in favour of a hardcoded `true`, and
`showcase_build_check.yml` was its only reader.
Deleting it rather than setting it to `true` is deliberate: a dead flag
that reads as authoritative is how this got mis-set in the first place,
and it is what made the state so hard to reason about here. With no
flag, **integration #21 cannot be added with LFS fetching silently off**
— there is nothing to set and nothing to forget. Both JSON blobs were
re-parsed after the edit (28 and 26 services, zero `lfs` keys).
**Residual risk, stated plainly:** every fetch site is now a literal
`lfs: true`, so correctness rests on a *new workflow* that serves
integration assets remembering to include it. That is a smaller and more
visible surface than a per-service flag, but it is not structurally
enforced. A CI guard asserting that any workflow checking out
`showcase/integrations/*` passes `lfs: true` would close it; not in this
PR.
## Verified affected list — 10 integrations, not 12
Pointer-committed `public/demo-files/sample.{png,pdf}` on `main`,
established from the **committed tree** (local LFS smudge makes the
working tree lie — `ls -l` shows 10083 B for a 130 B pointer):
`ag2`, `built-in-agent`, `claude-sdk-python`, `claude-sdk-typescript`,
`crewai-crews`, `langroid`, `llamaindex`, `ms-agent-harness-dotnet`,
`pydantic-ai`, `spring-ai`
Sizes via the LFS-unaware `git/trees` API: `sample.png` 130 B,
`sample.pdf` 129 B (real: 10083 B / 2486 B). Note the **contents** API
is LFS-aware and reports the resolved 10083 B for pointers too, so it
cannot be used to tell them apart.
The "12" figure conflated two root causes: these 10 have pointer stubs
(correct LFS state, harmless), while `agno`/`mastra` had the files
**absent** (`HTTP 404`, genuinely broken, #6157).
Already carrying per-integration overrides, untouched here:
`google-adk`, `langgraph-python`, `langgraph-fastapi`,
`langgraph-typescript`, `strands`, `strands-typescript`,
`ms-agent-dotnet`, `ms-agent-python`. Consolidating those onto the
uniform LFS convention is a separate follow-up.
## Bandwidth cost
Negligible for the assets themselves — `sample.png` 10083 B +
`sample.pdf` 2486 B ≈ **12.3 KB per integration**.
But `actions/checkout` with `lfs: true` fetches **every** LFS object in
the tree, not just demo files: **203 objects, ~495 MB** (largest are
multi-MB GIFs under `examples/` and `assets/`, up to 65 MB). That is the
real cost driver, and `showcase_build.yml` has already been paying it
for all ~20 integration builds since 2026-06-07 — this PR adds it to the
build-check and on-demand-e2e paths. Worth a follow-up look at whether
those two need a scoped `git lfs pull --include` instead of a blanket
fetch; flagging rather than solving it here, since it is a pre-existing
cost profile.
## Proof
**Local Docker builds were deliberately NOT used as evidence.** A local
build smudges LFS from the local checkout and passes regardless of what
CI does, so it proves nothing. All evidence above is CI-side or from
live prod/staging:
- CI checkout log showing `git lfs fetch` in the deploy build job
(quoted above).
- Live prod + staging HTTP probes returning real magic bytes and exact
byte sizes from CI-built images.
- Remote blob sizes read via the LFS-unaware `git/trees` API.
`actionlint` on all three edited workflows: **24 findings before, 24
after** — none introduced (all pre-existing shellcheck info/style plus
the custom `depot-*` runner label). YAML re-parsed and both
`ALL_SERVICES` blobs re-validated.
---
## CI-side before/after for the `build_check` path
Same integration (`crewai-crews`, one of the ten pointer-committed
ones), from real CI logs:
**BEFORE** — run `30132714792`, job `89610715113` (branch predating this
change):
```
lfs: false
```
No `git lfs install`, no `Fetching LFS objects` group. The image was
built from a pointer stub.
**AFTER** — run `30133303667`, job `89612372095` (this PR):
```
lfs: true
[command]/usr/bin/git lfs version
git-lfs/3.7.1 (GitHub; linux amd64; go 1.24.4)
[command]/usr/bin/git lfs install --local
Git LFS initialized.
##[group]Fetching LFS objects
[command]/usr/bin/git lfs fetch origin refs/remotes/pull/6160/merge
```
Build Check on this PR: **26 build jobs, 25 success, 0 failure** (one
still finishing at time of writing).
## Prod + staging multimodal cell state
Read from PocketBase (`showcase-pocketbase-production` /
`pocketbase-staging-eec0`), not from dashboard chip colours.
| | prod | staging |
|---|---|---|
| `e2e:*/multimodal` | **20/20 green** | **20/20 green** |
| `d5:*/multimodal` | 14 green, **6 red** | 15 green, **5 red** |
Red `d5` cells: `agno`, `built-in-agent`¹, `crewai-crews`, `llamaindex`,
`mastra`, `ms-agent-python` (¹ prod only — green on staging, i.e.
flapping).
**None of these are asset failures.** Every one reports the same
signature:
```
waitForTurnComplete: turn N did not complete within 60000ms (reason=dom-missing, runsFinished=0)
```
Not `is a Git LFS pointer, not the real asset`, and not `Could not fetch
sample … HTTP 404`. A scan of **500** prod status records found **zero**
occurrences of any LFS-pointer, `sample.png`/`sample.pdf` fetch, or
magic-byte error.
The correlation is absent in both directions:
| | |
|---|---|
| pointer-committed **and green** | 7 of 10 — `ag2`,
`claude-sdk-python`, `claude-sdk-typescript`, `langroid`,
`ms-agent-harness-dotnet`, `pydantic-ai`, `spring-ai` |
| **red** despite a real committed blob | `ms-agent-python` (has its own
override, real 10083 B / 2486 B assets) |
| real blob and green | `google-adk`, `langgraph-*`, `strands*`,
`ms-agent-dotnet` |
So pointer-committed status does not predict cell state, which is what
you'd expect once the deploy path fetches LFS.
**Reported separately and loudly:** four of those reds —
`built-in-agent`, `crewai-crews`, `llamaindex`, `ms-agent-python` — are
long-standing genuine failures with `fail_count` between **1713 and
2008** on prod, unrelated to this PR and unrelated to `agno`/`mastra`'s
missing files. They share the `waitForTurnComplete … dom-missing`
turn-timeout signature and need their own investigation. Staging shows
the same set at `fail_count=61`.
|
||
|
Ben Taylor
|
3341007c19 |
fix(core): execute frontend tools when backend returns placeholder result (#5440)
## Summary Fixes the remote-agent HITL case where a frontend tool is skipped because the backend returns the placeholder result `Forwarded to client`. The recovery is now marker-based. `processAgentResult` only removes an existing tool result when the content normalizes to that exact placeholder and there is an executable frontend handler on the current path. Every other backend result stays authoritative and blocks frontend re-execution. ## Implementation - `packages/core/src/core/run-handler.ts` keeps the current `executeFrontendTools` and follow-up flow, then replaces the old existence-only check with exact placeholder detection at https://github.com/CopilotKit/CopilotKit/blob/7449f6d790dbe80ab8e0950f66b04a9bf4acde2c/packages/core/src/core/run-handler.ts#L484-L599 - the placeholder is removed from both `newMessages` and `agent.messages` before the existing specific or wildcard execution path inserts the real result - `.changeset/fix-frontend-tool-placeholder.md` is dropped because current `main` no longer carries a `.changeset` tree ## Regression coverage - exact placeholder replacement for a named frontend tool: https://github.com/CopilotKit/CopilotKit/blob/7449f6d790dbe80ab8e0950f66b04a9bf4acde2c/packages/core/src/__tests__/core-frontend-tool-placeholder.test.ts#L22-L51 - placeholder passthrough when the tool has no handler: https://github.com/CopilotKit/CopilotKit/blob/7449f6d790dbe80ab8e0950f66b04a9bf4acde2c/packages/core/src/__tests__/core-frontend-tool-placeholder.test.ts#L53-L82 - genuine backend-result preservation for a named frontend tool: https://github.com/CopilotKit/CopilotKit/blob/7449f6d790dbe80ab8e0950f66b04a9bf4acde2c/packages/core/src/__tests__/core-frontend-tool-placeholder.test.ts#L85-L109 - existing edge-case guard for a non-placeholder result: https://github.com/CopilotKit/CopilotKit/blob/7449f6d790dbe80ab8e0950f66b04a9bf4acde2c/packages/core/src/__tests__/core-edge-cases.test.ts#L23-L54 - existing wildcard execution still works with no pre-existing result: https://github.com/CopilotKit/CopilotKit/blob/7449f6d790dbe80ab8e0950f66b04a9bf4acde2c/packages/core/src/__tests__/core-frontend-tool-placeholder.test.ts#L111-L154 - wildcard placeholder replacement also stays on the exact marker path: https://github.com/CopilotKit/CopilotKit/blob/7449f6d790dbe80ab8e0950f66b04a9bf4acde2c/packages/core/src/__tests__/core-frontend-tool-placeholder.test.ts#L156-L187 - the recursive follow-up starts with the assistant tool call intact and exactly one real tool message for the same `toolCallId`: https://github.com/CopilotKit/CopilotKit/blob/7449f6d790dbe80ab8e0950f66b04a9bf4acde2c/packages/core/src/__tests__/core-frontend-tool-placeholder.test.ts#L189-L250 Closes #3442. ## Validation - [x] `pnpm exec oxfmt --check packages/core/src/core/run-handler.ts packages/core/src/__tests__/core-edge-cases.test.ts packages/core/src/__tests__/core-frontend-tool-placeholder.test.ts` - [x] `pnpm exec oxlint packages/core/src/core/run-handler.ts packages/core/src/__tests__/core-edge-cases.test.ts packages/core/src/__tests__/core-frontend-tool-placeholder.test.ts` - [x] `pnpm -C packages/shared exec tsdown` - [x] `pnpm -C packages/core exec tsdown` - [x] `pnpm -C packages/core exec vitest run src/__tests__/core-frontend-tool-placeholder.test.ts src/__tests__/core-edge-cases.test.ts` 17/17 - [x] `pnpm -C packages/core exec vitest run` 54 files, 590 tests - [ ] CI green on the rebased head |
||
|
Ben Taylor
|
883ef8e609 |
fix(react-core): forward credentials to provisional agents (#6121)
## What does this PR do? Provisional agents returned by `useAgent` now receive the provider's `credentials` setting when they are created. Cached provisional agents also refresh that setting when the provider configuration changes, while keeping the same agent instance. The regression test covers both the initial value and a later `include` to `omit` update on the cached provisional agent. ### Verification - `pnpm nx run @copilotkit/react-core:test` - `pnpm nx run @copilotkit/react-core:check-types` - `pnpm nx run @copilotkit/react-core:build` - `pnpm check-format` ## Related PRs and Issues - Closes https://github.com/CopilotKit/CopilotKit/issues/6116 ## Checklist - [x] I have read the [Contribution Guide](https://github.com/copilotkit/copilotkit/blob/master/CONTRIBUTING.md) - [ ] If the PR changes or adds functionality, I have updated the relevant documentation - [x] "Allow edits by maintainers" is checked (lets us help iterate on your PR directly — faster turnaround for everyone) |
||
|
Ben Taylor
|
5e5d1de905 |
docs(langgraph): document emit API to v2 hook mapping end-to-end (#5483)
## What does this PR do? Adds end-to-end documentation showing how LangGraph SDK backend emit APIs (`copilotkit_emit_tool_call`, `copilotkit_customize_config`) connect to React v2 frontend hooks (`useFrontendTool`, `useComponent`). Includes a mapping table and a concrete code example demonstrating the full wiring from a LangGraph node to a frontend tool handler. Addresses the documentation gap reported in #3301. ## Related PRs and Issues - Closes #3301 ## Checklist - [x] I have read the Contribution Guide - [x] If the PR changes or adds functionality, I have updated the relevant documentation - [x] "Allow edits by maintainers" is checked |
||
|
Ethan qu
|
98663a43a9 |
fix(react-ui): re-assert sidebar header square corners at >=640px
Per BenTaylorDev's review on #5805, dropping the specificity of the sidebar variant created a tie at desktop widths with the @media (min-width:640px) base rule (both (0,1,0)); the later-authored base won and reverted the sidebar header's top corners from square (border-radius:0) back to rounded 8px. Re-assert the media-scoped sidebar override inside the @media block (mirroring window.css) so the sidebar header stays square at all widths. |
||
|
Ben Taylor
|
8806f668d1 |
fix(runtime): coalesce consecutive same-role Anthropic messages before dispatch (#5360)
## Summary
In multi-turn conversations using the Anthropic adapter, a model turn
that contains both assistant text and a tool call can be replayed as two
consecutive `{role: "assistant"}` entries in the Anthropic payload.
Anthropic expects one message object per turn with alternating roles, so
that split payload can blur turn boundaries on the next request. This PR
coalesces same-role Anthropic messages before dispatch so one assistant
turn stays one assistant message.
## Root cause
[`convertMessageToAnthropicMessage`](https://github.com/CopilotKit/CopilotKit/blob/005aebbededbfdd7978b3c1ee221580b74dd088d/packages/runtime/src/service-adapters/anthropic/utils.ts#L142-L219)
maps each CopilotKit message independently. A mixed assistant turn, a
`TextMessage(role=assistant)` followed by an `ActionExecutionMessage`,
therefore becomes two separate assistant entries. The Anthropic Messages
API requires all content blocks for one assistant turn to be sent in a
single message object: https://docs.anthropic.com/en/api/messages
[`AnthropicAdapter.process()`](https://github.com/CopilotKit/CopilotKit/blob/005aebbededbfdd7978b3c1ee221580b74dd088d/packages/runtime/src/service-adapters/anthropic/anthropic-adapter.ts#L282-L366)
already deduplicates `tool_result` blocks on the user side, but it
previously forwarded the mapped assistant-side payload without a
coalescing pass. Rebasing onto current `main` also exposed test-fixture
drift in the two new regression tests, so the final branch updates those
fixtures to the current object-form `TextMessage` constructor and
removes one stale `ActionInput` field while keeping the production fix
unchanged.
## Changes
- `packages/runtime/src/service-adapters/anthropic/utils.ts`: add
`coalesceConsecutiveSameRoleMessages(messages)` to merge adjacent
equal-role
Anthropic messages by concatenating their `content` arrays.
-
`packages/runtime/src/service-adapters/anthropic/anthropic-adapter.ts`:
call
`coalesceConsecutiveSameRoleMessages` before
`limitMessagesToTokenCount`.
-
`packages/runtime/tests/service-adapters/anthropic/anthropic-adapter.test.ts`:
add same-role regression coverage, align the new fixtures with the
current
object-form `TextMessage` constructor, and remove the stale `parameters`
field
from the action fixture.
- `.changeset/coalesce-anthropic-same-role-messages.md`: patch changeset
for
`@copilotkit/runtime`.
## Scope
The `tool_result` allowlist deduplication stays unchanged. Token
trimming and the orphan-removal post-processor still run after
coalescing, so `tool_use` and `tool_result` pairing remains intact. The
OpenAI, Google, LangChain, and Groq adapters are unaffected.
The reported regression and the explicit regression coverage are
assistant-side. The coalescing helper itself is role-agnostic for
adjacent equal-role array content, but this PR does not add a separate
user-side regression case.
## Related PRs and Issues
Prior attempted fix: #2864, which addressed unrelated message callbacks
rather than the adapter payload.
## Test plan
- [x] `pnpm -C packages/runtime exec vitest run
tests/service-adapters/anthropic/anthropic-adapter.test.ts`
10/10 tests pass, including the two same-role coalescing cases.
- [x] `pnpm -C packages/runtime exec vitest run
tests/service-adapters/anthropic/utils-token-trimming.test.ts`
9/9 tests pass.
- [ ] CI green (`static / quality`, `test / unit` on Node 20/22/24)
## Upstream
Closes #2910.
Reported by @alonronin.
|
||
|
Ben Taylor
|
6a5cccb68f |
fix(core): preserve tool result ordering for parallel frontend tool calls (#5374)
## Summary When an assistant message contains multiple parallel tool calls targeting frontend tools (`useFrontendTool`), the tool result messages are spliced into `agent.messages` in reverse order. This violates the ordering contract that some LLM providers (OpenAI via ADK + LiteLLM) enforce: tool results must appear in the same order as their corresponding `tool_call_id`s. The mismatch causes the provider to silently discard unmatched results during the follow-up run, making it appear as though only the first frontend tool executed. ## Root cause `executeSpecificTool` and `executeWildcardTool` both used the fixed insertion point `messageIndex + 1`, where `messageIndex` is the assistant message's position. Because the assistant message's index doesn't shift when a tool result is inserted after it, each successive splice pushed previously inserted results to the right, reversing the final order. ## Changes - `packages/core/src/core/run-handler.ts`: In both `executeSpecificTool` and `executeWildcardTool`, advance the insertion index past any tool-result messages already present after the parent assistant message before splicing. - `packages/core/src/__tests__/core-parallel-tool-order.test.ts`: New regression coverage for parallel tool-call ordering, including the follow-up path, with type-safe assistant and tool-message assertions on current `main`. - `.changeset/fix-parallel-tool-order.md`: Patch changeset for `@copilotkit/core`. ## What is NOT changed - Sequential execution of tool handlers is preserved; only the splice insertion position changes. - `runTool()` (programmatic API) is unaffected; it creates a single tool call per invocation. - The `newMessages.findIndex` guard that skips already-answered tool calls is unchanged. - Follow-up recursion logic is unchanged. ## Related PRs and Issues Closes #2809. ## Test plan - [x] `npx nx run @copilotkit/core:build` - [x] `pnpm -C packages/core exec vitest run src/__tests__/core-parallel-tool-order.test.ts` — 3/3 pass - [x] `pnpm -C packages/core exec vitest run` — 586/586 pass - [x] `pnpm -C packages/core exec tsc --noEmit` - [ ] CI green |
||
|
Varun Nuthalapati
|
4414d1b42b |
docs(langgraph): align frontend-tools system_prompt examples with current SDK
Switch the code samples from create_react_agent(middleware=..., prompt=...) to create_agent/createAgent with system_prompt/systemPrompt, matching the current pinned langchain 1.x API used elsewhere in the docs. Wire copilotkitMiddleware into the TypeScript example's middleware array (it was imported but never attached). Point the "Imperatively emitting tool calls" link at the correct /integrations/langgraph/ prefixed route. Addresses review feedback on #5469. |
||
|
Varun Nuthalapati
|
9f7fef178b |
docs(langgraph): fix emit-api-hook-mapping links and placeholder model
Point "See also" links at the /integrations/langgraph/ prefixed routes (the bare /langgraph/... links 404), drop the dead Python SDK reference link (no target page), and swap the placeholder "gpt-5.4" model string for a real one (gpt-4o) in both code samples. Addresses review feedback on #5483. The unrelated use-agent.tsx isReady change was dropped by rebasing this branch onto upstream/main and carrying forward only the docs commit — isReady already ships on main via #6041. |
||
|
Varun Nuthalapati
|
dda8754f4a |
docs(langgraph): document emit API to v2 hook mapping end-to-end
Adds a new page under integrations/langgraph/advanced/ explaining how LangGraph SDK backend emit functions (copilotkit_emit_tool_call, copilotkit_customize_config) connect to React v2 frontend hooks (useFrontendTool, useComponent). Includes a mapping table, two concrete end-to-end examples, and a decision guide. Closes #3301 |
||
|
Tushar-Khandelwal-2004
|
d2c2c9f005 | Treat blank tool content as empty inspector data |