mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
malabo
4758 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jordan Ritter
|
7282ecddfb |
ci(showcase): run the harness and shell-dashboard unit suites in CI (#6168)
## The gap
**No CI job ran the showcase unit suites.** Not "ran them partially" —
no job invoked them at all. Verified per suite against `91f0321397` (the
tip of `main` this branched from), by reading every workflow's actual
`run:` lines rather than trusting job names:
| Suite | Tests | Ran in CI before? | Where |
|---|---|---|---|
| `showcase/harness` | 3646 (3625 pass / 3 fail / 18 skip) | **NO** |
nowhere |
| `showcase/shell-dashboard` | 1331 (1330 pass / 1 skip) | **NO** |
nowhere |
| `showcase/shell` | ~57 (8 files) | **NO** | nowhere |
| `showcase/scripts` | — | yes | `showcase_validate.yml` → "Run build
pipeline tests" |
Why each candidate workflow misses them:
- **`test_unit.yml`** — the one whose name implies it covers this —
excludes them *twice over*: `paths-ignore` lists `showcase/**`, **and**
its nx selection is hard-scoped to `--projects='packages/**'`.
- **`showcase_validate.yml`** runs `pnpm exec vitest run` only in
`showcase/scripts`. It has two `working-directory: showcase/harness`
steps, but they are a CVDIAG perf bench (`vitest bench
src/cvdiag/emit-perf.bench.ts`) and an ESM boot-smoke — neither runs the
unit suite.
- **`static_quality.yml`** does run `nx run-many -t check-types`. This
resolves the reported name/invocation divergence: the *job* is called
`check-types` and the harness's script is called `typecheck`, so no
`check-types` target exists on the harness and **the harness is not
typechecked either**.
Net effect: ~5000 showcase unit tests gated nothing. A PR could carry
real defects in `showcase/harness/src/**` and still show an all-green
check list, because no job structurally could have caught them.
## What this wires
New workflow **`test / unit-showcase`**
(`.github/workflows/test_unit-showcase.yml`), two parallel jobs on PR +
push-to-main filtered to `showcase/**`:
- **`harness unit suite`** — `nx run
@copilotkit/showcase-harness:test:ci`, then `nx run
@copilotkit/showcase-harness:test:quarantine-ratchet`. Driven through nx
per the root `CLAUDE.md` task convention.
- **`shell-dashboard unit suite`** — `vitest run` directly; the
dashboard is deliberately outside the pnpm workspace (see the note in
`pnpm-workspace.yaml`) so it is not an nx project and has no target to
run.
New nx targets are declared in **`showcase/harness/package.json`'s
`nx.targets` block**, not in `nx.json` — they are harness-specific, and
a `nx.json` `targetDefaults` entry would silently apply to any future
project that happens to share the script name. Both are `cache: false`
on purpose: the nx `test` named-input covers `src/**` and `*.test.*` but
**not** `vitest.ci.config.ts` or `vitest.quarantine.json`, so a cached
result could survive an edit to the quarantine list.
### Why a separate workflow rather than steps in `showcase_validate.yml`
That file is a single ~900-line job already budgeted at 25 minutes and
is the busiest merge-path workflow in the repo. Splitting the unit
suites out gives them their own name in the check list, their own
concurrency group, and — since the two suites need different package
managers — lets them run in parallel instead of lengthening the critical
job. Naming follows the existing convention (`test_unit.yml`,
`test_unit-python-sdk.yml`, `test_unit-spring-ai.yml`).
**No `continue-on-error` and no `|| true` anywhere in the file.** A gate
that cannot fail is not a gate.
## Hazards, and how each is handled
### 1. The hanging spike test
*Correction to the reported location:* it is
`showcase/shell-dashboard/tests/runtime-env-switch.spike.test.ts`, not
under `showcase/harness/`. It reaches the dashboard suite via that
package's `include: ["src/**/*.test.{ts,tsx}",
"tests/**/*.spike.test.ts"]`.
**Reproduced locally.** vitest sat at **0.0% CPU with zero output** and
never even spawned the `next build` its `beforeAll` calls — no worker
fork, nothing. Killed at ~2 minutes; consistent with the reported
~25-minute stall.
**Handled by excluding it from the unit gate**, not by a hard timeout.
Justification: it is an integration spike by construction — one `next
build` plus two `next start` boots on fixed ports 3801/3802 — and this
package's own config comment already says it is *"too heavy for the
per-file unit suite"* even though the `include` glob pulls it in anyway.
A hard timeout would convert a 25-minute stall into a red gate with
nothing actionable in it; this suite has to stay a fast, trustworthy
unit signal. The spike needs its own job with a real server budget,
which is out of scope for wiring up the unit gate.
Note the `--exclude` values restate the package's existing config
excludes (`tests/visual/**`, `node_modules/**`) because vitest's CLI
`--exclude` **replaces** the config value rather than extending it —
dropping them would silently re-enable them.
### 2. Gitignored generated artifacts
`showcase/.gitignore` ignores `shell/src/data/*.json` and
`shell-dashboard/src/data/*.json`, but the suites consume them:
- `harness/src/probes/frontend-matrix.test.ts` **statically imports**
`shell/src/data/frontend-catalog.json` — a missing file is a module
*load error*, not a test failure.
- `harness/src/fleet/control-plane/d0-gone-predicate.test.ts` reads
`shell/src/data/registry.json` at runtime.
- `shell-dashboard/src/lib/docs-status.ts` **statically imports**
`@/data/docs-status.json`.
Generated as an explicit job step (`generate-registry.ts`, plus
`probe-docs.ts` for the dashboard), mirroring how
`showcase_validate.yml` drives the same generators. The dashboard job
invokes vitest directly rather than `npm test` specifically so the
`pretest` hook does not re-run `probe-docs.ts` and double its ~50
outbound HEAD requests.
Also load-bearing: `npm ci --ignore-scripts` in the dashboard job. That
package's `postinstall` is `cd ../scripts && npm install`, which would
lay an npm-resolved `showcase/scripts/node_modules` over the
pnpm-managed one.
### 3. The 3 pre-existing failures — quarantined behind a ratchet, none
deleted or weakened
Confirmed byte-identical to `origin/main` and pre-existing. A job that
is red on arrival gets ignored or disabled, so the three files are
excluded via `showcase/harness/vitest.quarantine.json` +
`vitest.ci.config.ts` — **and `scripts/quarantine-ratchet.ts` re-runs
each one and requires it to STILL FAIL.** The moment somebody fixes a
quarantined test, the ratchet goes red and names the entry to delete. An
entry therefore cannot outlive the failure it excuses. The ratchet also
fails on an entry pointing at a missing file, on an entry whose filter
matches more than one file, and on a malformed or under-documented
manifest (every entry must carry `file` / `since` / `reason` /
`unquarantineWhen`).
Local `pnpm test` still uses the unfiltered config, so a developer sees
the quarantined failures.
| File | Disposition | Why |
|---|---|---|
| `src/probes/frontend-matrix.test.ts` | quarantine | Asserts hard-coded
per-frontend counts (`react=660`, `angular=636`) against the
**generated** `frontend-catalog.json`, which has grown to 1302 runnable
cells (`react=664`). Stale literals, not a product defect — the two
assertions that express the test's stated intent ("without loss or
duplication": `length === metadata.runnable`, ids unique) both pass.
Exit criterion: derive the counts from the catalog; re-hard-coding fresh
numbers just resets the clock. |
| `src/probes/helpers/d5-mapping-drift.test.ts` | quarantine |
**Stranded by a completed refactor.** It regex-scrapes
`CATALOG_TO_D5_KEY` out of `shell-dashboard/src/lib/live-status.ts` to
prove the dashboard copy hasn't drifted from the harness's
`REGISTRY_TO_D5`. That file is now a one-line `export *` barrel over the
canonical `harness/src/shared/cell-model/live-status.ts` — there is no
second copy to drift and nothing for the regex to find, so the test
throws its own "could not locate" guard. Exit criterion: cell-model
owner deletes it (single-source is a strictly stronger guarantee) or
rewrites it to assert the barrel re-export. |
| `src/probes/helpers/d5-representatives.test.ts` | quarantine |
**Genuine product gap, deliberately not fixed here.**
`d5-browser-use-smoke` registers `browser-use-smoke` in `D5_REGISTRY`
but `D5_REPRESENTATIVES` has no entry, so
`getD5Representative('browser-use-smoke')` returns `undefined` and the
D5 driver has no representative fixture. The test is correct and
reporting a real hole; the fix belongs to the D5 owner, and this PR
touches no product source. |
### 4. New finding: a load-sensitive flaky test
Not one of the 3, and worth fixing separately:
`src/probes/loader/probe-invoker.test.ts` → *"times out at the invoker
level when enumerate() ignores abortSignal"* asserts a **wall-clock**
`elapsed < 150` against a 100ms invoker timeout racing a 200ms
enumerate. 50ms of headroom does not survive CPU contention — it passed
**5/5 in isolation** but failed twice at ~217ms while the machine was
running other vitest suites, and a 4-vCPU runner executing 173 test
files in parallel is exactly that condition.
Mitigated with `retry: 2` **in the CI config only** (base config
unchanged, so local runs still surface non-determinism). This is not
suppression: a genuinely broken test fails all three attempts and the
gate stays red, and vitest prints every retried test. The real fix is
for the probe-loader owner to drive that assertion off fake timers or
widen the bound, after which the `retry` should come out. Flagging the
tradeoff explicitly rather than burying it.
## Deliberate-breakage proof
A job that has never failed is not yet a gate. Scratch commit
`e100df27a7` broke two product-source files, then `4f092eccb1` reverted
it. The final branch diff is **578 insertions, 0 deletions, and zero
product source touched.**
| Run | SHA | `harness unit suite` | `shell-dashboard unit suite` |
Quarantine ratchet |
|---|---|---|---|---|
| baseline
[`30140087665`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140087665)
| `452a26115e` | **success** (3m33s) | **success** (1m35s) | success |
| **breakage**
[`30140218159`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140218159)
| `e100df27a7` | **FAILURE** (3m31s) | **FAILURE** (1m28s) | success |
The breakage — both reproduced locally before pushing:
- `showcase/harness/src/shared/cell-model/staleness.ts`:
`D4_STALE_AFTER_MS` 1h → 10h (a frozen-green D4 row would credit D4 for
ten hours instead of one)
- `showcase/shell-dashboard/src/lib/sort-order.ts`: `sortOrder` `mastra:
8` → `88`
What the RED run actually caught:
```
harness unit suite — Test Files 3 failed | 168 passed | 2 skipped (173)
Tests 4 failed | 3612 passed | 18 skipped (3634)
FAIL src/http/matrix.test.ts > stale green ladder: raw state='green' but /api/matrix folds to gray (U8)
FAIL src/shared/cell-model/cell-model.contribution.test.ts > family fold: one stale-green sibling forces STALE_DEGRADED
AssertionError: expected 'GREEN_FRESH' to be 'STALE_DEGRADED'
FAIL src/shared/cell-model/cell-model.equivalence.test.ts > golden-master identity > fixture: pos-d4-stale-green
FAIL src/shared/cell-model/cell-model.equivalence.test.ts > golden-master identity > fixture: stale-column
shell-dashboard unit suite — Test Files 3 failed | 64 passed (67)
Tests 4 failed | 1326 passed | 1 skipped (1331)
FAIL src/components/__tests__/depth-utils.test.ts > does not credit D4 when its green chat row is stale
AssertionError: expected 4 to be 3
FAIL src/lib/__tests__/cell-model.test.ts > comm-error staleness window > scopes the window PER ROW FAMILY (G3e)
AssertionError: expected 36060000 to be less than 21600000
```
Three things this establishes beyond "the job can go red":
1. **It catches exactly the class of defect that went undetected on
#6156** — a change to `showcase/harness/src/shared/cell-model/**` landed
4 failures in the harness gate, including two golden-master equivalence
fixtures.
2. **The dashboard job also caught the harness defect**, through the
`live-status.ts` re-export barrel. The shared cell-model now has two
independent gates over it.
3. **`retry: 2` is not suppression.** Every `FAIL` line above appears
three times in the log — the test was retried the full budget and failed
all three attempts. A genuinely broken test cannot pass through it. And
the **quarantine ratchet stayed green on the broken run**, confirming
the quarantine is scoped to its three files and does not swallow
unrelated regressions.
## Runtime
Measured on CI (Depot 4-vCPU), baseline run
[`30140087665`](https://github.com/CopilotKit/CopilotKit/actions/runs/30140087665)
— **both jobs green**:
| Job | CI wall-clock (whole job) | Suite only (local) | Files |
|---|---|---|---|
| `harness unit suite` | **3m33s** | 63.5s | 173 (171 pass / 2 skip,
3616 tests) |
| `shell-dashboard unit suite` | **1m35s** | 7.2s | 67 (1330 pass / 1
skip) |
They run in parallel, so the added critical-path cost is ~3m33s —
comfortably under `showcase_validate.yml`'s existing 25-minute budget
and nowhere near irritating-enough-to-disable. In both jobs the budget
is dominated by install and (for the harness) nx `^build`, not by the
tests.
**No sharding or `nx affected` needed.** `nx affected` would in fact be
actively wrong here: the entire point is that a change anywhere under
`showcase/**` should run the suite covering it, and the historical
failure mode was scope-narrowing, not slowness. If install time later
dominates, the lever is pnpm/npm store caching (already configured) —
not cutting test scope.
## Not covered
`showcase/shell`'s 8 test files (~57 tests) are **also ungated** and I
left them out to keep this change to one concern. They need a third
install path (`showcase/shell` is likewise outside the pnpm workspace)
and its `vitest.config.ts` already has a `globalSetup` that generates
its own `registry.json`. Cheap follow-up.
## Coordination
Concurrent work is adding a vacuity/drift text gate under
`showcase/scripts` and compile-time completeness exports. This PR owns
only the **suite invocation**. The one plausible cherry-pick conflict is
`showcase/harness/package.json` (I added two `scripts` entries and an
`nx.targets` block); `.github/workflows/test_unit-showcase.yml` is a new
file and `nx.json` is deliberately untouched.
|
||
|
Varun Nuthalapati
|
9f7fef178b |
docs(langgraph): fix emit-api-hook-mapping links and placeholder model
Point "See also" links at the /integrations/langgraph/ prefixed routes (the bare /langgraph/... links 404), drop the dead Python SDK reference link (no target page), and swap the placeholder "gpt-5.4" model string for a real one (gpt-4o) in both code samples. Addresses review feedback on #5483. The unrelated use-agent.tsx isReady change was dropped by rebasing this branch onto upstream/main and carrying forward only the docs commit — isReady already ships on main via #6041. |
||
|
Varun Nuthalapati
|
dda8754f4a |
docs(langgraph): document emit API to v2 hook mapping end-to-end
Adds a new page under integrations/langgraph/advanced/ explaining how LangGraph SDK backend emit functions (copilotkit_emit_tool_call, copilotkit_customize_config) connect to React v2 frontend hooks (useFrontendTool, useComponent). Includes a mapping table, two concrete end-to-end examples, and a decision guide. Closes #3301 |
||
|
Jordan Ritter
|
b04330abdb |
test(showcase): pin the polarity flip's D1/D2 case and make family folds mutation-detectable
The fail-safe polarity flip's widest case was frozen by nothing. A stripped-
`signal` red on a `health:`/`agent:` row flips a whole column green->red
(achieved 6->0, isRegression true) because those keys are integration-scoped,
but `red-signal-unknown` was only applied by `positionSweep` (D3-D6), and
`liveness-fresh-red-d1/-d2` use `row()`'s default `signal: null` -- which is
not `undefined`, so they classified FAIL_FRESH under the old polarity too.
Measured: restoring `!redSignalKnown -> NO_DATA` killed 4 golden-master entries,
all D3-D6, and left the entire D1/D2 leg byte-identical.
Underneath that sat a structural blind spot: every fixture variant maps ONE
uniform row shape over ALL keys of a rung, so no family anywhere contained two
DIFFERING red rows and no family contained a green row with an absent `signal`
key. A uniform set has nothing to aggregate, so three explicitly load-bearing
family folds were invisible at golden-master level -- `Math.max`->`Math.min` on
`maxNonInfraRedFailCount`, `allRedSoftClass` `every`->`any`, and re-widening the
provenance flag from red-row to family scope each left all 56 entries identical.
Adds 5 fixtures (56 -> 61 baseline entries):
- `liveness-red-signal-unknown-d1` / `-d2` -- a red D1/D2 row with `signal:
undefined` over a green D3-D6 ladder. Both now fail under a polarity
re-widening, joining the 4 existing entries.
- `hetero-d4-red-failcount-max` -- D4 {chat red fc1, tools red fc3}, straddling
D4_FIRST_STRIKE_THRESHOLD. MAX(1,3)=3 -> FAIL_FRESH/red; MIN(1,3)=1 ->
FIRST_STRIKE_FRESH/amber, i.e. a fresh sibling re-arming tolerance for an
already-confirmed failure, which the MAX doc explicitly forbids.
- `hetero-d4-green-stripped-infra-red` -- D4 {chat green signal-omitted, tools
red driver-error}, the production shape `classifyRung` cites verbatim and the
first fixture with a green row whose `signal` key is absent. Red-row scoping
keeps this gray; a family-scoped flag paints it red.
- `hetero-starter-mixed-soft-hard` -- starter {transport-error (soft),
smoke-failed (hard), rest green}, both below SOFT_MISS_TOLERANCE_THRESHOLD so
the soft-class quantifier is the only deciding factor.
Each mutation is now caught by exactly one fixture while its pre-existing
sibling stays blind, so the teeth come from the new shapes rather than
incidental coupling. Baseline entries were derived by hand from the source
first and inserted additively (140 insertions, 0 deletions) -- no wholesale
regeneration, so no result is self-approving.
Also closes the D1/D2 gap in the dashboard equivalence suite: the `fail-safe
polarity` fixtures seeded no `health:`/`agent:`/D5/D6 rows, so those rungs
contributed ABSENT -- also gray -- and the gray assertions passed via the
no-data path while naming the infra path. A `greenScaffold` helper makes every
rung but the one under test green-fresh, discriminating pill assertions prove
the gray sits over a present red pill, and a new test pins the whole-column
D1/D2 stripped-red case api-vs-browser (equal chip, achieved 0, regression
true).
Fixtures and baseline only -- no classification logic, `combine.ts`, or
coherence invariant touched.
|
||
|
Jordan Ritter
|
49dea09973 |
test(showcase): stop INV7 false-failing on the I1 ladder-gap shape
INV7 (`assertChipStripCoherent`) fired on a legitimate engine shape: with no
`e2e:` row, `scanWorst` breaks at the ABSENT D3 rung (invariant I1) and never
consults D4, so a fresh product-red `chat:<slug>` renders a gray chip over a red
D4 pill. INV7 demanded infra evidence for a red the chip provably never read.
Round-1 finding a4(iii), unfixed by the round-1 INV7 fix; re-raised as a8.
Adds a third, narrow carve-out beside the U8 all-stale one: the gray is excused
only when the ladder walk stopped for want of any observation STRICTLY BELOW the
shallowest red pill. `ladderGapDepth` tests exactly that ("no row for any of the
rung's keys"), which is provably equivalent to `classifyRung`'s ABSENT for a
D3/D4/D5 rung — deliberately not the strip's `status === null`, which also
covers the `anyExpectedMissing` NO_DATA collapse (a rung that does have an
observation and does not stop the walk). `combine.ts` is untouched: the
gap-break's own masking is a separate change needing a golden-master re-freeze.
The allowance is one-directional and canaried. A gap ABOVE the red excuses
nothing, and when the gap-break is eventually fixed the new case's chip stops
being gray, so its precondition assertion fails loudly rather than letting the
carve-out excuse a shape the engine no longer produces.
Also removes INV7's unreachable starter branch (b24). A starter cell has no
depth strip, so INV7's precondition is structurally unsatisfiable there; the
branch implied coverage that does not exist. It is now a loud throw, and a new
test pins the structural reason and fails if the starter axis gains a strip.
New cases: the gap shape (green, plus an in-suite mutation proof that the
allowance is what passes it), a gap above the red, a gapless gray-over-red, and
an UNPERTURBED engine model INV7 still rejects (an infra-red D4 grays a
product-red D6 — D6 is outside `scanWorst`'s scan). That last one corrects the
file's claim that the engine no longer produces a gray-over-red incoherence.
Exhaustive sweep over 295,245 rung combinations: 25,504 INV7 failures without
the allowance, 1,456 with. Every remaining one has its gap at or above the
shallowest red pill, i.e. none is gap-induced; they are two other pre-existing
engine masking mechanisms and INV7 keeps its teeth on them.
|
||
|
Jordan Ritter
|
ed10b61ffb |
test(showcase): give the initial-fetch page cap a two-sided guard
The cap "drift guard" did not guard. Every assertion in
useLiveStatus.supplemental-bounds.test.tsx was an UPPER bound
(toBeLessThanOrEqual / not.toContain(cap + 1)), so only an increase was
caught. Measured on the pre-fix branch, MAX_INITIAL_FETCH_PAGES could be
halved to 10 or cut to 5 with all 6 tests still green. A lowered cap is just
as much a silent behavior change -- it moves the cap toward the legitimate
read and starts truncating real collections. The bound assertions were also
vacuous, because Math.max of an empty array is -Infinity, so a supplemental
fetch that never ran at all satisfied them.
Both sites now assert the EXACT page sequence a full walk of the cap produces
(1..cap plus the one terminal lookahead), sorted because fan-out waves are
concurrent, and DISTINCT on the fail-loud bulk path because the throw re-runs
the whole loop once per reconnect attempt. Zero pages now fails, a halved cap
now fails. The literal is additionally pinned MECHANICALLY against the
exported constant, so the "MUST match" claim in its doc is checked rather
than trusted.
New file useLiveStatus.page-cap-boundary.test.tsx covers the boundary the
suites never reached -- a COMPLETE read that ends exactly on the cap page --
in both directions and at both levels.
- DIRECT on paginateStatusPages. Exactly cap x 500 rows is NOT truncated
(and issues one lookahead, then stops); one row past the cap IS truncated;
a read ending on a short page never issues the lookahead; a failed
lookahead fails safe to truncated.
- THROUGH the hook. A complete 10,000-row collection renders (status live,
all rows, red cell still RED from a signal-less bulk row) instead of the
permanent blank-dashboard outage; one row past the cap still fails LOUD
with an error and no rows.
It carries its own ../lib/pb mock rather than reusing the pb-query-eval fake
servers. The boundary needs 10,000 rows in the BULK response, which no shared
fixture serves, and nothing this file proves should be able to go green or red
because of a change to those servers' filter fidelity.
|
||
|
Jordan Ritter
|
75fdfb3aba |
fix(showcase): make truncated mean "rows unread", not "cap exhausted"
`fetchStatusPages` inferred truncation from cap exhaustion with a full last page. That is not evidence: the loop learns where the collection ends only from a SHORT page (`skipTotal: true` drops `totalItems`/`totalPages`), so a collection of exactly MAX_INITIAL_FETCH_PAGES x INITIAL_PAGE_SIZE rows is read COMPLETELY and still ends on a full page. The bulk consumer THROWS on truncation, so at exactly 10,000 rows the dashboard went offline deterministically and permanently -- every retry, every reload, because the condition is a pure function of the row count. The function's own JSDoc asserted the opposite of what it did. This is the same false-outage class matrix.ts already fixed server-side (any full final page read as truncation took /api/matrix down on every 500-boundary), reintroduced client-side. `truncated` now means exactly "there are rows we did not read", and it is proven rather than inferred: on cap exhaustion with a full last page the loop issues ONE lookahead read of page MAX_INITIAL_FETCH_PAGES + 1 whose rows are discarded and only whose emptiness is load-bearing. Chosen over `skipTotal: false` on page 1 (the matrix.ts shape) because that puts a COUNT(*) on the first-paint path of EVERY load -- the exact overhead `skipTotal: true` exists to drop -- to disambiguate a case that can only arise after 20 consecutive full pages; the lookahead puts the whole cost on the pathological path, where it is a rounding error against the 20 requests that preceded it. A FAILED lookahead fails safe (unproven completeness is reported as truncated), keeping the bulk path's fail-loud posture. The asymmetric on-hit semantics are unchanged and remain correct: supplemental truncation DEGRADES (reds stay red), bulk truncation THROWS (unread rows are whole cells that would render gray = masked failure). The false positive is fixed; the throw is not weakened. The pagination algorithm is hoisted out of the effect as `paginateStatusPages` over an injected page reader, so the `truncated` contract can be asserted directly rather than only through the hook's observable state -- and so every page of a read, INCLUDING the lookahead, provably carries the same `listOpts`. MAX_INITIAL_FETCH_PAGES is exported (for a mechanical, two-sided test pin) and its value is now justified against both bounds it actually has to satisfy: ~3x the measured production worst case below, and bounded runaway damage above; the value is unchanged at 20. |
||
|
Jordan Ritter
|
b8b75b7d45 |
test(showcase): collapse three PocketBase query evaluators into one, verified against a real server
The dashboard's `useLiveStatus` test doubles carried THREE divergent implementations of the same PocketBase query evaluator — the nominally-shared `__tests__/pb-query-eval.ts`, a second complete private copy inside `useLiveStatus.autocancel.test.tsx`, and a third partial one in the supplemental-bounds fakes. They disagreed on four axes, and the shared one was the weakest of the three. Three definitions of "what PocketBase does" is no definition at all. There is now exactly one, and every semantic in it is a measured observation from a real PocketBase v0.22.21 server (the version `showcase/pocketbase/Dockerfile` pins) plus the real SDK's own `filter()`. `pb-query-eval.test.ts` pins that oracle case by case, with the live result recorded next to each expectation. Fidelity defects fixed, all of which the module's own docs asserted backwards: - Single-quoted literals. `pb.filter()` wraps string params in SINGLE quotes with `\'` escaping and emits numbers/booleans/null BARE. The shared tokenizer handled only double quotes, so every dimension-scoped filter the hook builds threw `unsupported character "'"`. - LIKE is case-INSENSITIVE (it compiles to SQLite `LIKE`). - The implicit `%…%` wrap is triggered by `%` alone, and it ESCAPES `%` and `_` so they match literally. `_` does not suppress the wrap. Both prior copies got a different half of this wrong. - Ordering and equality follow the COLUMN type: `fail_count > 2` selects `fail_count` 10, which the string comparison in the private copy excluded. - `sort=-key` descends instead of silently no-opping. And it now genuinely fails loud rather than claiming to. An unterminated literal, a bare word (which real PB reads as a column reference and 400s on), a dangling operator or escape, an unbalanced paren, trailing input, an unsupported operator, a sort it cannot apply, a non-numeric `page`, and a field the row does not carry all throw — matching the real server, which answers every one of them with a 400. Serving that contract over HTTP is structural now, not per-fake discipline. `matchesPbFilter` used to be called bare inside two `createServer` handlers, so a parse error produced NO response: the hook hung and the test died on a 20 s `waitFor` timeout with the parse message nowhere in sight. Fakes go through `evaluatePbList`, which cannot throw — it returns either a 200 or the real server's own 400 carrying the message. All three supplemental-bounds fakes now answer through it, on both their bulk and their supplemental legs. Two of them ignored `filter` entirely while that file's preamble claimed all of them evaluated it, which left the page-cap tests proving themselves on fixture shape: with the fakes repaired, breaking the supplemental filter turns the supplemental cap test red, and breaking the bulk filter turns the bulk cap test red. Both were green before. |
||
|
Jordan Ritter
|
4f092eccb1 |
Revert "test(showcase): SCRATCH deliberate breakage to prove the new gate fails"
This reverts the scratch commit. Its only purpose was to prove the two new jobs fail on a real defect in product source — see the PR body for the RED run ID. The branch is back to a strictly additive CI change with no product diff. |
||
|
Jordan Ritter
|
e100df27a7 |
test(showcase): SCRATCH deliberate breakage to prove the new gate fails
TEMPORARY — reverted in the next commit. A job that has never failed is not yet
a gate, so this proves both new jobs actually catch a defect in product source.
harness: D4_STALE_AFTER_MS 1h -> 10h in shared/cell-model/staleness.ts
(a frozen-green D4 row would credit D4 for ten hours instead of one)
dashboard: sortOrder mastra 8 -> 88 in lib/sort-order.ts
Both reproduce locally before pushing: harness 3 failed / 177 passed in
src/shared/cell-model, dashboard 4 failed / 1326 passed.
|
||
|
Jordan Ritter
|
f5ff535ed6 |
test(showcase): add ratcheted quarantine config for the harness unit suite
Wiring the harness unit suite into CI (next commit) found it already RED on
main with exactly three failures. A job that is red on arrival gets ignored or
disabled, so this adds a CI-only vitest config that excludes those three files
— and a ratchet that stops the exclusion from becoming permanent.
vitest.quarantine.json the three entries, each with a date, a reason,
and an explicit exit criterion
vitest.ci.config.ts `test` minus the quarantined files
scripts/quarantine-ratchet.ts re-runs each quarantined file and requires it
to STILL FAIL
The ratchet is what makes the exclusion defensible: the moment somebody fixes a
quarantined test, it goes red and names the entry to delete, so an entry can
never outlive the failure it excuses. It also fails on an entry that points at a
missing file, on an entry whose filter matches more than one file, and on a
malformed manifest.
No test is deleted and no assertion is changed. The three failures are a stale
hard-coded cell count, a drift test stranded by a completed single-source
refactor, and a genuine `D5_REPRESENTATIVES` gap for `browser-use-smoke` that
belongs to the D5 owner.
`test:ci` and `test:quarantine-ratchet` are declared in the harness package.json
`nx.targets` block (project-local, not a workspace default) with `cache: false`:
the nx `test` named-input covers `src/**` but not the quarantine manifest, so a
cached result could survive an edit to the exclusion list.
Local `pnpm test` still uses the unfiltered config — a developer should see the
quarantined failures.
|
||
|
Jordan Ritter
|
938be2accf |
test(showcase): reconcile the supplemental-fetch discriminator across both dashboard suites
Integration defect, introduced by folding the autocancel fake-server fidelity
repair together with the bounded/projected supplemental fetch. Each was correct
alone; together they disagreed about how a supplemental request is IDENTIFIED.
The fidelity repair's `isSupplementalRequest` keyed on the ABSENCE of a `fields`
projection. That was true when it was written and is now false: the supplemental
fetch sends `STATUS_LIST_FIELDS,signal`. So all three faithful fake servers
classified the supplemental request as a BULK page, which:
- failed `expect(fields).not.toContain("signal")` on the "every bulk page
projects `signal` away" wire contract (the supplemental query was in the
bulk bucket), and
- left `supplementalQueries` / `firstElemSupplementalPages` EMPTY, so the two
`toEqual([1])` bounds assertions compared `[]` against `[1]`.
Whole autocancel suite: 3 failed / 3. Not a flake and not a pre-existing
failure — it is exactly the two-conventions collision the integration had to
resolve.
Resolved toward the convention the mocked sibling suite already uses after the
projection landed: a request is supplemental iff `signal` IS in its `fields`.
That is the property that actually defines the request rather than an artifact
of how it happened to be built, so it survives both the projection being added
and the bulk projection changing width. Both dashboard suites now assert the
SAME convention, and dropping `fields: STATUS_SIGNAL_FIELDS` from the hook turns
both of them red (verified), so neither can silently drift back.
Also corrected the three comments that still described the supplemental fetch as
unprojected.
Tests only — no production change. `git diff` against the hook is empty.
|
||
|
Jordan Ritter
|
51cc63d974 |
chore(showcase): store demo assets uniformly as Git LFS pointers
Demo assets under showcase/integrations/*/public/{demo-files,demo-audio}/
were stored two different ways. Ten integrations committed them as LFS
pointers (the root .gitattributes convention); eight carved themselves out
with a per-integration .gitattributes that re-declared the same paths
`-filter -diff -merge`, committing raw binaries instead.
Those carve-outs were added when the image build did not fetch LFS, so a
pointer stub shipped into the image and the multimodal sample-attachment
magic-bytes guard rejected it. That premise no longer holds: the deploy
build's Checkout step hardcodes `lfs: true` (
|
||
|
Jordan Ritter
|
2797618c5e |
docs(showcase): correct and anchor stale cold-load fetch doc claims
Every numeric claim below is re-derived from production PocketBase (showcase-pocketbase-production.up.railway.app) or from repo config on 2026-07-24, not copied from a neighbouring comment. - Row/page counts: 2455 rows / 5 pages -> 3082 rows (~3100) / 7 pages, with the exact source query recorded so the number can be refreshed. The INITIAL_FANOUT_BATCH no-over-fetch note now says that it holds because the CURRENT page count is odd, not as an algorithm property. - Byte estimate: ~1.29 MB -> ~1.6 MB. The retired figure applied the (correct) 371 KB -> 113 KB per-500-row basis to the OLD 2455-row collection. Measured all 7 pages both ways: 2.34 MB full vs 0.70 MB projected = 1.64 MB of signal. - Worst-case re-delivery window: ~29 min -> ~60 min. staleness.ts and harness/config/probes/*.yml schedule e2e-demos, starter_smoke and d6-all-pills-e2e HOURLY (10/40 * * * *); aimock-wiring is 6-hourly and the drift probes are weekly/monthly. - Restored the coverage-gap acknowledgement this PR deleted, rewritten for what is true today: the supplemental union is neither minimal nor complete. It is green-blind outside clause 1 (2646 rows carry a signal but sit outside it; the 731 green per-cell rows under the aggregate dimensions are excluded by BOTH clauses). Currently latent (0 uncovered rows carry __fleetCommError) but structurally open, and it is the hole that let the mis-scoped signal-provenance misreport ship green. - Anchored the 94%/6% product-vs-infra split with its query, date and partitioning predicate (336 vs 21 of 357 red rows), and flagged it as a snapshot the polarity argument does not depend on. Softened the unreproducible "~20% of the matrix" to the derivable bounds. - Version-qualified the STATUS_LIST_FIELDS omission-vs-null claim to the pinned PocketBase 0.22.21 and recorded the upgrade hazard: PB >=0.23's PublicExport isVisible/GetHidden path lets a field-level hidden flag omit signal independently of our projection, reintroducing the ambiguity the provenance flag exists to avoid. INTEGRATION RECONCILE (folded in when this was cherry-picked on top of the signalKnown-scope fix). Three conflicts in useLiveStatus.ts, all row/page-count prose, resolved to the measured 3082-row / 7-page anchor and to this commit's corrected INITIAL_FANOUT_BATCH note (the "page 7 ends its own wave" claim is only true at an ODD page count, which the superseded text asserted as an algorithm property). Additionally, every doc reference to `RawRung.signalKnown` — a field the scope fix DELETED — was rewritten to name the surviving red-row-scoped `FamilyFold.redSignalKnown`, so no comment describes a flag that no longer exists: live-status.ts STATUS_LIST_FIELDS doc (3 sites), useLiveStatus.ts coverage-gap note, api-matrix-equivalence.test.ts (2 sites). Comments only - no logic, no reformatting, no reordering. |
||
|
Jordan Ritter
|
3dea9f6424 |
test(showcase): make INV7 chip/strip coherence sound (scope + U8 stale exemption)
INV7 (added with the cold-load gray-masks-red fix) was unsound in two ways that only failed to bite by fixture accident: 1. WRONG QUANTIFICATION. Its doc says "every CONTRIBUTING red row", but it filtered over every row in `live.values()` — a whole-matrix map. Any red row belonging to a different column or feature could fail the invariant for a cell it contributes nothing to. Now quantified over the exact keyspace `buildCellModel` collects (`contributingKeys`). 2. MISSING isStaleCell EXEMPTION. `buildCellModel` deliberately force-grays an all-stale cell on every path (agent, starter, null-feature), folding ANY stale colour — red included — to the "re-sweep pending" gray. That gray is a recency claim about the cell, not an infra attribution for the red pill, so an all-stale cell containing a red row legitimately renders gray over a red pill. INV7 failed it. Exempted, with the reasoning inline. Both holes were latent: the current `stale-*` fixture variants are green/degraded, and the matrix has no cross-cell rows, so nothing hit them — the next fixture added would have tripped a false positive. Also adds an anti-vacuity guard (the narrowed quantification must still find at least one contributing red row whenever the strip reads red, so a future drift between `contributingKeys` and the engine's keyspace fails loudly instead of silently passing everything), and fixes the synthetic contribution helper `c()`, which set `freshestAgeMs: 0` for `ABSENT` — a shape `classifyRung` never produces (it returns `null`: an absent rung has no observation to age). New tests cover both false positives, a control, and four teeth cases proving INV7 still fails a genuine non-stale gray-chip-over-red-pill. |
||
|
Jordan Ritter
|
f5178765fc |
fix(showcase): bound the dashboard's supplemental signal fetch and stop its failure blanking the matrix
PR #6156 widened the browser's supplemental `signal` fetch from "comm-error aggregate rows only" to "comm-error aggregates ∪ every non-green row, all dimensions", and in doing so removed the early return that had bounded the work. What was left was a `for(;;)` with no page cap, unprojected, awaited before any row is returned — so it degrades exactly when the dashboard matters most. Measured against production: nominally 357 of 3082 rows (one page, 429 KB), but under a full-column outage the non-green set trends toward the whole collection — 7 pages, 1.73 MB — all of it on the critical path to first paint. Against a server that never returns a short page it does not terminate at all. Worse than slow: a supplemental rejection propagated through fetchInitial → connect() → the retry chain → `setRows([])` + `status: "error"`. A failure of an ENRICHMENT fetch blanked the ENTIRE dashboard, which directly contradicts the fail-safe rationale the PR was written to establish. (It also leaked an unhandled rejection per attempt, because the promise rejects while the bulk pages are still in flight and nothing is awaiting it yet.) Three changes, all on the dashboard hook: PAGE CAP. Both initial-fetch loops now share one bounded implementation (`fetchStatusPages`) and one cap, `MAX_INITIAL_FETCH_PAGES = 20`. Sized as a runaway guard rather than a budget: ~3x the real collection, so it never truncates a legitimate read including the degenerate all-non-green case. The supplemental loop also picks up the wave-concurrency and short-page merge guards the bulk loop already had and it did not, which is what made the parity structural instead of a promise two comment blocks made to each other. On-cap behavior is deliberately ASYMMETRIC: supplemental truncation DEGRADES (warn, use what arrived — the unreached rows keep their signal-less bulk copy, which classifyRung paints RED), while bulk truncation THROWS, because rows the bulk never read are whole cells that would render as no-data GRAY, i.e. exactly the masked-failure polarity this PR exists to remove. NON-FATAL ENRICHMENT. The rejection handler is attached at the fetch site, so a supplemental failure yields an empty row set, the merge degrades to "bulk rows, unenriched", and the dashboard RENDERS. Reds stay red via the classifier's fail-safe polarity; the residual cost is that a comm-error overlay on a green aggregate waits for its next SSE delta — the pre-CF7-F3-#1 behavior for the duration of one failed request, instead of a blank matrix. Attaching at creation also removes the unhandled rejection. PROJECTION. The supplemental fetch now sends `STATUS_LIST_FIELDS + signal` instead of no projection. That is exactly the declared StatusRow shape, derived from the drift-guarded constant so the two cannot diverge, which drops the columns PocketBase adds and nothing reads (collectionId, collectionName, created, updated, state_written_at, written_by): 429,085 → 362,982 bytes measured on the real non-green set, ~260 KB at incident scale. Deliberately NOT narrower — the merge treats a supplemental row as a COMPLETE row (replaces its bulk twin wholesale, appends one the bulk snapshot missed), so `key,observed_at,signal` would have saved another 58 KB by forcing a graft onto a row of a different vintage, which is the chimera `supplementalRowIsOlder` exists to prevent. It stays on the first-paint path. Enriching after first render would have removed it from the critical path, but a non-green row's chip can move once its signal lands (red → gray for a positively-infra red, red → amber for a soft-class first strike), and a cold-load verdict that corrects itself a beat later is the class of symptom this PR set out to eliminate. With the fetch bounded, wave-concurrent and projected it runs inside the bulk fetch's own round-trip budget, so gating first paint costs nothing on the happy path. Tests are REAL-SDK over real sockets, with fake endpoints that actually evaluate `filter`/`fields`/`sort` (new `__tests__/pb-query-eval.ts`) so a response is a function of the query the hook sent rather than of fixture layout. Verified by mutation: restoring the unbounded loop, the fatal rejection, or the missing projection each turns its own test red. |
||
|
Jordan Ritter
|
ebd7ac681d |
test(showcase): restore PB filter/fields fidelity to the autocancel fake servers
PR #6156 silently invalidated the dropped-tail pagination guards added by PR #4504. Regression in test COVERAGE, not production code — which is why CI stayed green. The three fake PB servers in the autocancel suite ignored the `filter` query param (and `fields`). #6156 made the supplemental `signal` fetch UNCONDITIONAL (dropping the `commAggregateFilter === null` early return and widening the filter to `state != "green"`), so a filter-ignoring server answered that now-always-issued fetch with the ENTIRE dataset — and `fetchInitial`'s leftover-append then REPAIRED whatever rows the bulk pagination dropped. Mutating the merge to drop the entire short page kept all three tests GREEN. The servers now answer a list request the way real PocketBase does: filter, then sort, then the page slice, then the `fields` projection, with the COUNT(*) envelope omitted only when `skipTotal` is set. The filter evaluator covers exactly the grammar `useLiveStatus` emits and THROWS on anything else (answered as 400) — a silently-unparsed clause would degrade to "match everything", i.e. straight back into the masking bug. Real-SDK quoting is matched (single-quoted string params, bare numbers/booleans, verified against the installed SDK); the sibling suite's double-quoting `pb.filter` stub flaw is deliberately NOT copied forward, and a wire assertion pins the convention. The supplemental fetch is discriminated by the ABSENCE of a `fields` projection — the sibling `useLiveStatus.test.tsx` convention, reused rather than reinvented. This replaces a `filter.includes("state !=")` substring check that the union filter had already made unreliable. Broken assertions repaired: - over-fetch instrumentation no longer conflates the supplemental serial walk with the bulk fan-out (separate page arrays), and the split is now asserted rather than assumed - the FIRST-wave-element server serves POISON rows on the page after the short page. Real PB pagination is monotonic, so that page is empty and appending it changes nothing — which is exactly why deleting the merge's boundary logic left the test green. Poison makes a merge that runs past the boundary observable - `.some(r => r.id === "f0699")` replaced with `rows.at(-1)?.id`, the LAST-row boundary check the comment already claimed, plus explicit poison-absence - the mid-wave suite no longer claims a boundary property its fixture cannot measure (its short page is the wave's last element); name and comment now state what it does measure, and it gained a real tail-boundary assertion Mutation red-green, both directions. Drop-the-entire-short-page: PASSED 3/3 before, now FAILS 3/3 (1300 to 1000, 1250 to 1000, 700 to 500). Boundary-logic-removed: PASSED before, now FAILS on the suite that claims it (700 to 1000). Reversed-merge-order (same row set): PASSED before, now FAILS on both new `.at(-1)` assertions. Mutations reverted: 3/3 pass. Test-only change — no production code touched. |
||
|
Jordan Ritter
|
2a68282d08 |
fix(showcase): scope the infra-red gray precondition to the RED rows
`classifyRung`'s U7 gray branch reads the `signal` blobs of a family's RED rows and nothing else, but the precondition qualifying what it read — "was `signal` actually delivered, or projected away?" — was `raw.signalKnown`, a boolean AND over EVERY present row of the family. A predicate about red-row evidence was answerable "no" by a row the branch never looks at. That broke exactly the case the precondition was written to protect. The dashboard's supplemental cold-load fetch restores `signal` for `state != "green"` only (by design — the classifier needs it solely in the red branch), so in a MIXED-state family the red rows arrive WITH attribution and the green siblings arrive WITHOUT it. The family-wide flag therefore read `false` precisely when it should have read `true`: D4 = green `chat` + infra-red `tools` skipped the gray branch, fell through first-strike (its only red is infra, so `maxNonInfraRedFailCount` is null), and landed on FAIL_FRESH — the browser painting RED a cell that `/api/matrix`, which always has the full `signal`, reports as gray. That is the §11.4 api-vs-render drift the read-model exists to forbid, and it made INFRA_RED_FRESH effectively unreachable in the browser for every multi-row family: D4, and any multi-pill D5/D6 (whose per-cell `d5:<slug>/<pill>` keys the supplemental filter's clause-1 `key !~ "%/%"` excludes). Only single-key D3 was unaffected — there the family's one row IS the red row and the two scopes coincide, which is why every fixture missed it. The fix derives the flag where the rows are, at the scope that consumes it: `foldFamily` now reports `redSignalKnown` over the RED rows only, alongside the red-scoped `hasNonInfraRed` / `maxNonInfraRedFailCount` / `allRedSoftClass` it already computed. `RawRung.signalKnown` and its `gatherRows` plumbing are removed rather than left dead — a family-scoped flag named `signalKnown` sitting next to a red-scoped consumer is the trap that produced this bug. `anyExpectedMissing` stays family-scoped, correctly: a missing sub-key IS a family property and it gates a family-scoped verdict. `redSignalKnown` is today IMPLIED by `!hasNonInfraRed`, since `signalHasInfraErrorClass(undefined)` is false and a stripped red row forces `hasNonInfraRed` on its own. It is kept explicit anyway — this is the masks-real-red guard and it must not rest on a coincidental property of a helper two modules away — and pinned directly on the fold so the scope cannot regress silently. Coverage. The previous tests named `signalKnown: false` but were insensitive to it: `raw.signalKnown &&` could be deleted outright and all 181 harness cell-model tests plus 58 dashboard equivalence tests stayed green, because every fixture stripped `signal` from ALL rows of the target rung and never from a green sibling. The new tests construct the mixed-state family that ships the bug, on the real derivation surface (`buildCellModel`, not a hand-passed flag): D4 chip, multi-key D5 chip, and the D6 `d6Effective` badge — D6 is the soft-parity top so its chip is amber either way, but its badge drifts red-vs-null. Each asserts browser == server == `/api/matrix`. `api-matrix-equivalence`'s `coldLoadChip === serverChip` guard is extended off single-key D3 to the D4 family. Five `foldFamily` tests pin the flag's scope directly, and the fail-safe polarity keeps its negative control: a red row whose OWN `signal` was stripped still renders RED. |
||
|
Jordan Ritter
|
510bb88553 |
fix(showcase): use each integration's own name in demo page titles
14 integrations' `src/app/demos/layout.tsx` hardcoded "LangChain - Python" in `generateMetadata` — a copy-paste leftover from langgraph-python, which the file was cloned from. Every `/demos/*` page in mastra, strands, ag2, agno and 10 others rendered `<title>LangChain - Python</title>`. Each now uses the display name from its own `manifest.yaml` `name:` field, matching the convention the already-correct integrations use (langgraph-typescript -> "LangGraph (TypeScript)", strands-typescript -> "AWS Strands (TypeScript)"). langgraph-python itself is included: its manifest name and root layout both say "LangGraph (Python)", so "LangChain - Python" (the legacy Notion partner-column label) was stale there too. |
||
|
Jordan Ritter
|
e1c801313b |
fix(showcase): bundle multimodal sample assets for agno and mastra
agno and mastra were the only 2 of 20 integrations with no
`public/demo-files/` directory ever committed. Their Attachments demo
fetches `/demo-files/sample.png` before building the message, so the
demo failed at the fetch with:
[multimodal-demo] sample-attachment send failed Error: Could not
fetch sample "sample.png" — HTTP 404. Is the file bundled under
public/demo-files/sample.png?
Nothing was ever sent to the agent — `userMsgCount: 0` — so the D5
probe timed out on turn 1. Both integrations already have working
vision-capable multimodal agents and correct aimock fixtures; this was
purely a missing-asset/packaging bug.
Copies sample.png / sample.pdf / README.md byte-for-byte from
langgraph-python (the canonical source) into each integration's own
`public/demo-files/` — a per-integration duplicated directory, not part
of the `shared/` symlink mechanism, which is exactly how two
integrations came to be missing it.
The assets are committed as plain Git LFS pointers, governed by the
repo-root `*.png` / `*.pdf` LFS rules, with no per-integration
`.gitattributes` override. LFS tracking stays uniform across the
fleet; the deploy-time problem (integration image builds do not fetch
LFS) is fixed in the build, not by carving these files out of LFS.
The pointer OIDs are the ones already stored for the identical assets
in the other integrations, so the backing LFS objects exist server-side
and are not dangling:
sample.png sha256:01aa5681de99461247543e9215c1e4da3242e26b2bee11593fcdbe209672d973 10083
sample.pdf sha256:3da2afae36a1a81fd2c02f15e54bfc38b6c22e41655c31a5b54ff1e0e3daab41 2486
NOTE: this change does not make agno/mastra pass on its own. It
requires the build-side LFS fetch fix to land first or together.
|
||
|
Jordan Ritter
|
233b509928 |
fix(showcase/mastra): restore shared-tools symlink, shrink erosion ratchet
`showcase/integrations/mastra/shared-tools` was a real committed directory where a symlink into the single source of truth belongs, violating the single-source symlink mechanism documented in showcase/AGENTS.md. Root cause: commit |
||
|
Jordan Ritter
|
006a42aeb6 |
test(showcase): track crewai-crews wired-count bound after multimodal unsupported
`generate-catalog.test.ts` pinned crewai-crews' wired-cell count at >= 30. Moving `multimodal` from `features` to `not_supported_features` reclassifies that cell from `wired` to `unsupported`, so the real count is 29 and CI's `showcase/scripts` vitest failed. The bound is a snapshot floor, not an invariant: the test's own comment says the partial parity tier only requires intersection >= 3 with the reference's wired set. 29 still clears that by a wide margin, so the floor tracks the manifest. Updated to 29 with a note recording which cell moved and why, so the next reader does not read the decrement as a regression. |
||
|
Jordan Ritter
|
4a40604aca |
chore(showcase): mark multimodal unsupported for llamaindex and crewai-crews
Attachments (`multimodal`) has never worked on either integration, and both
manifests were over-claiming it under `features`, so the D5/D6 probe correctly
reported red — a red chip that read as "this regressed" when the truth is "this
was never built".
Use the repo's existing unsupported mechanism — `not_supported_features` in the
integration manifest, which `determineCellStatus`
(showcase/harness/src/shared/catalog/catalog-flatten.ts:239) folds to
`status: "unsupported"` -> `isSupported: false` -> the `UNSUPPORTED` CellModel
(gray, `supported: false`, 🚫). Both cells now render as genuinely unsupported:
not green (no false claim of working) and not red (no false claim of
regression). Each entry carries an inline note recording WHY, so a future reader
does not mistake it for an oversight.
- llamaindex: upstream gap in `llama-index-protocols-ag-ui==0.2.2`
(`utils.py:82-85` passes AG-UI content parts straight into `ChatMessage`,
which rejects them on the `block_type` discriminator).
- crewai-crews: never implemented on our side — `src/agent_server.py` registers
no `/multimodal` route, so the demo route aliases the generic crew and fails.
Also corrects the llamaindex `multimodal_agent.py` docstring, which asserted a
normalization the upstream router does not perform.
Demo entries stay wired on both, matching every other `not_supported_features`
entry in these manifests.
|
||
|
Jordan Ritter
|
49e7b2174c |
fix(showcase/ms-agent-python): keep the user's prompt on the multimodal PDF turn
The multimodal PDF turn dropped the user's question out of the final outbound user message, so the model was handed a document dump with no question attached. Against aimock's strict mode that surfaced as `503 no_fixture_match` on turn 2 (turn 1, the image, passed); against a real LLM it would have silently answered the wrong thing. Root cause is a serialisation mismatch, not a fixture gap. `agent_framework_openai._chat_completion_client._prepare_message_for_openai` emits ONE OpenAI message per `Content` — it builds a fresh `args` dict on every iteration of its content loop. `_PdfFlattenChatMiddleware` appended the flattened `[Attached document]` text as a SECOND text `Content` beside the prompt, so one logical user turn serialised to two consecutive user messages: prompt-only, then document-only. Anything reading "the current user turn" off the tail of the list saw only the document. Merge the flattened document INTO the message's existing prompt text content instead, so the turn stays a single text content and serialises to a single user message reading `"<prompt>\n[Attached document]\n<body>"`. langgraph-python's equivalent agent is green precisely because LangChain keeps multiple text parts inside one message rather than splitting them. The merge copies the prompt `Content` rather than mutating it: the middleware restores the original `contents` list after the model call, and that restore only undoes the list swap — an in-place mutation would leak the raw PDF body into the AG-UI MESSAGES_SNAPSHOT and render it in the user's chat bubble. Also dedupe identical flattened blocks. The page's `LegacyConverterShim` appends a legacy `binary` mirror alongside every modern attachment part, so the same PDF arrives twice and its body was being sent to the model twice. No fixture change: the existing `userMessage` match key is correct and is what the corrected request shape satisfies. |
||
|
Jordan Ritter
|
3198b6bc2e |
fix(showcase): re-fetch signal for non-green rows on dashboard cold load
The dashboard's bulk initial fetch projects the heavy `signal` blob away (`STATUS_LIST_FIELDS`) for a real payload win — measured at ~70% of the response, 371 KB → 113 KB per 500-row page. A supplemental fetch then restored `signal` for the rows that are read at render time, but it only ever covered the comm-error AGGREGATE rows: dimensions `d6`/`d4`/`e2e-demos`/ `d5-single-pill-e2e`, narrowed by `key !~ "%/%"`. `classifyRung` also reads `signal`, to tell an INFRA red from a PRODUCT red, and the rows it needs were excluded twice over: they are mostly dimension `d5`/`e2e`/`health` (absent from FLEET_COMM_AGGREGATE_DIMENSIONS entirely) AND they are per-cell `<dim>:<slug>/<featureId>` keys that `key !~ "%/%"` filters out. So on every cold load those rungs arrived with no attribution at all, and self-corrected only when the probe's next sweep rewrote that specific row and the SSE delta redelivered it with `signal` — up to a full sweep interval (~29 min observed, ~14 min mean), again on every reload. The supplemental filter becomes a UNION: the existing comm-error clause OR `state != "green"`. Scoping by STATE is what keeps it cheap — `classifyRung` consults `signal` only in its red branch, so green rows never need it, and in production the non-green rows are ~360 of ~3100 (~430 KB, about a third of what shipping `signal` on every row would cost) with no schema change. `!= "green"` rather than an explicit red/degraded list also catches an out-of-vocabulary state, which `rankOfState` ranks WORST — the rows that matter most can never fall outside the fetch. Widening the EXISTING fetch rather than adding a second one reuses its freshness guard (`supplementalRowIsOlder`) and chimera-avoidance merge verbatim, keeps first paint to one extra request, and keeps the fail-loud retry posture. A dimension-scoped hook still narrows the comm-error clause to the matched literal, and drops it entirely when the scope sits outside the aggregate set — but the non-green clause always applies, so such a scope no longer skips the supplemental fetch altogether. Its test is updated accordingly. Also fixes two test doubles that identified the supplemental request by `filter` containing `key !~`. That marker now disappears for an out-of-set dimension scope, so both would silently misclassify a supplemental request as a BULK page and corrupt the fan-out instrumentation. They key off the absence of a `fields` projection instead — the property that structurally distinguishes the two requests and the whole reason this one exists. |
||
|
Jordan Ritter
|
3549f3e370 |
fix(showcase): a red rung with unknown infra-ness must render RED, not gray
`classifyRung` opened its red branch with `if (!raw.signalKnown) return
NO_DATA` — so whenever the dashboard's bulk fetch had projected the `signal`
blob away, a genuinely-failing rung was classified "no data" and painted the
muted gray "nothing to see here" chip, while the depth strip beside it
correctly rendered `1P ✗`.
That polarity is backwards twice over:
- `signalKnown === false` never meant "known to have no signal". PocketBase
omits the `signal` key ONLY under a `fields=` projection; a row that
genuinely has no signal arrives as `null`, and `null !== undefined`. So the
flag means "this row came from a projected fetch" — a PENDING attribution,
the absence of evidence about WHY the rung failed. It was never evidence
that the failure was infra.
- Measured against production, ~94% of reds are product-class and only ~6%
are infra-class. Graying every unattributed red lost 94 real failures to
suppress 6 false alarms — and a masked red is never investigated, while a
false alarm is looked at once and closed.
Graying a red now requires POSITIVE infra evidence: `hasNonInfraRed` is false
only when every contributing red row's blob actually carries an
INFRA_ERROR_CLASSES attribution, and `signalKnown` is retained as an explicit
second precondition so a missing blob can never be read as an infra
attribution. The residual cost is over-reporting, which is the direction a
health dashboard has to fail in.
Tests: the harness `§7 I5` case and the dashboard's api/render equivalence
case both ASSERTED the buggy gray as intended behaviour — which is why CI
never caught this. Both are inverted, with the reasoning recorded inline.
Four golden-master fixtures (`pos-d{3,4,5,6}-red-signal-unknown`) are
re-frozen; the regenerated baseline diff touches those 4 of 56 and nothing
else.
Adds coherence invariant INV7: a gray chip sitting above a red depth pill
requires positive infra evidence on every red row. INV1-INV6 only relate
chipColor to the other CHIP-side outputs and never inspect the d3/d4/d5/d6
pills, so the engine could return one object saying both `d5.status === "red"`
and `chipColor === "gray"` with every invariant intact. INV7 fails on the old
polarity and closes the gap in the only safe direction — by fixing the chip
upward, never by muting the strip.
|
||
|
David McKay
|
188bc6cb79 |
docs(react-native): expand into a production guide
Rewrites the React Native page into a production guide (Metro, polyfills,
provider options, runtime/model wiring, device + bench connectivity, frontend
tools, run lifecycle, voice), and corrects the source-backed inaccuracies found
in review. Every claim below was re-verified against the package source on main
(@copilotkit/react-native 1.63.2), not inferred.
Import surfaces (was "What's included" + "Headless imports")
- Documents all THREE entry points separately with the native peers each forces
Metro to resolve: /headless (none), root (expo-document-picker,
expo-file-system via useAttachments), /components (@gorhom/bottom-sheet,
react-native-streamdown).
- Removes the claim that the root barrel pulls @gorhom/bottom-sheet. It does
not -- the only import is src/components/CopilotModal.tsx, reachable solely
from /components. (The package's own source comments state this incorrectly;
that is what the previous draft was written from.)
- Warns that root's CopilotChat/CopilotModal are HEADLESS wrappers rendering
only children, while the same-named /components exports are the rendered
chat -- a silent blank-screen trap.
- The quickstart now imports from /headless throughout, matching its own advice
and keeping readers out of the release-bundle failure the page warns about.
- Corrects "the provider re-exports hooks" to the package, enumerates the
shared hooks, and drops "behave identically to the web SDK": React Native
ships its OWN useRenderTool (requires parameters, accepts handler, returns
ReactElement | null, no wildcard) and omits the web-only rendering hooks.
Metro
- Corrects the cause: jose is not a JWT/license dependency. It arrives via
telemetry (@copilotkit/shared -> @segment/analytics-node -> jose).
- Replaces the global unstable_conditionNames array with resolveRequest scoped
to jose (Metro's own documented recipe). unstable_conditionNames is an
UNORDERED SET of asserted conditions -- target priority comes from each
package's own exports key order, so reordering that array does nothing. The
previous config also applied browser globally and dropped Metro's default
react-native condition.
- Fixes version bounds: package exports arrived opt-in in RN 0.72 / Metro
0.76.1 and is default-on from Metro 0.82 / RN 0.79 (not "0.70+").
Polyfills
- Makes the quickstart canonical instead of correcting it later. Drops
"optional belt-and-suspenders".
- Documents the crypto ordering as a HARD requirement: CopilotKit's polyfill
and react-native-get-random-values are both first-writer-wins, so any
CopilotKit import evaluated first permanently locks in the non-cryptographic
Math.random fallback and silently no-ops the secure library.
Provider / runtime
- headers: a function form is evaluated when the PROVIDER RENDERS, memoized,
and pushed to the core -- never per request. Adds rotating-token guidance
(drive from state, or call copilotkit.setHeaders on refresh).
- cors: corrects the causal claim. CORS is browser-enforced; React Native's
native stack sends no Origin and never consults Access-Control-Allow-Origin,
so cors:true is irrelevant to native reachability (it matters for Expo Web).
Device connectivity
- Labels the adb reverse row Android-only and adds an iOS device row.
- Adds the missing `npx expo install expo-build-properties` step.
- Adds the iOS path, which was absent: ATS applies to debug and release alike,
expo-build-properties has no iOS ATS option, and since iOS 17 ATS rejects
raw IPs unless listed in NSExceptionDomains -- plus
NSLocalNetworkUsageDescription for the local-network permission gate. Notes
why NSAllowsLocalNetworking alone is insufficient.
Frontend tools
- Replaces the incorrect "useFrontendTool can carry a render function drawn as
the tool runs". The rendered RN chat reads a registry populated only by RN's
useRenderTool; a render passed to useFrontendTool is never drawn there and
yields the fallback "Called: <toolName>" stub.
Other
- Removes the useAgent({ threadId }) example: unsupported on main, does not
typecheck (TS2353), silently ignored. Recorded as a known limitation
pointing at #6141 instead, so this page no longer depends on that PR.
- Known limitations: drops the "No pre-built UI is required" positioning line;
corrects the markdown entry (the rendered chat DOES render markdown via
CopilotMarkdown/react-native-streamdown -- the old entry recommended the
wrong library); adds the threadId and web-only-hooks entries.
- Broadens the frontmatter description, which still promised only "get started".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Alem Tuzlak
|
d7ba17277c |
fix(showcase/built-in-agent): real beautiful-chat backend (verified vs real LLM) (#6150)
## Why Follow-up to #6106 / #6149. Prod testing (real LLM) revealed the flagship **beautiful-chat** cell was a **stub**: it was wired to the generic all-tools agent on a bare route, so the model reported `query_data` / Excalidraw / gen-UI tools "not available." It only passed local aimock D6 because fixtures scripted the tool calls — **aimock can't validate real-LLM tool availability**, which is why this slipped through. ## Fix Port the real beautiful-chat backend, mirroring `langgraph-python/src/agents/beautiful_chat.py`: - **New `createBeautifulChatAgent()`** — server tools `query_data` (sales data), `manage_todos`, `get_todos`, `search_flights` (A2UI fixed-schema flight cards); declares the runtime-injected tools (`render_a2ui`, `generateSandboxedUi`, Excalidraw MCP) so the model can actually call them; `manage_todos` → `STATE_DELTA /todos` for the app-mode board. Model `gpt-5.4`. - **Route** configures the combined runtime like LGP: `openGenerativeUI: true`, `a2ui.injectA2UITool: true` (catalog `copilotkit://app-dashboard-catalog`), `mcpApps` (Excalidraw). - Shared factory defaults unchanged (new opt-in factory). ## Verified against a REAL OpenAI LLM (not aimock) Drove a headless browser against the demo pointed at real OpenAI — all 6 flows fire the tool + render the component: | Flow | Result | |---|---| | Pie chart (`query_data`→`pieChart`) | ✅ real data | | Bar chart (`query_data`→`barChart`) | ✅ real data | | Calculator (`generateSandboxedUi`) | ✅ sandboxed iframe | | Flights (`search_flights` A2UI) | ✅ 2 FlightCards | | Excalidraw (`create_view` MCP) | ✅ MCP iframe | | Todos (`enableAppMode`→`manage_todos`→`get_todos`) | ✅ board populated | Zero "not available" apologies. ## Follow-up (tracked, not in this PR yet) Auditing the other generic-agent-backed demos against a real LLM for the same class of under-wiring; fixes will be added here or in a follow-up. |
||
|
Ben Taylor
|
a00592f115 |
docs(channels): add Channels SDK documentation (#6031)
## What does this PR do?
Adds documentation for the **`@copilotkit/channels` SDK** — running a
CopilotKit agent as a bot inside Slack, Teams, WhatsApp, Telegram, and
Discord. New `---Channels---` nav group under `showcase/shell-docs`.
**Guides**
- Overview (agent + JSX + channel, end to end), Quickstart (TanStack
factory agent on the Intelligence adapter)
- UI Library (JSX component rendering, callbacks, component
registration, `{ raw }` escape hatch)
- Interactive flows: Human-in-the-loop (+ agent-initiated
`onInterrupt`), Reactions (+ emoji helpers), Modals
- Commands & global reactions, Files & multimodality, MCP (agent-side),
Channel configuration, Tools & context
- Platforms: overview of what Intelligence handles for you, plus
per-adapter pages (Teams / WhatsApp / Slack / Telegram / Discord) with
built-in tools/context
**API reference** (signatures pulled from the installed `0.2.1` type
definitions): Channel, Thread, JSX callbacks.
All examples use the Intelligence adapter except the platform-specific
pages, which use each platform's own adapter. Every internal link
resolves; MDX validated.
## Related PRs and Issues
- N/A
## Checklist
- [x] I have read the Contribution Guide
- [x] If the PR changes or adds functionality, I have updated the
relevant documentation
- [x] "Allow edits by maintainers" is checked
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Benjamin Taylor
|
4f3c820908 |
docs(channels): add required agents:{} to runtime snippets + keep-alive note (adversarial review)
Adversarial pass caught that every new CopilotRuntime({...}) snippet omitted the
required non-optional `agents` field (TS2741 on copy-paste) — added `agents: {}`
matching examples/slack/app/managed.ts. Also noted that the managed index/mcp
snippets end at ready() with no server, so a reader's process would exit — added a
keep-alive pointer to the Quickstart's full listener mount.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Benjamin Taylor
|
6d1205d0f5 |
docs(channels): reconcile with Plan C — runtime-driven lifecycle, Intelligence key required
Channels run only through the Intelligence runtime now (PR #6145): public channel.start()/stop()/addAdapter() are removed. Updated the run path to new CopilotRuntime({ intelligence, channels }) + handler/listener.channels.ready()/stop() across quickstart, index, mcp, configuration, the 5 platform pages, and the channel reference; reframed the managed-vs-direct table (both require an Intelligence key — the difference is who holds the platform credentials); name is now required (the runtime keys lifecycle by it); surfaced the Intelligence-key (free tier) prerequisite. Standardized the managed pattern on zero-adapter createChannel({ name }) (runtime attaches the managed transport), matching examples/slack/app/managed.ts. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Lukas Moschitz
|
9ab49609f0 |
fix(showcase): make langgraph-fastapi get_stock_price scriptable
tool-rendering-custom-catchall renders the ticker quote in the wildcard card. fastapi's get_stock_price(ticker) had no scriptable args, so when the aimock fixture emitted price_usd/change_pct the @tool silently dropped the unknown kwargs and returned random numbers instead of the scripted quote — the card showed nondeterministic values, diverging from the north star. Port LGP's scriptable signature: get_stock_price(ticker, price_usd=None, change_pct=None) — echoes scripted values verbatim when supplied, random when omitted (backward-compatible with the other tool-rendering pills). Behavior byte-aligned to langgraph-python. D6 tool-rendering-custom-catchall green (two real ~8s runs); shared-backend siblings tool-rendering, tool-rendering-default-catchall, headless-complete all re-verified green. |
||
|
Lukas Moschitz
|
797d30ca63 |
fix(showcase): port langgraph-fastapi gen-ui-headless-complete to parity
Two compounding defects kept this cell red:
1. Backend missing get_revenue_chart. headless_complete.py never registered
the revenue-chart tool, so the chart-turn never emitted a tool call and the
card never mounted. Port LGP's get_revenue_chart tool + its system-prompt
routing rule (tool return shape byte-identical to LGP).
2. Stale turnIndex fixtures caused turn-2 and turn-4 loops to the recursion
limit. Align to LGP's canonical shape:
- tool-rendering.json (shared): AAPL first-leg turnIndex:0 -> hasToolResult:false
so it stops re-firing at turnIndex>=2 in the multi-pill thread.
- headless-complete.json: rewrite to LGP's structure (narration/toolCallId
fixtures first, tool-call legs after with userMessage+context only, no
stale turnIndex); drop the divergent fastapi-only highlight fixtures,
subsumed by the broader substring fixture.
D6 langgraph-fastapi:gen-ui-headless-complete now green (two real ~29s runs);
tool-rendering re-verified green (shared-fixture no regression).
|
||
|
Lukas Moschitz
|
243acc7771 |
fix(showcase): add langgraph-fastapi gen-ui-agent graph + recursion limit
gen-ui-agent had no dedicated backend graph: langgraph.json lacked a
gen_ui_agent entry and route.ts routed the name through the neutral-assistant
loop to sample_agent, which has no steps state or set_steps tool, so the
progress card never mounted (agent hit the default recursion limit of 25).
- Port LGP's gen_ui_agent.py (byte-identical) and register it in langgraph.json.
- route.ts: bind gen-ui-agent to createAgent("gen_ui_agent") and bake
assistantConfig.recursion_limit (default 100) into every LangGraphAgent —
the graph's Python with_config isn't visible to the server runs API, so the
multi-step set_steps walk overran 25. Mirrors langgraph-python.
- aimock: regenerate d6/gen-ui-agent.json from LGP (adds chunkSize:9999 on the
24 tool-call fixtures) and narrow the over-broad d4 chat.json "summarize" key
to "Summarize the sales pipeline" so it stops substring-shadowing the
competitor set_steps chain (and other summarize prompts). Matches LGP.
D6 langgraph-fastapi:gen-ui-agent now green (two real ~14s runs);
agent-config re-verified green (no regression).
|
||
|
Alem Tuzlak
|
2f2a703add |
fix(showcase/built-in-agent): real-LLM backend fixes from prod audit
Fixes the demos a real-LLM audit found broken (aimock had masked them). Each
verified against real OpenAI (plain next dev / next build; turbopack dev is
broken on Windows — unrelated):
- shared-state-read / shared-state-read-write: seed agent state on [agent] deps
(not []). useAgent returns a provisional agent during /info sync; the real
runtime-synced agent swaps in as a new ref, so a []-deps seed effect never
re-ran and runAgent serialized empty state. NOTE: diverges from LGP's
byte-identical frontend — BIA's stateless in-process backend needs re-seeding;
LGP masks it via server-side state. Documented in PARITY_NOTES.
- set_notes -> STATE_DELTA add /notes in tanstack-factory convertStream (mirrors
set_steps), so shared-state-read-write notes render.
- declarative-gen-ui / a2ui-recovery: drop response_format:json_object (the
Responses-API openaiText ignores it and returned empty) — enforce JSON via
system prompt + stripJsonFences, like the byoc factories. Surfaces now paint.
- reasoning-custom / -default / tool-rendering-reasoning-chain: switch to the
Responses API with reasoning summaries (reasoning:{effort:high,summary:auto})
+ custom STEP_* -> REASONING_MESSAGE_* converter; real OpenAI chat-completions
never streams reasoning_content. Reasoning blocks now render.
- cvdiag/*: drop .js import extensions so /api/copilotkit resolves under dev
bundlers.
- PARITY_NOTES: document all fixes + that auth/voice 500 is next-dev-turbopack
only (verified working under next build/start in prod).
|
||
|
Lukas Moschitz
|
71f44d46f7 |
fix(showcase): port langgraph-fastapi agent-config to v2 create_agent
The fastapi agent_config_agent still used the v1 StateGraph pattern reading tone/expertise/responseLength from RunnableConfig[configurable][properties]. The frontend (identical to LGP) publishes those knobs via the v2 useAgentContext hook and the route uses a plain LangGraphAgent, so the old graph received nothing on configurable and the run errored (turn never completed, D6 red). Port the backend to LGP's v2 shape: create_agent + CopilotKitMiddleware with a single static system prompt that reads the injected context entry. Update the route comment to describe the useAgentContext path (runtime code unchanged: plain LangGraphAgent + AGENT_URL env fallback preserved). D6 langgraph-fastapi:agent-config now green (two real ~28s runs). |
||
|
Lukas Moschitz
|
e4bf77f4e9 |
fix(showcase): stop langgraph-fastapi open-gen-ui D6 loop (fixture shadowing)
open-gen-ui and open-gen-ui-advanced (both probes load gen-ui-open.json) hung
with done-signal-missing (~12 agent runs, 1.3MB tool-call payload each turn).
Cause: two files (gen-ui-open.json + gen-ui-tool-based.json) both defined leg-1
generateSandboxedUi tool-call fixtures for the SAME userMessages ('3D axis…',
'Inline expression evaluator') under the same context but different toolCallIds.
aimock pools every d6/<slug>/*.json into one substring-match set, and the
turnIndex:0 fixture in gen-ui-tool-based.json got served on follow-up turns too
(turnIndex-relaxed), so the toolCallId-anchored terminal never won → no
finish_reason:stop → loop.
Fix (fastapi fixtures only, aligned to GOTCHAS 'canonical fixtures have no
turnIndex' + toolCallId-terminal-before-userMessage-leg1):
- gen-ui-tool-based.json: strip spurious turnIndex:0 from leg-1 tool-call
fixtures; add chunkSize:9999 so the large args JSON-parse in one chunk. Now
the single source for all open-gen-ui pills.
- gen-ui-open.json: empty its fixtures (keep a documenting _note) to remove the
duplicate leg-1/terminal for the 2 probed pills.
- gen-ui-open-advanced.json: delete orphan (no probe declares it; both open
probes use gen-ui-open.json).
D6 green + stable (two ~4s runs each); regression-checked gen-ui-custom
(gen-ui-tool-based route) still green. NOTE: LGP + likely all integrations have
the same open-gen-ui loop (fleet-wide fixture-shadow bug) — no sibling was green.
|
||
|
Lukas Moschitz
|
1dde897af3 |
fix(showcase): stop langgraph-fastapi frontend-tools-async D6 loop
fixture[8] (userMessage:"project planning", turnIndex:0) returned a query_notes toolCall. aimock matches userMessage by substring, so "project planning" also matched the test prompt "Find my notes about project planning"; as a turnIndex:0 'scripted turn' it beat the correct terminal fixture[2] (toolCallId-keyed, content-only) on the follow-up turn, so the agent kept getting another tool call and looped ~30x, never emitting the done signal (D6 done-signal-missing). Remove turnIndex:0 from fixture[8] so it's a plain fallback that loses to the terminal fixture by registration order (matching the langgraph-typescript shape). D6 now green: two real ~6s runs (was a 120s timeout). Note: langgraph-python has the identical bug (still red) — same one-line fix applies there; tracked separately. |
||
|
Lukas Moschitz
|
cd8c94a492 |
feat(showcase): make fe-parity exception-aware via PARITY_NOTES allowlist
fe-parity.ts now reads a per-integration allowlist from integrations/<slug>/PARITY_NOTES.md (an <!-- fe-parity-allow --> block of 'src-relative-path | reason' lines). Listed files report as ALLOWED (sanctioned per-integration divergence), not DRIFT; the gate passes on zero UNSANCTIONED drift instead of zero diffs. This stops the byte-identical goal from being impossible for demos that legitimately must differ (e.g. a2ui-recovery's per-slug prompt, which has no aimock context routing). Add langgraph-fastapi/PARITY_NOTES.md documenting the a2ui-recovery frontend divergence with reasons. D6 remains the correctness gate; fe-parity is the 'where to look' map. |
||
|
Alem Tuzlak
|
a486015563 |
feat(showcase/built-in-agent): real beautiful-chat backend (query_data + A2UI + OGUI + MCP)
The flagship beautiful-chat cell was a stub (generic all-tools agent on a bare route) that only passed local aimock D6 because fixtures scripted the tool calls; against a real LLM (prod) it said query_data/Excalidraw/etc. 'not available'. Port the real backend mirroring langgraph-python's beautiful_chat: - New createBeautifulChatAgent() with server tools query_data (sales data), manage_todos, get_todos, search_flights (A2UI fixed-schema flight cards), and declaration of the runtime-injected tools (render_a2ui, generateSandboxedUi, Excalidraw MCP) so the model can actually call them; manage_todos result -> STATE_DELTA /todos for the app-mode board. Model gpt-5.4. - Route configures the combined runtime: openGenerativeUI:true, a2ui.injectA2UITool:true (catalog copilotkit://app-dashboard-catalog), mcpApps (Excalidraw). VERIFIED against a real OpenAI LLM (not aimock): all 6 flows fire the tool + render the component (pie/bar via query_data, calculator OGUI iframe, flight A2UI cards, Excalidraw MCP iframe, todo board). Shared factory defaults unchanged. Also gitignore .env/.env*.local (local real-key testing). |
||
|
Alem Tuzlak
|
eedea568a2 |
fix(showcase/built-in-agent): use gpt-5.4 (staging returns empty on gpt-5.5)
Staging D4 chat probe returned 'empty assistant response' for every cell, capping all cells at D3. Root cause: the factories used gpt-5.5 (agentic-chat + generic createBuiltInAgent) and leftover gpt-4o/gpt-4o-mini (a2ui, byoc, mcp-apps, ogui, subagents). gpt-5.5 is unique to built-in-agent; every other integration — including the LGP reference that works on staging — uses gpt-5.4, which staging's OpenAI account serves. aimock ignores the model id, so this was invisible locally. Switch all built-in-agent openaiText() calls to gpt-5.4 (matches LGP, unblocks staging). reasoning-factory keeps gpt-5.2 (reasoning-capable, its demos are NSF-quarantined). Note: deviates from the general gpt-5.5 house rule because this is an LGP-parity integration and staging only serves gpt-5.4. |
||
|
Lukas Moschitz
|
eaa49e55b3 |
feat(showcase): port langgraph-fastapi declarative-gen-ui to canonical sales-analyst demo
Task 2 of OSS-582. Replace fastapi's old generic 'chart-picker' declarative-gen-ui with langgraph-python's canonical sales-analyst demo, so it matches the shared D6 probe (which asserts sales-dashboard / DataTable / StatusBadge / InfoRow per pill): - backend src/agents/src/a2ui_dynamic.py: generic prompt -> LGP sales-analyst graph (persona + dataset grounding + composition rules + generate_a2ui tool) - frontend definitions.ts/renderers.tsx/chat.tsx/suggestions.ts byte-identical to LGP, add sales-context.ts (fictional Vantage Threads dataset + composition rules) - d6 fixtures gen-ui-declarative.json + render-a2ui.json: LGP's, re-keyed to context=langgraph-fastapi Verified D6 green via control-plane (real 12s browser run, canonical testids asserted). a2ui-recovery intentionally NOT aligned: its fixtures have no context routing, so each integration must keep a unique pill prompt (sanctioned per-slug divergence). |
||
|
Tyler Slaton
|
831cfc0745 |
feat(showcase/built-in-agent): LGP parity — byte-identical frontends + named-agent backend (#6106)
## What & why
Brings the **built-in-agent** showcase integration to parity with the
**LangGraph-Python (LGP)** reference: byte-identical demo frontends + a
named-agent backend registry (BuiltInAgent + TanStack AI), so every demo
climbs the D0–D6 ladder against the LGP gold standard.
## Changes (4 commits)
1. **P0 pattern** — `agentic-chat` byte-identical + named agent
(`agentic_chat`); proven D6-green locally. Fixed `gpt-4o` → `gpt-5.5` in
the shared factory.
2. **Frontend migration (all demos)** — every LGP `src/app/demos/*`
copied verbatim (`diff -r` clean), plus shared `components/ui` (25
shadcn primitives) + `lib/utils`, byte-identical. Added the 5 demos BIA
lacked (`a2ui-recovery`, `declarative-hashbrown`,
`declarative-json-render`, `shared-state-read`,
`threadid-frontend-tool-roundtrip`); added the frontend deps the copied
UI needs (radix-ui, cmdk, embla-carousel-react, react-markdown,
remark-gfm, yaml, …).
3. **Named-agent backend** — `/api/copilotkit` registers 22 named agents
(generic all-tools, fixture-driven; reasoning trio via the reasoning
adapter). 8 dedicated routes re-keyed `default` → LGP agent id;
`mcp-apps` also serves `headless-complete`; `ogui` serves both
open-gen-ui ids; `byoc-*` routes renamed to `declarative-*`; new
`a2ui-recovery` + `beautiful-chat` routes reuse existing agents. Dropped
BIA-only extras (`byoc-*`, `hitl-in-chat-booking`).
4. **Reconcile** — `manifest.yaml` (37 features / 40 demos;
`generate-registry` + `validate-parity` pass) + `PARITY_NOTES.md`.
> **Note on "byte-identical":** frontends are verbatim LGP **modulo
BIA's `consistent-type-imports` ESLint rule** (type imports split into
`import type {}`) — required for a green lint/PR, semantically & DOM
identical.
## D6 status (local sweep)
- **~33/40 demos GREEN** on the first sweep — byte-identical frontends +
named agents + existing fixtures work broadly.
- **4 RED locally are an aimock-infra issue, not this integration:** the
deployed `ghcr.io/copilotkit/aimock:latest` has no
`context`/`--context-field` fixture scoping, so cross-slug `userMessage`
collisions let earlier-loaded (`ag2`/`d4`) fixtures shadow BIA's own.
BIA's fixtures are **correct** and converge under a context-aware aimock
(present on aimock `origin/main`). Affects
`tool-rendering-custom-catchall`, `headless-complete`, `gen-ui-agent`,
`frontend-tools`. **Action for infra: redeploy aimock from a
context-aware build.** Details in `PARITY_NOTES.md`.
- **2 downstream-host RED (kept as features, informational — mirrors
LGP):** `declarative-gen-ui` (A2UI renderer host) and `mcp-apps` (MCP
iframe host).
- Quarantined NSF unchanged: `gen-ui-interrupt`, `interrupt-headless`,
`shared-state-streaming`, reasoning-trio.
D6 is informational/weekly (not a merge gate); these are documented for
parity tracking.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
||
|
Austin Merrick
|
c89aa21658 | docs(showcase): explain existing-app hosted Intelligence setup | ||
|
Alem Tuzlak
|
bffb119091 |
fix(showcase): allowlist ms-agent-dotnet headless-complete custom-bubbles multi-file region
Restoring frontend parity re-added the @region[custom-bubbles] markers in both message-assistant.tsx and message-user.tsx (matching the langgraph reference and every other integration). Add the ms-agent-dotnet::headless-complete::custom-bubbles key to the multi-file-region allowlist so the demo-content bundler accepts it. |
||
|
Alem Tuzlak
|
d35935b35f |
fix(showcase): repoint ms-agent-dotnet manifest highlights to existing files
The stale-file cleanup removed chat-slots/custom-welcome-screen.tsx, the
top-level headless-complete/{message-list,use-rendered-messages}.tsx, and the
top-level gen-ui-interrupt/time-picker-card.tsx, but their manifest highlight
entries still pointed at them, failing the registry bundler (shell/shell-docs/
shell-dojo builds). Repoint to the real files (slot-wrappers, chat/chat,
hooks/use-tool-renderers, _components/time-picker-card).
|
||
|
Mike Ryan
|
98711fbefb | fix(showcase): stage Angular artifacts in deployment images | ||
|
Alem Tuzlak
|
1ab3fba04e |
feat(showcase): restore ms-agent-dotnet frontend parity with the langgraph reference
Restore the root shell (globals.css theme+brand, manifest-driven index, layout, middleware x-pathname, tsconfig), the declarative-gen-ui demo subtree, doc-snippet region markers, import-style, and subagents to match the langgraph-python reference; remove stale orphaned files. Load-bearing .NET adaptations (parse-json 3-layer unwrap, multimodal shim, tool-based hitl, agent-config properties) are preserved. |
||
|
Alem Tuzlak
|
b70b6913f7 |
feat(showcase): wire ms-agent-dotnet D6 routing, manifest, and threadid demo
Add the per-token createSharedStateStreamingAgent route shim, register the threadid-frontend-tool-roundtrip agent (frontend_tools passthrough) and its demo, drop shared-state-streaming from not_supported_features, and add the a2ui-recovery feature + demo entry to the manifest. |
||
|
Alem Tuzlak
|
55b4da3005 |
feat(showcase): add a2ui-recovery cell for ms-agent-dotnet
Implement the render/validate/retry recovery loop with a2ui_recovery_exhausted hard-fail as a raw-SSE endpoint (RecoveryAgent.cs, mounted in Program.cs), add the demo frontend + API route, the deterministic aimock fixture, and the unique per-slug PROMPTS entry in the shared d5-a2ui-recovery probe. |