## Summary
- **Restores ag2's `multimodal` D6 pill** from `skipped-incapable` (NSF)
to a working feature by adding a showcase-local ASGI middleware that
normalises AG-UI image/document/binary content parts to OpenAI Chat
Completions `image_url` parts before they hit AG2's `ConversableAgent`.
- **Surgical scope**: middleware mounted only on the multimodal sub-app
— other ag2 routes never see image parts and pay no body-buffer cost.
- **No upstream wait**: option (A) showcase shim, not an autogen PR.
autogen still lacks AG-UI image-part support; the moment they add it the
normalizer is a no-op and the RED-half regression pin flips to alert us.
- **Reverses commit d8a0a25db** for the multimodal half: removes
`multimodal` from `not_supported_features`, adds it back to `features`,
and restores the D6 aimock fixture pair.
`tool-rendering-reasoning-chain` stays quarantined (a different upstream
gap — no `REASONING_MESSAGE_*` events emitted by AGUIStream).
## What was failing
AG2's `autogen.code_utils.content_str` only accepts content-part types
`{"text", "input_text", "image_url", "input_image", "function",
"tool_call", "tool_calls"}`. The harness sends user messages whose
`content` carries:
- modern AG-UI: `{"type": "image" \| "document", "source": {"type":
"data" \| "url", "value": ..., "mime_type": ...}}`
- legacy mirror (appended by `legacy-converter-shim.tsx` for LangChain
integrations): `{"type": "binary", mimeType, data \| url}`
Both trip the gate with `ValueError("Wrong content format: unknown type
image within the content")` BEFORE the request reaches the vision model
— observed live on staging in the D6 multimodal probe. That's why the
feature was quarantined NSF in d8a0a25db.
## How the fix works
`agents/_multimodal_normalize.py` adds a raw-ASGI middleware (mirrors
the existing `RequestUserMessageMiddleware` pattern) that:
1. Buffers each inbound POST body.
2. Walks `messages[*].content` on user-role messages only.
3. Rewrites each AG-UI image/document/binary part to `{"type":
"image_url", "image_url": {"url": ...}}` — data sources become
`data:<mime>;base64,<value>` URLs; URL sources pass through unchanged.
4. Updates the request's `content-length` header.
5. Replays the rewritten body to the downstream AGUIStream endpoint.
Non-user messages, plain-text content, already-normalised parts, and
unknown shapes pass through untouched (identity-preserved on no-op
turns). Any body-parse failure logs at WARNING and replays the ORIGINAL
body so autogen's verbatim error surface stays intact — visibility, not
silent rewrite.
## RED → GREEN evidence
`tests/python/test_multimodal_normalize.py` — 14 unit tests, all pass:
| # | Test | What it pins |
|---|------|------|
| 1 | `test_autogen_rejects_raw_agui_image_part` | RED: `content_str`
raises the verbatim `ValueError` text the D6 probe surfaced |
| 2 | `test_normalized_content_is_accepted_by_autogen` | GREEN: after
normalize, `content_str` returns the rendered string with `<image>`
placeholder |
| 3-7 | shape coverage | image data/url, document data, binary data/url,
mimeType camelCase alias |
| 8-10 | passthrough | text-only, plain-string content, assistant/tool
messages |
| 11 | idempotency | re-running on already-normalised content is a no-op
|
| 12 | error path | unrecognised source → text placeholder (not a hard
fail) |
| 13 | tripwire | middleware class exposes `__init__(app)` + `__call__`
|
RED was independently verified by monkey-patching
`_normalize_content_part` to passthrough — that reproduces the exact
`ValueError("Wrong content format: unknown type image within the
content")` from the staging probe. Restoring the normalizer flips it
back to GREEN.
End-to-end ASGI smoke (run inline during development): a synthetic AGUI
POST body with a modern image part is sent through
`MultimodalContentNormalizerMiddleware` → inner ASGI app sees rewritten
body with correct `content-length`. PASS.
## Out of scope / follow-ups
- **PDF rendering**: PDFs ride through as
`data:application/pdf;base64,...` inside an `image_url` part — they
survive autogen's gate but the vision model can't read them natively.
Flattening PDFs to inline text (the pattern langgraph-python uses via
pypdf) is a separate enhancement; this PR's scope is unblocking the
image path that the D6 `multimodal` pill assertion checks.
- **Upstream**: autogen could fix this in `content_str` by accepting
AG-UI's `image`/`document`/`binary` content types directly. When/if that
lands, the normalizer becomes a no-op and the RED-half test will start
failing (which is the signal to delete the shim).
## Test plan
- [x] `cd showcase/integrations/ag2 && python -m pytest tests/python/` —
16 passed (2 pre-existing gen_ui guard tests + 14 new
multimodal_normalize tests)
- [x] `ruff format --check` on touched python files — clean
- [x] `ruff check` on touched python files — clean
- [x] `cd showcase/scripts && pnpm validate-manifests` — ag2 manifest
validates
- [x] `oxfmt --check showcase/aimock/d6/ag2/multimodal.json` — clean
- [x] Verified `multimodal_app.user_middleware` includes
`MultimodalContentNormalizerMiddleware` after import
- [x] End-to-end ASGI smoke: middleware rewrites body + updates
content-length, downstream app sees normalised payload
- [ ] Staging deploy: D6 `multimodal` pill flips from
`skipped-incapable` to GREEN with image fixture (1×1 PNG → "image
attachment shows a small abstract test pattern..."). Validated
post-merge via the staging deploy.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
- Remove unused imports: Iterable, ConversableAgent (from autogen),
AGStreamInput (from autogen.ag_ui.adapter) — none appear in executable
code, only in docstring prose.
- Fix raw_msgs possibly-unbound at dispatch guard: initialize to None
before the try block so the identity check at line 302 is always
safe even if model_dump raises before raw_msgs is assigned. Also
tighten the guard to `raw_msgs is not None` to make the no-normalization
fallback explicit.
autogen.ag_ui import unresolved and LLMConfig(dict) "Expected 0 positional
arguments" are ENVIRONMENT findings — autogen.ag_ui ships only in the
ag2[ag-ui] extra (present in the container, not in local Pyright's venv),
and LLMConfig({...}) is the codebase-wide pattern that works at runtime
with ag2>=0.9 as installed in the container.
AG2's ConversableAgent runs every user message through
``autogen.code_utils.content_str``, which only accepts content-part
types in {"text", "input_text", "image_url", "input_image", "function",
"tool_call", "tool_calls"}. CopilotChat / the AG-UI runtime emits image
and document attachments as the modern shape
{"type": "image" | "document", "source": {...}}
and the demo page's legacy-converter-shim.tsx ALSO appends a legacy
{"type": "binary", mimeType, data | url}
mirror alongside it (to keep the @ag-ui/langgraph converter happy on
LangChain-based integrations — it rides through on the ag2 path too).
Both shapes trip autogen's allowed-types gate with
ValueError("Wrong content format: unknown type image within the
content")
…BEFORE the request reaches the vision model — observed live in the
D6 multimodal probe (commit d8a0a25db, which originally quarantined
the feature as NSF).
Fix
---
Add ``agents/_multimodal_normalize.py``: a ``NormalizingAGUIStream``
subclass of ``AGUIStream`` that overrides ``dispatch()`` to normalize
AG-UI image/document/binary content parts to OpenAI Chat Completions
``image_url`` parts AFTER ``RunAgentInput`` Pydantic parsing and BEFORE
``AgentService`` serialises the messages for autogen.
This is the only correct interception point:
- Too early (ASGI body rewrite before Pydantic): ``RunAgentInput``
rejects ``image_url`` because it is not an AG-UI standard type —
the discriminated union only accepts image/document/binary/text.
- Too late (inside ConversableAgent): requires patching autogen
internals.
The override works by calling ``normalize_messages_for_autogen()`` on
the dict-serialised messages (same form as ``run_stream`` produces via
``model_dump()``) and re-injecting them via a ``_PatchedRunAgentInput``
wrapper that overrides only ``.messages``, delegating all other
attribute access to the original ``RunAgentInput``.
Conversions:
- {"type": "image", "source": {"type": "data", value, mime_type}} →
{"type": "image_url", "image_url": {"url": "data:<mime>;base64,<value>"}}
- {"type": "image", "source": {"type": "url", value}} →
{"type": "image_url", "image_url": {"url": value}}
- {"type": "document", "source": ...} → image_url with the document's
mime preserved (data:application/pdf;base64,...). The vision model
still can't natively read PDFs, but the request reaches the model
instead of being rejected upstream, which is the failure mode this
fix targets.
- {"type": "binary", mimeType, data | url} → image_url (the
legacy-shim parts ride through cleanly).
- {"type": "text", ...} and already-normalised image_url parts pass
through unchanged (identity-preserved on no-op turns).
Failure path: any normalization error is logged at WARNING and the
original messages are forwarded unchanged — autogen's own ValueError
fires verbatim with its error surface intact.
Manifest + fixture
------------------
- showcase/integrations/ag2/manifest.yaml: remove multimodal from
not_supported_features (with its now-stale comment) and add it back
to the features list next to voice.
- showcase/aimock/d6/ag2/multimodal.json: add the D6 fixture pair
using the actual autoPrompt strings from sample-attachment-buttons.tsx
("can you tell me what is in this demo image I just attached" /
"can you tell me what is in this demo pdf I just attached").
TDD evidence (red-green)
------------------------
showcase/integrations/ag2/tests/python/test_multimodal_normalize.py
contains 14 unit tests, pinned at three layers:
1. RED/GREEN against autogen's actual content gate:
* test_autogen_rejects_raw_agui_image_part — confirms
content_str([{type: image, source: ...}]) raises the verbatim
ValueError the D6 probe surfaced. This is the regression pin: if
autogen ever relaxes the gate, this test fails and we know to
revisit the normalizer.
* test_normalized_content_is_accepted_by_autogen — after
normalize_messages_for_autogen(...), content_str accepts every
part and renders "<image>" for the image_url part.
2. Shape coverage: modern image data/url, modern document, legacy
binary data/url, mimeType camelCase alias, plain-text passthrough,
plain-string content, assistant/tool messages untouched,
unrecognised source → text placeholder, idempotency.
3. NormalizingAGUIStream class surface tripwire.
Control-plane D6 RED→GREEN:
RED (no normalizer, pre-fix container): d6:ag2/multimodal → red
(HTTP 500 agent_run_error_event from content_str ValueError)
GREEN (NormalizingAGUIStream applied): d6:ag2/multimodal → green
## What
Adds a **Session-stack discipline / Cleanup after isolated runs**
subsection to `showcase/TESTING.md`, governing how `--isolate`/`--keep`
is used across a debugging/testing session.
## Why
`--keep` correctly lets an `--isolate <name>` stack survive a run so it
can be reused for a session-long test set. The leak was **agent
discipline**, not the flag:
1. Agents minted a **new** named kept stack per individual cell instead
of reusing ONE stack for the whole session — which is how Docker
accumulated `cvtest2`, `greenproof`, `conformred`, `conformgreen`,
`gp1`..`gp10`, `showcase-iso2/4`, etc.
2. When the session's work was done, the stacks it created were never
torn down — each one holds a slot and offset ports until the host fills
up.
## The discipline encoded
1. **One stack per session, reused** — choose ONE stable `--isolate
<session-name> --keep` and reuse it for ALL tests in the session (derive
the name from the primary slug, e.g. `--isolate <slug>-session`). Never
mint a new named stack per cell/feature/pill.
2. **`--keep` is for intra-session reuse only, never a license to leak**
— if you pass `--keep`, you OWN teardown at session end.
3. **Tear down at session end** — use the survival-notice command
`docker compose -p <name> down --remove-orphans --volumes && rm -rf
<run-dir> <slot-dir>`. `bin/showcase down` does NOT tear down isolated
stacks (it only stops the default `showcase-*` project). Bare
`--isolate` (no `--keep`) auto-cleans and is preferred for one-off
tests.
Teardown mechanics live once in `DEBUGGING.md → Cleanup` (cross-linked);
this section owns the discipline.
## Verification
This is a docs-only behavior change — no probe/Playwright/code red-green
surface applies. Doc quality gates run instead:
- `oxfmt --check showcase/TESTING.md` → passes (the file was correctly
formatted on `main`; the only formatter touch was `*new*` → `_new_`).
- Diff is purely additive (+49 lines, one file).
- Cross-link anchor `DEBUGGING.md#cleanup` verified against the `###
Cleanup` heading.
The three harness facts the guidance relies on were verified by reading
`scripts/cli/_common.sh`, `cmd-test.sh`, and `bin/showcase`: each
`--keep` run claims a fresh slot + idempotent pre-down + brings the
stack up (no attach); a same-name re-run against a still-live kept stack
fails loudly on the duplicate-name guard; the exact teardown command
matches the survival notice at `_common.sh:~1074`.
## Follow-up (not in this PR)
The harness could make this self-enforcing — e.g. warn when a session
uses >1 distinct kept `--isolate` name, or add a `bin/showcase slots
--reap-mine` convenience to tear down all stacks this user created.
Noted for later; no harness changes here.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
The `version_drift` probe fails on the fleet control-plane with
`probe.discovery-enumerate-failed` / `discoveryFailed:true` and writes 0
PB rows.
**Root cause:** the fleet `pnpm-workspace.yaml` carries a multi-segment
glob `examples/v2/*/apps/*`. The `pnpm-packages` discovery source's
`matchPattern` only supports literals and trailing `/*`, and throws
`DiscoverySourceSchemaError("unsupported pnpm-workspace glob pattern")`
for any pattern whose segment after the first `*` is not `""`/`"/"`.
That throw happens inside `expandPatterns` during enumeration —
**before** the probe's `pathPrefix: "packages/"` filter is applied — so
it aborts the entire enumeration. `version_drift` never gets to filter
to `packages/`; it just fails.
**Fix (option a):** thread `pathPrefix` into `expandPatterns` and skip
any pattern whose static (wildcard-free) leading prefix cannot intersect
the requested `pathPrefix`, **before** validating its glob shape.
`examples/v2/*/apps/*` (static prefix `examples/v2/`) is skipped when
the probe only wants `packages/`, so enumeration completes and returns
the `packages/` set.
Preserved behavior:
- An unsupported deep-glob that DOES overlap the requested prefix (e.g.
`packages/*/apps/*` with `pathPrefix: packages/`) still throws the
strict-shape SchemaError.
- With no `pathPrefix`, every pattern is in scope and the existing
strict-throw behavior is unchanged.
## Changes
- `showcase/harness/src/probes/discovery/pnpm-packages.ts` —
`staticPrefix` + `patternIntersectsPrefix` helpers; `expandPatterns` now
takes `pathPrefix` and skips out-of-prefix include/exclude patterns
before shape validation.
- `showcase/harness/src/probes/discovery/pnpm-packages.test.ts` —
red→green regression tests.
## Test plan
- [x] Red: new "skips out-of-prefix deep-glob" test fails against
unfixed source (throws SchemaError in `expandPatterns`)
- [x] Green: same test passes after the fix; out-of-prefix deep-glob
skipped, `packages/` packages still enumerated
- [x] Preserved: unsupported deep-glob overlapping the prefix still
throws; existing "non-trailing `*` throws" test (no pathPrefix)
unchanged
- [x] Full harness suite: 2137 passed (120 files)
- [x] `tsc -p tsconfig.build.json` clean
## Summary
Wires the starter-smoke probe's keyed `errorClass` into the dashboard
cell-state flip logic (`buildStarterBadge` in `live-status.ts`) so
transient SOFT failures get **two-miss tolerance**, reducing dashboard
flapping on transport hiccups.
- **SOFT** (`transport-error`, `aborted`): a *single* miss is tolerated
— the cell renders **amber `~`** instead of flipping red. Flips red only
on a **second consecutive** miss.
- **HARD** (`smoke-failed`): flips red **immediately**, no tolerance.
- A soft miss followed by a green tick renders a clean green ✓
(recovery).
## How the consecutive-miss count works (no new dashboard state)
`errorClass` was previously unused on the dashboard. The flip gate
reuses the **producer-maintained `fail_count`** — the harness
`status-writer`'s persisted consecutive-red counter (`1` on green→red,
incremented on sustained red, `0` on red→green). So
`resolveCell`/`buildStarterBadge` stay a **pure function of the current
row**: there is no dashboard-side counter to thread or reset. Tolerance
is applied as a `state` → `degraded` downgrade inside
`buildStarterBadge` (the same additive pattern as the existing
stale-green→degraded fold), so the connection/tooltip/drilldown-signal
are all preserved and a `.row.state` reader sees `degraded` (agreeing
with the amber tone), never a latent false-red.
Threshold: `fail_count >= 2` flips; `fail_count <= 1` tolerates.
## errorClass values used (soft/hard split)
Mirror of the harness `StarterFailureClass` union in
`showcase/harness/src/probes/drivers/starter-smoke.ts`:
| class | split | meaning |
|---|---|---|
| `transport-error` | **SOFT** | timeout / cold-start wake / connection
failure |
| `aborted` | **SOFT** | external-abort / outer-timeout |
| `smoke-failed` | **HARD** | real HTTP-level content regression |
Added as a dashboard-side mirror `STARTER_FAILURE_CLASSES` (the
dashboard imports only `@/*` and cannot reach across the package
boundary), guarded by a new **`starter-error-class-drift.test.ts`**
set-equality lint against the harness source — mirroring the existing
`commError-contract-drift.test.ts` pattern.
## Semantics chosen / ambiguity flagged (conservative defaults)
These were genuinely ambiguous; the most conservative sensible behavior
was chosen and is flagged here for review:
1. **A tolerated soft miss renders AMBER `~`, NOT green.** The probe
literally just failed, so claiming a green ✓ would be a false-green lie
(the codebase guards against false-green everywhere). Amber says
"transient, not yet actionable" — distinct from both the flap-to-red and
a dishonest green.
2. **Tolerance applies ONLY to an *explicit* soft `errorClass`.** A red
row with **no** `errorClass` (or an unrecognized value) flips
immediately as before — we only soften when the producer explicitly tags
the failure transient. This preserves all pre-existing red-row tests.
3. **`fail_count <= 1` (not strictly `== 1`) is tolerated** to guard the
legacy/edge boundary where a first failure reports `0`.
4. **Unsupported columns are unaffected** — the 🚫 mapping-derived state
still wins over any row data.
## Test plan (red → green)
- [x] RED first: the 3 single-soft-miss tolerance assertions failed
against current `main` (soft single miss flipped red); the 6
behavior-preserving assertions passed.
- [x] GREEN after implementation: all 9 new tests pass.
- [x] New drift guard `starter-error-class-drift.test.ts` passes
(set-equal vs harness `StarterFailureClass`).
- [x] Full dashboard vitest suite: **922 passed, 1 skipped (59 files)**
— incl. `STATUS_LIST_FIELDS` guard, comm-error contract tests, and all
pre-existing starter-badge tests.
- [x] `tsc --noEmit` clean.
## Reconciliation with peer "speedup" work
No speedup-owned symbols were modified: `summarizeSignal`,
`STATUS_LIST_FIELDS`, the `rowsAreNoop` signal-presence clause, and
`extractSignalFields` (which lives in `cell-drilldown.tsx`, not
`live-status.ts`) are all untouched. The tolerance logic is fully
self-contained (`toleratedSoftMissRow` + the taxonomy mirror) and layers
onto the existing badge path. The diff is `live-status.ts` (+119
additive), its test file, and one new drift test.
Wire the starter-smoke probe's keyed errorClass into the dashboard cell-state
flip logic so transient SOFT failures (transport-error / aborted) get two-miss
tolerance: a single soft miss renders amber ~ ("transient, not yet actionable")
instead of flapping the cell red, and only flips red on a second consecutive
miss. HARD failures (smoke-failed) and untagged reds flip immediately.
The flip gate reuses the producer-maintained fail_count (the persisted
consecutive-red counter: 1 on green->red, incremented on sustained red, 0 on
red->green) so the dashboard stays a pure function of the current row — no
dashboard-side counter to thread or reset. Tolerance is applied as a
state->degraded downgrade in buildStarterBadge (same pattern as the existing
stale-green fold), keeping the change additive and self-contained.
Adds STARTER_FAILURE_CLASSES as a dashboard-side mirror of the harness
StarterFailureClass union (the dashboard imports only @/*), guarded by a new
starter-error-class-drift.test.ts set-equality lint against the harness source.
The fleet pnpm-workspace.yaml carries a multi-segment glob
(`examples/v2/*/apps/*`) that the strict matcher rejects with a
SchemaError. Because that throw happens during enumeration — before the
probe's `pathPrefix` filter applies — it aborted the entire version_drift
discovery, surfacing as probe.discovery-enumerate-failed / discoveryFailed
with 0 PB rows.
Skip patterns whose static (wildcard-free) prefix cannot intersect the
requested `pathPrefix` BEFORE validating their glob shape, so a deep-glob
for an unrelated subtree no longer aborts a probe that only wants
`packages/`. An unsupported pattern that DOES overlap the requested prefix
still surfaces the strict-shape SchemaError, and behavior with no
pathPrefix is unchanged.
## Summary
Follow-up to #5832 which updated `generate_a2ui.py` to emit A2UI v0.9
nested op format. The 6 tests in
`showcase/integrations/langroid/tests/python/test_generate_a2ui.py`
still asserted the old flat format and were failing in CI.
- Updates assertions from flat format (`ops[0]["type"] ==
"create_surface"`, `ops[0]["surfaceId"]`, `ops[2]["data"]`) to v0.9
nested format (`ops[0]["version"] == "v0.9"`,
`ops[0]["createSurface"]["surfaceId"]`,
`ops[2]["updateDataModel"]["value"]`)
- No assertions were weakened — all structural checks were preserved and
extended to verify the full nested shape
## Red-Green Proof
**RED** (before fix): 6 failed, 0 passed
```
FAILED tests/python/test_generate_a2ui.py::test_generate_a2ui_happy_path_returns_operations
FAILED tests/python/test_generate_a2ui.py::test_generate_a2ui_happy_path_json_string_arguments_also_work
FAILED tests/python/test_generate_a2ui.py::test_generate_a2ui_legacy_function_call_path
FAILED tests/python/test_generate_a2ui.py::test_multi_tool_call_picks_first_and_warns
FAILED tests/python/test_generate_a2ui.py::test_tool_call_missing_function_attr_falls_through_to_legacy_path
FAILED tests/python/test_generate_a2ui.py::test_tool_call_with_function_arguments_none_falls_through_to_legacy_path
```
**GREEN** (after fix): 6 passed; full suite: **118 passed, 1 skipped**
Remove the Dependabot `github-actions` ecosystem config plus its
companion `dependabot-auto-merge` and `dependabot-major-analysis`
workflows. Renovate (via `renovate.json` → `local>CopilotKit/renovate`,
Dependency Dashboard #592) now owns github-actions updates.
Also cleans stale references to the deleted files:
- `.github/zizmor.yml`: drop the `dangerous-triggers` ignores for the
two dependabot workflows, remove the now-empty `dependabot-cooldown`
rule, and update the `unpinned-uses` comment to reference Renovate.
- `.github/workflows/security_zizmor.yml`: drop the
`.github/dependabot.yml` path triggers.
`.github/dependabot.yml` contained ONLY the github-actions ecosystem, so
it is deleted in full. No npm/pip/docker or other ecosystem was touched
— npm is untouched.
Rebased onto current main; all CI green (zizmor pass, commitlint pass,
build/types/unit/package-quality all pass).
## Summary
- Productizes the Claude SDK Python and TypeScript showcase demos with
LangGraph-parity frontends.
- Wires the Claude demo backends through the official Claude Agent
SDK/AG-UI adapter paths using `claude-sonnet-4.6`.
- Keeps Claude integration docs hidden for this PR and excludes
generated/authored docs artifacts from scope.
## Why
The goal is to bring the productized LangGraph demo surface to Claude
Agents SDKs without publishing integration docs in this pass. This keeps
the PR focused on local showcase demos, runtime behavior, fixtures, and
validation support.
## How
- Ported the demo frontend surfaces and local shell-dojo support for
Claude SDK Python/TypeScript.
- Added official Claude SDK adapter/backend wiring plus real-Claude
local compose support.
- Updated Claude aimock fixtures and validation ratchets for the
expanded demo set.
- Set both Claude manifests to `docs_mode: hidden` and removed docs
setup/snippet artifacts from the PR scope.
## Problem
\`@ag-ui/a2ui-middleware\` v0.0.10's \`getOperationSurfaceId()\` reads
only the A2UI v0.9 NESTED op format:
\`\`\`json
{"version": "v0.9", "createSurface": {"surfaceId": "...", "catalogId":
"..."}}
\`\`\`
The integrations were emitting the legacy FLAT format:
\`\`\`json
{"type": "create_surface", "surfaceId": "...", "catalogId": "..."}
\`\`\`
Result: all ops fell back to the \"default\" surface key → frontend
never mounted the named surface → \`surface-missing\` failure on
\`declarative-gen-ui\` across ~11 integrations.
## Fix
Convert all a2ui op builders and inline ops to the nested v0.9 format
in:
- \`tools/generate_a2ui.py\` — 9 integrations (agno, claude-sdk-python,
crewai-crews, langgraph-fastapi, langgraph-python, langroid, llamaindex,
pydantic-ai, strands)
- \`tools/search_flights.py\` — 11 integrations (ag2, agno,
claude-sdk-python, crewai-crews, langgraph-fastapi, langgraph-python,
langroid, llamaindex, ms-agent-python, pydantic-ai, strands)
- \`src/agents/a2ui_fixed_agent.py\` / \`a2ui_fixed.py\` /
\`beautiful_chat.py\` — agno, crewai-crews, langroid, pydantic-ai
Already-correct integrations skipped: google-adk,
ms-agent-python/generate\_a2ui.py, ag2/generate\_a2ui.py.
**Total: 25 files changed.**
## Verification
Zero flat-format ops remain in non-comment/non-test code. 96 occurrences
of \`"version": "v0.9"\` present in changed integrations (excluding
google-adk which was already correct).
## Red→Green
\`bin/showcase test\` runs against Docker containers — requires
infrastructure startup. The structural change is a mechanical
search-and-replace: \`getOperationSurfaceId()\` in \`a2ui-middleware\`
v0.0.10 reads \`op.createSurface?.surfaceId\` (nested), which is exactly
what these changes now emit. The old flat \`op.surfaceId\` path is not
read at all by the middleware, explaining the surface-missing fallback.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Commit `7c3edca2b7` (May 11) changed the `autoPrompt` strings in all
`sample-attachment-buttons.tsx` from `"describe the sample image"` /
`"summarize the sample document"` to `"can you tell me what is in this
demo image I just attached"` / `"can you tell me what is in this demo
pdf I just attached"` — but never updated the 41 aimock fixture files
- Every integration that uses the auto-send pattern was sending a
message matching no fixture → strict MISS → 404 → agent error banner →
`dom-missing` / `done-signal-missing` D6 timeouts
- Fixed 19 `multimodal.json` D6 fixtures, 20 `agentic-chat.json` D6
fallback fixtures, 1 shared D5 `multimodal.json`, and the
`split-fixtures.ts` router
## Red-Green Proof
**RED (before fix) — `langgraph-typescript:multimodal --d6 --direct`:**
```
turn 1: TIMEOUT dom-missing (60s) — aimock 404, no assistant text rendered
turn 2: TIMEOUT dom-missing (60s) — same
```
**GREEN (after fix) — `langgraph-typescript:multimodal --d6 --direct`:**
```
turn 1: PASS — assistant text "The attached image is the CopilotKit logo..." settled
turn 2: PASS — assistant text with document summary settled
```
## Remaining failures (out of scope, separate issues)
- `ms-agent-python`, `crewai-crews`: Python backend
`ChatClientException` when receiving binary (image/PDF) AG-UI content
parts — same class as active `wt-pydantic-multimodal` worktree
- `built-in-agent`, `claude-sdk-python`: DOM-inject only (no
`agent.addMessage` / `copilotkit.runAgent` auto-send) — probe design
mismatch, not a fixture issue
## Test plan
- [x] `langgraph-typescript:multimodal --d6 --direct` RED before / GREEN
after
- [ ] D6 repro sweep after merge to confirm cluster clears for auto-send
integrations
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Commit 7c3edca changed sample-attachment-buttons.tsx across all integrations
to auto-send via agent.addMessage with autoPrompt strings:
- "can you tell me what is in this demo image I just attached"
- "can you tell me what is in this demo pdf I just attached"
But the d5 harness fixture and all 19 d6 per-integration multimodal.json
fixtures still matched on the old strings:
- "describe the sample image"
- "summarize the sample document"
Aimock received requests with the new prompts, found no match, returned
a STRICT 404, and the agent emitted a streaming error back to the UI
(exact symptom: "An internal error has occurred while streaming events").
Also update agentic-chat.json across all 20 integrations (those files had
duplicate fallback entries for the old prompts) and fix split-fixtures.ts
to route the new strings to the "multimodal" feature bucket.
Local RED: ms-agent-python and crewai-crews both fail with fixture-miss
status=miss before this change.
Local GREEN: langgraph-typescript passes after this change (both turns
settle with "image" / "document" keywords confirmed in transcript).
Remaining failures after this fix are pre-existing Python backend issues
(ChatClientException on binary content parts in ms-agent-python; CrewAI
flow failure on binary content in crewai-crews) — unrelated to fixture
keys and tracked separately in the pydantic-ai multimodal work.
Remove the Dependabot github-actions ecosystem config and its companion
auto-merge / major-analysis workflows. Renovate (via
renovate.json -> local>CopilotKit/renovate, Dependency Dashboard #592)
now owns github-actions updates.
Also clean stale references to the deleted files:
- zizmor.yml: drop dangerous-triggers ignores for the two dependabot
workflows, remove the now-empty dependabot-cooldown rule, and update
the unpinned-uses comment to reference Renovate.
- security_zizmor.yml: drop the .github/dependabot.yml path trigger.
npm and other ecosystems are untouched (dependabot.yml had only the
github-actions ecosystem).
Semantically-identical reformat of `renovate.json` (compact single-line,
same content) to nudge Mend/Renovate to re-evaluate against the NEW
github-actions-only central preset.
Background: Renovate hasn't re-scanned since 2026-06-16 due to the
cached-old-preset gotcha, so no Dependency Dashboard has appeared yet
after the onboarding cutover (#5031). A no-op touch of the consumer
config forces re-evaluation.
Content is unchanged:
```json
{"$schema":"https://docs.renovatebot.com/renovate-schema.json","extends":["local>CopilotKit/renovate"]}
```
No behavior change; npm remains human-controlled.
Migrates this repo's Renovate configuration to extend the org-wide
central config at https://github.com/CopilotKit/renovate.
Phase 1 of the migration ([Notion
plan](https://www.notion.so/3613aa38185281a38863fcff2907021c)) scopes
Renovate to the github-actions ecosystem only; npm/pip remain on
Dependabot.
Replaces the previous renovate.json (which had `extends:
config:recommended` plus a global "ignore all packages initially" rule
that effectively disabled the prior Renovate install). The new config
inherits from the org-wide central preset, which is scoped to
github-actions for Phase 1.
Pre-existing open Renovate PRs (#4751, #3295) from the prior install can
be closed separately once this lands.
## Root Cause
`agno 2.6.20` removed `agno.os.interfaces.agui.utils`. The floating
`agno>=2.5.17` pin in `requirements.txt` caused staging to pull the
breaking version on the next build, causing a startup failure.
## Red-Green Proof
**RED** — with `agno>=2.6.20` installed:
```
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'agno.os.interfaces.agui.utils'
```
**GREEN** — with `agno==2.6.19` installed:
```
GREEN: all 3 symbols OK
```
(Symbols confirmed: `async_stream_agno_response_as_agui_events`,
`extract_agui_user_input`, `validate_agui_state`)
## Changes
- `showcase/integrations/agno/requirements.txt`: pinned `agno>=2.5.17` →
`agno==2.6.19` (exact pin, last version with `agui.utils`)
- `showcase/integrations/agno/src/agent_server.py`: added TODO comment
at line 73 import site noting migration to agno 2.6.20+ API is a
follow-up; no structural changes to imports
## Follow-up
Migration of `agent_server.py` imports to the agno 2.6.20+ API (once the
replacement for `agui.utils` is identified) is tracked in the TODO
comment at line 73.
## Note on CI
The `validate-pins` CI check will likely flag pre-existing non-exact
pins across ~15 other integrations (`openai ^5.9.0`, `crewai` ranges,
etc.). This is pre-existing debt not introduced by this PR.
## What & why
Resolves [OSS-132](https://linear.app/copilotkit/issue/OSS-132).
Investigated with systematic-debugging; every conclusion verified
against the **real** OpenAI Responses API.
**Net change: a TanStack version bump only.** No showcase schema change.
- `@tanstack/ai` `0.18.0` → `0.35.0`
- `@tanstack/ai-openai` `0.9.1` → `0.15.6`
- `package-lock.json` regenerated (Dockerfile uses `npm ci
--legacy-peer-deps`)
## The bug
The built-in-agent showcase 400s on every prompt against real OpenAI.
The state tools (`AGUISendStateSnapshot` / `AGUISendStateDelta` /
`set_steps`) declare arbitrary payloads as `z.any()`, which serializes
to a **typeless** JSON-Schema property (`{ "description": ... }`, no
`"type"`).
The old `@tanstack/openai-base`'s `isStrictModeCompatible()` only
screened for `oneOf/allOf/not/$ref/$defs`, so it missed the missing
`type`, sent the tool with `strict: true`, and OpenAI rejected it:
```
400 Invalid schema for function 'AGUISendStateSnapshot':
In context=('properties','snapshot'), schema must have a 'type' key.
```
This was **masked in production** because the deployed showcase runs
against aimock, which replays fixtures without validating the request
schema — a raw `curl` to prod returns a clean `RUN_FINISHED`, green for
the wrong reason.
The ticket's original framing (zod3/zod4 drift → typeless *root*, `got
"None"`) was already fixed by the zod-4 migration; this is the same
symptom one layer down (typeless *property*).
## The fix is upstream
`@tanstack/ai-openai@0.15.6` (via `@tanstack/openai-base@0.9.2`) fixes
`isStrictModeCompatible`: it now detects typeless / `z.any()` properties
and sends `strict: false`. OpenAI accepts typeless properties under
`strict: false` — so `z.any()` works again with no schema change on our
side.
(`@tanstack/ai-openai@0.15.5` also dropped `@tanstack/ai-client` from
its peerDependencies, so no `ai-client` dep is added.)
## Verification (real OpenAI, gpt-4o)
| Probe | Result |
|---|---|
| Typeless property, `strict: true` (raw OpenAI) | **400** — `schema
must have a 'type' key` |
| Typeless property, `strict: false` (raw OpenAI) | **ACCEPTED** —
confirms it was the strict flag, not the schema |
| `z.any()` tool on old adapter (0.9.1/0.15.4) | adapter sends `strict:
true` → **400** |
| `z.any()` tool on new adapter (0.15.6) | adapter sends **`strict:
false`** → **ACCEPTED**, model calls the tool |
| All 3 `z.any()` state tools attached, new adapter | **ACCEPTED**, no
400 |
## Not covered here
The showcase's aimock + Playwright e2e suite was **not** run locally
(this worktree has no installed toolchain). CI runs it on this PR;
please confirm the gen-ui / shared-state demos still pass before merge.
---
_Branch history shows an interim `z.string()` workaround that was
reverted once the upstream fix shipped; the net diff is the version bump
only. Squash-merge recommended._
One-line fix: `bin/railway`'s `run_staging_probe` invoked `npx --yes tsx
verify-deploy.ts` from the repo root with no `chdir`, so under Node 22
tsx failed to resolve (MODULE_NOT_FOUND in the ESM preload) → the
promoter misread it as 'staging not green' → hard REFUSE. This
tier-gated all prod promotes (incl. the langgraph fix in #5825). Fix
adds `chdir: File.expand_path("../scripts", __dir__)` so tsx resolves
from `showcase/scripts/node_modules`.
Red-green: from repo root `npm ls tsx` is empty and `npx tsx` crashes;
from showcase/scripts it resolves (tsx declared in
showcase/scripts/package.json).
NOTE: the agno `<2.6.20` pin (originally bundled here) was split out —
it edits a requirements file which trips the fleet-wide validate-pins
ratchet (pre-existing non-exact-pin debt across ~15 integrations).
Tracking separately.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
agno 2.6.20 removed agno.os.interfaces.agui.utils; the floating
agno>=2.5.17 pin in requirements.txt caused staging to pull the
breaking version. Pinned to 2.6.19 (last version with the module).
Added TODO comment at the import site for future migration.
Without chdir, npx resolves tsx from the repo root where it is not installed.
tsx is a dev dependency of showcase/scripts; chdir ensures npx resolves it correctly.
## What & why
Showcase services were being killed on Railway. Root causes, all fixed
here:
1. **langgraph-python / langgraph-fastapi — watchfiles log flood →
Railway 500-logs/sec replica kill.** `langgraph dev` ran with
hot-reload, emitting "1 change detected" per request; under D6 probe
fan-out this blew past Railway's 500 logs/sec cap and killed the
replica. Fix: `--no-reload` + `export
LANGGRAPH_DISABLE_FILE_PERSISTENCE=true` (also stops unbounded
pickle-state OOM).
2. **langgraph-typescript — `FileSystemPersistence` RangeError crash
loop.** `@langchain/langgraph-api` serialized unbounded thread state via
`JSON.stringify`; past V8's ~512MB string ceiling it threw `RangeError`
in a timer, hung the event loop, and the watchdog kill-looped (state
persisted on disk, so restarts re-crashed). Fix: boot-purge stale state
+ a **size-gated** restart (checks dir size, only restarts near the
ceiling — no in-flight-wiping timer, no unpinned `/internal/truncate`).
3 & 4. **Per-request proxy log flood across all integrations.**
`[copilotkit/route] POST` + `Response status` logged on every
sub-request, unconditionally, in 19 `route.ts`. Fix: gate them behind
`SHOWCASE_ROUTE_DEBUG` (off in prod) — **but keep non-2xx responses
logged unconditionally** so production errors stay visible, and gate the
health-probe GET too.
## Verification
- Every fix carries local red-green. langgraph-typescript entrypoint:
**18 mutation-sensitive subprocess tests** (reversed comparison / broken
du|awk / wrong-kill-target all caught; orphan-cleanup reaped). route.ts
gating verified on the real Next.js surface across ≥3 integrations
(non-2xx logged, 2xx+health gated, `SHOWCASE_ROUTE_DEBUG=1` restores
verbose).
- Code review: Round 1 (7 agents) → fixes → Round 2 (7-agent
confirmation) → fix → Round 3 (3-lens targeted) → fix → converged to
zero mandatory findings.
## ⚠ Before merge
The two entrypoint changes (`--no-reload` +
`LANGGRAPH_DISABLE_FILE_PERSISTENCE` on pinned `langgraph-cli 0.4.21`)
are **source-verified but could not be run locally** (the langgraph
packages are on a private index; `0.4.21`'s `--no-reload` was confirmed
only in public `0.4.3`). **Requires live-Railway validation** (branch
deploy: boots, serves 200, no watchfiles spam, no pickle files) before
merge. Kept as a **draft** until validated and the maintainer approves.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Gates per-request POST + 2xx Response-status + GET health-probe logs behind SHOWCASE_ROUTE_DEBUG across 19 integrations to stay under Railway's 500-logs/sec cap, while logging non-2xx responses unconditionally so production errors stay visible.
Boot-purge of stale .langgraph_api state plus a size-gated restart (du > threshold -> kill agent -> container restart -> purge), replacing an in-flight-wiping periodic truncate loop. Adds mutation-sensitive subprocess tests for the watchdog.
## What this does
Re-platforms the banking showcase's self-learning off the **abandoned**
offline-distill path (which targeted the now-closed Intelligence #192
`record → /annotate → sl-worker → /knowledge` pipeline) onto the
**shipped** memory substrate (`libs/memory`, Intelligence #294/#321).
The agent now saves a demonstrated over-limit procedure as a
`project`-scoped, `procedural` memory via `save_memory`, and
`recall_memory`s it at the start of later over-limit requests — so a
**fresh thread, or a different user on the same team, completes the
approval unaided**. That's the FOR-149 durable cross-thread + cross-user
proof.
## Verified live (local stack)
- Vendored memory-enabled Intelligence stack comes up healthy; `POST
/api/memories` → `201`, `/recall` → `200`, and
`save_memory`/`recall_memory`/`forget_memory` MCP tools attach
(`SL_ENABLED` + embedder).
- Cross-user: a project memory saved by one user recalls for a different
user.
- App boots in Intelligence mode; OSS fallback (`InMemoryAgentRunner`)
untouched and still the default.
## Changes
- **`docker-compose.yml`** — vendored stack cloned from the proven
`memory-chat` recipe (postgres/pgvector, redis, minio, TEI, composite
app-api + gateway). Hardened during a real bring-up: `minio-init`
DNS-race retry, **pluggable embedder** (`MEMORY_EMBEDDINGS_URL` + `tei`
dependency `required:false`, so RAM-constrained / Apple-Silicon machines
can point at a host TEI), and non-colliding `715x` host ports.
- **Runtime** (`route.ts`) — Intelligence branch gains `licenseToken` +
lock config + `generateThreadNames`; recall-first / save-on-teach
prompt; `recall_memory`/`save_memory` added to the tool list.
- **`saveLearnedWorkflow`** resolves a `status: saved` result that
drives the agent's `save_memory` call (Option A — agent-initiated),
keeping the already-approved guard.
- **Removed** the dead `record-user-action` `/annotate` seam (kept the
visual `useRecording` UX).
- **README** rewritten: one-command stack, host-TEI override, ports,
`.env`, cross-thread + cross-persona walkthrough, testing notes. Adds
`.env.example`.
## Tests
- **Deterministic E2E (CI gate):** `e2e/memory-learning.spec.ts` +
aimock fixtures — agent LLM served by `@copilotkit/aimock` (fixtured
`recall_memory` → exception → approve tool calls) against the **real**
local memory backend; asserts a fresh thread unlocks from recalled
memory with no recording offer.
- **Real-LLM drift smoke (manual, non-gating):**
`scripts/memory-drift-smoke.mjs`.
## ⚠️ Why draft — needs a green E2E run
The Task 7 E2E is **authored + statically validated** (`playwright test
--list` compiles spec + config; fixtures/JSON/launcher all valid) but
**has not had a green run yet** — it needs `@copilotkit/aimock`
installed, the docker stack up, and the dev server in Intelligence mode
(a 4-process orchestration). Each E2E file carries a `VERIFY ON FIRST
GREEN RUN` checklist (aimock fixture schema/launch API, chat + HITL
selectors, the `sequenceIndex` ordering key). Marking draft until that
passes.
## Out of scope (deferred)
- Per-run demo reset for a repeatable public embed (user-scope memory /
periodic DB reset / dashboard control).
- Managed-Intelligence target: PRD/handoff prefer
`api.intelligence.copilotkit.ai`; this PR ships the local-vendored stack
per direction. Reconciling for the V1 website/Railway deploy is a
follow-up.
- Pre-existing demo `tsc` looseness (`page.tsx`, `copilot-context.tsx`)
— untouched.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---
## Update — booth-bundle pass (2026-06-30)
Follow-up to make the demo booth-ready and reproducible from the
CopilotKit repo by teammates. Four commits on top of the above:
- **`/api/v1/dev/reset` now clears durable memory**, not just the
transaction store — so the full *fail → teach → succeed* arc replays for
each booth visitor. New scope-complete `forgetAllMemories` helper
enumerates via a bare `GET /api/memories` (the backend `400`s on
`?scope=` filters, so a single bare GET is inherently scope-complete)
and `DELETE`s each id; the route returns
`{ok,reset:["store","memory"],forgot:N}`, or a `502` on partial failure
so a half-reset state is never silently used. Live-validated
(`forgot:2`). **This resolves the "per-run demo reset" item listed as
deferred above.**
- **Memory `kind` migrated `operational` → `procedural`** to match the
Intelligence demo branch's current schema (`semantic | episodic |
procedural`). *(Supersedes the "operational" wording earlier in this
description.)*
- **Fixed the aimock E2E launcher** — `new LLMock({ fixtures })` ignores
`options.fixtures`, so the mock was serving 0 fixtures; now registers
via `addFixtures()`.
- **E2E status:** `test:unit` green; `test:self-learning` — the Glass
Engine inspector test passes; the autonomous-recall test has a known
aimock fixture-sequencing flake (harness-only, not a demo/backend bug).
The booth relies on the manual real-LLM arc.
**Build the Intelligence backend from `david/for-162-splat-demo`, not
`main`.** Verified by building both: the demo branch boots healthy and
runs the arc; `main` crash-loops with this compose — it requires the new
`INTELLIGENCE_DEPLOYMENT_MODE=self_hosted` auth contract (rejecting the
`DEPLOYMENT_MODE` + `DEFAULT_ORGANIZATION_ID` env this compose sets) and
its memory `kind` vocabulary is `topical/episodic/operational`.
Targeting `main` is a separate migration (compose auth env + org-seed
model + `kind` taxonomy). A full local-setup runbook exists for
teammates (internal Notion).
---
## Update — CR pass (2026-07-03)
A 7-agent review-and-fix loop converged (2 rounds + a bucket-(c)
promotion audit; 0 mandatory findings remaining). Four fixes landed,
each its own commit; `tsc`, unit tests (41/41), eslint, and `next build`
all green:
- **Recorder feed** — `handleApprove` in `transactions-list.tsx` and
`pending-approvals-chat.tsx` called `logStep()` *before*
`beginRecording()`, so the "Approved the charge" line was silently
dropped (`logStep` no-ops when inactive; `beginRecording` then resets
the feed). Reordered to `beginRecording → logStep → endRecording`; added
`recording-context.test.tsx` with red-green coverage.
- **Docs** — corrected the memory-learning spec path `tests/e2e/` →
`e2e/` (README, `.env.example`, smoke script), and the
top-of-description memory `kind` `operational` → `procedural`.
- **docker-compose header** — infra host-port comments corrected
`705x/706x` → the actual `715x/716x` mappings.
Deferred (pre-existing, out of this PR's subject; candidates for a
follow-up): the dual/divergent "current page" agent readable
(`copilot-context.tsx:96` vs `layout.tsx:147`), and the `PUT
/api/v1/transactions/[id]` error-swallow returning `undefined`.
---
## Update — migration to Intelligence `main` + presenter reset +
Apple-Silicon fresh-setup (2026-07-06)
This branch now targets Intelligence **`main`** (the earlier sections
assumed the `david/for-162-splat-demo` branch). Changes on top of the
above:
**Migration to `main`'s contract**
- **Compose auth:** `INTELLIGENCE_DEPLOYMENT_MODE=self_hosted` (legacy
`DEPLOYMENT_MODE` / `DEFAULT_ORGANIZATION_ID` removed — `main`'s
`loadAuthEnv` rejects them).
- **Memory `kind` renamed `semantic|procedural` →
`topical|operational`** to match `main`'s closed enum (`topical |
episodic | operational`). ⚠️ *This supersedes the earlier "migrated
operational → procedural" note (that was for the old branch): the
over-limit procedure is now **`operational`**, general facts
**`topical`**.*
- **Self-hosted memory is license-gated on `main`.** New
`scripts/mint-dev-license.mjs` (`pnpm mint-dev-license --write`) signs
an enterprise dev license (`features.memory=true`) with a throwaway key
and bakes the public half via `BAKED_LICENSE_KEYS_JSON`, which the local
(unbaked) app-api trusts. Drives the signer from the private
Intelligence source via `INTELLIGENCE_REPO` — no signing code vendored
into this public repo. Managed-Intelligence users instead supply a
CopilotKit-issued token and omit the baked key.
**Presenter reset button** (finishes the deferred "per-run demo reset",
now UI-driven)
- New `PRESENTER_RESET_ENABLED` flag gates **both** a sidebar reset
button **and** the `/api/v1/dev/reset` endpoint (403/hidden by default —
safe-off for public hosts).
- Full clean slate: re-seeds transactions + forgets memory for **both**
seeded personas (`SEEDED_USER_IDS`), with partial-progress reporting on
a mid-clear failure. TDD; spec + code-quality reviewed.
**Apple-Silicon fresh-setup fix**
- The bundled amd64 `tei` crash-loops under arm64 emulation (Candle
backend unavailable → ONNX/ORT backend → 404 on ONNX files
`Qwen3-Embedding-0.6B` doesn't publish). Gated it behind the
`cpu-fallback` profile (a bare `up` skips it), and added `run-demo.sh`
that runs a native Metal TEI on Apple Silicon (same 1.9.3 + model →
byte-identical embeddings) and the docker `tei` on amd64/CI. README
diagnosis corrected (was mis-attributed to OOM).
**Verification:** 55 unit tests green, `tsc` clean, `eslint` clean. A
from-scratch run (Intelligence `main` rebuild + clean `pnpm install` +
native TEI) was validated end-to-end — memory save/recall through the
native embedder, teach→recall arc, and reset all working. The
deterministic aimock e2e still has the known fixture-sequencing flake
(see follow-up comment below).
Tyler Slaton