Add --ci flag to eval orchestrator that skips Docker lifecycle and
assumes services are already running. Add ci-native-eval.sh helper
that installs deps, starts next dev + agent servers natively, health-
waits, then runs showcase eval --ci. Fix on-demand E2E workflow with
langgraph-python support and agent-type detection.
Picks up the router fix from CopilotKit/aimock#148 — `toolCallId` matchers
now only fire when the tool message is the *last* message in the request,
preventing stale tool_call_ids from history shadowing `userMessage`
matchers on new user turns.
Surfaced as: in beautiful-chat, clicking a second suggestion replayed the
prior chart's "Pie chart rendered above…" content fixture instead of
producing a new tool call. Once Railway rebuilds `ghcr.io/copilotkit/aimock:latest`
and restarts the service, demos will pick up the fix automatically.
- Refresh `pnpm-lock.yaml` resolutions (workspace `@copilotkit/runtime` devDep)
- Refresh `showcase/scripts/package-lock.json` to 1.16.4
- Bump the floor in `test_e2e-showcase-on-demand.yml` from `^1.14.3` → `^1.16.4`
so the `/test-aimock` PR-comment workflow always installs a build that
contains the fix
Pre-existing drift on main: count held at 134, hash shifted (one
FAIL healed, another regressed). Captured the new sorted-FAIL hash
locally with the same algorithm CI uses (sort -u | shasum -a 256)
and updated showcase/scripts/fail-baseline.json to match.
Spring-AI's DisplayFlightTool.java snippets bundled with language=text
because detectLanguage()'s extension map didn't list .java; the docs
Snippet renderer then fell back to highlightAuto, which produced no
hljs token classes. Same issue lurked for any future .xml file. Both
extensions added.
- claude-sdk-typescript manifest: G3's headless-complete and a2ui-fixed-schema entries pointed at -agent.ts paths but the actual files are -prompt.ts. Bundle-demo-content blew up on the missing files.
- generate-catalog test: update reference and crewai-crews wired-count assertions for the post-blitz state. Multiple integrations now tie at 39 wired; auto-detection picks the alphabetical winner (langgraph-fastapi) over langgraph-python. crewai wired count moved to 37.
- generate-registry: cross-validate not_supported_features against features list
- catalog-types: widen manifestation union to include 'starter'
- claude-sdk-python: drop invalid 'declarative-schema' from generative_ui enum
- crewai-crews: remove duplicate animated_preview_url key in mcp-apps demo
- built-in-agent + pydantic-ai: REASONING_MODEL env-var fallback so deployers can swap if gpt-5.2/gpt-5 isn't available
Adds a fourth cell status, "unsupported", to the showcase dashboard for
features whose framework cannot architecturally support them (e.g.,
no graph-interrupt API, no MCP tool runtime). This is distinct from
"unshipped" (just unbuilt).
- Manifest schema gains optional `not_supported_features: string[]`.
- generate-registry.ts: determineCellStatus checks `not_supported_features`
first; CatalogCell/CatalogMetadata gain `unsupported`; max_depth=0 for
unsupported cells; parity tier is computed against the supportable subset
of the reference (unsupported features no longer drag a framework's tier
down for gaps it legitimately cannot fill).
- depth-utils.ts: deriveDepth returns D0 with no regression for
unsupported cells.
- DepthChip: renders a dashed gray border + 🚫 glyph with
"Not supported by this framework" tooltip; data-status attribute
distinguishes unshipped vs unsupported.
- CoverageBar gains diagonal-stripe gray segment for unsupported.
- StatsBar / AdaptiveStatsBar / CellsView: separate "Unsupported" chip.
- CellMatrix gaps filter excludes unsupported cells (not work to do).
- Tests: cover unsupported in DepthChip, depth-utils, cell-matrix, and
catalog generator metadata/max_depth assertions.
## Summary
- Guard `preferences-card.tsx` against undefined `value.interests`
across all 17 integrations — crash prevented D5 shared-state probes from
completing
- Add missing D5 demo entries (`shared-state-read-write`, `hitl-in-app`,
`hitl-in-chat`, etc.) and feature IDs to manifests across 14
integrations — D5 probes couldn't discover features without these
entries
- Bump `@copilotkit/aimock` to v1.16.1 (turnIndex/sequenceIndex support
required for D5 fixture matching)
- Update snapshot test assertions for langgraph-python (38→40
features/demos) and crewai-crews (30→32 wired)
- Fix `docker-compose.local.yml`: mount D5 fixture volumes, add `--host
0.0.0.0`, replace curl healthcheck with Node fetch
- Delete redundant `scripts/dev-local.sh` (superseded by `bin/showcase`
CLI)
## Why
D5 probes were failing across all integrations due to three root causes:
(1) preferences-card crash on undefined interests, (2) missing manifest
entries preventing D5 probe discovery, and (3) aimock pinned at v1.10.0
missing turnIndex support. Local D5 verified GREEN on langgraph-python
(all 11 feature types pass).
## Test plan
- [ ] CI Validate Showcase passes (snapshot counts updated)
- [ ] CI build passes (aimock version bump)
- [ ] Local D5 probes pass for langgraph-python
- Mount D5/smoke/feature-parity fixtures into aimock via volumes
- Add --host 0.0.0.0 so other containers can reach aimock
- Switch healthcheck from curl to node fetch (aimock image has no curl)
- Delete dev-local.sh (redundant with bin/showcase CLI)
Needed for turnIndex/sequenceIndex fixture matching in D5 multi-turn
conversations. Updated in both packages/runtime (devDep) and
showcase/scripts (dep).
Remove residual showcase-starter-* references now that starters are
decommissioned:
- Delete showcase/ops/config/alerts/smoke-red-tick.yml (starter-only
alert rule with kind: starter filter)
- Delete showcase/ops/test/fixtures/rules/valid/smoke-red-tick.yml
(test fixture mirroring the deleted alert rule)
- Update showcase/ops/config/probes/smoke.yml comments to remove
showcase-starter-ag2 example and 34-service count
- Update showcase/ops/scripts/test-notify-ops-jq.sh to remove starter
build job fixtures and starter service names from jq partition test
- Update showcase/scripts/verify-railway-image-refs.ts to remove
showcase-starter-<slug> from canonical shape comment
## Summary
- Delete `test_smoke-starter-deployed.yml` workflow (runs every 6h,
would flood Slack with 17 red alerts once starter Railway services are
torn down)
- Remove `starter:` blocks from all 17 integration manifest.yaml files
(stale `demo_url` pointing to `showcase-starter-*` Railway services)
- Clean stale starter references from CI workflow comments
- Remove `"starter"` from dashboard's `manifestation` type union and
dead starter-handling code in cell-matrix
- Fix useLiveStatus test to match INITIAL_CAP=2000 (pre-existing from
#4362)
## Why
PR #4351 merged packages/starters into a unified integrations/
directory, eliminating starters as separate deployable units. The 17
`showcase-starter-*` Railway services are being deprovisioned. These
references must be cleaned up BEFORE teardown to prevent false alerts
and stale data propagation.
## Test plan
- [ ] Dashboard tests pass (15/15 useLiveStatus, depth-utils,
cell-matrix)
- [ ] No `showcase-starter-` URLs remain in generated data files
- [ ] No `starter:` blocks remain in manifest.yaml files
- [ ] CI green
The ADK source pin moved 1.55.2 → 1.56.4, so the corresponding
[FAIL] line in showcase validate-pins now reads "Dojo=1.56.4".
Same FAIL count (129), different hash. Updates the baseline so
CI accepts the new (still-failing-but-known) state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
remove dead starter scripts
Update probe configs, fixtures, and test drivers to reference
showcase/integrations/ instead of showcase/packages/.
Remove obsolete generate-starters test, starter-consistency
test, validate-workflow-starters, and rebuild-coord.sh.
Delete the pre-generated starters/ directory tree (previously
synced from packages/ by generate-starters.ts). Add
extract-starter.ts which produces a starter tarball on demand
from any integration. Move shared starter template files to
showcase/shared/starter-template/.
google-adk:
- tool-rendering: use v2 `parameters` instead of `args` in render
- hitl-in-app: align tool name to request_user_approval with
message/context params matching aimock fixtures
- headless-simple: add data-message-role attributes for probe
selector cascade
spring-ai:
- Derive SPRING_AI_OPENAI_BASE_URL from OPENAI_BASE_URL in
entrypoint.sh, stripping /v1 suffix
agno:
- Rename book_call parameter from attendee to name to match fixture
llamaindex:
- Add book_call frontend tool to agent router for hitl-text-input
1. Add missing @json-render/core and @json-render/react deps to
google-adk package.json (Docker build failed on byoc-json-render
demo imports)
2. Update catalog test expectations to match actual generated counts
(wired 482, crewai-crews 28, LGP 37 wired + 2 unshipped)
3. Regenerate google-adk starter from template to eliminate drift
detected by the starter consistency check
24 files in two byte-identical families across 12 integration trees:
- 12 `integrations/<fw>/threads.mdx` — 7 lines each, all md5 0371508d…,
body is just `<Threads />`.
- 12 `integrations/<fw>/premium/self-hosting.mdx` — 8 lines each, all
md5 06d17f98…, body is just `<SelfHosting />`.
Both render shared snippets the root-level pages already render. The
`buildFrameworkOverridesNav` filter dropped them from the merged
sidebar (root wins), and the framework-scoped router falls back to
root MDX when no per-framework override exists, so deletion is
invisible to users — `/built-in-agent/threads`,
`/langgraph-python/premium/self-hosting`, etc. continue to render
identically. Verified: each URL still returns 200 with full body
(340–465 KB) by curling the dev server before commit.
Add path-exclusion patterns to sync-docs-from-main.ts so future
upstream syncs don't resurrect them. Same shape as the BIA branch's
`(other)/` exclusion. Sample-tested the regexes against expected-
exclude and expected-keep paths — all classify correctly.
Audit covered every `integrations/<fw>/*.mdx` with body ≤ 2 non-blank
lines; no false-negative stubs of other shapes.
## Summary
Replace the hardcoded 200-line INTEGRATIONS array in
`integration-smoke.spec.ts` with a 6-line derivation from
`registry.json`. New demos automatically appear in smoke tests when
manifests are updated.
- Delete hardcoded Integration interface + 17-entry array
- Derive slug, name, backendUrl, deployed, hasToolRendering, demos from
registry
- Add unit tests verifying derivation correctness + regression guard
against re-hardcoding
## Test plan
- [x] New vitest tests pass (7/7)
- [x] Existing showcase build pipeline tests pass (1183/1183 across 26
files)
- [ ] CI green
Add missing feature IDs (hitl, hitl-in-chat-booking) to langgraph-python
manifest so it reclaims reference status from langgraph-fastapi. Update
catalog test expectations and validate-pins fail-baseline.json hash/count
after PR #4287 dependency changes.
Replace the 200-line hardcoded INTEGRATIONS array in integration-smoke.spec.ts
with a 6-line derivation from registry.json. New demos automatically appear in
smoke tests when manifests are updated — no manual maintenance needed.
Every framework under integrations/<fw>/ shipped its own copy of
contributing/ + telemetry/ content that was byte-identical (or
trivially divergent — a stray "cd" path, a legacy CLI name) to the
canonical copy at root (other)/. The duplicates were stale sync
artifacts with no framework-specific content: "how to contribute to
CopilotKit" and "how to configure telemetry" don't vary by agent
framework.
Beyond disk clutter (~2.5k lines across 8 frameworks), the duplicates
surfaced as a "Other" group nested under the framework-scoped
sidebar section whenever the merged nav built from meta.json — a
second copy of root's own "Other" section at the bottom of the
sidebar. Deleting the trees makes that UI bug disappear without any
filter patching.
Scope:
- Remove integrations/<fw>/(other)/ trees for ag2, agno, aws-strands,
crewai-flows, langgraph, llamaindex, mastra, microsoft-agent-framework.
- Drop ---Other--- + ...(other) entries from each framework's meta.json.
- Add a path-exclusion filter in sync-docs-from-main.ts so the next
sync run doesn't resurrect the subtrees when upstream edits touch
them. Upstream keeps its copies (removing them there means touching
all 13 parallel framework trees, out of scope for this branch).
Jordan Ritter