The validator was not considering the aimock match.toolName field when
checking for drift. When a fixture has toolName set, aimock only fires
it for agents that register that tool -- skip demos that don't have it.
The COMPOSE_CMD in apply_isolation was missing --project-name, so Docker
Compose would infer the project name from the directory and collide with
the base showcase stack (and other isolated runs). Adding --project-name
ensures containers, networks, and volumes are fully scoped to the
isolation slot.
apply_isolation previously mutated docker-compose.local.yml and
local-ports.json in-place with .iso-bak backups. If the process crashed
the originals stayed corrupted with +200 port offsets, breaking all
subsequent showcase commands.
Now writes modified copies to a temp directory and overrides
COMPOSE_FILE/PORTS_FILE shell variables so downstream code reads from
the overlay. Originals are never touched. restore_isolation just removes
the temp dir.
Also replaces hardcoded +200 port offset with atomic mkdir-based slot
allocation. Two parallel --isolate runs now get different port ranges
(slot 0 = +200, slot 1 = +400, etc.) instead of colliding on the same
ports. Container names include the slot number for collision-free Docker
naming. Stale slots from crashed runs are reclaimed via PID liveness
checks and a 2-hour age fallback.
TS harness files (config.ts, lifecycle.ts, doctor.ts) honor
LOCAL_PORTS_FILE env var so they read offset ports from the temp overlay.
PR #4562 removed the referenceCount variable but left a log line
referencing it, causing ReferenceError at build time. The dashboard
never rebuilt with langgraph-python as REF because the build crashed.
The auto-detect logic picked whichever integration had the most wired
features, with ties broken alphabetically. This caused ag2 to appear
as the reference when it matched langgraph-python's feature count.
langgraph-python is always the gold standard reference.
When passed to `showcase test`, creates an isolated Docker Compose
project with offset ports (+200) and renamed containers, allowing
multiple agents/sessions to run showcase tests simultaneously without
container conflicts.
Usage:
showcase test agno --d5 --isolate # auto-names isolate-<PID>
showcase test agno --d5 --isolate d5verify # explicit name
docs-only features (e.g. cli-start) exist purely for documentation
coverage tracking -- they have no route, no depth probes, and no
health signals. The catalog metadata was counting their 18 stub cells
in the headline wired/stub/unshipped/unsupported breakdown, inflating
the total and making the stats bar misleading.
Exclude docs-only cells from the headline counts. A new docs_only
field tracks the excluded count separately so the invariant
(wired + stub + unshipped + unsupported + docs_only == cells.length)
holds.
Before: total_cells=720 wired=673 stub=18 unsupported=29
After: total_cells=702 wired=673 stub=0 unsupported=29 docs_only=18
Add --ci flag to eval orchestrator that skips Docker lifecycle and
assumes services are already running. Add ci-native-eval.sh helper
that installs deps, starts next dev + agent servers natively, health-
waits, then runs showcase eval --ci. Fix on-demand E2E workflow with
langgraph-python support and agent-type detection.
Picks up the router fix from CopilotKit/aimock#148 — `toolCallId` matchers
now only fire when the tool message is the *last* message in the request,
preventing stale tool_call_ids from history shadowing `userMessage`
matchers on new user turns.
Surfaced as: in beautiful-chat, clicking a second suggestion replayed the
prior chart's "Pie chart rendered above…" content fixture instead of
producing a new tool call. Once Railway rebuilds `ghcr.io/copilotkit/aimock:latest`
and restarts the service, demos will pick up the fix automatically.
- Refresh `pnpm-lock.yaml` resolutions (workspace `@copilotkit/runtime` devDep)
- Refresh `showcase/scripts/package-lock.json` to 1.16.4
- Bump the floor in `test_e2e-showcase-on-demand.yml` from `^1.14.3` → `^1.16.4`
so the `/test-aimock` PR-comment workflow always installs a build that
contains the fix
Pre-existing drift on main: count held at 134, hash shifted (one
FAIL healed, another regressed). Captured the new sorted-FAIL hash
locally with the same algorithm CI uses (sort -u | shasum -a 256)
and updated showcase/scripts/fail-baseline.json to match.
Spring-AI's DisplayFlightTool.java snippets bundled with language=text
because detectLanguage()'s extension map didn't list .java; the docs
Snippet renderer then fell back to highlightAuto, which produced no
hljs token classes. Same issue lurked for any future .xml file. Both
extensions added.
- claude-sdk-typescript manifest: G3's headless-complete and a2ui-fixed-schema entries pointed at -agent.ts paths but the actual files are -prompt.ts. Bundle-demo-content blew up on the missing files.
- generate-catalog test: update reference and crewai-crews wired-count assertions for the post-blitz state. Multiple integrations now tie at 39 wired; auto-detection picks the alphabetical winner (langgraph-fastapi) over langgraph-python. crewai wired count moved to 37.
- generate-registry: cross-validate not_supported_features against features list
- catalog-types: widen manifestation union to include 'starter'
- claude-sdk-python: drop invalid 'declarative-schema' from generative_ui enum
- crewai-crews: remove duplicate animated_preview_url key in mcp-apps demo
- built-in-agent + pydantic-ai: REASONING_MODEL env-var fallback so deployers can swap if gpt-5.2/gpt-5 isn't available
Adds a fourth cell status, "unsupported", to the showcase dashboard for
features whose framework cannot architecturally support them (e.g.,
no graph-interrupt API, no MCP tool runtime). This is distinct from
"unshipped" (just unbuilt).
- Manifest schema gains optional `not_supported_features: string[]`.
- generate-registry.ts: determineCellStatus checks `not_supported_features`
first; CatalogCell/CatalogMetadata gain `unsupported`; max_depth=0 for
unsupported cells; parity tier is computed against the supportable subset
of the reference (unsupported features no longer drag a framework's tier
down for gaps it legitimately cannot fill).
- depth-utils.ts: deriveDepth returns D0 with no regression for
unsupported cells.
- DepthChip: renders a dashed gray border + 🚫 glyph with
"Not supported by this framework" tooltip; data-status attribute
distinguishes unshipped vs unsupported.
- CoverageBar gains diagonal-stripe gray segment for unsupported.
- StatsBar / AdaptiveStatsBar / CellsView: separate "Unsupported" chip.
- CellMatrix gaps filter excludes unsupported cells (not work to do).
- Tests: cover unsupported in DepthChip, depth-utils, cell-matrix, and
catalog generator metadata/max_depth assertions.
## Summary
- Guard `preferences-card.tsx` against undefined `value.interests`
across all 17 integrations — crash prevented D5 shared-state probes from
completing
- Add missing D5 demo entries (`shared-state-read-write`, `hitl-in-app`,
`hitl-in-chat`, etc.) and feature IDs to manifests across 14
integrations — D5 probes couldn't discover features without these
entries
- Bump `@copilotkit/aimock` to v1.16.1 (turnIndex/sequenceIndex support
required for D5 fixture matching)
- Update snapshot test assertions for langgraph-python (38→40
features/demos) and crewai-crews (30→32 wired)
- Fix `docker-compose.local.yml`: mount D5 fixture volumes, add `--host
0.0.0.0`, replace curl healthcheck with Node fetch
- Delete redundant `scripts/dev-local.sh` (superseded by `bin/showcase`
CLI)
## Why
D5 probes were failing across all integrations due to three root causes:
(1) preferences-card crash on undefined interests, (2) missing manifest
entries preventing D5 probe discovery, and (3) aimock pinned at v1.10.0
missing turnIndex support. Local D5 verified GREEN on langgraph-python
(all 11 feature types pass).
## Test plan
- [ ] CI Validate Showcase passes (snapshot counts updated)
- [ ] CI build passes (aimock version bump)
- [ ] Local D5 probes pass for langgraph-python
- Mount D5/smoke/feature-parity fixtures into aimock via volumes
- Add --host 0.0.0.0 so other containers can reach aimock
- Switch healthcheck from curl to node fetch (aimock image has no curl)
- Delete dev-local.sh (redundant with bin/showcase CLI)
Needed for turnIndex/sequenceIndex fixture matching in D5 multi-turn
conversations. Updated in both packages/runtime (devDep) and
showcase/scripts (dep).
Remove residual showcase-starter-* references now that starters are
decommissioned:
- Delete showcase/ops/config/alerts/smoke-red-tick.yml (starter-only
alert rule with kind: starter filter)
- Delete showcase/ops/test/fixtures/rules/valid/smoke-red-tick.yml
(test fixture mirroring the deleted alert rule)
- Update showcase/ops/config/probes/smoke.yml comments to remove
showcase-starter-ag2 example and 34-service count
- Update showcase/ops/scripts/test-notify-ops-jq.sh to remove starter
build job fixtures and starter service names from jq partition test
- Update showcase/scripts/verify-railway-image-refs.ts to remove
showcase-starter-<slug> from canonical shape comment
## Summary
- Delete `test_smoke-starter-deployed.yml` workflow (runs every 6h,
would flood Slack with 17 red alerts once starter Railway services are
torn down)
- Remove `starter:` blocks from all 17 integration manifest.yaml files
(stale `demo_url` pointing to `showcase-starter-*` Railway services)
- Clean stale starter references from CI workflow comments
- Remove `"starter"` from dashboard's `manifestation` type union and
dead starter-handling code in cell-matrix
- Fix useLiveStatus test to match INITIAL_CAP=2000 (pre-existing from
#4362)
## Why
PR #4351 merged packages/starters into a unified integrations/
directory, eliminating starters as separate deployable units. The 17
`showcase-starter-*` Railway services are being deprovisioned. These
references must be cleaned up BEFORE teardown to prevent false alerts
and stale data propagation.
## Test plan
- [ ] Dashboard tests pass (15/15 useLiveStatus, depth-utils,
cell-matrix)
- [ ] No `showcase-starter-` URLs remain in generated data files
- [ ] No `starter:` blocks remain in manifest.yaml files
- [ ] CI green
The ADK source pin moved 1.55.2 → 1.56.4, so the corresponding
[FAIL] line in showcase validate-pins now reads "Dojo=1.56.4".
Same FAIL count (129), different hash. Updates the baseline so
CI accepts the new (still-failing-but-known) state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
remove dead starter scripts
Update probe configs, fixtures, and test drivers to reference
showcase/integrations/ instead of showcase/packages/.
Remove obsolete generate-starters test, starter-consistency
test, validate-workflow-starters, and rebuild-coord.sh.
Delete the pre-generated starters/ directory tree (previously
synced from packages/ by generate-starters.ts). Add
extract-starter.ts which produces a starter tarball on demand
from any integration. Move shared starter template files to
showcase/shared/starter-template/.
google-adk:
- tool-rendering: use v2 `parameters` instead of `args` in render
- hitl-in-app: align tool name to request_user_approval with
message/context params matching aimock fixtures
- headless-simple: add data-message-role attributes for probe
selector cascade
spring-ai:
- Derive SPRING_AI_OPENAI_BASE_URL from OPENAI_BASE_URL in
entrypoint.sh, stripping /v1 suffix
agno:
- Rename book_call parameter from attendee to name to match fixture
llamaindex:
- Add book_call frontend tool to agent router for hitl-text-input
1. Add missing @json-render/core and @json-render/react deps to
google-adk package.json (Docker build failed on byoc-json-render
demo imports)
2. Update catalog test expectations to match actual generated counts
(wired 482, crewai-crews 28, LGP 37 wired + 2 unshipped)
3. Regenerate google-adk starter from template to eliminate drift
detected by the starter consistency check
24 files in two byte-identical families across 12 integration trees:
- 12 `integrations/<fw>/threads.mdx` — 7 lines each, all md5 0371508d…,
body is just `<Threads />`.
- 12 `integrations/<fw>/premium/self-hosting.mdx` — 8 lines each, all
md5 06d17f98…, body is just `<SelfHosting />`.
Both render shared snippets the root-level pages already render. The
`buildFrameworkOverridesNav` filter dropped them from the merged
sidebar (root wins), and the framework-scoped router falls back to
root MDX when no per-framework override exists, so deletion is
invisible to users — `/built-in-agent/threads`,
`/langgraph-python/premium/self-hosting`, etc. continue to render
identically. Verified: each URL still returns 200 with full body
(340–465 KB) by curling the dev server before commit.
Add path-exclusion patterns to sync-docs-from-main.ts so future
upstream syncs don't resurrect them. Same shape as the BIA branch's
`(other)/` exclusion. Sample-tested the regexes against expected-
exclude and expected-keep paths — all classify correctly.
Audit covered every `integrations/<fw>/*.mdx` with body ≤ 2 non-blank
lines; no false-negative stubs of other shapes.
Jordan Ritter