Commit Graph

650 Commits

Author SHA1 Message Date
Alem Tuzlak 6426281b74 test(showcase): regression coverage for open-gen-ui-advanced sandbox-function round-trips
The previous fixture regression (HTML+CSS only, no jsFunctions) slipped
past CI because the e2e suite only asserted "iframe mounts with non-empty
srcdoc" — which passes whether or not the iframe is interactive. Adds
two layers of guard so the same regression cannot land silently:

1. showcase/scripts/__tests__/open-gen-ui-advanced-fixtures.test.ts
   (vitest, runs in showcase_validate on every PR): asserts each of the
   three interactive fixture entries in d5-all.json ships jsFunctions
   referencing the matching host bridge (evaluateExpression / notifyHost).
   Catches "someone removed jsFunctions" at PR-time with no
   infrastructure dependencies.

2. showcase/integrations/langgraph-python/tests/e2e/open-gen-ui-advanced.spec.ts
   (playwright, runs in test_e2e-showcase-on-demand): adds three
   round-trip tests that drive the in-iframe controls and assert the
   host-side handler ran by capturing its console.log + verifying the
   iframe output element reflects the host response. Catches "the
   renderer fails to inject jsFunctions into the sandbox" too.

The e2e tests also switch the existing smoke tests off pill-click and
onto a textarea-driven fill+Enter path, following the same precedent as
commit 15db0bbf3 (gen-ui-headless-complete) — chip mounts diverge
between EmptyState and SuggestionBar surfaces, and Playwright's pill
click races React hydration. Using [data-testid="copilot-chat-textarea"]
with an explicit click + waitForLoadState("networkidle") makes the
suite reliable end-to-end (7/7 passing locally against the aimock-driven
stack).
2026-05-11 17:34:21 +02:00
Alem Tuzlak c99bea6670 fix(showcase/mcp-apps): route "Draw a flowchart" pill to create_view
The "Draw a flowchart" suggestion pill in the mcp-apps demo sent
"Use Excalidraw to draw a simple flowchart with three steps." which
had no matching create_view fixture in d5-all.json. aimock walked
through to feature-parity.json's `{userMessage: "steps"}` substring
fixture and returned a generic "Here is my plan..." content blurb
with no MCP tool call, so the runtime never invoked create_view, the
MCP middleware never fetched the UI resource, and the sandboxed
iframe never mounted.

Add a fixture pair in d5-all.json (and its harness mirror) keyed on
"draw a simple flowchart": turn 1 emits create_view with a three-
step Start -> Process -> End flowchart, turn 2 emits the narration
after the tool result. The distinctive substring beats the generic
feature-parity catch-alls under first-match-wins.

Adds a regression test that loads the same fixture files in the same
order as docker-compose.local.yml and asserts via aimock's matchFixture
that each mcp-apps pill routes to its create_view fixture on turn 1 and
its narration fixture on turn 2.
2026-05-11 15:21:18 +02:00
Tyler Slaton 70e2fb13c8 refactor(showcase): rename byoc-* slugs to declarative-* + sort index by manifest features
User-facing renames so the showcase reads the way a cold visitor would
expect:

- `byoc-hashbrown` → `declarative-hashbrown` (and `byoc-json-render` →
  `declarative-json-render`). The display titles already said
  "Declarative UI: …"; only the URL slugs and folder paths still
  leaked the internal BYOC ("Bring Your Own Components") jargon.
  Renamed:
    /demos/byoc-hashbrown          → /demos/declarative-hashbrown
    /demos/byoc-json-render        → /demos/declarative-json-render
    /api/copilotkit-byoc-*         → /api/copilotkit-declarative-*
    src/app/demos/byoc-*           → src/app/demos/declarative-*
    qa/byoc-*.md                   → qa/declarative-*.md
    tests/e2e/byoc-*.spec.ts       → tests/e2e/declarative-*.spec.ts
  Internal Python module names + langgraph graph IDs stay legacy
  (`byoc_hashbrown_agent.py`, `byoc_hashbrown`) — those are not
  user-facing and renaming them is a separate cross-codebase pass.
- `a2ui-fixed-schema` slug intentionally unchanged.
- Tool Rendering trio parenthetical rename (Default → Catch-all →
  Custom progression reads clearly as "how much do I customize?"):
    Tool Rendering (Default)        — unchanged
    Tool Rendering (Custom default) → Tool Rendering (Catch-all)
    Tool Rendering (Specific)       → Tool Rendering (Custom)
- `tool-rendering-reasoning-chain` cell renamed from
  "Generative UI: Rendering multiple tools" to
  "Generative UI: Tool calls + reasoning" (the demo is about combining
  reasoning + tool rendering, not about quantity of tools).
- `Open Generative UI: Default` / `Open Generative UI: Custom`
  descriptions expanded so a visitor understands how Open Generative UI
  differs from Tool Rendering (agent composes UI from a registered
  library vs. attaching a renderer to a *named* backend tool).
- Showcase index now sorts demos within each tag by `manifest.features`
  order. Previously demos appeared in manifest declaration order, which
  ignored the team's curated "polished flagship → simplest start →
  variants" arc.

Cross-cutting registry / harness / dashboard updates that fall out of
the rename:

- `shared/feature-registry.json` adds the two new IDs alongside the
  legacy `byoc-*` (so the catalog stays valid; the other 17
  integrations still declare `byoc-*` in their manifests).
- `shared/constraints.yaml` adds the new IDs to the
  generative-ui-approach allow-list.
- `scripts/__tests__/generate-catalog.test.ts` updates the cell-count
  expectations (45 features × 18 integrations = 810; 792 after docs-
  only exclusion; 45 LGP cells = 38 wired + 1 stub + 6 unshipped).
- Harness probe `d5-byoc.ts` + `d5-byoc.test.ts` now route both slug
  families through `preNavigateRoute` and exercise the new branches.
- `d5-feature-mapping.ts` and `shell-dashboard/live-status.ts` mirror
  the dual-ID mapping so both legacy and renamed slugs roll up under
  the same `byoc` D5 featureType.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 15:16:02 -07:00
Tyler Slaton 80a7f9af0e feat(showcase): align demo names + add Show Deprecated toggle
Two related changes that bring the dashboard's gold-standard view in
line with the desired naming convention and surface deprecated rows
behind a toggle (instead of hiding them at catalog generation).

## Naming alignment

Applied 28 renames in feature-registry.json + 20 in LGP manifest per
the user-provided mapping. Highlights:

- "Pre-Built CopilotChat" -> "Pre-Built: CopilotChat"
- "Headless Chat (Simple/Complete)" -> "Headless UI: Simple/Complete"
- "Multi-modal / File Uploads" -> "Attachements" (intentional spelling)
- "Controlled Gen-UI (Display)" -> "Generative UI: useComponent"
- "In-Chat HITL (use*)" -> "Human In/in the Loop: In-chat / Interrupts"
- "Headless Interrupt" -> "Human in the Loop: Headless Interrupts"
- "Declarative Generative UI (A2UI - *)" -> "Declarative UI: */* A2UI"
- "Fully Open-Ended Generative UI" -> "Open Generative UI: Default"
- "Tool Rendering ..." -> "Generative UI: Tool Rendering (...)"
- "Tool Rendering + Reasoning Chain" -> "Generative UI: Rendering multiple tools"
- "Agentic Generative UI ..." -> "Generative UI: Agent State"
- "Frontend Tools (...)" -> "Frontend Tools: ..."
- "Shared State (...)" -> "Shared State: ..."
- "State Streaming" -> "Shared State: Streaming"
- "Readonly State (Agent Context)" -> "Shared State: Frontend Context"
- "BYOC Hashbrown <-> json-render" -- labels intentionally swapped per
  user instruction (demos were historically reversed; new labels
  reflect what they actually do).

LGP manifest demos[].name updated to match feature-registry names so
the dojo and dashboard surface the same human-readable label.

## Show Deprecated toggle (feature-grid.tsx)

Added a checkbox in the matrix header -- default OFF -- that filters
feature rows where `feature.deprecated === true`. Toggle ON shows all
deprecated features across all integrations (audit trail); toggle OFF
hides those rows entirely so the gold-standard view stays clean.

Reverted the catalog-side filter from PR #4744 (which dropped LGP
cells for deprecated features at catalog-generation time). Now the
catalog emits cells uniformly for all (integration x feature) pairs,
and visibility is controlled at the dashboard layer. Toggling on
shows complete cross-integration data without missing-cell artifacts.

Affects 4 features marked deprecated:true in feature-registry.json:
agentic-chat-reasoning, hitl, hitl-in-chat-booking,
reasoning-default-render.

LGP cell count: back to 43 (38 wired + 1 stub + 4 unshipped). The 4
unshipped rows are hidden by default; toggle to surface them.

Tests: 18/18 catalog tests + 1588/1588 harness vitest passing.
validate-fixture-tool-surface clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 20:06:39 -07:00
github-actions[bot] 80eb7a12ad style: auto-fix formatting 2026-05-09 02:02:47 +00:00
Tyler Slaton 1ce83b8a73 fix(showcase): close 2 D5 fixture gaps + filter deprecated features from gold-standard view
Three fixes that follow up on PR #4743 to bring LGP closer to fully-green
on the dashboard:

1. tool-rendering-reasoning-chain probe: was failing with
   `expected [data-testid="reasoning-block"] to mount within 30000ms`.
   Root cause: the demo's `<ReasoningBlock>` slot only mounts when a
   reasoning-role message lands in the transcript, which requires
   aimock to emit REASONING_MESSAGE_* events, which in turn requires
   the fixture's first-leg response to carry a `reasoning` field. The
   weather/Tokyo and SFO/JFK first-leg fixtures were missing it.
   Mirrors the convention documented in reasoning-display.json:2.
   Patched both source (harness/fixtures/d5/) and bundle (aimock/d5-all.json).

2. gen-ui-interrupt source fixture: the source fixture file was missing
   the resume-leg toolCallId entries that already existed in the bundle.
   Cosmetic mirror so re-bundling stays consistent. Same chip prompts +
   same toolCallIds as interrupt-headless.json (both probes share the
   same agent and aimock fixture set; the difference is the FRONTEND
   rendering — useInterrupt inline vs useHeadlessInterrupt separate-pane).

3. Dashboard gold-standard filter: 4 deprecated/legacy features
   (agentic-chat-reasoning, hitl, hitl-in-chat-booking,
   reasoning-default-render) used to render as X-marked rows in the
   LGP gold-standard dashboard view because LGP intentionally does
   NOT implement them — they were consolidated into the modern shape
   (reasoning-custom + reasoning-default; hitl-in-chat with
   useHumanInTheLoop). Other 17 integrations still serve those legacy
   demos, so we don't yank the features from feature-registry.json
   entirely. Instead: marked them `deprecated: true` and updated
   generate-registry.ts to skip emitting cells when a deprecated
   feature is unshipped for an integration. LGP cells: 43 → 39 (the
   4 deprecated rows disappear). Other integrations: unchanged
   (audit trail preserved). Catalog total: 774 → 770.

Tests:
  - 1588/1588 harness vitest passing
  - 19/19 generate-catalog + generate-registry tests passing
    (counts updated for the 4 dropped LGP cells + new deprecated-
    feature filter test)
  - validate-fixture-tool-surface clean (282 fixtures × 627 demos)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 19:00:58 -07:00
Tyler Slaton 59ecd828e5 test(showcase/scripts): update catalog+registry counts post LGP feature additions
Validate Showcase CI was failing on hardcoded count assertions in
generate-catalog.test.ts and generate-registry.test.ts after this PR
added 3 features to the LGP manifest (interrupt-headless,
shared-state-read, tool-rendering-reasoning-chain) and 1 feature to
feature-registry.json (shared-state-read — the others were already
declared).

- LGP feature/demo count: 36 → 39
- Total feature-registry features: 42 → 43
- Catalog integrated cells: 756 → 774 (43 × 18)
- LGP integrated cells: 42 → 43 (38 wired + 1 stub + 4 unshipped)
- Metadata total_cells (docs-only excluded): 738 → 756

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 17:51:54 -07:00
Alem Tuzlak f8c711d2bb feat(showcase): aimock record/replay infra for deterministic D5 fixtures
Adds a record/replay loop on top of the existing local Docker stack so
each langgraph-python D5 demo can capture real-LLM responses once and
replay them deterministically thereafter. One fewer source of D5 flake
per demo: no more 'tests pass with real OpenAI / fail with aimock'
because the prompts have no fixture coverage.

What lands here:

- showcase/docker-compose.record.yml — overlays aimock with --record
  + --provider-openai/anthropic and a writable mount for the recording
  dir. Drops the baseline d5-all.json/feature-parity.json/smoke.json
  loads so prompts that already match a stale fixture can still proxy
  through to the real provider.
- showcase/docker-compose.replay.yml — same writable mount, no
  --record, no provider URLs; layers the per-demo fixtures alongside
  the baseline ones for normal probe runs.
- showcase/scripts/record-d5-fixtures.mjs — orchestrator. For each
  demo (catalog feature ID), drops any prior consolidated <slug>.json,
  restarts aimock to clear in-memory recorded fixtures, snapshots
  recorded/, runs the d5 probe through pnpm exec tsx, then merges
  every per-call file written under recorded/ into a single
  showcase/aimock/d5-recorded/<slug>.json (one fixture per LLM turn,
  in chronological order).
- showcase/aimock/d5-recorded/<slug>.json × 6 — initial recordings
  for the still-red bucket-C cells: beautiful-chat (8), gen-ui-interrupt
  (2), gen-ui-tool-based (1), headless-complete (1), reasoning-custom
  (1), tool-rendering-default-catchall (2). 15 fixtures total.
- showcase/aimock/d5-recorded/.gitignore — keeps the per-call
  recorded/ scratch dir out of the tree (orchestrator deletes per-call
  files after consolidation, this just guards against re-runs).

D5 impact: replaying the recordings flips
d5:langgraph-python/gen-ui-custom (catalog gen-ui-tool-based) from red
to green deterministically. The other five demos still fail their UI
or assertion-side checks, but their LLM-side responses are now fixed,
so the remaining work is probe/UI fixes against a stable baseline
rather than flake hunting.

Aimock recorder requires a one-line patch (turnIndex + hasToolResult
on each recorded fixture's match) for multi-turn flows to record
correctly. Upstream fix proposed for @copilotkit/aimock; until it
ships, the orchestrator probes for the patch and aborts loudly with
the missing-fields message rather than silently producing single-turn
fixtures. See the script's header comment for the exact patch payload.
2026-05-08 14:57:05 +02:00
Alem Tuzlak bbe128ca65 fix(showcase): repair langgraph-python manifest highlight paths and bump test snapshots
Three highlight paths in langgraph-python's manifest pointed at files
that don't exist after the PR #4694 reorganization:
  - hitl-in-chat → src/agents/hitl_in_chat.py (actually hitl_in_chat_agent.py)
  - chat-slots → custom-welcome-screen.tsx (file doesn't exist; use slot-wrappers.tsx)
  - mcp-apps → copilotkit-mcp-apps/route.ts (actually .../[[...slug]]/route.ts)

The bundler walks every highlight at build time; one missing path aborts
the whole CI step. Fix all three.

Test snapshot counts in generate-catalog and generate-registry hardcoded
40 features / 720 cells / 702 total. With the two new feature IDs added
to the registry (reasoning-default + reasoning-custom), counts shift to
42 / 756 / 738; the LGP-specific cell distribution moved from
39 wired + 1 stub + 0 unshipped to 35 wired + 1 stub + 6 unshipped, and
the registry-side LGP feature/demo count drops to 36 (PR #4694 trimmed
4 items from the manifest's features list).
2026-05-07 14:30:45 +02:00
Alem Tuzlak 45cc89dc99 chore(showcase): ratchet validate-pins baseline 97→98
Pre-existing drift on main (mastra/ms-agent/llamaindex/pydantic-ai/strands)
that the langgraph-python Dockerfile fix is unrelated to. Bump the
baseline in this PR to unblock CI.
2026-05-07 14:03:39 +02:00
Sam Julien e5715cbee0 Exclude mcp-server-setup snippet from upstream sync (PDX-117) 2026-05-06 16:21:21 -07:00
Alem Tuzlak 30666298da chore(shell-docs): sync upstream content (AgentCore + threads + misc) (#4514)
## Summary

Cleans up shell-docs after PR #4521 reintroduced content earlier PRs
intentionally retired, adds a Deploy section with an inlined AWS
AgentCore guide using a custom command-tabs component, and hardens the
sync script against future regressions of the same shape.

## Changes

### Deploy section + AgentCore page (custom UX)
- New `<AgentCoreCommandTabs />` component — framework-aware command
tabs with hljs highlighting and copy buttons, built on shell-docs's own
`<Tabs>`/`<Tab>` primitives. Upstream's version depends on fumadocs-ui,
which shell-docs doesn't install.
- Replaces main's `<Content />` stub at `/deploy/agentcore` with the
full inlined guide.
- Wires the Deploy section into the root sidebar via `deploy/meta.json`
+ a `---Deploy---` group in `meta.json`.
- Registers the component in `mdx-registry.tsx`.

### Remove duplicates reintroduced by the upstream sync
- Delete
`learn/{index,intelligence-platform,threads,tutorials/multi-conversation-chat}.mdx`.
Canonical homes are at `/premium/intelligence-platform`,
`/premium/threads-explained`, and `/tutorials/multi-conversation-chat`.
`next.config.ts` already redirects `/learn/*` to those destinations.
- Delete root `ag-ui-middleware.mdx`. Canonical home is
`/agentic-protocols/ag-ui-middleware`. `next.config.ts` already
redirects `/ag-ui-middleware`.

### Harden sync script
Add `PATH_EXCLUSIONS` regexes so next sync runs don't bring these paths
back:
- `docs/content/docs/learn/`
- `docs/content/docs/(root)/ag-ui-middleware.mdx`
- Orphan shared-state files:
`langgraph/shared-state/workflow-execution`,
`adk/shared-state/{workflow-execution,state-inputs-outputs}`,
`llamaindex/shared-state/state-inputs-outputs`
- AgentCore upstream sources (root shell, per-framework
`deploy-agentcore.mdx`, shared snippet) — shell-docs owns the canonical
inlined version

### Optional schema migrations Step
Adds an opt-in `(Optional) Enable schema migrations` Step to
`premium/self-hosting.mdx` covering `migrations.enabled: true` in the
Helm values + the verification behavior.

## Test plan
- [ ] `/deploy/agentcore` renders the inlined guide; both
`<AgentCoreCommandTabs />` blocks show framework tabs with highlighted
bash and a working copy button
- [ ] Deploy appears as a sidebar group between Premium and What's New
- [ ] `/learn/threads`, `/learn/intelligence-platform`,
`/learn/tutorials/multi-conversation-chat`, and `/learn` all redirect to
their canonical destinations (no longer 200 with duplicate content)
- [ ] `/ag-ui-middleware` redirects to
`/agentic-protocols/ag-ui-middleware`
- [ ] The optional schema migrations Step renders in
`/premium/self-hosting` between the secrets Step and the
install-the-chart Step
- [ ] Next dry-run of `sync-docs-from-main.ts` does not flag the
newly-excluded paths
2026-05-05 12:14:21 +02:00
Sam Julien 45f5cf711c showcase/sync: detect re-introduction of previously-deleted shell-docs paths
PATH_EXCLUSIONS is the durable mechanism for keeping retired upstream
paths out of shell-docs, but it requires the person retiring a page to
also add the regex — and that step has been missed (#4521 brought back
/learn/* and root /ag-ui-middleware.mdx after earlier PRs intentionally
removed them).

Add a safety net: before writing each upstream file, check if its target
shell-docs path exists in git's deletion history and isn't currently on
disk. If so, surface it in the auto-PR's review-items.txt under a new
'Files re-introduced from shell-docs deletion history' section and
include reintroduced.length in hasReviewItems so the PR is flagged
needs-review (exit 3) instead of auto-merged.

Doesn't block the write — content is still synced, the detector is
informational. The fix loop is: human reviews, decides intent. If
unwanted, add a PATH_EXCLUSIONS regex and delete the file; next sync
the detector picks it up again until the regex is in place.

Verified locally — current dry-run flags 3 real findings (root/index,
langgraph/index, microsoft-agent-framework/index) that were deleted in
favor of meta.json folder pages.
2026-05-04 16:32:57 -07:00
Sam Julien c5d2a06e14 showcase: harden sync exclusions to prevent retired paths from re-flowing
The previous sync brought back /learn/*, root /ag-ui-middleware.mdx, the
upstream AgentCore 3-shell + shared snippet, and several orphan
shared-state files that shell-docs deliberately doesn't carry. Add
matching PATH_EXCLUSIONS so future syncs don't reintroduce them.

Not bumping .docs-sync-sha — the previous sync's sha is still accurate
and the new exclusions take effect on the next run.
2026-05-04 15:09:04 -07:00
Jordan Ritter b1b7bb5cb8 Add Baseline tab, Coverage perf + depth fixes, dark mode toggle
Baseline tab (new):
- 33-feature x 25-partner correctness matrix from Notion Partner Hub
- E3 cell treatment: status emoji + colored letter badges (C/A/I/▶/D/T/✱)
- View/Edit toggle with commit/cancel accumulator (batch saves to PB)
- PocketBase baseline collection with public read/write, SSE live updates
- Stats bar, collapsible categories, fixed legend, 200ms CSS tooltips
- Full-row hover highlight, zebra stripes matching Coverage tab

Coverage tab fixes:
- Depth chip: green = at max achievable (not hardcoded per level)
- maxPossible computed from probe existence (CATALOG_TO_D5_KEY)
- D5 false positives: removed shared key aliases
- Stats bar derives green/amber/red from same logic as depth chips
- Hide badges for non-existent tests (was strikethrough)
- Render perf: memoize featuresByCategory, React.memo CategorySection
- Load perf: getFullList single call (was 7 sequential round-trips)

Both tabs:
- Dark/light/system theme toggle (upper right)
- Column order: Coverage first (1-18), Baseline extras (19-25)
- LangGraph naming (orchestration layer, not model router)
- Row hover highlight including sticky column
2026-05-01 22:51:23 -07:00
Ran Shem Tov 2358034310 chore: ratchet showcase pin baseline to 97 2026-05-01 13:01:30 +02:00
github-actions[bot] 374b85bec4 style: auto-fix formatting 2026-05-01 08:06:25 +00:00
Jordan Ritter 223c52b6c2 fix(showcase): update validate-pins baseline for google-adk openai dep
Adding openai to google-adk/package.json for the voice route introduced
one new pin validation failure (136 vs 135). Update the ratchet baseline.
2026-05-01 01:00:53 -07:00
Jordan Ritter 158b4daff9 fix(showcase): handle toolName match constraint in fixture drift validator
The validator was not considering the aimock match.toolName field when
checking for drift. When a fixture has toolName set, aimock only fires
it for agents that register that tool -- skip demos that don't have it.
2026-05-01 00:58:16 -07:00
Jordan Ritter d49dcc4fc9 fix(showcase): add --project-name to isolated compose commands
The COMPOSE_CMD in apply_isolation was missing --project-name, so Docker
Compose would infer the project name from the directory and collide with
the base showcase stack (and other isolated runs). Adding --project-name
ensures containers, networks, and volumes are fully scoped to the
isolation slot.
2026-05-01 00:09:30 -07:00
Jordan Ritter 7e4dad042c fix(showcase): temp overlay isolation + slot-based port allocation
apply_isolation previously mutated docker-compose.local.yml and
local-ports.json in-place with .iso-bak backups. If the process crashed
the originals stayed corrupted with +200 port offsets, breaking all
subsequent showcase commands.

Now writes modified copies to a temp directory and overrides
COMPOSE_FILE/PORTS_FILE shell variables so downstream code reads from
the overlay. Originals are never touched. restore_isolation just removes
the temp dir.

Also replaces hardcoded +200 port offset with atomic mkdir-based slot
allocation. Two parallel --isolate runs now get different port ranges
(slot 0 = +200, slot 1 = +400, etc.) instead of colliding on the same
ports. Container names include the slot number for collision-free Docker
naming. Stale slots from crashed runs are reclaimed via PID liveness
checks and a 2-hour age fallback.

TS harness files (config.ts, lifecycle.ts, doctor.ts) honor
LOCAL_PORTS_FILE env var so they read offset ports from the temp overlay.
2026-05-01 00:08:16 -07:00
Jordan Ritter 4b642bc2e0 fix(showcase): fix referenceCount variable in generate-registry after ref pin
PR #4562 removed the referenceCount variable but left a log line
referencing it, causing ReferenceError at build time. The dashboard
never rebuilt with langgraph-python as REF because the build crashed.
2026-04-30 22:59:51 -07:00
Jordan Ritter 19494f747f fix(showcase): pin reference integration to langgraph-python
The auto-detect logic picked whichever integration had the most wired
features, with ties broken alphabetically. This caused ag2 to appear
as the reference when it matched langgraph-python's feature count.
langgraph-python is always the gold standard reference.
2026-04-30 22:42:18 -07:00
Jordan Ritter f02c6fa906 chore(showcase): ratchet validate-pins baseline 134→135
New pin drift from ms-agent-dotnet V1→V2 import change.
2026-04-30 17:04:56 -07:00
Jordan Ritter f9578b2ab7 feat(showcase): add --isolate flag for parallel-safe local testing
When passed to `showcase test`, creates an isolated Docker Compose
project with offset ports (+200) and renamed containers, allowing
multiple agents/sessions to run showcase tests simultaneously without
container conflicts.

Usage:
  showcase test agno --d5 --isolate           # auto-names isolate-<PID>
  showcase test agno --d5 --isolate d5verify  # explicit name
2026-04-30 16:31:12 -07:00
Jordan Ritter 79c10dbce2 fix(showcase): exclude docs-only features from catalog metadata counts
docs-only features (e.g. cli-start) exist purely for documentation
coverage tracking -- they have no route, no depth probes, and no
health signals. The catalog metadata was counting their 18 stub cells
in the headline wired/stub/unshipped/unsupported breakdown, inflating
the total and making the stats bar misleading.

Exclude docs-only cells from the headline counts. A new docs_only
field tracks the excluded count separately so the invariant
(wired + stub + unshipped + unsupported + docs_only == cells.length)
holds.

Before: total_cells=720 wired=673 stub=18 unsupported=29
After:  total_cells=702 wired=673 stub=0  unsupported=29 docs_only=18
2026-04-30 14:38:18 -07:00
Jordan Ritter e9644b4822 feat(showcase): native CI execution for eval — --ci flag + helper script
Add --ci flag to eval orchestrator that skips Docker lifecycle and
assumes services are already running. Add ci-native-eval.sh helper
that installs deps, starts next dev + agent servers natively, health-
waits, then runs showcase eval --ci. Fix on-demand E2E workflow with
langgraph-python support and agent-type detection.
2026-04-30 11:13:22 -07:00
Alem Tuzlak 4722d4f4da chore(showcase): bump @copilotkit/aimock to 1.16.4
Picks up the router fix from CopilotKit/aimock#148 — `toolCallId` matchers
now only fire when the tool message is the *last* message in the request,
preventing stale tool_call_ids from history shadowing `userMessage`
matchers on new user turns.

Surfaced as: in beautiful-chat, clicking a second suggestion replayed the
prior chart's "Pie chart rendered above…" content fixture instead of
producing a new tool call. Once Railway rebuilds `ghcr.io/copilotkit/aimock:latest`
and restarts the service, demos will pick up the fix automatically.

- Refresh `pnpm-lock.yaml` resolutions (workspace `@copilotkit/runtime` devDep)
- Refresh `showcase/scripts/package-lock.json` to 1.16.4
- Bump the floor in `test_e2e-showcase-on-demand.yml` from `^1.14.3` → `^1.16.4`
  so the `/test-aimock` PR-comment workflow always installs a build that
  contains the fix
2026-04-30 16:28:08 +02:00
Alem Tuzlak bea43fb999 chore(showcase): ratchet validate-pins hash baseline
Pre-existing drift on main: count held at 134, hash shifted (one
FAIL healed, another regressed). Captured the new sorted-FAIL hash
locally with the same algorithm CI uses (sort -u | shasum -a 256)
and updated showcase/scripts/fail-baseline.json to match.
2026-04-30 11:41:03 +02:00
Jordan Ritter b4d5b9c0d4 fix(showcase): update pin ratchet baseline for restored eval files
Bump validatePinsFailCount 132→134 and hash to account for
eval-tiers.json and showcase_eval.yml re-added by eval restoration.
2026-04-29 20:22:47 -07:00
Jordan Ritter 5abf0409a8 fix(showcase): D5 test infrastructure and CLI improvements
- Rewrite cmd-test.sh to use harness CLI for D5 probes
- Add run-e2e-with-aimock.sh helper (removed stale version)
- Update fail-baseline hash for D5 coverage
- Fix provider base URLs in .env.example
- Update local-ports.json for spring-ai
- Remove eval tier system (superseded by D5 depth probes)
- Add D5 debugging runbook
2026-04-29 19:40:10 -07:00
Sam Julien 7965d70b26 fix(showcase): add .java + .xml to bundle-demo-content language detection
Spring-AI's DisplayFlightTool.java snippets bundled with language=text
because detectLanguage()'s extension map didn't list .java; the docs
Snippet renderer then fell back to highlightAuto, which produced no
hljs token classes. Same issue lurked for any future .xml file. Both
extensions added.
2026-04-29 08:15:16 -07:00
github-actions[bot] 58d1a8a87a style: auto-fix formatting 2026-04-29 11:34:09 +00:00
Alem Tuzlak 328304306f fix(showcase): claude-sdk-ts manifest paths + generate-catalog test fixtures
- claude-sdk-typescript manifest: G3's headless-complete and a2ui-fixed-schema entries pointed at -agent.ts paths but the actual files are -prompt.ts. Bundle-demo-content blew up on the missing files.
- generate-catalog test: update reference and crewai-crews wired-count assertions for the post-blitz state. Multiple integrations now tie at 39 wired; auto-detection picks the alphabetical winner (langgraph-fastapi) over langgraph-python. crewai wired count moved to 37.
2026-04-29 13:32:54 +02:00
Alem Tuzlak 7201a22294 chore(showcase): actually bump validate-pins baseline (prev commit's edit was a no-op) 2026-04-29 13:25:12 +02:00
Alem Tuzlak b375349e6a fix(showcase): cr-loop fixes — manifest validation, types, env-var fallbacks
- generate-registry: cross-validate not_supported_features against features list
- catalog-types: widen manifestation union to include 'starter'
- claude-sdk-python: drop invalid 'declarative-schema' from generative_ui enum
- crewai-crews: remove duplicate animated_preview_url key in mcp-apps demo
- built-in-agent + pydantic-ai: REASONING_MODEL env-var fallback so deployers can swap if gpt-5.2/gpt-5 isn't available
2026-04-29 12:58:31 +02:00
Alem Tuzlak 879b18740a feat(showcase-dashboard): add unsupported cell status distinct from unshipped
Adds a fourth cell status, "unsupported", to the showcase dashboard for
features whose framework cannot architecturally support them (e.g.,
no graph-interrupt API, no MCP tool runtime). This is distinct from
"unshipped" (just unbuilt).

- Manifest schema gains optional `not_supported_features: string[]`.
- generate-registry.ts: determineCellStatus checks `not_supported_features`
  first; CatalogCell/CatalogMetadata gain `unsupported`; max_depth=0 for
  unsupported cells; parity tier is computed against the supportable subset
  of the reference (unsupported features no longer drag a framework's tier
  down for gaps it legitimately cannot fill).
- depth-utils.ts: deriveDepth returns D0 with no regression for
  unsupported cells.
- DepthChip: renders a dashed gray border + 🚫 glyph with
  "Not supported by this framework" tooltip; data-status attribute
  distinguishes unshipped vs unsupported.
- CoverageBar gains diagonal-stripe gray segment for unsupported.
- StatsBar / AdaptiveStatsBar / CellsView: separate "Unsupported" chip.
- CellMatrix gaps filter excludes unsupported cells (not work to do).
- Tests: cover unsupported in DepthChip, depth-utils, cell-matrix, and
  catalog generator metadata/max_depth assertions.
2026-04-29 10:59:38 +02:00
Jordan Ritter 797295fa3b fix(showcase): D5 probe fixes — preferences-card crash guard + missing manifest demos (#4409)
## Summary

- Guard `preferences-card.tsx` against undefined `value.interests`
across all 17 integrations — crash prevented D5 shared-state probes from
completing
- Add missing D5 demo entries (`shared-state-read-write`, `hitl-in-app`,
`hitl-in-chat`, etc.) and feature IDs to manifests across 14
integrations — D5 probes couldn't discover features without these
entries
- Bump `@copilotkit/aimock` to v1.16.1 (turnIndex/sequenceIndex support
required for D5 fixture matching)
- Update snapshot test assertions for langgraph-python (38→40
features/demos) and crewai-crews (30→32 wired)
- Fix `docker-compose.local.yml`: mount D5 fixture volumes, add `--host
0.0.0.0`, replace curl healthcheck with Node fetch
- Delete redundant `scripts/dev-local.sh` (superseded by `bin/showcase`
CLI)

## Why

D5 probes were failing across all integrations due to three root causes:
(1) preferences-card crash on undefined interests, (2) missing manifest
entries preventing D5 probe discovery, and (3) aimock pinned at v1.10.0
missing turnIndex support. Local D5 verified GREEN on langgraph-python
(all 11 feature types pass).

## Test plan

- [ ] CI Validate Showcase passes (snapshot counts updated)
- [ ] CI build passes (aimock version bump)
- [ ] Local D5 probes pass for langgraph-python
2026-04-28 23:14:10 -07:00
Jordan Ritter 5bee392385 fix(showcase): fix docker-compose.local.yml aimock config + remove dev-local.sh
- Mount D5/smoke/feature-parity fixtures into aimock via volumes
- Add --host 0.0.0.0 so other containers can reach aimock
- Switch healthcheck from curl to node fetch (aimock image has no curl)
- Delete dev-local.sh (redundant with bin/showcase CLI)
2026-04-28 22:53:49 -07:00
Jordan Ritter 62724f2bd6 fix(showcase): update snapshot counts for new langgraph-python and crewai features
langgraph-python gained shared-state-read-write + hitl-in-app (38→40
features/demos, 37→39 wired, 2→0 unshipped). crewai-crews gained 2
wired features (30→32).
2026-04-28 22:29:36 -07:00
Jordan Ritter 2cb4e26469 feat(showcase): add aimock-rebuild, diff-logs, doctor, and fixtures CLI commands
cmd-aimock-rebuild: full rebuild-redeploy cycle with Depot bypass and
  health check timeout.
cmd-diff-logs: time-windowed container logs with --since last-test
  integration.
cmd-doctor: 10 diagnostic checks (Docker, Depot, ports, stale images,
  env vars, compose config, aimock health, disk, network, DNS).
cmd-fixtures: jq-based fixture validation (syntax, duplicates,
  turnIndex gaps, empty responses, orphaned sub-agent references).
2026-04-28 22:29:34 -07:00
Jordan Ritter 75504f900b feat(showcase): add logs, test, and recreate CLI commands
cmd-logs: container log tailing with --grep, --since, -n, --no-follow.
cmd-test: probe runner with --d5/--d6 filtering, --cycle for aimock log
  dump on failure, --timeout forwarding, .last-test-ts marker.
cmd-recreate: force-recreate with optional --build, routes aimock to
  its dedicated compose file.
2026-04-28 22:29:27 -07:00
Jordan Ritter 58b423d167 feat(showcase): add unified CLI dispatcher and shared utilities
bin/showcase auto-discovers cmd-*.sh plugins, routes built-in commands
(up/down/build/ps/ports), and provides stage_shared/restore_symlinks
for Docker Compose volume management.

_common.sh provides shared variables (SHOWCASE_ROOT, COMPOSE_CMD,
COMPOSE_FILE), output helpers (die/info/warn/success), and container
utilities (slug_to_container, slug_to_port, wait_healthy).
2026-04-28 22:29:21 -07:00
Jordan Ritter 8d7e850c22 chore: bump @copilotkit/aimock to latest (v1.16.1)
Needed for turnIndex/sequenceIndex fixture matching in D5 multi-turn
conversations. Updated in both packages/runtime (devDep) and
showcase/scripts (dep).
2026-04-28 22:21:02 -07:00
Jordan Ritter c62dd66dc4 Merge remote-tracking branch 'origin/main' into blitz/showcase-harness-rename/integration
# Conflicts:
#	pnpm-lock.yaml
#	showcase/harness/config/alerts/smoke-red-tick.yml
#	showcase/harness/scripts/test-notify-harness-jq.sh
#	showcase/harness/src/probes/drivers/e2e-chat-tools.test.ts
#	showcase/harness/src/probes/drivers/e2e-chat-tools.ts
#	showcase/harness/src/probes/drivers/e2e-demos.test.ts
#	showcase/harness/src/probes/drivers/e2e-demos.ts
#	showcase/harness/src/probes/drivers/e2e-readiness.test.ts
#	showcase/harness/src/probes/drivers/e2e-readiness.ts
#	showcase/harness/src/probes/drivers/e2e-smoke.test.ts
#	showcase/harness/src/probes/drivers/e2e-smoke.ts
#	showcase/harness/src/probes/drivers/liveness.test.ts
#	showcase/harness/src/probes/drivers/liveness.ts
#	showcase/harness/src/probes/drivers/smoke.test.ts
#	showcase/harness/src/probes/drivers/smoke.ts
#	showcase/harness/src/probes/liveness.test.ts
#	showcase/harness/src/probes/liveness.ts
#	showcase/harness/src/probes/smoke.test.ts
#	showcase/harness/src/probes/smoke.ts
#	showcase/harness/test/fixtures/rules/valid/smoke-red-tick.yml
#	showcase/ops/src/probes/drivers/e2e-demos.test.ts
#	showcase/ops/src/probes/drivers/e2e-demos.ts
#	showcase/ops/src/probes/drivers/e2e-smoke.test.ts
#	showcase/ops/src/probes/drivers/e2e-smoke.ts
#	showcase/ops/src/probes/drivers/smoke.test.ts
#	showcase/ops/src/probes/drivers/smoke.ts
#	showcase/ops/src/probes/smoke.test.ts
#	showcase/ops/src/probes/smoke.ts
#	showcase/scripts/verify-railway-image-refs.ts
2026-04-28 16:04:44 -07:00
Jordan Ritter b806358d59 chore: remove starter references from YAML configs, test fixtures, and scripts
Remove residual showcase-starter-* references now that starters are
decommissioned:

- Delete showcase/ops/config/alerts/smoke-red-tick.yml (starter-only
  alert rule with kind: starter filter)
- Delete showcase/ops/test/fixtures/rules/valid/smoke-red-tick.yml
  (test fixture mirroring the deleted alert rule)
- Update showcase/ops/config/probes/smoke.yml comments to remove
  showcase-starter-ag2 example and 34-service count
- Update showcase/ops/scripts/test-notify-ops-jq.sh to remove starter
  build job fixtures and starter service names from jq partition test
- Update showcase/scripts/verify-railway-image-refs.ts to remove
  showcase-starter-<slug> from canonical shape comment
2026-04-28 13:51:36 -07:00
Jordan Ritter 83b077db87 refactor(showcase): update scripts, docs, and aimock references for harness rename
Update comment references in showcase scripts (create-integration,
redirect-decommission, validate-pins, verify-railway-image-refs),
RAILWAY.md docs, aimock/d5-all.json fixture comment, and
built-in-agent route comment.
2026-04-28 13:48:30 -07:00
Jordan Ritter 22e557800a fix(showcase): remove stale starter service references before deprovisioning (#4378)
## Summary

- Delete `test_smoke-starter-deployed.yml` workflow (runs every 6h,
would flood Slack with 17 red alerts once starter Railway services are
torn down)
- Remove `starter:` blocks from all 17 integration manifest.yaml files
(stale `demo_url` pointing to `showcase-starter-*` Railway services)
- Clean stale starter references from CI workflow comments
- Remove `"starter"` from dashboard's `manifestation` type union and
dead starter-handling code in cell-matrix
- Fix useLiveStatus test to match INITIAL_CAP=2000 (pre-existing from
#4362)

## Why

PR #4351 merged packages/starters into a unified integrations/
directory, eliminating starters as separate deployable units. The 17
`showcase-starter-*` Railway services are being deprovisioned. These
references must be cleaned up BEFORE teardown to prevent false alerts
and stale data propagation.

## Test plan

- [ ] Dashboard tests pass (15/15 useLiveStatus, depth-utils,
cell-matrix)
- [ ] No `showcase-starter-` URLs remain in generated data files
- [ ] No `starter:` blocks remain in manifest.yaml files
- [ ] CI green
2026-04-28 12:31:09 -07:00
Jordan Ritter 0d2d7c9b1b fix: lower wired-count ratchet from 508 to 490 (17 starter cells removed) 2026-04-28 12:24:44 -07:00
Jordan Ritter 8d9ae3d5bb fix: update generate-catalog tests for starter removal (720 cells, not 737) 2026-04-28 12:17:24 -07:00