Root cause: gpt-4o-mini occasionally answered "Show me the sales
dashboard with metrics and a revenue chart" with a prose preamble
or clarifying question instead of the JSON spec the `@json-render`
renderer expects, which made the frontend fall back to
CopilotChatAssistantMessage (plain text) rather than a MetricCard +
BarChart tree. The single-component prompts (Revenue by category,
Expense trend) were terse enough that the model usually complied,
but the multi-component Sales Dashboard prompt was where it drifted.
Fix:
- Pass response_format={"type": "json_object"} via model_kwargs so
the OpenAI response is guaranteed to be a single JSON object,
removing any possibility of a plain-text fallback.
- Tighten the system prompt so every user turn is answered with JSON
(no clarifying questions, no prose) and components are referenced
by their exact case-sensitive names.
- Extend the Sales Dashboard worked example to include two MetricCards
plus a BarChart so the model has a concrete multi-component pattern
to mirror - the round-1 children forwarding in the registry wrapper
already supports arbitrarily many siblings under the root MetricCard.
The agent system prompt instructed the LLM to return <ui>...</ui> XML markup,
but the frontend renderer uses useJsonParser(message.content, kit.schema)
which expects a JSON object matching createUiKit(...).schema:
{
"ui": [
{ "metric": { "props": { "label": "...", "value": "..." } } },
{ "pieChart": { "props": { "title": "...", "data": "[{...}]" } } },
...
]
}
The XML syntax in useUiKit({ examples }) is hashbrowns prompt DSL for when
hashbrown drives the LLM directly (e.g. useUiChat); because this demo drives
the LLM via langgraph, the agent itself must emit the JSON wire format. With
XML content, useJsonParser returned { value: undefined } and the renderer
bailed to null, leaving the chat with only a loading indicator.
Rewrote the system prompt to produce the JSON envelope and updated the
renderer top-of-file docstring to match. Prop schemas tracked from the
current kit (metric: label+value, pieChart/barChart: title+data-as-JSON-string,
dealCard: title+stage+value, Markdown: children).
The multimodal demo failed three ways:
1. Try-with-sample-image rendered a broken thumbnail because the sample
PNG on disk is a Git LFS pointer stub when LFS is not pulled at build
time. The browser base64-encoded the 130-byte text stub, fed it into
the chat as a valid-looking image/png, and CopilotChat rendered it
as a broken <img>.
2. Try-with-sample-pdf attached in the composer but the agent said
"no image/document attached". Same broken-image pipeline also hid the
real symptom: the published @ag-ui/langgraph converter (0.0.x) only
understands the legacy { type: binary, mimeType, data | url }
AG-UI content-part shape. The modern { type: image | document,
source: {...} } parts CopilotChat emits are silently filtered out of
the LangChain message stream, so the LangGraph agent never saw them.
3. Manual drag-and-drop of an image failed for the same reason as #2.
Fixes:
- sample-attachment-buttons.tsx: reject Git LFS pointer stubs and
validate PNG/PDF magic bytes on the client before feeding them to the
attachment queue. Surfaces an actionable error pointing at
git-lfs-pull instead of silently producing a broken thumbnail.
- page.tsx: install an onRunInitialized subscriber on the active agent
that rewrites outgoing user-message image / document / audio / video
parts to the legacy binary shape the runtime converter preserves.
Everything else (CopilotChat UI, useAttachments upload pipeline,
paperclip + drag-drop + paste paths) is untouched; we only retarget
the wire format so the LangGraph side receives the attachment.
- multimodal_agent.py: after the rewrite, attachments arrive in the
agent as LangChain image_url content parts (data URLs) regardless of
upstream modality. Route on MIME instead of the AG-UI part type:
image/* forwards to GPT-4o natively, application/pdf is flattened to
text via pypdf. The modern document shape is still handled for
forward-compat when the runtime is upgraded.
Verified: classifier helpers pass inline unit checks; tsc --noEmit
introduces no new type errors relative to HEAD (module-resolution
errors for @copilotkit/shared and lucide-react are pre-existing on this
branch and unrelated to the multimodal demo).
Starting the /demos/auth cell unauthenticated meant the initial `/info`
handshake from CopilotKit returned 401 before any React error handler
could attach, crashing the page on load. Flip the default so the demo
mounts already signed in — the chat is immediately usable — and expose
Sign out / Sign in as the toggle for exercising the 401 path.
- use-demo-auth: default state flipped from unauthed to authed so the
page mounts with a valid Bearer header.
- page.tsx: add a ChatErrorBoundary around <CopilotChat /> that catches
any render-time throw from chat internals after signing out and shows
a friendly in-page message instead of white-screening. The boundary
resets on auth transitions so signing back in restores a live chat.
- auth-banner: rename Authenticate -> Sign in; status copy updated to
Signed in / Signed out to match the inverted default.
- tests/e2e/auth.spec.ts: flip assertions — page loads authenticated,
sign out triggers 401 on next send with banner still mounted, and
sign in clears the error and restores successful sends.
- qa/auth.md: refreshed checklist for the authenticated-default flow.
V1 CopilotRuntime wrapper silently drops `transcriptionService`, so the
composer never saw the mic button on /voice. Write it onto the V2 runtime
instance directly, and wrap the OpenAI service with a guard that throws an
"api key missing" error when OPENAI_API_KEY is absent (mapped by the
runtime error categorizer to AUTH_FAILED / 401) instead of the opaque 503
users were hitting.
Clicking any suggestion in the byoc-json-render demo crashed with
"useVisibility must be used within a VisibilityProvider". Renderer from
@json-render/react relies on StateProvider, VisibilityProvider,
ValidationProvider, and ActionProvider contexts that its internal
ElementRenderer consumes, but the assistant-message slot was rendering
Renderer bare.
Wrap it in JSONUIProvider (which composes all four required providers)
so streamed spec output renders without a context crash.
Also pass children through the MetricCard catalog wrapper. The agent's
Sales Dashboard worked example uses a MetricCard as the root with a
BarChart nested in its children array; previously that chart was
silently dropped by the registry wrapper, which only rendered the
MetricCard itself.
Wrap PieChart + BarChart with string-accepting wrappers that JSON-parse the
data prop, and change the hashbrown schema from s.streaming.array(...) to
s.string(). Drop the non-schema example props (trend on metric, assignee
+ dueDate on dealCard).
Wave 4a was ported from the starter, which used an older hashbrown where:
- optional props were declared via .optional()
- array props accepted stringified JSON in example JSX attributes
@hashbrownai/core@0.5.0-beta.4 dropped .optional() (props omitted from the
schema are treated as not-present) AND tightened example-prompt validation
to reject strings against array schemas. The build failed at prerender:
Error: Example prompt has 5 errors:
Prop 'trend' is not defined on <metric>
Prop 'data' on <pieChart>: Expected an array at: <root>
Prop 'data' on <barChart>: Expected an array at: <root>
Prop 'assignee' is not defined on <dealCard>
Prop 'dueDate' is not defined on <dealCard>
Modeling data as a string matches how the LLM streams it anyway (JSON
tokens through useJsonParser), and the wrappers are defensive about
partial tokens — they render nothing if the parse fails mid-stream.
The voice route was instantiating OpenAI() at module scope, which Next.js
triggers during 'collect page data' at build time. The Docker build context
does NOT have OPENAI_API_KEY set — only the running container does — so
every showcase-langgraph-python deploy has been failing since the voice
demo merged (#4224):
Error: Missing credentials. Please pass an `apiKey`, or set the
`OPENAI_API_KEY` environment variable.
[Error: Failed to collect page data for /api/copilotkit-voice]
Wrap the OpenAI + TranscriptionService + CopilotRuntime construction in
a getRuntime() function that runs on first request. Cached after first
call so there's no per-request cost in production.
This unblocks the full showcase deploy pipeline, not just voice.
Add *.wav to .gitattributes so audio assets follow the same LFS policy
as the existing *.png / *.pdf / *.gif / *.jpg / *.jpeg / *.mp4 / *.webm
rules. Re-stage sample.wav so the committed object becomes an LFS
pointer instead of a raw 87 KB blob.
- public/demo-audio/sample.wav (87KB, 16kHz mono, ~2.7s) — TTS "What is the weather in Tokyo?" for the voice demo's "Play sample" button
- public/demo-files/sample.png (10KB) — CopilotKit logo for the multimodal demo's "Try with sample image"
- public/demo-files/sample.pdf (2.4KB, one page, contains "CopilotKit" 9x) — quickstart excerpt for the multimodal demo's "Try with sample PDF"
Makes both demos fully self-contained end-to-end; the "Try with sample X" buttons now work without the user providing their own files.
Resolves conflicts:
- manifest.yaml / constraints.yaml / docs-links.json: keep both byoc-json-render
and main's re-wired open-gen-ui + open-gen-ui-advanced entries
- scripts/hooks/check-binaries.sh: keep both shell-docs and shell-dojo
demo-content exclusions (identical lines, different ordering)
- pnpm-lock.yaml (langgraph-python): regenerated via pnpm install
- shell/shell-dojo/shell-docs data JSONs: accept main's deletion (now
regenerated by nx targets), then regenerate via scripts/generate-registry.ts
and scripts/bundle-demo-content.ts
Bump generate-registry.test expected feature/demo count 32 -> 33 to account
for the new byoc-json-render demo.
Resolves textual conflicts in manifest.yaml and docs-links.json (keeps
both open-gen-ui* from main and agent-config from this branch). Accepts
main's deletion of generated shell*/src/data/*.json (now gitignored).
Bumps langgraph-python feature/demo count test from 32 to 33.
Resolved conflicts:
- showcase/packages/langgraph-python/manifest.yaml: kept auth (HEAD) +
open-gen-ui / open-gen-ui-advanced (main) in both features and demos.
- showcase/packages/langgraph-python/docs-links.json: kept auth entry
from HEAD alongside updated subagents shell_docs_path.
- showcase/shared/constraints.yaml: kept auth + open-gen-ui* entries.
- Deleted shell/shell-dojo/shell-docs generated JSONs per main (now
generated at build, no longer committed).
Bumped langgraph-python feature/demo count 32 -> 33 in generate-registry
test to reflect the added auth demo.
## Summary
First round of the snippet-linking sweep: every code fence in shell-docs
should be a `<Snippet>` pointing to real showcase source, not
hand-written inline ```tsx.
Scope on this PR is **prebuilt-components + the unselected twin** — 4
commits, easy per-commit review.
### Commits
- `7efe08ae8` — **Strip hand-written fences from Styling sections** on
all 6 prebuilt-components pages (base + unselected ×
chat/sidebar/popup). Keeps the `## Styling` header, intro copy, and
bullet links to `/custom-look-and-feel/*`; drops only the duplicative
slot-override code.
- `157cdae57` — **Wording pass**: replace "showcase cell" references
with neutral phrasing ("the example below", etc.) across 21 files.
- `f6086011a` — **Snippet-ify the Code example section on both chat.mdx
pages.** Adds a new `@region[chat-component]` to
`showcase/packages/langgraph-python/src/app/demos/agentic-chat/page.tsx`
wrapping the existing `Chat` helper (hook + render). Hand-written tsx
fences on `docs/prebuilt-components/chat.mdx` and
`docs/unselected/prebuilt-components/chat.mdx` replaced with `<Snippet
region="chat-component">`. The three `demo-content.json` bundles are
regenerated to include the new region.
- `ef0823372` — **Snippet-ify variant code blocks on
`unselected/prebuilt-components/index.mdx`.** Three hand-written tsx
fences (CopilotChat, CopilotSidebar, CopilotPopup variants) become
Snippets pointing at `chat-component` (agentic-chat),
`sidebar-basic-setup` (prebuilt-sidebar), and `popup-basic-setup`
(prebuilt-popup). Drops the redundant Deep customization inline example
in favor of a link to the Slots guide.
### Showcase-source change
Only one: `@region[chat-component]` / `@endregion[chat-component]`
markers added around the existing `Chat` helper in
`showcase/packages/langgraph-python/src/app/demos/agentic-chat/page.tsx`
(2 lines, no runtime behavior change).
## Test plan
Run shell-docs at `localhost:3003`.
Prebuilt-components pages (clear framework selection to reach
`/unselected/*`):
- [ ] `/langgraph-python/prebuilt-components/chat` — Basic setup =
`provider-setup` Snippet; Code example = new `chat-component` Snippet
showing the full `Chat` function; Styling = header + intro + 3 bullet
links, no code
- [ ] `/langgraph-python/prebuilt-components/sidebar` — Basic setup =
`sidebar-basic-setup` Snippet; Configuring = `sidebar-configuration`
Snippet; Styling = header + intro + link, no code
- [ ] `/langgraph-python/prebuilt-components/popup` — Basic setup =
`popup-basic-setup` Snippet; Styling = header + intro + link, no code
- [ ] `/unselected/prebuilt-components/chat`, `/sidebar`, `/popup` —
same expected rendering as the langgraph-python variants
Index page (URL-only, not in sidebar):
- [ ] `/unselected/prebuilt-components` — 3 variant subsections
(CopilotChat, CopilotSidebar, CopilotPopup), each showing a real-source
Snippet block; Deep customization is a one-paragraph pointer to Slots,
no code
Replaces the hand-written tsx fence in the 'Code example' section of both
docs/prebuilt-components/chat.mdx and docs/unselected/prebuilt-components/chat.mdx
with a <Snippet> reference to a new chat-component region in the
langgraph-python agentic-chat demo source.
The new @region[chat-component] wraps the whole Chat helper function in
showcase/packages/langgraph-python/src/app/demos/agentic-chat/page.tsx,
so readers see a self-contained real-code component (hook + render) with
the view-source link that a Snippet provides.
Also regenerates the three demo-content.json bundles to include the new
region.
Merge with main (-X theirs) dropped the voice entries from the hand-edited
manifest + docs-links files because the generated registry / demo-content
bundles on main diverged from my local copies. Restore the voice demo
entries (manifest features + demos, docs-links, regenerated shell bundles)
on top of main's state.
Wave 4a. Port the starter's hashbrown renderer into a dedicated
single-mode langgraph-python demo at /demos/byoc-hashbrown so the
byoc-hashbrown row goes green on the dashboard.
- @hashbrownai/core + @hashbrownai/react + recharts deps added.
- Dedicated /api/copilotkit-byoc-hashbrown route with byoc_hashbrown
graph (ChatOpenAI + CopilotKitMiddleware) and a catalog-aware
system prompt coaching the LLM to emit a <ui>...</ui> envelope.
- Ported MetricCard + bar-chart + pie-chart + chart-config from
showcase/starters/template/frontend with data-testid hooks on
chart roots for E2E coverage.
- Ported hashbrown renderer with local RenderMessageProps /
AssistantMessage types so @copilotkit/react-ui and @ag-ui/core
do not become direct deps of this package.
- Updated hashbrown schema calls to @0.5.0-beta.4 surface:
enumeration for SalesStage; description-first streaming.array +
object; dropped the non-existent .optional() chain.
- v2 CopilotChat uses messageView.assistantMessage slot and
useConfigureSuggestions for 3 canned prompts (sales dashboard /
revenue by category / expense trend).
- QA checklist + Playwright E2E authored (not run pre-deploy).
- Manifest / constraints / docs-links updated and derived registry +
demo-content + docs-status bundles regenerated.
- Extend scripts/hooks/check-binaries.sh whitelist to include the
shell-docs and shell-dojo demo-content bundles so the pre-commit
hook does not reject the regenerated 1.5 MB files it produced.
Wire CopilotChat AttachmentsConfig (image + PDF, inline base64) into a
dedicated /demos/multimodal cell backed by a vision-capable LangGraph
agent (gpt-4o). Scoped to its own runtime route so the vision cost
stays on this demo.
- Dedicated /api/copilotkit-multimodal route registering the new
multimodal graph under the multimodal-demo slug
- src/agents/multimodal_agent.py with an AgentMiddleware that flattens
PDF content parts to text server-side via pypdf
- Try with sample image / PDF buttons that drive CopilotChats own
hidden file input via DataTransfer + change event, so the sample and
paperclip paths exercise the same useAttachments pipeline
- public/demo-files/ placeholder — sample.png / sample.pdf binaries
are produced by the user at the end of the Wave 2b rollout
- Manifest + registry + constraints + docs-links wired; normalized
multi-modal -> multimodal in constraints to match the feature
registry id
- QA checklist + Playwright E2E spec (not run yet — pending deploy)
- aimock fixtures for sample prompts so CI runs are deterministic
- Extend the check-binaries hook allowlist to cover the shell-docs and
shell-dojo demo-content.json mirrors (same size as shell/src/data,
already allowed) so regenerated derived data can be committed
Integrates @json-render/{core,react} as an alternative BYOC generative-UI
rendering technology in langgraph-python, paired with Wave 4a's hashbrown
demo. Both demos share the same sales-dashboard catalog (MetricCard +
BarChart + PieChart) so the dashboard rows are directly comparable.
- Adds @json-render/{core,react} @ 0.18.0 pinned.
- New /demos/byoc-json-render page using CopilotChat's messageView.assistantMessage
slot to bridge CopilotKit output into @json-render/react Renderer.
- New /api/copilotkit-byoc-json-render route + byoc_json_render graph in
langgraph.json.
- Zod-validated catalog reusing @json-render/react/schema's prebuilt spec shape.
- System prompt with 3 worked examples inlined so the agent emits valid
{ root, elements } JSON deterministically.
- QA checklist + Playwright spec authored (E2E run deferred to post-deploy
stabilization).
- Adds shell-docs/shell-dojo demo-content.json to the check-binaries.sh
allowlist (same 1MB-allowed rationale as shell/demo-content.json).
Framework-native request authentication demo for the langgraph-python
showcase. Demonstrates both the unauthenticated (401) failure path and
the authenticated success path via a shared bearer token.
- New /demos/auth page with sticky AuthBanner + in-memory React auth state
- Dedicated /api/copilotkit-auth runtime route using createCopilotRuntimeHandler
from @copilotkit/runtime/v2 with an onRequest hook validating
Authorization: Bearer demo-token-123
- Shared DEMO_TOKEN constant imported by both client and server to prevent drift
- <CopilotKit headers={...}> injects Authorization reactively when authenticated
- Page-level error surface captures 401s via onError and renders a stable
data-testid=auth-demo-error for QA + E2E assertions
- QA checklist + Playwright E2E spec authored (E2E run deferred to post-deploy)
- Manifest, feature list, docs-links, constraints, and derived registry /
demo-content / constraints / docs-status data regenerated
- Allowlist shell-docs / shell-dojo demo-content.json in the binary-size
pre-commit hook (mirrors the Wave 2a voice commit; they are bundled
fixtures already over 1 MB on main)
Wire @copilotkit/voice into a langgraph-python /demos/voice route via a
dedicated runtime (/api/copilotkit-voice) that mounts
TranscriptionServiceOpenAI. The mic button auto-appears in the CopilotChat
composer because the runtime advertises audioFileTranscriptionEnabled=true.
- Per-demo runtime route /api/copilotkit-voice with transcriptionService
- Demo page at /demos/voice with <CopilotChat /> + Play-sample button
- SampleAudioButton fetches /demo-audio/sample.wav and POSTs the
single-route transcribe envelope to the runtime URL, then writes the
transcribed text into the chat textarea via the native setter +
synthetic input event so React state stays in sync
- Manifest, registry, constraints, docs-links wired; shell bundles regenerated
- QA checklist (qa/voice.md) + Playwright E2E spec (tests/e2e/voice.spec.ts)
- public/demo-audio/ placeholder committed; sample.wav to be added by user
- Allowlist shell-docs / shell-dojo demo-content.json in the binary-size
pre-commit hook (they are bundled fixtures, already over 1 MB on main)
20 primary per-package Playwright specs plus 3 testing-kind stubs
(~75 test blocks). Selectors discipline: testids, role-based,
verbatim text; never LLM-text assertions. Enter-key submit replaced
with copilot-send-button testid (Enter flakes on Railway slow paths).
All non-skipped tests green 3x against Railway.
22 substantive primary checklists at the qa/agentic-chat.md quality
bar (specific testids, timeouts, expected results) plus 3 short stubs
for testing-kind demos (interrupt-headless, tool-rendering-reasoning-chain,
reasoning-default-render). 25 files total.
Declare open-gen-ui and open-gen-ui-advanced in langgraph-python
manifest (code existed, was never registered). Add both to
constrained-explicit allowlist, fill shell_docs_path for 5 demos,
add hitl-in-app override, drop stale chat-customization-css fallback.
Regenerate registry.json, demo-content.json, constraints.json,
and docs-status.json across shell / shell-dojo / shell-docs.
Bump feature/demo count assertion 30→32 in generate-registry test.
Extend check-binaries.sh whitelist for sister-shell demo-content.
Substring-match fixtures (pie chart, bar chart, schedule, trip, etc.)
cross-fired across demos with different tool surfaces and returned tool
names the target agent never registered, causing demos to render nothing
in prod when aimock handles traffic.
Fixture changes (showcase/aimock/feature-parity.json):
- Replace generic pie-chart / bar-chart matches with per-suggestion
specific phrases so gen-ui-tool-based gets render_pie_chart /
render_bar_chart directly and beautiful-chat gets pieChart / barChart
with real data (skipping the query_data two-step that caused the
infinite loop on re-matching prompts).
- Narrow schedule+meeting to the Beautiful Chat 30-minute prompt
returning scheduleTime.
- Narrow flight+fly to flights-from-SFO-to-JFK.
- Narrow background to sunset-themed-gradient.
- Remove trip, sales, pipeline, todo: substring-false-firing across
unrelated demos; interrupt/A2UI demos fall through to real LLM.
Guardrail (showcase/scripts/validate-fixture-tool-surface.ts):
- Pure validate() cross-references every fixture's tool-call name
against the tool surface of each demo whose suggestion prompt contains
the fixture's match substring. Loud failure when the fixture returns a
tool the demo's agent does not register.
- CLI walks packages/ collecting suggestions from page.tsx + hooks/,
frontend tools from useComponent / useHumanInTheLoop / useFrontendTool
/ useRenderTool / useDefaultRenderTool, and backend tools via route.ts
agentId->graphId map + langgraph.json graph->file + @tool decorators.
- 7 vitest cases written TDD-first covering the drift detection,
content-only fixtures, case-insensitivity, and multi-tool responses.
- Current state: 33 fixtures x 191 demos, no drift. Counterfactual
(reverting the pie-chart fix) correctly flags gen-ui-tool-based and
declarative-gen-ui.
Also fixes a separate runtime bug in the langgraph-python package
Dockerfile: WORKDIR /app left /app owned by root; the app user could
not create the .langgraph_api cache dir LangGraph's in-memory runtime
needs, so the agent crashed on boot. Added a non-recursive chown
app:app /app (preserves the original perf intent of the explicit
--chown on COPY, which avoided a recursive chown).
Alem Tuzlak