Alem Tuzlak
7644d85f55
The langgraph-python multimodal-attachments demo had a stack of bugs that compounded each other. Fixing them required touching the local docker-compose, the aimock fixtures, the LangChain middleware, the client-side AG-UI shim, and the sample-attachment buttons. This commit lands the full set together because they only make sense as a unit — verified end-to-end against `showcase up langgraph-python` in a headed browser. New e2e suite pins each regression so the next person to refactor this won't re-introduce them silently. What was broken and what changed: 1. **Random uploads crashed with `Failed to fetch`.** When aimock had no fixture for a user prompt it returned HTTP 404, the OpenAI SDK in the agent treated that as `NotFoundError`, the AG-UI stream surfaced a `RUN_ERROR`, and the demo crashed. The deployed Railway aimock runs in `--proxy-only` mode (matched fixtures short-circuit, unmatched proxy to real OpenAI) but the local docker-compose didn't. Add `--proxy-only` and `--provider-openai https://api.openai.com` to the local aimock's command so the local stack mirrors prod and unmatched user prompts reach a real LLM instead of crashing. 2. **Bundled-sample fingerprints in aimock.** For the demo's two "Try with sample X" buttons we want deterministic responses without burning tokens. Replace the broad `userMessage: "this PDF" / "pdf" / "PDF"` fixtures (which collided with arbitrary user uploads — `architecture` contained `hi`, random PDFs that mention `dashboard` or `report` hit other broad fixtures, etc.) with one full-body PDF fingerprint (`"CopilotKit Quickstart\nAdd AI copilots..."`, unique to the bundled sample) and the existing `"this image"` matcher. Random uploads now fall through to the proxy. 3. **Sample buttons now auto-send via `useAgent`.** The previous DataTransfer-based path queued the attachment via the chat's hidden file input, then required clicking send while the attachment was still uploading — `CopilotChat.onSubmitInput` rejects submits during upload AND clears the input regardless, so the canned prompt was eaten. Rewrite to call `agent.addMessage(...)` + `copilotkit.runAgent({ agent })` directly with the base64'd content part, sidestepping the upload race entirely. 4. **PDF flattened text bled into the rendered user message.** `_PdfFlattenMiddleware` ran in `before_model` and returned `{"messages": rewritten}`, which persisted to agent state. The chat UI then rendered the `[Attached document]\n<pdf body>` text part inline with the user prompt. Switched to `wrap_model_call` so the PDF→text rewrite is scoped to the outgoing model request only and never pollutes state. 5. **Attachments doubled (and PDFs rendered as broken `<img>`).** The `@ag-ui/langgraph` round-trip translates outgoing `binary` parts to LangChain `image_url` and incoming `image_url` back to `image` AG-UI parts — regardless of mimeType, so PDFs came back as `type: "image"` with `mimeType: "application/pdf"` and were forced into `ImageAttachment`, where the load failed and the chat showed two "Failed to load image" boxes. Plus the user's original modern part survived alongside the round-tripped one, doubling visible chips. Added a `dedupeUserMessageMedia` subscriber that hooks both `onMessagesSnapshotEvent` and `onRunFinalized` to: - dedupe media parts by `source.value` (regardless of declared type) so the local + round-tripped copy collapse to one chip - re-key part `type` from `mimeType` so PDFs route to `DocumentAttachment` (icon + filename) and images to `ImageAttachment`. Also kept the existing `onRunInitialized` shim which APPENDS (rather than replaces) a legacy `binary` mirror beside the modern part so the converter still sees the attachment for the agent call. 6. **Regression suite (`tests/e2e/multimodal.spec.ts`).** Replaces the pre-rewrite suite with five focused tests: - page loads with all expected affordances - sample image: auto-sends, exactly ONE `<img>`, no broken-image fallback, assistant references the logo - sample PDF: auto-sends, exactly ONE `DocumentAttachment` chip ("PDF" label), NO `<img>`, no `[Attached document]` text bleed - image then PDF in the same session: each message keeps its own single chip, no cross-contamination - PDF then image in the same session: symmetric All 5 pass against the live local stack.
CopilotKit
Docs · Examples · Copilot Cloud · Discord
Build agent-native applications with generative UI, shared state, and human-in-the-loop workflows.
What is CopilotKit
CopilotKit is a best-in-class SDK for building full-stack agentic applications, Generative UI, and chat applications.
We are the company behind the AG-UI Protocol, adopted by Google, LangChain, AWS, Microsoft, Mastra, PydanticAI, and more!
https://github.com/user-attachments/assets/6f06c63f-9bd1-4762-99ac-24898ee227bc
Features:
- Chat UI – A React-based chat interface that supports message streaming, tool calls, and agent responses.
- Backend Tool Rendering – Enables agents to call backend tools that return UI components rendered directly in the client.
- Generative UI – Allows agents to generate and update UI components dynamically at runtime based on user intent and agent state.
- Shared State – A synchronized state layer that both agents and UI components can read from and write to in real time.
- Human-in-the-Loop – Lets agents pause execution to request user input, confirmation, or edits before continuing.
https://github.com/user-attachments/assets/55bf6714-62a7-4d5d-9232-07747cc0763b
Quick Start
New projects:
npx copilotkit@latest create -f <framework>
Existing projects:
npx copilotkit@latest init
https://github.com/user-attachments/assets/7372b27b-8def-40fb-a11d-1f6585f556ad
What this gives you:
- CopilotKit installed – Core packages are fully set up in your app
- Provider configured – Context, state, and hooks ready to use
- Agent <> UI connected – Agents can stream actions and render UI immediately
- Deployment-ready – Your app is ready to deploy
Complete getting started guide →
How it works:
CopilotKit connects your UI, agents, and tools into a single interaction loop.
This enables:
- Agents that ask users for input
- Tools that render UI
- Stateful workflows across steps and sessions
⭐️ useAgent Hook
The useAgent hook is a proper superset of useCoAgent and sits directly on AG-UI, giving more control over the agent connection.
// Programmatically access and control your agents
const { agent } = useAgent({ agentId: "my_agent" });
// Render and update your agent's state
return <div>
<h1>{agent.state.city}</h1>
<button onClick={() => agent.setState({ city: "NYC" })}>
Set City
</button>
</div>
Check out the useAgent docs to learn more.
https://github.com/user-attachments/assets/67928406-8abc-49a1-a851-98018b52174f
Generative UI
Generative UI is a core CopilotKit pattern that allows agents to dynamically render UI as part of their workflow.
https://github.com/user-attachments/assets/3cfacac0-4ffd-457a-96f9-d7951e4ab7b6
Compare the Three Types
Explore:
Generative UI educational repo →
🖥️ AG-UI: The Agent–User Interaction Protocol
Connect agent workflow to user-facing apps, with deep partnerships and 1st-party integrations across the agentic stack—including LangGraph, CrewAI, and more.
npx create-ag-ui-app my-agent-app
🤝 Community
Have questions or need help?
Join our Discord →Read the Docs →
Try Copilot Cloud →
Stay up to date with our latest releases!
Follow us on LinkedIn →Follow us on X →
🙋🏽♂️ Contributing
Thanks for your interest in contributing to CopilotKit! 💜
We value all contributions, whether it's through code, documentation, creating demo apps, or just spreading the word.
Here are a few useful resources to help you get started:
-
For code contributions, CONTRIBUTING.md.
-
For documentation-related contributions, check out the documentation contributions guide.
-
Want to contribute but not sure how? Join our Discord and we'll help you out!
📄 License
This repository's source code is available under the MIT License.