mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
Alem Tuzlak
7c3edca2b7
The langgraph-python multimodal-attachments demo had a stack of bugs that compounded each other. Fixing them required touching the local docker-compose, the aimock fixtures, the LangChain middleware, the client-side AG-UI shim, and the sample-attachment buttons. This commit lands the full set together because they only make sense as a unit — verified end-to-end against `showcase up langgraph-python` in a headed browser. New e2e suite pins each regression. Supersedes #4584 (the original fix from May 1 that never landed — this is a fresh port onto the post-refactor file layout where page.tsx is split into legacy-converter-shim.tsx, multimodal-chat.tsx, file-to-data-attachment.ts). What was broken and what changed: 1. Random uploads crashed with `Failed to fetch`. aimock returned HTTP 404 on no-match, the LangGraph SDK surfaced `NotFoundError`, the AG-UI stream surfaced a `RUN_ERROR`, the demo crashed. Added `--proxy-only` + `--provider-openai https://api.openai.com` to the local aimock command so unmatched user prompts fall through to real OpenAI (mirrors the Railway aimock setup). 2. Bundled-sample fixtures keyed on user-visible canned prompts. The auto-prompts are deliberately long, specific, and natural- reading ("can you tell me what is in this demo image/pdf I just attached") so they (a) render cleanly as the user message bubble, and (b) can't collide with arbitrary user prompts — random uploads phrase questions differently and fall through to the proxy. 3. Sample buttons now auto-send via `useAgent`. The previous DataTransfer-based path queued the attachment via the chat's hidden file input, then required clicking send while the attachment was still uploading — `CopilotChat.onSubmitInput` rejects submits during upload AND clears the input regardless, so the canned prompt was eaten. Rewrite to call `agent.addMessage(...)` + `copilotkit.runAgent({ agent })` directly with the base64'd content part, sidestepping the upload race entirely. 4. PDF flattened text bled into the rendered user message. `_PdfFlattenMiddleware` ran in `before_model` and returned `{"messages": rewritten}`, which persisted to agent state. The chat UI then rendered the `[Attached document]\n<pdf body>` text part inline with the user prompt. Switched to `wrap_model_call` so the PDF→text rewrite is scoped to the outgoing model request only and never pollutes state. 5. Attachments doubled (and PDFs rendered as broken `<img>`). The `@ag-ui/langgraph` round-trip translates outgoing `binary` parts to LangChain `image_url` and incoming `image_url` back to `image` AG-UI parts — regardless of mimeType, so PDFs came back as `type: "image"` with `mimeType: "application/pdf"` and were forced into `ImageAttachment`, where the load failed and the chat showed two "Failed to load image" boxes. Plus the user's original modern part survived alongside the round-tripped one, doubling visible chips. Added a `dedupeUserMessageMedia` subscriber on both `onMessagesSnapshotEvent` and `onRunFinalized` to: - dedupe media parts by `source.value` so the local + round- tripped copy collapse to one chip - re-key part `type` from `mimeType` so PDFs route to `DocumentAttachment` (icon + filename) and images to `ImageAttachment`. Also flipped the `onRunInitialized` shim from REPLACE to APPEND — keep the modern part for the UI AND emit a legacy `binary` sibling for the converter. 6. Regression suite (`tests/e2e/multimodal.spec.ts`). Replaces the pre-rewrite suite with five focused tests: - page loads with all expected affordances - sample image: auto-sends, EXACTLY ONE `<img>`, assistant references the logo - sample PDF: auto-sends, EXACTLY ONE `DocumentAttachment` chip ("PDF" label), NO `<img>`, no `[Attached document]` text bleed - image then PDF in the same session: each message keeps its own single chip, no cross-contamination - PDF then image in the same session: symmetric All 5 pass against the live local stack (15.4s).
472 KiB
472 KiB