mirror of
https://github.com/CopilotKit/CopilotKit.git
synced 2026-09-14 16:26:20 +08:00
Alem Tuzlak
1d32b0f057
Wire CopilotChat AttachmentsConfig (image + PDF, inline base64) into a dedicated /demos/multimodal cell backed by a vision-capable LangGraph agent (gpt-4o). Scoped to its own runtime route so the vision cost stays on this demo. - Dedicated /api/copilotkit-multimodal route registering the new multimodal graph under the multimodal-demo slug - src/agents/multimodal_agent.py with an AgentMiddleware that flattens PDF content parts to text server-side via pypdf - Try with sample image / PDF buttons that drive CopilotChats own hidden file input via DataTransfer + change event, so the sample and paperclip paths exercise the same useAttachments pipeline - public/demo-files/ placeholder — sample.png / sample.pdf binaries are produced by the user at the end of the Wave 2b rollout - Manifest + registry + constraints + docs-links wired; normalized multi-modal -> multimodal in constraints to match the feature registry id - QA checklist + Playwright E2E spec (not run yet — pending deploy) - aimock fixtures for sample prompts so CI runs are deterministic - Extend the check-binaries hook allowlist to cover the shell-docs and shell-dojo demo-content.json mirrors (same size as shell/src/data, already allowed) so regenerated derived data can be committed
3.7 KiB
3.7 KiB
QA: Multimodal Attachments — LangGraph (Python)
Prerequisites
- Demo deployed and accessible at
/demos/multimodal - Railway service
showcase-langgraph-pythonis healthy OPENAI_API_KEYset on the Railway service (vision calls require it)- Agent is using a vision-capable model (
gpt-4oor equivalent) — verifiable by inspecting the Railway logs forsrc/agents/multimodal_agent.pyor by running one image round-trip - Sample files are bundled under
public/demo-files/:sample.png— a small PNG the vision model can describesample.pdf— a small one-page PDF mentioning "CopilotKit"
Test Steps
1. Basic Functionality
- Navigate to
/demos/multimodal - Verify the page header "Multimodal attachments" is visible
- Verify the description text mentions image and PDF attachments
- Verify the sample-row (
[data-testid="multimodal-sample-row"]) is visible - Verify the "Try with sample image" and "Try with sample PDF" buttons are present and enabled
- Verify
<CopilotChat />renders a message composer with a paperclip / "Add attachments" button
2. Sample image path
- Click "Try with sample image"
- Button label briefly flips to "Loading…"
- Within 5 seconds an image attachment chip appears in the composer with a thumbnail preview
- Type "Describe this image"
- Click send
- Within 60 seconds the agent responds with a description referring to the image content (e.g. mentions a logo, colors, or brand mark)
3. Sample PDF path
- Click "Try with sample PDF"
- A document attachment chip appears in the composer (filename "sample.pdf" visible, document icon)
- Type "Summarize this document"
- Click send
- Within 60 seconds the agent responds with text that mentions "CopilotKit" (the sample PDF contains the word multiple times)
4. Paperclip / real upload path (manual)
- Click the paperclip / "Add attachments" button
- File picker opens filtered to
image/*andapplication/pdf - Select a local image under 10 MB
- Attachment chip renders in the composer within 2 seconds
- Type "What's in this image?" and send
- Agent responds with a description of the image content
5. Drag-and-drop
- Drag a local image onto the chat container
- Chat surface shows a drop affordance
- On drop, an attachment chip appears
- Sending a prompt works the same as the paperclip path
6. Multi-attachment
- Inject the sample image AND the sample PDF (click both buttons sequentially)
- Both chips are visible in the composer
- Type "What do these two attachments have in common?"
- Click send
- Agent response acknowledges both attachments
7. Error Handling
- Try to attach a file over 10 MB via the paperclip
- Verify
onUploadFailedfires (console warning from the page) and the file is rejected without corrupting the composer - Try to attach an unsupported type (e.g.
.exe) — the file picker filter excludes it oronUploadFailedfires withreason: "invalid-type" - Block
/demo-files/sample.pngin DevTools Network; click "Try with sample image"; themultimodal-sample-errorspan shows a helpful error without crashing the page
Expected Results
- Attachment chip appears within 2 seconds after upload or sample injection
- Thumbnail renders for image attachments; document icon for PDFs
- Agent response arrives within 60 seconds for single-attachment prompts (multimodal tokens are heavier than text-only)
- No console errors during successful flows (warnings from
onUploadFailedduring intentional error cases are acceptable) - Error states are visible and recoverable — the user can retry