Commit Graph

467 Commits

Author SHA1 Message Date
Alem Tuzlak 4b11b995a2 feat: multimodal attachments — images, audio, video, documents in CopilotChat (#3604)
## Summary

Adds full multimodal attachment support to CopilotKit's v2 chat
components. Users can send images, audio, video, and documents alongside
text messages to any AG-UI-compatible agent.

### Frontend
- **`attachments` prop** on `CopilotChat` — `{ enabled, accept, maxSize,
onUpload, onUploadFailed }`
- **`useAttachments` hook** — reusable, referentially stable hook for
custom chat UIs (all callbacks memoized, zero unnecessary re-renders)
- **Attachment queue** with lightbox previews (View Transition API),
video player, PDF/text document viewer
- **Drag-and-drop**, **clipboard paste**, and **file picker** upload
flows
- **Drop zone overlay** with visual indicator
- **Filename preservation** via `InputContent` metadata
- **`onUploadFailed` callback** with typed error reasons
(`file-too-large`, `invalid-type`, `upload-failed`)
- **`AttachmentUploadResult` discriminated union** for type-safe
`onUpload` returns
- Memoized attachment renderer components (v1 and v2)

### Backend
- `BuiltInAgent` properly converts multimodal `InputContent[]` to Vercel
AI SDK format
- Handles image, audio, video, document, and legacy binary content parts

### Deprecation lifecycle
- `@deprecated` JSDoc on all legacy image upload APIs (9 symbols) with
`@since 1.56.0`
- **Codemod** at `codemods/migrate-attachments.ts` — transforms props +
imports (18 tests)
- Migration guide moved to new **Migration Guides** docs section

### Docs
- New guide: `docs/(root)/multimodal-attachments.mdx`
- Updated migration guide with codemod instructions, new `onUpload`
type, metadata
- Cross-links from prebuilt-components page
- New **Migration Guides** section (moved from troubleshooting)

### AG-UI
- Bumped `@ag-ui/client`, `@ag-ui/core`, `@ag-ui/encoder`,
`@ag-ui/proto` to 0.0.51

### Example
- React Router example with dual agent setup (BuiltInAgent + TanStack
AI)
- Multimodal content properly forwarded to TanStack AI `chat()`

## Test plan

- [x] `useAttachments` hook stability tests — 9 tests (referential
stability, re-render counting, state defaults)
- [x] `CopilotChat.attachments` tests — 5 tests (onUploadFailed for
invalid-type, file-too-large, upload-failed, multiple rejections, valid
files)
- [x] Codemod tests — 18 tests (prop transforms, import renames,
idempotency, edge cases)
- [x] Full react-core suite — 966 tests passing
- [x] Tool call streaming tests pass (messagesMemoKey fingerprint fix)
2026-04-07 14:26:03 +02:00
Alem Tuzlak 773a4a6415 fix: format package.json files 2026-04-07 12:29:29 +02:00
Alem Tuzlak 0aca4e88bc fix: address review feedback — concurrent-safe consumeAttachments & scope-aware codemod
1. consumeAttachments: read from a ref mirror instead of side-effecting
   out of a setState updater, avoiding reliance on synchronous updater
   execution under React concurrent mode.

2. Codemod: skip declaration positions (variable, function, class, type,
   interface) and non-reference positions (object keys, member accesses).
   When a local declaration shadows the import name, only rename
   unambiguous type-position references to avoid corrupting unrelated code.
2026-04-07 12:14:14 +02:00
Alem Tuzlak 1317e51302 fix: revert copy-dts script, use typesVersions for legacy TS consumers
Reverts the copy-dts.mjs approach from #3612 in favor of typesVersions
which is the standard Node/TS mechanism for resolving subpath types
under legacy moduleResolution: "node".
2026-04-07 11:39:15 +02:00
Alem Tuzlak 136bfc1182 Merge branch 'main' into feat/multimodal-builtin-agent 2026-04-07 09:51:46 +02:00
Alem Tuzlak 9ae7a97823 fix: address PR review feedback from Martha
Bugs fixed:
- v1 Chat.tsx: onUpload now reads result.type/result.value (was result.data/result.url)
- v1 Chat.tsx: metadata captured from onUpload and forwarded in InputContent parts
- v1 Chat.tsx: onUploadFailed callback now invoked at all three error paths
- useAttachments: consumeAttachments preserves in-flight uploads (was dropping them)

Should-fix addressed:
- Lightbox: vtCounter replaced with React.useId() (concurrent mode + SSR safe)
- Lightbox: SSR guard before createPortal(document.body)
- useBlobUrl: useEffect deps changed to scalar values (was object reference)
- Codemod: dynamic inputFileAccept expressions preserved (was silently dropped)
- Codemod: scope-awareness limitation documented in comment

Tests: 966/966 react-core, 19/19 codemod
2026-04-07 09:42:28 +02:00
Benjamin Taylor 4094f30989 feat(runtime): Support configuring Threads lock TTL, prefix and heartbeat
feat(runtime): add defaults and max caps for thread lock TTL and heartbeat

lockTtlSeconds defaults to 20s (max 3600s / 1 hour) and
lockHeartbeatIntervalSeconds defaults to 15s (max 3000s / 50 minutes).
Both values are always set, so the heartbeat timer now starts
unconditionally and the paired-option validation is removed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 17:17:55 -07:00
Benjamin Taylor 3dc6963956 fix(threads): Improve useThreads DX with clearer names and direct type hints 2026-04-06 16:50:46 -07:00
Benjamin Taylor 9413015aa6 fix(react-core): clear copy button timeout on unmount
The CopyButton setTimeout fires after the jsdom test environment is
torn down, causing an unhandled ReferenceError on Node 20. Track the
timer in a ref and clear it on cleanup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 16:19:52 -07:00
Benjamin Taylor da2230ab32 feat(runtime): Add Segment tracking into v2 2026-04-06 16:19:52 -07:00
Martha Schumann c89f42d311 fix(tests): type MCPAppsActivityRenderer test renderer with Zod schema
Replace `ReactActivityMessageRenderer<unknown>` + `(content as any).resourceUri`
with `ReactActivityMessageRenderer<z.infer<typeof MCPAppsActivityContentSchema>>`
so the renderer prop is properly typed and the `as any` cast is eliminated.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 11:46:14 -07:00
Martha Schumann e6539d7f24 fix(tests): update MockMCPProxyAgent.clone() to share isRunning and runAgent state
After the useRenderActivityMessage fix, MCPAppsActivityRenderer receives the
per-thread clone instead of the registry agent. The mock's clone() did not share
isRunning state, causing waitForAgentIdle(clone) to never resolve (emit() only
updated registry.isRunning). Also, tests that monkey-patch runAgent before
renderWithCopilotKit need the clone to delegate proxied MCP requests to the
registry (to pick up the monkey-patch) while running user-message flows on the
clone itself (so clone.messages is updated and rendered by CopilotKit).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 10:11:56 -07:00
Martha Kelly Schumann 2d4782f68d Merge branch 'main' into fix/a2ui-activity-thread-clone 2026-04-06 10:04:55 -07:00
Martha Schumann 07b9ec13ec chore(a2ui): add /a2ui-demo page and DemoButtonAgent for manual fix validation
Adds a minimal demo at /a2ui-demo that uses the demo-button agent to
reproduce and validate the A2UI thread-clone bug fix. The agent renders
an A2UI surface with a Confirm button on first run; clicking it fires a
second run that emits a text confirmation message.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 09:45:23 -07:00
Alem Tuzlak 53a2e9c6e3 Merge branch 'main' into feat/multimodal-builtin-agent 2026-04-06 17:30:11 +02:00
Alem Tuzlak 626c33bda2 fix: stabilize useAttachments hook with useCallback, add stability tests
- All callbacks (processFiles, handleFileUpload, handleDragOver, handleDragLeave,
  handleDrop, removeAttachment, consumeAttachments) are now wrapped in useCallback
  with empty deps — referentially stable across renders
- Config values read from configRef to avoid dep array changes
- consumeAttachments no-ops on empty queue (returns same state reference)
- 9 new tests: referential stability across re-renders, re-render counting
  (consumeAttachments on empty queue triggers zero re-renders), initial state,
  consumeAttachments behavior, removeAttachment no-op
2026-04-06 16:20:42 +02:00
Alem Tuzlak d5449f3b5d fix: restore JSON.stringify messages memo to fix streaming tool call re-renders 2026-04-06 16:15:13 +02:00
Martha Schumann c1a560ba85 fix(a2ui): pass per-thread clone to activity message renderers
useRenderActivityMessage was calling copilotkit.getAgent() directly,
always returning the registry agent. When a user clicked an A2UI button,
handleAction → runAgent executed on the registry agent — messages
accumulated there while CopilotChat displayed from the per-thread clone
(created by useAgent), so responses appeared to silently vanish.

Apply the same getThreadClone(registryAgent, threadId) ?? registryAgent
pattern already used in useRenderCustomMessages.

Adds a regression test that asserts the renderer receives the clone,
and confirms the test catches the bug when the fix is reverted.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 07:01:22 -07:00
Alem Tuzlak d7578216d3 refactor: extract useAttachments hook, memoize attachment renderers
- Extract all attachment logic from CopilotChat into reusable useAttachments hook
  (state, processFiles, drag/drop, paste, consumeAttachments, removeAttachment)
- Export useAttachments from hooks/index.ts for custom chat UI builders
- Memoize v2 CopilotChatAttachmentRenderer sub-components
  (ImageAttachment, AudioAttachment, VideoAttachment, DocumentAttachment)
- Memoize v1 AttachmentRenderer sub-components to match v2 structure,
  align props (content → filename)
- Guard process.env in attachment-utils.ts for browser environments
- Remove debug logging (existing.debug = true) from use-agent.tsx
2026-04-06 15:17:56 +02:00
Alem Tuzlak 0fd3ddf960 refactor(docs): create migration-guides section, move migration pages from troubleshooting
- New docs/(root)/migration-guides/ section with migrate-to-v2, migrate-attachments,
  migrate-to-1.10.X, migrate-to-1.8.2
- Troubleshooting now only contains error-debugging, observability-connectors, common-issues
- Update all @deprecated JSDoc URLs from /troubleshooting/ to /migration-guides/
- Update doc cross-links (multimodal-attachments page, snippet import)
- Move snippet to snippets/shared/migration-guides/
- Set minimum version to v1.56.0 (was vX.Y.Z placeholder)
2026-04-06 14:55:09 +02:00
Alem Tuzlak 89d39d639b fix: address code review findings — error handling, codemod correctness, deprecation versions
- Restore console.error as safety net in processFiles catch blocks (errors no longer
  silently vanish when onUploadFailed callback is not provided)
- Add try/catch to useBlobUrl atob() — gracefully handles malformed base64 instead of
  crashing the component tree
- Fix DocumentLightboxContent: use blobUrl instead of undefined src variable
- Add .catch() to View Transition API transition.finished promise
- Codemod: preserve actual imageUploadsEnabled value (false, dynamic expressions)
  instead of hardcoding true — adds 3 new test cases (18 total)
- Replace all placeholder @deprecated versions (v1.x.0) with @since 1.56.0
- Add missing @since to ImageRenderer, ImageRendererProps, AIMessage.image tags
2026-04-06 14:55:08 +02:00
Alem Tuzlak 18c7a6001d feat: multimodal attachments — UI polish, deprecations, docs, codemod
Attachment queue & previews:
- Image lightbox with View Transition API morph animation
- Video lightbox with native controls and play button overlay
- Document lightbox (PDF via blob URL, text inline, info card fallback)
- Drop zone overlay with upload icon
- Filename preservation via InputContent metadata
- Proper video thumbnail sizing and play/pause indicator
- Fix attachment queue positioning (max-w-3xl constraint)
- Padding between X button and content for audio/document cards
- Document filenames wrap instead of truncating

Attachments config:
- onUploadFailed callback for validation/upload errors (file-too-large, invalid-type, upload-failed)
- onUpload accepts sync or async returns
- AttachmentUploadResult discriminated union with explicit interfaces
- Metadata field on Attachment and onUpload return type

AG-UI version bump:
- Bump @ag-ui/client, @ag-ui/core, @ag-ui/encoder, @ag-ui/proto to 0.0.51
- Remove process.env Vite workaround (fixed upstream in 0.0.51)

Deprecation lifecycle:
- @deprecated JSDoc on all legacy image upload APIs
- ImageRenderer, ImageRendererProps, ImageUpload type, imageUploadsEnabled prop,
  inputFileAccept prop, ImageRenderer prop, AIMessage.image, ImageData
- Codemod at codemods/migrate-attachments.ts (15 tests)
- Migration guide updated with codemod instructions and new type shapes

Docs:
- New guide: docs/(root)/multimodal-attachments.mdx
- Updated migration guide with onUpload return type, metadata, codemod section
- Cross-links from prebuilt-components and migration guide
- Label change: "Add photos or files" → "Add attachments"

Tests:
- CopilotChat.attachments.test.tsx — 5 tests for onUploadFailed
- migrate-attachments codemod — 15 tests
2026-04-06 14:55:06 +02:00
Alem Tuzlak 1603b63e33 style: format v2 multimodal changes 2026-04-06 14:54:46 +02:00
Alem Tuzlak cf0f8ad0b8 feat(react-core): export v2 attachment components and types 2026-04-06 14:54:45 +02:00
Alem Tuzlak c37d409bbf feat(react-core): add attachments prop, upload flow, drag-and-drop to v2 CopilotChat
Wire multimodal attachment support into the v2 CopilotChat component:
- Add AttachmentsConfig prop to CopilotChat for enabling file attachments
- Implement processFiles with accept filter, size validation, placeholder/ready lifecycle
- Support custom onUpload handlers and default base64 encoding
- Build InputContent[] when attachments accompany a message
- Add drag-and-drop handlers with visual feedback (dashed outline)
- Add clipboard paste handler for file items
- Render CopilotChatAttachmentQueue between scroll view and input
- Forward onAddFile through CopilotChatView to CopilotChatInput's AddMenuButton
- Omit internal attachment state props from CopilotChatProps public API
2026-04-06 14:54:44 +02:00
Alem Tuzlak ae1dc0ce4c feat(react-core): add v2 attachment renderer, update user message for multimodal, DRY shared utils 2026-04-06 14:54:43 +02:00
Alem Tuzlak 8f54f411ef feat(react-core): add v2 CopilotChatAttachmentQueue component 2026-04-06 14:54:42 +02:00
Alem Tuzlak f6c15cdbfe refactor(shared): move attachment utilities to shared package 2026-04-06 14:54:42 +02:00
Alem Tuzlak 7c774e9eb6 refactor(shared): move attachment types to shared package 2026-04-06 14:54:41 +02:00
Alem Tuzlak 152d1ade96 style: format code with oxfmt 2026-04-06 14:54:40 +02:00
Alem Tuzlak 928c385516 fix(react-ui): add baseline error logging, thumbnail diagnostics, rejected file feedback, mimeType fallback 2026-04-06 14:54:39 +02:00
Alem Tuzlak eeccbd3e36 fix(runtime): upgrade dropped content warnings to errors 2026-04-06 14:54:38 +02:00
Alem Tuzlak 2c546e44b5 test(react-ui): add tests for attachment utility functions 2026-04-06 14:54:37 +02:00
Alem Tuzlak e683f9386c fix(react-ui): address code review findings — stale closures, error handling, type safety
- Fix stale closure in paste handler by using processFilesRef pattern
- Wrap processFiles calls in try/catch across paste, file upload, and drop handlers
- Add early return for empty/falsy accept string in matchesAcceptFilter
- Block send while attachments are still uploading
- Add id field to Attachment interface, replace _id hack with randomUUID()
- Add 10s timeout to generateVideoThumbnail to prevent infinite hanging
- Show placeholder div for uploading attachments instead of rendering incomplete source
- Include filename context in upload error messages
2026-04-06 14:54:36 +02:00
Alem Tuzlak b6245d31fc fix(runtime): handle malformed URLs and unrecognized content parts defensively
Wrap all three new URL() call sites in try/catch so a malformed URL string
skips the part with a console.warn rather than crashing message conversion.
Add a default switch case to warn on unrecognized content part types.
Add a test covering the malformed-URL skip behaviour.
2026-04-06 14:54:36 +02:00
Alem Tuzlak 1d62da913d chore: add deprecation annotations and suppressable runtime warnings 2026-04-06 14:54:34 +02:00
Alem Tuzlak fc1cfa09d9 feat(react-ui): add attachments config, upload flow, drag-and-drop, and deprecation bridge 2026-04-06 14:54:33 +02:00
Alem Tuzlak ff236cfbe6 feat(react-ui): add AttachmentRenderer and update UserMessage for multimodal content 2026-04-06 14:54:32 +02:00
Alem Tuzlak c472f452f6 feat(react-ui): add AttachmentQueue component and styles 2026-04-06 14:54:31 +02:00
Alem Tuzlak 8bdb24ea87 feat(react-ui): add attachment types and utility functions 2026-04-06 14:54:30 +02:00
Alem Tuzlak c2c048f53f feat(runtime): support multimodal content in BuiltInAgent message conversion 2026-04-06 14:54:29 +02:00
Alem Tuzlak 0b05c3111d feat(shared): align UserMessage with AG-UI multimodal types
Re-export AG-UI multimodal input types (InputContent, TextInputPart, ImageInputPart,
AudioInputPart, VideoInputPart, DocumentInputPart, InputContentSource, etc.) from
@copilotkit/shared. UserMessage is now a plain pass-through of agui.UserMessage,
removing the CopilotKit-specific image?: ImageData extension. ImageData is kept with
@deprecated for backward compatibility with AIMessage.image.
2026-04-06 14:54:28 +02:00
Alem Tuzlak 7de4bfa9cf chore: bump @ag-ui/core and @ag-ui/client to 0.0.50-alpha.0 2026-04-06 14:54:27 +02:00
Ran Shemtov 2afc3c380e Merge branch 'main' into claude/explore-agentcore-cli-a25GL 2026-04-06 13:59:33 +02:00
Ran Shem Tov 1bf2bad15b fix(cli): add fixture tests for configureAgentCore config.yaml patching 2026-04-06 13:58:34 +02:00
Tyler Slaton 9c940dcaac Merge branch 'main' into claude/rebase-against-main-PGEp1 2026-04-05 21:46:05 -07:00
Claude ca2d081f8c fix: use preventScroll on focus calls so autoFocus opt-in doesn't scroll page
When users explicitly set autoFocus={true}, the textarea now focuses
with { preventScroll: true } so the page doesn't jump. Also applies
to the modal-open focus path. Added tests verifying default no-focus
behavior and that opt-in focus uses preventScroll.

https://claude.ai/code/session_01LPiJ8GcUNtWdS2W6MYSE4b
2026-04-05 00:39:36 +00:00
Claude f41133d356 fix: change autoFocus default to false in CopilotChatInput
Prevents the chat input from stealing focus on mount by default.
Users can opt-in via the autoFocus prop (e.g. input={{ autoFocus: true }}).

https://claude.ai/code/session_01LPiJ8GcUNtWdS2W6MYSE4b
2026-04-04 02:13:01 +00:00
Claude 7426486199 fix: remove scrollIntoView on chat input focus that forces page scroll
The v2 CopilotChatInput had a useEffect that called scrollIntoView()
whenever the textarea received focus. Combined with autoFocus defaulting
to true, this caused the entire page to scroll to the chat component on
load - the same bug that was previously fixed in v1.

https://claude.ai/code/session_01LPiJ8GcUNtWdS2W6MYSE4b
2026-04-04 02:07:38 +00:00
Maxim 208f9bc32f fix: restore .d.ts declaration files for legacy moduleResolution consumers
Copies .d.cts → .d.ts after each tsdown build so consumers on
moduleResolution "node" (e.g. DocuSign) can resolve types without
changing their tsconfig. Also points the "types" field back to .d.ts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 20:55:45 +02:00