Commit Graph

74 Commits

Author SHA1 Message Date
Alem Tuzlak 4b11b995a2 feat: multimodal attachments — images, audio, video, documents in CopilotChat (#3604)
## Summary

Adds full multimodal attachment support to CopilotKit's v2 chat
components. Users can send images, audio, video, and documents alongside
text messages to any AG-UI-compatible agent.

### Frontend
- **`attachments` prop** on `CopilotChat` — `{ enabled, accept, maxSize,
onUpload, onUploadFailed }`
- **`useAttachments` hook** — reusable, referentially stable hook for
custom chat UIs (all callbacks memoized, zero unnecessary re-renders)
- **Attachment queue** with lightbox previews (View Transition API),
video player, PDF/text document viewer
- **Drag-and-drop**, **clipboard paste**, and **file picker** upload
flows
- **Drop zone overlay** with visual indicator
- **Filename preservation** via `InputContent` metadata
- **`onUploadFailed` callback** with typed error reasons
(`file-too-large`, `invalid-type`, `upload-failed`)
- **`AttachmentUploadResult` discriminated union** for type-safe
`onUpload` returns
- Memoized attachment renderer components (v1 and v2)

### Backend
- `BuiltInAgent` properly converts multimodal `InputContent[]` to Vercel
AI SDK format
- Handles image, audio, video, document, and legacy binary content parts

### Deprecation lifecycle
- `@deprecated` JSDoc on all legacy image upload APIs (9 symbols) with
`@since 1.56.0`
- **Codemod** at `codemods/migrate-attachments.ts` — transforms props +
imports (18 tests)
- Migration guide moved to new **Migration Guides** docs section

### Docs
- New guide: `docs/(root)/multimodal-attachments.mdx`
- Updated migration guide with codemod instructions, new `onUpload`
type, metadata
- Cross-links from prebuilt-components page
- New **Migration Guides** section (moved from troubleshooting)

### AG-UI
- Bumped `@ag-ui/client`, `@ag-ui/core`, `@ag-ui/encoder`,
`@ag-ui/proto` to 0.0.51

### Example
- React Router example with dual agent setup (BuiltInAgent + TanStack
AI)
- Multimodal content properly forwarded to TanStack AI `chat()`

## Test plan

- [x] `useAttachments` hook stability tests — 9 tests (referential
stability, re-render counting, state defaults)
- [x] `CopilotChat.attachments` tests — 5 tests (onUploadFailed for
invalid-type, file-too-large, upload-failed, multiple rejections, valid
files)
- [x] Codemod tests — 18 tests (prop transforms, import renames,
idempotency, edge cases)
- [x] Full react-core suite — 966 tests passing
- [x] Tool call streaming tests pass (messagesMemoKey fingerprint fix)
2026-04-07 14:26:03 +02:00
Alem Tuzlak 773a4a6415 fix: format package.json files 2026-04-07 12:29:29 +02:00
Alem Tuzlak 0aca4e88bc fix: address review feedback — concurrent-safe consumeAttachments & scope-aware codemod
1. consumeAttachments: read from a ref mirror instead of side-effecting
   out of a setState updater, avoiding reliance on synchronous updater
   execution under React concurrent mode.

2. Codemod: skip declaration positions (variable, function, class, type,
   interface) and non-reference positions (object keys, member accesses).
   When a local declaration shadows the import name, only rename
   unambiguous type-position references to avoid corrupting unrelated code.
2026-04-07 12:14:14 +02:00
Alem Tuzlak 1317e51302 fix: revert copy-dts script, use typesVersions for legacy TS consumers
Reverts the copy-dts.mjs approach from #3612 in favor of typesVersions
which is the standard Node/TS mechanism for resolving subpath types
under legacy moduleResolution: "node".
2026-04-07 11:39:15 +02:00
Alem Tuzlak 136bfc1182 Merge branch 'main' into feat/multimodal-builtin-agent 2026-04-07 09:51:46 +02:00
Alem Tuzlak 9ae7a97823 fix: address PR review feedback from Martha
Bugs fixed:
- v1 Chat.tsx: onUpload now reads result.type/result.value (was result.data/result.url)
- v1 Chat.tsx: metadata captured from onUpload and forwarded in InputContent parts
- v1 Chat.tsx: onUploadFailed callback now invoked at all three error paths
- useAttachments: consumeAttachments preserves in-flight uploads (was dropping them)

Should-fix addressed:
- Lightbox: vtCounter replaced with React.useId() (concurrent mode + SSR safe)
- Lightbox: SSR guard before createPortal(document.body)
- useBlobUrl: useEffect deps changed to scalar values (was object reference)
- Codemod: dynamic inputFileAccept expressions preserved (was silently dropped)
- Codemod: scope-awareness limitation documented in comment

Tests: 966/966 react-core, 19/19 codemod
2026-04-07 09:42:28 +02:00
Benjamin Taylor 3dc6963956 fix(threads): Improve useThreads DX with clearer names and direct type hints 2026-04-06 16:50:46 -07:00
Benjamin Taylor 9413015aa6 fix(react-core): clear copy button timeout on unmount
The CopyButton setTimeout fires after the jsdom test environment is
torn down, causing an unhandled ReferenceError on Node 20. Track the
timer in a ref and clear it on cleanup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 16:19:52 -07:00
Martha Schumann c89f42d311 fix(tests): type MCPAppsActivityRenderer test renderer with Zod schema
Replace `ReactActivityMessageRenderer<unknown>` + `(content as any).resourceUri`
with `ReactActivityMessageRenderer<z.infer<typeof MCPAppsActivityContentSchema>>`
so the renderer prop is properly typed and the `as any` cast is eliminated.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 11:46:14 -07:00
Martha Schumann e6539d7f24 fix(tests): update MockMCPProxyAgent.clone() to share isRunning and runAgent state
After the useRenderActivityMessage fix, MCPAppsActivityRenderer receives the
per-thread clone instead of the registry agent. The mock's clone() did not share
isRunning state, causing waitForAgentIdle(clone) to never resolve (emit() only
updated registry.isRunning). Also, tests that monkey-patch runAgent before
renderWithCopilotKit need the clone to delegate proxied MCP requests to the
registry (to pick up the monkey-patch) while running user-message flows on the
clone itself (so clone.messages is updated and rendered by CopilotKit).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 10:11:56 -07:00
Martha Schumann 07b9ec13ec chore(a2ui): add /a2ui-demo page and DemoButtonAgent for manual fix validation
Adds a minimal demo at /a2ui-demo that uses the demo-button agent to
reproduce and validate the A2UI thread-clone bug fix. The agent renders
an A2UI surface with a Confirm button on first run; clicking it fires a
second run that emits a text confirmation message.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 09:45:23 -07:00
Alem Tuzlak 626c33bda2 fix: stabilize useAttachments hook with useCallback, add stability tests
- All callbacks (processFiles, handleFileUpload, handleDragOver, handleDragLeave,
  handleDrop, removeAttachment, consumeAttachments) are now wrapped in useCallback
  with empty deps — referentially stable across renders
- Config values read from configRef to avoid dep array changes
- consumeAttachments no-ops on empty queue (returns same state reference)
- 9 new tests: referential stability across re-renders, re-render counting
  (consumeAttachments on empty queue triggers zero re-renders), initial state,
  consumeAttachments behavior, removeAttachment no-op
2026-04-06 16:20:42 +02:00
Alem Tuzlak d5449f3b5d fix: restore JSON.stringify messages memo to fix streaming tool call re-renders 2026-04-06 16:15:13 +02:00
Martha Schumann c1a560ba85 fix(a2ui): pass per-thread clone to activity message renderers
useRenderActivityMessage was calling copilotkit.getAgent() directly,
always returning the registry agent. When a user clicked an A2UI button,
handleAction → runAgent executed on the registry agent — messages
accumulated there while CopilotChat displayed from the per-thread clone
(created by useAgent), so responses appeared to silently vanish.

Apply the same getThreadClone(registryAgent, threadId) ?? registryAgent
pattern already used in useRenderCustomMessages.

Adds a regression test that asserts the renderer receives the clone,
and confirms the test catches the bug when the fix is reverted.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 07:01:22 -07:00
Alem Tuzlak d7578216d3 refactor: extract useAttachments hook, memoize attachment renderers
- Extract all attachment logic from CopilotChat into reusable useAttachments hook
  (state, processFiles, drag/drop, paste, consumeAttachments, removeAttachment)
- Export useAttachments from hooks/index.ts for custom chat UI builders
- Memoize v2 CopilotChatAttachmentRenderer sub-components
  (ImageAttachment, AudioAttachment, VideoAttachment, DocumentAttachment)
- Memoize v1 AttachmentRenderer sub-components to match v2 structure,
  align props (content → filename)
- Guard process.env in attachment-utils.ts for browser environments
- Remove debug logging (existing.debug = true) from use-agent.tsx
2026-04-06 15:17:56 +02:00
Alem Tuzlak 89d39d639b fix: address code review findings — error handling, codemod correctness, deprecation versions
- Restore console.error as safety net in processFiles catch blocks (errors no longer
  silently vanish when onUploadFailed callback is not provided)
- Add try/catch to useBlobUrl atob() — gracefully handles malformed base64 instead of
  crashing the component tree
- Fix DocumentLightboxContent: use blobUrl instead of undefined src variable
- Add .catch() to View Transition API transition.finished promise
- Codemod: preserve actual imageUploadsEnabled value (false, dynamic expressions)
  instead of hardcoding true — adds 3 new test cases (18 total)
- Replace all placeholder @deprecated versions (v1.x.0) with @since 1.56.0
- Add missing @since to ImageRenderer, ImageRendererProps, AIMessage.image tags
2026-04-06 14:55:08 +02:00
Alem Tuzlak 18c7a6001d feat: multimodal attachments — UI polish, deprecations, docs, codemod
Attachment queue & previews:
- Image lightbox with View Transition API morph animation
- Video lightbox with native controls and play button overlay
- Document lightbox (PDF via blob URL, text inline, info card fallback)
- Drop zone overlay with upload icon
- Filename preservation via InputContent metadata
- Proper video thumbnail sizing and play/pause indicator
- Fix attachment queue positioning (max-w-3xl constraint)
- Padding between X button and content for audio/document cards
- Document filenames wrap instead of truncating

Attachments config:
- onUploadFailed callback for validation/upload errors (file-too-large, invalid-type, upload-failed)
- onUpload accepts sync or async returns
- AttachmentUploadResult discriminated union with explicit interfaces
- Metadata field on Attachment and onUpload return type

AG-UI version bump:
- Bump @ag-ui/client, @ag-ui/core, @ag-ui/encoder, @ag-ui/proto to 0.0.51
- Remove process.env Vite workaround (fixed upstream in 0.0.51)

Deprecation lifecycle:
- @deprecated JSDoc on all legacy image upload APIs
- ImageRenderer, ImageRendererProps, ImageUpload type, imageUploadsEnabled prop,
  inputFileAccept prop, ImageRenderer prop, AIMessage.image, ImageData
- Codemod at codemods/migrate-attachments.ts (15 tests)
- Migration guide updated with codemod instructions and new type shapes

Docs:
- New guide: docs/(root)/multimodal-attachments.mdx
- Updated migration guide with onUpload return type, metadata, codemod section
- Cross-links from prebuilt-components and migration guide
- Label change: "Add photos or files" → "Add attachments"

Tests:
- CopilotChat.attachments.test.tsx — 5 tests for onUploadFailed
- migrate-attachments codemod — 15 tests
2026-04-06 14:55:06 +02:00
Alem Tuzlak 1603b63e33 style: format v2 multimodal changes 2026-04-06 14:54:46 +02:00
Alem Tuzlak cf0f8ad0b8 feat(react-core): export v2 attachment components and types 2026-04-06 14:54:45 +02:00
Alem Tuzlak c37d409bbf feat(react-core): add attachments prop, upload flow, drag-and-drop to v2 CopilotChat
Wire multimodal attachment support into the v2 CopilotChat component:
- Add AttachmentsConfig prop to CopilotChat for enabling file attachments
- Implement processFiles with accept filter, size validation, placeholder/ready lifecycle
- Support custom onUpload handlers and default base64 encoding
- Build InputContent[] when attachments accompany a message
- Add drag-and-drop handlers with visual feedback (dashed outline)
- Add clipboard paste handler for file items
- Render CopilotChatAttachmentQueue between scroll view and input
- Forward onAddFile through CopilotChatView to CopilotChatInput's AddMenuButton
- Omit internal attachment state props from CopilotChatProps public API
2026-04-06 14:54:44 +02:00
Alem Tuzlak ae1dc0ce4c feat(react-core): add v2 attachment renderer, update user message for multimodal, DRY shared utils 2026-04-06 14:54:43 +02:00
Alem Tuzlak 8f54f411ef feat(react-core): add v2 CopilotChatAttachmentQueue component 2026-04-06 14:54:42 +02:00
Alem Tuzlak 7de4bfa9cf chore: bump @ag-ui/core and @ag-ui/client to 0.0.50-alpha.0 2026-04-06 14:54:27 +02:00
Claude ca2d081f8c fix: use preventScroll on focus calls so autoFocus opt-in doesn't scroll page
When users explicitly set autoFocus={true}, the textarea now focuses
with { preventScroll: true } so the page doesn't jump. Also applies
to the modal-open focus path. Added tests verifying default no-focus
behavior and that opt-in focus uses preventScroll.

https://claude.ai/code/session_01LPiJ8GcUNtWdS2W6MYSE4b
2026-04-05 00:39:36 +00:00
Claude f41133d356 fix: change autoFocus default to false in CopilotChatInput
Prevents the chat input from stealing focus on mount by default.
Users can opt-in via the autoFocus prop (e.g. input={{ autoFocus: true }}).

https://claude.ai/code/session_01LPiJ8GcUNtWdS2W6MYSE4b
2026-04-04 02:13:01 +00:00
Claude 7426486199 fix: remove scrollIntoView on chat input focus that forces page scroll
The v2 CopilotChatInput had a useEffect that called scrollIntoView()
whenever the textarea received focus. Combined with autoFocus defaulting
to true, this caused the entire page to scroll to the chat component on
load - the same bug that was previously fixed in v1.

https://claude.ai/code/session_01LPiJ8GcUNtWdS2W6MYSE4b
2026-04-04 02:07:38 +00:00
Maxim 208f9bc32f fix: restore .d.ts declaration files for legacy moduleResolution consumers
Copies .d.cts → .d.ts after each tsdown build so consumers on
moduleResolution "node" (e.g. DocuSign) can resolve types without
changing their tsconfig. Also points the "types" field back to .d.ts.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-03 20:55:45 +02:00
Tyler Slaton 180b719d50 Revert "fix: add types conditions to package exports for proper TypeScript resolution"
This reverts commit 499bca09ad.
2026-04-03 09:42:01 -07:00
Tyler Slaton 499bca09ad fix: add types conditions to package exports for proper TypeScript resolution
All @copilotkit/* packages were missing type declarations in their exports
map. This meant TypeScript couldn't resolve types for subpath imports like
@copilotkit/react-core/v2 without manual tsconfig paths workarounds
pointing to dist .d.ts files (which no longer exist after the tsup to
tsdown migration).

Uses nested conditional exports to map .d.mts for ESM and .d.cts for CJS
consumers, satisfying both attw and TypeScript module resolution under
nodenext and bundler modes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 14:56:09 -07:00
Maxim a2386017b4 Merge branch 'main' into feat/useagent-throttle-ms 2026-04-02 19:04:53 +02:00
Maxim 3f1c16881e fix: replace as any cast with typed RunAgentInput factory in throttle tests
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 18:46:35 +02:00
Maxim b2bc508f95 test: add missing throttleMs edge case coverage
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 18:11:31 +02:00
Maxim c2dcee2751 fix: unify effectiveThrottleMs to return 0, upgrade validation to console.error
Also fixes JSDoc inaccuracies: documents trailing-edge window restart,
uses enum names instead of callback names, adds notificationThrottle
interaction note.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 18:06:58 +02:00
Maxim 9b6d444c43 fix: trailing edge restarts throttle window to prevent back-to-back renders
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 18:01:43 +02:00
Alem Tuzlak 401a27c981 fix: resolve rebase conflicts and format files
- Remove stale eslint-config-custom refs from package.json files
- Regenerate pnpm-lock.yaml
- Format agentcore and other files with oxfmt
2026-04-02 16:43:34 +02:00
Alem Tuzlak 79ce60c580 chore: migrate from eslint+prettier to oxlint+oxfmt
Replace eslint and prettier with oxlint and oxfmt for faster linting
and formatting across the monorepo. Remove all eslint and prettier
configs, dependencies, and related packages. Add .oxlintrc.json and
.oxfmtrc.json for the new tooling. Update CI workflows and lefthook
hooks accordingly. Reformat codebase with oxfmt.

https://claude.ai/code/session_01GMkSf29p78HuMR1mbXn8He
2026-04-02 16:39:05 +02:00
Maxim a4109ad3be refactor: enhance message handling in useAgent tests
- Introduced message factory functions to eliminate `as any` type assertions for message literals.
- Updated test cases to utilize the new message factories for improved clarity and type safety.
- Added helper functions for subscriber notifications to streamline test setup.

This refactor improves code readability and maintainability in the test suite for the useAgent hook.
2026-04-02 16:33:13 +02:00
Maxim d1357083da fix: harden throttleMs implementation from code review
- Replace Date.now()-based throttle with flag-based leading+trailing
  pattern that is immune to wall-clock jumps
- Wrap throttleMs validation in useMemo to warn once per value change
  instead of on every render
- Remove phantom AbstractAgent.notificationThrottleMs JSDoc reference
- Reorder cleanup to clear timer before unsubscribing for safety
- Add tests for trailing-edge data freshness, timer precision, and
  dynamic throttleMs prop changes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 16:13:56 +02:00
Maxim f2b148c0d2 fix: improve throttleMs validation, simplify handlers, add invalid-input tests
- Fix JSDoc to accurately describe behavior for negative/non-finite values
- Remove NODE_ENV guard so console.warn fires in all environments
- Use derived effectiveThrottleMs instead of mutating destructured param
- Simplify handler wrappers: direct references instead of arrow wrappers
- Remove stale comment about content stripping
- Add parameterized tests for invalid throttleMs (NaN, Infinity, -1, -Infinity)
- Fix misleading test comment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 15:44:13 +02:00
Maxim 3cdaaf089d fix: consolidate throttleMs into single useEffect, add input validation and edge-case tests
Addresses code review findings:
- Merge two separate useEffect hooks into one to eliminate dual-subscription
  race window and simplify cleanup
- Add active boolean guard to prevent forceUpdate after unmount/re-run
- Validate throttleMs input (reject NaN, Infinity, negatives with dev warning)
- Expand JSDoc with scope, constraints, and notificationThrottleMs interaction
- Replace JSON.stringify(updateFlags) with direct ref in effect deps
- Strengthen unmount test with render counter instead of not.toThrow()
- Add 5 new test cases: explicit zero, rapid burst, new cycle after trailing,
  updates excluding OnMessagesChanged, subscription cleanup verification

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 15:34:43 +02:00
Maxim 30916a6ab7 feat: add throttleMs option to useAgent for streaming render throttling
Adds optional `throttleMs` parameter to `useAgent()` that coalesces
rapid onMessagesChanged notifications (per-token TEXT_MESSAGE_CHUNK
events) into fewer React re-renders using a leading+trailing throttle.

- throttleMs=0 (default): exact current behavior, zero overhead
- throttleMs>0: only onMessagesChanged is throttled; onStateChanged
  and onRunStatusChanged always fire immediately
- Leading+trailing: first call immediate, last call guaranteed

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 14:09:17 +02:00
Claude 0d9e3b88f6 chore(post-release): update version to 1.55.0-next.8
The previous publish workflow released 1.55.0-next.8 to npm but the
post-release commit failed to push to main (blocked by branch
protection). This applies the version bumps and pre.json updates that
the workflow would have committed, unblocking future releases.

https://claude.ai/code/session_01N8sLK9h9p6FdRjyHqzKzVV
2026-04-02 02:23:27 +00:00
Claude d55ce2c87b fix: add React keys to CopilotMessages children to suppress warning
CopilotMessages received an array of children (memoizedChildren +
RegisteredActionsRenderer) without keys, causing a React "unique key prop"
warning in dev mode. Wrap memoizedChildren in a keyed Fragment and add a
key to RegisteredActionsRenderer.

https://claude.ai/code/session_01N8sLK9h9p6FdRjyHqzKzVV
2026-04-02 01:32:52 +00:00
Martha Schumann 092956d299 style: fix prettier formatting across all changed files 2026-03-30 13:09:10 -07:00
Martha Schumann 6395a9a42d style: fix prettier formatting in CopilotChatMessageView after merge 2026-03-30 13:04:44 -07:00
Martha Schumann 83f58ee3de Merge branch 'main' into fix/cpk-7155-thread-shared-state
Incorporates the V1/V2 flat package consolidation (96885b595). All our
thread-isolation changes auto-merged to the new package paths. Three
content conflicts resolved (import paths updated from @/ aliases to
relative paths and @copilotkitnext/* to @copilotkit/*):
- packages/react-core/src/v2/hooks/use-agent.tsx
- packages/react-core/src/v2/hooks/use-render-custom-messages.tsx
- packages/react-core/src/v2/components/chat/CopilotChatMessageView.tsx

File location conflict resolved: use-agent-thread-isolation.test.tsx
moved from packages/v2/react/... to packages/react-core/src/v2/...
2026-03-30 12:33:16 -07:00
github-actions[bot] a1edf59eae chore(post-release): update version to 1.55.0-next.7 2026-03-29 00:19:22 +00:00
Claude 1ceb963a84 Revert "fix: resolve changeset version errors from stale @copilotkitnext package refs"
This reverts commit 1f6ae0d629.
2026-03-28 17:04:55 -07:00
Claude 369266f1be fix: resolve changeset version errors from stale @copilotkitnext package refs
After the V1/V2 consolidation merge, several changesets still referenced
old @copilotkitnext/* package names that no longer exist. Fixed package
references and ran changeset version to apply pending version bumps.

https://claude.ai/code/session_01A5aahBWFTmLPR3xZSeh4SD
2026-03-28 17:04:55 -07:00
Tyler Slaton 96885b5959 refactor: consolidate V1/V2 packages into flat @copilotkit/* structure
Flatten all packages from packages/v1/* and packages/v2/* into packages/* —
every package now lives directly under the @copilotkit/ scope with no v1/v2
subdirectories.

- Move all v1 packages (react-core, react-ui, runtime, shared, etc.) from
  packages/v1/* to packages/*
- Absorb v2 react code into packages/react-core/src/v2/ (exported via /v2 subpath)
- Absorb v2 agent code into packages/runtime/src/agent/ (exported via /v2 subpath)
- Move v2 packages (core, angular, demo-agents, etc.) to packages/*
- Replace all @copilotkitnext/* imports with @copilotkit/* equivalents
- Keep @copilotkitnext/angular as the sole exception (angular remains on next)
- Update CI workflows, renovate config, release scripts for flat structure
- No public API surface changes — all exports fields are preserved

Co-authored-by: Alem Tuzlak <t.zlak@hotmail.com>
Signed-off-by: Tyler Slaton <tyler@copilotkit.ai>
2026-03-28 16:45:10 -07:00