Commit Graph

4870 Commits

Author SHA1 Message Date
Ran Shem Tov e21ba4c9f6 chore: use latest ag-ui langgraph 2026-04-07 14:51:28 +02:00
Ran Shem Tov 5e962f0cda fix: ensure values are safe to stringify 2026-04-07 14:50:55 +02:00
Alem Tuzlak 44f84ede44 chore: add minor changeset for multimodal builtin agent (#3643)
## Summary
- Adds a minor version changeset for the multimodal builtin agent
feature that was merged without one
- Bumps `@copilotkit/runtime`, `@copilotkit/react-core`,
`@copilotkit/react-ui`, and `@copilotkit/shared` (fixed versioning
applies to all `@copilotkit/*` packages)

## Test plan
- [ ] Verify changeset is picked up by the release workflow
2026-04-07 14:42:10 +02:00
Alem Tuzlak 528979194a chore: add minor changeset for multimodal builtin agent 2026-04-07 14:30:30 +02:00
Alem Tuzlak 4b11b995a2 feat: multimodal attachments — images, audio, video, documents in CopilotChat (#3604)
## Summary

Adds full multimodal attachment support to CopilotKit's v2 chat
components. Users can send images, audio, video, and documents alongside
text messages to any AG-UI-compatible agent.

### Frontend
- **`attachments` prop** on `CopilotChat` — `{ enabled, accept, maxSize,
onUpload, onUploadFailed }`
- **`useAttachments` hook** — reusable, referentially stable hook for
custom chat UIs (all callbacks memoized, zero unnecessary re-renders)
- **Attachment queue** with lightbox previews (View Transition API),
video player, PDF/text document viewer
- **Drag-and-drop**, **clipboard paste**, and **file picker** upload
flows
- **Drop zone overlay** with visual indicator
- **Filename preservation** via `InputContent` metadata
- **`onUploadFailed` callback** with typed error reasons
(`file-too-large`, `invalid-type`, `upload-failed`)
- **`AttachmentUploadResult` discriminated union** for type-safe
`onUpload` returns
- Memoized attachment renderer components (v1 and v2)

### Backend
- `BuiltInAgent` properly converts multimodal `InputContent[]` to Vercel
AI SDK format
- Handles image, audio, video, document, and legacy binary content parts

### Deprecation lifecycle
- `@deprecated` JSDoc on all legacy image upload APIs (9 symbols) with
`@since 1.56.0`
- **Codemod** at `codemods/migrate-attachments.ts` — transforms props +
imports (18 tests)
- Migration guide moved to new **Migration Guides** docs section

### Docs
- New guide: `docs/(root)/multimodal-attachments.mdx`
- Updated migration guide with codemod instructions, new `onUpload`
type, metadata
- Cross-links from prebuilt-components page
- New **Migration Guides** section (moved from troubleshooting)

### AG-UI
- Bumped `@ag-ui/client`, `@ag-ui/core`, `@ag-ui/encoder`,
`@ag-ui/proto` to 0.0.51

### Example
- React Router example with dual agent setup (BuiltInAgent + TanStack
AI)
- Multimodal content properly forwarded to TanStack AI `chat()`

## Test plan

- [x] `useAttachments` hook stability tests — 9 tests (referential
stability, re-render counting, state defaults)
- [x] `CopilotChat.attachments` tests — 5 tests (onUploadFailed for
invalid-type, file-too-large, upload-failed, multiple rejections, valid
files)
- [x] Codemod tests — 18 tests (prop transforms, import renames,
idempotency, edge cases)
- [x] Full react-core suite — 966 tests passing
- [x] Tool call streaming tests pass (messagesMemoKey fingerprint fix)
2026-04-07 14:26:03 +02:00
Maxim 2485c486b6 fix: revert copy-dts script, use typesVersions for legacy TS consumers (#3640)
## Summary
- Reverts the `copy-dts.mjs` approach from #3612 — restores `"types"`
back to `.d.cts` and removes the post-build copy step from all 13
packages
- Adds `typesVersions` to `react-core` and `runtime` (the packages with
`./v2` subpath exports) so legacy `moduleResolution: "node"` consumers
can resolve subpath types
- Deletes `scripts/copy-dts.mjs`

From what I've read, `typesVersions` is the standard Node/TS mechanism
for this — we can safely revert the copy-dts changes and rely on it
instead.

@mxmzb heads up — this reverts your #3612 changes in favor of
`typesVersions`. Let me know if you have any concerns.

## Test plan
- [ ] Build all packages and verify `.d.cts` types are emitted (no
`.d.ts` copies)
- [ ] Verify `typesVersions` resolves `@copilotkit/react-core/v2` and
`@copilotkit/runtime/v2` types under `moduleResolution: "node"`
- [ ] Verify main entry types still resolve via the `"types"` field
pointing to `.d.cts`
2026-04-07 14:22:09 +02:00
Maxim 6c975bfd11 Merge branch 'main' into fix/revert-copy-dts-use-typesVersions 2026-04-07 13:54:12 +02:00
Ran Shemtov 1ccd28ea6e chore: loosen up langgraph dependencies on copilotkit and release (#3642) 2026-04-07 13:17:08 +02:00
Ran Shem Tov a7e3e131d5 chore: loosen up langgraph dependencies on copilotkit and release 2026-04-07 13:14:00 +02:00
Alem Tuzlak 773a4a6415 fix: format package.json files 2026-04-07 12:29:29 +02:00
Alem Tuzlak 31247f3663 chore: format codemods files 2026-04-07 12:29:28 +02:00
Alem Tuzlak 94c5bba46e fix: concurrent-safe consumeAttachments & scope-aware codemod renames (#3641)
## Summary
Addresses two review issues from #3604:

- **consumeAttachments race**: replaced the
side-effect-out-of-setState-updater pattern with a ref mirror
(`attachmentsRef`). `consumeAttachments` now reads from the ref
synchronously and uses the updater only for the state mutation. Safe
under React concurrent mode.
- **Codemod scope-awareness**: the rename logic now skips declaration
positions (variable, function, class, type, interface declarations),
object property keys, and member expression properties. When a local
declaration shadows the import name, only unambiguous type-position
references are renamed — value references are left alone to avoid
silently corrupting unrelated code.

## Test plan
- [x] All 21 codemod tests pass (including 2 new tests for
shadow/collision cases)
- [x] All 9 use-attachments hook tests pass
2026-04-07 12:18:07 +02:00
Alem Tuzlak 0aca4e88bc fix: address review feedback — concurrent-safe consumeAttachments & scope-aware codemod
1. consumeAttachments: read from a ref mirror instead of side-effecting
   out of a setState updater, avoiding reliance on synchronous updater
   execution under React concurrent mode.

2. Codemod: skip declaration positions (variable, function, class, type,
   interface) and non-reference positions (object keys, member accesses).
   When a local declaration shadows the import name, only rename
   unambiguous type-position references to avoid corrupting unrelated code.
2026-04-07 12:14:14 +02:00
Alem Tuzlak 1317e51302 fix: revert copy-dts script, use typesVersions for legacy TS consumers
Reverts the copy-dts.mjs approach from #3612 in favor of typesVersions
which is the standard Node/TS mechanism for resolving subpath types
under legacy moduleResolution: "node".
2026-04-07 11:39:15 +02:00
Alem Tuzlak 136bfc1182 Merge branch 'main' into feat/multimodal-builtin-agent 2026-04-07 09:51:46 +02:00
Alem Tuzlak 9ae7a97823 fix: address PR review feedback from Martha
Bugs fixed:
- v1 Chat.tsx: onUpload now reads result.type/result.value (was result.data/result.url)
- v1 Chat.tsx: metadata captured from onUpload and forwarded in InputContent parts
- v1 Chat.tsx: onUploadFailed callback now invoked at all three error paths
- useAttachments: consumeAttachments preserves in-flight uploads (was dropping them)

Should-fix addressed:
- Lightbox: vtCounter replaced with React.useId() (concurrent mode + SSR safe)
- Lightbox: SSR guard before createPortal(document.body)
- useBlobUrl: useEffect deps changed to scalar values (was object reference)
- Codemod: dynamic inputFileAccept expressions preserved (was silently dropped)
- Codemod: scope-awareness limitation documented in comment

Tests: 966/966 react-core, 19/19 codemod
2026-04-07 09:42:28 +02:00
Max Korp 5cbfdb1bcf fix: format files (#3635)
Looks like something got past oxfmt in
https://github.com/CopilotKit/CopilotKit/pull/3592
2026-04-06 17:26:35 -07:00
Max Korp a8876d4b97 feat(runtime): Support configuring Threads lock TTL, prefix and heartbeat (#3536) 2026-04-06 17:25:49 -07:00
Max Korp 3bd279d680 fix: format files 2026-04-06 17:20:06 -07:00
Benjamin Taylor 4094f30989 feat(runtime): Support configuring Threads lock TTL, prefix and heartbeat
feat(runtime): add defaults and max caps for thread lock TTL and heartbeat

lockTtlSeconds defaults to 20s (max 3600s / 1 hour) and
lockHeartbeatIntervalSeconds defaults to 15s (max 3000s / 50 minutes).
Both values are always set, so the heartbeat timer now starts
unconditionally and the paired-option validation is removed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 17:17:55 -07:00
Max Korp e1205b005e fix(threads): Improve useThreads DX with clearer names and direct type hints (#3556) 2026-04-06 17:17:30 -07:00
Benjamin Taylor 3dc6963956 fix(threads): Improve useThreads DX with clearer names and direct type hints 2026-04-06 16:50:46 -07:00
Max Korp 418c26fd50 feat(runtime): Add Segment tracking into v2 (#3540) 2026-04-06 16:49:56 -07:00
Benjamin Taylor 9413015aa6 fix(react-core): clear copy button timeout on unmount
The CopyButton setTimeout fires after the jsdom test environment is
torn down, causing an unhandled ReferenceError on Node 20. Track the
timer in a ref and clear it on cleanup.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-06 16:19:52 -07:00
Benjamin Taylor da2230ab32 feat(runtime): Add Segment tracking into v2 2026-04-06 16:19:52 -07:00
Jordan Ritter d9f58fb388 feat(showcase): add deep-agents-finance-erp showcase (#3592)
## Summary

Adds a full-featured **Finance ERP showcase** demonstrating CopilotKit
v2 deep agents with:

- **Multi-agent orchestrator** (research + projections subagents via
`deepagents`)
- **14 frontend tools** rendered inline in the CopilotSidebar — charts,
cash position cards, dashboard widgets, and human-in-the-loop approval
dialogs
- **Customizable dashboard** with drag/resize widgets controlled by the
AI agent
- **Component gallery** at `/components` showcasing all frontend tool
renderers

### Architecture

```
Orchestrator (FinanceOS AI)
├── research subagent — queries ERP data (invoices, accounts, inventory, HR)
├── projections subagent — forecasts, scenario analysis, trend computation
└── frontend tools (registered as backend stubs) — AG-UI streams events,
    CopilotKit frontend matches to useFrontendTool renderers
```

Frontend tools are registered as lightweight `@tool` stubs on the
orchestrator. When the model calls them, the AG-UI adapter streams
`ToolCall` events that CopilotKit matches to `useFrontendTool` /
`useHumanInTheLoop` renderers by name. HITL tools use
`copilotkit_interrupt()` to pause the graph for user approval.

### Key files

| Area | Files |
|---|---|
| Agent | `agent/agent.py`, `agent/prompts.py`, `agent/tools.py`,
`agent/frontend_tools.py` |
| Frontend hooks | `src/hooks/use-render-chart.ts`,
`use-render-cash-position.ts`, `use-approve-invoice-payment.ts`, ... (14
total) |
| Dashboard | `src/context/dashboard-context.tsx`,
`src/components/dashboard/` |
| Layout | `src/components/layout/shell.tsx` (tool registration +
context sharing) |

All changes are scoped to `examples/showcases/deep-agents-finance-erp/`
— no SDK or package modifications.

## Test plan

- [ ] `cd agent && pip install -e . && python main.py` — agent starts on
:8123
- [ ] `cd .. && npm install && npm run dev` — frontend starts on :3000
- [ ] "What's our current cash position?" → `CashPositionCard` renders
inline
- [ ] "Give me a cash flow projection" → `InlineChatChart` renders
- [ ] "Process payment for overdue invoices" → `InvoiceApprovalCard`
with Approve/Reject
- [ ] "Remove the expense breakdown" → dashboard widget removed
- [ ] `/components` page renders all tool previews

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-04-06 16:19:14 -07:00
GeneralJerel c186e20e0d feat(showcase): add deep-agents-finance-erp project
Finance ERP showcase powered by CopilotKit deep agents. Includes a
Next.js 16 frontend with 7 CopilotKit-registered tools, a LangGraph
multi-agent backend (orchestrator + research/projections subagents),
Postgres-backed dashboard gallery with 12 widget types and 4 templates,
and human-in-the-loop approval flows for invoices and inventory.
2026-04-06 16:16:12 -07:00
Jordan Ritter f17d577014 fix: redirect /whats-new/v1-50 to correct destination (#3632)
## Summary

- Fix the `/whats-new/v1-50` redirect destination from `/` (homepage) to
`/learn/whats-new/v1-50` (where the content actually lives)

This redirect was changed in commit `7a78f3f0` (Mar 20) to point to
root, but the v1.50 release notes page still exists at
`/learn/whats-new/v1-50`. Shared links and bookmarks to the old URL
currently land on the homepage instead of the content.

## Test plan

- [ ] Navigate to `/whats-new/v1-50` and verify it redirects to
`/learn/whats-new/v1-50`
- [ ] Verify `/learn/whats-new/v1-50` renders the v1.50 release notes

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-04-06 13:11:30 -07:00
Jordan Ritter b3ebc1cba9 fix: redirect /whats-new/v1-50 to /learn/whats-new/v1-50 instead of root 2026-04-06 13:00:15 -07:00
Martha Kelly Schumann 31d5cf401b fix(a2ui): pass per-thread clone to activity message renderers (#3630)
## Summary

Fixes A2UI button responses not appearing in chat after PR #3525
introduced per-thread agent cloning.

- **Root cause:** `useRenderActivityMessage` was passing the registry
agent to `ReactSurfaceHost` instead of the per-thread clone. When a
button was clicked, `handleAction` called `runAgent` on the registry
agent, but `CopilotChat` was rendering from the per-thread clone — so
responses were silently lost.
- **Fix:** Mirror the existing pattern from `useRenderCustomMessages`:
resolve `getThreadClone(registryAgent, threadId) ?? registryAgent`
before passing the agent prop down.
- **Regression test:** New test in
`CopilotChatActivityRendering.e2e.test.tsx` asserts that the `agent`
prop passed to an activity renderer is the per-thread clone, not the
registry agent.
- **Demo:** New page at `/a2ui-demo` with a `DemoButtonAgent` — renders
an A2UI button on first run, confirms the fix is working when clicked.

## Why this was missed in #3525

`useRenderCustomMessages` was updated with the thread-clone pattern but
`useRenderActivityMessage` was not. The two hooks serve similar roles
but the PR review focused on `useAgent` and `CopilotChat`, not on every
consumer of the registry agent.

## Impact of the temporary hack in #3588

PR #3588 (`ae48d4761e5`) returned `existing` (registry agent) from
`useAgent` unconditionally, making both `CopilotChat` and the activity
renderer use the same instance — so button clicks worked again. But this
discards the thread isolation from #3525: multiple `CopilotChat`
components sharing a `threadId` would bleed messages into each other.
This PR fixes the root cause so the #3588 workaround is no longer
needed.

## Test plan

- [x] Run `nx run @copilotkit/react-core:test --
CopilotChatActivityRendering` — all 4 tests pass
- [x] Run `nx run demo:dev`, open `/a2ui-demo`, type anything, click
Confirm — confirmation message appears in chat

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-04-06 12:01:00 -07:00
Martha Schumann c89f42d311 fix(tests): type MCPAppsActivityRenderer test renderer with Zod schema
Replace `ReactActivityMessageRenderer<unknown>` + `(content as any).resourceUri`
with `ReactActivityMessageRenderer<z.infer<typeof MCPAppsActivityContentSchema>>`
so the renderer prop is properly typed and the `as any` cast is eliminated.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 11:46:14 -07:00
Martha Schumann e6539d7f24 fix(tests): update MockMCPProxyAgent.clone() to share isRunning and runAgent state
After the useRenderActivityMessage fix, MCPAppsActivityRenderer receives the
per-thread clone instead of the registry agent. The mock's clone() did not share
isRunning state, causing waitForAgentIdle(clone) to never resolve (emit() only
updated registry.isRunning). Also, tests that monkey-patch runAgent before
renderWithCopilotKit need the clone to delegate proxied MCP requests to the
registry (to pick up the monkey-patch) while running user-message flows on the
clone itself (so clone.messages is updated and rendered by CopilotKit).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 10:11:56 -07:00
Martha Kelly Schumann 2d4782f68d Merge branch 'main' into fix/a2ui-activity-thread-clone 2026-04-06 10:04:55 -07:00
Martha Schumann 07b9ec13ec chore(a2ui): add /a2ui-demo page and DemoButtonAgent for manual fix validation
Adds a minimal demo at /a2ui-demo that uses the demo-button agent to
reproduce and validate the A2UI thread-clone bug fix. The agent renders
an A2UI surface with a Confirm button on first run; clicking it fires a
second run that emits a text confirmation message.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 09:45:23 -07:00
Alem Tuzlak 53a2e9c6e3 Merge branch 'main' into feat/multimodal-builtin-agent 2026-04-06 17:30:11 +02:00
Ran Shemtov 195d4526cb feat: add AgentCore framework support to CLI (#3600) 2026-04-06 17:28:47 +02:00
Alem Tuzlak 626c33bda2 fix: stabilize useAttachments hook with useCallback, add stability tests
- All callbacks (processFiles, handleFileUpload, handleDragOver, handleDragLeave,
  handleDrop, removeAttachment, consumeAttachments) are now wrapped in useCallback
  with empty deps — referentially stable across renders
- Config values read from configRef to avoid dep array changes
- consumeAttachments no-ops on empty queue (returns same state reference)
- 9 new tests: referential stability across re-renders, re-render counting
  (consumeAttachments on empty queue triggers zero re-renders), initial state,
  consumeAttachments behavior, removeAttachment no-op
2026-04-06 16:20:42 +02:00
Alem Tuzlak d5449f3b5d fix: restore JSON.stringify messages memo to fix streaming tool call re-renders 2026-04-06 16:15:13 +02:00
Martha Schumann c1a560ba85 fix(a2ui): pass per-thread clone to activity message renderers
useRenderActivityMessage was calling copilotkit.getAgent() directly,
always returning the registry agent. When a user clicked an A2UI button,
handleAction → runAgent executed on the registry agent — messages
accumulated there while CopilotChat displayed from the per-thread clone
(created by useAgent), so responses appeared to silently vanish.

Apply the same getThreadClone(registryAgent, threadId) ?? registryAgent
pattern already used in useRenderCustomMessages.

Adds a regression test that asserts the renderer receives the clone,
and confirms the test catches the bug when the fix is reverted.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-06 07:01:22 -07:00
Alem Tuzlak 5f464e6a0d fix(example): map AG-UI text parts to TanStack AI format (text → content) 2026-04-06 15:34:17 +02:00
Alem Tuzlak 39344c00f7 fix(example): preserve multimodal content arrays in TanStack AI agent 2026-04-06 15:30:08 +02:00
Alem Tuzlak 4eedc2ee45 chore: sync pnpm-lock.yaml after rebase 2026-04-06 15:23:28 +02:00
Alem Tuzlak d7578216d3 refactor: extract useAttachments hook, memoize attachment renderers
- Extract all attachment logic from CopilotChat into reusable useAttachments hook
  (state, processFiles, drag/drop, paste, consumeAttachments, removeAttachment)
- Export useAttachments from hooks/index.ts for custom chat UI builders
- Memoize v2 CopilotChatAttachmentRenderer sub-components
  (ImageAttachment, AudioAttachment, VideoAttachment, DocumentAttachment)
- Memoize v1 AttachmentRenderer sub-components to match v2 structure,
  align props (content → filename)
- Guard process.env in attachment-utils.ts for browser environments
- Remove debug logging (existing.debug = true) from use-agent.tsx
2026-04-06 15:17:56 +02:00
Alem Tuzlak 0fd3ddf960 refactor(docs): create migration-guides section, move migration pages from troubleshooting
- New docs/(root)/migration-guides/ section with migrate-to-v2, migrate-attachments,
  migrate-to-1.10.X, migrate-to-1.8.2
- Troubleshooting now only contains error-debugging, observability-connectors, common-issues
- Update all @deprecated JSDoc URLs from /troubleshooting/ to /migration-guides/
- Update doc cross-links (multimodal-attachments page, snippet import)
- Move snippet to snippets/shared/migration-guides/
- Set minimum version to v1.56.0 (was vX.Y.Z placeholder)
2026-04-06 14:55:09 +02:00
Alem Tuzlak 89d39d639b fix: address code review findings — error handling, codemod correctness, deprecation versions
- Restore console.error as safety net in processFiles catch blocks (errors no longer
  silently vanish when onUploadFailed callback is not provided)
- Add try/catch to useBlobUrl atob() — gracefully handles malformed base64 instead of
  crashing the component tree
- Fix DocumentLightboxContent: use blobUrl instead of undefined src variable
- Add .catch() to View Transition API transition.finished promise
- Codemod: preserve actual imageUploadsEnabled value (false, dynamic expressions)
  instead of hardcoding true — adds 3 new test cases (18 total)
- Replace all placeholder @deprecated versions (v1.x.0) with @since 1.56.0
- Add missing @since to ImageRenderer, ImageRendererProps, AIMessage.image tags
2026-04-06 14:55:08 +02:00
Alem Tuzlak a076a69852 fix: correct tsconfig extends paths for v1 node examples 2026-04-06 14:55:07 +02:00
Alem Tuzlak 18c7a6001d feat: multimodal attachments — UI polish, deprecations, docs, codemod
Attachment queue & previews:
- Image lightbox with View Transition API morph animation
- Video lightbox with native controls and play button overlay
- Document lightbox (PDF via blob URL, text inline, info card fallback)
- Drop zone overlay with upload icon
- Filename preservation via InputContent metadata
- Proper video thumbnail sizing and play/pause indicator
- Fix attachment queue positioning (max-w-3xl constraint)
- Padding between X button and content for audio/document cards
- Document filenames wrap instead of truncating

Attachments config:
- onUploadFailed callback for validation/upload errors (file-too-large, invalid-type, upload-failed)
- onUpload accepts sync or async returns
- AttachmentUploadResult discriminated union with explicit interfaces
- Metadata field on Attachment and onUpload return type

AG-UI version bump:
- Bump @ag-ui/client, @ag-ui/core, @ag-ui/encoder, @ag-ui/proto to 0.0.51
- Remove process.env Vite workaround (fixed upstream in 0.0.51)

Deprecation lifecycle:
- @deprecated JSDoc on all legacy image upload APIs
- ImageRenderer, ImageRendererProps, ImageUpload type, imageUploadsEnabled prop,
  inputFileAccept prop, ImageRenderer prop, AIMessage.image, ImageData
- Codemod at codemods/migrate-attachments.ts (15 tests)
- Migration guide updated with codemod instructions and new type shapes

Docs:
- New guide: docs/(root)/multimodal-attachments.mdx
- Updated migration guide with onUpload return type, metadata, codemod section
- Cross-links from prebuilt-components and migration guide
- Label change: "Add photos or files" → "Add attachments"

Tests:
- CopilotChat.attachments.test.tsx — 5 tests for onUploadFailed
- migrate-attachments codemod — 15 tests
2026-04-06 14:55:06 +02:00
Alem Tuzlak 085ac18057 feat(example): dual agent setup — BuiltInAgent + TanStack AI with agent toggle 2026-04-06 14:54:47 +02:00
Alem Tuzlak 64ffb9bee9 feat(example): update react-router example to test multimodal attachments 2026-04-06 14:54:47 +02:00
Alem Tuzlak 1603b63e33 style: format v2 multimodal changes 2026-04-06 14:54:46 +02:00