mirror of
https://github.com/ComposioHQ/composio.git
synced 2026-09-22 11:46:35 +08:00
Rahul Tarak
ea2e02071a
## Summary
- Switch the docs Eve agent from the AI Gateway `openai/gpt-5.4-mini`
string to an Inception Labs Mercury 2 OpenAI-compatible chat model.
- Keep tool calling on the chat-completions path and pass Mercury's
`reasoning_effort=medium` through the AI SDK OpenAI adapter.
- Add `DOCS_AGENT_MODEL_FLOW` so the same agent can run either `mercury`
or the old AI Gateway flow for eval comparisons.
- Add docs-agent eve evals covering grounded docs answers, docs
retrieval, citations, and account-specific support refusal.
- Replace the docs-agent retriever with an in-process BM25-style lexical
ranker that returns bounded full content for the top results, so Mercury
gets rich context in one fast tool call instead of a serial
`search_docs` → `read_doc` round trip.
- Precompute BM25 term counts/document frequencies into the generated
`agent/lib/docs-index.ts` snapshot at build time, removing deployed
cold-start corpus construction while keeping retrieval in-process.
- Add opt-in search perf logging (`DOCS_AGENT_SEARCH_PERF_LOG=1`,
optional `DOCS_AGENT_SEARCH_LOG_QUERY=1`) with timings for tokenization,
corpus load/cache, ranking, hydration, total duration, corpus source,
and top URLs.
- Add `eval:agent` and `eval:agent:flows` scripts; `eval:agent:flows`
can run local model-flow comparisons or remote target comparisons via
`DOCS_AGENT_EVAL_TARGETS`.
- Add `INCEPTION_API_KEY` / optional Mercury and gateway model knobs to
`docs/.env.example`, and move `@ai-sdk/openai` to runtime dependencies
for the agent import.
## Notes
- This is intentionally an experiment to see how Mercury's diffusion
model behaves with Eve tool calling (`search_docs` and `read_doc`).
- Preview/runtime environments need `INCEPTION_API_KEY`;
`INCEPTION_MODEL` and `INCEPTION_BASE_URL` are optional overrides.
- The custom fetch prevents accidentally falling back to
`OPENAI_API_KEY` against Inception's endpoint.
- The docs search is lexical/in-memory, not vector search. The slow path
was mostly serial model/tool round trips and cold index construction,
not embedding lookup.
- The generated BM25 snapshot is process-local once loaded: warm for the
lifetime of the running Node/Vercel function instance, and reset on cold
starts, redeploys, or process restarts. The expensive term-count corpus
is now built at docs build time.
- Perf logs omit raw user queries by default; set
`DOCS_AGENT_SEARCH_LOG_QUERY=1` only when you explicitly want raw
query/term logging.
- Local A/B-style eval run:
```bash
DOCS_AGENT_EVAL_FLOWS=gateway,mercury bun run eval:agent:flows --
--strict
```
- Live target comparison:
```bash
DOCS_AGENT_EVAL_TARGETS=baseline=https://<prod>,mercury=https://<preview>
bun run eval:agent:flows -- --strict
```
## Tests
- `bunx eslint scripts/build-agent-index.ts agent/lib/docs.ts
agent/tools/search_docs.ts`
- `bun scripts/build-agent-index.ts` (wrote 133 pages + 1000 toolkits +
1139 BM25 rows)
- `DOCS_AGENT_SEARCH_PERF_LOG=1 EVE_FORCE_BUNDLE=1 bun -e "const
tool=(await import('./agent/tools/search_docs.ts?log=' +
Date.now())).default; await tool.execute({query:'create a session with
github tools'}); await tool.execute({query:'auth config connected
account'});"` (logs cold and warm timing JSON)
- `EVE_FORCE_BUNDLE=1 bun -e "const tool=(await
import('./agent/tools/search_docs.ts?bundle=' + Date.now())).default;
const started=performance.now(); const r=await
tool.execute({query:'create a session with github tools'});
console.log(r.retrieval, r.results[0].url, r.results[0].content.length,
Math.round(performance.now()-started)+'ms');"` (precomputed bundle path,
~12ms)
- `bun -e "const tool=(await import('./agent/tools/search_docs.ts?live='
+ Date.now())).default; const started=performance.now(); const r=await
tool.execute({query:'create a session with github tools'});
console.log(r.retrieval, r.results[0].url, r.results[0].content.length,
Math.round(performance.now()-started)+'ms');"` (live-content path,
~34ms)
- `EVE_FORCE_BUNDLE=1 bun -e "const tool=(await
import('./agent/tools/search_docs.ts')).default; await
tool.execute({query:'create a session with github tools'}); const
started=performance.now(); const r=await tool.execute({query:'auth
config connected account'}); console.log(r.results[0].url,
r.results[0].content.length,
Math.round(performance.now()-started)+'ms');"` (warm path ~2ms)
-
`PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH
./node_modules/.bin/eve info --json` (reports `status: ready`, `model:
inception/mercury-2`, `errors: 0`)
- `DOCS_AGENT_MODEL_FLOW=gateway
PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH
./node_modules/.bin/eve info --json` (reports `status: ready`, `model:
openai/gpt-5.4-mini`, `errors: 0`)
-
`PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH
./node_modules/.bin/eve eval --list`
-
`PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH
bun scripts/eval-agent-flows.ts --list`
- `bun test tests/static/` (16 passed)
- `bun run types:check` currently fails on existing docs type-generation
errors in `app/(home)/docs/changelog/[...slug]/page.tsx`,
`app/(home)/examples/[[...slug]]/page.tsx`,
`app/(home)/toolkits/[[...slug]]/page.tsx`,
`app/llms.mdx/[[...slug]]/route.ts`, `lib/search-index.ts`, and
`lib/source.ts`; no new eval or `docs/agent/agent.ts` errors were
reported.
## Not run
- Real live model evals, because this local environment does not have
`INCEPTION_API_KEY` or AI Gateway credentials.
## Latest update
- Added default eager docs retrieval in the Eve HTTP channel: the server
runs the same BM25 search on the user's message before the first model
step and injects the results as one-turn context.
- Kept `search_docs` and `read_doc` available so Mercury can still
search/read more when the eager context is weak, ambiguous, or missing.
- Added `DOCS_AGENT_EAGER_SEARCH=0` as an escape hatch and labeled perf
logs with `invocation: "eager_context" | "tool"`.
- Updated the loading copy from “Searching the docs…” to “Thinking with
the docs…” so UI latency is not attributed solely to the search call.
## Latest tests
- `bun run lint -- agent/channels/eve.ts agent/tools/search_docs.ts
agent/lib/docs-search.ts components/eve-chat.tsx
evals/docs-agent/grounded-answers.eval.ts`
-
`PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH
node_modules/eve/bin/eve.js info --json` (reports `status: ready`,
`errors: 0`)
- `DOCS_AGENT_SEARCH_PERF_LOG=1 EVE_FORCE_BUNDLE=1 bun -e "import {
searchDocs } from './agent/lib/docs-search'; const r = searchDocs('How
do I create a session in Composio? Keep it brief.', { invocation:
'eager_context' }); console.log(JSON.stringify({count:r.results.length,
top:r.results[0]?.url, content: !!r.results[0]?.content}, null, 2));"`
- `DOCS_AGENT_SEARCH_PERF_LOG=1 bun -e "import { searchDocs } from
'./agent/lib/docs-search'; searchDocs('How do I create a session in
Composio? Keep it brief.', { invocation: 'eager_context' });
searchDocs('How do I create a session in Composio? Keep it brief.', {
invocation: 'tool' });"`
- `bun run types:check` still fails only on the pre-existing docs
type-generation issues listed above.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
112 lines
3.5 KiB
JSON
112 lines
3.5 KiB
JSON
{
|
|
"name": "@composio/docs",
|
|
"version": "0.0.0",
|
|
"private": true,
|
|
"engines": {
|
|
"node": ">=24"
|
|
},
|
|
"scripts": {
|
|
"build:agent-index": "bun scripts/build-agent-index.ts",
|
|
"prebuild": "bun scripts/build-agent-index.ts",
|
|
"build": "next build",
|
|
"predev": "bun scripts/build-agent-index.ts",
|
|
"dev": "next dev",
|
|
"start": "next start",
|
|
"types:check": "bun scripts/build-agent-index.ts && fumadocs-mdx && next typegen && tsc --noEmit",
|
|
"postinstall": "fumadocs-mdx",
|
|
"lint": "eslint",
|
|
"lint:links": "bun scripts/validate-links.ts",
|
|
"sync:search": "bun scripts/sync-algolia-search.ts",
|
|
"test:search": "bun scripts/test-algolia-search.ts",
|
|
"eval:agent": "eve eval docs-agent --skip-report",
|
|
"eval:agent:flows": "bun scripts/eval-agent-flows.ts",
|
|
"generate:toolkits": "bun scripts/generate-toolkits.ts",
|
|
"generate:meta-tools": "bun scripts/generate-meta-tools.ts",
|
|
"generate:api-index": "bun scripts/generate-api-index.ts",
|
|
"test": "bun test tests/static/",
|
|
"test:integration": "bun test tests/integration/ --timeout 30000"
|
|
},
|
|
"dependencies": {
|
|
"@ai-sdk/openai": "^3.0.9",
|
|
"@pierre/diffs": "^1.2.11",
|
|
"@pierre/trees": "^1.0.0-beta.4",
|
|
"@radix-ui/react-tabs": "^1.1.13",
|
|
"@shikijs/langs": "^4.2.0",
|
|
"@shikijs/themes": "^4.2.0",
|
|
"@shikijs/twoslash": "^3.21.0",
|
|
"@vercel/analytics": "^1.6.1",
|
|
"algoliasearch": "^5",
|
|
"class-variance-authority": "^0.7.1",
|
|
"clsx": "^2.1.1",
|
|
"diff": "^9.0.0",
|
|
"eve": "^0.13.3",
|
|
"framer-motion": "^12.39.0",
|
|
"fumadocs-core": "^16.4.2",
|
|
"fumadocs-mdx": "^14.2.4",
|
|
"fumadocs-openapi": "^10.2.2",
|
|
"fumadocs-ui": "^16.4.2",
|
|
"highlight.js": "^11.11.1",
|
|
"lucide-react": "^0.562.0",
|
|
"mermaid": "^11.12.3",
|
|
"motion": "^12.34.3",
|
|
"next": "16.2.6",
|
|
"nucleo-pixel": "^1.3.0",
|
|
"posthog-js": "^1.319.0",
|
|
"react": "^19.2.3",
|
|
"react-dom": "^19.2.3",
|
|
"react-markdown": "^10.1.0",
|
|
"remark-gfm": "^4.0.1",
|
|
"search-insights": "^2",
|
|
"shiki": "^3.20.0",
|
|
"tailwind-merge": "^3.4.0",
|
|
"zod": "^4.4.3"
|
|
},
|
|
"devDependencies": {
|
|
"@ai-sdk/anthropic": "^3.0.11",
|
|
"@ai-sdk/mcp": "^1.0.6",
|
|
"@ai-sdk/react": "^3.0.102",
|
|
"@anthropic-ai/claude-agent-sdk": "^0.2.5",
|
|
"@anthropic-ai/sdk": "^0.71.2",
|
|
"@composio/anthropic": "^0.5.5",
|
|
"@composio/claude-agent-sdk": "^0.6.3",
|
|
"@composio/core": "^0.9.1",
|
|
"@composio/google": "^0.5.5",
|
|
"@composio/langchain": "^0.5.5",
|
|
"@composio/llamaindex": "^0.5.5",
|
|
"@composio/mastra": "^0.5.5",
|
|
"@composio/openai": "^0.5.5",
|
|
"@composio/openai-agents": "^0.6.3",
|
|
"@composio/vercel": "^0.6.3",
|
|
"@google/genai": "^1.35.0",
|
|
"@langchain/core": "^1.1.12",
|
|
"@langchain/langgraph": "^1.0.15",
|
|
"@langchain/openai": "^1.2.1",
|
|
"@llamaindex/openai": "^0.4.22",
|
|
"@llamaindex/workflow": "^1.1.24",
|
|
"@mastra/core": "1.0.0-beta.21",
|
|
"@openai/agents": "^0.4.13",
|
|
"@shikijs/vitepress-twoslash": "^3.21.0",
|
|
"@tailwindcss/postcss": "^4.1.18",
|
|
"@types/mdx": "^2.0.13",
|
|
"@types/node": "^24.10.2",
|
|
"@types/react": "^19.2.7",
|
|
"@types/react-dom": "^19.2.3",
|
|
"ai": "7.0.0-beta.178",
|
|
"ajv": "^8.17.1",
|
|
"ajv-draft-04": "^1.0.0",
|
|
"eslint": "^9.39.2",
|
|
"eslint-config-next": "16.2.9",
|
|
"github-slugger": "^2.0.0",
|
|
"microsandbox": "^0.5.7",
|
|
"next-validate-link": "^1.6.4",
|
|
"openai": "^6.16.0",
|
|
"postcss": "^8.5.6",
|
|
"tailwindcss": "^4.1.18",
|
|
"tw-animate-css": "^1.4.0",
|
|
"typescript": "^6.0.3"
|
|
},
|
|
"overrides": {
|
|
"typescript-eslint": "^8.61.1"
|
|
}
|
|
}
|