mirror of
https://github.com/ComposioHQ/composio.git
synced 2026-09-22 11:46:35 +08:00
Rahul Tarak
ea2e02071a
## Summary
- Switch the docs Eve agent from the AI Gateway `openai/gpt-5.4-mini`
string to an Inception Labs Mercury 2 OpenAI-compatible chat model.
- Keep tool calling on the chat-completions path and pass Mercury's
`reasoning_effort=medium` through the AI SDK OpenAI adapter.
- Add `DOCS_AGENT_MODEL_FLOW` so the same agent can run either `mercury`
or the old AI Gateway flow for eval comparisons.
- Add docs-agent eve evals covering grounded docs answers, docs
retrieval, citations, and account-specific support refusal.
- Replace the docs-agent retriever with an in-process BM25-style lexical
ranker that returns bounded full content for the top results, so Mercury
gets rich context in one fast tool call instead of a serial
`search_docs` → `read_doc` round trip.
- Precompute BM25 term counts/document frequencies into the generated
`agent/lib/docs-index.ts` snapshot at build time, removing deployed
cold-start corpus construction while keeping retrieval in-process.
- Add opt-in search perf logging (`DOCS_AGENT_SEARCH_PERF_LOG=1`,
optional `DOCS_AGENT_SEARCH_LOG_QUERY=1`) with timings for tokenization,
corpus load/cache, ranking, hydration, total duration, corpus source,
and top URLs.
- Add `eval:agent` and `eval:agent:flows` scripts; `eval:agent:flows`
can run local model-flow comparisons or remote target comparisons via
`DOCS_AGENT_EVAL_TARGETS`.
- Add `INCEPTION_API_KEY` / optional Mercury and gateway model knobs to
`docs/.env.example`, and move `@ai-sdk/openai` to runtime dependencies
for the agent import.
## Notes
- This is intentionally an experiment to see how Mercury's diffusion
model behaves with Eve tool calling (`search_docs` and `read_doc`).
- Preview/runtime environments need `INCEPTION_API_KEY`;
`INCEPTION_MODEL` and `INCEPTION_BASE_URL` are optional overrides.
- The custom fetch prevents accidentally falling back to
`OPENAI_API_KEY` against Inception's endpoint.
- The docs search is lexical/in-memory, not vector search. The slow path
was mostly serial model/tool round trips and cold index construction,
not embedding lookup.
- The generated BM25 snapshot is process-local once loaded: warm for the
lifetime of the running Node/Vercel function instance, and reset on cold
starts, redeploys, or process restarts. The expensive term-count corpus
is now built at docs build time.
- Perf logs omit raw user queries by default; set
`DOCS_AGENT_SEARCH_LOG_QUERY=1` only when you explicitly want raw
query/term logging.
- Local A/B-style eval run:
```bash
DOCS_AGENT_EVAL_FLOWS=gateway,mercury bun run eval:agent:flows --
--strict
```
- Live target comparison:
```bash
DOCS_AGENT_EVAL_TARGETS=baseline=https://<prod>,mercury=https://<preview>
bun run eval:agent:flows -- --strict
```
## Tests
- `bunx eslint scripts/build-agent-index.ts agent/lib/docs.ts
agent/tools/search_docs.ts`
- `bun scripts/build-agent-index.ts` (wrote 133 pages + 1000 toolkits +
1139 BM25 rows)
- `DOCS_AGENT_SEARCH_PERF_LOG=1 EVE_FORCE_BUNDLE=1 bun -e "const
tool=(await import('./agent/tools/search_docs.ts?log=' +
Date.now())).default; await tool.execute({query:'create a session with
github tools'}); await tool.execute({query:'auth config connected
account'});"` (logs cold and warm timing JSON)
- `EVE_FORCE_BUNDLE=1 bun -e "const tool=(await
import('./agent/tools/search_docs.ts?bundle=' + Date.now())).default;
const started=performance.now(); const r=await
tool.execute({query:'create a session with github tools'});
console.log(r.retrieval, r.results[0].url, r.results[0].content.length,
Math.round(performance.now()-started)+'ms');"` (precomputed bundle path,
~12ms)
- `bun -e "const tool=(await import('./agent/tools/search_docs.ts?live='
+ Date.now())).default; const started=performance.now(); const r=await
tool.execute({query:'create a session with github tools'});
console.log(r.retrieval, r.results[0].url, r.results[0].content.length,
Math.round(performance.now()-started)+'ms');"` (live-content path,
~34ms)
- `EVE_FORCE_BUNDLE=1 bun -e "const tool=(await
import('./agent/tools/search_docs.ts')).default; await
tool.execute({query:'create a session with github tools'}); const
started=performance.now(); const r=await tool.execute({query:'auth
config connected account'}); console.log(r.results[0].url,
r.results[0].content.length,
Math.round(performance.now()-started)+'ms');"` (warm path ~2ms)
-
`PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH
./node_modules/.bin/eve info --json` (reports `status: ready`, `model:
inception/mercury-2`, `errors: 0`)
- `DOCS_AGENT_MODEL_FLOW=gateway
PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH
./node_modules/.bin/eve info --json` (reports `status: ready`, `model:
openai/gpt-5.4-mini`, `errors: 0`)
-
`PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH
./node_modules/.bin/eve eval --list`
-
`PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH
bun scripts/eval-agent-flows.ts --list`
- `bun test tests/static/` (16 passed)
- `bun run types:check` currently fails on existing docs type-generation
errors in `app/(home)/docs/changelog/[...slug]/page.tsx`,
`app/(home)/examples/[[...slug]]/page.tsx`,
`app/(home)/toolkits/[[...slug]]/page.tsx`,
`app/llms.mdx/[[...slug]]/route.ts`, `lib/search-index.ts`, and
`lib/source.ts`; no new eval or `docs/agent/agent.ts` errors were
reported.
## Not run
- Real live model evals, because this local environment does not have
`INCEPTION_API_KEY` or AI Gateway credentials.
## Latest update
- Added default eager docs retrieval in the Eve HTTP channel: the server
runs the same BM25 search on the user's message before the first model
step and injects the results as one-turn context.
- Kept `search_docs` and `read_doc` available so Mercury can still
search/read more when the eager context is weak, ambiguous, or missing.
- Added `DOCS_AGENT_EAGER_SEARCH=0` as an escape hatch and labeled perf
logs with `invocation: "eager_context" | "tool"`.
- Updated the loading copy from “Searching the docs…” to “Thinking with
the docs…” so UI latency is not attributed solely to the search call.
## Latest tests
- `bun run lint -- agent/channels/eve.ts agent/tools/search_docs.ts
agent/lib/docs-search.ts components/eve-chat.tsx
evals/docs-agent/grounded-answers.eval.ts`
-
`PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH
node_modules/eve/bin/eve.js info --json` (reports `status: ready`,
`errors: 0`)
- `DOCS_AGENT_SEARCH_PERF_LOG=1 EVE_FORCE_BUNDLE=1 bun -e "import {
searchDocs } from './agent/lib/docs-search'; const r = searchDocs('How
do I create a session in Composio? Keep it brief.', { invocation:
'eager_context' }); console.log(JSON.stringify({count:r.results.length,
top:r.results[0]?.url, content: !!r.results[0]?.content}, null, 2));"`
- `DOCS_AGENT_SEARCH_PERF_LOG=1 bun -e "import { searchDocs } from
'./agent/lib/docs-search'; searchDocs('How do I create a session in
Composio? Keep it brief.', { invocation: 'eager_context' });
searchDocs('How do I create a session in Composio? Keep it brief.', {
invocation: 'tool' });"`
- `bun run types:check` still fails only on the pre-existing docs
type-generation issues listed above.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OpenAPI Scripts
fetch-with-retry.ts
Shared fetchWithRetry helper used by the data-generation scripts
(generate-toolkits.ts, generate-meta-tools.ts). It wraps the global fetch
with rate-limit-aware retry/backoff: it retries 429 and transient 5xx
responses, honors the Retry-After header when present, otherwise falls back to
exponential backoff with jitter, and caps attempts so CI still fails fast when
the backend is genuinely down.
This matters because generate-toolkits.ts issues ~3000 requests per run
(2 + 3 per toolkit), which exceeds the staging limit of 2000 requests/minute.
Before this helper, runs failed with 429, and generate-meta-tools.ts — which
runs immediately after — inherited the exhausted rate-limit window.
fetch-openapi.mjs
Fetches the Composio OpenAPI spec and filters it for use in Fumadocs API reference documentation.
Usage
bun run scripts/fetch-openapi.mjs
This outputs public/openapi.json which is used by lib/openapi.ts.
Why Filtering is Needed
The raw OpenAPI spec from https://backend.composio.dev/api/v3/openapi.json has issues that break documentation generators:
- Endpoints with multiple tags - Causes duplicate entries in sidebar
- Internal endpoints exposed - CLI, Admin, Profiling endpoints shouldn't be in public docs
See OPENAPI_IMPROVEMENTS.md in the fumadocs root for planned fixes to the spec itself.
What Gets Filtered
Ignored Paths
These endpoints are completely removed:
/api/v3/mcp/validate/{uuid}/api/v3/cli/get-session/api/v3/cli/create-session/api/v3/auth/session/logout
Ignored Tags
Endpoints with only these tags are removed:
CLIAdminProfiling
Duplicate Prevention
If an endpoint has multiple tags, only the first tag is kept. This prevents the same endpoint appearing in multiple sidebar sections.
Configuration
The script uses environment variables:
| Variable | Default | Description |
|---|---|---|
OPENAPI_SPEC_URL |
https://backend.composio.dev/api/v3/openapi.json |
Source OpenAPI spec URL |
For staging deployments, set:
OPENAPI_SPEC_URL=https://staging.composio.dev/api/v3/openapi.json