3 Commits

Author	SHA1	Message	Date
Soham Basu	42ce7609a2	feat(docs): launch unified support knowledge MVP	2026-08-21 15:03:54 -07:00
Alberto Schiabel	de0bfa356d	fix(docs): resolve changelog links and Vercel deploys (#3811) ... This PR: - fixes copied changelog date URLs so they stay on `/reference/changelog` and target the selected date - resolves fragment links against the current page while preserving full site-relative links - upgrades the docs app to Vercel-supported `eve@0.18.0` - migrates docs-agent evals from the removed `t.completed()` assertion to `t.succeeded()` - verifies the docs static suite, TypeScript type-check, and production build --- [](https://github.com/EveryInc/compound-engineering-plugin) 	2026-07-13 13:53:54 +04:00
Rahul Tarak	ea2e02071a	docs: fast-search with mercury-2 (#3687) ... ## Summary - Switch the docs Eve agent from the AI Gateway `openai/gpt-5.4-mini` string to an Inception Labs Mercury 2 OpenAI-compatible chat model. - Keep tool calling on the chat-completions path and pass Mercury's `reasoning_effort=medium` through the AI SDK OpenAI adapter. - Add `DOCS_AGENT_MODEL_FLOW` so the same agent can run either `mercury` or the old AI Gateway flow for eval comparisons. - Add docs-agent eve evals covering grounded docs answers, docs retrieval, citations, and account-specific support refusal. - Replace the docs-agent retriever with an in-process BM25-style lexical ranker that returns bounded full content for the top results, so Mercury gets rich context in one fast tool call instead of a serial `search_docs` → `read_doc` round trip. - Precompute BM25 term counts/document frequencies into the generated `agent/lib/docs-index.ts` snapshot at build time, removing deployed cold-start corpus construction while keeping retrieval in-process. - Add opt-in search perf logging (`DOCS_AGENT_SEARCH_PERF_LOG=1`, optional `DOCS_AGENT_SEARCH_LOG_QUERY=1`) with timings for tokenization, corpus load/cache, ranking, hydration, total duration, corpus source, and top URLs. - Add `eval:agent` and `eval:agent:flows` scripts; `eval:agent:flows` can run local model-flow comparisons or remote target comparisons via `DOCS_AGENT_EVAL_TARGETS`. - Add `INCEPTION_API_KEY` / optional Mercury and gateway model knobs to `docs/.env.example`, and move `@ai-sdk/openai` to runtime dependencies for the agent import. ## Notes - This is intentionally an experiment to see how Mercury's diffusion model behaves with Eve tool calling (`search_docs` and `read_doc`). - Preview/runtime environments need `INCEPTION_API_KEY`; `INCEPTION_MODEL` and `INCEPTION_BASE_URL` are optional overrides. - The custom fetch prevents accidentally falling back to `OPENAI_API_KEY` against Inception's endpoint. - The docs search is lexical/in-memory, not vector search. The slow path was mostly serial model/tool round trips and cold index construction, not embedding lookup. - The generated BM25 snapshot is process-local once loaded: warm for the lifetime of the running Node/Vercel function instance, and reset on cold starts, redeploys, or process restarts. The expensive term-count corpus is now built at docs build time. - Perf logs omit raw user queries by default; set `DOCS_AGENT_SEARCH_LOG_QUERY=1` only when you explicitly want raw query/term logging. - Local A/B-style eval run: ```bash DOCS_AGENT_EVAL_FLOWS=gateway,mercury bun run eval:agent:flows -- --strict ``` - Live target comparison: ```bash DOCS_AGENT_EVAL_TARGETS=baseline=https://<prod>,mercury=https://<preview> bun run eval:agent:flows -- --strict ``` ## Tests - `bunx eslint scripts/build-agent-index.ts agent/lib/docs.ts agent/tools/search_docs.ts` - `bun scripts/build-agent-index.ts` (wrote 133 pages + 1000 toolkits + 1139 BM25 rows) - `DOCS_AGENT_SEARCH_PERF_LOG=1 EVE_FORCE_BUNDLE=1 bun -e "const tool=(await import('./agent/tools/search_docs.ts?log=' + Date.now())).default; await tool.execute({query:'create a session with github tools'}); await tool.execute({query:'auth config connected account'});"` (logs cold and warm timing JSON) - `EVE_FORCE_BUNDLE=1 bun -e "const tool=(await import('./agent/tools/search_docs.ts?bundle=' + Date.now())).default; const started=performance.now(); const r=await tool.execute({query:'create a session with github tools'}); console.log(r.retrieval, r.results[0].url, r.results[0].content.length, Math.round(performance.now()-started)+'ms');"` (precomputed bundle path, ~12ms) - `bun -e "const tool=(await import('./agent/tools/search_docs.ts?live=' + Date.now())).default; const started=performance.now(); const r=await tool.execute({query:'create a session with github tools'}); console.log(r.retrieval, r.results[0].url, r.results[0].content.length, Math.round(performance.now()-started)+'ms');"` (live-content path, ~34ms) - `EVE_FORCE_BUNDLE=1 bun -e "const tool=(await import('./agent/tools/search_docs.ts')).default; await tool.execute({query:'create a session with github tools'}); const started=performance.now(); const r=await tool.execute({query:'auth config connected account'}); console.log(r.results[0].url, r.results[0].content.length, Math.round(performance.now()-started)+'ms');"` (warm path ~2ms) - `PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH ./node_modules/.bin/eve info --json` (reports `status: ready`, `model: inception/mercury-2`, `errors: 0`) - `DOCS_AGENT_MODEL_FLOW=gateway PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH ./node_modules/.bin/eve info --json` (reports `status: ready`, `model: openai/gpt-5.4-mini`, `errors: 0`) - `PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH ./node_modules/.bin/eve eval --list` - `PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH bun scripts/eval-agent-flows.ts --list` - `bun test tests/static/` (16 passed) - `bun run types:check` currently fails on existing docs type-generation errors in `app/(home)/docs/changelog/[...slug]/page.tsx`, `app/(home)/examples/[[...slug]]/page.tsx`, `app/(home)/toolkits/[[...slug]]/page.tsx`, `app/llms.mdx/[[...slug]]/route.ts`, `lib/search-index.ts`, and `lib/source.ts`; no new eval or `docs/agent/agent.ts` errors were reported. ## Not run - Real live model evals, because this local environment does not have `INCEPTION_API_KEY` or AI Gateway credentials. ## Latest update - Added default eager docs retrieval in the Eve HTTP channel: the server runs the same BM25 search on the user's message before the first model step and injects the results as one-turn context. - Kept `search_docs` and `read_doc` available so Mercury can still search/read more when the eager context is weak, ambiguous, or missing. - Added `DOCS_AGENT_EAGER_SEARCH=0` as an escape hatch and labeled perf logs with `invocation: "eager_context" | "tool"`. - Updated the loading copy from “Searching the docs…” to “Thinking with the docs…” so UI latency is not attributed solely to the search call. ## Latest tests - `bun run lint -- agent/channels/eve.ts agent/tools/search_docs.ts agent/lib/docs-search.ts components/eve-chat.tsx evals/docs-agent/grounded-answers.eval.ts` - `PATH=/Users/cryogenicplanet/.vite-plus/js_runtime/node/24.15.0/bin:$PATH node_modules/eve/bin/eve.js info --json` (reports `status: ready`, `errors: 0`) - `DOCS_AGENT_SEARCH_PERF_LOG=1 EVE_FORCE_BUNDLE=1 bun -e "import { searchDocs } from './agent/lib/docs-search'; const r = searchDocs('How do I create a session in Composio? Keep it brief.', { invocation: 'eager_context' }); console.log(JSON.stringify({count:r.results.length, top:r.results[0]?.url, content: !!r.results[0]?.content}, null, 2));"` - `DOCS_AGENT_SEARCH_PERF_LOG=1 bun -e "import { searchDocs } from './agent/lib/docs-search'; searchDocs('How do I create a session in Composio? Keep it brief.', { invocation: 'eager_context' }); searchDocs('How do I create a session in Composio? Keep it brief.', { invocation: 'tool' });"` - `bun run types:check` still fails only on the pre-existing docs type-generation issues listed above. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-26 03:17:57 -07:00