Alberto Schiabel
82896858ec
This PR: - adds a shared `fetchWithRetry` helper for the docs data-sync scripts that retries `429` and transient 5xx responses, honoring `Retry-After` (with jitter) and falling back to exponential backoff - routes every request in `generate-toolkits.ts` and `generate-meta-tools.ts` through it, so the `Docs - Update Data` workflow rides out the staging rate limit (2000 req/min) instead of hard-failing - lowers toolkit fetch `batchSize` 10 → 5 (~30 → ~15 concurrent requests) to soften burst pressure on the rate limit - caps retry attempts so CI still fails fast when the backend is genuinely down - fixes the current `429 Too Many Requests` failure in `Generate meta tools reference`, which inherited the rate-limit window exhausted by the high-volume `generate-toolkits.ts` run - verified with `bun run types:check` and an offline retry/backoff behavior test (real end-to-end run needs the staging key in CI: `gh workflow run docs-update-data.yml`) - follow-up (separate PR): reduce per-toolkit request volume in `generate-toolkits.ts` — left out here because it needs consumer and search-quality validation
OpenAPI Scripts
fetch-with-retry.ts
Shared fetchWithRetry helper used by the data-generation scripts
(generate-toolkits.ts, generate-meta-tools.ts). It wraps the global fetch
with rate-limit-aware retry/backoff: it retries 429 and transient 5xx
responses, honors the Retry-After header when present, otherwise falls back to
exponential backoff with jitter, and caps attempts so CI still fails fast when
the backend is genuinely down.
This matters because generate-toolkits.ts issues ~3000 requests per run
(2 + 3 per toolkit), which exceeds the staging limit of 2000 requests/minute.
Before this helper, runs failed with 429, and generate-meta-tools.ts — which
runs immediately after — inherited the exhausted rate-limit window.
fetch-openapi.mjs
Fetches the Composio OpenAPI spec and filters it for use in Fumadocs API reference documentation.
Usage
bun run scripts/fetch-openapi.mjs
This outputs public/openapi.json which is used by lib/openapi.ts.
Why Filtering is Needed
The raw OpenAPI spec from https://backend.composio.dev/api/v3/openapi.json has issues that break documentation generators:
- Endpoints with multiple tags - Causes duplicate entries in sidebar
- Internal endpoints exposed - CLI, Admin, Profiling endpoints shouldn't be in public docs
See OPENAPI_IMPROVEMENTS.md in the fumadocs root for planned fixes to the spec itself.
What Gets Filtered
Ignored Paths
These endpoints are completely removed:
/api/v3/mcp/validate/{uuid}/api/v3/cli/get-session/api/v3/cli/create-session/api/v3/auth/session/logout
Ignored Tags
Endpoints with only these tags are removed:
CLIAdminProfiling
Duplicate Prevention
If an endpoint has multiple tags, only the first tag is kept. This prevents the same endpoint appearing in multiple sidebar sections.
Configuration
The script uses environment variables:
| Variable | Default | Description |
|---|---|---|
OPENAPI_SPEC_URL |
https://backend.composio.dev/api/v3/openapi.json |
Source OpenAPI spec URL |
For staging deployments, set:
OPENAPI_SPEC_URL=https://staging.composio.dev/api/v3/openapi.json