Alberto Schiabel
3335bb3013
This PR: - fixes a regression introduced by https://github.com/ComposioHQ/composio/pull/3956: fumadocs-openapi 11's `groupBy: 'tag'` silently skips operations whose tag is not declared in the document's top-level `tags` array, which 404'd all 16 Projects and Organization Management operation pages (e.g. `/reference/api-reference/projects/postProjectUsageSummary`) on both v3.1 and v3, and dropped them from the sitemap and search index - adds `declareOperationTags()` (`docs/lib/openapi-tags.ts`), applied at spec load time in `lib/openapi.ts` (covers build-time-fetched specs) and at sync time in `scripts/fetch-openapi.mjs`, which now also warns when the upstream payload omits tags; the checked-in specs are normalized accordingly (purely additive) - adds CI guards in `tests/static/api-reference-routes.test.ts`: spec invariants (every visible operation has a tag and `operationId`), bidirectional spec-vs-generated-routes set equality through the production loader path, a negative test pinning fumadocs' silent-drop behavior (fails when upstream fixes it, signaling the workaround can be retired), and validation that every `ApiEndpointsTable` quick-link href resolves to a generated page - extends `Docs - Check Links` with a nightly (02:30 UTC) + manual external-URL sweep via a new `lint:links:external` script; the validator only fails on evidence a link is dead (404/410 or repeated network errors, with UA/timeout/GET-fallback/retry and per-URL caching) and scheduled failures file a deduplicated tracking issue; PR runs keep the fast internal-only check - fixes six dead external links the first sweep found (moved Claude Agent SDK docs, stale dashboard settings URL ×2, a `master`-branch path now pinned to tag `0.5.0+post.1`, and two removed `ComposioHQ` example repos whose mentions were dropped — their code is embedded in the pages via `RepoBrowser`) ## Context fumadocs-openapi 10 generated a page for any tag string found on an operation; v11 requires the tag to be declared top-level and skips silently otherwise (`preset-auto.js`: `builder.fromTagName(tag)` → `continue`, no warning). The backend spec generator omits `Projects`, `Organization Management`, and `Invite Codes` from `tags`, so their operation pages vanished while the checked-in MDX tag landing pages kept rendering with dead quick links. With the normalizer, the generated URL set is byte-identical to the pre-#3956 set (234 routes, verified via `getReferenceSource().getPages()` diff); no revert needed. Existing checks missed this because `lint:links` only extracts markdown links and `Card` hrefs (the dead links live in `ApiEndpointsTable`'s array prop) and validates against the same loader that shrank, and the integration suite samples fixed routes under tags that survived. The new completeness guard derives expectations from the specs themselves, so routine data syncs don't churn it.
OpenAPI Scripts
fetch-with-retry.ts
Shared fetchWithRetry helper used by the data-generation scripts
(generate-toolkits.ts, generate-meta-tools.ts). It wraps the global fetch
with rate-limit-aware retry/backoff: it retries 429 and transient 5xx
responses, honors the Retry-After header when present, otherwise falls back to
exponential backoff with jitter, and caps attempts so CI still fails fast when
the backend is genuinely down.
This matters because generate-toolkits.ts issues ~6500 requests per run
(a few catalog pages + 3 per toolkit across a ~2.1k catalog), which can exceed
the backend request limit.
Before this helper, runs failed with 429, and generate-meta-tools.ts — which
runs immediately after — inherited the exhausted rate-limit window.
Toolkit versions
generate-toolkits.ts owns the complete toolkit catalog and always sources it
from the production API. For a version-only repair, run the narrower generator:
COMPOSIO_API_KEY=... bun run generate:toolkit-versions
Both paths share toolkit-versions.ts, so they fetch and apply changelog values
with identical semantics: a toolkit missing from the production changelog gets
version: null. Any COMPOSIO_API_BASE override must normalize to
https://backend.composio.dev/api/v3; non-production sources fail before a
request is made.
fetch-openapi.mjs
Fetches the Composio OpenAPI spec and filters it for use in Fumadocs API reference documentation.
Usage
bun run scripts/fetch-openapi.mjs
This outputs public/openapi.json which is used by lib/openapi.ts.
Why Filtering is Needed
The raw OpenAPI spec from https://backend.composio.dev/api/v3/openapi.json has issues that break documentation generators:
- Endpoints with multiple tags - Causes duplicate entries in sidebar
- Internal endpoints exposed - CLI, Admin, Profiling endpoints shouldn't be in public docs
See OPENAPI_IMPROVEMENTS.md in the fumadocs root for planned fixes to the spec itself.
What Gets Filtered
Ignored Paths
These endpoints are completely removed:
/api/v3/mcp/validate/{uuid}/api/v3/cli/get-session/api/v3/cli/create-session/api/v3/auth/session/logout
Ignored Tags
Endpoints with only these tags are removed:
CLIAdminProfiling
Duplicate Prevention
If an endpoint has multiple tags, only the first tag is kept. This prevents the same endpoint appearing in multiple sidebar sections.
Configuration
The script uses environment variables:
| Variable | Default | Description |
|---|---|---|
OPENAPI_SPEC_URL |
https://backend.composio.dev/api/v3/openapi.json |
Source OpenAPI spec URL |
For staging deployments, set:
OPENAPI_SPEC_URL=https://staging.composio.dev/api/v3/openapi.json