Commit Graph

3 Commits

Author SHA1 Message Date
Jeremias Menichelli 575ac8c645 [DOCS-979] feat(Docs): Generate new markdown files for reference pages (#46727)
In this PR:
- The `llms.ts` script is removed from the pipeline. It was just
iterating over the sections of the SDK JSON files to generate _dummy_
text files for LLMS to consume. But with a lot of inconsistencies and
mistakes.
- There's a new script that builds markdown files from the SDK
definition files.
- The Management API now has a definition file too, it was missing
before.
- On the www project now there are redirects for the reference text
files to the new markdown files in the docs project.
- NPM scripts have received better names so that prebuild ones are
smaller and all are easier to differentiate.
2026-06-09 10:39:55 +02:00
Pamela Chia d409836ca7 feat(www,docs): serve marketing pages as .md, advertise via link rel=alternate (#45277)
## Summary

Adds `/<page>.md` routes for 10 marketing/product pages (homepage, auth,
database, edge-functions, realtime, storage, vector, pricing,
modules/cron, modules/queues) so AI agents can fetch clean markdown
instead of parsing JS-rendered HTML. Also advertises the markdown
alternate via `<link rel="alternate" type="text/markdown">` on marketing
and docs pages so agents can discover it.

Pricing is generated dynamically via `generatePricingContent()` (single
source of truth with `/llms.txt` and `/llms-full.txt`); the other nine
slugs are bundled at build time from `content/md/*.md` into a
`MD_CONTENT` map.

Supersedes #44891 (rebased fresh off current master to avoid a 9-commit
replay over rename/rename conflicts created by #44897).

## Changes

- New `/api-v2/md/[...slug]` route handler returns the bundled markdown
(or dynamic pricing) with `Content-Type: text/markdown`,
`X-Content-Type-Options: nosniff`, and appropriate cache headers
- Middleware rewrites `/<slug>.md` and `Accept: text/markdown` to the
API route for the `MD_PAGES` allowlist; trailing-slash variants
(`/auth/`) are normalized so they resolve the same as `/auth`
- Build-time codegen `scripts/generateMdContent.mjs` scans `content/md/`
and emits `app/api-v2/md/content.generated.ts` exporting both
`MD_CONTENT` (Map) and `MD_PAGES` (Set, incl. dynamic `pricing`). Fails
the build on slug collision between `content/md/` and `DYNAMIC_SLUGS`.
Adding a new marketing `.md` is just dropping a file in `content/md/`
(also update `PRODUCT_OVERVIEW_LINKS` in `/llms.txt` since that list is
editorial).
- 8 permanent redirects `/llms/<product>.txt` → `/<product>.md` so
legacy URLs in caches and downstream `llms.txt` copies keep working
- `/llms.txt` product overview now references `.md` URLs (incl.
`modules/cron`, `modules/queues`); `/llms-full.txt` iterates
`MD_CONTENT.values()` (homepage first, then alphabetical) and appends
dynamic pricing
- `/llms/[slug]` route slimmed to proxy SDK reference files (`js.txt`,
`dart.txt`, etc.) since redirects handle product slugs and pricing;
pricing branch retained as fallback in case redirects are bypassed
- `apps/www/pages/_app.tsx` injects the alternate link conditionally
based on `MD_PAGES`; `/pricing` (app router) sets it via page metadata
- `apps/docs/app/page.tsx` (the `/docs` root) sets the text/markdown
alternate to `/llms-full.txt`; per-guide pages override with their
specific `.md` URL via `genGuideMeta` in `GuidesMdx.utils.tsx`. Other
docs pages (reference, troubleshooting) inherit nothing.
- `apps/www/.vercelignore`: replaces the prior `*.md`/`README.md` rules
with `*.md` + `!content/md/**/*.md` so Edge Function READMEs and future
scratch `.md` files aren't silently shipped to the build artifact
- Drops `apps/www/data/llms/*.txt` and the related
`outputFileTracingIncludes`
- Test coverage for the new middleware branches: `.md` suffix rewrite
(allowlisted vs. fall-through), `Accept: text/markdown` content
negotiation, trailing-slash normalization

## Testing (Vercel preview)

Local dev server smoke tests passing on `:3771` after each iteration.
Re-verified on the preview URL after the latest hardening commit:

- [x] `curl -I https://<preview>/llms/auth.txt` — expect `308 Permanent
Redirect` to `/auth.md`
- [x] `curl https://<preview>/auth.md | head -3` — expect `# Supabase
Auth`
- [x] `curl https://<preview>/pricing.md | head -3` — expect `# Supabase
Pricing` with current tier values
- [x] `curl https://<preview>/modules/cron.md | head -3` — expect `#
Supabase Cron`
- [x] `curl -H 'Accept: text/markdown' https://<preview>/ | head -3` —
expect `# Supabase` (homepage.md)
- [x] `curl https://<preview>/llms.txt` — Product Overview section lists
`.md` URLs and includes Cron + Queues
- [x] `curl https://<preview>/llms-full.txt | grep -E '^# Supabase
(Cron\|Queues\|Pricing)'` — Cron and Pricing each match once; Queues
matches twice (marketing module + existing docs guide)
- [x] View source on `/`, `/pricing`, `/database` — expect `<link
rel="alternate" type="text/markdown" href="/<slug>.md">`
- [x] View source on `/docs` — expect `<link rel="alternate"
type="text/markdown" href="/llms-full.txt">`
- [x] View source on a docs guide page (e.g., `/docs/guides/auth`) —
expect per-guide `.md` alternate; reference/troubleshooting pages should
NOT emit a markdown alternate
- [x] `curl -I https://<preview>/auth.md` — expect
`X-Content-Type-Options: nosniff`
- [x] `curl -I -L -H 'Accept: text/markdown' https://<preview>/auth/` —
should resolve to markdown content (trailing-slash normalization, with
Vercel's auto-redirect)

## Linear

- fixes GROWTH-760

## Follow-up (separate PR)

GROWTH-760 also asks about extending `.md` to blog/customers/events.
Different mechanism (path-prefix middleware, MDX read at request time
via `gray-matter`) so it deserves its own review. Will open a follow-up
PR after this lands.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Serve prebuilt and dynamic Markdown docs via new markdown endpoints
and routing; pages now advertise markdown alternates (including
pricing).
  * Added Cron and Queues module documentation pages.

* **Documentation**
  * Minor formatting tweaks to Realtime and Storage docs.

* **Chores**
* Added build-time Markdown content generation and adjusted
ignore/deploy rules for generated files.
* Added redirects from legacy text-based product URLs to new markdown
pages.

* **Tests**
* Expanded tests for markdown routing and content-negotiation behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-04-28 16:41:03 +09:00
Alaister Young cb34acd45c [GROWTH-773] chore(www): serve llms.txt content via API routes (#44897)
The docs build had a fragile implicit dependency on www's filesystem
(`../../../apps/www/public/llms`), flagged by the docs team in #44670.
Rather than formalising that dependency with a shared package, this PR
eliminates it entirely by making www the sole owner of llms content
assembly.

**How it works now:**

`/llms/[slug]` handles all `/llms/*.txt` requests via a 3-step cascade:
1. Dynamic content — `pricing.txt` generated at request time from
`shared-data` imports
2. Local file — product overviews read from `data/llms/`
3. Docs proxy — reference docs (guides, js, dart, etc.) fetched from the
docs app

No hardcoded slug lists, so adding new content just works.

**What changed:**

- `apps/docs/scripts/llms.ts` trimmed to only generate per-source
reference files — www now owns `llms.txt`, `llms-full.txt`, and product
overviews
- Removed `generateLlmsPricing.mjs` build script — pricing generated
dynamically from `shared-data`
- Removed llms rewrites from `rewrites.js` — routes handle everything
with consistent `Cache-Control: public, s-maxage=3600,
stale-while-revalidate=86400`
- Product overview `.txt` files moved from `public/llms/` → `data/llms/`
so all requests go through routes for consistent caching

**Docs team concerns from GROWTH-773:**

| Concern | Resolution |
|---------|-----------|
| Docs build depends on www files at a fragile relative path | Path
removed — docs no longer reads from www |
| www restructuring breaks docs with no obvious connection | Eliminated
— no cross-app filesystem dependency |
| No build order enforcement between www and docs | Not needed — docs
doesn't depend on www's build output |

## To test

- `curl <preview>/llms.txt` — markdown index with doc + product overview
links
- `curl <preview>/llms-full.txt` — combined product overviews + docs
content
- `curl <preview>/llms/pricing.txt` — dynamically generated pricing
tables
- `curl <preview>/llms/auth.txt` — product overview from local file
- `curl <preview>/llms/guides.txt` — proxied from docs app
- `curl <preview>/llms/nonexistent.txt` — 404
- Verify `Cache-Control` header on all responses

---------

Co-authored-by: Alaister Young <10985857+alaister@users.noreply.github.com>
2026-04-15 22:45:01 +09:00