Files
cloudflare__vinext/tests
Iago Lima Toledo 023aa76403 fix(metadata): add meta's 2024 crawler UAs to the html-limited bot list (#2666)
* fix(metadata): add meta's 2024 crawler UAs to the html-limited bot list

Meta introduced meta-externalagent and meta-externalfetcher in 2024. The
default html-limited bot list (mirrored from Next.js) only carries the
legacy facebookexternalhit, so the new UAs get streamed metadata. Meta's
crawlers don't execute JS, so every og:* tag lands after </head> and
link previews on WhatsApp/Instagram/Facebook render blank.

Deliberate, documented divergence from Next.js's default list (which
predates Meta's UA migration) per AGENTS.md. A user-provided
htmlLimitedBots config still replaces the whole list.

Test: new UAs get blocking metadata, the legacy UA stays covered, and a
custom htmlLimitedBots config still replaces the default.

* refactor(metadata): share html-limited bot detection

---------

Co-authored-by: MrIago <28714061+MrIago@users.noreply.github.com>
Co-authored-by: James <james@eli.cx>
2026-07-21 23:17:55 +01:00
..
2026-03-15 10:50:13 +00:00