Files
Mert Koseoglu 8476db7970 feat(fetch): finish the ladder — rung 2 recovers SPA pages browser-free
The measured gap: developer.apple.com/documentation/swiftui/view converts to
36 B of text from a 17,486 B shell, and reactnative.dev/docs/view ignores the
Accept header entirely. Both publish the article as a .md sibling. Rung 1
could not reach either; nothing below rung 1 existed.

Rung 2 now fires ONLY when the cheaper rungs did not produce an article, so
the happy path still costs exactly one request (asserted against a request
log, not inferred):

  2a  the page's .md sibling
  2b  the host's llms.txt, followed only when it names this page somewhere 2a
      did not already try

Acceptance is structural: a sibling is taken unless the server handed back an
HTML document. developer.apple.com serves its .md with an EMPTY Content-Type
and an HTML comment as its first bytes, so a 'starts with #' test would reject
a real article; angular.dev answers a missing .md with 200 + the SPA shell, so
status alone would accept a soft 404.

Every fetch now reports WHICH RUNG ANSWERED (stdout line 4) and which rung-2
urls were requested (line 5). The honest refusal names them instead of telling
the caller to go looking for files the ladder already asked for.

classifyExtraction is injected into the subprocess the same way classifyIp is,
so 'is this a shell?' has one definition and two callers. Its thresholds moved
inside the body to survive .toString() under esbuild minification — verified
against the shipped bundle, not just the source.

Also: 'SSRF blocked: redirect chain exceeded' no longer accuses an attack when
a benign locale redirect loop produces it (measured on Google devsite hosts).

Measured with scripts/measure-fetch-ladder.cjs over 36 documentation pages.
2026-08-12 23:39:27 +03:00
..