Mert Koseoglu 0ce043fe8c fix(fetch): stop reporting a JavaScript-rendered shell as a successful fetch ...

The only emptiness guard on the fetch path was `markdown.length === 0`.
excalidraw.com serves 6,862 bytes of bootstrap HTML whose entire text content
is its <title>; Turndown converts that to the 21 bytes "Excalidraw Whiteboard".
21 is not 0, so ctx_fetch_and_index answered "fetched successfully", indexed
the shell as though it were the page, and the caller had no way to tell that
apart from a genuinely short document.

Measured on this exact code path (bytes received -> markdown bytes out):

  excalidraw.com              6,862 ->     21   ( 0.31% yield)  <- shell
  app.diagrams.net            2,759 ->    476   (17.25% yield)
  nextjs.org                316,151 -> 13,587   ( 4.30% yield)
  developers.cloudflare.com 178,627 ->  8,389   ( 4.70% yield)

The subprocess now reports the pre-conversion byte count on stdout line 2.
Line 1 keeps its exact historical `__CM_CT__:<type>` shape, so the two
`header === "__CM_CT__:json"` comparisons downstream are untouched, and a
missing line 2 (older bundle) reads as "no evidence" rather than as a failure.

classifyExtraction() then requires BOTH signals before accusing: almost no
text came out (< 200 bytes) AND almost none of what came in survived (< 2%
yield). Neither works alone -- a ratio alone condemns a small valid page
(`<p>Hello</p>` is 5 bytes of text from 38 of markup and is perfectly fine),
a floor alone condemns a genuinely short document. Requiring both isolates
the shell case and leaves all three real pages above untouched. Arithmetic
only: no pattern matching, no markup sniffing, no word lists.

A shell is now `fetch_error` with reason "shell", carrying both byte counts
and the yield, and saying why a retry will return the same 21 bytes. Nothing
is indexed -- storing a bootstrap shell under the page's name poisons the
knowledge base and leaves the agent believing it has the document.

Verified live end-to-end after the change: excalidraw.com is refused with the
diagnosis, the other three are unaffected.

2026-08-12 22:26:31 +03:00

8.2 KiB

Raw Permalink Blame History

View Raw