mirror of
https://github.com/jackwener/OpenCLI.git
synced 2026-09-14 18:25:42 +08:00
jakevin
e83148a2c1
* feat(download): harden HTML→Markdown pipeline Inspired by the MD-This-Page / markdown-viewer-extension analysis, tighten the shared article→Markdown converter used by zhihu/weixin/web adapters: - enable turndown-plugin-gfm (tables, strikethrough, task lists) - strip script/style/noscript/iframe/canvas/form/button/dialog unconditionally - strip SVG via a dedicated rule (not in HTMLElementTagNameMap) - drop base64 data-URI images so they don't bloat .md output - post-process: collapse NBSP, lone bullet/middle-dot residue, trailing whitespace, and 3+ blank lines - frontmatter shape guarantees ≤2 consecutive newlines even when some metadata fields are absent Adds a minimal local .d.ts for turndown-plugin-gfm and 6 new tests covering GFM conversion, tag stripping, base64 drop, and whitespace cleanup. * fix(download): emit canonical markdown strikethrough * feat(download,browser): finish article pipeline polish Per the follow-up from the MD-This-Page / markdown-viewer-extension analysis, land the remaining items in the same PR instead of splitting: article-download.ts - extend STRIPPED_TAGS with header/footer/nav/aside (page chrome; the article's title/author/publishTime are supplied as separate fields on ArticleData, so duplicated DOM is redundant) - new option ArticleDownloadOptions.cleanSelectors — per-adapter CSS selector list removed before conversion, applied as a Turndown rule via node.matches so invalid selectors fail silently browser/article-extract.ts (new) - generic Readability-based extraction that runs in-page via CDP evaluate (no jsdom in Node) - short-circuits non-HTML documents (text/plain, JSON, XML) and the single-<pre> "browser rendering a plain text file" case - clones the document before any mutation (preserves live page state for subsequent snapshot / click) - isProbablyReaderable gate, Readability.parse on the clone, then a fallback chain main → [role="main"] → #main-content → … → body - library sources are JSON-embedded and eval'd inside a Function scope so their backticks / module.exports guards don't collide with the surrounding IIFE Tests - article-download: page-chrome strip, cleanSelectors match + invalid selector silently ignored (2 new) - article-extract: JS generation contents, default fallback chain, response normalization, null / malformed handling, and a Function() parse check to catch any template-literal break-out in the embedded Readability sources (8 new) * fix(download): honor selector cleanup in fallback paths * test(e2e): real-site regression for hardened article pipeline Adds tests/e2e/article-download-pipeline.test.ts driving `opencli web read` through 6 representative pages (example.com baseline, Wikipedia GFM tables, MDN metadata, GitHub fenced code, Vercel SSR blog, Ruan Yifeng CJK+images) and asserting the post-processing invariants: no base64/script/style leaks, no blank-line runs, no residue, no trailing whitespace, no NBSP. Graceful skip on bot detection / transient CDP errors, with a single retry. All 6 sites pass locally (37s total). * test(browser): add article extraction e2e fixtures