mirror of
https://github.com/jackwener/OpenCLI.git
synced 2026-09-14 18:25:42 +08:00
jakevin
e83148a2c1
* feat(download): harden HTML→Markdown pipeline Inspired by the MD-This-Page / markdown-viewer-extension analysis, tighten the shared article→Markdown converter used by zhihu/weixin/web adapters: - enable turndown-plugin-gfm (tables, strikethrough, task lists) - strip script/style/noscript/iframe/canvas/form/button/dialog unconditionally - strip SVG via a dedicated rule (not in HTMLElementTagNameMap) - drop base64 data-URI images so they don't bloat .md output - post-process: collapse NBSP, lone bullet/middle-dot residue, trailing whitespace, and 3+ blank lines - frontmatter shape guarantees ≤2 consecutive newlines even when some metadata fields are absent Adds a minimal local .d.ts for turndown-plugin-gfm and 6 new tests covering GFM conversion, tag stripping, base64 drop, and whitespace cleanup. * fix(download): emit canonical markdown strikethrough * feat(download,browser): finish article pipeline polish Per the follow-up from the MD-This-Page / markdown-viewer-extension analysis, land the remaining items in the same PR instead of splitting: article-download.ts - extend STRIPPED_TAGS with header/footer/nav/aside (page chrome; the article's title/author/publishTime are supplied as separate fields on ArticleData, so duplicated DOM is redundant) - new option ArticleDownloadOptions.cleanSelectors — per-adapter CSS selector list removed before conversion, applied as a Turndown rule via node.matches so invalid selectors fail silently browser/article-extract.ts (new) - generic Readability-based extraction that runs in-page via CDP evaluate (no jsdom in Node) - short-circuits non-HTML documents (text/plain, JSON, XML) and the single-<pre> "browser rendering a plain text file" case - clones the document before any mutation (preserves live page state for subsequent snapshot / click) - isProbablyReaderable gate, Readability.parse on the clone, then a fallback chain main → [role="main"] → #main-content → … → body - library sources are JSON-embedded and eval'd inside a Function scope so their backticks / module.exports guards don't collide with the surrounding IIFE Tests - article-download: page-chrome strip, cleanSelectors match + invalid selector silently ignored (2 new) - article-extract: JS generation contents, default fallback chain, response normalization, null / malformed handling, and a Function() parse check to catch any template-literal break-out in the embedded Readability sources (8 new) * fix(download): honor selector cleanup in fallback paths * test(e2e): real-site regression for hardened article pipeline Adds tests/e2e/article-download-pipeline.test.ts driving `opencli web read` through 6 representative pages (example.com baseline, Wikipedia GFM tables, MDN metadata, GitHub fenced code, Vercel SSR blog, Ruan Yifeng CJK+images) and asserting the post-processing invariants: no base64/script/style leaks, no blank-line runs, no residue, no trailing whitespace, no NBSP. Graceful skip on bot detection / transient CDP errors, with a single retry. All 6 sites pass locally (37s total). * test(browser): add article extraction e2e fixtures
60 lines
1.6 KiB
TypeScript
60 lines
1.6 KiB
TypeScript
import { defineConfig } from 'vitest/config';
|
|
|
|
const includeExtendedE2e = process.env.OPENCLI_E2E === '1';
|
|
|
|
export default defineConfig({
|
|
test: {
|
|
projects: [
|
|
{
|
|
test: {
|
|
name: 'unit',
|
|
include: ['src/**/*.test.ts'],
|
|
exclude: ['clis/**/*.test.{ts,js}'],
|
|
sequence: { groupOrder: 0 },
|
|
},
|
|
},
|
|
{
|
|
test: {
|
|
name: 'extension',
|
|
include: ['extension/src/**/*.test.ts'],
|
|
sequence: { groupOrder: 0 },
|
|
},
|
|
},
|
|
{
|
|
test: {
|
|
name: 'adapter',
|
|
include: ['clis/**/*.test.{ts,js}'],
|
|
sequence: { groupOrder: 1 },
|
|
},
|
|
},
|
|
{
|
|
test: {
|
|
name: 'e2e',
|
|
include: [
|
|
'tests/e2e/browser-public.test.ts',
|
|
'tests/e2e/band-auth.test.ts',
|
|
'tests/e2e/public-commands.test.ts',
|
|
'tests/e2e/management.test.ts',
|
|
'tests/e2e/output-formats.test.ts',
|
|
'tests/e2e/plugin-management.test.ts',
|
|
'tests/e2e/browser-tabs.test.ts',
|
|
'tests/e2e/article-download-pipeline.test.ts',
|
|
// Extended browser tests (20+ sites) — opt-in only:
|
|
// OPENCLI_E2E=1 npx vitest run
|
|
...(includeExtendedE2e ? ['tests/e2e/browser-public-extended.test.ts', 'tests/e2e/browser-auth.test.ts', 'tests/e2e/douban.test.ts'] : []),
|
|
],
|
|
maxWorkers: 2,
|
|
sequence: { groupOrder: 2 },
|
|
},
|
|
},
|
|
{
|
|
test: {
|
|
name: 'smoke',
|
|
include: ['tests/smoke/**/*.test.ts'],
|
|
sequence: { groupOrder: 3 },
|
|
},
|
|
},
|
|
],
|
|
},
|
|
});
|