* feat(download): harden HTML→Markdown pipeline
Inspired by the MD-This-Page / markdown-viewer-extension analysis, tighten
the shared article→Markdown converter used by zhihu/weixin/web adapters:
- enable turndown-plugin-gfm (tables, strikethrough, task lists)
- strip script/style/noscript/iframe/canvas/form/button/dialog unconditionally
- strip SVG via a dedicated rule (not in HTMLElementTagNameMap)
- drop base64 data-URI images so they don't bloat .md output
- post-process: collapse NBSP, lone bullet/middle-dot residue,
trailing whitespace, and 3+ blank lines
- frontmatter shape guarantees ≤2 consecutive newlines even when
some metadata fields are absent
Adds a minimal local .d.ts for turndown-plugin-gfm and 6 new tests
covering GFM conversion, tag stripping, base64 drop, and whitespace cleanup.
* fix(download): emit canonical markdown strikethrough
* feat(download,browser): finish article pipeline polish
Per the follow-up from the MD-This-Page / markdown-viewer-extension
analysis, land the remaining items in the same PR instead of splitting:
article-download.ts
- extend STRIPPED_TAGS with header/footer/nav/aside (page chrome; the
article's title/author/publishTime are supplied as separate fields on
ArticleData, so duplicated DOM is redundant)
- new option ArticleDownloadOptions.cleanSelectors — per-adapter CSS
selector list removed before conversion, applied as a Turndown rule
via node.matches so invalid selectors fail silently
browser/article-extract.ts (new)
- generic Readability-based extraction that runs in-page via CDP
evaluate (no jsdom in Node)
- short-circuits non-HTML documents (text/plain, JSON, XML) and the
single-<pre> "browser rendering a plain text file" case
- clones the document before any mutation (preserves live page state
for subsequent snapshot / click)
- isProbablyReaderable gate, Readability.parse on the clone, then a
fallback chain main → [role="main"] → #main-content → … → body
- library sources are JSON-embedded and eval'd inside a Function scope
so their backticks / module.exports guards don't collide with the
surrounding IIFE
Tests
- article-download: page-chrome strip, cleanSelectors match + invalid
selector silently ignored (2 new)
- article-extract: JS generation contents, default fallback chain,
response normalization, null / malformed handling, and a Function()
parse check to catch any template-literal break-out in the embedded
Readability sources (8 new)
* fix(download): honor selector cleanup in fallback paths
* test(e2e): real-site regression for hardened article pipeline
Adds tests/e2e/article-download-pipeline.test.ts driving `opencli web read`
through 6 representative pages (example.com baseline, Wikipedia GFM tables,
MDN metadata, GitHub fenced code, Vercel SSR blog, Ruan Yifeng CJK+images)
and asserting the post-processing invariants: no base64/script/style leaks,
no blank-line runs, no residue, no trailing whitespace, no NBSP.
Graceful skip on bot detection / transient CDP errors, with a single retry.
All 6 sites pass locally (37s total).
* test(browser): add article extraction e2e fixtures
* fix: sync package-lock.json with package.json dependencies
package-lock.json was missing @emnapi/core@1.9.2 and
@emnapi/runtime@1.9.2 (transitive deps of @emnapi/wasi-threads),
causing `npm ci` to fail on all CI jobs.
* fix: resolve remaining CI failures after TS-to-JS adapter migration
- vitest.config.ts: update adapter project include/exclude from .test.ts
to .test.{ts,js} to match converted adapter test files
- check-doc-coverage.sh: skip adapter directories containing only utility
files (prefixed with _), fixing false positive for clis/slock/
- linux-do/topic-content.test.js: fix hardcoded reference to topic.ts
(now topic.js after PR #928 migration)
* refactor: simplify core modules — remove root shims, consolidate error classification, streamline cascade/interceptor, clean up synthesize
1. Remove root-level shim files (errors.ts, logger.ts, registry.ts, types.ts, utils.ts, launcher.ts) — update all ~840 adapter imports to reference src/ directly
2. Consolidate interceptor: reuse shared DISGUISE_FN in tap interceptor instead of reimplementing
3. Unify error classification: single ClassifiedError type with icon/exitCode/hint lookup table, eliminating duplicated pattern matching between resolveExitCode and renderError
4. Simplify cascade probe: replace repetitive switch cases with PROBE_OPTIONS lookup map
12. Clean up synthesize.ts: remove deprecated snake_case field aliases (recommended_args, recommended_columns, recommendedColumnsLegacy) and unnecessary constant aliases
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs: update import paths in contributor docs and skill templates
Update all documentation and skill files to reference src/ directly,
matching the shim removal in the previous commit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update new Gemini adapter imports to use src/ paths
Fix imports in newly added deep-research adapter files that were
still referencing the deleted root shim files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update xiaohongshu tests for /search_result/ URL change
Tests now expect /search_result/<id> for bare note IDs (matching
the note-helpers.ts change from PR #774) and updated empty-shell
hint assertion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update LessWrong and hupu adapter imports to use src/ paths
Fix imports in newly merged LessWrong and hupu/mentions adapter
files that were still referencing the deleted root shim files.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(xueqiu): mock logger via src path
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat(hupu): add hupu cli adapter
* fix(hupu): prevent detail from returning the wrong thread
* refactor: deduplicate shared utilities in hupu adapter
- Merge postHupuJson and postHupuReplyJson into single function with mode parameter
- Move stripHtml and decodeHtmlEntities to utils.ts, remove duplicate definitions
* fix(hupu): add mentions command
* fix: move mentions.ts to clis/hupu/, remove src/clis/hupu duplicates
Post PR #782 restructure: adapter files live at root clis/, not src/clis/.
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* feat(hupu): add hupu cli adapter
* fix(hupu): prevent detail from returning the wrong thread
* refactor: deduplicate shared utilities in hupu adapter
- Merge postHupuJson and postHupuReplyJson into single function with mode parameter
- Move stripHtml and decodeHtmlEntities to utils.ts, remove duplicate definitions
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
turndown and @types/turndown were used in article-download.ts and
zhihu/download.test.ts but never declared in package.json, causing
CI failures on fresh npm ci installs.
Cleanup:
- Remove redundant double-retry in resolveTabId (was retrying data: URI
with the same data: URI)
- Fix stale comment (30s → 120s idle timeout)
- Remove verbose debug logging in resolveTabId
- Built extension is now smaller (16.66kB vs 17.18kB)
Extension conflict:
- Add hint to attach-failed error when chrome-extension:// URL is detected
- Add troubleshooting entry for extension conflicts (e.g. youmind, New Tab
Override) to both README.md and README.zh-CN.md
Ref: #249
When a new automation window is created, the initial tab URL may be
empty briefly while Chrome loads the data: URI. isDebuggableUrl('') was
returning false, causing ensureAttached to reject the tab.
Fix: only reject known non-debuggable URLs (chrome://, chrome-extension://).
Empty/undefined URLs are now treated as debuggable since they represent
tabs still loading.
Also adds 200ms delay after window creation to let Chrome populate the
tab URL.
Root cause: getAutomationWindow and resolveTabId used about:blank which
New Tab Override extensions intercept immediately, replacing it with
chrome-extension:// URLs that cannot be debugged.
Changes:
- Window creation: about:blank → data:text/html
- reuseTab fallback: about:blank → data:text/html
- newTab handler: about:blank → data:text/html
- Added diagnostic logging to resolveTabId for debugging
- Synced extension version to 1.2.4
Ref: #249
dependabot[bot]