Commit Graph

29 Commits

Author SHA1 Message Date
jakevin e83148a2c1 feat(download): harden HTML→Markdown pipeline (#1143)
* feat(download): harden HTML→Markdown pipeline

Inspired by the MD-This-Page / markdown-viewer-extension analysis, tighten
the shared article→Markdown converter used by zhihu/weixin/web adapters:

- enable turndown-plugin-gfm (tables, strikethrough, task lists)
- strip script/style/noscript/iframe/canvas/form/button/dialog unconditionally
- strip SVG via a dedicated rule (not in HTMLElementTagNameMap)
- drop base64 data-URI images so they don't bloat .md output
- post-process: collapse NBSP, lone bullet/middle-dot residue,
  trailing whitespace, and 3+ blank lines
- frontmatter shape guarantees ≤2 consecutive newlines even when
  some metadata fields are absent

Adds a minimal local .d.ts for turndown-plugin-gfm and 6 new tests
covering GFM conversion, tag stripping, base64 drop, and whitespace cleanup.

* fix(download): emit canonical markdown strikethrough

* feat(download,browser): finish article pipeline polish

Per the follow-up from the MD-This-Page / markdown-viewer-extension
analysis, land the remaining items in the same PR instead of splitting:

article-download.ts
- extend STRIPPED_TAGS with header/footer/nav/aside (page chrome; the
  article's title/author/publishTime are supplied as separate fields on
  ArticleData, so duplicated DOM is redundant)
- new option ArticleDownloadOptions.cleanSelectors — per-adapter CSS
  selector list removed before conversion, applied as a Turndown rule
  via node.matches so invalid selectors fail silently

browser/article-extract.ts (new)
- generic Readability-based extraction that runs in-page via CDP
  evaluate (no jsdom in Node)
- short-circuits non-HTML documents (text/plain, JSON, XML) and the
  single-<pre> "browser rendering a plain text file" case
- clones the document before any mutation (preserves live page state
  for subsequent snapshot / click)
- isProbablyReaderable gate, Readability.parse on the clone, then a
  fallback chain main → [role="main"] → #main-content → … → body
- library sources are JSON-embedded and eval'd inside a Function scope
  so their backticks / module.exports guards don't collide with the
  surrounding IIFE

Tests
- article-download: page-chrome strip, cleanSelectors match + invalid
  selector silently ignored (2 new)
- article-extract: JS generation contents, default fallback chain,
  response normalization, null / malformed handling, and a Function()
  parse check to catch any template-literal break-out in the embedded
  Readability sources (8 new)

* fix(download): honor selector cleanup in fallback paths

* test(e2e): real-site regression for hardened article pipeline

Adds tests/e2e/article-download-pipeline.test.ts driving `opencli web read`
through 6 representative pages (example.com baseline, Wikipedia GFM tables,
MDN metadata, GitHub fenced code, Vercel SSR blog, Ruan Yifeng CJK+images)
and asserting the post-processing invariants: no base64/script/style leaks,
no blank-line runs, no residue, no trailing whitespace, no NBSP.

Graceful skip on bot detection / transient CDP errors, with a single retry.

All 6 sites pass locally (37s total).

* test(browser): add article extraction e2e fixtures
2026-04-22 14:49:35 +08:00
zhengyu fbdb1b24dc fix(browser): harden multi-tab routing and target isolation (#1072)
* fix(browser): harden multi-tab routing and target isolation

- make daemon command ids collision-resistant and retry duplicate pending ids\n- add validated tab list/new/select/close flows with persisted default targets\n- keep untargeted browser commands on the default tab unless tab select changes it\n- document tab targeting and add unit, extension, and e2e coverage for concurrent multi-tab execution

* fix(browser): keep default tab stable after tab new

* fix(browser): close remaining tab routing gates

* docs(browser): align target id wording

* docs(browser): refine target id examples

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-04-19 18:48:45 +08:00
warkcod 0e38fd8c37 Feat/douban book subject (#993)
* chore: ignore local worktrees

* feat(douban): support book subject details
2026-04-14 20:41:14 +08:00
jakevin cc18ed67b7 fix: sync package-lock.json to unblock CI (#937)
* fix: sync package-lock.json with package.json dependencies

package-lock.json was missing @emnapi/core@1.9.2 and
@emnapi/runtime@1.9.2 (transitive deps of @emnapi/wasi-threads),
causing `npm ci` to fail on all CI jobs.

* fix: resolve remaining CI failures after TS-to-JS adapter migration

- vitest.config.ts: update adapter project include/exclude from .test.ts
  to .test.{ts,js} to match converted adapter test files
- check-doc-coverage.sh: skip adapter directories containing only utility
  files (prefixed with _), fixing false positive for clis/slock/
- linux-do/topic-content.test.js: fix hardcoded reference to topic.ts
  (now topic.js after PR #928 migration)
2026-04-10 23:29:09 +08:00
jakevin 555626f409 feat: replace tabId with targetId as cross-layer page identity (#899)
Use Chrome CDP targetId (UUID) as the canonical page identity across
all layers (extension → daemon → CLI), demoting tabId to an
extension-internal routing detail.

- Add extension/src/identity.ts: bidirectional targetId ↔ tabId mapping
  with lazy refresh via chrome.debugger.getTargets()
- Update protocol: Command.page and Result.page carry targetId
- Update background.ts: resolveCommandTabId() and pageScopedResult()
  helpers; all page-scoped handlers return targetId
- Add sendCommandFull() to daemon-client for responses with page identity
- Update Page class: _page stores targetId, goto/selectTab extract it
- Update record.ts: injectedPages tracks by targetId
- Add extension tests to vitest config and CI test scripts
2026-04-09 12:26:32 +08:00
jakevin 80eef46b4e refactor: monorepo adapter separation (clis/ at root) (#782)
* refactor: move adapters from src/clis/ to root clis/ for monorepo separation

Separates CLI adapters from the core runtime to prepare for independent
adapter distribution via postinstall fetch.

Key changes:
- Move src/clis/ → clis/ (adapters at repo root)
- Change tsconfig rootDir from "src" to "." so tsc compiles both
- Create root-level shim files (registry.ts, errors.ts, etc.) so adapter
  relative imports (../../registry.js) resolve correctly
- Update build-manifest.ts, main.ts paths for new dist/src/ structure
- Expand ensureUserCliCompatShims() to cover all adapter import targets
  (types, utils, logger, launcher, browser/*, download/*, pipeline/*)
- Add scripts/fetch-adapters.js postinstall for ~/.opencli/clis/ sync
- Update vitest.config.ts adapter test paths
- Add package.json files field to exclude adapters from npm package

Official adapter files are unconditionally overwritten on update;
user-created files not in the manifest are preserved.

* fix: add dist/clis/ and cli-manifest.json to npm files, harden fetch-adapters

- Add dist/clis/ and dist/cli-manifest.json to package.json files field
  so built-in adapters and manifest ship with the npm package
- Replace execSync with execFileSync to prevent command injection
- Add version check to skip redundant adapter fetches
- Track tmpRoot explicitly for reliable cleanup

* fix: address review blockers — manifest-based updates, global-only fetch, first-run fallback

1. Manifest-based update strategy:
   - Read old manifest to identify previously-official files
   - Clean up files removed upstream (in old manifest but not new)
   - User-created files (never in any manifest) remain untouched

2. Only run fetch-adapters on global install (npm_config_global=true)
   or explicit OPENCLI_FETCH=1, preventing heavy side effects for
   local/dev installs

3. First-run fallback in discovery.ts:
   - ensureUserAdapters() checks for adapter-manifest.json
   - If missing and ~/.opencli/clis/ is empty, spawns fetch-adapters.js
   - Guarantees adapters are available even with --ignore-scripts

* fix: remove OPENCLI_FETCH env var, use internal _OPENCLI_FIRST_RUN instead

* feat: also support OPENCLI_FETCH=1 for explicit adapter fetch trigger

* simplify: replace git clone with local copy from dist/clis/

Adapters already ship in the npm package (dist/clis/), so there's no
need to clone from GitHub. Copy directly from the installed package:

- Eliminates git, curl, tar dependencies
- No network calls in postinstall
- No timeout/offline issues
- Version always matches the installed CLI
- ~65 lines of clone/download code replaced by one cpSync loop
2026-04-05 01:46:36 +08:00
康厚超 440c001a20 feat(band): add Band.us adapter — bands, posts, mentions, post commands (#532)
* feat(band): add bands, posts, and mentions commands for band.us

- bands: lists all Bands via get_band_list_with_filter intercept
- posts: lists posts from a Band via get_posts_and_announcements intercept
- mentions: shows @mention notifications via get_news intercept

All use Strategy.INTERCEPT since band.us API requires an HMAC md header
generated by its own JS. SPA navigation to /band/{no}/post triggers the
band list and posts APIs; bell + @メンション tab click triggers mentions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(band): clean up all three band adapters

- Fix doc comments: Band uses XHR not fetch; clarify INTERCEPT rationale
- bands: replace for-loop with flatMap; explain why band page nav is needed
- posts: remove item.post ?? item fallback (API always wraps in post); rename
  finalRequests → requests for consistency; extract stripBandTags helper
- mentions: remove redundant ?? defaults (args have defaults defined); fix
  unreadOnly bug (was not applied to post/comment modes); consolidate Band tag
  stripping to single regex; cast kwargs types directly instead of converting;
  add comments explaining last-response strategy and 'referred' filter flag

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band/posts): handle mixed post/announcement items from API

get_posts_and_announcements returns both regular posts and announcements
that have different shapes — some lack post_no and wrap differently.
Restore item.post ?? item fallback and filter out items with no resolvable
identifier to prevent undefined in URLs and empty rows in output.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(band): add post command — full post export with comments and photo download

Exports the complete content of a single Band post:
- Post body (with Band markup tags stripped)
- All comments in chronological order
- Photo URLs shown inline, or downloaded with --output <dir>

Uses Strategy.INTERCEPT with a broad 'band.us' pattern to capture both the
batch request (embedding get_post) and get_comments in one SPA navigation.
Responses are identified client-side by shape: batch_result array vs items
array with comment_id fields.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(band): replace XHR interception with direct DOM extraction

- bands, posts, post: navigate directly to target URL instead of home→SPA detour
- All three switch from Strategy.INTERCEPT to Strategy.COOKIE with navigateBefore: false
  (bands uses framework pre-nav to home; posts/post disable it and goto target directly)
- DOM extraction polls for specific content elements rather than fixed waits
- post: confirm selectors via browser inspection (a.text, time.time, .sCommentList,
  .sReplyList for nested replies); add --comments flag to skip comment fetch
- posts: extract from rendered post list DOM; correct comment item selector (div.cComment)
- Fix: post empty-result guard changed from && to handle null data safely
- Fix: photo download now checks HTTP status code before piping to avoid writing
  redirect HTML into image files
- Fix: mentions unread client-side filter skipped for 'mentioned' mode since
  server already filtered via 未確認のみ button click

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): address code review feedback

- post: replace manual http/https download with shared downloadMedia utility
  (handles redirects, timeouts, stream errors correctly)
- post: fix photo URL resolution to use location.href as base, handling
  protocol-relative and relative URLs without throwing
- post: switch to node:-prefixed imports per repo convention
- post/posts: remove redundant ArgumentError guards — framework already
  validates required args before func() is called
- mentions: INTERCEPT strategy is intentional (Band HMAC prevents DOM-only
  approach for notifications; update PR description to clarify)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): address second round of code review feedback

- bands: tighten href selector to /band/{id}(?:/post)?$ so feed/post-detail
  links are excluded; only sidebar navigation links match
- mentions: replace fixed page.wait(2) sleeps with polling on
  getInterceptedRequests() — waits up to 8 s per action, exits as soon
  as the expected number of captures arrives (avoids flakiness on slow XHR)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): fix selector bugs found during testing

- bands: use a.bandCover._link + p.uriText + span.member em selectors
  (previous a[href*="/band/"] + .bandName combo leaked "メンバー" text)
- posts: use article.cContentsCard._postMainWrap + span.count selectors
  (previous li._postListItem selector matched nothing; DOM changed)
- mentions: fix page.wait(500) → page.wait(0.5) (was waiting 500s not ms);
  use timestamp-suffixed URL to force fresh page load each run so the
  notification panel is closed; fix get_news vs get_news_count capture
  ambiguity with result_data.news check; replace cumulative waitForCaptures
  with waitForOneCapture (getInterceptedRequests clears array on each call)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band/mentions): use CSS class selector for bell button instead of locale-dependent text match

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): address third round of code review feedback

- post: pass browser cookies to downloadMedia so Band's login-protected
  photo URLs don't fail with 401/403
- post: include photos.length in empty-result guard so photo-only posts
  are not falsely reported as not found
- mentions: accumulate captures across poll iterations so get_news_count
  responses don't cause early exit before the real get_news arrives
- mentions: update docstring to match actual implementation (client-side
  filtering, no tab-click)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): address fourth round of code review feedback

- mentions: fail fast with a clear error when bell button is not found,
  instead of silently no-op and waiting 8s before EmptyResultError
- post: use shared formatCookieHeader() instead of manual cookie string
  construction

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): address fifth round of code review feedback

- mentions: replace fixed page.wait(2) with polling for bell button
  readiness (up to 10s), eliminating the fixed sleep and fail-fast
  when the selector is missing
- mentions: add explicit !newsReq guard with a clear error message when
  get_news capture times out, instead of falling through to a misleading
  "No notifications found"
- posts: skip posts with no permalink href instead of emitting a bogus
  'https://www.band.us' URL

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): address sixth round of code review feedback

- post: only send Band cookies to *.band.us photo URLs; third-party CDN
  URLs are downloaded without cookies to avoid cross-domain cookie leakage
- bands: strip non-digit chars before parseInt so member counts like
  "1,234" parse correctly
- posts: same fix for comment counts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): address seventh round of code review feedback

- posts: check limit before push so --limit 0 returns empty result
- post: indent replies proportionally by depth ('  '.repeat(depth))
  so multi-level threads remain readable in table output

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band/bands): anchor href regex to prevent matching post-detail URLs

Pattern now requires /band/{id} or /band/{id}/post (with optional trailing
slash) so deeper paths like /band/{id}/post/{postNo} are excluded.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): address ninth round of code review feedback

- mentions: guard bell click with a boolean return so a disappearing
  element throws a clear EmptyResultError instead of a raw TypeError
- post: wait for comment list container instead of first .cComment so
  posts with zero comments don't incur a fixed 6s delay

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): use page.getCookies() for login detection across all commands

Replaces document.cookie.includes('band_session') with
page.getCookies({ domain: 'band.us' }) so login detection works even
if Band.us marks the session cookie as HttpOnly in the future.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): address eleventh round of code review feedback

- mentions: replace EmptyResultError with SelectorError for missing/
  disappeared bell button — produces a clearer SELECTOR error code
- post: assign per-photo filenames using a global index across both
  download batches so band-hosted and CDN photos don't overwrite each other

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band): address twelfth round of code review feedback

- post: derive file extension from URL path and include in filename
  (e.g. photo_1.jpg) so downloaded photos have correct extensions
- posts: remove dead code guard (!url && !content) — url is always
  non-empty here since href-empty posts are already skipped above

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(band/post): use url-scoped getCookies for photo download auth

Domain-scoped getCookies may omit host-only cookies scoped to www.band.us;
using url: 'https://www.band.us' ensures all relevant cookies are included
in the auth header for Band-hosted photo downloads.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(band): add adapter documentation and sidebar entry

Required by CI doc-check --strict: every adapter in src/clis/ must have
a corresponding docs/adapters/browser/*.md file.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(e2e): wire band auth coverage into default matrix

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-29 17:19:26 +08:00
jakevin 0f3021a086 fix: relax extension version check, enable all adapter tests (#520)
* feat: zero onboarding, extension version check, and update notifier

- Fail-fast guard in execution.ts: when daemon is running but extension
  is not connected, immediately surface a setup guide instead of waiting
  for the 30s connect timeout

- Extension version handshake: extension sends `hello` with its version
  on WebSocket connect; daemon stores it and exposes via /status; CLI
  warns on mismatch in both execution path and `opencli doctor`

- `opencli doctor` now shows extension version inline and reports
  version mismatch as an actionable issue

- Non-blocking npm update checker: registers a process exit hook so the
  update notice appears after command output (same pattern as npm/gh/yarn);
  background fetch writes to ~/.opencli/update-check.json for next run

- postinstall: print Browser Bridge setup instructions after shell
  completion install for first-time global install users

Bug fixes caught in review:
- discover.ts: add AbortController timeout to checkDaemonStatus() fetch,
  move clearTimeout after res.json() to cover body streaming
- daemon.ts: clear extensionVersion and reject pending requests in
  ws.on('error') handler, not just ws.on('close')
- update-check.ts: skip update notice when process exits with non-zero
  code; read cache once at module load to avoid double disk I/O;
  guard isNewer() against NaN from pre-release version strings

* fix: relax extension version check to major-only in doctor, remove from hot path

* test: enable all adapter tests via wildcard glob, fix apple-podcasts url field

* fix: clearTimeout in finally block, reset extensionVersion on reconnect, fix e2e regex
2026-03-28 00:08:10 +08:00
Lr_2002 310e136a6c feat(paperreview): add paperreview.ai adapter (#464)
* feat(paperreview): add paperreview.ai adapter

* fix(cli): normalize boolean command options

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-26 22:25:30 +08:00
AstroHan 0773616c1e feat(imdb): add IMDb adapter with 6 commands (#472)
* feat(imdb): add IMDb adapter with 6 commands

Add a public IMDb adapter using browser-based JSON-LD and __NEXT_DATA__
extraction. All commands use Strategy.PUBLIC with browser: true.

Commands:
- imdb search <query> — search movies, TV shows, and people
- imdb title <id> — get movie/show details (Movie, TVSeries, TVEpisode, TVMiniseries, TVMovie, etc.)
- imdb top — IMDb Top 250 chart
- imdb trending — Most Popular Movies
- imdb person <id> — actor/director info with filmography
- imdb reviews <id> — user reviews (first page, max 25)

Shared utils: ID normalization, ISO 8601 duration formatting, locale
forcing, JSON-LD extraction (supports type array filtering), and
anti-bot challenge detection.

* review: harden imdb adapter loading and tests

* test: unblock PR CI on merge head

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-26 22:21:52 +08:00
槑囿脑袋 c75fea90ad feat(douban): add photo listing and download commands (#474)
* feat(douban): add photo listing and download commands

* refactor(douban): remove unreachable empty download branch

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-26 21:59:20 +08:00
Conn Ho 15b9bc8e0c feat(producthunt): add Product Hunt CLI adapter (#462)
* feat(producthunt): add Product Hunt CLI adapter

Add three commands:
- posts: RSS feed with optional category filter
- today: latest day's posts from feed
- hot: today's top posts with vote counts (browser INTERCEPT strategy)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(producthunt): add browse command for category best products

Browse top-rated products in any Product Hunt category (e.g. vibe-coding,
ai-agents, developer-tools) with name, tagline, and review count.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(producthunt): add adapter documentation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(producthunt): rebase on main and stabilize selectors

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-26 16:02:33 +08:00
Conn Ho 22399cee1a feat(36kr): add 36氪 CLI adapter (#461)
* feat(36kr): add 36氪 CLI adapter with 4 commands

- news: latest articles via public RSS feed (no browser needed), includes title/summary/date/url
- hot: trending articles via INTERCEPT strategy, supports --type renqi/zonghe/shoucang/catalog
- search: keyword search via INTERCEPT + DOM scraping
- article: fetch article detail (title/author/date/body) by ID or URL

Also adds vitest adapter project entry for 36kr tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(36kr): add adapter documentation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(36kr): use Shanghai hot-list dates and complete docs

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-26 15:34:46 +08:00
Xeron ed89157804 fix(jd): filter avif images only from pcpubliccms CDN (#453)
* feat(jd): add item adapter for JD.com product details

Support fetching:
- Product title, price, shop
- Product specifications (品牌, 型号, 规格参数 etc.)
- Main product images
- Detail images from product page

Usage: opencli jd item <sku>

* fix(jd): update test to expect avifImages column

* review: tighten jd item image contract

* fix: stabilize extension packaging and Chinese-site e2e

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-26 15:12:41 +08:00
Xiao Han 7348231b08 feat(twitter): add likes command (#448)
* feat(twitter): add likes command

* review: harden twitter likes query resolution

* refactor(twitter): share query id resolution

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-26 12:38:35 +08:00
glwlg 784bbc45f4 fix(xiaohongshu): improve image-text publish flow (#447)
* fix(xiaohongshu): improve image-text publish flow

Match visible 图文 tab labels instead of relying on narrow class selectors, fail early when the page is still on the video publish surface, and avoid injecting images into a generic file input. Add regression coverage for the image-text tab flow and the video-page failure case.

* test(xiaohongshu): include publish tests in adapter project

* fix(xiaohongshu): wait for image-text surface before upload

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-26 12:25:57 +08:00
tiaot33 e64046219d feat(linux-do): refactor adapters with unified feed, tags, user commands (#434)
* feat(linux-do): refactor adapters with unified feed, tags, user commands

- Replace hot/latest/category with unified `feed` command (tag/category/view routing)
- Add `tags`, `user-topics`, `user-posts` commands
- Add static data files for categories and tags lookup
- Fix error handling: use CliError subclasses instead of raw Error
- Fix Discourse API field mapping in search (tags, created)
- Add strategy: cookie to all YAML adapters
- Update docs and README command listings
- Update E2E tests for new command signatures

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* review: resolve linux-do feed from live metadata

* fix: restore linux-do CI

* fix: harden linux-do compatibility

* refactor: stabilize linux-do command migration

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-26 00:45:34 +08:00
Conn Ho 824dc38aab fix(weread): restore positional book-id coverage (#433)
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-25 23:58:40 +08:00
jakevin dba93c2739 fix(test): limit default e2e to bilibili/zhihu/v2ex, gate extended browser tests (#423)
Split browser-public.test.ts: core sites (bilibili, zhihu, v2ex) run
by default; all other 20+ site tests moved to browser-public-extended
and gated behind OPENCLI_E2E=1 to prevent AI agents from launching
dozens of browser instances.
2026-03-25 16:13:45 +08:00
jakevin 03d94ba2e1 chore: trim adapter test suite to bilibili, zhihu, v2ex only (#421)
Remove other adapter sites from vitest config to keep test runs
focused and avoid flaky failures from live site changes.
2026-03-25 16:01:15 +08:00
nianyi(likai) 39ca8330c5 feat(douyin): add Douyin creator center adapter (14 commands, 8-phase publish pipeline) (#416)
* feat(douyin): add Douyin creator center adapter (14 commands, 8-phase publish pipeline)

- publish: 8-phase pipeline (STS2 → TOS multipart upload w/ resume → ImageX cover → transcode poll → safety check → create_v2)
- draft: save as draft (phases 1-6 + is_draft:1, no timing)
- videos/drafts/delete/profile/update: content management
- hashtag (search/suggest/hot) / location / activities / collections / stats: discovery & analytics
- _shared: tos-upload (AWS Sig V4, multipart, resume), imagex-upload, transcode poller (encode=2), browser-fetch, sts2, creation-id, timing, text-extra
- 124 tests, tsc clean

* fix(douyin): accept unix timestamp strings

* docs(douyin): add browser adapter guide

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-25 15:36:07 +08:00
Ryan Tan d36a43e805 feat(pixiv): add Pixiv adapter (#403)
* feat(pixiv): add Pixiv adapter with 6 commands

Add support for Pixiv (pixiv.net) with the following commands:
- ranking: daily/weekly/monthly illustration rankings
- search: search illustrations by keyword/tag
- user: view artist profile info
- illusts: list illustrations by artist
- detail: view illustration details (tags, stats)
- download: download original-quality images

All commands use COOKIE strategy to reuse Chrome's logged-in session.
YAML adapters for simple API fetches (ranking, detail, user), TypeScript
for complex logic (search, illusts, download with Referer header).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test(pixiv): add unit tests and E2E auth failure tests

- search.test.ts: auth error, result parsing, limit, empty results (4 tests)
- illusts.test.ts: auth error, empty user, two-step fetch, limit (4 tests)
- download.test.ts: auth error, no images, Referer header, partial failure (4 tests)
- Add pixiv to vitest adapter project include list
- Add 5 pixiv commands to E2E browser-auth graceful failure tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(pixiv): correct ranking API path and YAML arg naming

- ranking: use /ranking.php?format=json (not /ajax/ranking which 404s)
- ranking: fix JSON path from data.body.contents to data.contents
- user/detail: rename hyphenated args (user-id → uid, illust-id → id)
  to fix YAML template evaluation (dot access doesn't support hyphens)

All 6 commands verified working against live Pixiv API.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(pixiv): use JSON.stringify to prevent code injection in page.evaluate

Address CodeRabbit review: all user inputs (query, userId, illustId,
idsParam) passed to page.evaluate are now serialized via JSON.stringify
instead of direct string interpolation, preventing code injection in
browser context.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(pixiv): address code review feedback

- ranking.yaml: add | json filter to page/limit args for defense-in-depth
- user.yaml: guard illusts/manga/novels with typeof check for robustness
- Extract shared createPageMock to test-utils.ts, deduplicate across 3 test files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(pixiv): use minimal page mock and add download E2E test

- test-utils.ts: slim down to minimal mock (goto, evaluate, getCookies)
  with overrides support, matching upstream's pragmatic mock style
- Add missing download command to E2E browser-auth graceful failure tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(pixiv): address all remaining CodeRabbit review comments

- detail.yaml: add url to columns to match description mentioning "URLs"
- All adapters: differentiate HTTP errors — 401/403 → AuthRequiredError,
  404 → "not found", others → generic "request failed (HTTP N)"
- Tests: use beforeAll to cache registry lookup, avoiding repeated reads
  from global singleton
- Tests: assert error type (AuthRequiredError) not just message content
- Tests: add dedicated test cases for non-auth errors (500) and 404

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(pixiv): add adapter docs and indexes

- Add pixiv.md documentation page under docs/adapters/browser/
- Update docs/adapters/index.md with pixiv entry
- Add Pixiv to sidebar in docs/.vitepress/config.mts
- Update README.md and README.zh-CN.md adapter tables
- Add pixiv to download support tables in both READMEs

Completes the documentation checklist for the pixiv adapter PR.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(pixiv): address code review findings

- Use CommandExecutionError instead of raw Error for HTTP failures
- Add page.goto() before page.evaluate() to establish browser context
- Fix search keyword double-encoding in URL construction
- Fix ranking.yaml using rating_count instead of illust_bookmark_count
- Throw on batch detail fetch failure instead of silent empty return
- Add beforeEach mock reset in download tests
- Add novels column to user.yaml output

Ensures pixiv adapter follows upstream CliError conventions and handles
edge cases correctly before submitting to upstream.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(pixiv): improve download description in READMEs

- Replace technical Referer header detail with user-facing description
- Describe what users care about: original quality and multi-page support

Technical details belong in code comments, not user-facing docs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(pixiv): expand usage examples with all options

- Add ranking mode examples including R18 variants
- Add search filter examples (mode, order, pagination)
- Organize examples by command category for readability

Users need to know available options without reading source code.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(pixiv): address second round of CodeRabbit review comments

- Validate illust-id is numeric to prevent path traversal
- Move URL parsing inside per-item try block for graceful error handling
- Add auth error handling for batch detail request (consistent with step 1)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(pixiv): extract shared pixivFetch helper, add input validation & batch support

- Create utils.ts with pixivFetch() for unified navigate + fetch + error handling
- Refactor search.ts, illusts.ts, download.ts to use pixivFetch (DRY)
- Add user-id/illust-id numeric validation in TS adapters
- Add batch pagination in illusts.ts for limit > 48 (Pixiv server limit)
- Add comment explaining Pixiv search API dual keyword requirement
- Update tests: new invalid-ID test cases, aligned mock format with pixivFetch

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-25 12:52:21 +08:00
AstroHan 3a7a5e135b fix(grok): preserve conversation across repeated ask calls (#332)
* fix(grok): preserve conversation across repeated ask calls (#330)

The adapter unconditionally navigated to grok.com/ on every invocation,
destroying the existing conversation URL even when --new was not passed.
Since the browser daemon already reuses the same Chrome tab, skipping
navigation lets the tab stay on the current chat thread.

- Only navigate to grok.com/ when --new is true or tab is not on grok.com
- Add tryStartFreshChat to the default path's --new branch (was dead code)
- Add isOnGrok helper with hostname-based domain matching
- Add unit tests for isOnGrok

* test(grok): add adapter to vitest project config

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-24 19:19:40 +08:00
Piotr Yordanov 8869d3b457 feat(linkedin): add timeline feed command (#342)
* feat(linkedin): add timeline feed command

* test(linkedin): add timeline adapter unit tests

Add shape tests and utility function tests for the new timeline command.
Include linkedin in the vitest adapter project config.

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-24 15:54:51 +08:00
jakevin 1b3f74cd6a test: focus adapter coverage on four priority sites (#339) 2026-03-24 12:03:26 +08:00
jackwener e0395ce5ed test: make Vitest project order explicit
Add explicit group ordering for Vitest projects so unit tests run before e2e tests, while keeping the e2e ordering fix from PR #38.\n\nCo-authored-by: RbBtSn0w <hamiltonsnow@gmail.com>
2026-03-17 17:46:03 +08:00
jackwener fbec2f6f5d feat(snapshot): filter contentinfo subtrees, bilibili ad URLs, boilerplate buttons
- Add contentinfo to subtree-level noise filtering (biggest single win)
  - Reuters: 51% → 62%, Google: 57% → 70%, Netflix: 48% → 60%
- Add cm.bilibili.com/cm/api/fees/ ad URL pattern
- Add 广告 keyword to ad detection
- Add back-to-top / 回到顶部 boilerplate button filtering
- Unify ad/boilerplate/contentinfo into single subtree-skip mechanism
- Add vitest config and comprehensive test suite (33 tests)
- Fixture tests skip gracefully when snapshot files are absent

Bump to v0.7.1
2026-03-17 01:26:49 +08:00
jackwener 7eec7ce89f fix: restore tests/ in vitest include for CI compatibility
vitest run tests/e2e/ intersects the CLI path with include patterns,
so tests/ must be in the include glob for CI to find test files.
2026-03-16 17:37:48 +08:00
jackwener 0fd9adf0f5 feat: add 8 new sites — bbc, weibo, boss, yahoo-finance, reuters, smzdm, ctrip, youtube
New adapters ported from bb-sites:
- bbc/news: BBC News RSS headlines (public API)
- weibo/hot: 微博热搜 (browser)
- boss/search: BOSS直聘搜索职位 (browser)
- yahoo-finance/quote: 股票行情 (browser)
- reuters/search: 路透社新闻搜索 (browser)
- smzdm/search: 什么值得买搜索好价 (browser)
- ctrip/search: 携程旅行搜索 (browser)
- youtube/search: YouTube视频搜索 (browser)

Also: build script adds clean-yaml step, docs updated to 28 commands / 16 sites
2026-03-15 08:46:13 +08:00