Commit Graph

448 Commits

Author SHA1 Message Date
yin1991 f8bf66390d fix(xiaohongshu): improve search login-wall handling and detail output (#298)
* fix(xiaohongshu): improve search login-wall handling and detail output

* fix(xiaohongshu/search): keep login-wall detection & URL improvements, remove serial per-note enrichment

- Detect login wall and throw a clear error message (from original PR)
- Preserve search_result/ URL with xsec_token instead of degrading to /explore/<id>
- Add author_url to results
- Remove readNoteDetail() + sequential page.goto() per note (caused 60s+ delays
  for default limit=20 with 3s wait each)
- Simplify and unify DOM extraction logic (remove unused fallback anchor scan)
- Update tests: cover login-wall, URL preservation (assert single goto), and limit/filter

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-23 17:34:23 +08:00
jicaiji1-max 22f5c7ade0 fix: ensure standard PATH is available for external CLIs (#285)
Some environments (GUI apps, cron, IDE terminals) launch with a minimal
PATH that excludes standard directories like /usr/local/bin and /usr/sbin.
This causes external CLIs to fail when they try to run system commands
(e.g. sysctl).

Fix by ensuring standard system paths exist in process.env.PATH at
startup. This is a one-time fix that benefits ALL child processes —
isBinaryInstalled(), installExternalCli(), daemon spawn, etc. — without
needing per-call env patching.

Fixes #284

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-23 15:59:57 +08:00
jakevin 8ab0cd2a50 Revert "feat: add xianyu-cli as external CLI (#292)" (#295)
This reverts commit d4b06be049.
2026-03-23 15:45:09 +08:00
donquijote2557-web d4b06be049 feat: add xianyu-cli as external CLI (#292)
Add xianyu (闲鱼/Goofish) CLI tool as an external CLI integration.

Features: search, messaging, agent-flow auto-pricing pipeline,
QR code login, WebSocket real-time messaging with auto-reconnect.

Repo: https://github.com/Donquijote-coder/xianyu-cli

Co-authored-by: donquijote2557-web <donquijote2557-web@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 15:40:39 +08:00
AstroHan 7073645e30 docs: add gws to External CLI Hub table in README (#286)
* docs: add gws to External CLI Hub table in README

The Google Workspace CLI (gws) was registered in external-clis.yaml
but missing from the README table. Closes #120.

* docs: add gws to Chinese README External CLI Hub table

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-23 15:36:46 +08:00
jakevin 3a21be624e fix(xiaohongshu): scope image selector to avoid downloading avatars (#293)
Narrow '#noteContainer img[src*="xhscdn"]' to
'#noteContainer .media-container img[src*="xhscdn"]'
to exclude user avatars and sidebar icons from downloads.

Closes #281
2026-03-23 15:29:43 +08:00
AstroHan 127a974ecd feat(hackernews): add new, best, ask, show, jobs, search, user commands (#290)
Expand HackerNews from 1 command to 8, covering all major HN use cases.
All YAML adapters, strategy: public, browser: false.

- new/best/ask/show/jobs: Firebase API list endpoints with deleted/dead filtering
- search: Algolia API with query + sort (relevance/date)
- user: Firebase user profile with date formatting
- top.yaml: add filter for deleted/dead items + dynamic pre-fetch limit
- E2E tests for all 7 new commands
- Update README, README.zh-CN, adapter docs

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-23 15:13:47 +08:00
jakevin fcb5a9d409 docs: sync adapter lists with codebase (#291)
- Add 9 missing adapters: facebook, google, instagram, tiktok, lobsters,
  medium, sinablog, substack, doubao-app
- Add missing xiaohongshu publish command
- Fix grok mode: Desktop → Browser
- Fix boss commands in docs/adapters/index.md (was incomplete)
- Add doubao-app to Desktop Adapters table
2026-03-23 15:03:23 +08:00
jakevin 66c4b841f2 feat(doubao-app): add Doubao AI desktop app CLI adapter (#289)
- Commands: status, send, read, new, ask, screenshot, dump
- Uses Strategy.UI for desktop CDP connection
- Shared common.ts with selectors and evaluate script builders
- Requires Doubao launched with --remote-debugging-port=9226
2026-03-23 14:52:58 +08:00
jakevin 2a52906ed2 fix: add turndown dependency to package.json (#288)
turndown and @types/turndown were used in article-download.ts and
zhihu/download.test.ts but never declared in package.json, causing
CI failures on fresh npm ci installs.
2026-03-23 14:46:08 +08:00
helloimcx 9cdc1274b2 feat: add doubao browser adapter (#277) 2026-03-23 12:34:27 +08:00
stometaverse a6d993f37f feat(xiaohongshu): add publish command for 图文 note automation (#276)
Adds `opencli xiaohongshu publish` which automates posting a 图文 (image+text)
note via the creator center UI (creator.xiaohongshu.com/publish/publish).

Features:
- --title (required, max 20 chars)
- positional content argument
- --images comma-separated local file paths (jpg/png/gif/webp, max 9)
- --topics comma-separated hashtag names (without #)
- --draft flag to save as draft instead of publishing

Image upload uses DataTransfer injection into the file input element, converting
local files to base64 in Node.js and creating File blobs in the browser context.
Text fields use document.execCommand('insertText') for contenteditable editors.
Graceful debug screenshots on failure (/tmp/xhs_publish_*_debug.png).

Requires: opencli browser session logged into creator.xiaohongshu.com.
2026-03-23 12:26:02 +08:00
jakevin b7c6c02370 feat: add weixin article download adapter & abstract download helpers (#280)
- New: src/clis/weixin/download.ts — WeChat article to Markdown adapter
- New: src/download/article-download.ts — shared article download helper
  (TurndownService, image localization, frontmatter, customizable labels)
- New: src/download/media-download.ts — shared media download helper
  (batch download, ProgressTracker, yt-dlp routing, auto cookie export)
- Refactor: migrate zhihu/download to use downloadArticle()
- Refactor: migrate xiaohongshu/download to use downloadMedia()
- Refactor: migrate twitter/download to use downloadMedia()
- Refactor: migrate bilibili/download to use downloadMedia()
- Docs: add weixin to README, README.zh-CN, download docs, adapter docs
2026-03-23 12:11:43 +08:00
jakevin 722c180a0a docs: clarify release follow-up steps (#279) 2026-03-23 11:47:26 +08:00
jakevin 788126198b chore: release v1.3.1 (#273) v1.3.1 2026-03-23 00:33:04 +08:00
jakevin 98ecfab8ad chore: bump version to 1.3.0 (#272)
* chore: bump version to 1.3.0

* chore: bump version to 1.3.0
2026-03-23 00:27:55 +08:00
jakevin 4b976da04f perf: smart page settle via DOM stability detection (#271)
Replace fixed settleMs sleep in goto() with MutationObserver-based DOM
stability detection. The page is considered settled when no DOM mutations
occur for quietMs (default 500ms), with settleMs as a hard timeout cap.

Changes:
- Add waitForDomStableJs() shared helper to dom-helpers.ts
- Update Page.goto() and CDPPage.goto() to use smart settle
- No IPage interface changes (implementation detail only)

Key improvements over naive approach:
- Timer starts AFTER MutationObserver.observe() to avoid race condition
- Falls back to sleep(maxMs) if document.body is not available
- Monitors attributes in addition to childList/subtree
- quietMs defaults to 500ms (conservative) for async request buffering
2026-03-23 00:24:09 +08:00
Zhang ShengYan 3bedaccc25 docs: refresh testing guide (#223)
* docs: refresh testing guide

* docs: include missing twitter/timeline.test.ts in test inventory

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-23 00:11:44 +08:00
jakevin 40bd11dfbe fix(daemon): harden security against browser CSRF attacks (#268) (#270)
- Add Origin header check: reject HTTP/WS from non chrome-extension:// origins
- Require X-OpenCLI custom header on all HTTP requests
- Remove Access-Control-Allow-Origin: * from all responses
- Add WebSocket verifyClient to reject malicious connections at upgrade
- Add 1MB body size limit to prevent OOM
- Update file header with security model documentation

Closes #268
2026-03-22 23:56:39 +08:00
AlexYue 9a77dba139 ci: trigger website rebuild on release (#269) 2026-03-22 23:46:48 +08:00
jakevin 49c2dc7426 docs: remove all --live references (now default behavior) (#267) 2026-03-22 23:03:53 +08:00
jakevin e4a13cb6f0 fix: remove duplicate horizontal rules and extra blank lines in READMEs (#266) 2026-03-22 22:58:54 +08:00
jakevin 637161f0ab fix: update doctor tests for auto-start daemon and --no-live default (#265)
- Fix skip message assertion: 'skipped (--no-live)' instead of old text
- Fix auto-start test: mock checkDaemonStatus for both initial and final calls
2026-03-22 22:57:33 +08:00
jakevin a778f617ca chore: update package-lock.json (#264) 2026-03-22 22:55:36 +08:00
jakevin b4a8089224 refactor: doctor defaults to live mode, remove setup command entirely (#263)
- Remove setup command completely (no backward compat needed)
- Doctor now runs live connectivity test by default
- Add --no-live flag to skip if needed
- Update SKILL.md docs
2026-03-22 22:51:28 +08:00
jakevin 428b831f85 refactor: deprecate opencli setup, enhance doctor with daemon auto-start (#262)
- Delete setup.ts (fully redundant with doctor)
- opencli setup now prints deprecation warning and delegates to doctor
- doctor auto-starts daemon if not running (no more false 'not connected')
- Update all doc references (README, SKILL.md, docs/)
2026-03-22 22:49:14 +08:00
jakevin 7ebe8134cc docs: clean up READMEs - remove redundant sections (#261)
Removed from both EN/CN READMEs:
- Table of Contents (GitHub auto-generates TOC)
- Method 2: Load from npm Package (keep recommended + dev only)
- Bloomberg detailed note (too specific for README)
- Pipeline Step YAML example (developer-internal)
- Releasing New Versions (belongs in CONTRIBUTING.md)

EN README only:
- Simplified Testing section to one-liner + link to TESTING.md
2026-03-22 22:26:49 +08:00
jakevin c44bc62b60 chore: code cleanup + extension conflict troubleshooting (#260)
Cleanup:
- Remove redundant double-retry in resolveTabId (was retrying data: URI
  with the same data: URI)
- Fix stale comment (30s → 120s idle timeout)
- Remove verbose debug logging in resolveTabId
- Built extension is now smaller (16.66kB vs 17.18kB)

Extension conflict:
- Add hint to attach-failed error when chrome-extension:// URL is detected
- Add troubleshooting entry for extension conflicts (e.g. youmind, New Tab
  Override) to both README.md and README.zh-CN.md

Ref: #249
v1.2.6
2026-03-22 22:18:22 +08:00
jakevin 7f57e76485 fix: treat empty tab URL as debuggable (fixes first-run doctor --live failure) (#259)
When a new automation window is created, the initial tab URL may be
empty briefly while Chrome loads the data: URI. isDebuggableUrl('') was
returning false, causing ensureAttached to reject the tab.

Fix: only reject known non-debuggable URLs (chrome://, chrome-extension://).
Empty/undefined URLs are now treated as debuggable since they represent
tabs still loading.

Also adds 200ms delay after window creation to let Chrome populate the
tab URL.
v1.2.5
2026-03-22 22:10:30 +08:00
jakevin e9818c1b41 chore: remove CRX from release pipeline and docs (#258)
CRX files cannot be installed in modern Chrome without Chrome Web Store
publishing. Updated all docs to recommend 'Load unpacked' installation
method only. Added npm package loading method as alternative.

- Removed CRX build step from build-extension.yml workflow
- Removed CRX from artifact upload and release attachment
- Updated README.md, README.zh-CN.md, browser-bridge docs (en/zh)
- Added 'Load from npm package' as installation method
2026-03-22 22:07:33 +08:00
jakevin 3e91876d13 fix: replace all about:blank with data: URI to prevent New Tab Override interception (#257)
Root cause: getAutomationWindow and resolveTabId used about:blank which
New Tab Override extensions intercept immediately, replacing it with
chrome-extension:// URLs that cannot be debugged.

Changes:
- Window creation: about:blank → data:text/html
- reuseTab fallback: about:blank → data:text/html
- newTab handler: about:blank → data:text/html
- Added diagnostic logging to resolveTabId for debugging
- Synced extension version to 1.2.4

Ref: #249
v1.2.4
2026-03-22 22:04:25 +08:00
jakevin 7c02588105 chore: bump version to 1.2.3 (#256) v1.2.3 2026-03-22 21:48:17 +08:00
jakevin 112fdefa8d fix: harden resolveTabId against New Tab Override extension interception (#255)
resolveTabId's reuseTab path now verifies the URL is actually debuggable
after navigating to about:blank. If a New Tab Override extension intercepts
it (setting it back to chrome-extension://), falls back to a data: URI,
then creates a fresh tab as last resort.

This fixes the persistent 'attach failed: Cannot access chrome-extension://'
error for users with New Tab Override extensions installed.

Ref: #249
2026-03-22 21:47:44 +08:00
jakevin e077ad2336 chore: bump version to 1.2.2 (#254) v1.2.2 2026-03-22 21:27:03 +08:00
jakevin 81384ede00 chore: bump version to 1.2.1 (#252) v1.2.1 2026-03-22 21:24:42 +08:00
jakevin 71b2c3961b fix: harden browser automation pipeline (resolves #249) (#251)
- resolveTabId: validate URL even for explicit tabId, fall through to
  auto-resolve when tab is not debuggable or has been closed
- handleNavigate: wait for URL change before checking 'complete' status
  to avoid race condition with stale about:blank
- ensureAttached: pre-check tab URL, verify cached attach with probe,
  invalidate cache on URL change via onUpdated listener
- daemon-client: recognize transient extension errors (disconnected,
  attach failed) as retryable with 1500ms delay; fresh command ID per attempt
- pipeline executor: add per-step retry for browser steps (up to 2 retries
  on transient errors); cleanup automation window on pipeline failure
- page.ts: selectTab/newTab/closeTab properly update/invalidate _tabId
- daemon.ts: add WebSocket ping/pong heartbeat (15s interval, 2-miss disconnect)
- Increase automation window idle timeout from 30s to 120s
- Fix timeout param edge cases in BrowserBridge._ensureDaemon
- Remove unused chalk import; fix trailing import placement

Closes #249
2026-03-22 21:23:21 +08:00
jakevin b3b9892836 docs: add star history chart (#246) 2026-03-22 19:22:41 +08:00
jakevin 520622ac75 ci: update GitHub Actions runtime versions (#245) 2026-03-22 19:02:21 +08:00
jakevin 2d1b8c1e76 chore: prepare v1.2 release (#244) v1.2.0 2026-03-22 18:40:32 +08:00
ykfnxx 70651d3ba8 feat(douban): add movie adapter with search, top250, subject, marks, reviews commands (#239)
* feat(douban): add movie adapter with search, top250, subject, marks, reviews commands

- search: search movies by keyword
- top250: get top 250 movies
- subject: get movie details by id
- marks: export personal viewing marks
- reviews: export personal movie reviews

* review: resolve douban adapter blockers

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-22 17:57:34 +08:00
plat1ko 9696db9ed4 feat: make primary args positional across all CLIs (#242)
* feat: make primary args positional across all CLIs

Convert primary arguments from named options (--arg) to positional
arguments for a more natural CLI experience.

Affected sites: antigravity, bilibili, boss, chaoxing, coupang, grok,
hf, instagram, jike, jimeng, linkedin, linux-do, tiktok, twitter,
xiaohongshu, youtube

Also adds Arg Design Convention to CONTRIBUTING.md and earnings-date
to xueqiu command list in READMEs.

Usage examples:
  opencli xueqiu search '茅台'       (was: --query '茅台')
  opencli twitter followers elonmusk  (was: --user elonmusk)
  opencli bilibili download BV1xxx    (was: --bvid BV1xxx)

* review: keep config args named in positional cleanup

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-22 17:04:56 +08:00
jakevin c76f86c9cb refactor: fail fast on invalid pipeline steps (#237) 2026-03-22 14:53:57 +08:00
jakevin 4cd0409ded fix: harden twitter timeline review findings (#236) 2026-03-22 14:30:03 +08:00
VK ea113a6471 feat(devto): add devto adapter (#234)
* feat(devto): add devto adapter

* refactor(devto): improve adapters to match project conventions

- Make tag/username args positional for natural CLI usage:
  opencli devto tag javascript (instead of --tag javascript)
  opencli devto user ben (instead of --username ben)
- Add rank field (index + 1) matching hackernews/lobsters pattern
- Add tags field from tag_list for richer output
- Remove redundant author column from user command (already filtering by user)
- Use type: str (project convention) instead of type: string
- Increase default limit from 10 to 20 (matching other adapters)
- Update docs with positional arg examples

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-22 14:18:47 +08:00
AstroHan a439286398 docs(plugin): add juejin plugin to example plugins (#207) 2026-03-22 14:17:12 +08:00
AstroHan 1d56dd77a8 fix(wikipedia): fix search arg name + add random and trending commands (#231)
* fix(wikipedia): fix search arg name + add random and trending commands

- fix: search.ts referenced `args.keyword` but the argument is defined
  as `query`, causing the search term to always be undefined
- feat: add `random` command (random article summary via REST API)
- feat: add `trending` command (most-read articles, yesterday's data)

All commands are PUBLIC strategy, no browser required, reuse wikiFetch.

* refactor(wikipedia): extract shared types + add docs for random/trending

- Extract WikiSummary, WikiMostReadArticle types to utils.ts
- Extract EXTRACT_MAX_LEN/DESC_MAX_LEN constants
- Add formatSummaryRow() helper to eliminate duplicate mapping in
  summary.ts and random.ts
- Update docs with random and trending command examples

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-22 14:14:21 +08:00
AstroHan e98cf756e9 feat(twitter): add --type flag to timeline command (#83) (#232)
Support switching between For You (algorithmic) and Following
(chronological) timelines via `--type for-you|following`.

Both endpoints share the same response structure; only the GraphQL
endpoint name and queryId differ. QueryId is resolved dynamically
from fa0311/twitter-openapi with a hardcoded fallback, and validated
against /^[A-Za-z0-9_-]+$/ to prevent injection from upstream.
2026-03-22 12:11:21 +08:00
Zhang ShengYan 387aa0d6e5 fix: resolve inconsistent doctor --live report (fix #121) (#224)
* fix(doctor): refresh status after live check to resolve #121

* refactor: reorder live check before status read for natural consistency

Instead of calling checkDaemonStatus() twice (before and after the
connectivity check), reorder so that the live connectivity check runs
first, then read daemon status only once. This:
- Eliminates redundant checkDaemonStatus() call
- Naturally avoids the timing inconsistency (fixes #121)
- Also fixes the sessions query using stale status
- Simplifies test assertions to avoid over-coupling to exact wording

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-22 10:59:59 +08:00
jakevin 1ecac25df5 fix: correct SKILL.md github reference and add missing adapter docs (#230)
- SKILL.md: replace non-existent 'opencli github search' with correct
  'opencli gh' external CLI passthrough examples
- SKILL.md: remove 'github search' from public API commands list
- Add missing docs for douban, sinablog, substack adapters (fixes
  doc-check CI failure: 47/50 → 50/50)
- Add new adapter pages to VitePress sidebar config
2026-03-22 10:54:03 +08:00
jakevin 9921e5d696 remove broken desktop adapters (#221) 2026-03-22 04:14:07 +08:00