59 Commits

Author SHA1 Message Date
jackwener 92408e7ee9 fix: improve E2E testing infrastructure
- buildMcpArgs: use CI env var instead of token detection (preserves local behavior)
- browser.test.ts: align tests with CI-based mode detection
- api-health.test.ts: remove duplicate xueqiu entry
- browser-public.test.ts: unify BBC test with tryBrowserCommand
- vitest.config.ts: default to unit tests only (E2E via explicit path)
- ci.yml: remove unnecessary needs:build for unit-test, use composite action
- e2e-headed.yml: use composite action, increase timeout to 20min
- Add .github/actions/setup-chrome composite action for shared CI steps
2026-03-16 17:33:25 +08:00
AlexYue 257e9ec4f8 feat: add E2E testing infrastructure with real Chrome in CI (#1)
# feat: add E2E testing infrastructure with real Chrome in CI

## What

Establish a comprehensive E2E testing framework for opencli using **real Chrome + xvfb virtual display** in GitHub Actions CI.

## Changes

### E2E Test Suite (~52 test cases)
- `public-commands.test.ts` — Public API commands (hackernews, v2ex)
- `browser-public.test.ts` — Browser commands for public data across all 18 sites (21 tests)
- `browser-auth.test.ts` — Graceful failure verification for login-required commands (14 tests)
- `management.test.ts` — Full coverage of management commands (list/validate/verify/version/help)
- `output-formats.test.ts` — Output format validation (json/yaml/csv/md)
- `smoke/api-health.test.ts` — Scheduled API health checks

### Auto-detect Browser Mode
- `buildMcpArgs` automatically selects mode based on `PLAYWRIGHT_MCP_EXTENSION_TOKEN`:
  - Token present → `--extension` (local user, connects to logged-in Chrome)
  - Token absent → standalone (CI launches its own browser)
- No extra environment variables needed

### CI Pipeline
- `e2e-headed.yml` — Real Chrome via `browser-actions/setup-chrome` + `xvfb-run` in headed mode
- `ci.yml` — build + unit-test (2 shards) + smoke-test (scheduled/manual)
- Browser commands that return empty data due to geo-blocking or bot detection gracefully warn + pass without blocking CI

### Documentation
- New `TESTING.md` — Architecture, coverage, local setup, how to add tests, CI explanation
- Updated `README.md` — Added Testing section with quick-start commands
2026-03-16 16:22:00 +08:00
jackwener 6f629260e7 feat: add interactive 'opencli setup' command with TUI checkbox
New zero-dependency TUI checkbox component (src/tui.ts) with:
  - ↑↓/jk navigation, Space toggle, Tab toggle+move
  - 'a' toggle all, Enter confirm, q/Esc cancel

New 'opencli setup' command (src/setup.ts) that:
  - Auto-discovers token from Chrome LevelDB extension
  - Shows interactive multi-select for config files
  - Displays tool names (Codex, Cursor, Claude Code, etc.)
  - Color-coded status (green=ok, yellow=mismatch, red=missing)
  - Applies changes only to selected files
2026-03-16 13:53:03 +08:00
jackwener 1e0b83bb84 feat(doctor): add Antigravity and Gemini CLI config paths
Add ~/.gemini/settings.json (Gemini CLI) and
~/.gemini/antigravity/mcp_config.json (Antigravity) to the
default MCP config scan list.
2026-03-16 13:45:02 +08:00
jackwener 1ff184e24d feat(doctor): add Claude Code and project .mcp.json config paths
Add ~/.claude.json (Claude Code user-scoped MCP config) and
.mcp.json (Claude Code project-scoped MCP config) to the
default config scan list.
2026-03-16 13:43:03 +08:00
jackwener 8e84145ccc feat(doctor): auto-discover extension token from Chrome LevelDB
Scan Chrome/Edge/Chromium localStorage LevelDB files to extract the
Playwright MCP Bridge auth-token directly from the extension's storage.
Uses a fast 'strings | grep' shell pipeline (~200ms) on macOS/Linux
with a pure-Node fallback for Windows.

The discovered token is now shown in 'opencli doctor' output and takes
priority as the recommended token when using '--fix'.
2026-03-16 13:38:48 +08:00
jackwener c77c8a8e3a feat: add Coupang search and add-to-cart adapters
Add browser-backed Coupang adapters for search and add-to-cart workflows.
- coupang search: multi-strategy data collection (API/JSON-LD/bootstrap/DOM), structured fields (price, rating, rocket, delivery), pagination and rocket filter support
- coupang add-to-cart: logged-in browser session reuse, stops before checkout
- coupang.ts: comprehensive data normalization layer with badge/rocket/delivery mapping
- browser-tab.ts: withTemporaryTab utility for isolated tab operations
- Unit tests for core normalization functions

Co-authored-by: CodeBBakGoSu <127713112+CodeBBakGoSu@users.noreply.github.com>
2026-03-16 13:30:42 +08:00
jackwener 34a9bff2b3 refactor: eliminate code duplication, improve type safety, add tests
- NEW: src/interceptor.ts — unified XHR/Fetch interceptor (was duplicated 3x)
- NEW: src/version.ts — centralized PKG_VERSION (was duplicated 2x)
- NEW: src/constants.ts — shared VOLATILE_PARAMS, FIELD_ROLES etc.
- NEW: src/engine.test.ts, src/registry.test.ts — 14 new unit tests

- browser.ts: use shared normalizeEval, interceptor, withTimeoutMs, PKG_VERSION
- intercept.ts, tap.ts: use shared interceptor generators
- cascade.ts: extract shared buildFetchProbeJs (90% dedup)
- engine.ts: use InternalCliCommand (no more 'as any' casts)
- executor.ts: remove all 15 'as StepHandler' type casts
- runtime.ts: add withTimeoutMs, IBrowserFactory interface
- registry.ts: add InternalCliCommand type for internal fields
- validate.ts: add pipeline step name validation
- output.ts: remove 'null as any' and '.filter(() => true)' hacks
- explore.ts, synthesize.ts: use shared constants
- docs: fix V2EX commands (3→6), SKILL.md version, verify example

Tests: 88 passed (was 74), tsc --noEmit: 0 errors
2026-03-16 13:22:12 +08:00
jackwener b0c51ddf19 fix: use Playwright MCP --executable-path flag (kebab-case)
Extract buildMcpArgs() helper and fix --executablePath → --executable-path
to match the Playwright MCP CLI's expected flag format.

Closes #16

Co-authored-by: KasumiChen <KasumiChen@users.noreply.github.com>
2026-03-16 12:58:18 +08:00
jackwener 7ee0c9c96b fix: tab cleanup regex to match actual Playwright MCP tab format
The tab list from Playwright MCP uses '- N: (current) [title](url)' format,
but extractTabEntries only matched 'Tab N ...' format. This caused
_initialTabIdentities to always be empty, so tabs were never cleaned up.

Now supports both formats.
2026-03-16 12:54:44 +08:00
jackwener 95fb841fc7 refactor: remove all CDP code, keep extension-only mode
- Remove CDP auto-discovery (discoverChromeEndpoint, isCdpApiAvailable, isPortReachable)
- Remove CDP readiness probe and autoAllowCdpDialog
- Remove forceExtension plumbing from registry, runtime, main, and adapters
- Remove CDP diagnostics from doctor (extractTokenViaCdp, remoteDebugging fields)
- Simplify connect() to always use --extension mode
- Remove CDP env vars (OPENCLI_USE_CDP, OPENCLI_CDP_ENDPOINT, OPENCLI_FORCE_EXTENSION)
- Update README.md, README.zh-CN.md, SKILL.md to remove CDP sections
- Update tests to remove CDP-related assertions

Current state preserved in extension-and-cdp branch.
2026-03-16 12:44:09 +08:00
jackwener 67d50191af fix: CDP mode hanging when using chrome://inspect remote debugging
- Add isCdpApiAvailable() to verify CDP HTTP JSON API before using endpoint
- Chrome's chrome://inspect#remote-debugging writes DevToolsActivePort but
  its CDP endpoint is incompatible with Playwright connectOverCDP (init
  succeeds but all tool calls hang silently)
- Add CDP readiness probe in connect() to catch unresponsive endpoints
- Skip tab cleanup in close() for CDP mode (no bridge tabs to clean)
- Add PlaywrightMCPMode tracking for extension vs CDP lifecycle policy
2026-03-16 02:28:28 +08:00
jackwener f8491f01ae fix: add 5s timeout for initial tabs fetch during CDP connect
CDP mode on @playwright/mcp v0.0.68 hangs on browser_tabs calls.
This timeout prevents the entire connect() from blocking for 30s.
2026-03-16 01:51:22 +08:00
jackwener db8194337b Improve browser session lifecycle handling 2026-03-15 23:31:52 +08:00
jackwener cef50dc8bf feat(doctor): print cli version in doctor report 2026-03-15 21:51:21 +08:00
jackwener 1aaafed2db feat(twitter): implement post, reply, delete, following, followers and notifications adapters 2026-03-15 21:42:48 +08:00
jackwener ee6669f45d feat(core): support YAML format output and optimize manifest builder 2026-03-15 21:05:36 +08:00
jackwener eb9050fadd feat(v2ex): add daily, me, and notifications session adapters 2026-03-15 20:59:39 +08:00
jackwener 53cd006a23 fix(boss): resolve limit=60 infinite loop and global CLI hangs
- search.ts: Added deduplication break condition so if the API returns duplicate pages, it stops instead of looping forever.
- search.ts: Removed unsafe Promise.race around page.goto which caused 'context closed' errors.
- browser.ts: Added robust static cleanup hooks to guarantee process and MCP lock destruction on crashed or ctrl+c'd runs.
2026-03-15 20:33:22 +08:00
jackwener c9710a656b feat(boss): add pagination loop to support fetching more than 15 results
- The API endpoint only returns 15 results per page.
- Added a loop to fetch multiple pages until the requested limit is reached.
- Added random 1-3 second delays between page fetches to avoid rate limits.
2026-03-15 20:19:30 +08:00
jackwener 00f0ac9133 fix(boss): prioritize fresh browser cookie extraction over boss-cli cache 2026-03-15 19:58:46 +08:00
jackwener 64d655ae0d feat(boss): rewrite search adapter to use boss-cli native cookie extraction
- BOSS Zhipin aggressively detects CDP connections and kills pages
- Completely rewrote boss/search.ts to use browser: false (bypassing Playwright)
- Implemented native Python subprocess to extract cookies via browser-cookie3
- Falls back to boss-cli ~/.config/boss-cli/credential.json if available
- Uses direct Node.js fetch with anti-detection headers to avoid bans
2026-03-15 19:55:46 +08:00
jackwener a6e9c3b611 feat(browser): enhance CDP auto-discovery with port probe & opt-in priority
- Add TCP port reachability check to handle stale DevToolsActivePort files
- Make discoverChromeEndpoint() async with port probe before returning
- Change connection priority: extension mode is default, CDP is opt-in
  - OPENCLI_CDP_ENDPOINT: explicit CDP endpoint
  - OPENCLI_USE_CDP=1: enable DevToolsActivePort auto-discovery
  - Default: --extension mode (Playwright MCP Bridge)
- Add verbose logging for CDP discovery when -v flag is used
- Fix indentation inconsistency in connect() method
- Update README.md, README.zh-CN.md, SKILL.md with new connection docs
2026-03-15 19:44:27 +08:00
jackwener da155e96f0 feat: add forceExtension to bypass CDP detection for anti-bot sites
- Added forceExtension flag to CliCommand, CliOptions, browserSession, and PlaywrightMCP.connect()
- BOSS Zhipin detects CDP mode and drops connections; marked boss/search with forceExtension: true
- Also supports OPENCLI_FORCE_EXTENSION=1 env var for global override
- Fixed async discoverChromeEndpoint() placement outside Promise constructor
- Enhanced boss search with city names, experience, degree, salary, etc.
2026-03-15 19:39:39 +08:00
backtime1993 c20eaee393 feat(browser): Chrome 144+ auto-discovery via DevToolsActivePort
Chrome 144+ allows enabling remote debugging from chrome://inspect
without any command-line flags (--remote-debugging-port). Chrome writes
the active port and browser GUID to a DevToolsActivePort file in the
user data directory.

This change reads that file to auto-discover the CDP WebSocket endpoint,
eliminating the need for the Playwright MCP Bridge browser extension in
most cases.

Connection priority:
1. OPENCLI_CDP_ENDPOINT env var (manual override)
2. DevToolsActivePort auto-discovery (Chrome/Edge, cross-platform)
3. --extension mode fallback (original behavior)

Supports Windows, macOS, and Linux with Chrome, Edge, and Chromium.

Co-authored-by: kensei <backtime1993@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 19:34:14 +08:00
jackwener 15d8c45b95 feat(boss): add city name mapping, experience/degree/salary filters, page support
- 30+ Chinese city names → code mapping (e.g. --city 杭州)
- Experience filter: 应届/1-3年/3-5年/5-10年/10年以上
- Degree filter: 大专/本科/硕士/博士
- Salary filter: 3K以下/3-5K/5-10K/10-20K/20-50K/50K以上
- Page number support for pagination
- Area now shows city·district·business district
- Skills column added to output
2026-03-15 19:31:05 +08:00
jackwener 2d638e3a0a fix(manifest): extract TS adapter args at build time for proper CLI registration
The manifest builder now statically parses TS source files to extract
args, description, columns, domain, and strategy metadata. Previously
TS adapters had empty args in the manifest, causing commander to reject
their options (e.g. --keyword) before the lazy module could load.

Also includes the xiaohongshu search DOM fallback fix from earlier.

Co-authored-by: jack4world <jack4world@users.noreply.github.com>
2026-03-15 18:59:15 +08:00
jackwener b3850d9c53 fix(xiaohongshu): switch search from broken XHR interception to DOM extraction (Fixes #10)
Co-authored-by: jack4world <jack4world@users.noreply.github.com>
2026-03-15 18:55:29 +08:00
jackwener f8ee50b776 feat(bilibili): add following adapter — get user's followed accounts
Discovered via Playwright MCP Bridge workflow (CLI-CREATOR.md):
 - Navigate to space.bilibili.com/{uid}/fans/follow
 - Sniff network: GET /x/relation/followings (Cookie-tier, no Wbi needed)
 - Verify API response: total=1342, fields: mid/uname/sign

Also fixed .d.ts loading bug in build-manifest.ts and engine.ts:
 - build-manifest now scans .js files in dist/ (not .ts)
 - engine.ts skips .d.js declaration stubs
2026-03-15 18:38:34 +08:00
jackwener ad9e9605fc feat: extreme architectural optimizations
Phase 1: Smart autoScroll with MutationObserver (single IPC, instant DOM change detection)
Phase 2: Build-time CLI manifest compiler + lazy-loading engine (instant cold start)
Phase 3: Batch IPC fetch (N browser fetches in single evaluate() call)
2026-03-15 18:25:58 +08:00
jackwener 4fc4c4b92f fix: apply remaining code review fixes to subtitle.ts and explore.ts
- subtitle.ts: URL injection fix (JSON.stringify), --lang option,
  debug console.error removal, comment numbering
- explore.ts: clickLabels targeted clicking, empty-value penalty
  moved to scoreEndpoint(), removed console.log debug output
- Deleted stale test_subtitle.ts
2026-03-15 18:12:40 +08:00
jackwener a47d687513 perf: extreme architectural optimizations
Phase 1: Smart autoScroll with MutationObserver (single IPC, instant DOM change detection)
Phase 2: Build-time CLI manifest compiler + lazy-loading engine (instant cold start)
Phase 3: Batch IPC fetch (N browser fetches in single evaluate() call)
2026-03-15 18:12:36 +08:00
jackwener 2dacba8244 fix: code review improvements (#1-#8)
- #1 Fix URL injection in subtitle.ts via JSON.stringify
- #2 Remove debug console.error from production code
- #3 Delete stale test_subtitle.ts
- #4 Add --lang option for multi-language subtitle selection
- #5 Fix duplicate comment numbering (two '// 4.')
- #6 Add clickLabels targeted clicking + --click flag to explore
- #7 Move empty-value penalty into scoreEndpoint() (affects filtering)
- #8 Add cascade request code template to CLI-CREATOR.md
2026-03-15 18:09:59 +08:00
jackwener e64fc65554 feat: add Bilibili subtitle adapter with Wbi signing & explore fuzzing
- New: src/clis/bilibili/subtitle.ts - cascading BVID→CID→subtitle fetch
  using apiGet with automatic Wbi signature (w_rid)
- Enhanced: src/explore.ts - added INTERACT_FUZZ_JS for interactive
  API discovery via simulated clicks (--auto flag)
- Enhanced: src/explore.ts - empty-value score penalty for anti-bot detection
- Fixed: src/main.ts - restored explore command syntax, added --auto option
- Updated: CLI-CREATOR.md - cascading requests & anti-bot sections
2026-03-15 18:03:25 +08:00
jackwener 790c94d644 feat: Add Xueqiu site with 6 new commands and a Twitter bookmarks command, updating documentation. 2026-03-15 16:03:45 +08:00
jackwener aef110a8f8 feat(browser): support OPENCLI_BROWSER_EXECUTABLE_PATH env var to specify custom Playwright MCP browser path (Resolves #8) 2026-03-15 15:54:08 +08:00
jackwener 71f2f3d6e6 fix: integrate community bugfixes for PRs 1, 4, 6 and Issue 7
- Fix zhihu id truncation issue beyond safe integer
- Add json filter to template engine
- Escape evaluate args in browser scripts using json filter
- Resolve module not found bug for index.ts

Co-authored-by: 袁鑫 <eric.yuanxin@gmail.com>
Co-authored-by: zhenxing jiang <jiangzhx@gmail.com>
Co-authored-by: wangruotian <wangruotian@meituan.com>
Co-authored-by: 滕文证 <83223457+tenwz@users.noreply.github.com>
2026-03-15 15:43:33 +08:00
jackwener 829172fc59 feat: introduce browser interception and auto-scroll utilities, enable dynamic loading of TypeScript CLIs, and add a JSON template filter 2026-03-15 15:42:19 +08:00
jackwener 20f7d58dd6 feat: Add new CLIs for Bilibili, Reddit, Twitter, and Xiaohongshu, incorporating an intercept strategy for Twitter. 2026-03-15 15:33:30 +08:00
jackwener 1361fe6c19 feat: release v0.2.0 with all new adapters and fixes 2026-03-15 08:54:35 +08:00
jackwener 46ae0a0aa3 fix: test import path 2026-03-15 08:46:39 +08:00
jackwener 0fd9adf0f5 feat: add 8 new sites — bbc, weibo, boss, yahoo-finance, reuters, smzdm, ctrip, youtube
New adapters ported from bb-sites:
- bbc/news: BBC News RSS headlines (public API)
- weibo/hot: 微博热搜 (browser)
- boss/search: BOSS直聘搜索职位 (browser)
- yahoo-finance/quote: 股票行情 (browser)
- reuters/search: 路透社新闻搜索 (browser)
- smzdm/search: 什么值得买搜索好价 (browser)
- ctrip/search: 携程旅行搜索 (browser)
- youtube/search: YouTube视频搜索 (browser)

Also: build script adds clean-yaml step, docs updated to 28 commands / 16 sites
2026-03-15 08:46:13 +08:00
jackwener 999d340a34 refactor: remove xiaohongshu me, add lessons learned, fix docs
- Remove xiaohongshu/me (profile page async loading too unreliable)
- CLI-CREATOR: add 4 lessons to 常见陷阱 (stale dist YAML, TS IIFE
  format, async polling, YAML vs TS threshold)
- Fix build: clean-yaml before copy to prevent stale overrides
- Fix docs: MCP token is configured via MCP config, not env var
- Update all command lists (README, README.zh-CN, SKILL.md)
2026-03-15 08:10:40 +08:00
jackwener fd94a5d27b fix: auto-close tabs after command, add login requirement reminders
- browser.ts: close all tabs opened during session (not just extension tabs)
  uses _initialTabCount to restore Chrome to pre-command state
- README/README.zh-CN: add ⚠️ login requirement warning with examples
- SKILL.md: add login note and tab auto-close mention
2026-03-15 07:57:12 +08:00
jackwener 9df96db598 refactor: pipeline error propagation, me.yaml→me.ts, docs overhaul
Code:
- pipeline.ts: warn on stderr when step returns {error} object
- xiaohongshu/me.yaml → me.ts (evaluate code belongs in TS)
- index.ts: register xiaohongshu/me.ts

Docs:
- CLI-CREATOR: YAML vs TS decision tree, tap step docs with
  real example, 'explore store first' debugging workflow
- SKILL.md: add tap to Pipeline Steps table, update XHS commands
- README + README.zh-CN: update xiaohongshu command list
2026-03-15 07:51:20 +08:00
jackwener 7bc670ee22 feat(xiaohongshu): add me + notifications, remove broken adapters
- me: personal profile info (navigate + evaluate from Pinia store)
- notifications: browse notifications by type (mentions/likes/connections)
  uses tap pipeline to call getNotification store action
- removed note, comments, user-posts (store action names unverified)
2026-03-15 07:38:53 +08:00
jackwener 9124dcc018 chore: clean up duplicate adapters
- Remove xiaohongshu/search.yaml (keep .ts with XHR interception)
- Remove zhihu/search.ts (keep .yaml, self-contained)
- Clean ~/.opencli/clis/ generated candidates (5 dirs)

24 commands across 8 sites, no duplicates
2026-03-15 07:30:35 +08:00
jackwener 0988fcb334 feat(xiaohongshu): add me, note, comments, user-posts commands
All 4 commands use YAML tap pipeline with Pinia store interception:
- me: personal profile info
- note --id: note detail
- comments --id: note comments
- user-posts --uid: user's published notes
2026-03-15 07:29:42 +08:00
jackwener d35d78d836 docs: polish README and SKILL for v0.1.0
README:
- Cleaner structure, accurate 25 commands / 13 sites
- Correct strategy labels (v2ex=public, not cookie)
- Explore artifacts documentation
- Better formatting and English/Chinese mix

SKILL:
- 5-tier authentication strategy table
- Public API YAML pattern example
- TS escape hatch guidance
- intercept pipeline step docs
- Troubleshooting section
2026-03-15 07:18:17 +08:00
jackwener 4b85ddd678 feat: add xiaohongshu search + v2ex topic + test fixes
- xiaohongshu/search.ts: Pinia store + XHR interception (from bb-sites)
- v2ex/topic.yaml: public API topic details
- Register xiaohongshu/search in index.ts
- 25 commands across 13 sites

Tested: bilibili hot/search ✅, zhihu hot ✅, v2ex hot/topic ✅,
hackernews top ✅, github search ✅
2026-03-15 07:16:03 +08:00