Commit Graph

380 Commits

Author SHA1 Message Date
jackwener eb2c3fdf89 fix: support Chrome Dev and Chrome Beta browser variants
Add Chrome Dev and Chrome Beta profile paths to discoverExtensionToken()
and checkExtensionInstalled() across macOS, Linux, and Windows.

Closes #30
2026-03-17 16:36:19 +08:00
jackwener de962eb5fb feat: support commands completion (#32)
Add full shell tab-completion for opencli, supporting Bash, Zsh, and Fish.

Co-authored-by: RinChanNOWWW <rin_chan_now@outlook.com>

Co-authored-by: Claude <noreply@anthropic.com>
2026-03-17 16:33:19 +08:00
jackwener 25bd872a24 fix: doctor/setup edge cases — format detection, dynamic profiles, fish shell
- upsertJsonConfigToken: detect format by file path (opencode → mcp format,
  others → mcpServers). Previously empty files always got OpenCode format.
- Dynamic Chrome profile enumeration: scan for Default/Profile N directories
  instead of hardcoding 4 profiles.
- Fish shell: use 'set -gx' syntax for config.fish, not 'export'.
- Pass filePath through all callers (setup.ts, applyBrowserDoctorFix).
- Reduce setup auto-verify timeout from 8s to 5s.
- Add 7 new tests (19 total): empty file format, opencode path detection,
  claude.json path detection, fish shell set/replace/append, zshrc fallback.
2026-03-17 16:13:05 +08:00
jackwener ff3e5c6887 feat: enhance setup with precise token scan errors and auto-verify
- When token scan fails, diagnose exact cause via checkExtensionInstalled()
  (extension not installed vs token not in LevelDB)
- Show actionable fix instructions instead of generic warning
- Auto-verify browser connectivity after writing configs (Step 7)
- Simplify README setup flow to 2 steps (install + setup)
2026-03-17 16:07:38 +08:00
SonicKang 341c42c62f fix(opencode): use 'environment' instead of 'env' for MCP config (#29)
OpenCode config schema uses 'environment' property for MCP server
environment variables, not 'env'.

Schema reference: https://opencode.ai/config.json
2026-03-17 14:46:53 +08:00
jackwener 50b71c0936 fix: use binary read for LevelDB token discovery on all platforms
The previous strings+grep pipeline failed because LevelDB's internal
encoding fragments ASCII strings like 'auth-token' and the extension ID
across byte boundaries. Replace extractTokenViaStrings with a unified
binary read approach that scans for the extension ID prefix and searches
a 500-byte window for base64url tokens.

Also removes unused execSync import.
2026-03-17 14:44:34 +08:00
jackwener 981c167a0b feat(doctor): add extension install check and token connectivity test
- checkExtensionInstalled(): scans Chrome/Edge/Chromium Extensions dirs
- checkTokenConnectivity(): actual MCP handshake via --live flag
- Updated DoctorReport type and report rendering
- Added unit tests for new rendering (12/12 pass)
2026-03-17 14:38:16 +08:00
Ji 8e7490407c feat(youtube): add video metadata and transcript commands (#25)
Add two new YouTube adapters:

- **youtube video**: fetch metadata (title, views, description, etc.) from ytInitialPlayerResponse and ytInitialData
- **youtube transcript**: fetch subtitles via Android InnerTube API to bypass PoToken requirement on Web client caption URLs
  - Two output modes: --mode grouped (sentence merging, speaker detection, chapter headings) and --mode raw (precise sub-second timestamps)
  - CJK support with 30s time-window fallback for unpunctuated captions
  - Language selection with --lang and stderr warning on fallback
  - URL normalization for watch, youtu.be, shorts, embed, live formats

Co-authored-by: Ji Zhang <jizhang.work@gmail.com>
2026-03-17 13:26:30 +08:00
Ji 14dcd2bc5f feat(reddit): add threaded comment tree to read command (#26)
Replace flat top-level-only read.yaml with recursive tree walker:
- Configurable depth and breadth (--depth, --replies)
- Replies sorted by score, top-K selected at each level
- Hidden replies surfaced as [+N more replies]
- Multiline bodies preserve indentation at all depths
- Configurable --max_length (was hard-coded 500 chars)
- Input validation: all numeric params clamped to safe minimums
2026-03-17 13:18:11 +08:00
SiweiMa e9b9beedfe feat(linkedin): add job search adapter (#28)
* feat: add linkedin job search adapter

* fix(linkedin): fix parseCsvArg undefined bug, regex escapes in page.evaluate, replace hardcoded wait, add IPage type

* refactor(linkedin): extract evaluate logic, add progress logging, improve code structure

- Extract Voyager query/URL building into typed standalone functions
- Split fetchJobCards into its own function with per-batch evaluate
- Add SearchInput interface for type safety
- Add progress logging to enrichJobDetails (stderr)
- Add section comments for code organization
- Deduplicate normalize helpers in evaluate strings

---------

Co-authored-by: Siwei Ma <siweima@Siweis-MacBook-Pro.local>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-03-17 13:07:16 +08:00
jackwener 7555f14369 refactor: deep code review improvements
- Add *.log to .gitignore, remove debug.log from tracking
- Fix dev-mode FS scan to discover .ts adapter files (not just .js)
- Deduplicate CONNECT_TIMEOUT: browser.ts now uses runtime.ts constant
- Fix CSV output: escape newlines in field values per RFC 4180
- Add proper type interfaces for validate/verify (remove any types)
- Remove unused hadOuterQuotes variable in snapshotFormatter
- Derive CliOptions from CliCommand via Omit+Partial to reduce duplication
- Expand dense one-liner action callbacks in main.ts for readability
2026-03-17 01:38:23 +08:00
jackwener fbec2f6f5d feat(snapshot): filter contentinfo subtrees, bilibili ad URLs, boilerplate buttons
- Add contentinfo to subtree-level noise filtering (biggest single win)
  - Reuters: 51% → 62%, Google: 57% → 70%, Netflix: 48% → 60%
- Add cm.bilibili.com/cm/api/fees/ ad URL pattern
- Add 广告 keyword to ad detection
- Add back-to-top / 回到顶部 boilerplate button filtering
- Unify ad/boilerplate/contentinfo into single subtree-skip mechanism
- Add vitest config and comprehensive test suite (33 tests)
- Fixture tests skip gracefully when snapshot files are absent

Bump to v0.7.1
2026-03-17 01:26:49 +08:00
jackwener 0c845d58c8 feat(twitter): implement article, profile, follow, unfollow, bookmark, & unbookmark adapters
This commit introduces the long-form Article adapter, a rewritten Profile adapter, and 4 new UI-based Write commands for managing relationships and bookmarks. Also adds support for positional arguments across the dynamic CLI engine.
2026-03-16 20:17:38 +08:00
jackwener 7f55950fed feat(reddit): add 11 new adapters borrowed from rdt-cli
Phase 1 - YAML adapters (read-only):
- popular: /r/popular feed
- read: read post + comments by ID
- user: view user profile (karma, account age)
- user-posts: user's submitted posts
- user-comments: user's comment history
- search: enhanced with sort/time/subreddit params
- subreddit: enhanced with time filter for top/controversial

Phase 2 - TypeScript adapters (write operations):
- upvote: upvote/downvote posts via /api/vote
- save: save/unsave posts via /api/save
- comment: post comments via /api/comment
- subscribe: subscribe/unsubscribe subreddits
- saved: browse saved posts (auto-resolves username)
- upvoted: browse upvoted posts (auto-resolves username)

Reddit adapters: 4 → 15
2026-03-16 19:56:55 +08:00
jackwener 77193a0003 Merge PR #20: feat(boss): add detail adapter + security_id in search
Closes #20

Added boss detail command with fixes:
- district/address field dedup
- template string injection safety
- empty jobInfo guard
- IPage-compatible wait
2026-03-16 19:33:17 +08:00
jackwener 05b7f1bccf fix(boss): improve detail adapter quality
- Fix district/address field duplication (district now uses areaDistrict·businessDistrict)
- Fix template string injection risk in evaluate script (use JSON.stringify)
- Add jobInfo empty guard with user-friendly error message
- Replace raw setTimeout with page.wait for IPage compatibility
- Update README docs to include boss detail command
2026-03-16 19:33:00 +08:00
jackwener 61ea05bff7 fix: URL injection, strictNullChecks, cross-platform build, +34 tests
Security:
- Fix URL injection in fetch.ts and bilibili.ts (JSON.stringify instead of string interpolation)
- Fix unused scroll() amount parameter

TypeScript:
- Enable strictNullChecks in tsconfig
- Change CliCommand.func signature to IPage (non-null) for browser adapters
- Fix 93 compile errors across all adapters

Build:
- Remove || true from build-manifest (report failures instead of silencing)
- Replace Unix shell commands with Node.js scripts for cross-platform builds

Code quality:
- Remove error-object special detection from pipeline executor
- Unify error handling to throw pattern

Tests:
- New interceptor.test.ts (11 tests)
- New executor.test.ts (13 tests)
- Rewrite output.test.ts with comprehensive coverage (10 tests)
2026-03-16 18:14:27 +08:00
xuelin e781d40408 feat(boss): add security_id to search output
Expose securityId in search results so users can pipe it to
`boss detail` for full job information.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 18:12:46 +08:00
xuelin c230f3e5ad feat(boss): add job detail adapter
Add `boss detail` command to fetch full job posting details using
securityId from search results.

Fields returned: job description, skills, welfare, boss info (name,
title, active time), company info (industry, scale, stage), address.

Tested with real API calls against multiple job postings.

Usage:
  opencli boss detail --security_id <id_from_search>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-16 18:09:49 +08:00
AlexYue 788b069c02 feat: add E2E testing infrastructure with real Chrome in CI
## Changes

### E2E Test Suite (~52 test cases)
- public-commands.test.ts — Public API commands (hackernews, v2ex)
- browser-public.test.ts — Browser commands for public data across all sites
- browser-auth.test.ts — Graceful failure verification for login-required commands
- management.test.ts — Full coverage of management commands
- output-formats.test.ts — Output format validation (json/yaml/csv/md)
- smoke/api-health.test.ts — Scheduled API health checks

### Auto-detect Browser Mode
- buildMcpArgs uses CI env var to select mode:
  - Local (no CI) → --extension (connect to user's Chrome)
  - CI → standalone (launches its own browser)

### CI Pipeline
- e2e-headed.yml — Real Chrome via setup-chrome + xvfb in headed mode
- ci.yml — build + unit-test (2 shards) + smoke-test (scheduled/manual)
- Composite action for shared Chrome + xvfb setup

### Documentation
- New TESTING.md — Architecture, coverage, local setup, how to add tests

Co-authored-by: AlexYue <yj976240184@qq.com>
2026-03-16 17:35:16 +08:00
jackwener 486e513d07 fix(setup): only pre-select shell RC, let user choose other configs 2026-03-16 14:15:43 +08:00
jackwener 6486a42def fix(setup): clear screen before TUI to prevent page jumping 2026-03-16 14:04:38 +08:00
jackwener b308d5594a refactor(doctor/setup): polish UX and dedup code
- Doctor: chalk-colored output ([OK] green, [MISSING] red, etc.)
- Doctor: paths shortened with ~ and tool labels ([Codex], etc.)
- Doctor --fix: skip already-configured files
- TUI: hide cursor during interaction, proper Ctrl+C exit
- Setup: source hint after shell write, dedup shared helpers
- Tests: strip ANSI for assertions
2026-03-16 14:02:02 +08:00
jackwener 6f629260e7 feat: add interactive 'opencli setup' command with TUI checkbox
New zero-dependency TUI checkbox component (src/tui.ts) with:
  - ↑↓/jk navigation, Space toggle, Tab toggle+move
  - 'a' toggle all, Enter confirm, q/Esc cancel

New 'opencli setup' command (src/setup.ts) that:
  - Auto-discovers token from Chrome LevelDB extension
  - Shows interactive multi-select for config files
  - Displays tool names (Codex, Cursor, Claude Code, etc.)
  - Color-coded status (green=ok, yellow=mismatch, red=missing)
  - Applies changes only to selected files
2026-03-16 13:53:03 +08:00
jackwener 1e0b83bb84 feat(doctor): add Antigravity and Gemini CLI config paths
Add ~/.gemini/settings.json (Gemini CLI) and
~/.gemini/antigravity/mcp_config.json (Antigravity) to the
default MCP config scan list.
2026-03-16 13:45:02 +08:00
jackwener 1ff184e24d feat(doctor): add Claude Code and project .mcp.json config paths
Add ~/.claude.json (Claude Code user-scoped MCP config) and
.mcp.json (Claude Code project-scoped MCP config) to the
default config scan list.
2026-03-16 13:43:03 +08:00
jackwener 8e84145ccc feat(doctor): auto-discover extension token from Chrome LevelDB
Scan Chrome/Edge/Chromium localStorage LevelDB files to extract the
Playwright MCP Bridge auth-token directly from the extension's storage.
Uses a fast 'strings | grep' shell pipeline (~200ms) on macOS/Linux
with a pure-Node fallback for Windows.

The discovered token is now shown in 'opencli doctor' output and takes
priority as the recommended token when using '--fix'.
2026-03-16 13:38:48 +08:00
jackwener c77c8a8e3a feat: add Coupang search and add-to-cart adapters
Add browser-backed Coupang adapters for search and add-to-cart workflows.
- coupang search: multi-strategy data collection (API/JSON-LD/bootstrap/DOM), structured fields (price, rating, rocket, delivery), pagination and rocket filter support
- coupang add-to-cart: logged-in browser session reuse, stops before checkout
- coupang.ts: comprehensive data normalization layer with badge/rocket/delivery mapping
- browser-tab.ts: withTemporaryTab utility for isolated tab operations
- Unit tests for core normalization functions

Co-authored-by: CodeBBakGoSu <127713112+CodeBBakGoSu@users.noreply.github.com>
2026-03-16 13:30:42 +08:00
jackwener 34a9bff2b3 refactor: eliminate code duplication, improve type safety, add tests
- NEW: src/interceptor.ts — unified XHR/Fetch interceptor (was duplicated 3x)
- NEW: src/version.ts — centralized PKG_VERSION (was duplicated 2x)
- NEW: src/constants.ts — shared VOLATILE_PARAMS, FIELD_ROLES etc.
- NEW: src/engine.test.ts, src/registry.test.ts — 14 new unit tests

- browser.ts: use shared normalizeEval, interceptor, withTimeoutMs, PKG_VERSION
- intercept.ts, tap.ts: use shared interceptor generators
- cascade.ts: extract shared buildFetchProbeJs (90% dedup)
- engine.ts: use InternalCliCommand (no more 'as any' casts)
- executor.ts: remove all 15 'as StepHandler' type casts
- runtime.ts: add withTimeoutMs, IBrowserFactory interface
- registry.ts: add InternalCliCommand type for internal fields
- validate.ts: add pipeline step name validation
- output.ts: remove 'null as any' and '.filter(() => true)' hacks
- explore.ts, synthesize.ts: use shared constants
- docs: fix V2EX commands (3→6), SKILL.md version, verify example

Tests: 88 passed (was 74), tsc --noEmit: 0 errors
2026-03-16 13:22:12 +08:00
jackwener b0c51ddf19 fix: use Playwright MCP --executable-path flag (kebab-case)
Extract buildMcpArgs() helper and fix --executablePath → --executable-path
to match the Playwright MCP CLI's expected flag format.

Closes #16

Co-authored-by: KasumiChen <KasumiChen@users.noreply.github.com>
2026-03-16 12:58:18 +08:00
jackwener 7ee0c9c96b fix: tab cleanup regex to match actual Playwright MCP tab format
The tab list from Playwright MCP uses '- N: (current) [title](url)' format,
but extractTabEntries only matched 'Tab N ...' format. This caused
_initialTabIdentities to always be empty, so tabs were never cleaned up.

Now supports both formats.
2026-03-16 12:54:44 +08:00
jackwener 95fb841fc7 refactor: remove all CDP code, keep extension-only mode
- Remove CDP auto-discovery (discoverChromeEndpoint, isCdpApiAvailable, isPortReachable)
- Remove CDP readiness probe and autoAllowCdpDialog
- Remove forceExtension plumbing from registry, runtime, main, and adapters
- Remove CDP diagnostics from doctor (extractTokenViaCdp, remoteDebugging fields)
- Simplify connect() to always use --extension mode
- Remove CDP env vars (OPENCLI_USE_CDP, OPENCLI_CDP_ENDPOINT, OPENCLI_FORCE_EXTENSION)
- Update README.md, README.zh-CN.md, SKILL.md to remove CDP sections
- Update tests to remove CDP-related assertions

Current state preserved in extension-and-cdp branch.
2026-03-16 12:44:09 +08:00
jackwener 67d50191af fix: CDP mode hanging when using chrome://inspect remote debugging
- Add isCdpApiAvailable() to verify CDP HTTP JSON API before using endpoint
- Chrome's chrome://inspect#remote-debugging writes DevToolsActivePort but
  its CDP endpoint is incompatible with Playwright connectOverCDP (init
  succeeds but all tool calls hang silently)
- Add CDP readiness probe in connect() to catch unresponsive endpoints
- Skip tab cleanup in close() for CDP mode (no bridge tabs to clean)
- Add PlaywrightMCPMode tracking for extension vs CDP lifecycle policy
2026-03-16 02:28:28 +08:00
jackwener f8491f01ae fix: add 5s timeout for initial tabs fetch during CDP connect
CDP mode on @playwright/mcp v0.0.68 hangs on browser_tabs calls.
This timeout prevents the entire connect() from blocking for 30s.
2026-03-16 01:51:22 +08:00
jackwener db8194337b Improve browser session lifecycle handling 2026-03-15 23:31:52 +08:00
jackwener cef50dc8bf feat(doctor): print cli version in doctor report 2026-03-15 21:51:21 +08:00
jackwener 1aaafed2db feat(twitter): implement post, reply, delete, following, followers and notifications adapters 2026-03-15 21:42:48 +08:00
jackwener ee6669f45d feat(core): support YAML format output and optimize manifest builder 2026-03-15 21:05:36 +08:00
jackwener eb9050fadd feat(v2ex): add daily, me, and notifications session adapters 2026-03-15 20:59:39 +08:00
jackwener 53cd006a23 fix(boss): resolve limit=60 infinite loop and global CLI hangs
- search.ts: Added deduplication break condition so if the API returns duplicate pages, it stops instead of looping forever.
- search.ts: Removed unsafe Promise.race around page.goto which caused 'context closed' errors.
- browser.ts: Added robust static cleanup hooks to guarantee process and MCP lock destruction on crashed or ctrl+c'd runs.
2026-03-15 20:33:22 +08:00
jackwener c9710a656b feat(boss): add pagination loop to support fetching more than 15 results
- The API endpoint only returns 15 results per page.
- Added a loop to fetch multiple pages until the requested limit is reached.
- Added random 1-3 second delays between page fetches to avoid rate limits.
2026-03-15 20:19:30 +08:00
jackwener 00f0ac9133 fix(boss): prioritize fresh browser cookie extraction over boss-cli cache 2026-03-15 19:58:46 +08:00
jackwener 64d655ae0d feat(boss): rewrite search adapter to use boss-cli native cookie extraction
- BOSS Zhipin aggressively detects CDP connections and kills pages
- Completely rewrote boss/search.ts to use browser: false (bypassing Playwright)
- Implemented native Python subprocess to extract cookies via browser-cookie3
- Falls back to boss-cli ~/.config/boss-cli/credential.json if available
- Uses direct Node.js fetch with anti-detection headers to avoid bans
2026-03-15 19:55:46 +08:00
jackwener a6e9c3b611 feat(browser): enhance CDP auto-discovery with port probe & opt-in priority
- Add TCP port reachability check to handle stale DevToolsActivePort files
- Make discoverChromeEndpoint() async with port probe before returning
- Change connection priority: extension mode is default, CDP is opt-in
  - OPENCLI_CDP_ENDPOINT: explicit CDP endpoint
  - OPENCLI_USE_CDP=1: enable DevToolsActivePort auto-discovery
  - Default: --extension mode (Playwright MCP Bridge)
- Add verbose logging for CDP discovery when -v flag is used
- Fix indentation inconsistency in connect() method
- Update README.md, README.zh-CN.md, SKILL.md with new connection docs
2026-03-15 19:44:27 +08:00
jackwener da155e96f0 feat: add forceExtension to bypass CDP detection for anti-bot sites
- Added forceExtension flag to CliCommand, CliOptions, browserSession, and PlaywrightMCP.connect()
- BOSS Zhipin detects CDP mode and drops connections; marked boss/search with forceExtension: true
- Also supports OPENCLI_FORCE_EXTENSION=1 env var for global override
- Fixed async discoverChromeEndpoint() placement outside Promise constructor
- Enhanced boss search with city names, experience, degree, salary, etc.
2026-03-15 19:39:39 +08:00
backtime1993 c20eaee393 feat(browser): Chrome 144+ auto-discovery via DevToolsActivePort
Chrome 144+ allows enabling remote debugging from chrome://inspect
without any command-line flags (--remote-debugging-port). Chrome writes
the active port and browser GUID to a DevToolsActivePort file in the
user data directory.

This change reads that file to auto-discover the CDP WebSocket endpoint,
eliminating the need for the Playwright MCP Bridge browser extension in
most cases.

Connection priority:
1. OPENCLI_CDP_ENDPOINT env var (manual override)
2. DevToolsActivePort auto-discovery (Chrome/Edge, cross-platform)
3. --extension mode fallback (original behavior)

Supports Windows, macOS, and Linux with Chrome, Edge, and Chromium.

Co-authored-by: kensei <backtime1993@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 19:34:14 +08:00
jackwener 15d8c45b95 feat(boss): add city name mapping, experience/degree/salary filters, page support
- 30+ Chinese city names → code mapping (e.g. --city 杭州)
- Experience filter: 应届/1-3年/3-5年/5-10年/10年以上
- Degree filter: 大专/本科/硕士/博士
- Salary filter: 3K以下/3-5K/5-10K/10-20K/20-50K/50K以上
- Page number support for pagination
- Area now shows city·district·business district
- Skills column added to output
2026-03-15 19:31:05 +08:00
jackwener 2d638e3a0a fix(manifest): extract TS adapter args at build time for proper CLI registration
The manifest builder now statically parses TS source files to extract
args, description, columns, domain, and strategy metadata. Previously
TS adapters had empty args in the manifest, causing commander to reject
their options (e.g. --keyword) before the lazy module could load.

Also includes the xiaohongshu search DOM fallback fix from earlier.

Co-authored-by: jack4world <jack4world@users.noreply.github.com>
2026-03-15 18:59:15 +08:00
jackwener b3850d9c53 fix(xiaohongshu): switch search from broken XHR interception to DOM extraction (Fixes #10)
Co-authored-by: jack4world <jack4world@users.noreply.github.com>
2026-03-15 18:55:29 +08:00
jackwener f8ee50b776 feat(bilibili): add following adapter — get user's followed accounts
Discovered via Playwright MCP Bridge workflow (CLI-CREATOR.md):
 - Navigate to space.bilibili.com/{uid}/fans/follow
 - Sniff network: GET /x/relation/followings (Cookie-tier, no Wbi needed)
 - Verify API response: total=1342, fields: mid/uname/sign

Also fixed .d.ts loading bug in build-manifest.ts and engine.ts:
 - build-manifest now scans .js files in dist/ (not .ts)
 - engine.ts skips .d.js declaration stubs
2026-03-15 18:38:34 +08:00