* feat(xiaohongshu): use CDP DOM.setFileInputFiles for image upload
Replace base64 DataTransfer injection with CDP DOM.setFileInputFiles,
which lets Chrome read image files directly from the local filesystem.
This eliminates payload size limits that caused "fetch failed" errors
when uploading large images (>500KB) through the browser bridge.
Changes:
- Add 'set-file-input' action to protocol, extension handler, and CDP executor
- Add Page.setFileInput() method for CLI-side usage
- Rewrite publish image upload to use CDP path, with base64 fallback
for older extension versions that don't support the new action
- Add clear warning when falling back to base64 with large payloads
Closes#542 (partially — image upload reliability)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: cover cdp file input upload path
* fix: keep image upload on image-only inputs
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
* refactor: slim CI matrix, extract shared utils, unify logging, remove __test__ from public API
- CI: unit-test uses dynamic matrix (PR=ubuntu+22 only, push=full 3OS×2Node);
adapter-test reduced to ubuntu-latest (OS doesn't affect pure unit tests)
- _shared/common.ts: add sleep() and clampToRange() shared adapter utilities;
douban/utils.ts and sinablog/utils.ts now use clampToRange instead of duplicate clampLimit
- browser/daemon-client.ts: replace inline setTimeout Promise with local sleep()
- execution.ts: replace conditional console.error with log.debug
- browser/index.ts: remove __test__ from public barrel export;
browser.test.ts now imports internal helpers directly from source files
* fix: remove unused afterEach import, fix schedule/dispatch CI matrix, clarify clampToRange docs
* refactor: move sleep to src/utils.ts, simplify clamp signature to match lodash convention
* fix(browser): retry settle probe after SPA client-side redirect
SPA sites like creator.xiaohongshu.com can trigger a client-side
redirect after chrome.tabs reports status 'complete', invalidating
the CDP target. The waitForDomStable probe in page.goto() was
unprotected, causing -32000 "Inspected target navigated or closed".
Wrap the settle probe in try/catch with a single 200ms-delayed retry,
consistent with the existing stealth injection error handling pattern.
The retry gives the SPA redirect time to complete, while the outer
catch ensures settle failure never crashes goto() since navigation
itself already succeeded.
Closes#502
* review: narrow settle retry to target redirects
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* feat: zero onboarding, extension version check, and update notifier
- Fail-fast guard in execution.ts: when daemon is running but extension
is not connected, immediately surface a setup guide instead of waiting
for the 30s connect timeout
- Extension version handshake: extension sends `hello` with its version
on WebSocket connect; daemon stores it and exposes via /status; CLI
warns on mismatch in both execution path and `opencli doctor`
- `opencli doctor` now shows extension version inline and reports
version mismatch as an actionable issue
- Non-blocking npm update checker: registers a process exit hook so the
update notice appears after command output (same pattern as npm/gh/yarn);
background fetch writes to ~/.opencli/update-check.json for next run
- postinstall: print Browser Bridge setup instructions after shell
completion install for first-time global install users
Bug fixes caught in review:
- discover.ts: add AbortController timeout to checkDaemonStatus() fetch,
move clearTimeout after res.json() to cover body streaming
- daemon.ts: clear extensionVersion and reject pending requests in
ws.on('error') handler, not just ws.on('close')
- update-check.ts: skip update notice when process exits with non-zero
code; read cache once at module load to avoid double disk I/O;
guard isNewer() against NaN from pre-release version strings
* fix: relax extension version check to major-only in doctor, remove from hot path
* test: enable all adapter tests via wildcard glob, fix apple-podcasts url field
* fix: clearTimeout in finally block, reset extensionVersion on reconnect, fix e2e regex
* perf: smart pre-navigation — skip redundant domain nav + remove hardcoded 2s wait
- Add `getCurrentUrl()` to IPage, Page, and CDPPage to check current browser URL
- Skip pre-navigation entirely if the browser is already on the target domain
- Remove the hardcoded `page.wait(2)` after pre-navigation — `page.goto()` already
includes smart DOM-settle detection via `waitForDomStable`, making the fixed
2-second sleep redundant
- Saves ~2s per browser command in the common case (consecutive commands on the
same site), and ~1-2s even on cold navigation
* perf: smart page.wait() — DOM-stable early return for waits >= 1s
For page.wait(N) where N >= 1 second, use DOM MutationObserver-based
stability detection instead of a fixed sleep. The original wait time
becomes a hard cap, but the call returns as soon as the DOM stops
mutating (500ms quiet period).
This benefits ~200 hardcoded sleep calls across ~40 adapters without
changing any adapter code. A typical page.wait(5) now completes in
<1s when the page is already stable, instead of always waiting 5s.
Short waits (< 1s) are kept as fixed sleeps — these are typically
UI animation delays or anti-bot throttling where DOM-ready is irrelevant.
* refactor: getCurrentUrl() uses in-memory tracking instead of round-trip
Replace the sendCommand('exec', 'window.location.href') call with a
local _lastUrl field set during goto(). This eliminates a daemon HTTP
round-trip for the domain check, making isAlreadyOnDomain() zero-cost.
On fresh tabs (about:blank), _lastUrl is null so we correctly fall
through to navigation — no special-casing needed.
* feat: zero onboarding, extension version check, and update notifier
- Fail-fast guard in execution.ts: when daemon is running but extension
is not connected, immediately surface a setup guide instead of waiting
for the 30s connect timeout
- Extension version handshake: extension sends `hello` with its version
on WebSocket connect; daemon stores it and exposes via /status; CLI
warns on mismatch in both execution path and `opencli doctor`
- `opencli doctor` now shows extension version inline and reports
version mismatch as an actionable issue
- Non-blocking npm update checker: registers a process exit hook so the
update notice appears after command output (same pattern as npm/gh/yarn);
background fetch writes to ~/.opencli/update-check.json for next run
- postinstall: print Browser Bridge setup instructions after shell
completion install for first-time global install users
Bug fixes caught in review:
- discover.ts: add AbortController timeout to checkDaemonStatus() fetch,
move clearTimeout after res.json() to cover body streaming
- daemon.ts: clear extensionVersion and reject pending requests in
ws.on('error') handler, not just ws.on('close')
- update-check.ts: skip update notice when process exits with non-zero
code; read cache once at module load to avoid double disk I/O;
guard isNewer() against NaN from pre-release version strings
* fix: reduce fail-fast timeout to 300ms and guard stderr.write in exit hook
Add search element heuristics and label/span wrapper detection
- Add SEARCH_INDICATORS set to detect search-related elements
- Add isSearchElement function for heuristic detection
- Add hasFormControlDescendant to detect wrapped form controls
- Enhance isInteractive for label/span wrapper patterns
Ref: browser-use ClickableElementDetector research
Review: @codex
- Remove unused re-exports from registry.ts (serializeArg, serializeCommand, etc.)
- Unify FormatOptions into SnapshotOptions from types.ts; rename dom-snapshot's
SnapshotOptions to DomSnapshotOptions to avoid name collision
- Extract shared analysis.ts module from explore.ts and record.ts, eliminating
~200 lines of duplicated logic (urlToPattern, findArrayPath, inferCapabilityName,
inferStrategy, detectAuth*, classifyQueryParams)
- Merge snapshotFormatter from 7-pass to 4-pass pipeline by combining parse+filter
with ad/boilerplate subtree skipping, and merging three dedup passes into one
- Rename all CLI adapter shared files to consistent utils.ts naming
(boss/common.ts, douban/shared.ts, doubao*/common.ts, jike/shared.ts,
medium/shared.ts, sinablog/shared.ts, substack/shared.ts)
- Merge douban/shared.ts into douban/utils.ts
* feat(browser): human-like delay system for anti-detection
Adds a framework-level delay/jitter system using log-normal distribution
to simulate natural browsing patterns, addressing issue #59 (P0).
- New `HumanDelay` class with configurable profiles (none/fast/moderate/cautious/stealth)
- Log-normal distribution for realistic delay variance (not uniform)
- Periodic "breaks" that simulate reading/thinking pauses
- Auto-injected between page.goto() navigations
- Configurable via OPENCLI_DELAY_PROFILE env var
- Boss search adapter migrated from hardcoded jitter to framework delay
- 10 unit tests covering all profiles and edge cases
Real-world validation against a major job board (cookie-authenticated,
aggressive bot detection):
| Scenario | Without jitter | With jitter |
|-----------------------|--------------------|--------------------|
| 50 detail pages | ✅ OK | ✅ OK |
| 200 detail pages | ❌ Banned (code 32) | ✅ OK |
| 850 requests over 5h | N/A (banned early) | ✅ Zero detection |
| 4-day sustained crawl | N/A | ✅ 1800+ records |
Closes#59
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: disable human delay in CI environment to prevent E2E timeouts
In CI environments (CI=true), resolveProfile() now defaults to the
'none' profile instead of 'moderate'. This prevents the 1-8s per-
navigation delay from causing E2E test timeouts (30s limit).
Users can override this by setting OPENCLI_DELAY_PROFILE explicitly.
---------
Co-authored-by: toolmanlab <toolmanlab@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
- I1: Log pre-navigation failures in debug mode instead of silently swallowing
- I2: Validate env var timeout values, fallback on NaN/negative
- I4: Guard against indexOf returning -1 for unknown strategies in cascade
- I5: Fix shouldReplaceManifestEntry returning true for same-type entries
- I6: Prevent infinite loop in parseTsArgsBlock cursor advancement
- I7: Skip redundant Page.enable calls in CDP goto
- I8: Fix wait({time:0}) being treated as falsy
- I10: Warn when cookiesFile path doesn't exist before fallback
- I11: Sanitize tab/newline chars in cookie name/value for Netscape format
- I12: Use DEFAULT_DAEMON_PORT constant instead of hardcoded port in error
- I15: Log npm install failures in plugin lifecycle instead of swallowing
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1. execution.ts: Guard lazy-loaded func commands against null page — if a
lazy module incorrectly requires browser context, throw a clear error
instead of a cryptic TypeError on page.goto().
2. daemon.ts: Fix readBody race condition — add aborted flag to prevent
req.destroy() from triggering both reject (via error) and resolve
(via end event) on the same Promise, which could process truncated data.
3. browser/cdp.ts: Prevent CDPBridge.connect() reentry — throw if already
connected instead of silently leaking the previous WebSocket and its
message handlers.
4. interceptor.ts: Store intercept pattern in a separate global variable
so subsequent installInterceptor calls with different patterns update
the match condition without being blocked by the patchGuard.
5. record.ts: Always call cleanupEnter() after Promise.race — previously
only called in the timeout path, leaving readline open when user pressed
Enter, potentially blocking process exit. Also removed unused enterRace.
6. generate.ts: Fix undefined entering String.includes() — when c.name is
undefined, toLowerCase() returns undefined which gets coerced to the
string "undefined" by includes(), causing false positive matches.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- Only override navigator.plugins when empty (don't replace real user
browser plugins with fakes)
- Replace Error.prepareStackTrace (V8/Node-only) with
Error.prototype.stack getter override that works in browser context
- Fix \\n escaping in template literal for stack trace split/join
- Dynamic cdc_ variable scan via getOwnPropertyNames instead of
hardcoded names
- Update tests to cover 7 patches
Add stealth.ts module that patches browser globals to hide automation
fingerprints when opencli controls a browser via CDP or daemon extension.
Patches applied:
- navigator.webdriver → undefined (CDP sets it to true)
- window.chrome stub (only if missing)
- navigator.plugins fake list (only if empty)
- navigator.languages guarantee (only if empty)
- Permissions.query normalization for notifications
- Cleanup __playwright/__puppeteer/cdc_* artifacts
CDP mode: stealth registered via Page.addScriptToEvaluateOnNewDocument
(runs before any page JS on every navigation).
Daemon mode: stealth injected via exec after navigation, with guard
flag to prevent double-injection.
- Add getErrorMessage() to errors.ts (used in 5 files)
- Add DEFAULT_DAEMON_PORT to constants.ts (used in 5 files)
- Reduces code duplication and improves maintainability
Replace fixed settleMs sleep in goto() with MutationObserver-based DOM
stability detection. The page is considered settled when no DOM mutations
occur for quietMs (default 500ms), with settleMs as a hard timeout cap.
Changes:
- Add waitForDomStableJs() shared helper to dom-helpers.ts
- Update Page.goto() and CDPPage.goto() to use smart settle
- No IPage interface changes (implementation detail only)
Key improvements over naive approach:
- Timer starts AFTER MutationObserver.observe() to avoid race condition
- Falls back to sleep(maxMs) if document.body is not available
- Monitors attributes in addition to childList/subtree
- quietMs defaults to 500ms (conservative) for async request buffering
- Add Origin header check: reject HTTP/WS from non chrome-extension:// origins
- Require X-OpenCLI custom header on all HTTP requests
- Remove Access-Control-Allow-Origin: * from all responses
- Add WebSocket verifyClient to reject malicious connections at upgrade
- Add 1MB body size limit to prevent OOM
- Update file header with security model documentation
Closes#268
- resolveTabId: validate URL even for explicit tabId, fall through to
auto-resolve when tab is not debuggable or has been closed
- handleNavigate: wait for URL change before checking 'complete' status
to avoid race condition with stale about:blank
- ensureAttached: pre-check tab URL, verify cached attach with probe,
invalidate cache on URL change via onUpdated listener
- daemon-client: recognize transient extension errors (disconnected,
attach failed) as retryable with 1500ms delay; fresh command ID per attempt
- pipeline executor: add per-step retry for browser steps (up to 2 retries
on transient errors); cleanup automation window on pipeline failure
- page.ts: selectTab/newTab/closeTab properly update/invalidate _tabId
- daemon.ts: add WebSocket ping/pong heartbeat (15s interval, 2-miss disconnect)
- Increase automation window idle timeout from 30s to 120s
- Fix timeout param edge cases in BrowserBridge._ensureDaemon
- Remove unused chalk import; fix trailing import placement
Closes#249
* feat: plugin system (Stage 0-2)
- Stage 0: discoverPlugins() scans ~/.opencli/plugins/ at startup
- Stage 1: demo plugin repos (github-trending, hot-digest)
- Stage 2: opencli plugin install/uninstall/list commands
- package.json exports ./registry for TS plugin peerDep support
- 17 new/updated tests, tsc --noEmit clean
* fix: CDPBridge connect timeout unit mismatch (seconds vs ms)
opts.timeout is passed in seconds from runtime.ts but CDPBridge
was using it as milliseconds, causing instant timeout (30ms).
* feat: add registry-api public entry point for TS plugin peerDep support
- Add src/registry-api.ts: re-exports core registration API (cli, Strategy,
getRegistry) without transitive side-effects, safe for plugin imports
- Update package.json exports: './registry' -> './dist/registry-api.js'
- Update src/registry.ts: use globalThis shared registry to ensure single
instance across npm-linked plugin modules
- Update .gitignore for plugin-related artifacts
* fix: symlink host opencli into plugin node_modules on install
After npm install, replace the npm-installed @jackwener/opencli
with a symlink to the running host's package root. This ensures
TS plugins always resolve '@jackwener/opencli/registry' against
the host installation, avoiding version mismatches when the
published npm package lags behind.
* fix: transpile TS plugins to JS on install, deduplicate .ts/.js discovery
- installPlugin: after symlinking host opencli, transpile any .ts files
to .js using esbuild from the host's node_modules/.bin/
- discoverPluginDir: skip .ts files when a .js sibling exists (production
node cannot load .ts directly)
- scanPluginCommands: deduplicate basenames via Set to avoid showing
'aggregate, aggregate' when both .ts and .js exist
* docs: add plugin system user guide
- New docs/guide/plugins.md covering:
- Installation/uninstallation commands
- Creating YAML plugins (zero-dep)
- Creating TS plugins (with peerDep)
- TS plugin install lifecycle (clone → deps → symlink → transpile)
- Example plugins and troubleshooting
- Add Plugins to VitePress sidebar (EN + ZH)
- Link from getting-started.md Next Steps
* fix: address review issues in plugin system
- Security: replace execSync with execFileSync to prevent shell injection
- Replace deprecated npm --production with --omit=dev
- Tighten parseSource regex to [\w.-]+ to reject special chars
- Fix ZH sidebar plugin link (/guide/plugins → /zh/guide/plugins)
- Return plugin name from installPlugin() to avoid duplicated logic
- Use execFileSync for esbuild transpilation
- Fix misleading comment in linkHostOpencli
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* fix(intercept): use evaluate() for IIFE wrapping in installInterceptor/getInterceptedRequests
Root cause: daemon migration changed these methods from this.evaluate()
to direct sendCommand('exec'), losing the wrapForEval() IIFE wrapping.
CDP received bare arrow functions that were never invoked.
Fixes#98
* fix(twitter): SPA navigation, data path, and author resolution for INTERCEPT commands
- followers/following: install interceptor on profile page, then click
followers/following link (SPA navigation preserves JS context).
Use JSON.stringify for targetUser to prevent injection. Throw on
navigation failure. Update selector: /verified_followers.
- notifications: install interceptor on home, then pushState+popstate
to /notifications. Validate navigation URL.
- search: fix author resolution (core.screen_name, not legacy).
- All: fix GraphQL data path (remove extra .data level), update author
resolution to try core.screen_name before legacy.screen_name.
- followers: remove erroneous .filter(r => r?.url) — interceptor stores
response body JSON, URL filtering happens at capture time.
- Add 'close-window' action to extension protocol and background.ts
- Add Page.closeWindow() method to send close-window command
- browserSession() now closes automation window in cleanup
- Remove domain pre-navigation + 2s wait from main.ts (CDP handles
cross-domain cookies natively, no same-origin workaround needed)
- Net effect: commands run faster, no stale windows left behind
- Delete unused extension/src/executor.ts (chrome.scripting experiment)
- Remove 15 no-op backward-compat exports from doctor.ts
- Remove getTokenFingerprint no-op from browser/index.ts
- Rename PlaywrightMCP → BrowserBridge across all source files
(backward-compat alias kept in mcp.ts and browser/index.ts)
- Remove unnecessary host_permissions from extension manifest
- Sync extension package.json version to 0.2.0
- All 14 tests pass
process.execPath is always plain 'node' even under tsx,
so .ts files could not be executed. Use --import tsx/esm
flag to enable TypeScript loading in spawned daemon.
Exponential backoff:
- Reconnect delay: 2s, 4s, 8s, 16s, ..., capped at 60s
- Resets to base delay on successful connection
- Reduces idle CPU waste vs fixed 3s reconnect
Screenshot via CDP Page.captureScreenshot:
- New 'screenshot' action in protocol (5th action)
- Supports format (png/jpeg), quality, fullPage
- Full-page: uses Emulation.setDeviceMetricsOverride for scroll height
- CLI-side: page.screenshot() with optional file save
- Extension build: 9.81KB (+1.7KB from 8.11KB)
Inspired by bb-browser's architecture patterns.
* feat(browser): add CDP remote connection support for server environments
This feature enables OpenCLI to connect to a Chrome browser running on a
different machine (e.g., your local computer) from a headless server
environment via Chrome DevTools Protocol (CDP).
Server environments (CI, cloud VMs, headless Linux) cannot run Chrome with
a GUI or install the Playwright MCP Bridge extension. This makes it
impossible to use OpenCLI commands that require browser authentication.
Add support for the `OPENCLI_CDP_ENDPOINT` environment variable, which
tells OpenCLI to connect to a remote Chrome instance via CDP instead of
using the local extension mode.
1. Start Chrome with remote debugging on local machine:
```
chrome --remote-debugging-port=9222 --user-data-dir="$HOME/chrome-debug"
```
2. Create SSH tunnel to forward port to server:
```
ssh -R 9222:localhost:9222 your-server
```
3. Run OpenCLI on server:
```
export OPENCLI_CDP_ENDPOINT="http://localhost:9222"
opencli bilibili hot --limit 5
```
- src/browser.ts: Add CDP endpoint detection in buildMcpArgs()
- src/doctor.ts: Show CDP mode status in doctor report
- README.md: Add "Remote Chrome (Server/Headless)" section
- README.zh-CN.md: Add corresponding Chinese documentation
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* docs: extract CDP connection guide into separate files
* docs: clarify CDP vs SSH/Proxy distinction in CDP guides
* docs: restructure CDP guides into 3 distinct phases (preparation, tunnel, execution)
---------
Co-authored-by: ByteYue <yj976240184@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
James