* chore: fix pre-existing biome lint in template.ts
- isNaN → Number.isNaN (2 occurrences)
- string concatenation → template literal
- biome-ignore for intentional control chars in sanitize regex
* fix(pipeline): evaluate chained || in template engine (#303)
The || handler in evalExpr returned the right side as a literal string
instead of recursively evaluating it. This broke chained fallbacks like
`item.a || item.b || 'default'` — when item.a was falsy, the entire
`item.b || 'default'` was returned as text.
Fix: call evalExpr on the right side so chained || works at any depth.
* perf(pipeline): fast-path string literals in evalExpr to skip VM
When the right side of || is a quoted string like 'N/A', detect it
with a simple regex and return directly instead of falling through
to evalJsExpr which spins up a node:vm sandbox.
* refactor(pipeline): simplify evalExpr by removing hand-rolled operator parsing
Replace the manual regex-based || and arithmetic handlers with a
streamlined flow: pipe filters → fast-path literals → resolvePath →
evalJsExpr (VM). The VM already handles ||, ??, arithmetic, ternary,
etc. natively, so reimplementing them with regex was redundant and
bug-prone (see issue #303).
Key improvements:
- Fix pipe | vs || disambiguation with lookbehind/lookahead regex
(?<!|)|(?!|) so "item.a || item.b | upper" works correctly
- Remove ~20 lines of manual operator handling
- Add numeric literal fast path
- Pipe filter handler now uses evalExpr recursively (not just
resolvePath), enabling filters on complex expressions
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* feat(xueqiu): add danjuan fund account commands
* refactor(xueqiu): convert danjuan fund YAML adapters to TS
- Replace 3 YAML files with 4 TS files (shared utils + 3 commands)
- Extract shared helpers: fetchDanjuanApi, fetchAssetGain, collectHoldings
- Fix double-navigation by using navigateBefore instead of pipeline navigate
- Unify error messages to English with Hint pattern
- Mask real account ID in docs example
- Add explicit default for --account arg
* refactor(xueqiu): optimize danjuan fund adapters
- Single page.evaluate with Promise.all for parallel account fetching
(1 browser round-trip instead of N+1)
- Merge fund-accounts into fund-holdings (account info visible per row)
- 3 files: danjuan-utils.ts (shared), fund-holdings.ts, fund-snapshot.ts
- Strong TypeScript interfaces for all data shapes
- Update docs to reflect 2-command design
* fix(xueqiu): preserve danjuan pre-navigation metadata
* fix(xueqiu): fail on incomplete danjuan snapshots
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* feat(tiktok): add video URL to search results
Add a 'url' field to the TikTok search adapter output, constructed from
the author's uniqueId and the video id returned by the API. This allows
downstream consumers (AI agents, pipelines, scripts) to link directly to
each video instead of only having the author handle.
The URL format is: https://www.tiktok.com/@{author}/video/{videoId}
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: guard against empty uniqueId/id producing invalid URL
When uniqueId or id is missing, return empty string instead of
a malformed URL like "https://www.tiktok.com/@/video/".
---------
Co-authored-by: Allen Song (Beyondsoft) <v-songjun@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
* chore: ignore worktree directory
* fix(pipeline): check HTTP status in fetch step
* fix(pipeline): align fetch error semantics
* fix(pipeline): use CliError and add warn logging in fetch step
- Replace bare Error with CliError('FETCH_ERROR') for consistent CLI output
- Return error status from browser evaluate instead of throwing inside it
- Add log.warn() for batch item failures in both browser and non-browser paths
* chore: remove unrelated .worktrees/ from .gitignore
* refactor(fetch): use getErrorMessage(), unify sentinel naming to __httpError
- Use project's existing getErrorMessage() utility instead of manual instanceof checks
- Rename sentinel from __fetchError to __httpError for consistency with other adapters
- Simplify sentinel structure (url already available in outer scope, no need to pass through evaluate)
- Add comment explaining why getErrorMessage() can't be used inside evaluate()
- Add comment explaining CDP error message rewriting behavior
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* feat(pixiv): add Pixiv adapter with 6 commands
Add support for Pixiv (pixiv.net) with the following commands:
- ranking: daily/weekly/monthly illustration rankings
- search: search illustrations by keyword/tag
- user: view artist profile info
- illusts: list illustrations by artist
- detail: view illustration details (tags, stats)
- download: download original-quality images
All commands use COOKIE strategy to reuse Chrome's logged-in session.
YAML adapters for simple API fetches (ranking, detail, user), TypeScript
for complex logic (search, illusts, download with Referer header).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test(pixiv): add unit tests and E2E auth failure tests
- search.test.ts: auth error, result parsing, limit, empty results (4 tests)
- illusts.test.ts: auth error, empty user, two-step fetch, limit (4 tests)
- download.test.ts: auth error, no images, Referer header, partial failure (4 tests)
- Add pixiv to vitest adapter project include list
- Add 5 pixiv commands to E2E browser-auth graceful failure tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(pixiv): correct ranking API path and YAML arg naming
- ranking: use /ranking.php?format=json (not /ajax/ranking which 404s)
- ranking: fix JSON path from data.body.contents to data.contents
- user/detail: rename hyphenated args (user-id → uid, illust-id → id)
to fix YAML template evaluation (dot access doesn't support hyphens)
All 6 commands verified working against live Pixiv API.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(pixiv): use JSON.stringify to prevent code injection in page.evaluate
Address CodeRabbit review: all user inputs (query, userId, illustId,
idsParam) passed to page.evaluate are now serialized via JSON.stringify
instead of direct string interpolation, preventing code injection in
browser context.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(pixiv): address code review feedback
- ranking.yaml: add | json filter to page/limit args for defense-in-depth
- user.yaml: guard illusts/manga/novels with typeof check for robustness
- Extract shared createPageMock to test-utils.ts, deduplicate across 3 test files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(pixiv): use minimal page mock and add download E2E test
- test-utils.ts: slim down to minimal mock (goto, evaluate, getCookies)
with overrides support, matching upstream's pragmatic mock style
- Add missing download command to E2E browser-auth graceful failure tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(pixiv): address all remaining CodeRabbit review comments
- detail.yaml: add url to columns to match description mentioning "URLs"
- All adapters: differentiate HTTP errors — 401/403 → AuthRequiredError,
404 → "not found", others → generic "request failed (HTTP N)"
- Tests: use beforeAll to cache registry lookup, avoiding repeated reads
from global singleton
- Tests: assert error type (AuthRequiredError) not just message content
- Tests: add dedicated test cases for non-auth errors (500) and 404
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(pixiv): add adapter docs and indexes
- Add pixiv.md documentation page under docs/adapters/browser/
- Update docs/adapters/index.md with pixiv entry
- Add Pixiv to sidebar in docs/.vitepress/config.mts
- Update README.md and README.zh-CN.md adapter tables
- Add pixiv to download support tables in both READMEs
Completes the documentation checklist for the pixiv adapter PR.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(pixiv): address code review findings
- Use CommandExecutionError instead of raw Error for HTTP failures
- Add page.goto() before page.evaluate() to establish browser context
- Fix search keyword double-encoding in URL construction
- Fix ranking.yaml using rating_count instead of illust_bookmark_count
- Throw on batch detail fetch failure instead of silent empty return
- Add beforeEach mock reset in download tests
- Add novels column to user.yaml output
Ensures pixiv adapter follows upstream CliError conventions and handles
edge cases correctly before submitting to upstream.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(pixiv): improve download description in READMEs
- Replace technical Referer header detail with user-facing description
- Describe what users care about: original quality and multi-page support
Technical details belong in code comments, not user-facing docs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* docs(pixiv): expand usage examples with all options
- Add ranking mode examples including R18 variants
- Add search filter examples (mode, order, pagination)
- Organize examples by command category for readability
Users need to know available options without reading source code.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(pixiv): address second round of CodeRabbit review comments
- Validate illust-id is numeric to prevent path traversal
- Move URL parsing inside per-item try block for graceful error handling
- Add auth error handling for batch detail request (consistent with step 1)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor(pixiv): extract shared pixivFetch helper, add input validation & batch support
- Create utils.ts with pixivFetch() for unified navigate + fetch + error handling
- Refactor search.ts, illusts.ts, download.ts to use pixivFetch (DRY)
- Add user-id/illust-id numeric validation in TS adapters
- Add batch pagination in illusts.ts for limit > 48 (Pixiv server limit)
- Add comment explaining Pixiv search API dual keyword requirement
- Update tests: new invalid-ID test cases, aligned mock format with pixivFetch
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
* ci: add cross-platform matrix (Linux/macOS/Windows) to build, unit-test, adapter-test
Add OS matrix with ubuntu-latest, macos-latest, and windows-latest to
the build, unit-test, and adapter-test CI jobs. This ensures cross-
platform compatibility is verified on every push and PR.
Smoke tests remain Linux-only due to xvfb dependency.
Relates to #392 (Windows plugin path issues).
* test: replace hardcoded /tmp with os.tmpdir() for Windows compatibility
Fix Windows CI failures caused by hardcoded '/tmp' paths that don't
exist on Windows. Use os.tmpdir() which returns the correct platform-
specific temp directory on all operating systems.
Files fixed:
- src/engine.test.ts: 3 occurrences (mkdtemp, discoverClis path)
- src/plugin.test.ts: 2 occurrences (getCommitHash test, mock condition)
* test: fix remaining Windows path issues in test files
- engine.test.ts: use pathToFileURL().href for dynamic import paths
(path.join produces backslashes on Windows, breaking ES module imports)
- download.test.ts: replace hardcoded '/tmp' with os.tmpdir() + path.join
* fix(plugin): resolve Windows path and symlink issues
- Replace `new URL(import.meta.url).pathname` with `fileURLToPath()` from
node:url — the former returns `/C:/Users/...` on Windows (leading slash
before drive letter), breaking path resolution for host linking and
esbuild binary lookup.
- Use junction (`'junction'`) instead of directory symlink (`'dir'`) on
Windows in linkHostOpencli — junctions don't require admin privileges,
while `fs.symlinkSync(..., 'dir')` does on Windows.
- Use `where` instead of `which` on Windows for global esbuild lookup.
All changes are platform-conditional and preserve existing Unix behavior.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix(plugin): additional Windows fixes found during UAT
- npm execFileSync needs shell:true on Windows (.cmd wrapper)
- esbuild binary is a shebang script, needs shell:true on Windows
- resolveEsbuildBin: prefer .cmd in node_modules/.bin/ on Windows
over import.meta.resolve (which returns a shebang script)
- Updated test to accept .cmd extension on Windows
Found during UAT testing on Windows 11.
* fix: handle multi-line output from 'where' on Windows
'where esbuild' on Windows can return multiple matching paths, one per
line. Take only the first match to get a valid single path for
resolveEsbuildBin().
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: ByteYue <yj976240184@gmail.com>
* test(plugin): add E2E integration tests for plugin lifecycle
Add plugin-management.test.ts covering the full plugin lifecycle
using real GitHub clone of opencli-plugin-hot-digest:
- plugin install from github:ByteYue/opencli-plugin-hot-digest
- plugin list (table and JSON formats)
- plugin update on installed plugin
- plugin uninstall with cleanup verification
- error paths: invalid source, non-existent plugin, missing args
Tests safely backup/restore existing plugin state to avoid
interfering with user's real installed plugins.
Update TESTING.md to document the new test file.
* test(plugin): isolate lifecycle e2e from user home
* fix(plugin): respect HOME env var for test isolation
The E2E tests for plugin management were failing because os.homedir()
doesn't respect the HOME environment variable. This made test isolation
impossible since all tests would use the real ~/.opencli directory.
Added getHomeDir() helper that checks process.env.HOME first before
falling back to os.homedir(). Updated readLockFile() and writeLockFile()
to use this new function.
Fixes test failures in plugin-management.test.ts where:
- plugin install would write to real home instead of temp dir
- lock file assertions would fail with ENOENT
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* feat(plugin): add lifecycle hooks API (onStartup, onBeforeExecute, onAfterExecute)
Introduce a hooks system that allows plugins to tap into opencli's
execution lifecycle without modifying core code.
New files:
- src/hooks.ts: hook registration, emission, and globalThis singleton
- src/hooks.test.ts: 10 unit tests covering registration, ordering,
error isolation, async support, and globalThis sharing
Modified files:
- src/execution.ts: emit onBeforeExecute/onAfterExecute around command execution
- src/main.ts: emit onStartup after discoverPlugins()
- src/registry-api.ts: export hooks API for plugin consumption
Example plugin: https://github.com/ByteYue/opencli-plugin-audit-log
* fix(discovery): load plugin files that register lifecycle hooks
The isCliModule() check only matched files containing 'cli(' calls,
silently skipping hook-only files like audit-hooks.ts that register
onBeforeExecute/onAfterExecute without any cli() command registration.
Renamed CLI_MODULE_PATTERN → PLUGIN_MODULE_PATTERN and extended the
regex to also match onStartup(, onBeforeExecute(, onAfterExecute(.
* fix(plugin): tighten lifecycle hook semantics
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
Add search element heuristics and label/span wrapper detection
- Add SEARCH_INDICATORS set to detect search-related elements
- Add isSearchElement function for heuristic detection
- Add hasFormControlDescendant to detect wrapped form controls
- Enhance isInteractive for label/span wrapper patterns
Ref: browser-use ClickableElementDetector research
Review: @codex
- Remove unused re-exports from registry.ts (serializeArg, serializeCommand, etc.)
- Unify FormatOptions into SnapshotOptions from types.ts; rename dom-snapshot's
SnapshotOptions to DomSnapshotOptions to avoid name collision
- Extract shared analysis.ts module from explore.ts and record.ts, eliminating
~200 lines of duplicated logic (urlToPattern, findArrayPath, inferCapabilityName,
inferStrategy, detectAuth*, classifyQueryParams)
- Merge snapshotFormatter from 7-pass to 4-pass pipeline by combining parse+filter
with ad/boilerplate subtree skipping, and merging three dedup passes into one
- Rename all CLI adapter shared files to consistent utils.ts naming
(boss/common.ts, douban/shared.ts, doubao*/common.ts, jike/shared.ts,
medium/shared.ts, sinablog/shared.ts, substack/shared.ts)
- Merge douban/shared.ts into douban/utils.ts
Remove kubectl from:
- README.md highlights and external CLI examples table
- README.zh-CN.md highlights and external CLI examples table
- src/external-clis.yaml external CLI registry
kubectl is not relevant to the opencli project scope and should not be showcased as a primary example.
The dictionary adapters commit (3d39574) introduced a duplicate
`}, 30_000);` at line 524 of public-commands.test.ts, causing
the vite:oxc transformer to fail with [PARSE_ERROR] Unexpected token
in the E2E Headed Chrome CI workflow.
* feat(browser): human-like delay system for anti-detection
Adds a framework-level delay/jitter system using log-normal distribution
to simulate natural browsing patterns, addressing issue #59 (P0).
- New `HumanDelay` class with configurable profiles (none/fast/moderate/cautious/stealth)
- Log-normal distribution for realistic delay variance (not uniform)
- Periodic "breaks" that simulate reading/thinking pauses
- Auto-injected between page.goto() navigations
- Configurable via OPENCLI_DELAY_PROFILE env var
- Boss search adapter migrated from hardcoded jitter to framework delay
- 10 unit tests covering all profiles and edge cases
Real-world validation against a major job board (cookie-authenticated,
aggressive bot detection):
| Scenario | Without jitter | With jitter |
|-----------------------|--------------------|--------------------|
| 50 detail pages | ✅ OK | ✅ OK |
| 200 detail pages | ❌ Banned (code 32) | ✅ OK |
| 850 requests over 5h | N/A (banned early) | ✅ Zero detection |
| 4-day sustained crawl | N/A | ✅ 1800+ records |
Closes#59
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: disable human delay in CI environment to prevent E2E timeouts
In CI environments (CI=true), resolveProfile() now defaults to the
'none' profile instead of 'moderate'. This prevents the 1-8s per-
navigation delay from causing E2E test timeouts (30s limit).
Users can override this by setting OPENCLI_DELAY_PROFILE explicitly.
---------
Co-authored-by: toolmanlab <toolmanlab@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
* fix(grok): preserve conversation across repeated ask calls (#330)
The adapter unconditionally navigated to grok.com/ on every invocation,
destroying the existing conversation URL even when --new was not passed.
Since the browser daemon already reuses the same Chrome tab, skipping
navigation lets the tab stay on the current chat thread.
- Only navigate to grok.com/ when --new is true or tab is not on grok.com
- Add tryStartFreshChat to the default path's --new branch (was dead code)
- Add isOnGrok helper with hostname-based domain matching
- Add unit tests for isOnGrok
* test(grok): add adapter to vitest project config
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
The search command defined its argument as `query` but referenced
`args.keyword`, causing the search term to be undefined.
Closes#334
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add "Why opencli?" section and comparison guide (#238)
- Add "Why opencli?" section to README.md and README.zh-CN.md
(between Highlights and Prerequisites)
- Add docs/comparison.md with 5-scenario honest evaluation
- Add Comparison entry to VitePress sidebar
* docs: refine positioning — use approximate numbers, emphasize broad coverage
- Replace specific counts (300+, 55, 20+) with approximate descriptions
- Emphasize broad global + Chinese platform coverage instead of singling out Chinese sites
- Fix Firecrawl description to mention self-hosted option
- Replace "sub-second" / "milliseconds" with accurate "seconds" / "fast deterministic"
- Add testing and AI workflow to Further Reading links
- Add "easy to extend" point to strengths
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
The favorite command was using up_mid: 0 which returns empty results. Now it correctly fetches the current user UID using getSelfUid().
Co-authored-by: 章晖 <zhanghui@MacBook-Pro.local>
* feat(linkedin): add timeline feed command
* test(linkedin): add timeline adapter unit tests
Add shape tests and utility function tests for the new timeline command.
Include linkedin in the vitest adapter project config.
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
The C2 fix in PR #337 added a null-page guard after lazy-loading TS
modules, but it threw unconditionally — breaking all browser:false
commands (bloomberg, apple-podcasts, google, yollomi, etc.) that
use func() with a null page. Guard now checks updated.browser !== false.
Also fixes apple-podcasts top E2E flake: when the command times out on
CI, stderr is empty and the guard didn't catch it.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add generic `web read` command for any URL → Markdown
Adds a new `opencli web read --url <any-url>` command that fetches any
web page and exports it as clean Markdown with optional image download.
Uses browser-side DOM heuristics for content extraction:
1. <article> element
2. [role="main"] element
3. <main> element
4. Largest text-dense block fallback
Pipes through the existing article-download pipeline (Turndown + image
localization), so it inherits code block handling, frontmatter generation,
and concurrent image downloading for free.
Tested on: Anthropic blog, OpenAI blog, general news sites.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: improve web read dedup for sites with duplicated DOM paragraphs
Anthropic's blog renders each paragraph twice (a normal version + a
line-broken animation version). The previous substring-based dedup
missed these because whitespace differences changed string lengths.
Fix: compare texts after stripping ALL whitespace, and keep the
version with more proper spacing (more spaces = better formatted).
Result on Anthropic blog: 98.4KB → 53.7KB (45% reduction).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Harrison <harrison@HarrisondeMacBook-Pro.local>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- I1: Log pre-navigation failures in debug mode instead of silently swallowing
- I2: Validate env var timeout values, fallback on NaN/negative
- I4: Guard against indexOf returning -1 for unknown strategies in cascade
- I5: Fix shouldReplaceManifestEntry returning true for same-type entries
- I6: Prevent infinite loop in parseTsArgsBlock cursor advancement
- I7: Skip redundant Page.enable calls in CDP goto
- I8: Fix wait({time:0}) being treated as falsy
- I10: Warn when cookiesFile path doesn't exist before fallback
- I11: Sanitize tab/newline chars in cookie name/value for Netscape format
- I12: Use DEFAULT_DAEMON_PORT constant instead of hardcoded port in error
- I15: Log npm install failures in plugin lifecycle instead of swallowing
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1. execution.ts: Guard lazy-loaded func commands against null page — if a
lazy module incorrectly requires browser context, throw a clear error
instead of a cryptic TypeError on page.goto().
2. daemon.ts: Fix readBody race condition — add aborted flag to prevent
req.destroy() from triggering both reject (via error) and resolve
(via end event) on the same Promise, which could process truncated data.
3. browser/cdp.ts: Prevent CDPBridge.connect() reentry — throw if already
connected instead of silently leaking the previous WebSocket and its
message handlers.
4. interceptor.ts: Store intercept pattern in a separate global variable
so subsequent installInterceptor calls with different patterns update
the match condition without being blocked by the patchGuard.
5. record.ts: Always call cleanupEnter() after Promise.race — previously
only called in the timeout path, leaving readline open when user pressed
Enter, potentially blocking process exit. Also removed unused enterRace.
6. generate.ts: Fix undefined entering String.includes() — when c.name is
undefined, toLowerCase() returns undefined which gets coerced to the
string "undefined" by includes(), causing false positive matches.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
AstroHan