35 Commits

Author SHA1 Message Date
AlexYue 257e9ec4f8 feat: add E2E testing infrastructure with real Chrome in CI (#1)
# feat: add E2E testing infrastructure with real Chrome in CI

## What

Establish a comprehensive E2E testing framework for opencli using **real Chrome + xvfb virtual display** in GitHub Actions CI.

## Changes

### E2E Test Suite (~52 test cases)
- `public-commands.test.ts` — Public API commands (hackernews, v2ex)
- `browser-public.test.ts` — Browser commands for public data across all 18 sites (21 tests)
- `browser-auth.test.ts` — Graceful failure verification for login-required commands (14 tests)
- `management.test.ts` — Full coverage of management commands (list/validate/verify/version/help)
- `output-formats.test.ts` — Output format validation (json/yaml/csv/md)
- `smoke/api-health.test.ts` — Scheduled API health checks

### Auto-detect Browser Mode
- `buildMcpArgs` automatically selects mode based on `PLAYWRIGHT_MCP_EXTENSION_TOKEN`:
  - Token present → `--extension` (local user, connects to logged-in Chrome)
  - Token absent → standalone (CI launches its own browser)
- No extra environment variables needed

### CI Pipeline
- `e2e-headed.yml` — Real Chrome via `browser-actions/setup-chrome` + `xvfb-run` in headed mode
- `ci.yml` — build + unit-test (2 shards) + smoke-test (scheduled/manual)
- Browser commands that return empty data due to geo-blocking or bot detection gracefully warn + pass without blocking CI

### Documentation
- New `TESTING.md` — Architecture, coverage, local setup, how to add tests, CI explanation
- Updated `README.md` — Added Testing section with quick-start commands
2026-03-16 16:22:00 +08:00
jackwener c77c8a8e3a feat: add Coupang search and add-to-cart adapters
Add browser-backed Coupang adapters for search and add-to-cart workflows.
- coupang search: multi-strategy data collection (API/JSON-LD/bootstrap/DOM), structured fields (price, rating, rocket, delivery), pagination and rocket filter support
- coupang add-to-cart: logged-in browser session reuse, stops before checkout
- coupang.ts: comprehensive data normalization layer with badge/rocket/delivery mapping
- browser-tab.ts: withTemporaryTab utility for isolated tab operations
- Unit tests for core normalization functions

Co-authored-by: CodeBBakGoSu <127713112+CodeBBakGoSu@users.noreply.github.com>
2026-03-16 13:30:42 +08:00
jackwener 34a9bff2b3 refactor: eliminate code duplication, improve type safety, add tests
- NEW: src/interceptor.ts — unified XHR/Fetch interceptor (was duplicated 3x)
- NEW: src/version.ts — centralized PKG_VERSION (was duplicated 2x)
- NEW: src/constants.ts — shared VOLATILE_PARAMS, FIELD_ROLES etc.
- NEW: src/engine.test.ts, src/registry.test.ts — 14 new unit tests

- browser.ts: use shared normalizeEval, interceptor, withTimeoutMs, PKG_VERSION
- intercept.ts, tap.ts: use shared interceptor generators
- cascade.ts: extract shared buildFetchProbeJs (90% dedup)
- engine.ts: use InternalCliCommand (no more 'as any' casts)
- executor.ts: remove all 15 'as StepHandler' type casts
- runtime.ts: add withTimeoutMs, IBrowserFactory interface
- registry.ts: add InternalCliCommand type for internal fields
- validate.ts: add pipeline step name validation
- output.ts: remove 'null as any' and '.filter(() => true)' hacks
- explore.ts, synthesize.ts: use shared constants
- docs: fix V2EX commands (3→6), SKILL.md version, verify example

Tests: 88 passed (was 74), tsc --noEmit: 0 errors
2026-03-16 13:22:12 +08:00
jackwener 76eefed83d docs: add opencli doctor hint after token setup 2026-03-16 12:47:59 +08:00
jackwener 95fb841fc7 refactor: remove all CDP code, keep extension-only mode
- Remove CDP auto-discovery (discoverChromeEndpoint, isCdpApiAvailable, isPortReachable)
- Remove CDP readiness probe and autoAllowCdpDialog
- Remove forceExtension plumbing from registry, runtime, main, and adapters
- Remove CDP diagnostics from doctor (extractTokenViaCdp, remoteDebugging fields)
- Simplify connect() to always use --extension mode
- Remove CDP env vars (OPENCLI_USE_CDP, OPENCLI_CDP_ENDPOINT, OPENCLI_FORCE_EXTENSION)
- Update README.md, README.zh-CN.md, SKILL.md to remove CDP sections
- Update tests to remove CDP-related assertions

Current state preserved in extension-and-cdp branch.
2026-03-16 12:44:09 +08:00
jackwener e2192a2e8d docs: rename CLI-CREATOR → CLI-EXPLORER, add CLI-ONESHOT refs to READMEs
- Rename CLI-CREATOR.md → CLI-EXPLORER.md (explorer better reflects its purpose)
- Update all references across SKILL.md, README.md, README.zh-CN.md, CLI-ONESHOT.md
- Add quick-mode (CLI-ONESHOT) and full-mode (CLI-EXPLORER) entries in both READMEs
2026-03-15 23:58:10 +08:00
jackwener 5a676b9f4b docs: clarify that both env var and MCP config are required for token 2026-03-15 22:12:59 +08:00
jackwener 59a02d3a5d feat(twitter): implement like adapter 2026-03-15 21:53:42 +08:00
jackwener 1aaafed2db feat(twitter): implement post, reply, delete, following, followers and notifications adapters 2026-03-15 21:42:48 +08:00
jackwener ee6669f45d feat(core): support YAML format output and optimize manifest builder 2026-03-15 21:05:36 +08:00
jackwener 4f570fa75e docs: add instruction to get token by clicking extension icon 2026-03-15 20:08:30 +08:00
jackwener 730285c0ca docs: set MCP Bridge as primary connection method, CDP as fallback 2026-03-15 20:07:32 +08:00
jackwener c5915d9ee3 docs: clarify that MCP token needs to be configured in both MCP config and env var 2026-03-15 20:06:32 +08:00
jackwener b7c8067bc9 docs: restore playwright MCP token configuration guide 2026-03-15 20:05:08 +08:00
jackwener c28ccd4b23 docs: huge improvement on README structure, badges and readability 2026-03-15 19:59:02 +08:00
jackwener 9c78ffdd39 chore: change license to BSD-3-Clause 2026-03-15 19:54:34 +08:00
jackwener a6e9c3b611 feat(browser): enhance CDP auto-discovery with port probe & opt-in priority
- Add TCP port reachability check to handle stale DevToolsActivePort files
- Make discoverChromeEndpoint() async with port probe before returning
- Change connection priority: extension mode is default, CDP is opt-in
  - OPENCLI_CDP_ENDPOINT: explicit CDP endpoint
  - OPENCLI_USE_CDP=1: enable DevToolsActivePort auto-discovery
  - Default: --extension mode (Playwright MCP Bridge)
- Add verbose logging for CDP discovery when -v flag is used
- Fix indentation inconsistency in connect() method
- Update README.md, README.zh-CN.md, SKILL.md with new connection docs
2026-03-15 19:44:27 +08:00
jackwener da155e96f0 feat: add forceExtension to bypass CDP detection for anti-bot sites
- Added forceExtension flag to CliCommand, CliOptions, browserSession, and PlaywrightMCP.connect()
- BOSS Zhipin detects CDP mode and drops connections; marked boss/search with forceExtension: true
- Also supports OPENCLI_FORCE_EXTENSION=1 env var for global override
- Fixed async discoverChromeEndpoint() placement outside Promise constructor
- Enhanced boss search with city names, experience, degree, salary, etc.
2026-03-15 19:39:39 +08:00
jackwener 3ac5aa3023 docs: reduce excessive emoji usage across markdown files 2026-03-15 19:30:32 +08:00
jackwener 9a6e157af6 docs: add strong AI Agent prompts to read CLI-CREATOR.md before creating adapters 2026-03-15 19:28:05 +08:00
jackwener 7ec04c66cb docs: deep review fixes — sync command tables to actual 47 commands
- Command count 46 → 47 (added following)
- bilibili row: add 'following' command
- github row: remove non-existent 'trending', change mode to Public
- SKILL.md: add following, xueqiu hot/search commands
- Fix github section from 'trending=browser' to 'public'
2026-03-15 18:41:14 +08:00
jackwener ee2e0a3bfc docs: update command tables to 46 commands, add subtitle/dynamic/ranking/frontpage/subreddit/profile/timeline/me/user, update SKILL.md version 2026-03-15 18:16:26 +08:00
jackwener 790c94d644 feat: Add Xueqiu site with 6 new commands and a Twitter bookmarks command, updating documentation. 2026-03-15 16:03:45 +08:00
jackwener 829172fc59 feat: introduce browser interception and auto-scroll utilities, enable dynamic loading of TypeScript CLIs, and add a JSON template filter 2026-03-15 15:42:19 +08:00
jackwener 92c23e8f9b docs: fix CLI creator guide links in READMEs 2026-03-15 08:59:13 +08:00
jackwener 0fd9adf0f5 feat: add 8 new sites — bbc, weibo, boss, yahoo-finance, reuters, smzdm, ctrip, youtube
New adapters ported from bb-sites:
- bbc/news: BBC News RSS headlines (public API)
- weibo/hot: 微博热搜 (browser)
- boss/search: BOSS直聘搜索职位 (browser)
- yahoo-finance/quote: 股票行情 (browser)
- reuters/search: 路透社新闻搜索 (browser)
- smzdm/search: 什么值得买搜索好价 (browser)
- ctrip/search: 携程旅行搜索 (browser)
- youtube/search: YouTube视频搜索 (browser)

Also: build script adds clean-yaml step, docs updated to 28 commands / 16 sites
2026-03-15 08:46:13 +08:00
jackwener 999d340a34 refactor: remove xiaohongshu me, add lessons learned, fix docs
- Remove xiaohongshu/me (profile page async loading too unreliable)
- CLI-CREATOR: add 4 lessons to 常见陷阱 (stale dist YAML, TS IIFE
  format, async polling, YAML vs TS threshold)
- Fix build: clean-yaml before copy to prevent stale overrides
- Fix docs: MCP token is configured via MCP config, not env var
- Update all command lists (README, README.zh-CN, SKILL.md)
2026-03-15 08:10:40 +08:00
jackwener fd94a5d27b fix: auto-close tabs after command, add login requirement reminders
- browser.ts: close all tabs opened during session (not just extension tabs)
  uses _initialTabCount to restore Chrome to pre-command state
- README/README.zh-CN: add ⚠️ login requirement warning with examples
- SKILL.md: add login note and tab auto-close mention
2026-03-15 07:57:12 +08:00
jackwener 129df91247 docs: fix site count from 13 to 8 in READMEs 2026-03-15 07:53:03 +08:00
jackwener 9df96db598 refactor: pipeline error propagation, me.yaml→me.ts, docs overhaul
Code:
- pipeline.ts: warn on stderr when step returns {error} object
- xiaohongshu/me.yaml → me.ts (evaluate code belongs in TS)
- index.ts: register xiaohongshu/me.ts

Docs:
- CLI-CREATOR: YAML vs TS decision tree, tap step docs with
  real example, 'explore store first' debugging workflow
- SKILL.md: add tap to Pipeline Steps table, update XHS commands
- README + README.zh-CN: update xiaohongshu command list
2026-03-15 07:51:20 +08:00
jackwener 7c88c603c8 docs: add CLI-CREATOR.md + fix MCP token config instructions
- CLI-CREATOR.md: comprehensive adapter development guide with
  5-tier strategy decision tree, YAML/TS templates, XHR interception
  pattern, common pitfalls, and AI Agent auto-generation workflow
- Fix all docs: MCP Bridge connects via PLAYWRIGHT_MCP_EXTENSION_TOKEN
  in MCP config (not by clicking to approve)
2026-03-15 07:23:16 +08:00
jackwener ba620f4498 feat: add tap pipeline step and probe command
- tap: declarative Store Action Bridge (Pinia/Vuex)
  - fetch + XHR dual interception, auto-cleanup
  - 4-line YAML replaces 30+ line JS monkey-patching
- probe: AI-powered site intelligence command
  - auto-discovers Pinia/Vuex stores and actions
  - correlates stores → API endpoints → capabilities
  - outputs stores.json artifact
- synthesize: generates tap-based pipelines for store-action sites
- xiaohongshu: feed.yaml and search.yaml using tap
- build: copy YAML adapters to dist/
2026-03-15 07:20:00 +08:00
jackwener d35d78d836 docs: polish README and SKILL for v0.1.0
README:
- Cleaner structure, accurate 25 commands / 13 sites
- Correct strategy labels (v2ex=public, not cookie)
- Explore artifacts documentation
- Better formatting and English/Chinese mix

SKILL:
- 5-tier authentication strategy table
- Public API YAML pattern example
- TS escape hatch guidance
- intercept pipeline step docs
- Troubleshooting section
2026-03-15 07:18:17 +08:00
jackwener 1ee4919a24 docs: update README and SKILL with new features
- Fix directory path: ai-native-cli → opencli
- Add reddit, zhihu/question to command tables
- Add AI Agent workflow section with cascade command
- Add intercept pipeline step to reference
- Add reddit examples to SKILL.md
2026-03-15 07:01:01 +08:00
jackwener d4133f955b feat: opencli TypeScript rewrite with 20 commands across 11 sites 2026-03-15 06:23:17 +08:00