mirror of https://github.com/jackwener/OpenCLI.git synced 2026-09-14 18:25:42 +08:00

Files

T

jackwener 34a9bff2b3 refactor: eliminate code duplication, improve type safety, add tests

- NEW: src/interceptor.ts — unified XHR/Fetch interceptor (was duplicated 3x)
- NEW: src/version.ts — centralized PKG_VERSION (was duplicated 2x)
- NEW: src/constants.ts — shared VOLATILE_PARAMS, FIELD_ROLES etc.
- NEW: src/engine.test.ts, src/registry.test.ts — 14 new unit tests

- browser.ts: use shared normalizeEval, interceptor, withTimeoutMs, PKG_VERSION
- intercept.ts, tap.ts: use shared interceptor generators
- cascade.ts: extract shared buildFetchProbeJs (90% dedup)
- engine.ts: use InternalCliCommand (no more 'as any' casts)
- executor.ts: remove all 15 'as StepHandler' type casts
- runtime.ts: add withTimeoutMs, IBrowserFactory interface
- registry.ts: add InternalCliCommand type for internal fields
- validate.ts: add pipeline step name validation
- output.ts: remove 'null as any' and '.filter(() => true)' hacks
- explore.ts, synthesize.ts: use shared constants
- docs: fix V2EX commands (3→6), SKILL.md version, verify example

Tests: 88 passed (was 74), tsc --noEmit: 0 errors

2026-03-16 13:22:12 +08:00

12 KiB

Raw Permalink Blame History

name, description, version, author, tags

name

description

version

author

OpenCLI

Make any website your CLI. Reuse Chrome login, zero risk, AI-powered discovery.

Caution

AI Agent 必读：创建或修改任何适配器之前，你必须先阅读 CLI-EXPLORER.md！ 该文档包含完整的 API 发现工作流（必须使用 Playwright MCP Bridge 浏览器探索）、5 级认证策略决策树、平台 SDK 速查表、tap 步骤调试流程、分页 API 模板、级联请求模式、以及常见陷阱。 本文件（SKILL.md）仅提供命令参考和简化模板，不足以正确开发适配器。

Install & Run

# npm global install (recommended)
npm install -g @jackwener/opencli
opencli <command>

# Or from source
cd ~/code/opencli && npm install
npx tsx src/main.ts <command>

# Update to latest
npm update -g @jackwener/opencli

Prerequisites

Browser commands require:

Chrome browser running (logged into target sites)
Playwright MCP Bridge extension installed and configured

Note

: You must be logged into the target website in Chrome before running commands. Tabs opened during command execution are auto-closed afterwards.

Public API commands (hackernews, github search, v2ex) need no browser.

Commands Reference

Data Commands

# Bilibili (browser)
opencli bilibili hot --limit 10          # B站热门视频
opencli bilibili search --keyword "rust"  # 搜索视频
opencli bilibili me                       # 我的信息
opencli bilibili favorite                 # 我的收藏
opencli bilibili history --limit 20       # 观看历史
opencli bilibili feed --limit 10          # 动态时间线
opencli bilibili user-videos --uid 12345  # 用户投稿
opencli bilibili subtitle --bvid BV1xxx   # 获取视频字幕 (支持 --lang zh-CN)
opencli bilibili dynamic --limit 10       # 动态
opencli bilibili ranking --limit 10       # 排行榜
opencli bilibili following --limit 20     # 我的关注列表 (支持 --uid 查看他人)

# 知乎 (browser)
opencli zhihu hot --limit 10             # 知乎热榜
opencli zhihu search --keyword "AI"      # 搜索
opencli zhihu question --id 34816524     # 问题详情和回答

# 小红书 (browser)
opencli xiaohongshu search --keyword "美食"  # 搜索笔记
opencli xiaohongshu notifications             # 通知（mentions/likes/connections）
opencli xiaohongshu feed --limit 10           # 推荐 Feed
opencli xiaohongshu me                         # 我的信息
opencli xiaohongshu user --uid xxx             # 用户主页

# 雪球 Xueqiu (browser)
opencli xueqiu hot-stock --limit 10      # 雪球热门股票榜
opencli xueqiu stock --symbol SH600519   # 查看股票实时行情
opencli xueqiu watchlist                 # 获取自选股/持仓列表
opencli xueqiu feed                      # 我的关注 timeline
opencli xueqiu hot --limit 10            # 雪球热榜
opencli xueqiu search --keyword "特斯拉"  # 搜索

# GitHub (public)
opencli github search --keyword "cli"    # 搜索仓库

# Twitter/X (browser)
opencli twitter trending --limit 10      # 热门话题
opencli twitter bookmarks --limit 20     # 获取收藏的书签推文
opencli twitter search --keyword "AI"    # 搜索推文
opencli twitter profile --username elonmusk  # 用户资料
opencli twitter timeline --limit 20      # 时间线

# Reddit (browser)
opencli reddit hot --limit 10            # 热门帖子
opencli reddit hot --subreddit programming  # 指定子版块
opencli reddit frontpage --limit 10      # 首页
opencli reddit search --keyword "AI"     # 搜索
opencli reddit subreddit --name rust     # 子版块浏览

# V2EX (public + browser)
opencli v2ex hot --limit 10              # 热门话题
opencli v2ex latest --limit 10           # 最新话题
opencli v2ex topic --id 1024             # 主题详情
opencli v2ex daily                       # 每日签到 (browser)
opencli v2ex me                          # 我的信息 (browser)
opencli v2ex notifications --limit 10    # 通知 (browser)

# Hacker News (public)
opencli hackernews top --limit 10        # Top stories

# BBC (public)
opencli bbc news --limit 10             # BBC News RSS headlines

# 微博 (browser)
opencli weibo hot --limit 10            # 微博热搜

# BOSS直聘 (browser)
opencli boss search --query "AI agent"  # 搜索职位

# YouTube (browser)
opencli youtube search --query "rust"   # 搜索视频

# Yahoo Finance (browser)
opencli yahoo-finance quote --symbol AAPL  # 股票行情

# Reuters (browser)
opencli reuters search --query "AI"     # 路透社搜索

# 什么值得买 (browser)
opencli smzdm search --keyword "耳机"    # 搜索好价

# 携程 (browser)
opencli ctrip search --query "三亚"      # 搜索目的地

Management Commands

opencli list                # List all commands
opencli list --json         # JSON output
opencli list -f yaml        # YAML output
opencli validate            # Validate all CLI definitions
opencli validate bilibili   # Validate specific site

AI Agent Workflow

# Deep Explore: network intercept → response analysis → capability inference
opencli explore <url> --site <name>

# Synthesize: generate evaluate-based YAML pipelines from explore artifacts
opencli synthesize <site>

# Generate: one-shot explore → synthesize → register
opencli generate <url> --goal "hot"

# Strategy Cascade: auto-probe PUBLIC → COOKIE → HEADER
opencli cascade <api-url>

# Explore with interactive fuzzing (click buttons to trigger lazy APIs)
opencli explore <url> --auto --click "字幕,CC,评论"

# Verify: validate adapter definitions
opencli verify

Output Formats

All built-in commands support --format / -f with table, json, yaml, md, and csv. The list command supports the same formats and also keeps --json as a compatibility alias.

opencli list -f yaml            # YAML command registry
opencli bilibili hot -f table   # Default: rich table
opencli bilibili hot -f json    # JSON (pipe to jq, feed to AI agent)
opencli bilibili hot -f yaml    # YAML (readable structured output)
opencli bilibili hot -f md      # Markdown
opencli bilibili hot -f csv     # CSV

Verbose Mode

opencli bilibili hot -v         # Show each pipeline step and data flow

Creating Adapters

Tip

快速模式：如果你只想为一个具体页面生成一个命令，直接看 CLI-ONESHOT.md。只需要一个 URL + 一句话描述，4 步搞定。

Important

完整模式 — 在写任何代码之前，先阅读 CLI-EXPLORER.md。 它包含：① AI Agent 浏览器探索工作流（必须用 Playwright MCP 抓包验证 API）② 认证策略决策树 ③ 平台 SDK（如 Bilibili 的 apiGet/fetchJson）④ YAML vs TS 选择指南 ⑤ tap 步骤调试方法 ⑥ 级联请求模板 ⑦ 常见陷阱表。 下方仅为简化模板参考，直接使用极易踩坑。

YAML Pipeline (declarative, recommended)

Create src/clis/<site>/<name>.yaml:

site: mysite
name: hot
description: Hot topics
domain: www.mysite.com
strategy: cookie        # public | cookie | header | intercept | ui
browser: true

args:
  limit:
    type: int
    default: 20
    description: Number of items

pipeline:
  - navigate: https://www.mysite.com

  - evaluate: |
      (async () => {
        const res = await fetch('/api/hot', { credentials: 'include' });
        const d = await res.json();
        return d.data.items.map(item => ({
          title: item.title,
          score: item.score,
        }));
      })()

  - map:
      rank: ${{ index + 1 }}
      title: ${{ item.title }}
      score: ${{ item.score }}

  - limit: ${{ args.limit }}

columns: [rank, title, score]

For public APIs (no browser):

strategy: public
browser: false

pipeline:
  - fetch:
      url: https://api.example.com/hot.json
  - select: data.items
  - map:
      title: ${{ item.title }}
  - limit: ${{ args.limit }}

TypeScript Adapter (programmatic)

Create src/clis/<site>/<name>.ts. It will be automatically dynamically loaded (DO NOT manually import it in index.ts):

import { cli, Strategy } from '../../registry.js';

cli({
  site: 'mysite',
  name: 'search',
  strategy: Strategy.INTERCEPT, // Or COOKIE
  args: [{ name: 'keyword', required: true }],
  columns: ['rank', 'title', 'url'],
  func: async (page, kwargs) => {
    await page.goto('https://www.mysite.com/search');
    
    // Inject native XHR/Fetch interceptor hook
    await page.installInterceptor('/api/search');
    
    // Auto scroll down to trigger lazy loading
    await page.autoScroll({ times: 3, delayMs: 2000 });
    
    // Retrieve intercepted JSON payloads
    const requests = await page.getInterceptedRequests();
    
    let results = [];
    for (const req of requests) {
      results.push(...req.data.items);
    }
    return results.map((item, i) => ({
      rank: i + 1, title: item.title, url: item.url,
    }));
  },
});

When to use TS: XHR interception (page.installInterceptor), infinite scrolling (page.autoScroll), cookie extraction, complex data transforms (like GraphQL unwrapping).

Pipeline Steps

Step	Description	Example
`navigate`	Go to URL	`navigate: https://example.com`
`fetch`	HTTP request (browser cookies)	`fetch: { url: "...", params: { q: "..." } }`
`evaluate`	Run JavaScript in page	`evaluate: \| (async () => { ... })()`
`select`	Extract JSON path	`select: data.items`
`map`	Map fields	`map: { title: "${{ item.title }}" }`
`filter`	Filter items	`filter: item.score > 100`
`sort`	Sort items	`sort: { by: score, order: desc }`
`limit`	Cap result count	`limit: ${{ args.limit }}`
`intercept`	Declarative XHR capture	`intercept: { trigger: "navigate:...", capture: "api/hot" }`
`tap`	Store action + XHR capture	`tap: { store: "feed", action: "fetchFeeds", capture: "homefeed" }`
`snapshot`	Page accessibility tree	`snapshot: { interactive: true }`
`click`	Click element	`click: ${{ ref }}`
`type`	Type text	`type: { ref: "@1", text: "hello" }`
`wait`	Wait for time/text	`wait: 2` or `wait: { text: "loaded" }`
`press`	Press key	`press: Enter`

Template Syntax

# Arguments with defaults
${{ args.keyword }}
${{ args.limit | default(20) }}

# Current item (in map/filter)
${{ item.title }}
${{ item.data.nested.field }}

# Index (0-based)
${{ index }}
${{ index + 1 }}

5-Tier Authentication Strategy

Tier	Name	Method	Example
1	`public`	No auth, Node.js fetch	Hacker News, V2EX
2	`cookie`	Browser fetch with `credentials: include`	Bilibili, Zhihu
3	`header`	Custom headers (ct0, Bearer)	Twitter GraphQL
4	`intercept`	XHR interception + store mutation	小红书 Pinia
5	`ui`	Full UI automation (click/type/scroll)	Last resort

Environment Variables

Variable	Default	Description
`OPENCLI_BROWSER_CONNECT_TIMEOUT`	30	Browser connection timeout (sec)
`OPENCLI_BROWSER_COMMAND_TIMEOUT`	45	Command execution timeout (sec)
`OPENCLI_BROWSER_EXPLORE_TIMEOUT`	120	Explore timeout (sec)
`PLAYWRIGHT_MCP_EXTENSION_TOKEN`	—	Auto-approve extension connection

Troubleshooting

Issue	Solution
`npx not found`	Install Node.js: `brew install node`
`Timed out connecting to browser`	1) Chrome must be open 2) Install MCP Bridge extension and configure token
`Target page context` error	Add `navigate:` step before `evaluate:` in YAML
Empty table data	Check if evaluate returns JSON string (MCP parsing) or data path is wrong

12 KiB Raw Permalink Blame History Unescape Escape