Restore the original icons (commit b2fa7da) that were replaced by the
v1.6.8 "refresh icons" change in e9867dc. Per user feedback, the original
neon `>_` design read more clearly and was preferred over the abstract
arrow + dash variant.
Reverts only the four icon PNGs (16/32/48/128); manifest, popup, and
extension version stay where they are.
* fix(chatgpt-app): use AX send flow and support zh-CN generating state
* fix(chatgpt-app): fail fast on stale AX send path
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* feat(weixin): add publish (create draft with cover) and drafts (list drafts)
Closes#441
* fix(weixin): rename publish to create-draft to match issue #441 proposal
* fix(weixin): fail fast on draft auth and empty states
* test(weixin): align adapter imports with repo style
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* fix(deepseek): fix history titles and resume conversation on ask
- history: use link.innerText instead of link.querySelector('div') for
title extraction. DeepSeek changed sidebar DOM; the first child div
is now an empty ds-focus-ring element, causing all titles to show as
(untitled).
- ask: when workspace is recycled (idle timeout) and --new is false
(default), click the most recent sidebar conversation link to resume
it instead of staying on the blank new-chat page. Skip model
selection when inside an existing conversation since the selector is
only rendered on the new-chat page.
- ensureOnDeepSeek: return boolean indicating whether navigation
occurred, so callers can react to workspace recycling.
Closes#1149
* fix(deepseek): fail fast on explicit model resume
* fix(cli): expose only explicit option sources
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* fix(sinafinance): match stock symbol in addition to name
The scoring function only compared user input against the Chinese
display name (p[4] from suggest API), so searching "AAPL" matched
"AAPLU" (score 0.8) over Apple Inc. whose name field is "苹果"
(score 0). Check the symbol field first for exact and partial matches.
Fixes#1157
* docs(sinafinance): add missing commands to adapter index
The index table only listed `news` for sinafinance. Added the other
three commands (`rolling-news`, `stock`, `stock-rank`) and updated
the mode from Public to hybrid since rolling-news and stock-rank
require a browser.
Fixes#1156
* test(sinafinance): lock stock symbol matching
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* feat(web,download): absorb #1048 media + --stdout into web read
Distill the useful pieces of the abandoned PR #1048 (`web md`) into the
existing shared pipeline instead of introducing a parallel command:
- Turndown rules for <video> / <audio> / <iframe>. Video and audio are
emitted as inline HTML so renderers that support it keep playback,
and iframes degrade to markdown links (title + src) so embedded
content (YouTube, CodePen, …) stays reachable. `iframe` moves out of
STRIPPED_TAGS since it's now handled explicitly.
- `stdout` option on ArticleDownloadOptions: writes the full markdown
to process.stdout, skips image download + mkdir + file write, and
reports saved='-'. Remote image URLs stay intact so piped output is
self-contained.
- `web read --stdout` wires the above through.
- Lazy-load src rewrite: the extractor now promotes data-src /
data-original / data-lazy-src / data-srcset onto `src` before the
HTML is frozen, so the markdown body and the image-download list
reference the same URL (previously a page with placeholder.gif +
data-src produced broken image links in the output).
Nothing in #1048 that overlapped with the already-merged #1143
hardening was kept — no new Readability wiring, no duplicate Turndown
config, no new command.
* fix(web): keep stdout streaming output clean
* fix(tests): update iframe e2e assertion and drop relative src import
- article-extract e2e fixture test: iframe now converts to a markdown
link instead of being stripped, so assert the YouTube embed link
survives rather than asserting its absence.
- clis/web/read.test.js: replace vi.importActual('../../src/registry.js')
with a direct __test__.command export from read.js; the relative
import into src/ tripped the package-exports adapter guardrail.
* fix(deepseek): separate thinking process from response in --think mode (#1124)
When --think is enabled, the response now includes separate fields:
- response: clean final answer only
- thinking: chain-of-thought reasoning content
- thinking_time: time spent thinking (e.g. '1')
Supports both English ('Thought for X seconds') and Chinese
('已思考（用时 X 秒）') thinking header patterns.
Fixes#1124
* chore: regenerate cli-manifest.json
* fix(deepseek): DOM-level think/response separation, dynamic columns
Blocker 1: Replace fragile split(/\n\n+/) heuristic in parseThinkingResponse()
with DOM-level extraction in waitForResponse(). The page evaluate now queries
distinct DOM nodes (.ds-markdown--think vs .ds-markdown) for thinking and
response content. The text-level parser falls back to treating everything
after the header as thinking (no split), avoiding silent corruption of
multi-paragraph content.
Blocker 2: Remove static columns declaration from askCommand. The renderer
infers columns from row keys, so non-think output only shows 'response'
while think output shows all three columns.
Tests added for multi-paragraph thinking, multi-paragraph answer, and
non-think column regression guard.
* chore: regenerate cli-manifest.json
* feat(download): harden HTML→Markdown pipeline
Inspired by the MD-This-Page / markdown-viewer-extension analysis, tighten
the shared article→Markdown converter used by zhihu/weixin/web adapters:
- enable turndown-plugin-gfm (tables, strikethrough, task lists)
- strip script/style/noscript/iframe/canvas/form/button/dialog unconditionally
- strip SVG via a dedicated rule (not in HTMLElementTagNameMap)
- drop base64 data-URI images so they don't bloat .md output
- post-process: collapse NBSP, lone bullet/middle-dot residue,
trailing whitespace, and 3+ blank lines
- frontmatter shape guarantees ≤2 consecutive newlines even when
some metadata fields are absent
Adds a minimal local .d.ts for turndown-plugin-gfm and 6 new tests
covering GFM conversion, tag stripping, base64 drop, and whitespace cleanup.
* fix(download): emit canonical markdown strikethrough
* feat(download,browser): finish article pipeline polish
Per the follow-up from the MD-This-Page / markdown-viewer-extension
analysis, land the remaining items in the same PR instead of splitting:
article-download.ts
- extend STRIPPED_TAGS with header/footer/nav/aside (page chrome; the
article's title/author/publishTime are supplied as separate fields on
ArticleData, so duplicated DOM is redundant)
- new option ArticleDownloadOptions.cleanSelectors — per-adapter CSS
selector list removed before conversion, applied as a Turndown rule
via node.matches so invalid selectors fail silently
browser/article-extract.ts (new)
- generic Readability-based extraction that runs in-page via CDP
evaluate (no jsdom in Node)
- short-circuits non-HTML documents (text/plain, JSON, XML) and the
single-<pre> "browser rendering a plain text file" case
- clones the document before any mutation (preserves live page state
for subsequent snapshot / click)
- isProbablyReaderable gate, Readability.parse on the clone, then a
fallback chain main → [role="main"] → #main-content → … → body
- library sources are JSON-embedded and eval'd inside a Function scope
so their backticks / module.exports guards don't collide with the
surrounding IIFE
Tests
- article-download: page-chrome strip, cleanSelectors match + invalid
selector silently ignored (2 new)
- article-extract: JS generation contents, default fallback chain,
response normalization, null / malformed handling, and a Function()
parse check to catch any template-literal break-out in the embedded
Readability sources (8 new)
* fix(download): honor selector cleanup in fallback paths
* test(e2e): real-site regression for hardened article pipeline
Adds tests/e2e/article-download-pipeline.test.ts driving `opencli web read`
through 6 representative pages (example.com baseline, Wikipedia GFM tables,
MDN metadata, GitHub fenced code, Vercel SSR blog, Ruan Yifeng CJK+images)
and asserting the post-processing invariants: no base64/script/style leaks,
no blank-line runs, no residue, no trailing whitespace, no NBSP.
Graceful skip on bot detection / transient CDP errors, with a single retry.
All 6 sites pass locally (37s total).
* test(browser): add article extraction e2e fixtures
* feat(51job): add comprehensive 51job adapter (search / hot / detail / company)
Four adapters covering the main 51job surface:
- `51job search <keyword>` — keyword job search via we.51job.com/api/job/search-pc.
Rich filters: --area (40+ city name/alias → 6-digit code), --salary, --experience,
--degree, --companyType, --companySize, --sort, --page, --limit. Response already
carries full jobDescribe + HR + company + encCoId, so most callers won't need detail.
- `51job hot` — same endpoint with empty keyword, returns 51job's recommendation feed.
- `51job detail <jobId>` — scrapes jobs.51job.com/x/<jobId>.html. Returns description,
welfare tags, category, address, age requirement, company meta.
- `51job company <encCoId>` — scrapes jobs.51job.com/all/co<encCoId>.html. Job cards
carry a `sensorsdata` JSON attribute, so we parse that instead of fragile DOM text.
Company meta from `.c-info.ellipsis`, intro from `#companyIntroRef`.
All four are Strategy.COOKIE + browser:true + navigateBefore:false. 51job sits
behind Aliyun WAF — bare curl / Node-side fetch always hits the slider challenge
(tried copying acw_sc__v2 + ssxmod_itna cookies to Node, WAF also checks TLS
fingerprint and JS execution). Only reliable path is browser-context fetch via
`page.evaluate(fetch(url, {credentials:'include'}))`, so utils.js exports
`pageFetchJson` that wraps this pattern + detects WAF-served HTML.
Verify fixtures included (~/.opencli/sites/51job/verify/*.json) — four adapters
pass `opencli browser verify 51job/<cmd>` with rowCount / columns / types /
patterns / notEmpty checks. Eyeballed jobId 171699769 on jobs.51job.com/suzhou
matches adapter output.
* fix(51job): tighten city handling and docs
* chore: regenerate cli-manifest.json after 51job column cleanup
* feat(weread): add ai-outline command for AI-generated book outlines
Two-step API flow: fetch chapter UIDs via authenticated chapterInfos,
then retrieve hierarchical AI outline from public outline endpoint.
Supports --depth to control detail level (2=topics, 3=key points,
4=full details) and --raw for structured output (chapter/idx/level/text)
suitable for programmatic consumption.
Closes#1140
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(weread): tighten ai-outline auth contract
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
* fix(weread/book): add fallback selectors for reader page without cover
When the private API session expires, `loadReaderFallbackResult` navigates
to the reader URL. The page now sometimes skips the cover/flyleaf and
renders reading content directly, causing the wait for cover/flyleaf title
selectors to time out.
- Add `.readerTopBar_title_link` to `page.wait` selector (always present)
- Use cascading `firstText()` for title: cover → flyleaf → outline → top bar
- Use cascading `firstText()` for author: cover → flyleaf → outline → document.title
Fixes#1137
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(weread/book): parse author from trailing title segments
* fix(weread): avoid author guess from document title
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
* fix(xiaoyuzhou): correct podcast-episodes API endpoint
The endpoint `/v1/podcast/listEpisode` returns 404. The correct
endpoint is `/v1/episode/list` (verified against Xiaoyuzhou iOS app
traffic; also matches the `episode-list` implementation in
ultrazg/xyz, a widely-used Xiaoyuzhou API wrapper).
Additionally, the server requires an `order` field in the request
body (returns 400 if omitted). Add `order: 'desc'` so callers get
the latest episodes first, matching typical UX for a podcast feed.
Before: podcast-episodes -> HTTP 404 for every podcast
After: podcast-episodes returns the N most recent episodes
Tested against real podcast 626b46ea9cbbf0451cf5a962
(张小珺｜商业访谈录) — now returns 140 episodes correctly.
* test(xiaoyuzhou): lock podcast episodes endpoint
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* feat: agent-native retrospective — analyze / verify guards / fixture content checks
Post-mortem on slow 1point3acres + 51job adapter sessions, consolidated
into one PR. Scope is "reduce uncertainty and catch silent failures"
— the two things that sink agent success rate on first-time adapters.
Changes:
- `browser analyze <url>` — one command returns pattern (A/B/C/D),
anti-bot vendor (Aliyun/Cloudflare/Akamai/Geetest), nearest adapter,
and a single-sentence recommended_next_step. Replaces the three-step
open/wait/network recon loop when it can reach a confident verdict.
- `browser wait xhr <regex>` — poll for a specific XHR URL instead of
blind `wait time N`, so SPA data-arrival barriers are deterministic.
- Fixture `mustNotContain` / `mustBeTruthy` — catch two silent-failure
modes `notEmpty` misses: content contamination (sibling DOM bleed)
and `|| 0` / `|| false` fallbacks.
- `browser verify` post-success site-memory check + `--strict-memory`
— verify-green no longer hides the case where `~/.opencli/sites/`
was never written back. Memory only materializes if authors write it.
- CI: guard that committed `cli-manifest.json` matches a fresh build.
Main was already drifted (#1118 left stale ordering + a missing arg);
this PR regenerates the manifest and will catch the next drift.
Docs (opencli-adapter-author + opencli-autofix skills):
- `success-rate-pitfalls.md` — 10 concrete silent-failure scenarios
seen in real adapter sessions, each with defense via fixture /
adapter patterns.
- `autofix` gains discipline rule #6: verify pattern failure means
tighten the adapter, never loosen the fixture.
- `site-recon.md` leads with `browser analyze`; `api-discovery.md`
adds a §0 covering WAF vendor detection and cross-subdomain CORS
(the two gotchas that burned the 51job session).
- `wait time 3` → `wait time 2`, with `wait xhr` as the robust choice.
* fix: make output-dir defaults host-independent in manifest
Three adapters (chatgpt/image, gemini/image, instagram/download) baked
`path.join(os.homedir(), ...)` into the `default` field of their args.
The committed manifest therefore carried my personal `/Users/jakevin/...`
paths — which agents running on a different host saw as surprising
defaults. The drift guard I just added to CI caught it on the first run.
Runtime behavior is unchanged: each adapter still falls back to
`path.join(os.homedir(), …)` inside `func` when the kwarg is absent.
Only the displayed / registered default becomes a tilde-path.
* fix(cli): enforce strict-memory without fixture
* fix(browser): harden analyze and xhr guards
* fix(browser): fallback to interceptor buffer
* feat(verify): fixture-based value validation + skill docs for COOKIE pitfalls
`opencli browser verify` now loads `~/.opencli/sites/<site>/verify/<cmd>.json`
when present and validates row count / columns / types / patterns / notEmpty
against the live adapter output. Without a fixture, behavior is unchanged
(just runs the adapter and prints). New flags `--write-fixture`,
`--update-fixture`, `--no-fixture` seed / refresh / bypass the spec.
Motivation: previous verify only checked that the adapter exited 0 and
produced *something* — shape regressions (author name bleeding across rows,
a column silently becoming null after a site refresh, duplicated thread-level
time on every post) all passed "✓ Adapter works!" and shipped broken.
Skill doc updates (opencli-adapter-author):
- adapter-template.md: new "COOKIE adapter 骨架" section — HttpOnly +
dual-domain cookie read via `page.getCookies`, Node-side fetch for HTML
(explaining why `page.evaluate(fetch(...))` is the wrong tool when
`navigateBefore: false` or the response is non-UTF-8), and empty-state
sentinel row over `[]`
- api-discovery.md §4: note that BBS engines (Discuz/phpBB/vBulletin) set
auth cookies on the root domain + HttpOnly, so single-domain `getCookies`
calls silently miss them
- SKILL.md Step 10/12: make `--write-fixture` part of the runbook, forbid
debug dumps outside `~/.opencli/sites/<site>/fixtures/` or `/tmp/`
* fix(verify): support positional argv in fixture args + site-memory docs
Reviewer feedback blocker: fixture.args was Record<string, unknown>,
expanded as --key value only, so positional-subject adapters
(<tid>/<url>/<query>) couldn't be verified. Repo convention is
"主语优先 positional".
- verify-fixture.ts: args now accepts Record<string, unknown> | unknown[].
Object → --k v pairs; array → verbatim passthrough. New helper
expandFixtureArgs() centralizes the branching.
- cli.ts verify action: swap inline expansion for expandFixtureArgs().
- verify-fixture.test.ts: 6 new cases covering array form, mixed
positional+flag, empty shapes, passthrough stringification.
- site-memory.md: Layer 2 tree now lists verify/<cmd>.json; new schema
block distinguishes it from fixtures/<cmd>-<ts>.json; runbook timing
section gets a Step 10 verify-write row. Repo-tree debug-dump ban
clarified.
- adapter-template.md: new "Verify fixture" section with named-flag and
positional recipes, honest about --write-fixture only seeding named.
Smoke-tested 1point3acres/thread (positional <tid>): fixture round-trip
green (args=["1173710","--limit","2"]).
Two issues surfaced post-merge of #1110 by the Copilot reviewer:
1. Help text and docs advertise `video URL` as a valid input for
`opencli bilibili video <bvid>`, but the original implementation
delegated the whole input to `resolveBvid()` — which only recognises
bare `BV...` IDs and `b23.tv` short codes. A canonical bilibili URL
like `https://www.bilibili.com/video/BV.../` therefore got rewritten
to `https://b23.tv/www.bilibili.com/video/BV.../` and failed before
ever calling the view API.
Fix: pre-extract the BV ID from `bilibili.com/video/<BV>...` and
`bilibili.com/bangumi/play/<BV>...` URLs (www / m. / with or without
query string) in `video.js`, and fall through to `resolveBvid()`
only for bare BV IDs and `b23.tv` links.
2. `description` was being truncated to 200 chars with whitespace
collapsed before being returned. JSON/YAML consumers silently lost
the full `desc` value. Other bilibili adapters return raw fields.
Fix: return the full `d.desc` verbatim and let consumers/display
layers handle formatting.
Adds four regression tests for the URL paths (full URL, URL with query
string, m.bilibili.com mobile URL) and one for description integrity
(> 200 chars, preserved verbatim).
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(deepseek): use position-based model selection instead of text matching
Fixes#1111
* fix(deepseek): preserve explicit instant model contract
* fix(deepseek): guard expert selector arity
---------
Co-authored-by: Benjamin Liu <beneecs@Benjamins-Mac-mini.local>
Co-authored-by: jackwener <jakevingoo@gmail.com>
Adds two additive columns to the Twitter read commands (search, timeline,
tweets, thread, likes):
- has_media: boolean — true if the tweet contains any photo, video, or GIF
- media_urls: string[] — photo URLs and mp4 variant URLs for videos/GIFs,
extracted from legacy.extended_entities.media (falls back to entities.media)
The INTERCEPT/COOKIE payloads already carry this data; this change only
extends the row-mapping layer, so no new network work is needed. Pattern
mirrors #465 (time column).
Shared extraction helper lives in clis/twitter/shared.js so all five
adapters stay consistent, with unit coverage for photo, video (mp4 variant
selection), animated_gif, entities.media fallback, and the empty case.
Closes#1107
* docs(skills): restore and rewrite opencli-usage as orientation skill
The original opencli-usage skill was deleted in PR #1094 as part of the
skill consolidation, but its role (top-level orientation to what opencli
is, how to discover adapters, what flags/env/formats are universal, and
which specialized skill to load next) was not covered elsewhere. Restore
it, but deliberately NOT as a verbatim copy:
- Drop the hand-maintained 100-adapter table. There are 100+ adapters
and the list moves every week — `opencli list -f json` is the source
of truth agents should call at the start of a task.
- Replace it with the meta-structure agents actually need: the three
pillars (adapters / browser driving / external CLI passthrough), the
strategy tags (PUBLIC | COOKIE | HEADER | INTERCEPT | UI | LOCAL)
and what each implies for prerequisites, universal flags (-f, -v),
output formats, env vars, self-repair hook, adapter authoring paths,
plugins, external CLI passthrough.
- Explicitly list the commands PR #1094 removed (`explore`, `record`,
`web` / `desktop` top-level groups) so agents don't attempt them.
- Cross-link to the four post-consolidation skills: opencli-browser
(ad-hoc driving), opencli-adapter-author (writing adapters),
opencli-autofix (repair flow), smart-search (search routing).
Adapter-author description updated to stop claiming it replaces
opencli-usage.
* docs(skills): tighten opencli-usage validate + doctor scope per review
- validate: describe as registry-level semantic check (description, domain,
pipeline step names, func|pipeline|_lazy presence, arg duplicates), not
YAML/TS syntax check — matches src/validate.ts
- doctor: narrow to browser-bridge diagnostic; PUBLIC/LOCAL adapters, list,
validate, verify, plugins, and external-CLI passthrough do not need it
Backfill the 1.7.6 section that was missing from the release PR.
Covers window lifecycle flags, selector-first browser interactions,
agent-native payload, compound form fields, three new adapter commands,
four fixes, skill doc updates, and extension 1.0.2 body-truncation
contract unification.
--live (OPENCLI_LIVE=1) keeps the automation window open after an adapter
command finishes, so agents or humans can inspect the page state. Default
behavior (immediate closeWindow) is unchanged.
--focus (OPENCLI_WINDOW_FOCUSED=1) surfaces the existing env-var toggle as a
CLI flag so users don't need to shell-export to see the window in foreground.
Both flags are parsed early in main.ts and stripped from argv, so they can be
placed anywhere on the command line and work on any subcommand (adapter or
browser).
Restore `skills/opencli-browser/SKILL.md`, deleted in #1094, rewritten for
the post-#1116 browser CLI surface: selector-first target contract,
`match_level { exact | stable | reidentified }`, compound fields for
date/time/select/file, structured error codes with `available` vs
`candidates`, new `find` / `extract` / `network --filter` commands,
html tree budgets, tabs/frames, cost guide, recipes, pitfalls.
Review tightened two contract-drift bugs before merge:
- `browser tab list` envelope field is `page`, not `targetId`
- `network --ttl` default is `24h`, not `~5min`
2 reviewers green (codex-mini1, First-principles-1); CI all-green.
* feat(browser): compound expansion + cascading stale-ref + bbox 0.99 dedup
Three agent-native upgrades inspired by browser-use, landed as one PR
because they share the same target / snapshot / find surface.
1. Compound expansion (compound.ts)
Date/time/datetime-local/month/week, select, and file inputs now
emit a `compound` JSON field on `browser find --css` entries with
format, current value, min/max (date family), full options list
+ selected (select), accept / multiple / files[] (file). Kills
the three biggest form-page failure modes (wrong date format,
guessed options, re-uploaded files) without extra round-trips.
2. Cascading stale-ref (target-resolver.ts)
Numeric ref resolution now walks three tiers before giving up:
exact → stable (tag + strong id match, soft signals drifted) →
reidentified (original ref lost, fingerprint uniquely found a
live element; re-tag + refresh identity). Every success envelope
carries `match_level` so callers can tell which tier matched.
SPA re-renders / i18n label swaps no longer stall agents.
3. BBox 0.99 containment for interactive descendants (dom-snapshot.ts)
Adds a second dedup tier on top of the existing 0.95 non-interactive
one. When a parent is a propagator (tag a/button OR role button/
link/menuitem/tab/option) and a child is interactive but
undistinctive (no aria-label/id/testid/name/form-control), fold
it into the parent — removes `[1]<button> [2]<svg> [3]<span>`
noise on icon buttons.
Tests: 287/287 pass (src/browser + src/cli.test.ts). Typecheck clean.
* fix(browser): address reviewer blockers on PR #1116
- compound select: walk ALL options to collect selected labels, not just
the first 50 we serialize. Fixes dropdowns where the selected entry
sits past COMPOUND_SELECT_OPTIONS_CAP (e.g. country lists, timezones)
reporting current: "" even though the user picked a valid option.
- match_level: propagate the cascading match tier
(exact / stable / reidentified) through IPage.click/typeText/scrollTo,
BasePage, and the cli command envelopes (click / type / select /
get text|value|html|attributes). Agents now see in JSON that the
resolver had to fall back, instead of the tier being swallowed.
- compound contract is now also emitted by `browser state`
(per-ref compounds: sidecar) and by `browser get html --as json`
(compound field on each node), not only by `browser find --css`.
Closes the gap where agents using the default snapshot still
round-tripped `find` for every date / select / file control.
Adds targeted regression tests for each blocker + updates cli.test.ts
mocks to the new envelope shape.
The resolveTwitterQueryId() function in shared.js fetches an external JSON
file from GitHub without a timeout. If the network request stalls, the
function never resolves and the twitter article command hangs indefinitely.
Add a 5-second AbortController timeout so the fetch fails fast and falls
back to the local script-scanning strategy. This fixes the reported hang
when opencli twitter article loses network connectivity.
* feat(bilibili): add video command
Add `opencli bilibili video <bvid|url|short-link>` to fetch one
video's metadata via the public /x/web-interface/view endpoint.
Returns title, author, category, publish time, duration, view /
danmaku / reply / like / coin / favorite / share counts, parts,
thumbnail, and description as a key/value table.
Reuses `resolveBvid` and `apiGet` from clis/bilibili/utils.js to
stay consistent with the existing bilibili adapters
(subtitle/search/etc. all follow the same navigate + apiGet
pattern). Non-zero API codes surface as CommandExecutionError.
Fills a visible gap: existing bilibili commands cover search,
hot, subtitle, ranking, user-videos etc., but nothing returned
metadata for a single video — `web read` only gets a DOM shell.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(manifest): register bilibili video command
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
* WIP: deepseek file upload (blocked by 30s idle timeout)
* feat(deepseek): add file upload support via --file flag
Closes#1092
* fix(deepseek): use native file input path for --file
---------
Co-authored-by: Benjamin Liu <beneecs@Benjamins-Mac-mini.local>
Co-authored-by: jackwener <jakevingoo@gmail.com>
* fix(youtube): fall back to Videos tab when Home tab has no videos (#1108)
Some channels have no video shelves on their Home tab, causing
`opencli youtube channel <id>` to return an empty `recent_videos` list
even though the channel has videos visible in the browser.
When the Home tab extraction finds zero videos, the command now makes
a second InnerTube browse request to the Videos tab and extracts from
its richGridRenderer format.
* fix(youtube): make Videos tab fallback locale-safe
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* feat(browser): selector-first find + get/click/type/select with JSON envelope
A2: new `browser find --css <sel>` — structured JSON (matches_n + entries[]) so
agents can go from semantic selector directly to a list of candidates without
parsing free-text snapshot output. Per-entry shape: nth/ref/tag/role/text/attrs/
visible. Attr whitelist kept small (11 high-signal fields), invisible elements
still returned so agents can reason about offscreen vs missing.
A3: get text/value/attributes now accept a selector-first <target> (numeric ref
OR CSS) and emit `{value, matches_n}`. Bonus scope (approved by reviewers):
click/type/select share the same contract with `--nth <n>`, emitting
`{clicked|typed|selected, target, matches_n, ...}` on success.
Unified structured error envelope across all selector-first commands:
{ error: { code, message, hint?, candidates?, matches_n? } }
with codes invalid_selector / selector_not_found / selector_ambiguous /
selector_nth_out_of_range (CSS) plus not_found / stale_ref (numeric ref).
Write commands reject multi-match CSS without `--nth` as selector_ambiguous;
reads default to "first match wins" but always expose matches_n so agents
notice ambiguity. `resolveTargetJs` is the single source of truth; click /
typeText / scrollTo share a `runResolve` helper in BasePage.
No back-compat shims per design directive.
125 targeted tests green; tsc clean.
* fix(browser): unify selector surface + allocate fresh refs in find
Two blockers from PR #1112 review:
1. First-principles-1 (blocker): `browser find --css` now allocates fresh
numeric refs for untagged matches. It scans `window.__opencli_ref_identity`
(and any stray `data-opencli-ref` attrs) for the current max, allocates
`max+1` upward, writes `data-opencli-ref` on the element, and populates
the identity map with the same fingerprint shape snapshot uses (tag,
role, text, ariaLabel, id, testId). `find -> click <ref>` now works on
fresh pages without requiring `browser state` first. Type changed from
`ref: number | null` to `ref: number`.
2. codex-mini1 (blocker): removed the `isCssLike` regex
(`^[a-zA-Z#.\[]`) in `resolveTargetJs`. Valid selectors like `:root`,
`:has(...)`, `*` used to short-circuit to "Cannot parse target" before
reaching `querySelectorAll`, so `find --css` accepted them but
`get/click/type/select` did not. Now: numeric → ref path, everything
else → querySelectorAll, and the browser parser decides. Same selector
surface across all selector-first commands.
Tests added:
- target-resolver: pseudo-selectors flow into CSS branch (not rejection)
- find: ref allocation writes attribute + identity map; fingerprint shape matches resolver
- cli: find envelope now expects numeric refs
127 targeted tests green; tsc clean.
* feat(browser): agent-native payload — network bodies, html tree budgets, extract command
Three fixes/additions driven by agent-usage gaps, as one complete change:
- network (P0 fix): lift silent 4000-char body truncation in CDP + extension
paths to an 8MB memory-guard cap, and surface body_truncated / body_full_size
/ body_truncation_reason in the --detail envelope so the agent sees when a
body was cut. List view also exposes body_truncated_count and per-entry flag.
Adds --max-body flag for explicit caller-side capping.
- get html --as json (P1): add --depth / --children-max / --text-max budget
knobs on the tree serializer, plus a truncated={depth,children_dropped,
text_truncated} envelope that only appears when a budget is hit. Lets the
agent narrow DOM output without walking away empty-handed.
- extract (P2 new command): agent-native article/content channel. Scope →
denoise (strip nav/header/footer/scripts/forms/etc.) → HTML→markdown via
existing htmlToMarkdown → paragraph-boundary-aware chunk with stateless
next_start_char resume cursor. Agents no longer misuse `get html` to read.
* fix(browser): unify body-truncation signal contract across raw/detail/fallback
Addresses review blockers on #1104:
- NETWORK_INTERCEPTOR_JS fallback no longer silently drops bodies above the
per-entry cap. Raised cap to 1 MiB (ring stays at 200 entries), and on
overflow keeps the string prefix + sets `bodyTruncated` / `bodyFullSize`
so `browser network` propagates the same agent-visible signal the CDP /
extension paths emit.
- `CachedNetworkEntry` schema switches from internal camelCase
`bodyTruncated` to the user-facing `body_truncated` / `body_full_size`
fields. `--raw` emits cache entries verbatim, so this removes the
snake_case/camelCase split across list / --detail / --raw.
- Adds a `--raw` truncation-contract test that also asserts the camelCase
fields do not leak through.
Agents often know what fields a target request's body should contain
but not which captured request carries it. --filter lets them declare
the field set and get back only matching entries.
Matching is "any-segment": a field matches when it equals any segment
name of any inferShape() path (ignoring root $, array indices, and
bracket-quoted key syntax). Multiple fields AND together. Case-sensitive.
- invalid_filter for empty / commas-only values
- invalid_args when combined with --detail (mutually exclusive)
- 0 matches is a valid empty result, not an error
- persisted cache stays unfiltered so later --detail lookups still resolve
Envelope gains `filter` (echo) and `filter_dropped` (count of entries
passing the static-resource filter but not --filter). Existing --raw
and --all compose normally.
* feat(browser): remove silent html truncation, add --as json tree output
`browser get html` had two agent-hostile defaults:
1. A silent 50000-char cap on the returned HTML — agents that got a
truncated page had no signal they were looking at half the DOM.
2. Only raw HTML string output, forcing agents to re-parse for
structured extraction.
Changes:
- Default output is now the full outerHTML, no truncation
- `--max <n>` opts in to a character cap; when the cap actually
trips, the HTML is prepended with
`<!-- opencli: truncated N of M chars; re-run without --max ... -->`
so agents always see the signal
- `--as json` returns `{selector, matched, tree}` where `tree` is
`{tag, attrs, text, children}` recursively. `matched` is the full
count of selector matches so agents know when more elements exist
beyond the first. `text` is the node's own direct text children,
whitespace-collapsed; child elements live in `children`.
- `--selector` not matching any element now emits structured
`{error:{code:"selector_not_found", ...}}` with a non-zero exit
code, in both raw and json modes (was `(empty)` stdout previously,
indistinguishable from empty element)
- Invalid `--as` / negative `--max` emit structured
`invalid_format` / `invalid_max` error codes
Extracted the tree serializer as `src/browser/html-tree.ts` so the
JS expression can be unit-tested against a DOM stub.
* fix(browser get html): structured errors for invalid selector & strict --max
Both edges previously bypassed the structured-error contract introduced in
#1102, which agents rely on for branching:
- Invalid CSS selector: querySelector(All) would throw SyntaxError through
page.evaluate into the generic exception path. Wrap the lookup in try/catch
inside page context for both raw and --as json paths; surface as
{error:{code:"invalid_selector", message}} + non-zero exit.
- --max validation: parseInt silently accepted "1.5" -> 1 and "10abc" -> 10.
Switch to a strict /^\\d+$/ check so fractional, negative, and non-numeric
values all return {error:{code:"invalid_max"}}; validation runs up front so
bad values never reach the page.
Covered by new unit tests in cli.test.ts (fractional, non-numeric, invalid
selector on raw + json) and html-tree.test.ts (SyntaxError -> invalidSelector
envelope).
Co-authored-by: freemandealer <freeman.zhang1992@gmail.com>
---------
Co-authored-by: freemandealer <freeman.zhang1992@gmail.com>
* feat(browser): rewrite network command for agent-native discovery
Replace the index-based list + pretty-printed --detail flow with a
structured JSON interface built around stable keys, body-shape previews,
and a persistent capture cache. Agents can now reference captured
requests by operationName (GraphQL) or `METHOD host+pathname` (REST)
instead of array indexes that shift on every rerun.
- `browser network` now emits JSON: `{workspace, captured_at, count,
filtered_out, entries: [{key, method, status, url, ct, size, shape}],
detail_hint}` — no body payloads by default
- Shape inference (src/browser/shape.ts) walks response JSON into a
flat path -> descriptor map with depth cap 6 and a 2KB budget per
entry, so agents see structure without paying body tokens
- Stable key generator (src/browser/network-key.ts) derives
`operationName` from graphql URLs and `METHOD host+pathname`
elsewhere, disambiguating collisions with `#N` suffixes
- Persistent cache (src/browser/network-cache.ts) snapshots every
capture to `~/.opencli/cache/browser-network/<workspace>.json` with
a 24h TTL, so `--detail <key>` survives later commands
- `--detail <key>` returns `{key, url, method, status, ct, size, shape,
body}` with structured error codes (cache_missing / cache_expired /
cache_corrupt / key_not_found, the latter including available_keys)
- Add `--raw` for agents that want every full body inline, `--ttl` for
cache lookups
- Update opencli-adapter-author + opencli-autofix skill docs to
reference `--detail <key>` and the shape-first discovery flow
Supersedes the cache prototype in #1051.
Co-authored-by: freemandealer <freeman.zhang1992@gmail.com>
* fix(browser): structured errors for capture/save, shape budget guard
Self-review findings on the network refactor:
- captureNetworkItems throwing (browser crashed / CDP dropped) now emits
`error.code: capture_failed` on stdout rather than leaking a bare
stderr line from browserAction's generic handler — agents get a
parseable JSON blob on every failure path, matching the design goal.
- saveNetworkCache throwing (disk full, read-only path) is a soft
failure: the captured data is already in hand, so surface a
`cache_warning` field in the envelope and keep going instead of
aborting. `--detail` lookups on that run will miss the cache but the
listing still reaches the agent.
- shape.ts: guard the sub-walk on `add()`'s return value so the
"budget hits on the array/object descriptor itself" path can never
emit a stray child without its parent marker.
- network-key.ts: document that `#N` suffixes start at `#2` — the first
occurrence stays bare, there is no `#1`. Matches test + code.
Added regression tests: `capture_failed` on readNetworkCapture throw,
`cache_warning` on persistence failure, shape budget hit on array descriptor.
Co-authored-by: freemandealer <freeman.zhang1992@gmail.com>
---------
Co-authored-by: freemandealer <freeman.zhang1992@gmail.com>
* feat(twitter): add tweets command for fetching a user's recent posts
Adds `opencli twitter tweets <username> [--limit N]` to pull a user's
most recent chronological tweets via the UserTweets GraphQL endpoint.
Long posts resolve via note_tweet, pinned entries are skipped, and
retweets are flagged. QueryIds resolve dynamically through
`resolveTwitterQueryId` with hardcoded fallbacks.
* fix(twitter): expose retweet flag in tweets output
* refactor: consolidate 6 skills into 3, remove mechanical commands
Replaces opencli-oneshot / opencli-explorer / opencli-browser /
opencli-usage with a single opencli-adapter-author skill that takes
the AI agent end-to-end: site recon, API discovery, field decoding,
adapter coding, and `opencli browser verify`.
Removes the mechanical commands (`explore`, `synthesize`, `generate`,
`cascade`, `record`) and their src/tests — they were codegen scaffolding
meant for agents, which the new skill handles more flexibly via
`opencli browser` primitives.
Skill highlights:
- Top-level decision tree + 12-step runbook
- 5 site patterns (SPA / SSR / JSONP / Token / Streaming)
- 5-layer API discovery (network → initial state → bundle → token → interceptor)
- Field decode playbook (self-explanatory → codes → sort-key comparison)
- Output design guide (columns, types, order, ≤15 per adapter)
- Two-layer site memory: in-repo seeds for eastmoney/xueqiu/bilibili/tonghuashun
plus local `~/.opencli/sites/<site>/` runtime workspace
Kept skills: opencli-autofix (now points to adapter-author for rewrites),
smart-search. Kept primitives: `browser *`, `doctor`, `list`, `validate`,
`verify`, `<site> <cmd>`, `plugin *`, `completion`.
No backward compatibility shims. Full test suite (1605 tests) passes.
* review fixes: honest coverage, hard memory-hit path, typo, stale docs
- site-memory hit path no longer jumps to writing adapter; forces Step 5
endpoint re-verification + Step 7 field check, and 30-day expiry
- site-memory.md now specifies exact schemas for endpoints.json /
field-map.json / notes.md / fixtures + write-back timing rules
- coverage-matrix.md marks unverified patterns as 🟡 with an evidence
section citing coingecko dry run + PR #1091 eastmoney + bilibili
- eastmoney seed typo: resolveSecids -> resolveSecid (and splitSymbols)
- docs/developer/ai-workflow.md rewritten to teach the adapter-author
skill + opencli browser * primitives (dropped generate/synthesize/
cascade/explore references)
- ts-adapter.md, getting-started.md, CHANGELOG.md:87 updated to point
at opencli-adapter-author
* fix(ci): resync package-lock + drop stale built-in list reference
- Regenerate package-lock.json to restore @emnapi/core + @emnapi/runtime
entries that got dropped during the rebase — `npm ci` was failing on all
CI jobs (build / audit / docs-build / bun-test / unit-test)
- docs/guide/getting-started.md: built-in list dropped `explore`, now
reads (list, validate, verify, browser, doctor, plugin...)
* fix(ci): restore package-lock.json from main (unrelated lockfile churn)
Mirror the remaining 13 read-oriented adapters and the shared _secid.js
helper from the author's local workspace into the repo, so that
clis/eastmoney/ becomes the full Phase A codegen regression oracle
described in OpenCLI Improvement Spec v1.1 §B.10.
Total repo oracle after this PR: 14 adapters under clis/eastmoney/
(hot-rank.js already exists; this PR adds the other 13) plus the
_secid.js normalize helper.
Covers the two schema-expressiveness gaps discovered during prep:
- CSV row_format: kline.js decodes "YYYYMMDD,open,close,..." strings
- :row_index source: convertible.js derives rank = i + 1
_secid.js is the canonical example of the v1.1 §B.7 helper contract
(pure normalize/derive function, serializable I/O, no env/fs/net/session
access, does not drive pagination/retry/fallback).
This PR is oracle-only, carries no framework changes. Phase A framework
PR depends on this merging first so the codegen diff target is stable.
Refs: task #177 / spec v1.1 §B.10
* feat(download): show saved file path in web read and weixin download output
Closes#1038
* test(download): cover saved article path
---------
Co-authored-by: Benjamin Liu <beneecs@Benjamins-Mac-mini.local>
Co-authored-by: jackwener <jakevingoo@gmail.com>
jakevin