PR #1297 introduced a CI gate that fails when a site has both a listing
and a detail command but the listing rows don't carry an id-shaped column.
The gate came with a 10-entry EXEMPT map (topic-string trending,
profile-attribute rows, UI-only sessions, ...) where each exemption
recorded a "why this listing legitimately doesn't pair" reason.
By the same filter that closed PR #1311 (write-without-delete-pair gate):
Is "listing should pair with detail" a *permanent* anti-pattern, or
case-by-case business judgment?
It's case-by-case. Topic-string listings and profile-attribute rows
genuinely don't pair with a detail command. The fact that we needed an
EXEMPT map with 10 entries and individual reason strings is the smell —
it's not the rule winning, it's the rule failing. Forcing every adapter
PR to either add an id column or file an exemption was a higher cognitive
cost than the silent-loss bugs the rule actually catches.
Changes:
- .github/workflows/ci.yml — drop the "Check listing↔detail id pairing"
step. Other gates (silent-column-drop, typed-error-lint) stay in place.
- package.json — rename the script from `check:listing-id-pairing` to
`advise:listing-id-pairing` to make the advisory nature explicit.
- scripts/check-listing-id-pairing.mjs — drop the `--strict` flag and the
EXEMPT map. The script now always exits 0 and prints an advisory report
of listings that don't carry an id-shaped column. Reviewers/authors use
it as guidance, not a gate.
- docs/conventions/listing-detail-id-pairing.md — rewrite from "MUST" to
"soft convention". Adds an explicit "why advisory, not a gate" section
that lists the legitimate non-pairing categories so future readers know
the rule's boundary.
- docs/developer/ts-adapter.md — match the advisory tone in the
adapter-author guidance.
The doc, the script, and the column patterns table all stay — agents and
adapter authors can still consult them. What's gone is the CI failure and
the per-PR exempt-list maintenance burden.
Net diff: -34 lines (gate + EXEMPT map removed, advisory-tone doc adds
a small "why advisory" section).
Adds a baseline CI gate for convention-audit typed-error lint findings. Also refreshes the silent-column-drop baseline for dianping changes already on main.
* feat(convention): listing↔detail id pairing rule + CI gate
Adds a hard convention: when a site exposes both a listing-class command
(search / hot / top / recent / ...) and a detail-class command (read /
paper / article / view / ...), every listing row MUST surface an id-shaped
column whose value round-trips into the detail command. Without that, an
agent has no way to follow up on a listing row except re-searching by
title or scraping URLs out of band — both of which break the agent-native
contract.
What's in this PR
- docs/conventions/listing-detail-id-pairing.md — full rule, examples
table, why-it-matters, what counts as id-shaped, exemption taxonomy,
how to add an id column to a listing.
- scripts/check-listing-id-pairing.mjs — validator that reads
cli-manifest.json, classifies each entry as listing / detail / other,
and fails when a listing on a site that also has a read-detail command
is missing an id-shaped column. Exemption allowlist records WHY each
pair is exempt so future maintainers know what to verify.
- npm run check:listing-id-pairing — strict-mode wrapper.
- CI: new step in build job runs the validator after the manifest
freshness check on Linux.
- docs/developer/ts-adapter.md — cross-link from the adapter authoring
guide.
- docs/.vitepress/config.mts — sidebar entries for the new conventions
section.
Fixes brought to zero violations
- 1688/search: add offer_id (already extracted, just surfaced)
- bluesky/user: add uri (AT URI round-trips into bluesky/thread)
- tieba/search: add id + url (thread_id already extracted)
- tieba/hot: add url (rows are topics, not threads — url is the
best-effort round-trip handle, doc'd as such)
Exemptions (intentional, doc'd in EXEMPT map with rationale)
- nowcoder/hot, bluesky/trending, twitter/trending — listing rows are
topic strings, not posts.
- lesswrong/user, reddit/user — rows are profile-attribute key/value
pairs, addressed by the username arg.
- discord-app/search — desktop UI session, message ids not extractable.
- notion/search — Strategy.UI Quick Find, page ids not exposed in DOM.
Validator output after this PR: 32 sites scanned, 75 listings checked,
7 exempted, 0 violations.
* fix(convention): tighten listing id gate
* fix(convention): close url-derived id loophole
The previous implementation silently skipped any adapter whose import
failed (catch + warn-to-stderr + return []), then printed a successful
"✅ Manifest compiled: N entries". When dist/ was stale (e.g. after
renaming an export the JS adapters re-import) every adapter using that
export would fail to load, get skipped, and the script still exited 0.
An agent reading exit codes to gate work would commit the resulting
manifest and silently delete dozens of unrelated adapter entries.
Three layers of defense:
1. Distinguish skip kinds. Files that don't call `cli(...)` are still
silently dropped (helpers / type modules). Files that look like CLI
modules but fail to import now throw `ManifestImportError`. The
batch scanner aggregates failures and `main()` exits 1 with an
explicit list, leaving the existing manifest on disk untouched.
2. Net-deletion safety net. `main()` diffs the new entries against the
committed manifest and refuses to overwrite when entries would be
removed. `--allow-removals=N` (or bare `--allow-removals` for any)
is the explicit opt-in; the error message tells the caller exactly
what value to pass.
3. Runtime dist guard. `node dist/src/build-manifest.js` now refuses
to run with a clear pointer at `npm run build-manifest` (which uses
tsx). The npm script itself is migrated to `tsx src/build-manifest.ts`
so no project-level command points at the compiled copy anymore.
Release CI gains a manifest-drift gate (build-manifest + git diff
--exit-code) so a tag push can never publish stale or silently-shrunk
manifests. The existing CI check on PRs is preserved.
`ManifestEntry` is split into `src/manifest-types.ts` so runtime code
(discovery.ts) imports the type without pulling the build-time
compiler module.
Tests:
- `loadManifestEntries` throws ManifestImportError on import failure
- helper modules without cli() are still silently skipped
- `scanClisDir` aggregates per-adapter failures
- `diffRemovedEntries` returns expected site/name diff
- `parseBuildManifestArgs` reads --allow-removals[=N]
* feat: agent-native retrospective — analyze / verify guards / fixture content checks
Post-mortem on slow 1point3acres + 51job adapter sessions, consolidated
into one PR. Scope is "reduce uncertainty and catch silent failures"
— the two things that sink agent success rate on first-time adapters.
Changes:
- `browser analyze <url>` — one command returns pattern (A/B/C/D),
anti-bot vendor (Aliyun/Cloudflare/Akamai/Geetest), nearest adapter,
and a single-sentence recommended_next_step. Replaces the three-step
open/wait/network recon loop when it can reach a confident verdict.
- `browser wait xhr <regex>` — poll for a specific XHR URL instead of
blind `wait time N`, so SPA data-arrival barriers are deterministic.
- Fixture `mustNotContain` / `mustBeTruthy` — catch two silent-failure
modes `notEmpty` misses: content contamination (sibling DOM bleed)
and `|| 0` / `|| false` fallbacks.
- `browser verify` post-success site-memory check + `--strict-memory`
— verify-green no longer hides the case where `~/.opencli/sites/`
was never written back. Memory only materializes if authors write it.
- CI: guard that committed `cli-manifest.json` matches a fresh build.
Main was already drifted (#1118 left stale ordering + a missing arg);
this PR regenerates the manifest and will catch the next drift.
Docs (opencli-adapter-author + opencli-autofix skills):
- `success-rate-pitfalls.md` — 10 concrete silent-failure scenarios
seen in real adapter sessions, each with defense via fixture /
adapter patterns.
- `autofix` gains discipline rule #6: verify pattern failure means
tighten the adapter, never loosen the fixture.
- `site-recon.md` leads with `browser analyze`; `api-discovery.md`
adds a §0 covering WAF vendor detection and cross-subdomain CORS
(the two gotchas that burned the 51job session).
- `wait time 3` → `wait time 2`, with `wait xhr` as the robust choice.
* fix: make output-dir defaults host-independent in manifest
Three adapters (chatgpt/image, gemini/image, instagram/download) baked
`path.join(os.homedir(), ...)` into the `default` field of their args.
The committed manifest therefore carried my personal `/Users/jakevin/...`
paths — which agents running on a different host saw as surprising
defaults. The drift guard I just added to CI caught it on the first run.
Runtime behavior is unchanged: each adapter still falls back to
`path.join(os.homedir(), …)` inside `func` when the kwarg is absent.
Only the displayed / registered default becomes a tilde-path.
* fix(cli): enforce strict-memory without fixture
* fix(browser): harden analyze and xhr guards
* fix(browser): fallback to interceptor buffer
* fix: remove duplicate extension zip from releases
The release and build-extension workflows were creating both
opencli-extension.zip and opencli-extension-v{version}.zip (identical
content), causing both to be uploaded. Keep only the versioned filename.
* docs: update extension zip filename to versioned format
Update all references from opencli-extension.zip to
opencli-extension-v{version}.zip to match the workflow change.
* feat: decouple extension version from CLI version
Extension and CLI had tightly coupled version numbers (both 1.7.2),
requiring manual sync across 3 files on every release. This decouples
them so each can release independently.
Changes:
- Extension version reset to 1.0.0 with independent versioning
- Extension sends compatRange (e.g. ">=1.7.0") in hello message
so doctor can check CLI/extension compatibility
- Daemon stores and exposes extensionCompatRange via /status
- Doctor uses compatRange for compatibility checks (falls back to
major-version check for older extensions without compatRange)
- Doctor shows extension update availability from cached GitHub
Releases data
- release.yml always builds and attaches extension zip to every
CLI release, so users always find both in the same release page
- build-extension.yml triggers on ext-v* tags (not v*) to avoid
duplicate builds
* fix: version extension release assets
* fix: include adapter tests in default npm test
`npm test` only ran unit + extension projects, so adapter tests
(clis/**/*.test.js) were never exercised by the default test command.
Add --project adapter so they run alongside unit and extension tests.
* test: include adapter project in default npm test
* fix: project hygiene — docs, lint, daemon restart, code fence
- Update Node version requirement from >= 20 to >= 21 in 7 doc files
(README, README.zh-CN, installation guides, troubleshooting)
- Update adapter count from 79+ to 87+ in READMEs
- Remove duplicate `lint` script (identical to `typecheck`)
- Fix TESTING.md CI matrix: Node ['22'] instead of ['20', '22']
- Fix autofix SKILL.md code fence escaping (\``` → ~~~)
- Add daemon restart to postinstall so updated adapters are picked up
- Fix preuninstall to respect OPENCLI_DAEMON_PORT env var
* fix: align docs and skills with JS-first adapter contract
Adapters are now .js files (not .ts). Update all references across:
- README.md, README.zh-CN.md, CONTRIBUTING.md
- docs/guide/getting-started.md, docs/index.md
- skills/opencli-browser/SKILL.md, skills/opencli-explorer/SKILL.md
The runtime (discovery.ts) only loads .js from user clis/ directories,
and `opencli browser init` generates .js scaffolds. Documentation was
still teaching users to create .ts files.
* fix: update CI matrix to Node 22 only (drop Node 20)
package.json requires Node >= 21 (styleText dependency). The CI matrix
was still testing Node 20 which doesn't meet this requirement.
* fix: revert incorrect daemon restart from postinstall
The daemon (browser bridge) only handles CDP communication — it has no
knowledge of adapters. Adapter discovery, loading, and execution all
happen in the CLI process, which is fresh each invocation. The
_loadedModules cache in execution.ts is process-local and not a real
staleness concern. Remove the unnecessary restartDaemon() call.
* refactor: slim CI matrix, extract shared utils, unify logging, remove __test__ from public API
- CI: unit-test uses dynamic matrix (PR=ubuntu+22 only, push=full 3OS×2Node);
adapter-test reduced to ubuntu-latest (OS doesn't affect pure unit tests)
- _shared/common.ts: add sleep() and clampToRange() shared adapter utilities;
douban/utils.ts and sinablog/utils.ts now use clampToRange instead of duplicate clampLimit
- browser/daemon-client.ts: replace inline setTimeout Promise with local sleep()
- execution.ts: replace conditional console.error with log.debug
- browser/index.ts: remove __test__ from public barrel export;
browser.test.ts now imports internal helpers directly from source files
* fix: remove unused afterEach import, fix schedule/dispatch CI matrix, clarify clampToRange docs
* refactor: move sleep to src/utils.ts, simplify clamp signature to match lodash convention
* feat(runtime): add runtime detection utility for Bun/Node.js
Add runtime-detect.ts module that detects whether opencli is running
under Bun or Node.js via globalThis.Bun check. Includes helper
functions for version string and label formatting.
Add corresponding unit tests that work correctly under both runtimes.
* feat(runtime): integrate Bun runtime support into CLI tooling
- doctor: show runtime label (e.g. 'node v22.13.0') in diagnostic output
- package.json: add dev:bun, start:bun, test:bun convenience scripts
- E2E helpers: support OPENCLI_TEST_RUNTIME env var for runtime selection
* ci: add Bun compatibility test job and document runtime support
- ci.yml: add bun-test job using oven-sh/setup-bun@v2
- README.md: update Prerequisites to mention Bun, add Runtime Support
section with usage examples for dev:bun, start:bun, test:bun
* ci: pin Bun version in compatibility job
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
* fix(ci): include popup assets in extension release
Copy popup assets into the packaged Chrome extension zip and validate that manifest-referenced files exist before publishing the artifact.
Co-authored-by: Codex <noreply@openai.com>
* fix: restore executable permission on bin entries after tsc build (#446) (#452)
tsc does not preserve the +x bit when compiling, so after clean-dist
removes dist/ and tsc regenerates it, dist/main.js loses its executable
permission. This causes 'Permission denied' when users run 'npm run build'
in the installed directory.
Fix: read bin entries from package.json at the end of build-manifest and
chmod 0o755 them (skipped on Windows). Wrapped in try/catch so it never
breaks the build.
Closes#446
* fix: correct positional arg usage in tests (#449)
* fix yahoo-finance quote e2e invocation
* fix positional args in v2ex topic tests
* fix(ci): script extension release packaging
---------
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: jakevin <jakevingoo@gmail.com>
Co-authored-by: pi-dal <hi@pi-dal.com>
* ci: add cross-platform support for E2E and smoke tests
Make headed browser tests (E2E and smoke) runnable on Linux, macOS,
and Windows:
- setup-chrome action: only install xvfb on Linux (macOS/Windows
have native GUI sessions and don't need a virtual display)
- e2e-headed.yml: add OS matrix, use xvfb-run wrapper only on Linux
- ci.yml smoke-test: add OS matrix, use xvfb-run wrapper only on Linux
The browser-actions/setup-chrome action already supports all three
platforms natively.
* ci: exclude Windows from E2E/smoke matrix (Chrome install hangs)
browser-actions/setup-chrome hangs indefinitely during Chrome MSI
installation on Windows runners (observed 10+ min with no progress).
This is a known limitation of Windows CI runners.
Keep Linux + macOS for headed browser tests. Windows is still covered
by build, unit-test, and adapter-test jobs.
* ci: add cross-platform matrix (Linux/macOS/Windows) to build, unit-test, adapter-test
Add OS matrix with ubuntu-latest, macos-latest, and windows-latest to
the build, unit-test, and adapter-test CI jobs. This ensures cross-
platform compatibility is verified on every push and PR.
Smoke tests remain Linux-only due to xvfb dependency.
Relates to #392 (Windows plugin path issues).
* test: replace hardcoded /tmp with os.tmpdir() for Windows compatibility
Fix Windows CI failures caused by hardcoded '/tmp' paths that don't
exist on Windows. Use os.tmpdir() which returns the correct platform-
specific temp directory on all operating systems.
Files fixed:
- src/engine.test.ts: 3 occurrences (mkdtemp, discoverClis path)
- src/plugin.test.ts: 2 occurrences (getCommitHash test, mock condition)
* test: fix remaining Windows path issues in test files
- engine.test.ts: use pathToFileURL().href for dynamic import paths
(path.join produces backslashes on Windows, breaking ES module imports)
- download.test.ts: replace hardcoded '/tmp' with os.tmpdir() + path.join
CRX files cannot be installed in modern Chrome without Chrome Web Store
publishing. Updated all docs to recommend 'Load unpacked' installation
method only. Added npm package loading method as alternative.
- Removed CRX build step from build-extension.yml workflow
- Removed CRX from artifact upload and release attachment
- Updated README.md, README.zh-CN.md, browser-bridge docs (en/zh)
- Added 'Load from npm package' as installation method
* docs: add missing adapter docs, fix sidebar 404s, add doc-check CI
- Add doc pages for 11 undocumented adapters: arxiv, barchart,
chaoxing, grok, hf, jike, jimeng, linux-do, sinafinance,
stackoverflow, weread, wikipedia
- Update adapters/index.md with all new adapter entries
- Update VitePress sidebar config with 12 new entries
- Remove broken zh/ sidebar refs (troubleshooting, testing)
- Add doc-check CI workflow (adapter coverage + build + link check)
- Add scripts/check-doc-coverage.sh for adapter doc enforcement
- Enhance PR template with adapter doc checklist
* fix(ci): use --root-dir instead of --base for lychee link checker
lychee v0.23 requires --base to be a URL or absolute path.
Use --root-dir for resolving root-relative links in local files.
* fix(ci): remove lychee link-check job, rely on VitePress build
VitePress links use extension-less paths (e.g. /adapters/browser/twitter)
which lychee cannot resolve. The docs-build job already catches all
broken internal links via VitePress dead link detection during build.
* chore(ci): add Dependabot for npm and GitHub Actions updates
- Weekly npm dependency updates with PR limit of 10
- Weekly GitHub Actions version updates with PR limit of 5
- Conventional commit prefixes (chore(deps), chore(ci))
* ci: add security audit workflow
- Run npm audit on push/PR and weekly schedule
- Fail on high-severity vulnerabilities using audit-ci
- Only audit production dependencies
* ci: add release-please for automated changelog and versioning
- Auto-generate CHANGELOG.md from Conventional Commits
- Create version bump PRs on push to main
- Works alongside existing release.yml for npm publish
* ci: add concurrency controls and Node.js version matrix
- Add concurrency groups to ci, e2e-headed, security workflows
to cancel duplicate runs on the same branch
- Test unit tests across Node 18/20/22 with fail-fast: false
- Update test step name to show Node version
* chore: bump minimum Node.js version from 18 to 20
- Update engines.node in package.json to >=20.0.0
- Update prerequisites in README.md and README.zh-CN.md
- Remove Node 18 from CI test matrix
* review: fix release token and prod-only audit scope
* docs: align Node 20 troubleshooting guidance
---------
Co-authored-by: jackwener <jakevingoo@gmail.com>
## Changes
### E2E Test Suite (~52 test cases)
- public-commands.test.ts — Public API commands (hackernews, v2ex)
- browser-public.test.ts — Browser commands for public data across all sites
- browser-auth.test.ts — Graceful failure verification for login-required commands
- management.test.ts — Full coverage of management commands
- output-formats.test.ts — Output format validation (json/yaml/csv/md)
- smoke/api-health.test.ts — Scheduled API health checks
### Auto-detect Browser Mode
- buildMcpArgs uses CI env var to select mode:
- Local (no CI) → --extension (connect to user's Chrome)
- CI → standalone (launches its own browser)
### CI Pipeline
- e2e-headed.yml — Real Chrome via setup-chrome + xvfb in headed mode
- ci.yml — build + unit-test (2 shards) + smoke-test (scheduled/manual)
- Composite action for shared Chrome + xvfb setup
### Documentation
- New TESTING.md — Architecture, coverage, local setup, how to add tests
Co-authored-by: AlexYue <yj976240184@qq.com>
jakevin