Commit Graph

7 Commits

Author SHA1 Message Date
Ocean 4a6cfe8060 fix(xhs,youtube): 把合法空数据语义切到 EmptyResultError (#1674)
* fix(xhs,youtube): 把"合法空数据"语义切到 EmptyResultError

把 xiaohongshu/user 和 youtube/transcript 跟 bilibili/subtitle 已有的
structured-error 模式对齐,让下游能区分"用户/视频没内容"和"fetch 真的挂了"。

## xiaohongshu user

返回 `No public notes found for this Xiaohongshu user` 的场景——目标用户
零公开笔记(销号 / 私密 / 全删)——原来抛 plain `Error`,下游无法和
"真的 fetch 失败 / cookie 死"区分。

改抛 `EmptyResultError`,exit code 变 66,stderr 携带 `code: EMPTY_RESULT`,
跟 `bilibili subtitle` empty 同 shape。

## youtube transcript

`No captions available for this video`(作者没开 CC、YouTube 也没自动生成)
是数据条件,不是基础设施失败。原来跟 HTTP/解析错误一样抛
`CommandExecutionError`,造成调用方反复重试。

改这个特定 case 抛 `EmptyResultError`;其他 caption 错误(HTTP / parse /
empty response)继续走 `CommandExecutionError` 触发重试。

## 为什么 downstream 需要这个

调用方(如自动化采集流水线)通常对 "data.length === 0 && exitCode !== 0"
做 soft-rate-limit 启发式:N 次连续 soft fail 触发 24h 平台跳过。当 seed 列表
里有变质条目(XHS 账号销号 / YouTube 视频丢失字幕),"empty" 响应堆积会
误触跳过——cookies 和平台本身都健康。EmptyResultError 让调用方能区分
"这个用户没内容"和"API 挂了"。

## 测试

- `npx vitest run clis/xiaohongshu clis/youtube` —— 全过
- `npx tsc --noEmit` 干净

* fix(xhs): distinguish empty user notes from parser drift

* fix(empty): tighten legal empty evidence

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-19 19:02:32 +08:00
Ocean dc645a5bcf fix(youtube/transcript): 把 timedtext URL 匹配限定到当前 videoId,修跨视频字幕串台 (#1655)
* fix(youtube/transcript): scope timedtext URL match to current videoId

YouTube watch-page is an SPA — page.goto between watch URLs preserves
performance.getEntriesByType('resource') entries from prior videos.
findTimedtextUrl filtered only by lang, so a previously-viewed
same-language video's timedtext URL could be picked up by the polling
loop before the current video's fetch hook captured a fresh one,
returning the wrong video's captions to the caller.

Fix: require URLs to contain v=<currentVideoId> across all three paths:
  - in-page findTimedtextUrl (resource-buffer scan)
  - in-page isJson3TimedtextUrl (fetch/XHR hook)
  - Node-side extractSegmentsFromNetworkCapture (CDP capture)

Most likely to hit callers that reuse a single daemon tab to fetch
many transcripts back-to-back (e.g. ml-scout). Confirmed in the wild:
a Fox News Ukraine clip got Whisper Flow promo captions written to
its row when the prior call on the same tab pulled an English
Whisper Flow video.

Adds one source-contract assertion (both in-page sites use a shared
videoIdMarker) and one behavioral test (CDP capture buffer with a
stale 'v=prev' entry alongside the current 'v=abc' returns only the
current video's captions).

* fix(youtube): exact-match transcript timedtext video id

---------

Co-authored-by: ml-scout <ml-scout@anthropic.com>
Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-19 00:53:39 +08:00
Yabin Zheng 68ef95659f Fix YouTube transcript caption fetching (#1499)
* fix(youtube): unwrap transcript caption results

* fix(youtube): validate transcript caption info shape

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-15 17:49:02 +08:00
Kagura 6d84009ee8 fix(youtube): request srv3 format for caption URLs (#1420) (#1422)
* fix(youtube): request srv3 format for caption URLs (#1420)

YouTube may return empty responses when caption URLs lack an explicit format
parameter. This adds fmt=srv3 (standard YouTube XML caption format) to the
caption URL when no fmt parameter is already present, with a fallback to the
original URL if srv3 also returns empty.

Also adds HTTP status checking before reading the response body, preventing
silent failures on non-200 responses.

Fixes #1420

* fix(youtube): preserve caption fetch failures

---------

Co-authored-by: jackwener <jakevingoo@gmail.com>
2026-05-12 02:44:21 +08:00
Kagura abfd0e2180 fix(youtube): use watch page HTML for transcript captions (#1378)
Fixes #1376 — YouTube transcript command failed with `No captions available for this video` for all videos.

## Root cause
Transcript adapter used InnerTube `/youtubei/v1/player` API with Android client context (`clientName: 'ANDROID'`, version `20.10.38`) to retrieve caption track URLs. YouTube has restricted/deprecated this approach; the Android client no longer reliably returns captions data.

## Fix
Replace Step 1 (caption track retrieval) with watch page HTML bootstrap parsing — fetch `/watch?v=...` with cookies and extract `ytInitialPlayerResponse.captions.playerCaptionsTracklistRenderer`. This is the same approach used by sibling `clis/youtube/video.js`, so it's an alignment to existing site-local stable pattern, not a new invention.

## 2 head iteration
- `cf77f5e8` initial fix (Step 1 caption retrieval switch + 18/18 unit tests)
- `bb30788c` lead test hardening — source-contract regression test in `transcript.test.js`:
  - **positive lock**: must fetch `/watch?v=...`, parse `ytInitialPlayerResponse`, read `playerCaptionsTracklistRenderer`
  - **negative lock**: must NOT use `/youtubei/v1/player` or `clientName: 'ANDROID'` (prevents regression)
  - stale Android-InnerTube file header comment also updated

## Better-solution evaluation
- Official YouTube Data API captions surface (`developers.google.com/youtube/v3/docs/captions/download`) is owner-authorized API, NOT a public transcript replacement
- yt-dlp also relies on watch-page bootstrap path
- Existing `youtube/video.js` already uses the same `ytInitialPlayerResponse` extraction → this PR aligns transcript with stable site-local pattern instead of inventing a new path

## Typed failure / no-silent-empty boundaries
- watch HTML HTTP failure / missing `ytInitialPlayerResponse` / no `captionTracks` → `CommandExecutionError` (typed fail)
- Empty parsed XML → `EmptyResultError` (existing path, preserved)
- `Strategy.COOKIE` matches YouTube adapter family + `video.js`; cookies/session/consent unavailable → typed fail not silent empty success illusion

## Diff containment
Runtime change limited to Step 1 caption track discovery. XML fetch, segment parsing, chapters, raw/grouped formatting all unchanged.

## Verification
Local: YouTube adapter tests `19/19` (+1 from new test), `npm run build`, typed-error-lint `192/192`, silent-column-drop `103/103`, doc coverage `140/140`, `docs:build`, listing-id advisory unchanged `13`, `git diff --check`, merge-tree clean.
GitHub: build × 3 OS, unit × 2 shards, bun-test, adapter-test, audit, doc-coverage, docs-build all SUCCESS. PR CLEAN/MERGEABLE.

Author: kagura-agent (fork). Lead: codex-mini0. Aux: First-principles-0. Coordination: pr-monitor.
2026-05-07 21:50:30 +08:00
jakevin eea9ff8bfe feat(cli): add command access metadata (#1296) 2026-05-04 19:47:08 +08:00
jakevin d2974a9ff6 refactor(adapters): convert adapter layer from TypeScript to JavaScript (#928)
* refactor(adapters): convert adapter layer from TypeScript to JavaScript

Core framework stays TypeScript; adapter layer moves to JS-first.
Adapters are essentially "executable config + browser scripts" that
barely use TS features — this simplifies the build/distribution pipeline
by removing the dist/clis/ intermediate compilation step.

Changes:
- Convert all 753 adapter files in clis/ from .ts to .js
- Update tsconfig to exclude clis/ from compilation
- Simplify build-manifest to scan clis/*.js directly (no dist/clis/)
- Update discovery, main, fetch-adapters to load JS adapters from clis/
- Update generate-verified to output .js artifacts
- Update package.json files field: dist/clis/ → clis/
- Fix all test files for the .ts → .js transition

* fix(main): use findPackageRoot for BUILTIN_CLIS path

The previous relative path (../../clis from __dirname) only worked for
dist/src/main.js but broke dev mode (tsx src/main.ts) where __dirname
is <repo>/src — resolving to /clis instead of <repo>/clis.

Use findPackageRoot() which works for both dev and prod paths.
2026-04-10 14:52:18 +08:00