mirror of
https://github.com/jackwener/OpenCLI.git
synced 2026-09-14 18:25:42 +08:00
jakevin
f594e500a8
* feat: AutoResearch framework + V2EX test suite (40 tasks) AutoResearch framework (Karpathy-style autonomous iteration): - engine.ts: 8-phase loop (review → modify → commit → verify → guard → decide → log) - config.ts: typed config + CLI parser + metric extraction - logger.ts: TSV append-only results log - commands/run.ts: main loop spawning Claude Code per iteration - commands/plan.ts: interactive config wizard - commands/fix.ts: auto-detect broken state, iteratively fix - commands/debug.ts: hypothesis-driven debugging for failing tasks V2EX test suite (5 layers, 40 tasks): - L1 Atomic (10): open, state, click, scroll, eval, back, wait - L2 Single Page (10): hot topics, node list, topic meta, pagination - L3 Multi-Step (10): click-read, navigate-node, tab-then-topic, pagination - L4 Write Ops (5): reply typing, favorite detection, form detection - L5 Complex Chain (5): cross-page collect, multi-node compare, full workflow Presets: operate-reliability, skill-quality, v2ex-reliability * test: V2EX test suite 60/60 — fix selectors, add harder tasks - Fix v2ex-collect-hot-authors selector (pathname-based member link detection) - Fix v2ex-wait-text judge (accept "appeared") - Fix trailing commas in eval step strings - Add 20 harder tasks: state+click interaction + long chain workflows - Baseline: 60/60 across all layers * feat: Zhihu test suite — 60 tasks across 8 layers, 60/60 passing Knowledge-intensive Chinese Q&A site (React SPA, lazy loading, complex DOM): - L1 Atomic (10): open, state, title, url, scroll, tab, back, wait, keys, screenshot - L2 Feed (8): feed titles, hot list, metrics, tabs, authors, content types, avatar, search - L3 Question (8): title, meta, answer, votes, buttons, descriptions, answer count - L4 Navigation (8): hot→question, feed→question, author profile, search, topic, user, back - L5 Write (6): upvote/follow/comment/bookmark/write-answer/share button detection - L6 Chain (8): read-answer-author, author-profile, multi-hot, search-then-read, scroll-answers - L7 Search (6): basic, people, topic, click-result, filter, back - L8 Complex (6): full workflow, deep author chain, cross-question, search-read, 3-page, scroll-deep Key fixes during development: - Zhihu search page needs 5s+ wait (SPA lazy loading) - Back navigation goes to about:blank (daemon init page), fixed with direct navigate - User profile answers page needs 4s wait for content - Broader selectors needed (h2 a instead of specific class names) * feat: combined eval-all runner + combined-reliability preset * experiment(operate): fix extract-npm-description + nav-click-link-example Round 1: Fix 2 remaining browse-tasks failures: - extract-npm-description: use generic <p> selector instead of class-based - nav-click-link-example: include URL in output (title is 'Example Domains', not 'IANA') * experiment(operate): fix bench-imdb-matrix — use broader selectors for year/rating Round 2: IMDB page selectors were too specific (data-testid changed). Use generic h1 for title, link text match for year, broader class match for rating. * experiment(operate): add edge cases + fix SPA navigation timing Round 3: Add 10 edge case tasks (5 V2EX + 5 Zhihu): - rapid-navigate: 3 consecutive opens - eval-after-click: verify URL changes after SPA click - scroll-and-extract: extract after deep scroll - structured extraction: multi-field JSON from dynamic content - lazy-load answers: scroll triggers more content Key finding: Zhihu SPA click() doesn't update location.pathname immediately. Use window.location.href = a.href for reliable navigation. V2EX: 65/65, Zhihu: 65/65, Browse: 59/59 = 189/189 * experiment(operate): add agent-style tasks using state+click+type (no eval for interaction) Round 4-5: Add 5 tasks that test the actual agent workflow: - agent-click-first-topic: find topic index via data-opencli-ref - agent-type-search: type into search using state index - agent-click-navigate-back: click by ref, verify navigation - agent-state-has-interactive: verify state output format - agent-state-after-scroll: verify scroll position in state V2EX: 70/70 tasks * fix: review fixes — extractVerdict, stderr, dead code - eval-skill.ts: remove dead TASKS_FILE variable (skill-tasks.yaml never existed) - eval-skill.ts: rewrite extractVerdict to use brace-counting JSON.parse instead of regex (handles escaped quotes in explanation) - eval-browse.ts: include stderr in runCommand error output for debuggability
28 lines
777 B
TypeScript
28 lines
777 B
TypeScript
/**
|
|
* Preset: Combined Reliability (browse + V2EX + Zhihu)
|
|
*
|
|
* Optimizes across ALL test suites simultaneously.
|
|
* Current baseline: 57/59 + 60/60 + 60/60 = 177/179
|
|
* Target: 179/179 (100%)
|
|
*/
|
|
|
|
import type { AutoResearchConfig } from '../config.js';
|
|
|
|
export const combinedReliability: AutoResearchConfig = {
|
|
goal: 'Fix all remaining test failures across browse + V2EX + Zhihu (177/179 → 179/179)',
|
|
scope: [
|
|
'src/browser/dom-snapshot.ts',
|
|
'src/browser/dom-helpers.ts',
|
|
'src/browser/base-page.ts',
|
|
'src/browser/page.ts',
|
|
'src/cli.ts',
|
|
'autoresearch/browse-tasks.json',
|
|
],
|
|
metric: 'pass_count',
|
|
direction: 'higher',
|
|
verify: 'npx tsx autoresearch/eval-all.ts 2>&1 | tail -1',
|
|
guard: 'npm run build',
|
|
iterations: 10,
|
|
minDelta: 1,
|
|
};
|