mirror of
https://github.com/jackwener/OpenCLI.git
synced 2026-09-14 18:25:42 +08:00
jakevin
f594e500a8
* feat: AutoResearch framework + V2EX test suite (40 tasks) AutoResearch framework (Karpathy-style autonomous iteration): - engine.ts: 8-phase loop (review → modify → commit → verify → guard → decide → log) - config.ts: typed config + CLI parser + metric extraction - logger.ts: TSV append-only results log - commands/run.ts: main loop spawning Claude Code per iteration - commands/plan.ts: interactive config wizard - commands/fix.ts: auto-detect broken state, iteratively fix - commands/debug.ts: hypothesis-driven debugging for failing tasks V2EX test suite (5 layers, 40 tasks): - L1 Atomic (10): open, state, click, scroll, eval, back, wait - L2 Single Page (10): hot topics, node list, topic meta, pagination - L3 Multi-Step (10): click-read, navigate-node, tab-then-topic, pagination - L4 Write Ops (5): reply typing, favorite detection, form detection - L5 Complex Chain (5): cross-page collect, multi-node compare, full workflow Presets: operate-reliability, skill-quality, v2ex-reliability * test: V2EX test suite 60/60 — fix selectors, add harder tasks - Fix v2ex-collect-hot-authors selector (pathname-based member link detection) - Fix v2ex-wait-text judge (accept "appeared") - Fix trailing commas in eval step strings - Add 20 harder tasks: state+click interaction + long chain workflows - Baseline: 60/60 across all layers * feat: Zhihu test suite — 60 tasks across 8 layers, 60/60 passing Knowledge-intensive Chinese Q&A site (React SPA, lazy loading, complex DOM): - L1 Atomic (10): open, state, title, url, scroll, tab, back, wait, keys, screenshot - L2 Feed (8): feed titles, hot list, metrics, tabs, authors, content types, avatar, search - L3 Question (8): title, meta, answer, votes, buttons, descriptions, answer count - L4 Navigation (8): hot→question, feed→question, author profile, search, topic, user, back - L5 Write (6): upvote/follow/comment/bookmark/write-answer/share button detection - L6 Chain (8): read-answer-author, author-profile, multi-hot, search-then-read, scroll-answers - L7 Search (6): basic, people, topic, click-result, filter, back - L8 Complex (6): full workflow, deep author chain, cross-question, search-read, 3-page, scroll-deep Key fixes during development: - Zhihu search page needs 5s+ wait (SPA lazy loading) - Back navigation goes to about:blank (daemon init page), fixed with direct navigate - User profile answers page needs 4s wait for content - Broader selectors needed (h2 a instead of specific class names) * feat: combined eval-all runner + combined-reliability preset * experiment(operate): fix extract-npm-description + nav-click-link-example Round 1: Fix 2 remaining browse-tasks failures: - extract-npm-description: use generic <p> selector instead of class-based - nav-click-link-example: include URL in output (title is 'Example Domains', not 'IANA') * experiment(operate): fix bench-imdb-matrix — use broader selectors for year/rating Round 2: IMDB page selectors were too specific (data-testid changed). Use generic h1 for title, link text match for year, broader class match for rating. * experiment(operate): add edge cases + fix SPA navigation timing Round 3: Add 10 edge case tasks (5 V2EX + 5 Zhihu): - rapid-navigate: 3 consecutive opens - eval-after-click: verify URL changes after SPA click - scroll-and-extract: extract after deep scroll - structured extraction: multi-field JSON from dynamic content - lazy-load answers: scroll triggers more content Key finding: Zhihu SPA click() doesn't update location.pathname immediately. Use window.location.href = a.href for reliable navigation. V2EX: 65/65, Zhihu: 65/65, Browse: 59/59 = 189/189 * experiment(operate): add agent-style tasks using state+click+type (no eval for interaction) Round 4-5: Add 5 tasks that test the actual agent workflow: - agent-click-first-topic: find topic index via data-opencli-ref - agent-type-search: type into search using state index - agent-click-navigate-back: click by ref, verify navigation - agent-state-has-interactive: verify state output format - agent-state-after-scroll: verify scroll position in state V2EX: 70/70 tasks * fix: review fixes — extractVerdict, stderr, dead code - eval-skill.ts: remove dead TASKS_FILE variable (skill-tasks.yaml never existed) - eval-skill.ts: rewrite extractVerdict to use brace-counting JSON.parse instead of regex (handles escaped quotes in explanation) - eval-browse.ts: include stderr in runCommand error output for debuggability
848 lines
36 KiB
JSON
848 lines
36 KiB
JSON
[
|
|
{
|
|
"_comment": "=== L1: Atomic Operations (10 tasks) ==="
|
|
},
|
|
{
|
|
"name": "zhihu-open-home",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/"
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "Navigated to"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-get-title",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"document.title\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "知乎"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-state",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate state"
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\\[@?\\d+\\]"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-get-url",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate get url"
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "zhihu.com/hot"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-scroll-down",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate scroll down --amount 500",
|
|
"opencli operate eval \"window.scrollY > 100 ? 'scrolled' : 'not scrolled'\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "scrolled"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-click-tab-hot",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('nav a[href*=hot]');if(a){a.click();return 'clicked';}return 'not found';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "clicked"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-back-navigation",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate back",
|
|
"opencli operate get url"
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "zhihu\\.com/?$"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-wait-page-load",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate wait text 推荐",
|
|
"opencli operate eval \"document.querySelector('nav')?.textContent?.includes('推荐') ? 'loaded' : 'empty'\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "loaded"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-keys-escape",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate keys Escape"
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "pressed|Pressed"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-screenshot",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate screenshot /tmp/zhihu-test.png"
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"_comment": "=== L2: Homepage & Feed Extraction (8 tasks) ==="
|
|
},
|
|
{
|
|
"name": "zhihu-feed-titles",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('h2.ContentItem-title a, .ContentItem-title a');var r=[];for(var i=0;i<Math.min(items.length,5);i++){r.push(items[i].textContent.trim().slice(0,60));}return JSON.stringify(r);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "arrayMinLength",
|
|
"minLength": 3
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-hot-list",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('.HotItem-content a');var r=[];for(var i=0;i<Math.min(items.length,5);i++){r.push({title:items[i].textContent.trim().slice(0,50),href:items[i].pathname});}return JSON.stringify(r);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "arrayMinLength",
|
|
"minLength": 5
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-hot-metrics",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var metrics=document.querySelectorAll('.HotItem-metrics');var r=[];for(var i=0;i<Math.min(metrics.length,5);i++){r.push(metrics[i].textContent.trim());}return JSON.stringify(r);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "arrayMinLength",
|
|
"minLength": 3
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-nav-tabs",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"(function(){var tabs=document.querySelectorAll('nav a');var r=[];for(var i=0;i<tabs.length;i++){r.push(tabs[i].textContent.trim());}return JSON.stringify(r);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "推荐"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-feed-with-authors",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('.ContentItem');var r=[];for(var i=0;i<Math.min(items.length,3);i++){var title=items[i].querySelector('h2 a')?.textContent?.trim()||'';var author=items[i].querySelector('.AuthorInfo-name')?.textContent?.trim()||'';if(title)r.push({title:title.slice(0,40),author:author});}return JSON.stringify(r);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "arrayMinLength",
|
|
"minLength": 2
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-feed-types",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"(function(){var links=document.querySelectorAll('h2.ContentItem-title a, .ContentItem-title a');var types={question:0,article:0,other:0};for(var i=0;i<links.length;i++){var h=links[i].pathname||'';if(h.includes('/question/'))types.question++;else if(h.includes('/p/'))types.article++;else types.other++;}return JSON.stringify(types);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"question\":\\d+"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-user-avatar",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"(function(){var img=document.querySelector('img[alt*=\\\"头像\\\"],img[alt*=\\\"主页\\\"],img[class*=\\\"Avatar\\\"]');return img?img.src:'no avatar';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-search-input-exists",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"(function(){var input=document.querySelector('input[role=combobox],input[type=search]');return input?'search found':'no search';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "search found"
|
|
}
|
|
},
|
|
{
|
|
"_comment": "=== L3: Question Page Operations (8 tasks) ==="
|
|
},
|
|
{
|
|
"name": "zhihu-question-title",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');return a?a.href:'none';})()\"",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"document.querySelector('.QuestionHeader-title')?.textContent?.trim()?.slice(0,80)||document.title\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-question-meta",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var title=document.querySelector('.QuestionHeader-title')?.textContent?.trim()||'';var answerCount=document.querySelector('.List-headerText')?.textContent?.trim()||'';var followers=document.querySelector('[class*=FollowButton]')?.textContent?.trim()||'';return JSON.stringify({title:title.slice(0,60),answerCount:answerCount,followers:followers});})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"title\":"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-first-answer",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var content=document.querySelector('.RichContent-inner')?.textContent?.trim()?.slice(0,200)||'';var author=document.querySelector('.AuthorInfo-name')?.textContent?.trim()||'';return JSON.stringify({author:author,content:content});})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"content\":"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-answer-votes",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var btns=document.querySelectorAll('button[class*=VoteButton]');var r=[];for(var i=0;i<Math.min(btns.length,6);i++){var label=btns[i].getAttribute('aria-label')||btns[i].textContent.trim();if(label)r.push(label.slice(0,30));}return JSON.stringify(r);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "arrayMinLength",
|
|
"minLength": 2
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-question-buttons",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var btns=document.querySelectorAll('button');var r=[];for(var i=0;i<btns.length;i++){var t=btns[i].textContent.trim();if(t.length>0&&t.length<25)r.push(t);}return JSON.stringify(r.slice(0,15));})()\""
|
|
],
|
|
"judge": {
|
|
"type": "arrayMinLength",
|
|
"minLength": 3
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-multiple-answers",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var answers=document.querySelectorAll('.List-item .RichContent-inner');var r=[];for(var i=0;i<Math.min(answers.length,3);i++){r.push(answers[i].textContent.trim().slice(0,80));}return JSON.stringify(r);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "arrayMinLength",
|
|
"minLength": 1
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-question-description",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var desc=document.querySelector('.QuestionRichText')?.textContent?.trim()?.slice(0,200)||'no description';return desc;})()\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-answer-count-number",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var header=document.querySelector('.List-headerText');if(!header)return '0';var m=header.textContent.match(/\\\\d+/);return m?m[0]:'0';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\\d+"
|
|
}
|
|
},
|
|
{
|
|
"_comment": "=== L4: Multi-Step Navigation (8 tasks) ==="
|
|
},
|
|
{
|
|
"name": "zhihu-hot-to-question",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){a.click();return 'clicked: '+a.textContent.trim().slice(0,30);}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"document.querySelector('.QuestionHeader-title')?.textContent?.trim()?.slice(0,80)||document.title\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-feed-to-question",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('h2.ContentItem-title a');if(a){a.click();return 'clicked';}return 'none';})()\"",
|
|
"opencli operate wait time 3",
|
|
"opencli operate eval \"document.querySelector('.QuestionHeader-title')?.textContent?.trim()?.slice(0,80)||document.title\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-question-to-author",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var author=document.querySelector('.AuthorInfo-name a, .UserLink-link');if(author){var name=author.textContent.trim();window.location.href=author.href;return name;}return 'no author';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"document.title\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-search-navigate",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/search?type=content&q=TypeScript",
|
|
"opencli operate wait time 5",
|
|
"opencli operate scroll down --amount 300",
|
|
"opencli operate wait time 1",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('h2 a');var r=[];var seen={};for(var i=0;i<items.length;i++){var t=items[i].textContent.trim();if(t.length>5&&t.length<100&&!seen[t]){seen[t]=1;r.push(t.slice(0,50));}}return JSON.stringify(r.slice(0,5));})()\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-topic-page",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/topic/19552832/hot",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var title=document.querySelector('.TopicName, .ContentItem-title, h1')?.textContent?.trim()||document.title;return title;})()\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-user-profile",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/people/excited-vczh",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var name=document.querySelector('.ProfileHeader-title .ProfileHeader-name')?.textContent?.trim()||document.querySelector('h1')?.textContent?.trim()||'';var bio=document.querySelector('.ProfileHeader-headline')?.textContent?.trim()||'';return JSON.stringify({name:name,bio:bio.slice(0,100)});})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"name\":"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-question-and-back",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){a.click();return 'clicked';}return 'none';})()\"",
|
|
"opencli operate wait time 3",
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate get url"
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "zhihu.com/hot"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-scroll-load-more",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"document.querySelectorAll('.HotItem-content').length\"",
|
|
"opencli operate scroll down --amount 2000",
|
|
"opencli operate wait time 1",
|
|
"opencli operate eval \"document.querySelectorAll('.HotItem-content').length\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\\d+"
|
|
}
|
|
},
|
|
{
|
|
"_comment": "=== L5: Write Operations (6 tasks, requires login) ==="
|
|
},
|
|
{
|
|
"name": "zhihu-upvote-button-find",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var btn=document.querySelector('button[aria-label*=赞同]');return btn?JSON.stringify({text:btn.textContent.trim(),ariaLabel:btn.getAttribute('aria-label')}):'no upvote button';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "赞同"
|
|
},
|
|
"note": "Finds upvote button but does NOT click"
|
|
},
|
|
{
|
|
"name": "zhihu-follow-question-find",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var btns=document.querySelectorAll('button');for(var i=0;i<btns.length;i++){if(btns[i].textContent.includes('关注问题'))return 'found: '+btns[i].textContent.trim();}return 'not found';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "关注问题"
|
|
},
|
|
"note": "Finds follow button but does NOT click"
|
|
},
|
|
{
|
|
"name": "zhihu-comment-button-find",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var btns=document.querySelectorAll('button');for(var i=0;i<btns.length;i++){if(btns[i].textContent.includes('评论'))return 'found: '+btns[i].textContent.trim();}return 'not found';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "评论"
|
|
},
|
|
"note": "Finds comment button but does NOT click"
|
|
},
|
|
{
|
|
"name": "zhihu-bookmark-find",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var btns=document.querySelectorAll('button');for(var i=0;i<btns.length;i++){var t=btns[i].textContent.trim();if(t.includes('收藏')||t.includes('Bookmark'))return 'found: '+t;}return 'not found';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "收藏|found"
|
|
},
|
|
"note": "Finds bookmark button but does NOT click"
|
|
},
|
|
{
|
|
"name": "zhihu-write-answer-btn",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var btns=document.querySelectorAll('button');for(var i=0;i<btns.length;i++){if(btns[i].textContent.includes('写回答'))return 'found: '+btns[i].textContent.trim();}return 'not found';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "写回答"
|
|
},
|
|
"note": "Finds write answer button but does NOT click"
|
|
},
|
|
{
|
|
"name": "zhihu-share-find",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var btns=document.querySelectorAll('button');for(var i=0;i<btns.length;i++){if(btns[i].textContent.includes('分享'))return 'found: '+btns[i].textContent.trim();}return 'not found';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "分享"
|
|
},
|
|
"note": "Finds share button but does NOT click"
|
|
},
|
|
{
|
|
"_comment": "=== L6: Long Chain Workflows (8 tasks) ==="
|
|
},
|
|
{
|
|
"name": "zhihu-hot-read-answer-author",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){a.click();return 'clicked';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var author=document.querySelector('.AuthorInfo-name')?.textContent?.trim()||'';var content=document.querySelector('.RichContent-inner')?.textContent?.trim()?.slice(0,100)||'';return JSON.stringify({author:author,content:content});})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"author\":"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-hot-to-author-profile",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var link=document.querySelector('.AuthorInfo-name a, .UserLink-link');if(link){window.location.href=link.href;return 'going to author';}return 'no author link';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var name=document.querySelector('h1, .ProfileHeader-name')?.textContent?.trim()||document.title;return name;})()\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-multi-hot-topics",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('.HotItem-content a');var r=[];for(var i=0;i<Math.min(items.length,3);i++){r.push(items[i].textContent.trim().slice(0,40));}return JSON.stringify(r);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "arrayMinLength",
|
|
"minLength": 3
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-search-then-read",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/search?type=content&q=Python",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.ContentItem-title a, .SearchResult-Card h2 a');if(a){a.click();return 'clicked';}return 'none';})()\"",
|
|
"opencli operate wait time 3",
|
|
"opencli operate eval \"document.querySelector('.QuestionHeader-title, h1')?.textContent?.trim()?.slice(0,80)||document.title\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-question-scroll-answers",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate scroll down --amount 1000",
|
|
"opencli operate wait time 1",
|
|
"opencli operate eval \"document.querySelectorAll('.RichContent-inner').length\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\\d+"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-compare-tabs",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('h2.ContentItem-title a');return a?a.textContent.trim().slice(0,40):'none';})()\"",
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');return a?a.textContent.trim().slice(0,40):'none';})()\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-user-answers",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/people/excited-vczh/answers",
|
|
"opencli operate wait time 4",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('h2 a, [class*=title] a, [class*=Title] a');var r=[];for(var i=0;i<items.length;i++){var t=items[i].textContent.trim();if(t.length>10&&t.length<100)r.push(t.slice(0,50));}return JSON.stringify(r.slice(0,3));})()\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-topic-questions",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/topic/19552832/hot",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('.ContentItem-title a, h2 a');var r=[];for(var i=0;i<Math.min(items.length,3);i++){var t=items[i].textContent.trim();if(t.length>5)r.push(t.slice(0,50));}return JSON.stringify(r);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "arrayMinLength",
|
|
"minLength": 1
|
|
}
|
|
},
|
|
{
|
|
"_comment": "=== L7: Search Workflows (6 tasks) ==="
|
|
},
|
|
{
|
|
"name": "zhihu-search-basic",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/search?type=content&q=AI",
|
|
"opencli operate wait time 5",
|
|
"opencli operate scroll down --amount 300",
|
|
"opencli operate wait time 1",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('h2 a');var r=[];var seen={};for(var i=0;i<items.length;i++){var t=items[i].textContent.trim();if(t.length>5&&t.length<100&&!seen[t]){seen[t]=1;r.push(t.slice(0,50));}}return JSON.stringify(r.slice(0,5));})()\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-search-people",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/search?type=people&q=Python",
|
|
"opencli operate wait time 3",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('[class*=UserItem] a, [class*=user] a, .List-item a');var r=[];for(var i=0;i<Math.min(items.length,10);i++){var t=items[i].textContent.trim();if(t.length>1&&t.length<30)r.push(t);}return JSON.stringify(r.slice(0,3));})()\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-search-topic",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/search?type=topic&q=编程",
|
|
"opencli operate wait time 3",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('a');var r=[];for(var i=0;i<items.length;i++){var t=items[i].textContent.trim();if(t.length>2&&t.length<30&&(t.includes('编程')||items[i].pathname?.includes('/topic/')))r.push(t);}return JSON.stringify(r.slice(0,3));})()\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-search-click-result",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/search?type=content&q=Rust编程",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.ContentItem-title a, h2 a');if(a){a.click();return 'clicked';}return 'none';})()\"",
|
|
"opencli operate wait time 3",
|
|
"opencli operate eval \"document.title\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-search-filter-answers",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/search?type=content&q=Docker",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('.ContentItem');return JSON.stringify({total:items.length,hasAnswers:items.length>0});})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"total\":\\d+"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-search-and-back",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/search?type=content&q=前端开发",
|
|
"opencli operate wait time 5",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('h2 a, [class*=title] a');if(a){a.click();return 'clicked';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate open https://www.zhihu.com/search?type=content&q=前端开发",
|
|
"opencli operate wait time 3",
|
|
"opencli operate get url"
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "search"
|
|
}
|
|
},
|
|
{
|
|
"_comment": "=== L8: Complex Long Chain (6 tasks) ==="
|
|
},
|
|
{
|
|
"name": "zhihu-full-browse-workflow",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('.HotItem-content a');var r=[];for(var i=0;i<Math.min(items.length,3);i++){r.push(items[i].textContent.trim().slice(0,30));}return JSON.stringify(r);})()\"",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){a.click();return 'clicked';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var title=document.querySelector('.QuestionHeader-title')?.textContent?.trim()?.slice(0,60)||'';var answers=document.querySelectorAll('.RichContent-inner').length;return JSON.stringify({title:title,answers:answers});})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"title\":"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-deep-author-chain",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'step1';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var link=document.querySelector('.AuthorInfo-name a');if(link){var name=link.textContent.trim();window.location.href=link.href;return 'step2: '+name;}return 'no author';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var answers=document.querySelectorAll('.ContentItem-title a');var r=[];for(var i=0;i<Math.min(answers.length,2);i++){r.push(answers[i].textContent.trim().slice(0,40));}return JSON.stringify({profile:document.title,recentAnswers:r});})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"profile\":"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-cross-question-compare",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('.HotItem-content a');return items.length>=2?JSON.stringify([items[0].textContent.trim().slice(0,30),items[1].textContent.trim().slice(0,30)]):'not enough';})()\"",
|
|
"opencli operate eval \"(function(){var a=document.querySelectorAll('.HotItem-content a')[0];if(a){window.location.href=a.href;return 'q1';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){return JSON.stringify({q1_title:document.querySelector('.QuestionHeader-title')?.textContent?.trim()?.slice(0,40)||'',q1_answers:document.querySelectorAll('.RichContent-inner').length});})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"q1_title\":"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-search-read-chain",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/search?type=content&q=Claude",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('.ContentItem-title a, h2 a');var r=[];for(var i=0;i<Math.min(items.length,3);i++){r.push(items[i].textContent.trim().slice(0,40));}return JSON.stringify(r);})()\"",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.ContentItem-title a, h2 a');if(a){a.click();return 'clicked';}return 'none';})()\"",
|
|
"opencli operate wait time 3",
|
|
"opencli operate eval \"document.querySelector('.QuestionHeader-title, h1')?.textContent?.trim()?.slice(0,60)||document.title\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-3-page-chain",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate eval \"document.title\"",
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"document.querySelectorAll('.HotItem-content').length\"",
|
|
"opencli operate open https://www.zhihu.com/people/excited-vczh",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"document.title\""
|
|
],
|
|
"judge": {
|
|
"type": "nonEmpty"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-hot-scroll-deep-read",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate scroll down --amount 1000",
|
|
"opencli operate eval \"document.querySelectorAll('.HotItem-content a').length\"",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('.HotItem-content a');var last=items[items.length-1];if(last){last.click();return 'clicked last';}return 'none';})()\"",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"(function(){var title=document.querySelector('.QuestionHeader-title')?.textContent?.trim()||document.title;var firstAnswer=document.querySelector('.RichContent-inner')?.textContent?.trim()?.slice(0,100)||'';return JSON.stringify({title:title.slice(0,60),firstAnswer:firstAnswer});})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"title\":"
|
|
}
|
|
},
|
|
{
|
|
"_comment": "=== Edge Cases: SPA lazy load, dynamic content ==="
|
|
},
|
|
{
|
|
"name": "zhihu-rapid-navigate",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/",
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate open https://www.zhihu.com/people/excited-vczh",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"location.pathname\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "/people/"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-hot-click-verify-url",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 3",
|
|
"opencli operate eval \"location.pathname.startsWith('/question/') ? 'on question page' : 'wrong: '+location.pathname\""
|
|
],
|
|
"judge": {
|
|
"type": "contains",
|
|
"value": "on question page"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-scroll-lazy-answers",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 3",
|
|
"opencli operate eval \"document.querySelectorAll('.RichContent-inner').length\"",
|
|
"opencli operate scroll down --amount 2000",
|
|
"opencli operate wait time 2",
|
|
"opencli operate eval \"document.querySelectorAll('.RichContent-inner').length\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\\d+"
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-extract-structured",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var items=document.querySelectorAll('.HotItem-content');var r=[];for(var i=0;i<Math.min(items.length,3);i++){var a=items[i].querySelector('a');var m=items[i].closest('[class*=HotItem]')?.querySelector('[class*=metrics]');r.push({title:(a?.textContent||'').trim().slice(0,40),heat:(m?.textContent||'').trim()});}return JSON.stringify(r);})()\""
|
|
],
|
|
"judge": {
|
|
"type": "arrayMinLength",
|
|
"minLength": 3
|
|
}
|
|
},
|
|
{
|
|
"name": "zhihu-question-answer-chain",
|
|
"steps": [
|
|
"opencli operate open https://www.zhihu.com/hot",
|
|
"opencli operate eval \"(function(){var a=document.querySelector('.HotItem-content a');if(a){window.location.href=a.href;return 'navigating';}return 'none';})()\"",
|
|
"opencli operate wait time 3",
|
|
"opencli operate eval \"(function(){var title=document.querySelector('.QuestionHeader-title')?.textContent?.trim()||'';var answers=document.querySelectorAll('.RichContent-inner');var first=answers[0]?.textContent?.trim()?.slice(0,100)||'';var count=answers.length;return JSON.stringify({title:title.slice(0,50),firstAnswer:first,answerCount:count});})()\""
|
|
],
|
|
"judge": {
|
|
"type": "matchesPattern",
|
|
"pattern": "\"answerCount\":\\d+"
|
|
}
|
|
}
|
|
] |