expose proxy-attempts + use-profile CORS headers; simplify DDG url builders

feat: update DuckDuckGo search logic, use "more results" button for next page results (#41 )
* feat: implement windowOrganicResults function and update DuckDuckGo search logic * Add fallback ddg "more results" selector. Update test HTML + add test --------- Co-authored-by: Rustem Kamalov <karust13@gmail.com>
2026-08-28 10:36:23 +08:00 · 2026-07-22 15:17:16 +03:00 · 2026-07-22 00:11:20 +03:00 · 2026-07-19 04:18:50 +03:00 · 2026-07-18 21:44:46 +03:00 · 2026-07-18 21:31:46 +03:00
105 changed files with 5222 additions and 2841 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -39,3 +39,4 @@ core/testdata/*
 .tmp-gocache
 .tmpcache/
 google/data/geotargets-2026-05-28.csv
+demo.yml
--- a/.golangci.yml
+++ b/.golangci.yml
@@ -1,6 +1,19 @@
 run:
  tests: true
  timeout: 5m
+  go: "1.24"
+
+# Pinned set so CI and local runs agree
+linters:
+  disable-all: true
+  enable:
+    - govet
+    - staticcheck
+    - gosimple
+    - errcheck
+    - ineffassign
+    - unused
+    - misspell

 issues:
  max-issues-per-linter: 0
--- a/2
+++ b/2
@@ -12,7 +12,7 @@ ARG TARGETARCH
 RUN CGO_ENABLED=0 GOOS=${TARGETOS:-linux} GOARCH=${TARGETARCH:-amd64} go build -trimpath -ldflags="-s -w" -o /app/openserp .

 # `chromedp/headless-shell:stable` also works here
-FROM chromedp/headless-shell:stable@sha256:aac539266027f91cf47610da1129dce360d23f45f8f150683cca94223fa2f1e2
+FROM chromedp/headless-shell:stable@sha256:f7e7ac721b023cb8717f8108aef8b3e49995fb1e5a912f41e570c29e45d24961

 WORKDIR /usr/src/app

--- a/README.md
+++ b/README.md
@@ -1,49 +1,49 @@
 ![OpenSERP](./logo.svg)

-# OpenSERP (Search Engine Results)
+# OpenSERP

-[![Go Report Card](https://goreportcard.com/badge/github.com/karust/openserp)](https://goreportcard.com/report/github.com/karust/openserp)
-[![Go Reference](https://pkg.go.dev/badge/github/karust/openserp?style=for-the-badge)](https://pkg.go.dev/github.com/karust/openserp)
+[![Go Reference](https://pkg.go.dev/badge/github.com/karust/openserp.svg)](https://pkg.go.dev/github.com/karust/openserp)
 [![release](https://img.shields.io/github/v/release/karust/openserp)](https://github.com/karust/openserp/releases)
 [![Docker Pulls](https://img.shields.io/docker/v/karust/openserp)](https://hub.docker.com/r/karust/openserp)
 [![CI](https://github.com/karust/openserp/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/karust/openserp/actions/workflows/ci.yml)
+[![Telegram](https://img.shields.io/badge/Telegram-openserp__cloud-26A5E4?logo=telegram&logoColor=white)](https://t.me/openserp_cloud)

-**OpenSERP** is a free, open-source API and CLI for accessing normalized search engine results from **Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia**.
+**OpenSERP** is a free, open-source SERP API and CLI for Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia.

-Run it locally, self-host it, or use the optional hosted API when you do not want to manage infrastructure.
+No API keys, no per-search billing: one command gives you live, structured search results on localhost - including engines the paid APIs don't cover. Use it as a search tool for LLMs and agents, or as a backend for SEO rank tracking. If you'd rather not run infrastructure, there is a [hosted version](https://openserp.org/cloud) with the same API.

-**Official website:** [openserp.org](https://openserp.org)
-
-**Feedback:** [GitHub Issues](https://github.com/karust/openserp/issues) or [feedback@openserp.org](mailto:feedback@openserp.org)
-
-**Latest updates, usage examples**: [Telegram](https://t.me/+RJEKspw3mUlhZDMy)
-
-> 💡 OpenSERP is free and open-source. Only links listed in this repository and on the official website are associated with the project.
+![OpenSERP CLI demo](./docs/demo.gif)

 ## Features

- 🔍 **Multi-engine** - search with dedicated endpoints for each engine
- 🌐 **Megasearch** - cross-engine aggregation with deduplication
- 🖼 **Images** - image search is also available
- 🎯 **Advanced filters** - language, date range, file type, and site queries
- ✨ **SERP features** - AI summaries, answer boxes, people-also-ask, and related searches in a response
- 📄 **URL extraction** - turn target pages into clean markdown/text for grounding and automation
- 🌍 **Configurable** - proxy, cache, and resilient mode
- 🐳 **Docker-ready** - local and container deployment
- 📝 **Data Formats** - JSON, Markdown, Text, NdJSON response formats
+- Dedicated endpoints for six engines, same JSON schema across all of them
+- **Megasearch** - one query across several engines at once, merged and deduped
+- **URL extraction** - search results plus clean markdown of the target pages in one call
+- SERP features: AI summaries, answer boxes, people-also-ask, related searches
+- Image search, filters (language, date range, file type, site), **JSON/Markdown/Text/NdJSON** output
+- Proxies, cache, resilient mode, prebuilt Docker images

-## ⚡ Quick Start
+## Quick Start

 ### Docker

+Prebuilt images are published to [docker hub: `karust/openserp`](https://hub.docker.com/r/karust/openserp).
+
 ```sh
 # Run the API server via prebuilt image
 docker run --rm -p 127.0.0.1:7000:7000 karust/openserp:latest serve -a 0.0.0.0 -p 7000

-# Or use docker-compose (pulls the prebuilt image)
+# Or
 docker compose up
 ```

+### Go install
+
+```sh
+go install github.com/karust/openserp@latest
+openserp search duckduckgo "open source serp api" --format markdown
+```
+
 ### From Source

 ```sh
@@ -53,32 +53,124 @@ go build -o openserp .
 ./openserp serve
 ```

-## Deployment Options
+### First request

- **Self-hosted (this repo)** - free, MIT-licensed, with full control over runtime, proxies, cache, and scaling.
- **[Hosted API](https://openserp.org/cloud)** - optional managed version from the project maintainers, with the same API shape.
+```sh
+# mode=any returns the first engine that responds
+curl "http://127.0.0.1:7000/mega/search?engines=bing,google&text=golang+vs+rust&extract=1&mode=any"
+```

-The hosted API helps fund continued development of the open-source project. Same endpoints, same response schema, and client code can migrate either direction.
+<details>
+<summary>Example JSON response</summary>

-## API Docs
+```json
+{
+  "query": {
+    "text": "golang vs rust",
+    "engines_requested": ["bing", "google"]
+  },
+  "meta": {
+    "request_id": "019ecdc0-a66d-79a4-9d2b-9e9b480d495e",
+    "requested_at": "2026-06-16T00:06:55Z",
+    "took_ms": 720,
+    "engines_responded": ["bing"],
+    "engines_failed": [],
+    "version": "2.1"
+  },
+  "results": [
+    {
+      "id": "s_5a8273f16b19ab64",
+      "rank": 1,
+      "type": "organic",
+      "title": "The Go Programming Language",
+      "url": "https://go.dev/",
+      "display_url": "go.dev",
+      "snippet": "Get Started Playground Tour Stack Overflow Help Packages Standard Library …",
+      "domain": "go.dev",
+      "favicon": "https://go.dev/favicon.ico",
+      "position": {
+        "absolute": 1
+      },
+      "engine": "bing",
+      "domain_info": {
+        "tld": "dev",
+        "sld": "go",
+        "category": ""
+      },
+      "extracted": {
+        "title": "Build simple, secure, scalable systems with Go",
+        "format": "markdown",
+        "content": "## Build simple, secure, scalable systems with Go\n\n![Go Gopher climbing a ladder.](https://go.dev/images/gophers/ladder.svg)\n\n- “At the time, no single team member knew Go, but **within a month, everyone was writing in Go** and we were building out the endpoints. ........",
+        "mode_used": "fast",
+        "fetched_at": "2026-06-16T00:06:56Z"
+      }
+    },
+    {
+      "id": "s_1a364ebcb3035539",
+      "rank": 2,
+      "type": "organic",
+      "title": "Go (programming language) - Wikipedia",
+      "url": "https://en.wikipedia.org/wiki/Go_(programming_language)",
+      "display_url": "en.wikipedia.org › wiki › Go_(programming_language)",
+      "snippet": "In Go's package system, each package has a path (e.g., \"compress/bzip2\" or \"golang.org/x/net/html\") and a name (e.g., bzip2 or html). …",
+      "domain": "en.wikipedia.org",
+      "favicon": "https://en.wikipedia.org/favicon.ico",
+      "position": {
+        "absolute": 2
+      },
+      "engine": "bing",
+      "domain_info": {
+        "tld": "org",
+        "sld": "wikipedia",
+        "category": ""
+      },
+      "classification": {
+        "content_type": "article",
+        "source_hint": "encyclopedia"
+      }
+    },
+    ...
+  ],
+  "serp_features": [],
+  "pagination": {
+    "page": 1,
+    "has_more": false,
+    "next_start": 10
+  },
+  "clusters": [
+    {
+      "id": "c_f20b23a020101dce",
+      "canonical_url": "https://go.dev/",
+      "domain": "go.dev",
+      "title": "The Go Programming Language",
+      "occurrences": [
+        {
+          "engine": "bing",
+          "rank": 1,
+          "result_id": "s_5a8273f16b19ab64"
+        }
+      ],
+      "engines_count": 1,
+      "best_rank": 1,
+      "score": 0.5
+    },
+    ...
+  ]
+}
+```

-Once the server is running, the interactive docs are available locally:
-
- Swagger UI: `http://127.0.0.1:7000/docs`
- OpenAPI YAML: `http://127.0.0.1:7000/openapi.yaml`
-
-To browse the spec without running the server, see [docs/openapi.yaml](./docs/openapi.yaml). For a higher-level overview of how OpenSERP works internally, see the [architecture docs](https://openserp.org/docs/architecture/).
+</details>

 ## SDKs & Examples

 Official client packages. Each works against your self-hosted server (set `baseUrl`) or the [hosted API](https://openserp.org/cloud) (set `apiKey`):

-| Type                        | Package                                                                                      | Install                         |
-| --------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------- |
-| JavaScript / TypeScript SDK | [`@openserp/sdk`](https://www.npmjs.com/package/@openserp/sdk)                               | `npm install @openserp/sdk`     |
-| Python SDK                  | [`openserp`](https://pypi.org/project/openserp/)                                             | `pip install openserp`          |
-| MCP server (AI agents)      | [`@openserp/mcp`](https://www.npmjs.com/package/@openserp/mcp)                               | `npx @openserp/mcp`             |
-| n8n community node          | [`@openserp/n8n-nodes-openserp`](https://www.npmjs.com/package/@openserp/n8n-nodes-openserp) | Install via n8n community nodes |
+| Type                        | Package                                                                                      | Source                                                              | Install                         |
+| --------------------------- | -------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | ------------------------------- |
+| JavaScript / TypeScript SDK | [`@openserp/sdk`](https://www.npmjs.com/package/@openserp/sdk)                               | [openserpapi/sdk-js](https://github.com/openserpapi/sdk-js)         | `npm install @openserp/sdk`     |
+| Python SDK                  | [`openserp`](https://pypi.org/project/openserp/)                                             | [openserpapi/sdk-python](https://github.com/openserpapi/sdk-python) | `pip install openserp`          |
+| MCP server (AI agents)      | [`@openserp/mcp`](https://www.npmjs.com/package/@openserp/mcp)                               | [openserpapi/mcp](https://github.com/openserpapi/mcp)               | `npx @openserp/mcp`             |
+| n8n community node          | [`@openserp/n8n-nodes-openserp`](https://www.npmjs.com/package/@openserp/n8n-nodes-openserp) | [openserpapi/n8n](https://github.com/openserpapi/n8n)               | Install via n8n community nodes |

 See [**examples**](./examples) for small JavaScript and Python use cases covering search, AI grounding, SEO, content extraction, and image search.

@@ -109,16 +201,28 @@ curl "http://127.0.0.1:7000/bing/image?text=golang+logo&limit=10"
 Megasearch:

 ```bash
-# Search all configured engines
 curl "http://127.0.0.1:7000/mega/search?text=golang&limit=10"
+```

-# Fast mode: only one fastest engine is queried
+`/mega/search` returns the same envelope as engine endpoints plus `clusters`: results are deduplicated by normalized URL, and clusters keep the per-engine occurrences and ranks.
+
+| Mode       | Best for                             | Behavior                                       |
+| ---------- | ------------------------------------ | ---------------------------------------------- |
+| `balanced` | Most multi-engine SERP workflows     | Queries engines in parallel and merges results |
+| `fast`     | Lowest latency                       | Uses the fastest available engine              |
+| `any`      | Fallback-style availability checking | Tries engines sequentially until one responds  |
+
+<details>
+<summary>More megasearch examples</summary>
+
+```bash
+# Fast mode
 curl "http://127.0.0.1:7000/mega/search?text=golang&mode=fast&engines=google,bing,yandex"

-# Any mode: sequential fallback in provided order (default order if none provided)
+# Any mode
 curl "http://127.0.0.1:7000/mega/search?text=golang&mode=any&engines=google,yandex,bing"

-# Balanced mode (default): parallel all engines with aggregation controls
+# Balanced mode with aggregation controls
 curl "http://127.0.0.1:7000/mega/search?text=golang&mode=balanced&dedupe=true&merge=true"

 # Advanced filtering
@@ -128,6 +232,8 @@ curl "http://127.0.0.1:7000/mega/search?text=golang&engines=google,bing&limit=20
 curl "http://127.0.0.1:7000/mega/image?text=golang+logo&limit=20"
 ```

+</details>
+
 List engines:

 ```bash
@@ -143,28 +249,100 @@ curl "http://127.0.0.1:7000/extract?url=https://example.com&mode=auto"
 # Return clean page markdown
 curl "http://127.0.0.1:7000/extract?url=https://example.com&format=markdown"

+# Extract several URLs at once - returns a bare [{page_content, metadata}] array
+# (Open WebUI external loader compatible); failed URLs become items with metadata.error
+curl -X POST "http://127.0.0.1:7000/extract/batch" \
+  -H "Content-Type: application/json" \
+  -d '{"urls":["https://example.com","https://go.dev"],"mode":"fast"}'
+
 # Embed extracted content under the top search results
-curl "http://127.0.0.1:7000/google/search?text=llm+observability&extract=true&extract_top=2&format=markdown"
+curl "http://127.0.0.1:7000/google/search?text=llm+observability&extract=2&format=markdown"
 ```

-## 🔍 Query Parameters
+## CLI Search
+
+No server required - query an engine straight from the terminal. The CLI shares the same engines, formats, and filters as the API.
+
+```sh
+openserp search ecosia "weather in london" --format markdown
+```
+
+<details>
+<summary>CLI output and more examples</summary>
+
+```markdown
+# Search results for "weather in london"
+
+**Query:** weather in london - **Engines:** ecosia - **Took:** 866ms
+
+## Results
+
+### 1. London - BBC Weather
+
+**bbc.com › weather › 2643743** - organic
+
+Latest forecast for London ... Tonight will continue dry, and there will be mainly clear skies. Just a few patches of cloud drifting in from the north at times.
+
+-> https://www.bbc.com/weather/2643743
+
+### 2. London (Greater London) weather - Met Office
+
+**weather.metoffice.gov.uk › forecast › gcpvj0v07** - organic
+
+Remaining warm with light winds and dry. Possibly cloudy at times Monday and Tuesday, then Wednesday sunnier conditions are likely.
+
+-> https://weather.metoffice.gov.uk/forecast/gcpvj0v07
+
+### 3. London, London, United Kingdom Weather Forecast
+
+**accuweather.com › en › gb › london › ec4a-2 › wea…** - organic
+
+London, London, United Kingdom Weather Forecast, with current conditions, wind, air quality, and what to expect for the next 3 days.
+
+-> https://www.accuweather.com/en/gb/london/ec4a-2/weather-forecast/328328
+```
+
+More CLI examples:
+
+```sh
+# JSON is the default format
+openserp search google "golang generics" --limit 20
+
+# Plain text, German results
+openserp search yandex "wetter berlin" --format text --lang DE --region DE
+
+# Restrict to a site and stream NdJSON
+openserp search bing "release notes" --site github.com --format ndjson
+
+# Embed clean page content from the top 2 results
+openserp search google "llm observability" --extract 2 --format markdown
+
+# Browserless (raw HTTP) mode through a proxy
+# (raw mode: google, yandex, baidu, ecosia)
+openserp search ecosia "weather in london" --raw --proxy http://user:pass@127.0.0.1:8080
+```
+
+</details>
+
+Run `openserp search --help` for the full flag list. Engine names: `google`, `yandex`, `baidu`, `bing`, `duckduckgo`, `ecosia`.
+
+## Query Parameters

 Common parameters:

-| Parameter      | Description                                                                                                                          | Example                              |
-| -------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------ |
-| `text`         | Search query                                                                                                                         | `golang programming`                 |
-| `lang`         | Language code                                                                                                                        | `EN`, `DE`, `RU`, `ES`               |
-| `region`       | Market/location hint. Countries/locales work across engines; Google also accepts city names via `uule`; Yandex accepts numeric `lr`. | `DE`, `en-GB`, `Berlin`, `213`       |
-| `date`         | Date range                                                                                                                           | `20250101..20251231`                 |
-| `file`         | File extension                                                                                                                       | `pdf`, `doc`, `xls`                  |
-| `site`         | Site-specific search                                                                                                                 | `github.com`                         |
-| `limit`        | Number of organic results, max 100. When omitted or `<=10`, only the first SERP page is parsed.                                      | `25`, `50`                           |
-| `start`        | Pagination offset                                                                                                                    | `0`, `10`, `20`                      |
-| `format`       | Output format                                                                                                                        | `json`, `markdown`, `text`, `ndjson` |
-| `extract`      | Fetch and embed target-page content for top web results                                                                              | `true`                               |
-| `extract_top`  | Number of top web results to extract, clamped to 1-5                                                                                 | `3`                                  |
-| `extract_mode` | Extraction strategy: raw HTTP first, raw only, or browser-rendered                                                                   | `auto`, `fast`, `rendered`           |
+| Parameter      | Description                                                                                                                                                                                             | Example                              |
+| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------ |
+| `text`         | Search query                                                                                                                                                                                            | `golang programming`                 |
+| `lang`         | Language code                                                                                                                                                                                           | `EN`, `DE`, `RU`, `ES`               |
+| `region`       | Market/location hint. Countries/locales work across engines; Google also accepts city names via `uule`; Yandex accepts numeric `lr`.                                                                    | `DE`, `en-GB`, `Berlin`, `213`       |
+| `date`         | Date range                                                                                                                                                                                              | `20250101..20251231`                 |
+| `file`         | File extension                                                                                                                                                                                          | `pdf`, `doc`, `xls`                  |
+| `site`         | Site-specific search                                                                                                                                                                                    | `github.com`                         |
+| `limit`        | Number of organic results, max 100. When omitted or `<=10`, only the first SERP page is parsed.                                                                                                         | `25`, `50`                           |
+| `start`        | Pagination offset                                                                                                                                                                                       | `0`, `10`, `20`                      |
+| `format`       | Output format                                                                                                                                                                                           | `json`, `markdown`, `text`, `ndjson` |
+| `extract`      | Fetch and embed target-page content for top web results. Bool or int depth: `0`/`false` off, `true`/`1` top result, `N` top N (1-5). `extract_mode`/`min_runes` imply `extract=true` unless `extract=0` | `1`, `3`, `true`                     |
+| `extract_mode` | Extraction strategy: raw HTTP first, raw only, or browser-rendered                                                                                                                                      | `auto`, `fast`, `rendered`           |

 Engine-specific parameters:

@@ -173,132 +351,7 @@ Engine-specific parameters:
 | `filter`   | `google`          | Duplicate filter: `true` hides similar results, `false` includes them. |
 | `features` | browser `Search`  | Populate `serp_features[]` from the live page. Defaults to `true`.     |

-## Search Response Example
-
-<details>
-<summary>Search response example</summary>
-
-```json
-{
-  "query": {
-    "text": "golang",
-    "engines_requested": ["google"]
-  },
-  "meta": {
-    "request_id": "019dc6c1-da45-706e-a57c-d671fa2862ee",
-    "requested_at": "2026-04-25T22:27:52Z",
-    "took_ms": 6410,
-    "engines_failed": [],
-    "version": "2.1"
-  },
-  "results": [
-    {
-      "id": "s_78341aa47c336101",
-      "rank": 1,
-      "type": "organic",
-      "title": "Documentation - The Go Programming Language",
-      "url": "https://go.dev/doc/",
-      "display_url": "go.dev > doc",
-      "snippet": "Official Go documentation, tutorials, references, and release notes.",
-      "domain": "go.dev",
-      "favicon": "https://go.dev/favicon.ico",
-      "position": {
-        "absolute": 1
-      },
-      "engine": "google",
-      "domain_info": {
-        "tld": "dev",
-        "sld": "go",
-        "category": ""
-      }
-    }
-  ],
-  "pagination": {
-    "page": 1,
-    "has_more": true,
-    "next_start": 25
-  }
-}
-```
-
-</details>
-
-## Mega Response Notes
-
-`/mega/search` returns the same envelope plus `clusters`. Results are deduplicated by normalized URL; clusters keep the per-engine occurrences.
-
-<details>
-<summary>Cluster example</summary>
-
-```json
-{
-  "id": "c_a1b2c3d4e5f6a1b2",
-  "canonical_url": "https://go.dev/",
-  "domain": "go.dev",
-  "title": "The Go Programming Language",
-  "occurrences": [
-    { "engine": "google", "rank": 1, "result_id": "s_78341aa47c336101" },
-    { "engine": "bing", "rank": 2, "result_id": "s_20f9f15f0c3d9f6d" }
-  ],
-  "engines_count": 2,
-  "best_rank": 1,
-  "score": 0.75
-}
-```
-
-</details>
-
-## Image Response Example
-
-<details>
-<summary>Image result example</summary>
-
-```json
-{
-  "id": "i_a1b2c3d4e5f6a1b2",
-  "rank": 1,
-  "type": "image",
-  "title": "Go Gopher Logo",
-  "image": {
-    "url": "https://example.com/images/go-logo.png",
-    "thumbnail": "https://example.com/images/go-logo-thumb.png",
-    "width": 1200,
-    "height": 800
-  },
-  "source": {
-    "page_url": "https://go.dev/brand/",
-    "domain": "go.dev"
-  },
-  "engine": "bing"
-}
-```
-
-</details>
-
-## Error Responses
-
-`400 Bad Request`:
-
-```json
-{
-  "error": "bad_request",
-  "code": 400,
-  "message": "EMPTY_QUERY: query cannot be empty: provide text, site, or file parameter",
-  "reason": "EMPTY_QUERY"
-}
-```
-
-`503 Service Unavailable`:
-
-```json
-{
-  "error": "service_unavailable",
-  "code": 503,
-  "message": "captcha found, please stop sending requests for a while: captcha detected"
-}
-```
-
-## 🌍 Proxy Support
+## Proxy Support

 OpenSERP supports HTTP and SOCKS5 proxies.

@@ -311,18 +364,21 @@ Simple global proxy:

 Advanced proxy configuration is available in [config.yaml](./config.yaml). You can enable tagged proxy pools and per-request override via `X-Use-Proxy: <tag>` or `X-Use-Proxy: direct`.

-A [managed API](https://openserp.org/cloud) is also available for teams that do not want to operate infrastructure.
+## API Docs

-## Health & Stats
+Once the server is running, the interactive docs are available locally:

-```bash
-curl -i "http://127.0.0.1:7000/health"
-curl "http://127.0.0.1:7000/ready"
-curl "http://127.0.0.1:7000/stats"
-curl "http://127.0.0.1:7000/stats/cache"
-curl "http://127.0.0.1:7000/stats/proxy"
-curl "http://127.0.0.1:7000/stats/cb"
-```
+- Swagger UI: `http://127.0.0.1:7000/docs` - full schemas, error shapes, and the `/health`, `/ready`, `/stats/*` endpoints
+- OpenAPI YAML: `http://127.0.0.1:7000/openapi.yaml`
+
+To browse the spec without running the server, see [docs/openapi.yaml](./docs/openapi.yaml). For a higher-level overview of how OpenSERP works internally, see the [architecture docs](https://openserp.org/docs/architecture/).
+
+## Self-Hosted or Cloud
+
+- **Self-hosted (this repo)** - free, MIT-licensed, full control over runtime, proxies, cache, and scaling.
+- **[OpenSERP Cloud](https://openserp.org/cloud)** - same endpoints and response schema, no infrastructure to run.
+
+Client code migrates in either direction, so you are never locked in.

 ## License

@@ -335,7 +391,7 @@ Contributions are welcome. See [docs/CONTRIBUTING.md](./docs/CONTRIBUTING.md).
 ## Feedback & Updates

 - [GitHub Issues](https://github.com/karust/openserp/issues) - bugs, feature ideas, and reproducible issues.
- [Telegram channel](https://t.me/openserp_cloud) - OpenSERP news, release notes, and project updates. Direct messages are open for quick feedback and hosted API questions.
 - [feedback@openserp.org](mailto:feedback@openserp.org) - private notes, longer feedback, or anything that does not fit GitHub Issues.
+- [Telegram](https://t.me/openserp_cloud) - OpenSERP news, release notes, and project updates.

-###### _"OpenSERP" is the name of this open-source project. The official website is [openserp.org](https://openserp.org). Resources not linked on this page are not affiliated with the project._
+> OpenSERP is free and open-source. Only links listed in this repository and on [openserp.org](https://openserp.org) are associated with the project.
--- a/baidu/features.go
+++ b/baidu/features.go
@@ -6,7 +6,7 @@ import (
 )

 func extractBaiduFeatures(doc *goquery.Document) []core.SerpFeature {
-	features := core.ExtractSerpFeaturesBySelectors(doc, []core.SerpFeatureSelector{
+	return core.ExtractSerpFeaturesBySelectors(doc, []core.SerpFeatureSelector{
 		{
 			Type:          core.ResultTypeAISummary,
 			Title:         "AI summary",
@@ -36,5 +36,4 @@ func extractBaiduFeatures(doc *goquery.Document) []core.SerpFeature {
 			Confidence:   0.75,
 		},
 	})
-	return core.DeduplicateSerpFeatures(features)
 }
--- a/baidu/parse_html.go
+++ b/baidu/parse_html.go
@@ -1,6 +1,7 @@
 package baidu

 import (
+	"errors"
 	"io"
 	"strings"

@@ -15,9 +16,28 @@ func ParseHTML(r io.Reader) ([]core.SearchResult, error) {
 	if err != nil {
 		return nil, err
 	}
+	pageStatus := classifyBaiduDocument(doc)
+	if errors.Is(pageStatus, core.ErrEmptyResult) {
+		return []core.SearchResult{}, nil
+	}
+	if pageStatus != nil {
+		return nil, pageStatus
+	}
 	return parseBaiduDocument(doc), nil
 }

+func classifyBaiduDocument(doc *goquery.Document) error {
+	if doc.Find(Selectors.Captcha).Length() > 0 {
+		return core.ErrCaptcha
+	}
+	if doc.Find(Selectors.Timeout).Length() > 0 {
+		return core.ErrSearchTimeout
+	}
+	return core.ClassifyChallengeDocument(doc, core.DocSignals{
+		EmptySelectors: []string{Selectors.NoResults},
+	})
+}
+
 func parseBaiduDocument(doc *goquery.Document) []core.SearchResult {
 	features := extractBaiduFeatures(doc)
 	// Match all result-card variants in one pass so DOM order is preserved and
@@ -41,9 +61,7 @@ func baiduResultSelector() string {

 func parseBaiduSelection(sel *goquery.Selection) []core.SearchResult {
 	var results []core.SearchResult
-	rank := 1
-	adRank := 1
-	absoluteRank := 1
+	rank := core.NewRankState(0)

 	sel.Each(func(_ int, item *goquery.Selection) {
 		isAd := baiduSelectionHasAdMarker(item)
@@ -120,14 +138,7 @@ func parseBaiduSelection(sel *goquery.Selection) []core.SearchResult {
 			desc = strings.TrimSpace(strings.Replace(full, title, "", 1))
 		}

-		resultRank := rank
-		if isAd {
-			resultRank = adRank
-			adRank++
-		} else {
-			rank++
-		}
-
+		resultRank, absoluteRank := rank.Next(isAd)
 		results = append(results, core.SearchResult{
 			Rank:         resultRank,
 			AbsoluteRank: absoluteRank,
@@ -136,7 +147,6 @@ func parseBaiduSelection(sel *goquery.Selection) []core.SearchResult {
 			Description:  desc,
 			Ad:           isAd,
 		})
-		absoluteRank++
 	})

 	// Re-rank sequentially after dedup so callers get a clean 1..N sequence
--- a/baidu/search.go
+++ b/baidu/search.go
@@ -3,6 +3,7 @@ package baidu
 import (
 	"context"
 	"encoding/json"
+	"errors"
 	"fmt"
 	"regexp"
 	"strings"
@@ -53,32 +54,25 @@ func (baid *Baidu) Name() string {
 	return "baidu"
 }

-func (baid *Baidu) isCaptcha(page *rod.Page) bool {
-	has, _, _ := page.Has(Selectors.Captcha)
-	return has
-}
-
-func (baid *Baidu) isTimeout(page *rod.Page) bool {
-	has, _, _ := page.Has(Selectors.Timeout)
-	return has
-}
-
+// classifyBlockPage runs the same captcha/timeout/no-results rules the raw
+// HTML path uses (classifyBaiduDocument), against a snapshot of the live
+// page, so both paths can't drift. ErrEmptyResult is not a block: callers
+// handle that themselves once they've also checked for parsed results.
 func (baid *Baidu) classifyBlockPage(page *rod.Page, url string) error {
-	if baid.isCaptcha(page) {
-		baid.logger.Error("Captcha detected: %s", url)
-		return core.ErrCaptcha
+	err := core.ClassifyFromPage(page, classifyBaiduDocument)
+	if errors.Is(err, core.ErrEmptyResult) {
+		return nil
 	}
-	if baid.isTimeout(page) {
-		baid.logger.Error("Timeout occurred: %s", url)
-		return core.ErrSearchTimeout
+	if err != nil {
+		baid.logger.Error("Page classified as %v: %s", err, url)
 	}
-	return nil
+	return err
 }

 // Search executes a Baidu web search and returns normalized search results.
 // It may return core.ErrCaptcha or core.ErrSearchTimeout.
 func (baid *Baidu) Search(ctx context.Context, query core.Query) (results []core.SearchResult, err error) {
-	ctx = core.PrepareEngineContext(ctx, query, baid.Name(), true)
+	ctx = core.PrepareEngineContext(ctx, query, baid.Name())
 	scoped := *baid
 	scoped.logger = baid.logger.WithRequest(ctx)
 	if scoped.Browser.WaitLoadTime == 0 || scoped.Browser.WaitLoadTime > 250*time.Millisecond {
@@ -114,6 +108,7 @@ func (baid *Baidu) Search(ctx context.Context, query core.Query) (results []core
 		}
 		searchResults[i].Rank = query.Start + searchResults[i].Rank
 	}
+	searchResults = core.StripResultFeatures(searchResults, query.Features)
 	return searchResults, nil
 }

@@ -133,17 +128,21 @@ func (baid *Baidu) waitForParsedSearchResults(ctx context.Context, page *rod.Pag
 			if parseErr == nil && len(results) > 0 {
 				return results, nil
 			}
+			if parseErr != nil && !errors.Is(parseErr, core.ErrEmptyResult) {
+				baid.logger.Error("Page classified as %v: %s", parseErr, url)
+				return nil, parseErr
+			}
 			lastErr = parseErr
 		} else {
 			lastErr = err
+			if blockErr := baid.classifyBlockPage(page, url); blockErr != nil {
+				return nil, blockErr
+			}
 		}

 		if core.HasAnySelector(page, baiduResultSelectors()) {
 			sawResultContainer = true
 		}
-		if blockErr := baid.classifyBlockPage(page, url); blockErr != nil {
-			return nil, blockErr
-		}
 		if !time.Now().Before(deadline) {
 			break
 		}
@@ -171,7 +170,7 @@ func (baid *Baidu) waitForParsedSearchResults(ctx context.Context, page *rod.Pag
 // SearchImage executes a Baidu image search and returns normalized image
 // results. It may return core.ErrCaptcha or core.ErrSearchTimeout.
 func (baid *Baidu) SearchImage(ctx context.Context, query core.Query) ([]core.SearchResult, error) {
-	ctx = core.PrepareEngineContext(ctx, query, baid.Name(), true)
+	ctx = core.PrepareEngineContext(ctx, query, baid.Name())
 	scoped := *baid
 	scoped.logger = baid.logger.WithRequest(ctx)
 	baid = &scoped
--- a/baidu/search_raw.go
+++ b/baidu/search_raw.go
@@ -15,17 +15,11 @@ func classifyBaiduRawHTML(body []byte) error {
 	if err != nil {
 		return err
 	}
-	if doc.Find(Selectors.Captcha).Length() > 0 || doc.Find(Selectors.Timeout).Length() > 0 {
-		return core.ErrCaptcha
-	}
-	if doc.Find("div.content_none, div.nors").Length() > 0 {
-		return core.ErrEmptyResult
-	}
-	return nil
+	return classifyBaiduDocument(doc)
 }

 func Search(ctx context.Context, query core.Query) (results []core.SearchResult, err error) {
-	ctx = core.PrepareEngineContext(ctx, query, "baidu", false)
+	ctx = core.PrepareEngineContext(ctx, query, "baidu")

 	searchURL, err := BuildURL(query)
 	if err != nil {
@@ -76,5 +70,6 @@ func Search(ctx context.Context, query core.Query) (results []core.SearchResult,
 		fmt.Sprintf("Baidu Raw results : %v", parsedResults),
 	)

-	return core.DeduplicateResults(parsedResults), nil
+	deduped := core.StripResultFeatures(core.DeduplicateResults(parsedResults), query.Features)
+	return deduped, nil
 }
--- a/baidu/search_raw_test.go
+++ b/baidu/search_raw_test.go
@@ -17,12 +17,12 @@ func TestBaiduParseHTMLFixtures(t *testing.T) {
 	t.Parallel()

 	tests := []struct {
-		name     string
-		fixture  string
-		wantZero bool
+		name    string
+		fixture string
+		wantErr error
 	}{
-		{name: "no results", fixture: "search_no_results.html", wantZero: true},
-		{name: "captcha page", fixture: "search_captcha.html", wantZero: true},
+		{name: "no results", fixture: "search_no_results.html"},
+		{name: "captcha page", fixture: "search_captcha.html", wantErr: core.ErrCaptcha},
 	}

 	for _, tt := range tests {
@@ -30,10 +30,16 @@ func TestBaiduParseHTMLFixtures(t *testing.T) {
 			t.Parallel()

 			results, err := ParseHTML(testutil.ResponseFromFixture(t, tt.fixture).Body)
+			if tt.wantErr != nil {
+				if !errors.Is(err, tt.wantErr) {
+					t.Fatalf("expected %v for %s, got %v", tt.wantErr, tt.fixture, err)
+				}
+				return
+			}
 			if err != nil {
 				t.Fatalf("ParseHTML() error = %v", err)
 			}
-			if tt.wantZero && len(results) != 0 {
+			if len(results) != 0 {
 				t.Fatalf("expected zero results for %s, got %d", tt.fixture, len(results))
 			}
 		})
--- a/baidu/selectors.go
+++ b/baidu/selectors.go
@@ -4,6 +4,7 @@ package baidu
 var Selectors = struct {
 	Captcha       string
 	Timeout       string
+	NoResults     string
 	Results       string
 	ResultsAlt    []string
 	AdMarkers     []string
@@ -16,6 +17,7 @@ var Selectors = struct {
 }{
 	Captcha:       "div.passMod_dialog-wrapper",
 	Timeout:       "button.timeout-button",
+	NoResults:     "div.content_none, div.nors",
 	Results:       "#content_left div.result.c-container",
 	ResultsAlt:    []string{"#content_left div.result-op.c-container", "div.c-container.new-pmd"},
 	AdMarkers:     []string{"[data-tuiguang]", "[data-click*='tuiguang']", ".ec-tuiguang", ".c-icon-bear-p"},
--- a/bing/bing_test.go
+++ b/bing/bing_test.go
@@ -33,9 +33,6 @@ func TestBuildURL(t *testing.T) {
 				if got := params.Get("q"); got != "golang тест site:example.com filetype:pdf" {
 					t.Fatalf("unexpected q value: %q", got)
 				}
-				if got := params.Get("pq"); got != params.Get("q") {
-					t.Fatalf("pq should match q, got %q vs %q", got, params.Get("q"))
-				}
 				if got := params.Get("setlang"); got != "ru" {
 					t.Fatalf("unexpected setlang value: %q", got)
 				}
@@ -54,14 +51,10 @@ func TestBuildURL(t *testing.T) {
 				if got := params.Get("first"); got != "" {
 					t.Fatalf("first should be omitted when Start=0, got %q", got)
 				}
-				if got := params.Get("form"); got != "QBLH" {
-					t.Fatalf("unexpected form value: %q", got)
-				}
-				if got := params.Get("qs"); got != "HS" {
-					t.Fatalf("unexpected qs value: %q", got)
-				}
-				if got := params.Get("sp"); got != "-1" {
-					t.Fatalf("unexpected sp value: %q", got)
+				for _, key := range []string{"form", "qs", "sp", "pq"} {
+					if got := params.Get(key); got != "" {
+						t.Fatalf("expected %s to be omitted, got %q", key, got)
+					}
 				}
 			},
 		},
@@ -76,9 +69,6 @@ func TestBuildURL(t *testing.T) {
 				if got := params.Get("q"); got != "megadeth tickets" {
 					t.Fatalf("unexpected q value: %q", got)
 				}
-				if got := params.Get("pq"); got != "megadeth tickets" {
-					t.Fatalf("unexpected pq value: %q", got)
-				}
 				if got := params.Get("filters"); got != `ex1:"ez5_20454_20570"` {
 					t.Fatalf("unexpected filters value: %q", got)
 				}
--- a/bing/features.go
+++ b/bing/features.go
@@ -1,19 +1,18 @@
 package bing

 import (
+	"context"
+
 	"github.com/PuerkitoBio/goquery"
 	"github.com/go-rod/rod"
 	"github.com/karust/openserp/core"
 )

 func extractBingFeatures(doc *goquery.Document) []core.SerpFeature {
-	features := core.ExtractSerpFeaturesBySelectors(doc, []core.SerpFeatureSelector{
+	return core.ExtractSerpFeaturesBySelectors(doc, []core.SerpFeatureSelector{
 		{
 			Type: core.ResultTypeAnswerBox,
-			// Only treat a b_ans block as an answer box when it carries an
-			// actual answer/entity payload. A bare li.b_ans also wraps related
-			// modules ("Searches you might like", "Get a detailed look at ..."),
-			// so require a focus/fact/xl text node to be present.
+			// li.b_ans also wraps related modules; require answer payload.
 			Container:     []string{"li.b_ans:has(.b_focusTextLarge)", "li.b_ans:has(.b_focusLabel)", "li.b_ans:has(.b_xlText)", "li.b_ans:has(.b_factrow)"},
 			TitleSelector: []string{".b_focusLabel", "h2"},
 			TextSelector:  []string{".b_focusTextLarge", ".b_xlText", ".b_vPanel .b_factrow", ".b_caption p"},
@@ -32,33 +31,25 @@ func extractBingFeatures(doc *goquery.Document) []core.SerpFeature {
 		{
 			Type:         core.ResultTypeRelatedSearches,
 			Title:        "Related searches",
-			Container:    []string{"#brsv3", "li.b_rs", "ol#b_rs"},
-			ItemSelector: []string{"li a", "a"},
+			Container:    []string{"#brsv3", "#rs_root", "#inline_rs", "#brs", "#b_rs", "ol#b_rs", "li.b_rs"},
+			ItemSelector: []string{"li.rslist a", "li a", "a"},
 			LinkSelector: []string{"a[href^='http']", "a"},
 			Confidence:   0.75,
+			SingleMatch:  true,
 		},
 		{
-			// Bing's "developer answer" / rich answer card is AI-generated
-			// ("This summary was generated using AI based on multiple online
-			// sources"). Title sits in h2.b_topTitle; cited sources are the
-			// numbered superscript anchors. Copilot chat (#b_sydConvCont) is kept
-			// as a fallback for SERPs that render the chat answer inline instead.
 			Type:          core.ResultTypeAISummary,
 			Title:         "AI answer",
-			Container:     []string{".developer_answercard_wrapper", "#b_sydConvCont", ".b_sydConvCont", "[data-testid='bing-chat-answer']"},
+			Container:     []string{".developer_answercard_wrapper", "#ca_main", ".ca_container", "#b_sydConvCont", ".b_sydConvCont", "[data-testid='bing-chat-answer']"},
 			TitleSelector: []string{"h2.b_topTitle", ".b_sydAns"},
-			// The full generated answer lives in .devmag_card_content, split across
-			// many <p>/<li> inside span.devmag_cntnt_snip; take the wrapper's whole
-			// collapsed text so the body isn't truncated to the first paragraph.
-			TextSelector: []string{".devmag_card_content", ".rd_def_list", ".b_sydAns", "[data-testid='answer']", "p"},
-			LinkSelector: []string{".rd_cnt_srcs a[href^='http']", ".rd_gencon_attr a[href^='http']", "h2.b_topTitle a[href^='http']", "a[href^='http']"},
-			Position:     1,
-			Confidence:   0.6,
+			TextSelector:  []string{".devmag_card_content", ".rd_def_list", ".b_sydAns", "[data-testid='answer']", ".ca_div", "p"},
+			LinkSelector:  []string{".rd_cnt_srcs a[href^='http']", ".rd_gencon_attr a[href^='http']", "h2.b_topTitle a[href^='http']", "a[href^='http']"},
+			Position:      1,
+			Confidence:    0.6,
 		},
 	})
-	return core.DeduplicateSerpFeatures(features)
 }

-func extractBingFeaturesFromPage(page *rod.Page) []core.SerpFeature {
-	return core.FeaturesFromPage(page, extractBingFeatures)
+func extractBingFeaturesFromPage(ctx context.Context, page *rod.Page) []core.SerpFeature {
+	return core.FeaturesFromPageWithWait(ctx, page, extractBingFeatures)
 }
--- a/bing/parse_html.go
+++ b/bing/parse_html.go
@@ -1,6 +1,7 @@
 package bing

 import (
+	"errors"
 	"io"
 	"strings"

@@ -16,14 +17,27 @@ func ParseHTML(r io.Reader) ([]core.SearchResult, error) {
 	if err != nil {
 		return nil, err
 	}
+	pageStatus := classifyBingDocument(doc)
+	if errors.Is(pageStatus, core.ErrEmptyResult) {
+		return []core.SearchResult{}, nil
+	}
+	if pageStatus != nil {
+		return nil, pageStatus
+	}
 	return parseBingDocument(doc), nil
 }

+func classifyBingDocument(doc *goquery.Document) error {
+	return core.ClassifyChallengeDocument(doc, core.DocSignals{
+		CaptchaSelectors: Selectors.Captcha,
+		CaptchaMarkers:   Selectors.CaptchaMarkers,
+		EmptyMarkers:     Selectors.NoResultsMarkers,
+	})
+}
+
 func parseBingDocument(doc *goquery.Document) []core.SearchResult {
 	var results []core.SearchResult
-	rank := 1
-	adRank := 1
-	absoluteRank := 1
+	rank := core.NewRankState(0)

 	doc.Find(Selectors.ResultItems).Each(func(_ int, item *goquery.Selection) {
 		isAd := item.Is(Selectors.Ads)
@@ -41,85 +55,89 @@ func parseBingDocument(doc *goquery.Document) []core.SearchResult {
 		if titleTag.Length() == 0 {
 			return
 		}
+		href, _ := titleTag.Attr("href")
+		title := bingDocumentTitle(item, titleTag)
+		desc := bingDocumentDescription(item, title)

-		link, exists := titleTag.Attr("href")
-		if !exists || link == "" || link == "#" {
-			return
+		if res, ok := assembleBingRow(href, title, desc, isAd, rank); ok {
+			results = append(results, res)
 		}
-
-		title := titleTag.Text()
-		if title == "" {
-			title = extractFirstText(item, Selectors.TitleFallbacks)
-		}
-		if title == "" {
-			return
-		}
-
-		desc := descriptionFromItem(item, title)
-
-		resultRank := rank
-		if isAd {
-			resultRank = adRank
-			adRank++
-		} else {
-			rank++
-		}
-
-		results = append(results, core.SearchResult{
-			Rank:         resultRank,
-			AbsoluteRank: absoluteRank,
-			URL:          link,
-			Title:        title,
-			Description:  desc,
-			Ad:           isAd,
-		})
-		absoluteRank++
 	})

 	return core.AttachFeaturesToFirstResult(core.DeduplicateResults(results), extractBingFeatures(doc))
 }

-func extractFirstText(item *goquery.Selection, selectors []string) string {
-	for _, selector := range selectors {
-		if tag := item.Find(selector).First(); tag.Length() > 0 {
-			if text := strings.TrimSpace(tag.Text()); text != "" {
-				return text
-			}
-			if label, exists := tag.Attr("aria-label"); exists {
-				if label = strings.TrimSpace(label); label != "" {
-					return label
-				}
-			}
+// assembleBingRow validates an already-extracted Bing row and assigns ranks.
+// Shared by the rod (browser) and goquery (raw / parse) parsers, which differ
+// only in how they pull title/href/desc out of the DOM.
+func assembleBingRow(href, title, desc string, isAd bool, rank *core.RankState) (core.SearchResult, bool) {
+	url := strings.TrimSpace(href)
+	if url == "" || url == "#" || strings.HasPrefix(url, "javascript:") {
+		return core.SearchResult{}, false
+	}
+	if title == "" {
+		return core.SearchResult{}, false
+	}
+
+	resultRank, absoluteRank := rank.Next(isAd)
+	return core.SearchResult{
+		Rank:         resultRank,
+		AbsoluteRank: absoluteRank,
+		URL:          url,
+		Title:        title,
+		Description:  desc,
+		Ad:           isAd,
+	}, true
+}
+
+// bingDocumentTitle reproduces the rod path's title fallback for goquery: the
+// title anchor's aria-label/title attribute, then its text, then any fallback
+// selector's text or aria-label.
+func bingDocumentTitle(item, titleTag *goquery.Selection) string {
+	if title := firstNonEmptyAttr(titleTag, "aria-label", "title"); title != "" {
+		return title
+	}
+	if title := core.NormalizeWhitespace(titleTag.Text()); title != "" {
+		return title
+	}
+	for _, selector := range Selectors.TitleFallbacks {
+		tag := item.Find(selector).First()
+		if tag.Length() == 0 {
+			continue
+		}
+		if text := core.NormalizeWhitespace(tag.Text()); text != "" {
+			return text
+		}
+		if label := firstNonEmptyAttr(tag, "aria-label", "title"); label != "" {
+			return label
 		}
 	}
 	return ""
 }

-// descriptionFromItem extracts a description using the same 4-step fallback
-// chain as the rod-based browser parser. Bing renders snippet text with heavy
-// source-indentation whitespace, so each candidate is whitespace-collapsed.
-func descriptionFromItem(item *goquery.Selection, title string) string {
-	if descTag := item.Find(Selectors.DescPrimary).First(); descTag.Length() > 0 {
-		if text := normalizeWhitespace(descTag.Text()); text != "" {
-			return text
+// bingDocumentDescription reproduces the rod path's 3-selector description
+// fallback plus the strip-title structural fallback. Bing renders snippet text
+// with heavy source-indentation whitespace, so each candidate is collapsed.
+func bingDocumentDescription(item *goquery.Selection, title string) string {
+	for _, selector := range []string{Selectors.DescPrimary, Selectors.DescFallback, Selectors.DescAny} {
+		if tag := item.Find(selector).First(); tag.Length() > 0 {
+			if text := core.NormalizeWhitespace(tag.Text()); text != "" {
+				return text
+			}
 		}
 	}
-	if descTag := item.Find(Selectors.DescFallback).First(); descTag.Length() > 0 {
-		if text := normalizeWhitespace(descTag.Text()); text != "" {
-			return text
-		}
-	}
-	if descTag := item.Find(Selectors.DescAny).First(); descTag.Length() > 0 {
-		if text := normalizeWhitespace(descTag.Text()); text != "" {
-			return text
-		}
-	}
-	// Structural fallback: strip title from full text
-	return normalizeWhitespace(strings.Replace(item.Text(), title, "", 1))
+	return core.NormalizeWhitespace(strings.Replace(item.Text(), title, "", 1))
 }

-// normalizeWhitespace collapses runs of whitespace (including the newlines and
-// indentation Bing leaves in snippet markup) into single spaces.
-func normalizeWhitespace(s string) string {
-	return strings.Join(strings.Fields(s), " ")
+func firstNonEmptyAttr(item *goquery.Selection, attrs ...string) string {
+	for _, attr := range attrs {
+		value, exists := item.Attr(attr)
+		if !exists {
+			continue
+		}
+		if value = core.NormalizeWhitespace(value); value != "" {
+			return value
+		}
+	}
+	return ""
 }
--- a/bing/parse_html_test.go
+++ b/bing/parse_html_test.go
@@ -2,9 +2,12 @@ package bing

 import (
 	"bytes"
+	"errors"
 	"os"
 	"strings"
 	"testing"
+
+	"github.com/karust/openserp/core"
 )

 func TestParseBingHTML(t *testing.T) {
@@ -56,6 +59,29 @@ func TestParseBingHTMLEmpty(t *testing.T) {
 	}
 }

+func TestParseBingHTMLCaptcha(t *testing.T) {
+	t.Parallel()
+
+	html := `<html><body><div class="captcha">Enter the characters you see</div></body></html>`
+	results, err := ParseHTML(bytes.NewReader([]byte(html)))
+	if !errors.Is(err, core.ErrCaptcha) {
+		t.Fatalf("expected ErrCaptcha, got results=%d err=%v", len(results), err)
+	}
+}
+
+func TestParseBingHTMLNoResults(t *testing.T) {
+	t.Parallel()
+
+	html := `<html><body><main>There are no results for this search.</main></body></html>`
+	results, err := ParseHTML(bytes.NewReader([]byte(html)))
+	if err != nil {
+		t.Fatalf("ParseHTML() error = %v", err)
+	}
+	if len(results) != 0 {
+		t.Fatalf("expected zero results, got %d", len(results))
+	}
+}
+
 func TestParseBingHTMLAds(t *testing.T) {
 	t.Parallel()

@@ -145,3 +171,49 @@ func TestParseBingHTMLTitleFallback(t *testing.T) {
 		t.Fatalf("title = %q, want fallback", results[0].Title)
 	}
 }
+
+func TestParseBingHTMLPrefersTitleAttribute(t *testing.T) {
+	t.Parallel()
+
+	html := `
+<ol id="b_results">
+  <li class="b_algo">
+    <h2><a aria-label="Real SERP Title" href="https://example.com/result">example.com</a></h2>
+    <div class="b_caption"><p>Snippet</p></div>
+  </li>
+</ol>`
+
+	results, err := ParseHTML(bytes.NewReader([]byte(html)))
+	if err != nil {
+		t.Fatalf("ParseHTML() error = %v", err)
+	}
+	if len(results) != 1 {
+		t.Fatalf("expected 1 result, got %d", len(results))
+	}
+	if results[0].Title != "Real SERP Title" {
+		t.Fatalf("title = %q, want attribute title", results[0].Title)
+	}
+}
+
+func TestParseBingHTMLDescriptionFallsThroughEmptyPrimary(t *testing.T) {
+	t.Parallel()
+
+	html := `
+<ol id="b_results">
+  <li class="b_algo">
+    <h2><a href="https://example.com/result">Result title</a></h2>
+    <div class="b_caption"><p>   </p><div>Useful snippet text</div></div>
+  </li>
+</ol>`
+
+	results, err := ParseHTML(bytes.NewReader([]byte(html)))
+	if err != nil {
+		t.Fatalf("ParseHTML() error = %v", err)
+	}
+	if len(results) != 1 {
+		t.Fatalf("expected 1 result, got %d", len(results))
+	}
+	if results[0].Description != "Useful snippet text" {
+		t.Fatalf("description = %q, want fallback snippet", results[0].Description)
+	}
+}
--- a/bing/search.go
+++ b/bing/search.go
@@ -42,6 +42,8 @@ func (bing *Bing) getTotalResults(page *rod.Page) (int, error) {
 	return len(results), nil
 }

+// checkCaptcha classifies the live page the same way classifyBingDocument
+// (parse_html.go) classifies raw HTML, so /bing/search and /bing/parse agree.
 func (bing *Bing) checkCaptcha(page *rod.Page) bool {
 	if page == nil {
 		return false
@@ -62,10 +64,38 @@ func (bing *Bing) checkCaptcha(page *rod.Page) bool {
 		}
 	}

+	return pageTextContainsAny(page, Selectors.CaptchaMarkers)
+}
+
+// checkNoResults reports whether the page text matches Bing's no-results
+// phrasing, mirroring classifyBingDocument's text-marker check.
+func (bing *Bing) checkNoResults(page *rod.Page) bool {
+	return pageTextContainsAny(page, Selectors.NoResultsMarkers)
+}
+
+func pageTextContainsAny(page *rod.Page, markers []string) bool {
+	if page == nil || len(markers) == 0 {
+		return false
+	}
+	html, err := page.HTML()
+	if err != nil {
+		return false
+	}
+	text := strings.ToLower(html)
+	for _, marker := range markers {
+		if strings.Contains(text, marker) {
+			return true
+		}
+	}
 	return false
 }

 func (bing *Bing) acceptCookies(ctx context.Context, page *rod.Page) error {
+	// Probe first so a banner-less SERP returns immediately instead of blocking
+	// .Element for the full Timeout/10.
+	if has, _, err := page.Has(Selectors.CookieBtn); err != nil || !has {
+		return nil
+	}
 	consentBtn, err := page.Timeout(bing.Timeout / 10).Element(Selectors.CookieBtn)
 	if err != nil {
 		return nil
@@ -85,7 +115,7 @@ func bingElementMatches(el *rod.Element, selector string) bool {
 	return err == nil && matches
 }

-func (bing *Bing) parseResultElement(el *rod.Element, isAd bool, rank, absoluteRank int) (core.SearchResult, bool) {
+func (bing *Bing) parseResultElement(el *rod.Element, isAd bool, rank *core.RankState) (core.SearchResult, bool) {
 	titleSelector := Selectors.Title
 	if isAd {
 		titleSelector = Selectors.AdTitle
@@ -96,56 +126,36 @@ func (bing *Bing) parseResultElement(el *rod.Element, isAd bool, rank, absoluteR
 		bing.logger.Debug("Missing title")
 		return core.SearchResult{}, false
 	}
-
 	href, err := titleElem.Property("href")
 	if err != nil {
 		bing.logger.Debug("Missing URL")
 		return core.SearchResult{}, false
 	}
-	url := strings.TrimSpace(href.String())
-	if url == "" || url == "#" || strings.HasPrefix(url, "javascript:") {
-		return core.SearchResult{}, false
-	}

-	title, _ := titleElem.Text()
-	title = strings.TrimSpace(title)
+	title := core.ElementAttribute(titleElem, "aria-label", "title")
+	if title == "" {
+		title = core.ElementText(titleElem)
+	}
 	if title == "" {
 		title = core.FirstNonEmptyText(el, Selectors.TitleFallbacks...)
 	}
 	if title == "" {
 		title = core.FirstNonEmptyAttribute(el, "aria-label", Selectors.TitleFallbacks...)
 	}
-	if title == "" {
-		bing.logger.Debug("Missing title text")
-		return core.SearchResult{}, false
-	}

-	desc := ""
-	if descElem, err := el.Element(Selectors.DescPrimary); err == nil {
-		desc, _ = descElem.Text()
-	} else if descElem, err := el.Element(Selectors.DescFallback); err == nil {
-		desc, _ = descElem.Text()
-	} else if descElem, err := el.Element(Selectors.DescAny); err == nil {
-		desc, _ = descElem.Text()
-	} else {
+	desc := core.FirstNonEmptyText(el, Selectors.DescPrimary, Selectors.DescFallback, Selectors.DescAny)
+	if desc == "" {
 		fullText, _ := el.Text()
-		desc = strings.TrimSpace(strings.Replace(fullText, title, "", 1))
+		desc = core.NormalizeWhitespace(strings.Replace(fullText, title, "", 1))
 	}

-	return core.SearchResult{
-		Rank:         rank,
-		AbsoluteRank: absoluteRank,
-		URL:          url,
-		Title:        title,
-		Description:  strings.TrimSpace(desc),
-		Ad:           isAd,
-	}, true
+	return assembleBingRow(href.String(), title, desc, isAd, rank)
 }

 // Search executes a Bing web search and returns normalized search results.
 // It may return core.ErrCaptcha or core.ErrSearchTimeout.
 func (bing *Bing) Search(ctx context.Context, query core.Query) (results []core.SearchResult, err error) {
-	ctx = core.PrepareEngineContext(ctx, query, bing.Name(), false)
+	ctx = core.PrepareEngineContext(ctx, query, bing.Name())
 	scoped := *bing
 	scoped.logger = bing.logger.WithRequest(ctx)
 	bing = &scoped
@@ -176,11 +186,15 @@ func (bing *Bing) Search(ctx context.Context, query core.Query) (results []core.

 	resultElements, _, err := core.WaitForElements(ctx, page, []string{Selectors.ResultItems, Selectors.Results}, bing.GetSelectorTimeout())
 	if err != nil {
-		// Re-check captcha on timeout - Bing interstitials can render after WaitLoad.
+		// Re-check captcha/no-results on timeout - Bing interstitials and
+		// no-results pages can both render after WaitLoad.
 		if bing.checkCaptcha(page) {
 			bing.logger.Error("Captcha detected: %s", url)
 			return nil, core.ErrCaptcha
 		}
+		if bing.checkNoResults(page) {
+			return []core.SearchResult{}, nil
+		}
 		bing.logger.Error("Cannot parse organic results: %s", err)
 		return nil, core.ErrSearchTimeout
 	}
@@ -191,9 +205,7 @@ func (bing *Bing) Search(ctx context.Context, query core.Query) (results []core.
 	}
 	bing.logger.Info("Found %d organic result containers", totalResults)

-	rank := query.Start
-	adRank := 1
-	absoluteRank := query.Start + 1
+	rank := core.NewRankStateAt(query.Start, query.Start+1)
 	for _, result := range resultElements {
 		isAd := bingElementMatches(result, Selectors.Ads)
 		isOrganic := bingElementMatches(result, Selectors.Results)
@@ -201,21 +213,11 @@ func (bing *Bing) Search(ctx context.Context, query core.Query) (results []core.
 			continue
 		}

-		resultRank := rank + 1
-		if isAd {
-			resultRank = adRank
-		}
-		srchRes, ok := bing.parseResultElement(result, isAd, resultRank, absoluteRank)
+		srchRes, ok := bing.parseResultElement(result, isAd, rank)
 		if !ok {
 			continue
 		}
 		searchResults = append(searchResults, srchRes)
-		absoluteRank++
-		if isAd {
-			adRank++
-		} else {
-			rank++
-		}
 	}

 	// Deduplicate results
@@ -224,7 +226,7 @@ func (bing *Bing) Search(ctx context.Context, query core.Query) (results []core.
 	deduped = core.LimitOrganicResults(deduped, query.Limit)

 	if query.Features {
-		deduped = core.AttachFeaturesToFirstResult(deduped, extractBingFeaturesFromPage(page))
+		deduped = core.AttachFeaturesToFirstResult(deduped, extractBingFeaturesFromPage(ctx, page))
 	}
 	return deduped, nil
 }
@@ -254,7 +256,7 @@ func resolveImageLinkElement(container *rod.Element) (*rod.Element, error) {
 // SearchImage executes a Bing image search and returns normalized image
 // results. It may return core.ErrCaptcha or core.ErrSearchTimeout.
 func (bing *Bing) SearchImage(ctx context.Context, query core.Query) ([]core.SearchResult, error) {
-	ctx = core.PrepareEngineContext(ctx, query, bing.Name(), false)
+	ctx = core.PrepareEngineContext(ctx, query, bing.Name())
 	scoped := *bing
 	scoped.logger = bing.logger.WithRequest(ctx)
 	bing = &scoped
--- a/bing/selectors.go
+++ b/bing/selectors.go
@@ -2,21 +2,33 @@ package bing

 // Selectors is the single source of truth for Bing SERP CSS selectors.
 var Selectors = struct {
-	Captcha        []string
-	CookieBtn      string
-	ResultItems    string
-	Results        string
-	Ads            string
-	ImageResults   string
-	Title          string
-	TitleFallbacks []string
-	DescPrimary    string
-	DescFallback   string
-	DescAny        string
-	AdTitle        string
+	Captcha          []string
+	CaptchaMarkers   []string
+	NoResultsMarkers []string
+	CookieBtn        string
+	ResultItems      string
+	Results          string
+	Ads              string
+	ImageResults     string
+	Title            string
+	TitleFallbacks   []string
+	DescPrimary      string
+	DescFallback     string
+	DescAny          string
+	AdTitle          string
 }{
-	Captcha:      []string{"div.captcha", "div.captcha_header"},
-	CookieBtn:    "button#bnp_btn_accept",
+	Captcha: []string{"div.captcha", "div.captcha_header"},
+	// CaptchaMarkers/NoResultsMarkers are checked against lowercased page text
+	// as a fallback when the CSS selectors above don't match a challenge page.
+	CaptchaMarkers: []string{
+		"verify that you are not a robot",
+		"enter the characters you see",
+	},
+	NoResultsMarkers: []string{
+		"there are no results for",
+		"no results found for",
+	},
+	CookieBtn: "button#bnp_btn_accept",
 	// ResultItems matches the main-column children only, so carousels and
 	// "related searches" cards that reuse b_algo-style markup are excluded.
 	ResultItems:  "#b_results > li.b_algo, #b_results > li.b_ad",
--- a/bing/serp_features_test.go
+++ b/bing/serp_features_test.go
@@ -9,10 +9,7 @@ import (
 	"github.com/karust/openserp/core"
 )

-// TestParseHTMLFixtureExtractsRealFeatures guards selectors against the
-// sanitized real-SERP fixture (related searches present; the noisy b_ans
-// "detailed look"/"searches you might like" modules must not be emitted as
-// answer boxes).
+// TestParseHTMLFixtureExtractsRealFeatures guards the live SERP fixture.
 func TestParseHTMLFixtureExtractsRealFeatures(t *testing.T) {
 	t.Parallel()
 	f, err := os.Open("testdata/search_results.html")
@@ -25,19 +22,22 @@ func TestParseHTMLFixtureExtractsRealFeatures(t *testing.T) {
 	if err != nil {
 		t.Fatalf("ParseHTML() error = %v", err)
 	}
-	assertFeatureType(t, results, core.ResultTypeRelatedSearches)
-	// The fixture carries a developer/rich answer card flagged as AI-generated.
-	assertFeatureType(t, results, core.ResultTypeAISummary)
+
+	related := findFeature(results, core.ResultTypeRelatedSearches)
+	if related == nil {
+		t.Fatalf("expected related_searches feature in fixture")
+	}
+	if len(related.Items) == 0 {
+		t.Fatalf("related_searches feature has no items: %#v", related)
+	}
 	for _, r := range results {
 		for _, feature := range r.Features {
-			if feature.Type == core.ResultTypeAnswerBox && feature.Title == "Get a detailed look atpizza delivery" {
+			if feature.Type == core.ResultTypeAnswerBox && strings.Contains(feature.Title, "searches you might like") {
 				t.Fatalf("related-search module leaked into answer_box: %#v", feature)
 			}
 		}
 	}

-	// Descriptions must be whitespace-collapsed: Bing leaves raw newlines and
-	// source indentation in snippet markup, which previously surfaced verbatim.
 	for i, r := range results {
 		if strings.ContainsAny(r.Description, "\n\t") {
 			t.Fatalf("result %d description has raw whitespace: %q", i, r.Description)
@@ -76,6 +76,72 @@ func TestParseHTMLExtractsSerpFeatures(t *testing.T) {
 	assertFeatureType(t, results, core.ResultTypeRelatedQuestions)
 }

+// TestParseHTMLExtractsRelatedSearchesFromBrsContainer guards the classic footer.
+func TestParseHTMLExtractsRelatedSearchesFromBrsContainer(t *testing.T) {
+	t.Parallel()
+
+	html := `
+<ol id="b_results">
+  <li class="b_algo">
+    <h2><a href="https://example.com/result">Organic result</a></h2>
+    <div class="b_caption"><p>Snippet</p></div>
+  </li>
+</ol>
+<div id="brs">
+  <ul>
+    <li><a href="https://www.bing.com/search?q=best+languages+2026">best languages 2026</a></li>
+    <li><a href="https://www.bing.com/search?q=easiest+language+to+learn">easiest language to learn</a></li>
+  </ul>
+</div>`
+
+	results, err := ParseHTML(bytes.NewReader([]byte(html)))
+	if err != nil {
+		t.Fatalf("ParseHTML() error = %v", err)
+	}
+	assertFeatureType(t, results, core.ResultTypeRelatedSearches)
+}
+
+// TestParseHTMLExtractsSimilarSearchesFromInlineRail guards the inline rail.
+func TestParseHTMLExtractsSimilarSearchesFromInlineRail(t *testing.T) {
+	t.Parallel()
+
+	html := `
+<ol id="b_results">
+  <li class="b_algo">
+    <h2><a href="https://example.com/result">Organic result</a></h2>
+    <div class="b_caption"><p>Snippet</p></div>
+  </li>
+  <li class="b_ans">
+    <div id="inline_rs" class="b_hide">
+      <div id="rs_root" class="rsExplr">
+        <h2><a>Users also search for</a><a>Close</a></h2>
+        <ul>
+          <li class="rslist"><a href="https://www.bing.com/ck/a?u=a1aHR0cA"><span class="b_suggestionText">learn coding free</span></a></li>
+          <li class="rslist"><a href="https://www.bing.com/ck/a?u=a1aHR0cB"><span class="b_suggestionText">where to start programming</span></a></li>
+        </ul>
+      </div>
+    </div>
+  </li>
+</ol>`
+
+	results, err := ParseHTML(bytes.NewReader([]byte(html)))
+	if err != nil {
+		t.Fatalf("ParseHTML() error = %v", err)
+	}
+	related := findFeature(results, core.ResultTypeRelatedSearches)
+	if related == nil {
+		t.Fatalf("expected related_searches from inline rail")
+	}
+	if len(related.Items) != 2 {
+		t.Fatalf("expected 2 rail items (header chrome excluded), got %d: %#v", len(related.Items), related.Items)
+	}
+	for _, it := range related.Items {
+		if strings.Contains(it.Text, "Users also search") || it.Text == "Close" {
+			t.Fatalf("rail header chrome leaked as item: %q", it.Text)
+		}
+	}
+}
+
 func TestParseHTMLOrganicOnlyHasNoSerpFeatures(t *testing.T) {
 	t.Parallel()

@@ -94,6 +160,17 @@ func TestParseHTMLOrganicOnlyHasNoSerpFeatures(t *testing.T) {
 	assertNoFeatures(t, results)
 }

+func findFeature(results []core.SearchResult, want core.ResultType) *core.SerpFeature {
+	for _, result := range results {
+		for i := range result.Features {
+			if result.Features[i].Type == want {
+				return &result.Features[i]
+			}
+		}
+	}
+	return nil
+}
+
 func assertFeatureType(t *testing.T, results []core.SearchResult, want core.ResultType) {
 	t.Helper()
 	for _, result := range results {
--- a/bing/url.go
+++ b/bing/url.go
@@ -101,11 +101,7 @@ func BuildURL(q core.Query) (string, error) {
 		params.Add("filters", filter)
 	}

-	// Bing-specific parameters for consistent results
-	params.Add("form", "QBLH")        // Standard search form
-	params.Add("qs", "HS")            // Query suggestions
-	params.Add("sp", "-1")            // Search provider
-	params.Add("pq", params.Get("q")) // Previous query
+	// Omit autosuggest state; fresh sessions can reuse stale q otherwise.

 	base.RawQuery = params.Encode()
 	return base.String(), nil
--- a/cmd/engines.go
+++ b/cmd/engines.go
@@ -0,0 +1,78 @@
+package cmd
+
+import (
+	"context"
+	"io"
+
+	"github.com/karust/openserp/baidu"
+	"github.com/karust/openserp/bing"
+	"github.com/karust/openserp/core"
+	"github.com/karust/openserp/duckduckgo"
+	"github.com/karust/openserp/ecosia"
+	"github.com/karust/openserp/google"
+	"github.com/karust/openserp/yandex"
+)
+
+// engineSpec is the single registry row for a search engine, driving CLI search,
+// raw dispatch, serve's browserEngineSpecs, and the alias/validation strings.
+// cfg points into the live config global; rawSearchFn is nil when an engine has
+// no browserless mode.
+type engineSpec struct {
+	name        string
+	aliases     []string
+	factory     func(core.Browser, core.SearchEngineOptions) core.SearchEngine
+	rawSearchFn func(context.Context, core.Query) ([]core.SearchResult, error)
+	parseHTMLFn func(io.Reader) ([]core.SearchResult, error)
+	cfg         *EngineConfig
+}
+
+func (s engineSpec) opts() core.SearchEngineOptions {
+	return s.cfg.SearchEngineOptions
+}
+
+func engineSpecs() []engineSpec {
+	return []engineSpec{
+		{name: "google", factory: newEngine(google.New), rawSearchFn: google.Search, parseHTMLFn: google.ParseHTML, cfg: &config.GoogleConfig},
+		{name: "yandex", factory: newEngine(yandex.New), rawSearchFn: yandex.Search, parseHTMLFn: yandex.ParseHTML, cfg: &config.YandexConfig},
+		{name: "baidu", factory: newEngine(baidu.New), rawSearchFn: baidu.Search, parseHTMLFn: baidu.ParseHTML, cfg: &config.BaiduConfig},
+		{name: "bing", factory: newEngine(bing.New), parseHTMLFn: bing.ParseHTML, cfg: &config.BingConfig},
+		{name: "duckduckgo", aliases: []string{"duck", "ddg"}, factory: newEngine(duckduckgo.New), parseHTMLFn: duckduckgo.ParseHTML, cfg: &config.DuckDuckGoConfig},
+		{name: "ecosia", factory: newEngine(ecosia.New), rawSearchFn: ecosia.Search, parseHTMLFn: ecosia.ParseHTML, cfg: &config.EcosiaConfig},
+	}
+}
+
+// newEngine adapts a concrete pkg.New (returning *Engine) to the
+// core.SearchEngine-typed factory the registry stores.
+func newEngine[T core.SearchEngine](ctor func(core.Browser, core.SearchEngineOptions) T) func(core.Browser, core.SearchEngineOptions) core.SearchEngine {
+	return func(b core.Browser, o core.SearchEngineOptions) core.SearchEngine {
+		return ctor(b, o)
+	}
+}
+
+// engineValidArgs returns every accepted engine token (canonical names +
+// aliases) for cobra's OnlyValidArgs validation.
+func engineValidArgs() []string {
+	specs := engineSpecs()
+	args := make([]string, 0, len(specs))
+	for _, s := range specs {
+		args = append(args, s.name)
+		args = append(args, s.aliases...)
+	}
+	return args
+}
+
+// resolveEngineSpec returns the spec whose canonical name or alias matches raw
+// (case/space already normalized by the caller), or false when unknown.
+func resolveEngineSpec(raw string) (engineSpec, bool) {
+	for _, s := range engineSpecs() {
+		if s.name == raw {
+			return s, true
+		}
+		for _, alias := range s.aliases {
+			if alias == raw {
+				return s, true
+			}
+		}
+	}
+	return engineSpec{}, false
+}
--- a/cmd/root.go
+++ b/cmd/root.go
@@ -17,7 +17,7 @@ import (
 )

 const (
-	version               = "0.8.3"
+	version               = "0.8.12"
 	defaultConfigFilename = "config"
 	envPrefix             = "OPENSERP"
 )
@@ -51,6 +51,7 @@ type ServerConfig struct {
 	ConfigPath    string `mapstructure:"config_path"`
 	IsDebug       bool   `mapstructure:"debug"`
 	IsVerbose     bool   `mapstructure:"verbose"`
+	IsQuiet       bool   `mapstructure:"quiet"`
 	IsRawRequests bool   `mapstructure:"raw_requests"`
 	Insecure      bool   `mapstructure:"insecure"`
 }
@@ -115,6 +116,7 @@ var flagToConfigKey = map[string]string{
 	"profiles-json":           "app.profiles",
 	"verbose":                 "server.verbose",
 	"debug":                   "server.debug",
+	"quiet":                   "server.quiet",
 	"head":                    "app.head",
 	"leakless":                "app.leakless",
 	"raw":                     "server.raw_requests",
@@ -154,12 +156,24 @@ var RootCmd = &cobra.Command{
 		}
 		config.App.LogFormat = logFormat

-		core.InitLogger(config.Server.IsVerbose, config.Server.IsDebug, config.App.LogFormat)
+		// One-shot CLI commands default to quiet so stdout is payload-only.
+		// Server mode keeps request logs unless server.quiet is set.
+		quiet := config.Server.IsQuiet
+		if commandDefaultsToQuiet(cmd) && !cmd.Flags().Changed("quiet") {
+			quiet = true
+		}
+		config.Server.IsQuiet = quiet
+
+		core.InitLogger(config.Server.IsVerbose, config.Server.IsDebug, quiet, config.App.LogFormat)
 		logrus.WithField("config", sanitizedConfigForLog(config)).Debug("Final config")
 		return nil
 	},
 }

+func commandDefaultsToQuiet(cmd *cobra.Command) bool {
+	return cmd != nil && cmd.Name() != serveCMD.Name()
+}
+
 func sanitizedConfigForLog(cfg Config) map[string]interface{} {
 	return map[string]interface{}{
 		"server": cfg.Server,
@@ -279,7 +293,7 @@ func initializeConfig(cmd *cobra.Command) error {
 		envKey := envPrefix + "_" + strings.ToUpper(strings.ReplaceAll(key, ".", "_"))
 		err := v.BindEnv(key, envKey)
 		if err != nil {
-			logrus.WithError(err).Error(fmt.Sprintf("Unable to bind ENV valye: %v", err))
+			logrus.WithError(err).Error(fmt.Sprintf("Unable to bind ENV value: %v", err))
 		}
 	}

@@ -380,6 +394,7 @@ func setConfigDefaults(v *viper.Viper) {
 	v.SetDefault("server.port", 7070)
 	v.SetDefault("server.debug", false)
 	v.SetDefault("server.verbose", false)
+	v.SetDefault("server.quiet", false)
 	v.SetDefault("server.raw_requests", false)
 	v.SetDefault("server.insecure", false)
 	v.SetDefault("app.log_format", "")
@@ -436,8 +451,9 @@ func init() {
 	RootCmd.PersistentFlags().StringVar(&config.App.ProfilesJSON, "profiles", "", "Path to browser profile catalog JSON")
 	RootCmd.PersistentFlags().BoolVarP(&config.Server.IsVerbose, "verbose", "v", false, "Use verbose output")
 	RootCmd.PersistentFlags().BoolVarP(&config.Server.IsDebug, "debug", "d", false, "Use debug output. Disable headless browser")
+	RootCmd.PersistentFlags().BoolVarP(&config.Server.IsQuiet, "quiet", "q", false, "Suppress info logs on stderr (default for CLI commands)")
 	RootCmd.PersistentFlags().BoolVarP(&config.App.IsBrowserHead, "head", "", false, "Enable browser UI")
-	RootCmd.PersistentFlags().BoolVarP(&config.App.IsLeakless, "leakless", "l", false, "Use leakless mode to insure browser instances are closed after search")
+	RootCmd.PersistentFlags().BoolVarP(&config.App.IsLeakless, "leakless", "l", false, "Use leakless mode to ensure browser instances are closed after search")
 	RootCmd.PersistentFlags().BoolVarP(&config.Server.IsRawRequests, "raw", "r", false, "Disable browser usage, use HTTP requests")
 	RootCmd.PersistentFlags().BoolVarP(&config.App.IsLeaveHead, "leave", "", false, "Leave browser and tabs opened after search is made")
 	RootCmd.PersistentFlags().StringVarP(&config.Config2Capcha.ApiKey, "2captcha_key", "", "", "2 captcha api key")
--- a/cmd/search.go
+++ b/cmd/search.go
@@ -4,42 +4,91 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
-	"os"
+	"net/url"
 	"strings"
+	"sync"
 	"time"

-	"github.com/karust/openserp/baidu"
-	"github.com/karust/openserp/bing"
+	"github.com/google/uuid"
 	"github.com/karust/openserp/core"
-	"github.com/karust/openserp/duckduckgo"
-	"github.com/karust/openserp/ecosia"
-	"github.com/karust/openserp/google"
-	"github.com/karust/openserp/yandex"
+	extractpkg "github.com/karust/openserp/extract"
 	"github.com/sirupsen/logrus"
 	"github.com/spf13/cobra"
 )

-var searchCMD = &cobra.Command{
-	Use:     "search",
-	Aliases: []string{"find"},
-	Short:   "Search results using chosen web search engine (google, yandex, baidu, bing, duckduckgo, ecosia)",
-	Args:    cobra.MatchAll(cobra.OnlyValidArgs, cobra.ExactArgs(2)),
-	Run:     search,
+// searchFlags holds the per-invocation CLI flags for the search command.
+type searchFlags struct {
+	limit    int
+	lang     string
+	region   string
+	start    int
+	site     string
+	filetype string
+	format   string
+	full     bool
+	features bool
+	extract  int
+	timeout  int
 }

-func search(cmd *cobra.Command, args []string) {
+var searchOpts searchFlags
+
+var searchCMD = &cobra.Command{
+	Use:     "search [engine] [query]",
+	Aliases: []string{"find"},
+	Short:   "Search results using chosen web search engine (google, yandex, baidu, bing, duckduckgo, ecosia)",
+	// Validate the engine ourselves; cobra.OnlyValidArgs would also reject the
+	// query arg. ValidArgs still feeds shell completion.
+	Args:      cobra.MatchAll(cobra.ExactArgs(2), validateEngineArg),
+	ValidArgs: engineValidArgs(),
+	RunE:      search,
+}
+
+// validateEngineArg checks args[0] against the registry with a clear error,
+// without rejecting the query arg.
+func validateEngineArg(cmd *cobra.Command, args []string) error {
+	if len(args) == 0 {
+		return nil
+	}
+	if _, ok := resolveEngineSpec(normalizeEngineArg(args[0])); !ok {
+		return fmt.Errorf("unknown engine %q; valid: %s", args[0], strings.Join(engineValidArgs(), ", "))
+	}
+	return nil
+}
+
+func search(cmd *cobra.Command, args []string) error {
+	// Already validated by validateEngineArg, so this can't miss.
 	engineType := normalizeEngineArg(args[0])
+	spec, _ := resolveEngineSpec(engineType)
+
+	format, err := normalizeSearchFormat(searchOpts.format)
+	if err != nil {
+		return err
+	}
+
+	limit := searchOpts.limit
+	if limit <= 0 {
+		limit = 10
+	}
 	query := core.Query{
 		Text:     args[1],
-		Limit:    10,
+		LangCode: searchOpts.lang,
+		Region:   searchOpts.region,
+		Site:     searchOpts.site,
+		Filetype: searchOpts.filetype,
+		Limit:    limit,
+		Start:    searchOpts.start,
 		Filter:   true,
+		Features: searchOpts.features,
 		Insecure: config.Server.Insecure,
 	}
+	if err := applyCLIExtractFlag(&query, searchOpts.extract); err != nil {
+		return err
+	}

 	captchaSolverEnabled, captchaSolverAPIKey, err := resolveCaptchaSolverConfig()
 	if err != nil {
-		logrus.WithError(err).Error(fmt.Sprintf("Error validating captcha solver config: %v", err))
-		os.Exit(1)
+		return fmt.Errorf("validate captcha solver config: %w", err)
 	}

 	proxyRuntime := core.ProxyRuntimeBrowser
@@ -49,39 +98,45 @@ func search(cmd *cobra.Command, args []string) {

 	proxyCfg, err := buildNormalizedProxyConfig(proxyRuntime)
 	if err != nil {
-		logrus.WithError(err).Error(fmt.Sprintf("Error validating proxy config: %v", err))
-		return
+		return fmt.Errorf("validate proxy config: %w", err)
 	}

 	policy := resolveEngineProxyPolicy(proxyCfg, engineType)

 	selectedProxy, err := selectCLIProxy(proxyCfg, policy)
 	if err != nil {
-		logrus.WithError(err).Error(fmt.Sprintf("Error selecting proxy for %s: %v", engineType, err))
-		return
+		return fmt.Errorf("select proxy for %s: %w", engineType, err)
 	}

 	if config.Server.IsRawRequests {
 		query.ProxyURL = selectedProxy
 	}

+	// Bound the whole search so a wedged Chrome can't hang the CLI forever.
+	timeoutSec := searchOpts.timeout
+	if timeoutSec <= 0 {
+		timeoutSec = 60
+	}
+	ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutSec)*time.Second)
+	defer cancel()
+
 	logrus.WithFields(logrus.Fields{
 		"engine":     engineType,
 		"query_hash": core.QueryHashFromQuery(query),
 	}).Info(fmt.Sprintf("Starting SERP search request using %s engine for query: %s", engineType, query.Text))

+	startedAt := time.Now()
 	var results []core.SearchResult
 	if config.Server.IsRawRequests {
 		logrus.WithField("engine", engineType).Info(fmt.Sprintf("Using raw requests mode for %s search", engineType))
-		results, err = searchRaw(engineType, query)
+		results, err = searchRaw(ctx, spec, query)
 	} else {
 		logrus.WithField("engine", engineType).Info(fmt.Sprintf("Using browser mode for %s search", engineType))
-		results, err = searchBrowser(engineType, query, selectedProxy, captchaSolverEnabled, captchaSolverAPIKey)
+		results, err = searchBrowser(ctx, spec, query, selectedProxy, captchaSolverEnabled, captchaSolverAPIKey)
 	}

 	if err != nil {
-		logrus.WithError(err).WithField("engine", engineType).Error(fmt.Sprintf("Error during %s search: %s", engineType, err))
-		return
+		return fmt.Errorf("%s search: %w", engineType, err)
 	}

 	logrus.WithFields(logrus.Fields{
@@ -89,17 +144,203 @@ func search(cmd *cobra.Command, args []string) {
 		"results_count": len(results),
 	}).Info(fmt.Sprintf("Successfully completed SERP search using %s engine, returned %d results", engineType, len(results)))

-	b, err := json.MarshalIndent(results, "", " ")
-	if err != nil {
-		logrus.Error(err)
-		return
+	env := buildCLIEnvelope(spec.name, query, results, startedAt)
+	if query.Extract {
+		if err := enrichCLIEnvelopeWithExtraction(ctx, env, query, format, selectedProxy, captchaSolverEnabled, captchaSolverAPIKey); err != nil {
+			return fmt.Errorf("extract search results: %w", err)
+		}
 	}
-
-	fmt.Println(string(b))
+	payload := renderCLIEnvelope(env, format, searchOpts.full)
+	fmt.Println(strings.TrimRight(string(payload), "\n"))
+	return nil
 }

-func searchBrowser(engineType string, query core.Query, browserProxyURL string, captchaSolverEnabled bool, captchaSolverAPIKey string) ([]core.SearchResult, error) {
-	var engine core.SearchEngine
+func buildCLIEnvelope(engineName string, query core.Query, results []core.SearchResult, startedAt time.Time) *core.Envelope {
+	env := core.NewEnvelope(query, uuid.NewString(), startedAt, []string{engineName})
+	ectx := core.EnrichContext{Engine: engineName, Query: query}
+	for _, r := range results {
+		core.AppendEnrichedSearchResult(env, r, ectx, startedAt)
+	}
+	env.Finalize(startedAt, query)
+	return env
+}
+
+// renderCLIEnvelope renders a v2.1 envelope. JSON/ndjson always carry the full
+// envelope; text/markdown omit serp_features unless --full.
+func renderCLIEnvelope(env *core.Envelope, format string, full bool) []byte {
+	if !full && format != "json" && format != "ndjson" {
+		env.SerpFeatures = nil
+	}
+
+	switch format {
+	case "text":
+		return core.RenderText(env)
+	case "markdown":
+		return core.RenderMarkdown(env)
+	case "ndjson":
+		return core.RenderNDJSON(env)
+	default: // json
+		b, err := json.MarshalIndent(env, "", "  ")
+		if err != nil {
+			logrus.WithError(err).Error("marshal envelope")
+			return nil
+		}
+		return b
+	}
+}
+
+const maxCLIExtractTop = 5
+
+func applyCLIExtractFlag(query *core.Query, extractTop int) error {
+	top, err := normalizeCLIExtractTop(extractTop)
+	if err != nil {
+		return err
+	}
+	if top == 0 {
+		return nil
+	}
+	if !config.Extract.Enabled {
+		return fmt.Errorf("extraction is disabled in config")
+	}
+	query.Extract = true
+	query.ExtractTop = top
+	query.ExtractMode = string(extractpkg.ModeAuto)
+	return nil
+}
+
+func normalizeCLIExtractTop(raw int) (int, error) {
+	if raw < 0 {
+		return 0, fmt.Errorf("--extract must be a non-negative integer")
+	}
+	if raw > maxCLIExtractTop {
+		return maxCLIExtractTop, nil
+	}
+	return raw, nil
+}
+
+func enrichCLIEnvelopeWithExtraction(ctx context.Context, env *core.Envelope, query core.Query, format string, proxyURL string, captchaSolverEnabled bool, captchaSolverAPIKey string) error {
+	if env == nil || !query.Extract {
+		return nil
+	}
+	query.ProxyURL = proxyURL
+	extractor, closeExtractor, err := newCLIExtractor(captchaSolverEnabled, captchaSolverAPIKey)
+	if err != nil {
+		return err
+	}
+	defer closeExtractor()
+
+	// Same depth bounds, batch deadline, and candidate fill-in as the HTTP server.
+	core.EnrichEnvelopeWithExtraction(ctx, env, query, format, extractor, config.Extract)
+	return nil
+}
+
+// newCLIExtractor builds an Extractor backed by a lazily-created, single-use
+// browser. The raw path delegates to core.RawExtractFetch; the rendered path
+// validates the target, gates auth'd SOCKS, then reuses core.RenderExtractHTML.
+func newCLIExtractor(captchaSolverEnabled bool, captchaSolverAPIKey string) (extractpkg.Extractor, func(), error) {
+	cfg := config.Extract.Normalized()
+	var browserMu sync.Mutex
+	var browser *core.Browser
+
+	closeExtractor := func() {
+		browserMu.Lock()
+		defer browserMu.Unlock()
+		if browser == nil {
+			return
+		}
+		if err := browser.Close(); err != nil {
+			logrus.WithError(err).Debug("Extraction browser close error")
+		}
+		browser = nil
+	}
+
+	extractor := extractpkg.Extractor{
+		Cfg: cfg,
+		RawFetch: func(ctx context.Context, req extractpkg.ExtractRequest) (*extractpkg.FetchResponse, error) {
+			return core.RawExtractFetch(ctx, req, cfg, config.Server.Insecure)
+		},
+		RenderedFetch: func(ctx context.Context, req extractpkg.ExtractRequest) (*extractpkg.FetchResponse, error) {
+			if err := validateCLIExtractTargetURL(ctx, req.URL, cfg.AllowPrivateNetworks); err != nil {
+				return nil, err
+			}
+			if core.IsAuthenticatedSocksProxyURL(req.ProxyURL) {
+				return nil, fmt.Errorf(
+					"%w: browser runtime does not support authenticated SOCKS proxy %s",
+					core.ErrProxyUnavailable,
+					core.MaskProxyURL(req.ProxyURL),
+				)
+			}
+
+			browserMu.Lock()
+			if browser == nil {
+				created, err := newCLIExtractBrowser(cfg, req.ProxyURL, captchaSolverEnabled, captchaSolverAPIKey)
+				if err != nil {
+					browserMu.Unlock()
+					return nil, err
+				}
+				browser = created
+			}
+			current := browser
+			browserMu.Unlock()
+
+			return core.RenderExtractHTML(ctx, current, req)
+		},
+	}
+	return extractor, closeExtractor, nil
+}
+
+func newCLIExtractBrowser(cfg extractpkg.Config, proxyURL string, captchaSolverEnabled bool, captchaSolverAPIKey string) (*core.Browser, error) {
+	blockedResourceTypes, err := core.ParseBlockedResourceTypes(config.App.BlockResources)
+	if err != nil {
+		return nil, fmt.Errorf("invalid block_resources config: %w", err)
+	}
+	opts := core.BrowserOpts{
+		IsHeadless:           !config.App.IsBrowserHead && !config.Server.IsDebug,
+		IsLeakless:           config.App.IsLeakless,
+		Timeout:              cfg.Timeout,
+		LeavePageOpen:        false,
+		CaptchaSolverEnabled: captchaSolverEnabled,
+		CaptchaSolverApiKey:  captchaSolverAPIKey,
+		BrowserPath:          config.App.BrowserPath,
+		ProxyURL:             proxyURL,
+		Insecure:             config.Server.Insecure,
+		BlockResourceTypes:   blockedResourceTypes,
+		BlockTrackers:        config.App.BlockTrackers,
+	}
+	return core.NewBrowser(opts)
+}
+
+func validateCLIExtractTargetURL(ctx context.Context, rawURL string, allowPrivateNetworks bool) error {
+	targetURL := extractpkg.NormalizeURL(strings.TrimSpace(rawURL))
+	if allowPrivateNetworks {
+		parsed, err := url.ParseRequestURI(targetURL)
+		if err != nil {
+			return fmt.Errorf("invalid url: %w", err)
+		}
+		if parsed.Scheme != "http" && parsed.Scheme != "https" {
+			return fmt.Errorf("url must use http or https")
+		}
+		return nil
+	}
+	return core.ValidatePublicHTTPURL(ctx, targetURL)
+}
+
+func normalizeSearchFormat(raw string) (string, error) {
+	switch strings.ToLower(strings.TrimSpace(raw)) {
+	case "", "json":
+		return "json", nil
+	case "text", "txt":
+		return "text", nil
+	case "markdown", "md":
+		return "markdown", nil
+	case "ndjson", "jsonl":
+		return "ndjson", nil
+	default:
+		return "", fmt.Errorf("invalid --format %q; valid: json, text, markdown, ndjson", raw)
+	}
+}
+
+func searchBrowser(ctx context.Context, spec engineSpec, query core.Query, browserProxyURL string, captchaSolverEnabled bool, captchaSolverAPIKey string) ([]core.SearchResult, error) {
 	blockedResourceTypes, err := core.ParseBlockedResourceTypes(config.App.BlockResources)
 	if err != nil {
 		return nil, fmt.Errorf("invalid block_resources config: %w", err)
@@ -134,49 +375,24 @@ func searchBrowser(engineType string, query core.Query, browserProxyURL string,
 	if err != nil {
 		return nil, err
 	}
+	// Close the browser so Chromium never outlives the CLI run.
+	defer func() {
+		if closeErr := browser.Close(); closeErr != nil {
+			logrus.WithError(closeErr).Debug("Browser close error")
+		}
+	}()

-	switch strings.ToLower(engineType) {
-	case "yandex":
-		engine = yandex.New(*browser, config.YandexConfig.SearchEngineOptions)
-	case "google":
-		engine = google.New(*browser, config.GoogleConfig.SearchEngineOptions)
-	case "baidu":
-		engine = baidu.New(*browser, config.BaiduConfig.SearchEngineOptions)
-	case "bing":
-		engine = bing.New(*browser, config.BingConfig.SearchEngineOptions)
-	case "duckduckgo":
-		engine = duckduckgo.New(*browser, config.DuckDuckGoConfig.SearchEngineOptions)
-	case "ecosia":
-		engine = ecosia.New(*browser, config.EcosiaConfig.SearchEngineOptions)
-	default:
-		return nil, fmt.Errorf("no %q search engine found", engineType)
-	}
-
-	return engine.Search(context.Background(), query)
+	engine := spec.factory(*browser, spec.opts())
+	return engine.Search(ctx, query)
 }

-func searchRaw(engineType string, query core.Query) ([]core.SearchResult, error) {
+func searchRaw(ctx context.Context, spec engineSpec, query core.Query) ([]core.SearchResult, error) {
 	logrus.Warn("Browserless results are very inconsistent or may not even work!")
-	ctx := context.Background()
-
-	switch strings.ToLower(engineType) {
-	case "yandex":
-		return yandex.Search(ctx, query)
-	case "google":
-		return google.Search(ctx, query)
-	case "baidu":
-		return baidu.Search(ctx, query)
-	case "ecosia":
-		return ecosia.Search(ctx, query)
-	case "bing":
-		logrus.Warn("Bing does not support raw HTTP requests mode. Please use browser mode instead.")
-		return nil, fmt.Errorf("bing does not support raw requests mode")
-	case "duckduckgo":
-		logrus.Warn("DuckDuckGo does not support raw HTTP requests mode. Please use browser mode instead.")
-		return nil, fmt.Errorf("duckduckgo does not support raw requests mode")
-	default:
-		return nil, fmt.Errorf("no %q search engine found", engineType)
+	if spec.rawSearchFn == nil {
+		logrus.Warnf("%s does not support raw HTTP requests mode. Please use browser mode instead.", spec.name)
+		return nil, fmt.Errorf("%s does not support raw requests mode", spec.name)
 	}
+	return spec.rawSearchFn(ctx, query)
 }

 func selectCLIProxy(proxyCfg core.ProxyConfig, policy core.ProxyPolicy) (string, error) {
@@ -201,14 +417,20 @@ func selectCLIProxy(proxyCfg core.ProxyConfig, policy core.ProxyPolicy) (string,
 }

 func normalizeEngineArg(raw string) string {
-	switch strings.ToLower(strings.TrimSpace(raw)) {
-	case "duck":
-		return "duckduckgo"
-	default:
-		return strings.ToLower(strings.TrimSpace(raw))
-	}
+	return strings.ToLower(strings.TrimSpace(raw))
 }

 func init() {
+	searchCMD.Flags().IntVar(&searchOpts.limit, "limit", 10, "Maximum number of results")
+	searchCMD.Flags().StringVar(&searchOpts.lang, "lang", "", "Language hint (e.g. EN, DE, RU)")
+	searchCMD.Flags().StringVar(&searchOpts.region, "region", "", "Region/market hint (e.g. RU, en-US)")
+	searchCMD.Flags().IntVar(&searchOpts.start, "start", 0, "Pagination start offset")
+	searchCMD.Flags().StringVar(&searchOpts.site, "site", "", "Restrict results to a domain (e.g. github.com)")
+	searchCMD.Flags().StringVar(&searchOpts.filetype, "file", "", "File type filter (e.g. pdf)")
+	searchCMD.Flags().StringVar(&searchOpts.format, "format", "json", "Output format: json, text, markdown, ndjson")
+	searchCMD.Flags().BoolVar(&searchOpts.full, "full", false, "Include SERP features in text/markdown output")
+	searchCMD.Flags().BoolVar(&searchOpts.features, "features", false, "Parse SERP feature modules (browser mode)")
+	searchCMD.Flags().IntVar(&searchOpts.extract, "extract", 0, "Extract clean content from the top N results using auto mode (1-5)")
+	searchCMD.Flags().IntVar(&searchOpts.timeout, "search-timeout", 60, "Overall search timeout in seconds")
 	RootCmd.AddCommand(searchCMD)
 }
--- a/cmd/search_test.go
+++ b/cmd/search_test.go
@@ -0,0 +1,82 @@
+package cmd
+
+import (
+	"strings"
+	"testing"
+
+	"github.com/karust/openserp/core"
+	extractpkg "github.com/karust/openserp/extract"
+)
+
+func TestNormalizeCLIExtractTop(t *testing.T) {
+	tests := []struct {
+		name    string
+		raw     int
+		want    int
+		wantErr bool
+	}{
+		{name: "disabled", raw: 0, want: 0},
+		{name: "one", raw: 1, want: 1},
+		{name: "clamped", raw: 20, want: maxCLIExtractTop},
+		{name: "negative", raw: -1, wantErr: true},
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.name, func(t *testing.T) {
+			got, err := normalizeCLIExtractTop(tc.raw)
+			if tc.wantErr {
+				if err == nil {
+					t.Fatal("expected error")
+				}
+				return
+			}
+			if err != nil {
+				t.Fatalf("unexpected error: %v", err)
+			}
+			if got != tc.want {
+				t.Fatalf("normalizeCLIExtractTop(%d) = %d, want %d", tc.raw, got, tc.want)
+			}
+		})
+	}
+}
+
+func TestApplyCLIExtractFlagSetsAutoMode(t *testing.T) {
+	previous := config
+	config.Extract = extractpkg.DefaultConfig()
+	defer func() { config = previous }()
+
+	query := core.Query{Text: "weather today"}
+	if err := applyCLIExtractFlag(&query, 2); err != nil {
+		t.Fatalf("applyCLIExtractFlag() error = %v", err)
+	}
+	if !query.Extract {
+		t.Fatal("expected query.Extract")
+	}
+	if query.ExtractTop != 2 {
+		t.Fatalf("ExtractTop = %d, want 2", query.ExtractTop)
+	}
+	if query.ExtractMode != string(extractpkg.ModeAuto) {
+		t.Fatalf("ExtractMode = %q, want auto", query.ExtractMode)
+	}
+}
+
+func TestApplyCLIExtractFlagRequiresEnabledConfig(t *testing.T) {
+	previous := config
+	config.Extract = extractpkg.Config{Enabled: false}
+	defer func() { config = previous }()
+
+	query := core.Query{Text: "weather today"}
+	err := applyCLIExtractFlag(&query, 1)
+	if err == nil {
+		t.Fatal("expected error")
+	}
+	if !strings.Contains(err.Error(), "disabled") {
+		t.Fatalf("error = %q, want disabled message", err.Error())
+	}
+}
+
+func TestSearchCommandHasExtractFlag(t *testing.T) {
+	if searchCMD.Flags().Lookup("extract") == nil {
+		t.Fatal("expected search command to expose --extract")
+	}
+}
--- a/cmd/serve.go
+++ b/cmd/serve.go
@@ -2,6 +2,8 @@ package cmd

 import (
 	"context"
+	"crypto/sha256"
+	"encoding/hex"
 	"errors"
 	"fmt"
 	"io"
@@ -13,13 +15,7 @@ import (
 	"syscall"
 	"time"

-	"github.com/karust/openserp/baidu"
-	"github.com/karust/openserp/bing"
 	"github.com/karust/openserp/core"
-	"github.com/karust/openserp/duckduckgo"
-	"github.com/karust/openserp/ecosia"
-	"github.com/karust/openserp/google"
-	"github.com/karust/openserp/yandex"
 	"github.com/sirupsen/logrus"
 	"github.com/spf13/cobra"
 	"golang.org/x/time/rate"
@@ -35,18 +31,11 @@ type rawEngine struct {
 func (r *rawEngine) Search(ctx context.Context, q core.Query) ([]core.SearchResult, error) {
 	q.Insecure = config.Server.Insecure

-	switch r.name {
-	case "google":
-		return google.Search(ctx, q)
-	case "yandex":
-		return yandex.Search(ctx, q)
-	case "baidu":
-		return baidu.Search(ctx, q)
-	case "ecosia":
-		return ecosia.Search(ctx, q)
-	default:
+	spec, ok := resolveEngineSpec(r.name)
+	if !ok || spec.rawSearchFn == nil {
 		return nil, fmt.Errorf("unsupported engine: %s", r.name)
 	}
+	return spec.rawSearchFn(ctx, q)
 }

 func (r *rawEngine) SearchImage(_ context.Context, _ core.Query) ([]core.SearchResult, error) {
@@ -302,12 +291,12 @@ func newBrowserPool(base core.BrowserOpts, defaultLaunchProxyURL string, laneSto
 		stopSweeper:  make(chan struct{}),
 		sweeperDone:  make(chan struct{}),
 	}
-	// A configured global proxy (legacy) becomes a pre-bound entry on the
-	// shared "direct" key so requests without a per-request proxy still use it.
 	if launchURL := strings.TrimSpace(defaultLaunchProxyURL); launchURL != "" {
-		pool.browsers[directBrowserKey] = &pooledBrowser{
-			launchProxyURL: launchURL,
-			lastUsedAt:     time.Now(),
+		if key := browserPoolKey(launchURL); key != directBrowserKey {
+			pool.browsers[key] = &pooledBrowser{
+				launchProxyURL: launchURL,
+				lastUsedAt:     time.Now(),
+			}
 		}
 	}
 	if idleTTL > 0 {
@@ -319,7 +308,8 @@ func newBrowserPool(base core.BrowserOpts, defaultLaunchProxyURL string, laneSto
 }

 // browserPoolKey derives the pool key from a request's proxy URL. Authenticated
-// HTTP/HTTPS proxies get their own Chrome keyed by scheme+host+port+username.
+// HTTP/HTTPS proxies get their own Chrome keyed by scheme+host+port+username
+// plus a short auth hash, so provider session tokens do not share one Chrome.
 // Empty/unauthenticated/SOCKS request URLs fall through to the shared
 // "direct" Chrome.
 func browserPoolKey(requestProxyURL string) string {
@@ -343,8 +333,10 @@ func browserPoolKey(requestProxyURL string) string {
 	if parsed.User == nil {
 		return directBrowserKey
 	}
-	username := parsed.User.Username()
-	return fmt.Sprintf("%s|%s|%s", parsed.Scheme, parsed.Host, username)
+	// Hash the full userinfo so a rotating password gets its own Chrome without
+	// leaking credentials into the key. Scheme+host+username stay readable.
+	sum := sha256.Sum256([]byte(parsed.User.String()))
+	return fmt.Sprintf("%s|%s|%s|%s", parsed.Scheme, parsed.Host, parsed.User.Username(), hex.EncodeToString(sum[:])[:16])
 }

 // browserLaunchURL returns the URL to pass to launcher.Proxy for a given
@@ -626,56 +618,17 @@ type browserEngineSpec struct {
 }

 func browserEngineSpecs() []browserEngineSpec {
-	return []browserEngineSpec{
-		{
-			name: "google",
-			opts: config.GoogleConfig.SearchEngineOptions,
-			factory: func(browser core.Browser, opts core.SearchEngineOptions) core.SearchEngine {
-				return google.New(browser, opts)
-			},
-			parseHTMLFn: google.ParseHTML,
-		},
-		{
-			name: "yandex",
-			opts: config.YandexConfig.SearchEngineOptions,
-			factory: func(browser core.Browser, opts core.SearchEngineOptions) core.SearchEngine {
-				return yandex.New(browser, opts)
-			},
-			parseHTMLFn: yandex.ParseHTML,
-		},
-		{
-			name: "baidu",
-			opts: config.BaiduConfig.SearchEngineOptions,
-			factory: func(browser core.Browser, opts core.SearchEngineOptions) core.SearchEngine {
-				return baidu.New(browser, opts)
-			},
-			parseHTMLFn: baidu.ParseHTML,
-		},
-		{
-			name: "bing",
-			opts: config.BingConfig.SearchEngineOptions,
-			factory: func(browser core.Browser, opts core.SearchEngineOptions) core.SearchEngine {
-				return bing.New(browser, opts)
-			},
-			parseHTMLFn: bing.ParseHTML,
-		},
-		{
-			name: "duckduckgo",
-			opts: config.DuckDuckGoConfig.SearchEngineOptions,
-			factory: func(browser core.Browser, opts core.SearchEngineOptions) core.SearchEngine {
-				return duckduckgo.New(browser, opts)
-			},
-			parseHTMLFn: duckduckgo.ParseHTML,
-		},
-		{
-			name: "ecosia",
-			opts: config.EcosiaConfig.SearchEngineOptions,
-			factory: func(browser core.Browser, opts core.SearchEngineOptions) core.SearchEngine {
-				return ecosia.New(browser, opts)
-			},
-			parseHTMLFn: ecosia.ParseHTML,
-		},
+	specs := engineSpecs()
+	out := make([]browserEngineSpec, 0, len(specs))
+	for _, s := range specs {
+		out = append(out, browserEngineSpec{
+			name:        s.name,
+			opts:        s.opts(),
+			factory:     s.factory,
+			parseHTMLFn: s.parseHTMLFn,
+		})
 	}
+	return out
 }

 func buildBrowserEngines(baseOpts core.BrowserOpts, proxyCfg core.ProxyConfig) ([]core.SearchEngine, func() error, core.BrowserResolver, error) {
--- a/cmd/serve_test.go
+++ b/cmd/serve_test.go
@@ -19,30 +19,55 @@ func TestRawEngineCachesRateLimiter(t *testing.T) {
 	}
 }

+func TestCommandDefaultsToQuiet(t *testing.T) {
+	if !commandDefaultsToQuiet(searchCMD) {
+		t.Fatal("expected search command to default to quiet")
+	}
+	if commandDefaultsToQuiet(serveCMD) {
+		t.Fatal("expected serve command to keep server logging by default")
+	}
+}
+
 func TestBrowserPoolKey(t *testing.T) {
 	cases := []struct {
-		name string
-		raw  string
-		want string
+		name       string
+		raw        string
+		wantDirect bool
+		wantPrefix string
 	}{
-		{"empty -> direct", "", directBrowserKey},
-		{"unauth http -> direct", "http://proxy.example:8080", directBrowserKey},
-		{"unauth socks -> direct", "socks5://proxy.example:1080", directBrowserKey},
-		{"auth socks -> direct (rejected upstream)", "socks5://user:pass@proxy.example:1080", directBrowserKey},
-		{"auth http", "http://user:pass@proxy.example:8080", "http|proxy.example:8080|user"},
-		{"auth https different scheme", "https://user:pass@proxy.example:8443", "https|proxy.example:8443|user"},
-		{"different password same key", "http://user:other-pass@proxy.example:8080", "http|proxy.example:8080|user"},
-		{"different user different key", "http://user2:pass@proxy.example:8080", "http|proxy.example:8080|user2"},
-		{"different host different key", "http://user:pass@proxy2.example:8080", "http|proxy2.example:8080|user"},
-		{"different port different key", "http://user:pass@proxy.example:9090", "http|proxy.example:9090|user"},
+		{"empty -> direct", "", true, ""},
+		{"unauth http -> direct", "http://proxy.example:8080", true, ""},
+		{"unauth socks -> direct", "socks5://proxy.example:1080", true, ""},
+		{"auth socks -> direct (rejected upstream)", "socks5://user:pass@proxy.example:1080", true, ""},
+		{"auth http", "http://user:pass@proxy.example:8080", false, "http|proxy.example:8080|user|"},
+		{"auth https different scheme", "https://user:pass@proxy.example:8443", false, "https|proxy.example:8443|user|"},
+		{"different user different key", "http://user2:pass@proxy.example:8080", false, "http|proxy.example:8080|user2|"},
+		{"different host different key", "http://user:pass@proxy2.example:8080", false, "http|proxy2.example:8080|user|"},
+		{"different port different key", "http://user:pass@proxy.example:9090", false, "http|proxy.example:9090|user|"},
 	}
 	for _, tc := range cases {
 		t.Run(tc.name, func(t *testing.T) {
-			if got := browserPoolKey(tc.raw); got != tc.want {
-				t.Fatalf("browserPoolKey(%q) = %q, want %q", tc.raw, got, tc.want)
+			got := browserPoolKey(tc.raw)
+			if tc.wantDirect {
+				if got != directBrowserKey {
+					t.Fatalf("browserPoolKey(%q) = %q, want %q", tc.raw, got, directBrowserKey)
+				}
+				return
+			}
+			if !strings.HasPrefix(got, tc.wantPrefix) {
+				t.Fatalf("browserPoolKey(%q) = %q, want prefix %q", tc.raw, got, tc.wantPrefix)
+			}
+			if strings.Contains(got, "pass") {
+				t.Fatalf("browser pool key leaked password material: %q", got)
 			}
 		})
 	}
+
+	first := browserPoolKey("http://user:pass@proxy.example:8080")
+	second := browserPoolKey("http://user:other-pass@proxy.example:8080")
+	if first == second {
+		t.Fatalf("expected different passwords to produce different browser pool keys: %q", first)
+	}
 }

 func TestBrowserLaunchURL(t *testing.T) {
@@ -67,6 +92,28 @@ func TestBrowserLaunchURL(t *testing.T) {
 	}
 }

+func TestBrowserPoolGlobalProxyDoesNotOccupyDirectSlot(t *testing.T) {
+	globalProxy := "http://user:pass@proxy.example:8080"
+	pool := newBrowserPool(core.BrowserOpts{}, globalProxy, nil, 2, 0)
+	defer func() {
+		if err := pool.close(); err != nil {
+			t.Fatalf("close pool: %v", err)
+		}
+	}()
+
+	if _, ok := pool.browsers[directBrowserKey]; ok {
+		t.Fatalf("global proxy must not occupy %q browser slot", directBrowserKey)
+	}
+	key := browserPoolKey(globalProxy)
+	entry, ok := pool.browsers[key]
+	if !ok {
+		t.Fatalf("expected global proxy slot %q to be pre-bound", key)
+	}
+	if entry.launchProxyURL != globalProxy {
+		t.Fatalf("expected launch proxy %q, got %q", globalProxy, entry.launchProxyURL)
+	}
+}
+
 func TestBrowserPoolEvictLRU(t *testing.T) {
 	// Pre-populate with bare entries (browser=nil) so we exercise eviction
 	// without launching real Chrome. closePooledBrowser handles nil safely.
--- a/config.yaml
+++ b/config.yaml
@@ -2,7 +2,7 @@ server:
  host: 0.0.0.0 # API host to bind
  port: 7000 # API port to bind
  debug: false # Enable debug logs and force browser UI mode
-  verbose: true # Enable info-level request logs
+  verbose: false # Enable debug-level request logs
  raw_requests: false # true = raw HTTP mode, false = browser mode
  insecure: true # Allow insecure TLS connections

@@ -14,12 +14,12 @@ app:
  head: false # Headful mode
  leakless: false # Force browser process cleanup after request
  leave_head: false # Keep tabs open after request
+  block_resources: "image,font,css,media" # Block heavy subresources in browser mode
+  block_trackers: true # Block known tracker domains
  max_processes: 6 # Concurrent Chrome processes
  idle_ttl: 5m # close a Chrome that has not served traffic for this long

  mega_timeout: 90s # max total wait for /mega/* requests; slow engines return partial results
-  block_trackers: true
-  block_resources: "image,font,css,media"

 extract:
  enabled: true
@@ -29,7 +29,7 @@ extract:
  max_concurrent: 2

 proxies:
-  allow_request_proxy_url: false
+  allow_request_proxy_url: true
  # Force a single proxy for all engines.
  # Same behavior as passing --proxy on the CLI.
  #global: http://127.0.0.1:8080
@@ -64,7 +64,7 @@ cors:
  enabled: true
  allow_origins: "*"
  allow_methods: "GET, POST, OPTIONS"
-  allow_headers: "Origin, Content-Type, Accept, Authorization, X-Use-Proxy, X-Proxy-URL, X-Proxy-Country, X-Proxy-Class, X-Proxy-Provider, X-Proxy-Session-ID, X-Request-ID, X-Tenant"
+  allow_headers: "Origin, Content-Type, Accept, Authorization, X-Use-Proxy, X-Proxy-URL, X-Proxy-Country, X-Proxy-Class, X-Proxy-Provider, X-Proxy-Session-ID, X-Request-ID, X-Tenant, X-Use-Profile"
  max_age: 86400

 # 2captcha:
--- a/core/browser.go
+++ b/core/browser.go
@@ -2,13 +2,13 @@ package core

 import (
 	"context"
-	"encoding/json"
 	"errors"
 	"fmt"
 	"net/http"
 	"net/url"
 	"os"
 	"regexp"
+	"runtime"
 	"runtime/debug"
 	"strings"
 	"sync"
@@ -30,7 +30,7 @@ type BrowserOpts struct {
 	IsLeakless bool
 	// Timeout is applied to browser connect and page navigation operations.
 	Timeout time.Duration
-	// LanguageCode sets Accept-Language for emulated requests.
+	// LanguageCode selects the regional browser lane and search locale.
 	LanguageCode string
 	// WaitRequests waits for request-idle state after navigation.
 	WaitRequests bool
@@ -252,8 +252,19 @@ type browserConnection struct {
 	laneProfiles map[string]browserprofile.Profile
 	authCancel   context.CancelFunc
 	authStopped  chan struct{}
+	// lastOK is when a CDP call last succeeded; the health ping is skipped while
+	// it is within healthPingSkipWindow.
+	lastOK time.Time
 }

+const (
+	// healthPingTimeout bounds the per-call connection ping so a wedged Chrome
+	// can't stall navigations while holding the connection lock.
+	healthPingTimeout = 3 * time.Second
+	// healthPingSkipWindow skips the ping when a CDP call succeeded this recently.
+	healthPingSkipWindow = 5 * time.Second
+)
+
 // NewBrowser launches a new Chromium process via Rod launcher and returns a
 // Browser wrapper configured with proxy and captcha solver settings.
 func NewBrowser(opts BrowserOpts) (*Browser, error) {
@@ -268,17 +279,17 @@ func NewBrowser(opts BrowserOpts) (*Browser, error) {
 		return nil, err
 	}

-	// Create launcher.
-	// headless=new uses the full Chrome renderer; legacy --headless disables the
-	// GPU process entirely, making WebGL context creation fail even with SwiftShader.
-	// use-angle=swiftshader-webgl (Chrome ≥112) enables a software WebGL renderer.
+	// headless=new keeps the full renderer, so WebGL falls back to SwiftShader
+	// when there's no GPU - which is what the swiftshader profiles describe.
+	// Legacy --headless kills the GPU process and breaks WebGL entirely.
+	// Leakless defaults to false: the helper binary trips antivirus on Windows.
 	l := launcher.New().Leakless(opts.IsLeakless).
+		Set("lang", browserLaunchLanguage(opts)).
 		Set("disable-blink-features", "AutomationControlled").
-		Delete("enable-automation").
-		Set("use-angle", "swiftshader-webgl").
-		Set("ignore-gpu-blocklist")
+		Delete("enable-automation")
 	if opts.IsHeadless {
-		l = l.HeadlessNew(true)
+		// Initial window only; per-page setDeviceMetricsOverride is what pages see. Sized for the 1920x1080 swiftshader profiles that run on Docker.
+		l = l.HeadlessNew(true).Set("window-size", "1920,1040")
 	} else {
 		l = l.Headless(false)
 	}
@@ -351,6 +362,12 @@ func browserOptsLogFields(opts BrowserOpts) logrus.Fields {
 	}
 }

+func browserLaunchLanguage(_ BrowserOpts) string {
+	// Speech voices are per-process and CDP can't change them, so pin the
+	// launch locale to the catalog rather than the request hint.
+	return "en-US"
+}
+
 func maskedProxyLogValue(proxyURL string) string {
 	if strings.TrimSpace(proxyURL) == "" {
 		return ""
@@ -557,18 +574,18 @@ func (b *Browser) ensureConnectedBrowser(ctx context.Context, forceReconnect boo
 			return nil, err
 		}
 		state.browser = connected
+		state.lastOK = time.Now()
 		return state.browser, nil
 	}

-	// Bound just this health-ping with a per-call timeout so a wedged browser
-	// can't block the connection lock forever. Use a fresh derived context each
-	// call (not browser.Timeout, which would bake a permanent deadline into the
-	// persistent connection — see newRodBrowser).
-	pingTimeout := b.Timeout
-	if pingTimeout <= 0 {
-		pingTimeout = 30 * time.Second
+	// A recent successful CDP call means the connection is alive; skip the ping.
+	if !state.lastOK.IsZero() && time.Since(state.lastOK) < healthPingSkipWindow {
+		return state.browser, nil
 	}
-	pingCtx, cancelPing := context.WithTimeout(EnsureContext(ctx), pingTimeout)
+
+	// Fresh derived context per call (not browser.Timeout, which would bake a
+	// permanent deadline into the persistent connection — see newRodBrowser).
+	pingCtx, cancelPing := context.WithTimeout(EnsureContext(ctx), healthPingTimeout)
 	_, pingErr := state.browser.Context(pingCtx).Version()
 	cancelPing()
 	if pingErr != nil {
@@ -579,6 +596,7 @@ func (b *Browser) ensureConnectedBrowser(ctx context.Context, forceReconnect boo
 		}
 		state.browser = connected
 	}
+	state.lastOK = time.Now()

 	return state.browser, nil
 }
@@ -638,7 +656,6 @@ func (b *Browser) laneProfile(ctx context.Context, browser *rod.Browser) (browse
 		profile, ok := browserprofile.ProfileByID(forcedID)
 		if ok {
 			profile = applyRuntimeBrowserVersion(profile, browser)
-			profile = applyProfileLanguageHint(profile, region)
 			if overrideUA := strings.TrimSpace(b.UserAgent); overrideUA != "" {
 				profile.UserAgent = overrideUA
 			}
@@ -648,9 +665,8 @@ func (b *Browser) laneProfile(ctx context.Context, browser *rod.Browser) (browse

 	if laneKey := proxyLaneKeyFromContext(ctx); !laneKey.Empty() && b.ProxyLaneStore != nil {
 		profile := b.ProxyLaneStore.Profile(laneKey, func() browserprofile.Profile {
-			selected := browserprofile.SelectProfileForSession(engine, region, laneKey.SessionID)
+			selected := browserprofile.SelectProfileForSessionHeadless(engine, region, laneKey.SessionID, b.IsHeadless)
 			selected = applyRuntimeBrowserVersion(selected, browser)
-			selected = applyProfileLanguageHint(selected, region)
 			if overrideUA := strings.TrimSpace(b.UserAgent); overrideUA != "" {
 				selected.UserAgent = overrideUA
 			}
@@ -676,9 +692,8 @@ func (b *Browser) laneProfile(ctx context.Context, browser *rod.Browser) (browse
 	// RWMutex-guarded catalog, and applyRuntimeBrowserVersion makes a CDP
 	// round-trip (browser.Version). Holding state.mu over network I/O would
 	// serialize all concurrent Navigate calls.
-	profile := browserprofile.SelectProfileForSession(engine, region, laneKey)
+	profile := browserprofile.SelectProfileForSessionHeadless(engine, region, laneKey, b.IsHeadless)
 	profile = applyRuntimeBrowserVersion(profile, browser)
-	profile = applyProfileLanguageHint(profile, region)
 	if overrideUA := strings.TrimSpace(b.UserAgent); overrideUA != "" {
 		profile.UserAgent = overrideUA
 	}
@@ -697,17 +712,46 @@ func (b *Browser) laneProfile(ctx context.Context, browser *rod.Browser) (browse
 	return profile, laneKey
 }

+// removeChromeBrand drops the "Google Chrome" client-hint brand from a profile,
+// used when the launched binary is Chromium rather than Google Chrome.
+func removeChromeBrand(profile browserprofile.Profile) browserprofile.Profile {
+	profile.UACHBrands = removeBrand(profile.UACHBrands, "google chrome")
+	profile.UACHFullVerList = removeBrand(profile.UACHFullVerList, "google chrome")
+	return profile
+}
+
+func removeBrand(values []browserprofile.BrandVersion, unwanted string) []browserprofile.BrandVersion {
+	out := make([]browserprofile.BrandVersion, 0, len(values))
+	for _, value := range values {
+		if strings.EqualFold(strings.TrimSpace(value.Brand), unwanted) {
+			continue
+		}
+		out = append(out, value)
+	}
+	return out
+}
+
 func applyRuntimeBrowserVersion(profile browserprofile.Profile, browser *rod.Browser) browserprofile.Profile {
 	fullVersion := ""
+	product := ""
 	if browser != nil {
 		version, err := browser.Version()
 		if err == nil && version != nil {
+			product = strings.TrimSpace(version.Product)
 			fullVersion = extractChromeVersion(version.UserAgent)
 			if fullVersion == "" {
-				fullVersion = extractChromeVersion(version.Product)
+				fullVersion = extractChromeVersion(product)
 			}
 		}
 	}
+
+	// version.Product tells us the real binary: "Chrome/..." for genuine Chrome,
+	// "Chromium/..." or "HeadlessChrome/..." otherwise. On non-Chrome builds we
+	// drop the "Google Chrome" UA-CH brand so the hints match the binary.
+	return applyRuntimeBrowserVersionValues(profile, product, fullVersion)
+}
+
+func applyRuntimeBrowserVersionValues(profile browserprofile.Profile, product, fullVersion string) browserprofile.Profile {
 	if fullVersion == "" {
 		fullVersion = extractChromeVersion(profile.UserAgent)
 	}
@@ -720,12 +764,39 @@ func applyRuntimeBrowserVersion(profile browserprofile.Profile, browser *rod.Bro
 		return profile
 	}

-	profile.UserAgent = replaceChromeUserAgentVersion(profile.UserAgent, major+".0.0.0")
-	profile.UACHBrands = patchBrandVersions(profile.UACHBrands, major, false)
-	profile.UACHFullVerList = patchBrandVersions(profile.UACHFullVerList, fullVersion, true)
+	if template := strings.TrimSpace(profile.UserAgentTemplate); template != "" {
+		profile.UserAgent = strings.ReplaceAll(template, "{chrome_major}", major)
+	} else {
+		profile.UserAgent = replaceChromeUserAgentVersion(profile.UserAgent, major+".0.0.0")
+	}
+	if len(profile.UACHBrands) == 0 {
+		profile.UACHBrands = runtimeUACHBrands(major, false)
+	} else {
+		profile.UACHBrands = patchBrandVersions(profile.UACHBrands, major, false)
+	}
+	if len(profile.UACHFullVerList) == 0 {
+		profile.UACHFullVerList = runtimeUACHBrands(fullVersion, true)
+	} else {
+		profile.UACHFullVerList = patchBrandVersions(profile.UACHFullVerList, fullVersion, true)
+	}
+	if product != "" && !strings.HasPrefix(product, "Chrome/") {
+		profile = removeChromeBrand(profile)
+	}
 	return profile
 }

+func runtimeUACHBrands(version string, full bool) []browserprofile.BrandVersion {
+	notABrandVersion := "24"
+	if full {
+		notABrandVersion = "24.0.0.0"
+	}
+	return []browserprofile.BrandVersion{
+		{Brand: "Not_A Brand", Version: notABrandVersion},
+		{Brand: "Chromium", Version: version},
+		{Brand: "Google Chrome", Version: version},
+	}
+}
+
 func extractChromeVersion(value string) string {
 	matches := chromeVersionPattern.FindStringSubmatch(strings.TrimSpace(value))
 	if len(matches) < 2 {
@@ -793,9 +864,6 @@ type profileDisplayMetrics struct {
 	ViewportHeight int
 	ScreenWidth    int
 	ScreenHeight   int
-	AvailWidth     int
-	AvailHeight    int
-	AvailTop       int
 	OuterWidth     int
 	OuterHeight    int
 	PositionX      int
@@ -834,9 +902,6 @@ func profileDisplayMetricsFor(profile browserprofile.Profile) profileDisplayMetr
 		ViewportHeight: viewportHeight,
 		ScreenWidth:    screenWidth,
 		ScreenHeight:   screenHeight,
-		AvailWidth:     screenWidth,
-		AvailHeight:    availHeight,
-		AvailTop:       availTop,
 		OuterWidth:     screenWidth,
 		OuterHeight:    availHeight,
 		PositionX:      0,
@@ -882,12 +947,12 @@ func applyProfileLanguageHint(profile browserprofile.Profile, langCode string) b
 	return profile
 }

-func applyProfile(page *rod.Page, profile browserprofile.Profile, minimal bool) error {
+func applyProfile(page *rod.Page, profile browserprofile.Profile, headless bool) error {
 	if page == nil {
 		return fmt.Errorf("page is nil")
 	}

-	navigatorLangs := profileNavigatorLanguages(profile)
+	navigatorLangs := profileNavigatorLanguagesForRuntime(profile, runtime.GOOS, headless)
 	acceptLanguage := strings.TrimSpace(profile.AcceptLanguage)
 	if acceptLanguage == "" {
 		acceptLanguage = navigatorLangs[0]
@@ -898,6 +963,18 @@ func applyProfile(page *rod.Page, profile browserprofile.Profile, minimal bool)
 	}

 	metrics := profileDisplayMetricsFor(profile)
+	// setWindowBounds can close the target under headless, and there's no real
+	// window anyway - setDeviceMetricsOverride below covers those dimensions.
+	if !headless {
+		if err := page.SetWindow(&proto.BrowserBounds{
+			Left:   &metrics.PositionX,
+			Top:    &metrics.PositionY,
+			Width:  &metrics.OuterWidth,
+			Height: &metrics.OuterHeight,
+		}); err != nil {
+			logrus.WithError(err).Debug("set browser window unsupported")
+		}
+	}

 	metadata := &proto.EmulationUserAgentMetadata{
 		Brands:          toProtoBrandVersions(profile.UACHBrands),
@@ -909,9 +986,11 @@ func applyProfile(page *rod.Page, profile browserprofile.Profile, minimal bool)
 		Mobile:          profile.Mobile,
 	}

+	// This only seeds navigator.languages (Chrome strips q-values), so it's the
+	// plain tag list. The wire header with q-weights is set below.
 	if err := (proto.NetworkSetUserAgentOverride{
 		UserAgent:         strings.TrimSpace(profile.UserAgent),
-		AcceptLanguage:    acceptLanguage,
+		AcceptLanguage:    strings.Join(navigatorLangs, ","),
 		Platform:          navigatorPlatformForProfile(profile),
 		UserAgentMetadata: metadata,
 	}).Call(page); err != nil {
@@ -943,6 +1022,14 @@ func applyProfile(page *rod.Page, profile browserprofile.Profile, minimal bool)
 		return fmt.Errorf("set device metrics failed: %w", err)
 	}

+	if err := (proto.EmulationSetEmulatedMedia{Features: []*proto.EmulationMediaFeature{
+		{Name: "prefers-reduced-motion", Value: "no-preference"},
+		{Name: "prefers-color-scheme", Value: "light"},
+		{Name: "forced-colors", Value: "none"},
+	}}).Call(page); err != nil {
+		return fmt.Errorf("set emulated media failed: %w", err)
+	}
+
 	if err := (proto.NetworkSetExtraHTTPHeaders{
 		Headers: proto.NetworkHeaders{
 			"Accept-Language": gson.New(acceptLanguage),
@@ -951,14 +1038,6 @@ func applyProfile(page *rod.Page, profile browserprofile.Profile, minimal bool)
 		return fmt.Errorf("set extra headers failed: %w", err)
 	}

-	if minimal {
-		return nil
-	}
-
-	if err := evalPatchScript(page, profile, navigatorLangs, metrics); err != nil {
-		return err
-	}
-
 	return nil
 }

@@ -1008,14 +1087,7 @@ type networkUsageWatcher struct {
 	done   chan struct{}
 }

-type workerPatchWatcher struct {
-	cancel context.CancelFunc
-	done   chan struct{}
-	page   *rod.Page
-}
-
 var pageNetworkUsageWatchers sync.Map
-var pageWorkerPatchWatchers sync.Map

 func startMainDocumentStatusWatcher(ctx context.Context, page *rod.Page) *mainDocumentStatusWatcher {
 	watchCtx, cancel := context.WithCancel(EnsureContext(ctx))
@@ -1083,95 +1155,6 @@ func startNetworkUsageWatcher(ctx context.Context, page *rod.Page) *networkUsage
 	return watcher
 }

-func startWorkerPatchWatcher(ctx context.Context, page *rod.Page, script string) (*workerPatchWatcher, error) {
-	if page == nil || strings.TrimSpace(script) == "" {
-		return nil, nil
-	}
-
-	watchCtx, cancel := context.WithCancel(EnsureContext(ctx))
-	watcher := &workerPatchWatcher{
-		cancel: cancel,
-		done:   make(chan struct{}),
-		page:   page,
-	}
-
-	scopedPage := page.Context(watchCtx)
-	started := make(chan struct{})
-	go func() {
-		defer close(watcher.done)
-		wait := scopedPage.EachEvent(func(e *proto.TargetAttachedToTarget) bool {
-			if e == nil || e.SessionID == "" {
-				return false
-			}
-			go injectWorkerPatch(watchCtx, scopedPage.Browser(), e, script)
-			return false
-		})
-		close(started)
-		wait()
-	}()
-
-	<-started
-	if err := (proto.TargetSetAutoAttach{
-		AutoAttach:             true,
-		WaitForDebuggerOnStart: true,
-		Flatten:                true,
-		Filter:                 workerTargetFilter(),
-	}).Call(scopedPage); err != nil {
-		cancel()
-		<-watcher.done
-		return nil, fmt.Errorf("enable worker auto-attach: %w", err)
-	}
-
-	return watcher, nil
-}
-
-func workerTargetFilter() proto.TargetTargetFilter {
-	return proto.TargetTargetFilter{
-		{Type: "worker"},
-		{Type: string(proto.TargetTargetInfoTypeSharedWorker)},
-		{Type: string(proto.TargetTargetInfoTypeServiceWorker)},
-	}
-}
-
-func injectWorkerPatch(ctx context.Context, browser *rod.Browser, e *proto.TargetAttachedToTarget, script string) {
-	if browser == nil || e == nil || e.SessionID == "" {
-		return
-	}
-
-	injectCtx, cancel := context.WithTimeout(EnsureContext(ctx), 3*time.Second)
-	defer cancel()
-
-	if isPatchableWorkerTarget(e.TargetInfo) {
-		eval := proto.RuntimeEvaluate{
-			Expression:                  script,
-			Silent:                      true,
-			AllowUnsafeEvalBlockedByCSP: true,
-		}
-		if _, err := browser.Call(injectCtx, string(e.SessionID), eval.ProtoReq(), eval); err != nil && !errors.Is(err, context.Canceled) {
-			logrus.WithError(err).Debug("Worker profile patch failed")
-		}
-	}
-
-	if e.WaitingForDebugger {
-		run := proto.RuntimeRunIfWaitingForDebugger{}
-		if _, err := browser.Call(injectCtx, string(e.SessionID), run.ProtoReq(), run); err != nil && !errors.Is(err, context.Canceled) {
-			logrus.WithError(err).Debug("Resume worker after profile patch failed")
-		}
-	}
-}
-
-func isPatchableWorkerTarget(info *proto.TargetTargetInfo) bool {
-	if info == nil {
-		return true
-	}
-	switch string(info.Type) {
-	case "worker", string(proto.TargetTargetInfoTypeSharedWorker), string(proto.TargetTargetInfoTypeServiceWorker):
-		return true
-	default:
-		return false
-	}
-}
-
 func rememberNetworkUsageWatcher(page *rod.Page, watcher *networkUsageWatcher) {
 	if page == nil || watcher == nil {
 		return
@@ -1179,13 +1162,6 @@ func rememberNetworkUsageWatcher(page *rod.Page, watcher *networkUsageWatcher) {
 	pageNetworkUsageWatchers.Store(page, watcher)
 }

-func rememberWorkerPatchWatcher(page *rod.Page, watcher *workerPatchWatcher) {
-	if page == nil || watcher == nil {
-		return
-	}
-	pageWorkerPatchWatchers.Store(page, watcher)
-}
-
 func stopNetworkUsageWatcher(page *rod.Page) {
 	if page == nil {
 		return
@@ -1199,19 +1175,6 @@ func stopNetworkUsageWatcher(page *rod.Page) {
 	}
 }

-func stopWorkerPatchWatcher(page *rod.Page) {
-	if page == nil {
-		return
-	}
-	raw, ok := pageWorkerPatchWatchers.LoadAndDelete(page)
-	if !ok {
-		return
-	}
-	if watcher, ok := raw.(*workerPatchWatcher); ok {
-		watcher.Stop()
-	}
-}
-
 func (w *networkUsageWatcher) Stop() {
 	if w == nil {
 		return
@@ -1223,27 +1186,6 @@ func (w *networkUsageWatcher) Stop() {
 	}
 }

-func (w *workerPatchWatcher) Stop() {
-	if w == nil {
-		return
-	}
-	if w.page != nil {
-		disableCtx, cancel := context.WithTimeout(context.Background(), time.Second)
-		_ = (proto.TargetSetAutoAttach{
-			AutoAttach:             false,
-			WaitForDebuggerOnStart: false,
-			Flatten:                true,
-			Filter:                 workerTargetFilter(),
-		}).Call(w.page.Context(disableCtx))
-		cancel()
-	}
-	w.cancel()
-	select {
-	case <-w.done:
-	case <-time.After(100 * time.Millisecond):
-	}
-}
-
 func (w *mainDocumentStatusWatcher) Status() int {
 	if w == nil {
 		return 0
@@ -1266,21 +1208,22 @@ func classifyMainDocumentStatus(status int) error {

 func profileNavigatorLanguages(profile browserprofile.Profile) []string {
 	langs := make([]string, 0, len(profile.NavigatorLangs))
+	seen := make(map[string]struct{}, len(profile.NavigatorLangs))
 	for _, language := range profile.NavigatorLangs {
 		trimmed := strings.TrimSpace(language)
 		if trimmed == "" {
 			continue
 		}
+		if _, ok := seen[trimmed]; ok {
+			continue
+		}
+		seen[trimmed] = struct{}{}
 		langs = append(langs, trimmed)
 	}
-	if len(langs) > 0 {
-		return langs
-	}

 	acceptLanguage := strings.TrimSpace(profile.AcceptLanguage)
 	if acceptLanguage != "" {
 		parts := strings.Split(acceptLanguage, ",")
-		langs = make([]string, 0, len(parts))
 		for _, part := range parts {
 			part = strings.TrimSpace(part)
 			if part == "" {
@@ -1290,12 +1233,16 @@ func profileNavigatorLanguages(profile browserprofile.Profile) []string {
 				part = strings.TrimSpace(part[:idx])
 			}
 			if part != "" {
+				if _, ok := seen[part]; ok {
+					continue
+				}
+				seen[part] = struct{}{}
 				langs = append(langs, part)
 			}
 		}
-		if len(langs) > 0 {
-			return langs
-		}
+	}
+	if len(langs) > 0 {
+		return langs
 	}

 	if locale := strings.TrimSpace(profile.Locale); locale != "" {
@@ -1304,6 +1251,15 @@ func profileNavigatorLanguages(profile browserprofile.Profile) []string {
 	return []string{"en-US"}
 }

+func profileNavigatorLanguagesForRuntime(profile browserprofile.Profile, goos string, headless bool) []string {
+	langs := profileNavigatorLanguages(profile)
+	// Linux headless-shell keeps workers at the process locale only.
+	if headless && goos == "linux" {
+		return langs[:1]
+	}
+	return langs
+}
+
 func toProtoBrandVersions(values []browserprofile.BrandVersion) []*proto.EmulationUserAgentBrandVersion {
 	out := make([]*proto.EmulationUserAgentBrandVersion, 0, len(values))
 	for _, value := range values {
@@ -1320,60 +1276,6 @@ func toProtoBrandVersions(values []browserprofile.BrandVersion) []*proto.Emulati
 	return out
 }

-func buildProfilePatchScript(profile browserprofile.Profile, langs []string, metrics profileDisplayMetrics) (string, error) {
-	langsJSON, err := json.Marshal(langs)
-	if err != nil {
-		return "", fmt.Errorf("marshal navigator languages: %w", err)
-	}
-
-	webGLVendor := strings.TrimSpace(profile.WebGLVendor)
-	if webGLVendor == "" {
-		webGLVendor = "Intel Inc."
-	}
-	webGLRenderer := strings.TrimSpace(profile.WebGLRenderer)
-	if webGLRenderer == "" {
-		webGLRenderer = "Intel Iris OpenGL Engine"
-	}
-
-	webGLVendorJSON, err := json.Marshal(webGLVendor)
-	if err != nil {
-		return "", fmt.Errorf("marshal webgl vendor: %w", err)
-	}
-	webGLRendererJSON, err := json.Marshal(webGLRenderer)
-	if err != nil {
-		return "", fmt.Errorf("marshal webgl renderer: %w", err)
-	}
-
-	return fmt.Sprintf("(() => {\nconst __langs = %s;\nconst __w = %d;\nconst __h = %d;\nconst __screenW = %d;\nconst __screenH = %d;\nconst __availW = %d;\nconst __availH = %d;\nconst __availTop = %d;\nconst __outerW = %d;\nconst __outerH = %d;\nconst __webglVendor = %s;\nconst __webglRenderer = %s;\n%s\n})();",
-		string(langsJSON),
-		metrics.ViewportWidth,
-		metrics.ViewportHeight,
-		metrics.ScreenWidth,
-		metrics.ScreenHeight,
-		metrics.AvailWidth,
-		metrics.AvailHeight,
-		metrics.AvailTop,
-		metrics.OuterWidth,
-		metrics.OuterHeight,
-		string(webGLVendorJSON),
-		string(webGLRendererJSON),
-		string(browserprofile.PatchJS),
-	), nil
-}
-
-func evalPatchScript(page *rod.Page, profile browserprofile.Profile, langs []string, metrics profileDisplayMetrics) error {
-	script, err := buildProfilePatchScript(profile, langs, metrics)
-	if err != nil {
-		return err
-	}
-
-	_, err = page.EvalOnNewDocument(script)
-	if err != nil {
-		return fmt.Errorf("eval patch script: %w", err)
-	}
-	return nil
-}
-
 // Navigate connects to Chromium, creates a page, applies a coherent profile and
 // proxy auth, then navigates to URL. It returns an initialized page ready for
 // selector queries, or an error when browser setup/navigation fails.
@@ -1422,7 +1324,6 @@ func (b *Browser) Navigate(ctx context.Context, URL string) (*rod.Page, error) {
 	// context first causes Chrome to kill the page target before our Close call,
 	// producing a spurious "target closed" error on the page.Close() that follows.
 	closeOnErr := func() {
-		stopWorkerPatchWatcher(page)
 		stopNetworkUsageWatcher(page)
 		if cerr := page.Close(); cerr != nil && !isBrowserClosedError(cerr) {
 			WithRequest(ctx).WithError(cerr).Debug("Close page after navigate error failed")
@@ -1431,15 +1332,22 @@ func (b *Browser) Navigate(ctx context.Context, URL string) (*rod.Page, error) {
 			WithRequest(ctx).WithError(derr).Debug("Dispose browser context after navigate error failed")
 		}
 	}
+	// LeavePageOpen keeps the Chrome tab open for manual inspection, but the
+	// background watchers attached to it must still stop or they leak goroutines.
+	stopWatchersOnErr := func() {
+		if b.LeavePageOpen {
+			stopNetworkUsageWatcher(page)
+		} else {
+			closeOnErr()
+		}
+	}

 	profile, laneKey := b.laneProfile(ctx, browser)
 	SetBrowserProfileID(ctx, profile.ID)
-	minimalProfile := minimalBrowserProfileFromContext(ctx)
 	WithRequest(ctx).WithFields(logrus.Fields{
-		"lane_id":         laneKey,
-		"minimal_profile": minimalProfile,
+		"lane_id": laneKey,
 	}).Info("Browser profile selected")
-	if err := applyProfile(page, profile, minimalProfile); err != nil {
+	if err := applyProfile(page, profile, b.IsHeadless); err != nil {
 		closeOnErr()
 		return nil, fmt.Errorf("apply profile %s (%s) failed: %w", profile.ID, laneKey, err)
 	}
@@ -1449,20 +1357,6 @@ func (b *Browser) Navigate(ctx context.Context, URL string) (*rod.Page, error) {
 	}

 	page = page.Context(ctx)
-	if !minimalProfile {
-		metrics := profileDisplayMetricsFor(profile)
-		patchScript, err := buildProfilePatchScript(profile, profileNavigatorLanguages(profile), metrics)
-		if err != nil {
-			closeOnErr()
-			return nil, err
-		}
-		workerPatchWatcher, err := startWorkerPatchWatcher(ctx, page, patchScript)
-		if err != nil {
-			closeOnErr()
-			return nil, err
-		}
-		rememberWorkerPatchWatcher(page, workerPatchWatcher)
-	}
 	if err := b.configureRequestBlocking(ctx, page); err != nil {
 		closeOnErr()
 		return nil, fmt.Errorf("configure request blocking failed: %w", err)
@@ -1478,7 +1372,7 @@ func (b *Browser) Navigate(ctx context.Context, URL string) (*rod.Page, error) {
 	timedPage := page.Timeout(b.Timeout)

 	if err := timedPage.Navigate(URL); err != nil {
-		closeOnErr()
+		stopWatchersOnErr()
 		return nil, classifyProxyNetworkError(err)
 	}

@@ -1503,20 +1397,24 @@ func (b *Browser) Navigate(ctx context.Context, URL string) (*rod.Page, error) {
 		wait()
 	}

-	// WaitStable internally waits for page load too, so cap it separately.
-	// Selector parsing still decides whether a partially loaded page is usable.
-	stableWaitTimeout := minPositiveDuration(b.Timeout, b.WaitLoadTime+time.Second)
-	if err := page.Timeout(stableWaitTimeout).WaitStable(800 * time.Millisecond); err != nil {
-		WithRequest(ctx).WithError(err).Debug("WaitStable returned early; continuing")
-	}
 	if err := classifyMainDocumentStatus(statusWatcher.Status()); err != nil {
-		closeOnErr()
+		stopWatchersOnErr()
 		return nil, err
 	}
 	b.saveLaneCookies(ctx, page, URL)
+	b.markConnectionOK()
 	return page, nil
 }

+// markConnectionOK records a successful CDP round trip so the next
+// ensureConnectedBrowser can skip its health ping (see healthPingSkipWindow).
+func (b *Browser) markConnectionOK() {
+	state := b.connectionState()
+	state.mu.Lock()
+	state.lastOK = time.Now()
+	state.mu.Unlock()
+}
+
 // Close closes the active browser connection.
 func (b *Browser) Close() error {
 	if b == nil || b.browserAddr == "" {
@@ -1574,7 +1472,6 @@ func ClosePageWithTimeout(ctx context.Context, page *rod.Page, timeout time.Dura
 	if page == nil {
 		return nil
 	}
-	stopWorkerPatchWatcher(page)
 	stopNetworkUsageWatcher(page)
 	if timeout <= 0 {
 		timeout = time.Second
--- a/core/browser/patch.js
+++ b/core/browser/patch.js
@@ -1,273 +0,0 @@
-// Stealth patches injected via EvalOnNewDocument.
-// Arguments are injected as a leading const block by the Go caller:
-//   const __langs = [...];          // navigator_langs from profile
-//   const __w = 1920;               // viewport/content width
-//   const __h = 955;                // viewport/content height
-//   const __screenW = 1920;         // screen.width
-//   const __screenH = 1080;         // screen.height
-//   const __availW = 1920;          // screen.availWidth
-//   const __availH = 1040;          // screen.availHeight
-//   const __availTop = 0;           // screen.availTop
-//   const __outerW = 1920;          // window.outerWidth
-//   const __outerH = 1040;          // window.outerHeight
-//   const __webglVendor = "...";    // UNMASKED_VENDOR_WEBGL spoof
-//   const __webglRenderer = "...";  // UNMASKED_RENDERER_WEBGL spoof
-//
-// Scope: only patches that fix detectors with high ROI and low introspection
-// surface. Notably absent: Function.prototype.toString proxy, custom Worker
-// constructor, mass plugin/mimeType arrays, iframe contentWindow patching.
-// Those triggered server-side detection on Google in commit 612e0dc.
-(() => {
-  'use strict';
-
-  const sealGetter = (target, prop, fn) => {
-    try {
-      Object.defineProperty(target, prop, {
-        get: fn,
-        set: undefined,
-        enumerable: true,
-        configurable: true,
-      });
-      Object.defineProperty(target, prop, { configurable: false });
-    } catch (_) {}
-  };
-
-  // --- navigator.webdriver ---
-  // headless Chrome sets this to true. Delete it so getter returns undefined.
-  // (We rely on `disable-blink-features=AutomationControlled` already turning
-  // this off via the launcher; this is belt-and-braces in case it leaks.)
-  try {
-    if (typeof navigator.webdriver !== 'undefined') {
-      delete Object.getPrototypeOf(navigator).webdriver;
-    }
-  } catch (_) {}
-
-  // --- navigator.language / navigator.languages ---
-  // CDP setUserAgentOverride sets the HTTP header but not these JS props.
-  const primary = __langs.length ? __langs[0] : 'en-US';
-  const patchLangs = (target) => {
-    if (!target) return;
-    sealGetter(target, 'language', () => primary);
-    sealGetter(target, 'languages', () => Object.freeze(__langs.slice()));
-  };
-  patchLangs(Object.getPrototypeOf(navigator));
-  if (typeof WorkerNavigator !== 'undefined') {
-    patchLangs(WorkerNavigator.prototype);
-  }
-
-  // --- screen dimensions ---
-  // EmulationSetDeviceMetricsOverride sets the CSS viewport but leaves
-  // window.screen.* at headless defaults.
-  const screenProto = typeof Screen !== 'undefined' ? Screen.prototype : null;
-  if (screenProto) {
-    sealGetter(screenProto, 'width', () => __screenW);
-    sealGetter(screenProto, 'height', () => __screenH);
-    sealGetter(screenProto, 'availWidth', () => __availW);
-    sealGetter(screenProto, 'availHeight', () => __availH);
-    sealGetter(screenProto, 'availLeft', () => 0);
-    sealGetter(screenProto, 'availTop', () => __availTop);
-    // Headless reports colorDepth/pixelDepth=24 already in most builds, but
-    // some checks see 0 in WSL/Docker. Lock to 24 which matches real Chrome.
-    sealGetter(screenProto, 'colorDepth', () => 24);
-    sealGetter(screenProto, 'pixelDepth', () => 24);
-  }
-  if (typeof window !== 'undefined') {
-    sealGetter(window, 'outerWidth', () => __outerW);
-    sealGetter(window, 'outerHeight', () => __outerH);
-  }
-
-  // --- WebGL vendor/renderer spoof ---
-  // Patches getParameter(37445=UNMASKED_VENDOR_WEBGL, 37446=UNMASKED_RENDERER_WEBGL)
-  // on WebGLRenderingContext.prototype and WebGL2RenderingContext.prototype.
-  // Only delegates to the original for all other params, so behavioral
-  // signals (extension list, real pixel rendering) still pass through.
-  const patchWebGLProto = (proto) => {
-    if (!proto) return;
-    const original = proto.getParameter;
-    if (typeof original !== 'function') return;
-    const replacement = function getParameter(parameter) {
-      if (parameter === 37445) return __webglVendor;
-      if (parameter === 37446) return __webglRenderer;
-      return original.apply(this, arguments);
-    };
-    try {
-      Object.defineProperty(proto, 'getParameter', {
-        value: replacement,
-        writable: true,
-        enumerable: false,
-        configurable: true,
-      });
-    } catch (_) {}
-  };
-  if (typeof WebGLRenderingContext !== 'undefined') {
-    patchWebGLProto(WebGLRenderingContext.prototype);
-  }
-  if (typeof WebGL2RenderingContext !== 'undefined') {
-    patchWebGLProto(WebGL2RenderingContext.prototype);
-  }
-
-  // --- navigator.plugins / mimeTypes ---
-  // Match real Chrome 136 exactly: 5 plugins (all "internal-pdf-viewer"), each
-  // exposing application/pdf + text/pdf. navigator.mimeTypes dedupes to 2.
-  // Each MimeType.enabledPlugin must back-reference the FIRST plugin that owns
-  // that type (Chrome's invariant). Plugin order is fixed.
-  try {
-    const PluginArrayProto = typeof PluginArray !== 'undefined' ? PluginArray.prototype : null;
-    const PluginProto = typeof Plugin !== 'undefined' ? Plugin.prototype : null;
-    const MimeTypeArrayProto = typeof MimeTypeArray !== 'undefined' ? MimeTypeArray.prototype : null;
-    const MimeTypeProto = typeof MimeType !== 'undefined' ? MimeType.prototype : null;
-
-    if (PluginArrayProto && PluginProto && MimeTypeArrayProto && MimeTypeProto) {
-      const pluginNames = [
-        'PDF Viewer',
-        'Chrome PDF Viewer',
-        'Chromium PDF Viewer',
-        'Microsoft Edge PDF Viewer',
-        'WebKit built-in PDF',
-      ];
-      const mimeSpecs = [
-        { type: 'application/pdf', suffixes: 'pdf', description: 'Portable Document Format' },
-        { type: 'text/pdf', suffixes: 'pdf', description: 'Portable Document Format' },
-      ];
-
-      // Two MimeType instances, each enabledPlugin points to the first plugin
-      // (PDF Viewer) per Chrome's invariant: navigator.mimeTypes[i].enabledPlugin
-      // === navigator.plugins[0] for every PDF mime.
-      const sharedMimes = mimeSpecs.map((spec) => {
-        const m = Object.create(MimeTypeProto);
-        Object.defineProperty(m, 'type', { value: spec.type, enumerable: true });
-        Object.defineProperty(m, 'suffixes', { value: spec.suffixes, enumerable: true });
-        Object.defineProperty(m, 'description', { value: spec.description, enumerable: true });
-        return m;
-      });
-
-      const plugins = pluginNames.map((name) => {
-        const p = Object.create(PluginProto);
-        Object.defineProperty(p, 'name', { value: name, enumerable: true });
-        Object.defineProperty(p, 'filename', { value: 'internal-pdf-viewer', enumerable: true });
-        Object.defineProperty(p, 'description', { value: 'Portable Document Format', enumerable: true });
-        Object.defineProperty(p, 'length', { value: sharedMimes.length, enumerable: true });
-        sharedMimes.forEach((m, i) => {
-          Object.defineProperty(p, String(i), { value: m, enumerable: true });
-          Object.defineProperty(p, m.type, { value: m });
-        });
-        return p;
-      });
-
-      // Set enabledPlugin AFTER plugins are constructed, pointing to plugins[0].
-      sharedMimes.forEach((m) => {
-        Object.defineProperty(m, 'enabledPlugin', { value: plugins[0], enumerable: true });
-      });
-
-      const pluginArr = Object.create(PluginArrayProto);
-      Object.defineProperty(pluginArr, 'length', { value: plugins.length, enumerable: true });
-      plugins.forEach((p, i) => {
-        Object.defineProperty(pluginArr, String(i), { value: p, enumerable: true });
-        Object.defineProperty(pluginArr, p.name, { value: p });
-      });
-
-      const mimeArr = Object.create(MimeTypeArrayProto);
-      Object.defineProperty(mimeArr, 'length', { value: sharedMimes.length, enumerable: true });
-      sharedMimes.forEach((m, i) => {
-        Object.defineProperty(mimeArr, String(i), { value: m, enumerable: true });
-        Object.defineProperty(mimeArr, m.type, { value: m });
-      });
-
-      sealGetter(Object.getPrototypeOf(navigator), 'plugins', () => pluginArr);
-      sealGetter(Object.getPrototypeOf(navigator), 'mimeTypes', () => mimeArr);
-    }
-  } catch (_) {}
-
-  // --- navigator.permissions.query notifications fix ---
-  // headless returns 'denied' for notifications when Notification.permission is
-  // 'default'. Real Chrome returns 'prompt' in that case. Sannysoft checks this
-  // mismatch (permissions_new / headchr_permissions).
-  try {
-    if (navigator.permissions && typeof navigator.permissions.query === 'function') {
-      const proto = Object.getPrototypeOf(navigator.permissions);
-      const desc = Object.getOwnPropertyDescriptor(proto, 'query');
-      if (desc && typeof desc.value === 'function') {
-        const original = desc.value;
-        const replacement = function query(parameters) {
-          if (parameters && parameters.name === 'notifications' &&
-              typeof Notification !== 'undefined' && Notification.permission === 'default') {
-            return Promise.resolve({ state: 'prompt', onchange: null });
-          }
-          return original.apply(this, arguments);
-        };
-        Object.defineProperty(proto, 'query', {
-          value: replacement,
-          writable: desc.writable,
-          enumerable: desc.enumerable,
-          configurable: desc.configurable,
-        });
-      }
-    }
-  } catch (_) {}
-
-  // --- getBoundingClientRect / getClientRects subpixel jitter ---
-  // Headless Chrome returns integer-valued rects; real Chrome returns subpixel
-  // floats due to CSS layout fractions. Fingerprinters hash rect tuples; even
-  // a sub-pixel offset breaks the canonical "headless rect" hash.
-  // Jitter is deterministic per-element (based on element identity) so the
-  // same element returns the same value across calls within the page lifetime.
-  try {
-    const rectProto = typeof DOMRect !== 'undefined' ? DOMRect.prototype : null;
-    const elProto = typeof Element !== 'undefined' ? Element.prototype : null;
-    if (rectProto && elProto) {
-      const wmJitter = new WeakMap();
-      const jitterFor = (el) => {
-        let j = wmJitter.get(el);
-        if (!j) {
-          // Tiny noise in [-0.05, +0.05) — well below visual threshold but
-          // changes hash output. Generated once per element.
-          j = {
-            x: (Math.random() - 0.5) * 0.1,
-            y: (Math.random() - 0.5) * 0.1,
-          };
-          wmJitter.set(el, j);
-        }
-        return j;
-      };
-      const origGBCR = elProto.getBoundingClientRect;
-      Object.defineProperty(elProto, 'getBoundingClientRect', {
-        value: function getBoundingClientRect() {
-          const r = origGBCR.apply(this, arguments);
-          const j = jitterFor(this);
-          // DOMRect is mutable; nudge x/y. width/height left intact so layout
-          // calculations don't drift.
-          try { r.x = r.x + j.x; r.y = r.y + j.y; } catch (_) {}
-          return r;
-        },
-        writable: true,
-        enumerable: false,
-        configurable: true,
-      });
-    }
-  } catch (_) {}
-
-  // --- window.chrome.runtime stub ---
-  // Real Chrome exposes window.chrome with a .runtime sub-object.
-  // headless leaves window.chrome empty, which sannysoft (chrome_new,
-  // headchr_chrome_obj) flags. A minimal runtime stub satisfies the check
-  // without touching method behavior.
-  try {
-    if (typeof window !== 'undefined') {
-      if (!window.chrome) {
-        Object.defineProperty(window, 'chrome', { value: {}, writable: true, configurable: true });
-      }
-      if (window.chrome && !window.chrome.runtime) {
-        Object.defineProperty(window.chrome, 'runtime', {
-          value: {
-            OnInstalledReason: { CHROME_UPDATE: 'chrome_update', INSTALL: 'install', UPDATE: 'update' },
-            OnRestartRequiredReason: { APP_UPDATE: 'app_update', OS_UPDATE: 'os_update', PERIODIC: 'periodic' },
-            PlatformOs: { ANDROID: 'android', CROS: 'cros', LINUX: 'linux', MAC: 'mac', WIN: 'win' },
-          },
-          writable: true,
-          enumerable: true,
-          configurable: true,
-        });
-      }
-    }
-  } catch (_) {}
-})();
--- a/core/browser/profile.go
+++ b/core/browser/profile.go
@@ -23,24 +23,23 @@ type Viewport struct {
 }

 type Profile struct {
-	ID              string         `json:"id"`
-	UserAgent       string         `json:"user_agent"`
-	UACHBrands      []BrandVersion `json:"uach_brands"`
-	UACHFullVerList []BrandVersion `json:"uach_full_version_list"`
-	Platform        string         `json:"platform"`
-	PlatformVersion string         `json:"platform_version"`
-	Architecture    string         `json:"architecture"`
-	Bitness         string         `json:"bitness"`
-	Mobile          bool           `json:"mobile"`
-	AcceptLanguage  string         `json:"accept_language"`
-	NavigatorLangs  []string       `json:"navigator_langs"`
-	Locale          string         `json:"locale"`
-	Timezone        string         `json:"timezone"`
-	Viewport        Viewport       `json:"viewport"`
-	WebGLVendor     string         `json:"webgl_vendor"`
-	WebGLRenderer   string         `json:"webgl_renderer"`
-	Tags            []string       `json:"tags"`
-	Weight          int            `json:"weight"`
+	ID                string         `json:"id"`
+	UserAgentTemplate string         `json:"user_agent_template"`
+	UserAgent         string         `json:"user_agent,omitempty"`
+	UACHBrands        []BrandVersion `json:"uach_brands,omitempty"`
+	UACHFullVerList   []BrandVersion `json:"uach_full_version_list,omitempty"`
+	Platform          string         `json:"platform"`
+	PlatformVersion   string         `json:"platform_version"`
+	Architecture      string         `json:"architecture"`
+	Bitness           string         `json:"bitness"`
+	Mobile            bool           `json:"mobile"`
+	AcceptLanguage    string         `json:"accept_language"`
+	NavigatorLangs    []string       `json:"navigator_langs"`
+	Locale            string         `json:"locale"`
+	Timezone          string         `json:"timezone"`
+	Viewport          Viewport       `json:"viewport"`
+	Tags              []string       `json:"tags"`
+	Weight            int            `json:"weight"`
 }

 type catalogConfig struct {
@@ -51,18 +50,13 @@ type catalogConfig struct {

 const (
 	ProfileChromeWinUS   = "chrome-win-uhd620"
-	ProfileChromeWinRU   = "chrome-win-ru"
 	ProfileChromeMacUS   = "chrome-macos-intel-iris"
 	ProfileChromeLinuxUS = "chrome-linux-mesa-uhd620"
-	ProfileChromeLinuxRU = "chrome-linux-ru"
 )

 //go:embed profiles.json
 var defaultProfilesJSON []byte

-//go:embed patch.js
-var PatchJS []byte
-
 var profileCatalogMu sync.RWMutex

 var catalog = map[string]Profile{}
@@ -110,8 +104,8 @@ func loadProfilesFromJSONBytes(data []byte) error {
 		if profile.ID == "" {
 			return fmt.Errorf("profiles[%d].id is empty", i)
 		}
-		if strings.TrimSpace(profile.UserAgent) == "" {
-			return fmt.Errorf("profiles[%d].user_agent is empty", i)
+		if strings.TrimSpace(profile.UserAgentTemplate) == "" && strings.TrimSpace(profile.UserAgent) == "" {
+			return fmt.Errorf("profiles[%d] has no user_agent_template or user_agent", i)
 		}
 		if _, exists := nextCatalog[profile.ID]; exists {
 			return fmt.Errorf("duplicate profile id %q", profile.ID)
@@ -201,6 +195,13 @@ func SelectProfile(engine string, region string) Profile {
 // Non-empty salt uses weighted selection seeded by FNV-1a hash of salt,
 // giving each session a stable but varied profile.
 func SelectProfileForSession(engine, region, salt string) Profile {
+	return SelectProfileForSessionHeadless(engine, region, salt, false)
+}
+
+// SelectProfileForSessionHeadless is SelectProfileForSession, but headless Linux
+// (the Docker deployment) renders WebGL via SwiftShader, so it only picks
+// swiftshader-tagged profiles. Runtimes with a real GPU exclude them.
+func SelectProfileForSessionHeadless(engine, region, salt string, headless bool) Profile {
 	engine = NormalizeEngine(engine)
 	region = NormalizeRegion(region)
 	if region == "" {
@@ -215,7 +216,7 @@ func SelectProfileForSession(engine, region, salt string) Profile {
 		return profileByID(profileID)
 	}

-	pool := eligibleProfiles(engine, region)
+	pool := eligibleProfiles(runtime.GOOS, headless)
 	return pickWeighted(pool, salt)
 }

@@ -224,10 +225,11 @@ type weightedProfile struct {
 	weight  int
 }

-// eligibleProfiles builds the weighted pool for (engine, region).
-// Linux profiles are preferred 4x on linux runtime; Windows 4x on windows; macOS 4x on darwin.
-// Profiles tagged "ru" are included only when region == "ru"; "ru"-tagged profiles are excluded otherwise.
-func eligibleProfiles(engine, region string) []weightedProfile {
+// eligibleProfiles builds the pool for the runtime platform. Headless Linux
+// keeps only swiftshader-tagged profiles (SwiftShader WebGL); a real GPU
+// excludes them. We no longer spoof WebGL, so the GPU sub-tag (nvidia/amd/mesa)
+// only steers selection - it matches reality on Docker, cosmetic on a headful box.
+func eligibleProfiles(goos string, headless bool) []weightedProfile {
 	profileCatalogMu.RLock()
 	snap := make([]Profile, 0, len(catalog))
 	for _, p := range catalog {
@@ -240,50 +242,52 @@ func eligibleProfiles(engine, region string) []weightedProfile {
 		return strings.Compare(a.ID, b.ID)
 	})

-	goos := runtime.GOOS
+	platformTag := runtimePlatformTag(goos)
+	wantSwiftShader := headless && goos == "linux"

 	var pool []weightedProfile
 	for _, p := range snap {
-		isRu := slices.Contains(p.Tags, "ru")
-		if region == "ru" && !isRu {
+		if platformTag != "" && !slices.Contains(p.Tags, platformTag) {
 			continue
 		}
-		if region != "ru" && isRu {
+		if slices.Contains(p.Tags, "swiftshader") != wantSwiftShader {
 			continue
 		}
-
 		w := p.Weight
 		if w <= 0 {
 			w = 1
 		}

-		platformLower := strings.ToLower(p.Platform)
-		switch goos {
-		case "linux":
-			if platformLower == "linux" {
-				w *= 4
-			}
-		case "windows":
-			if platformLower == "windows" {
-				w *= 4
-			}
-		case "darwin":
-			if platformLower == "macos" {
-				w *= 4
-			}
-		}
-
 		pool = append(pool, weightedProfile{profile: p, weight: w})
 	}

+	// Fall back to the platform pool if no swiftshader profile exists yet, so
+	// selection never returns empty.
+	if len(pool) == 0 && wantSwiftShader {
+		return eligibleProfiles(goos, false)
+	}
+
 	return pool
 }

+func runtimePlatformTag(goos string) string {
+	switch strings.ToLower(strings.TrimSpace(goos)) {
+	case "windows":
+		return "windows"
+	case "darwin":
+		return "macos"
+	case "linux":
+		return "linux"
+	default:
+		return ""
+	}
+}
+
 // pickWeighted selects a profile from pool using FNV-1a hash of salt modulo total weight.
 // Empty salt returns the first profile in the pool (deterministic for tests).
 func pickWeighted(pool []weightedProfile, salt string) Profile {
 	if len(pool) == 0 {
-		return profileByID(defaultProfileID("us"))
+		return profileByID(defaultProfileID())
 	}
 	if salt == "" {
 		return pool[0].profile
@@ -384,7 +388,7 @@ func profileByID(profileID string) Profile {
 	if profile, ok := catalog[profileID]; ok {
 		return profile
 	}
-	if fallback, ok := catalog[defaultProfileID("us")]; ok {
+	if fallback, ok := catalog[defaultProfileID()]; ok {
 		return fallback
 	}
 	for _, profile := range catalog {
@@ -393,24 +397,13 @@ func profileByID(profileID string) Profile {
 	return Profile{}
 }

-func defaultProfileID(region string) string {
-	region = NormalizeRegion(region)
-	if region == "" {
-		region = "us"
-	}
-
+func defaultProfileID() string {
 	switch runtime.GOOS {
 	case "windows":
-		if region == "ru" {
-			return ProfileChromeWinRU
-		}
 		return ProfileChromeWinUS
 	case "darwin":
 		return ProfileChromeMacUS
 	default:
-		if region == "ru" {
-			return ProfileChromeLinuxRU
-		}
 		return ProfileChromeLinuxUS
 	}
 }
--- a/core/browser/profile_coherence_test.go
+++ b/core/browser/profile_coherence_test.go
@@ -51,12 +51,12 @@ func TestProfileCoherence(t *testing.T) {
 		region string
 	}{
 		{
-			name:   "windows lane",
+			name:   "ru lane",
 			engine: "google",
 			region: "ru",
 		},
 		{
-			name:   "mac lane",
+			name:   "us lane",
 			engine: "bing",
 			region: "en-US",
 		},
@@ -84,7 +84,7 @@ func TestProfileCoherence(t *testing.T) {
 			}

 			expected := selectedProfileFromContext(t, ctx)
-			expected.UserAgent = expectedUserAgentForRuntime(expected.UserAgent, got.UserAgent)
+			expected.UserAgent = expectedUserAgentForRuntime(expectedProfileUserAgent(expected, got.UserAgent), got.UserAgent)
 			if got.UserAgent != expected.UserAgent {
 				t.Fatalf("navigator.userAgent mismatch:\nexpected: %s\nactual:   %s", expected.UserAgent, got.UserAgent)
 			}
@@ -106,11 +106,26 @@ func TestProfileCoherence(t *testing.T) {
 			if got.NavigatorLanguages[0] != expected.NavigatorLangs[0] {
 				t.Fatalf("navigator.languages[0] mismatch: expected %q got %q", expected.NavigatorLangs[0], got.NavigatorLanguages[0])
 			}
-			if got.WebdriverType != "undefined" {
-				t.Fatalf("navigator.webdriver expected undefined, got %q", got.WebdriverType)
+			// Real Chrome exposes navigator.webdriver as an inherited boolean
+			// that reads false; --disable-blink-features=AutomationControlled
+			// gives us that. Deleting the property (typeof undefined) is itself a
+			// bot tell, so we assert the genuine-browser shape instead.
+			if got.WebdriverType != "boolean" {
+				t.Fatalf("navigator.webdriver expected boolean, got %q", got.WebdriverType)
+			}
+			if got.WebdriverValue {
+				t.Fatal("navigator.webdriver should read false")
 			}
 			if got.WebdriverOwnPropPresent {
-				t.Fatal("navigator own property 'webdriver' should not be present")
+				t.Fatal("navigator.webdriver should be inherited, not an own property")
+			}
+			// hardwareConcurrency is left native (no override), so main and
+			// worker must simply agree on the machine's real core count.
+			if got.HardwareConcurrency <= 0 {
+				t.Fatal("navigator.hardwareConcurrency should be reported")
+			}
+			if got.WorkerHardwareConcurrency != got.HardwareConcurrency {
+				t.Fatalf("worker hardwareConcurrency mismatch: main %d worker %d", got.HardwareConcurrency, got.WorkerHardwareConcurrency)
 			}
 			if got.WorkerUserAgent != got.UserAgent {
 				t.Fatalf("worker userAgent mismatch: main %q worker %q", got.UserAgent, got.WorkerUserAgent)
@@ -127,11 +142,11 @@ func TestProfileCoherence(t *testing.T) {
 			if got.WorkerTimezone != got.Timezone {
 				t.Fatalf("worker timezone mismatch: main %q worker %q", got.Timezone, got.WorkerTimezone)
 			}
-			if got.WorkerWebGLVendor != expected.WebGLVendor {
-				t.Fatalf("worker WebGL vendor mismatch: expected %q got %q", expected.WebGLVendor, got.WorkerWebGLVendor)
+			if got.WorkerWebGLVendor != got.WebGLVendor {
+				t.Fatalf("worker WebGL vendor mismatch: main %q worker %q", got.WebGLVendor, got.WorkerWebGLVendor)
 			}
-			if got.WorkerWebGLRenderer != expected.WebGLRenderer {
-				t.Fatalf("worker WebGL renderer mismatch: expected %q got %q", expected.WebGLRenderer, got.WorkerWebGLRenderer)
+			if got.WorkerWebGLRenderer != got.WebGLRenderer {
+				t.Fatalf("worker WebGL renderer mismatch: main %q worker %q", got.WebGLRenderer, got.WorkerWebGLRenderer)
 			}
 			if got.InnerHeight >= got.OuterHeight {
 				t.Fatalf("innerHeight should be smaller than outerHeight, got inner=%d outer=%d", got.InnerHeight, got.OuterHeight)
@@ -139,32 +154,37 @@ func TestProfileCoherence(t *testing.T) {
 			if got.OuterHeight > got.ScreenAvailHeight {
 				t.Fatalf("outerHeight should fit in screen.availHeight, got outer=%d avail=%d", got.OuterHeight, got.ScreenAvailHeight)
 			}
-			if got.ScreenAvailHeight >= got.ScreenHeight {
-				t.Fatalf("screen.availHeight should be smaller than screen.height, got avail=%d screen=%d", got.ScreenAvailHeight, got.ScreenHeight)
+			if got.ScreenAvailHeight > got.ScreenHeight {
+				t.Fatalf("screen.availHeight should fit in screen.height, got avail=%d screen=%d", got.ScreenAvailHeight, got.ScreenHeight)
 			}
 		})
 	}
 }

 type profileSurface struct {
-	UserAgent               string   `json:"userAgent"`
-	Platform                string   `json:"platform"`
-	NavigatorPlatform       string   `json:"navigatorPlatform"`
-	NavigatorLanguages      []string `json:"navigatorLanguages"`
-	Timezone                string   `json:"timezone"`
-	Locale                  string   `json:"locale"`
-	WebdriverType           string   `json:"webdriverType"`
-	WebdriverOwnPropPresent bool     `json:"webdriverOwnPropPresent"`
-	WorkerUserAgent         string   `json:"workerUserAgent"`
-	WorkerPlatform          string   `json:"workerPlatform"`
-	WorkerNavigatorLangs    []string `json:"workerNavigatorLangs"`
-	WorkerTimezone          string   `json:"workerTimezone"`
-	WorkerWebGLVendor       string   `json:"workerWebGLVendor"`
-	WorkerWebGLRenderer     string   `json:"workerWebGLRenderer"`
-	InnerHeight             int      `json:"innerHeight"`
-	OuterHeight             int      `json:"outerHeight"`
-	ScreenHeight            int      `json:"screenHeight"`
-	ScreenAvailHeight       int      `json:"screenAvailHeight"`
+	UserAgent                 string   `json:"userAgent"`
+	Platform                  string   `json:"platform"`
+	NavigatorPlatform         string   `json:"navigatorPlatform"`
+	NavigatorLanguages        []string `json:"navigatorLanguages"`
+	Timezone                  string   `json:"timezone"`
+	Locale                    string   `json:"locale"`
+	WebdriverType             string   `json:"webdriverType"`
+	WebdriverValue            bool     `json:"webdriverValue"`
+	WebdriverOwnPropPresent   bool     `json:"webdriverOwnPropPresent"`
+	HardwareConcurrency       int      `json:"hardwareConcurrency"`
+	WorkerHardwareConcurrency int      `json:"workerHardwareConcurrency"`
+	WorkerUserAgent           string   `json:"workerUserAgent"`
+	WorkerPlatform            string   `json:"workerPlatform"`
+	WorkerNavigatorLangs      []string `json:"workerNavigatorLangs"`
+	WorkerTimezone            string   `json:"workerTimezone"`
+	WebGLVendor               string   `json:"webGLVendor"`
+	WebGLRenderer             string   `json:"webGLRenderer"`
+	WorkerWebGLVendor         string   `json:"workerWebGLVendor"`
+	WorkerWebGLRenderer       string   `json:"workerWebGLRenderer"`
+	InnerHeight               int      `json:"innerHeight"`
+	OuterHeight               int      `json:"outerHeight"`
+	ScreenHeight              int      `json:"screenHeight"`
+	ScreenAvailHeight         int      `json:"screenAvailHeight"`
 }

 func browserProfileSurface(page *rod.Page) (profileSurface, error) {
@@ -205,6 +225,25 @@ func selectedProfileFromContext(t *testing.T, ctx context.Context) browserprofil
 	return profile
 }

+// expectedProfileUserAgent resolves the profile's user agent, expanding the
+// {chrome_major} template with the runtime's major when the profile carries a
+// template rather than a literal UA.
+func expectedProfileUserAgent(profile browserprofile.Profile, runtimeUserAgent string) string {
+	if ua := strings.TrimSpace(profile.UserAgent); ua != "" {
+		return ua
+	}
+	template := strings.TrimSpace(profile.UserAgentTemplate)
+	if template == "" {
+		return ""
+	}
+	major := chromeToken(runtimeUserAgent)
+	major = strings.TrimPrefix(major, "Chrome/")
+	if idx := strings.IndexByte(major, '.'); idx >= 0 {
+		major = major[:idx]
+	}
+	return strings.ReplaceAll(template, "{chrome_major}", major)
+}
+
 func expectedUserAgentForRuntime(profileUserAgent, runtimeUserAgent string) string {
 	runtimeChrome := chromeToken(runtimeUserAgent)
 	if runtimeChrome == "" {
--- a/core/browser/profile_surface_test.js
+++ b/core/browser/profile_surface_test.js
@@ -1,4 +1,17 @@
 async () => {
+  const readWebGL = (canvas) => {
+    try {
+      const gl = canvas ? (canvas.getContext('webgl') || canvas.getContext('experimental-webgl') || canvas.getContext('webgl2')) : null;
+      const debugInfo = gl && gl.getExtension('WEBGL_debug_renderer_info');
+      return {
+        vendor: gl && debugInfo ? (gl.getParameter(debugInfo.UNMASKED_VENDOR_WEBGL) || '') : '',
+        renderer: gl && debugInfo ? (gl.getParameter(debugInfo.UNMASKED_RENDERER_WEBGL) || '') : '',
+      };
+    } catch (_) {
+      return { vendor: '', renderer: '' };
+    }
+  };
+  const webGL = readWebGL(document.createElement('canvas'));
  const workerData = await new Promise((resolve) => {
    try {
      const source = [
@@ -19,6 +32,7 @@ async () => {
        "platform: self.navigator.platform || '',",
        "navigatorLanguages: Array.from(self.navigator.languages || []),",
        "timezone: Intl.DateTimeFormat().resolvedOptions().timeZone || '',",
+        "hardwareConcurrency: self.navigator.hardwareConcurrency || 0,",
        "webGLVendor,",
        "webGLRenderer,",
        "});",
@@ -51,11 +65,16 @@ async () => {
    timezone: Intl.DateTimeFormat().resolvedOptions().timeZone || "",
    locale: Intl.DateTimeFormat().resolvedOptions().locale || "",
    webdriverType: typeof navigator.webdriver,
+    webdriverValue: navigator.webdriver === true,
    webdriverOwnPropPresent: Object.getOwnPropertyNames(navigator).includes("webdriver"),
+    hardwareConcurrency: navigator.hardwareConcurrency || 0,
+    workerHardwareConcurrency: workerData.hardwareConcurrency || 0,
    workerUserAgent: workerData.userAgent || "",
    workerPlatform: workerData.platform || "",
    workerNavigatorLangs: Array.from(workerData.navigatorLanguages || []),
    workerTimezone: workerData.timezone || "",
+    webGLVendor: webGL.vendor,
+    webGLRenderer: webGL.renderer,
    workerWebGLVendor: workerData.webGLVendor || "",
    workerWebGLRenderer: workerData.webGLRenderer || "",
    innerHeight: window.innerHeight || 0,
--- a/core/browser/profile_test.go
+++ b/core/browser/profile_test.go
@@ -69,14 +69,73 @@ func TestSelectProfileForSession(t *testing.T) {
 		}
 	})

-	t.Run("ru region returns ru-tagged profile", func(t *testing.T) {
+	t.Run("locale does not change hardware profile pool", func(t *testing.T) {
 		p := SelectProfileForSession("yandex", "ru", "some-session")
-		if !slices.Contains(p.Tags, "ru") {
-			t.Fatalf("expected ru-tagged profile, got ID=%q tags=%v", p.ID, p.Tags)
+		if slices.Contains(p.Tags, "ru") {
+			t.Fatalf("expected locale-neutral profile, got ID=%q tags=%v", p.ID, p.Tags)
 		}
 	})
 }

+func TestEligibleProfilesMatchRuntimePlatform(t *testing.T) {
+	tests := []struct {
+		goos string
+		tag  string
+	}{
+		{goos: "linux", tag: "linux"},
+		{goos: "windows", tag: "windows"},
+		{goos: "darwin", tag: "macos"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.goos, func(t *testing.T) {
+			pool := eligibleProfiles(tt.goos, false)
+			if len(pool) == 0 {
+				t.Fatal("expected eligible profiles")
+			}
+			for _, candidate := range pool {
+				if !slices.Contains(candidate.profile.Tags, tt.tag) {
+					t.Fatalf("profile %q does not match %s", candidate.profile.ID, tt.goos)
+				}
+			}
+		})
+	}
+}
+
+// Headless Linux (the Docker deployment) has no real GPU, so only
+// swiftshader-tagged profiles are eligible; headful Linux excludes them.
+func TestEligibleProfilesHeadlessLinuxUsesSwiftShader(t *testing.T) {
+	headless := eligibleProfiles("linux", true)
+	if len(headless) == 0 {
+		t.Fatal("expected a Linux SwiftShader profile")
+	}
+	for _, candidate := range headless {
+		if !slices.Contains(candidate.profile.Tags, "swiftshader") {
+			t.Fatalf("headless profile %q is not a SwiftShader profile", candidate.profile.ID)
+		}
+	}
+	if len(headless) < 2 {
+		t.Fatalf("expected multiple SwiftShader profiles so Docker is not a single fingerprint, got %d", len(headless))
+	}
+	timezones := make(map[string]struct{}, len(headless))
+	for _, candidate := range headless {
+		timezones[candidate.profile.Timezone] = struct{}{}
+	}
+	if len(timezones) < 2 {
+		t.Fatalf("expected distinct CDP-visible SwiftShader profiles, got timezones %v", timezones)
+	}
+
+	headful := eligibleProfiles("linux", false)
+	if len(headful) == 0 {
+		t.Fatal("expected headful Linux profiles")
+	}
+	for _, candidate := range headful {
+		if slices.Contains(candidate.profile.Tags, "swiftshader") {
+			t.Fatalf("headful profile %q should not be a SwiftShader profile", candidate.profile.ID)
+		}
+	}
+}
+
 func TestNormalizeRegion(t *testing.T) {
 	tests := []struct {
 		input string
--- a/core/browser/profiles.json
+++ b/core/browser/profiles.json
@@ -2,399 +2,96 @@
  "profiles": [
    {
      "id": "chrome-linux-mesa-uhd620",
-      "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Linux",
-      "platform_version": "6.1.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1920, "height": 1080},
-      "webgl_vendor": "Intel Inc.",
-      "webgl_renderer": "Mesa Intel(R) UHD Graphics 620 (KBL GT2)",
-      "tags": ["linux", "mesa", "integrated"],
-      "weight": 3
+      "user_agent_template": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "Linux", "platform_version": "6.1.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1920, "height": 1080}, "tags": ["linux", "mesa", "integrated"], "weight": 3
    },
    {
      "id": "chrome-linux-mesa-iris-xe",
-      "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Linux",
-      "platform_version": "6.1.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1920, "height": 1080},
-      "webgl_vendor": "Intel Inc.",
-      "webgl_renderer": "Mesa Intel(R) Graphics (RPL-S)",
-      "tags": ["linux", "mesa", "integrated"],
-      "weight": 2
+      "user_agent_template": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "Linux", "platform_version": "6.1.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1920, "height": 1080}, "tags": ["linux", "mesa", "integrated"], "weight": 2
    },
    {
      "id": "chrome-linux-nvidia",
-      "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Linux",
-      "platform_version": "6.1.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1920, "height": 1080},
-      "webgl_vendor": "Google Inc. (NVIDIA Corporation)",
-      "webgl_renderer": "ANGLE (NVIDIA Corporation, NVIDIA GeForce GTX 1660 SUPER/PCIe/SSE2, OpenGL 4.5.0 NVIDIA 535.86.05)",
-      "tags": ["linux", "nvidia"],
-      "weight": 2
+      "user_agent_template": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "Linux", "platform_version": "6.1.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1920, "height": 1080}, "tags": ["linux", "nvidia"], "weight": 2
    },
    {
      "id": "chrome-linux-amd",
-      "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Linux",
-      "platform_version": "6.1.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1920, "height": 1080},
-      "webgl_vendor": "Google Inc. (AMD)",
-      "webgl_renderer": "ANGLE (AMD, AMD Radeon RX 6600 (radeonsi, navi23, ACO, DRM 3.42.0, 5.15.0-91-generic), OpenGL 4.6 (Core Profile))",
-      "tags": ["linux", "amd"],
-      "weight": 1
+      "user_agent_template": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "Linux", "platform_version": "6.1.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1920, "height": 1080}, "tags": ["linux", "amd"], "weight": 1
    },
    {
      "id": "chrome-linux-swiftshader",
-      "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Linux",
-      "platform_version": "6.1.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1366, "height": 768},
-      "webgl_vendor": "Google Inc. (Google)",
-      "webgl_renderer": "ANGLE (Google, Vulkan 1.3.0 (SwiftShader Device (Subzero) (0x0000C0DE)), SwiftShader driver)",
-      "tags": ["linux", "swiftshader", "headless"],
-      "weight": 1
+      "user_agent_template": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "Linux", "platform_version": "6.1.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1920, "height": 1080}, "tags": ["linux", "swiftshader", "headless"], "weight": 1
    },
    {
-      "id": "chrome-linux-ru",
-      "user_agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Linux",
-      "platform_version": "6.0.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7",
-      "navigator_langs": ["ru-RU"],
-      "locale": "ru-RU",
-      "timezone": "Europe/Moscow",
-      "viewport": {"width": 1920, "height": 1080},
-      "webgl_vendor": "Intel Inc.",
-      "webgl_renderer": "Mesa Intel(R) UHD Graphics 620 (KBL GT2)",
-      "tags": ["linux", "ru"],
-      "weight": 1
+      "id": "chrome-linux-swiftshader-fhd",
+      "user_agent_template": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "Linux", "platform_version": "6.1.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/Chicago",
+      "viewport": {"width": 1920, "height": 1080}, "tags": ["linux", "swiftshader", "headless"], "weight": 1
    },
    {
      "id": "chrome-win-uhd620",
-      "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Windows",
-      "platform_version": "15.0.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1920, "height": 1080},
-      "webgl_vendor": "Intel Inc.",
-      "webgl_renderer": "ANGLE (Intel, Intel(R) UHD Graphics 620 Direct3D11 vs_5_0 ps_5_0, D3D11)",
-      "tags": ["windows"],
-      "weight": 2
+      "user_agent_template": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "Windows", "platform_version": "15.0.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1920, "height": 1080}, "tags": ["windows"], "weight": 2
    },
    {
      "id": "chrome-win-nvidia",
-      "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Windows",
-      "platform_version": "15.0.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1920, "height": 1080},
-      "webgl_vendor": "Google Inc. (NVIDIA)",
-      "webgl_renderer": "ANGLE (NVIDIA, NVIDIA GeForce RTX 3060 Direct3D11 vs_5_0 ps_5_0, D3D11)",
-      "tags": ["windows", "nvidia"],
-      "weight": 2
+      "user_agent_template": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "Windows", "platform_version": "15.0.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1920, "height": 1080}, "tags": ["windows", "nvidia"], "weight": 2
    },
    {
      "id": "chrome-win-iris-xe",
-      "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Windows",
-      "platform_version": "15.0.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1536, "height": 864},
-      "webgl_vendor": "Google Inc. (Intel)",
-      "webgl_renderer": "ANGLE (Intel, Intel(R) Iris(R) Xe Graphics Direct3D11 vs_5_0 ps_5_0, D3D11)",
-      "tags": ["windows", "integrated"],
-      "weight": 1
+      "user_agent_template": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "Windows", "platform_version": "15.0.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1920, "height": 1080}, "tags": ["windows", "integrated"], "weight": 1
    },
    {
      "id": "chrome-win-amd",
-      "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Windows",
-      "platform_version": "15.0.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1920, "height": 1080},
-      "webgl_vendor": "Google Inc. (AMD)",
-      "webgl_renderer": "ANGLE (AMD, AMD Radeon RX 6600 Direct3D11 vs_5_0 ps_5_0, D3D11)",
-      "tags": ["windows", "amd"],
-      "weight": 1
-    },
-    {
-      "id": "chrome-win-ru",
-      "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "Windows",
-      "platform_version": "15.0.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7",
-      "navigator_langs": ["ru-RU"],
-      "locale": "ru-RU",
-      "timezone": "Europe/Moscow",
-      "viewport": {"width": 1920, "height": 1080},
-      "webgl_vendor": "Intel Inc.",
-      "webgl_renderer": "ANGLE (Intel, Intel(R) UHD Graphics 620 Direct3D11 vs_5_0 ps_5_0, D3D11)",
-      "tags": ["windows", "ru"],
-      "weight": 1
+      "user_agent_template": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "Windows", "platform_version": "15.0.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1920, "height": 1080}, "tags": ["windows", "amd"], "weight": 1
    },
    {
      "id": "chrome-macos-m1",
-      "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "macOS",
-      "platform_version": "13.0.0",
-      "architecture": "arm",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1440, "height": 900},
-      "webgl_vendor": "Google Inc. (Apple)",
-      "webgl_renderer": "ANGLE (Apple, ANGLE Metal Renderer: Apple M1, Unspecified Version)",
-      "tags": ["macos", "apple-silicon"],
-      "weight": 2
+      "user_agent_template": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "macOS", "platform_version": "13.0.0", "architecture": "arm", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1440, "height": 900}, "tags": ["macos", "apple-silicon"], "weight": 2
    },
    {
      "id": "chrome-macos-m2",
-      "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "macOS",
-      "platform_version": "14.0.0",
-      "architecture": "arm",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1512, "height": 982},
-      "webgl_vendor": "Google Inc. (Apple)",
-      "webgl_renderer": "ANGLE (Apple, ANGLE Metal Renderer: Apple M2, Unspecified Version)",
-      "tags": ["macos", "apple-silicon"],
-      "weight": 2
+      "user_agent_template": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "macOS", "platform_version": "14.0.0", "architecture": "arm", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1512, "height": 982}, "tags": ["macos", "apple-silicon"], "weight": 2
    },
    {
      "id": "chrome-macos-intel-iris",
-      "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      "uach_brands": [
-        {"brand": "Not_A Brand", "version": "24"},
-        {"brand": "Chromium", "version": "136"},
-        {"brand": "Google Chrome", "version": "136"}
-      ],
-      "uach_full_version_list": [
-        {"brand": "Not_A Brand", "version": "24.0.0.0"},
-        {"brand": "Chromium", "version": "136.0.0.0"},
-        {"brand": "Google Chrome", "version": "136.0.0.0"}
-      ],
-      "platform": "macOS",
-      "platform_version": "14.0.0",
-      "architecture": "x86",
-      "bitness": "64",
-      "mobile": false,
-      "accept_language": "en-US,en;q=0.9",
-      "navigator_langs": ["en-US"],
-      "locale": "en-US",
-      "timezone": "America/New_York",
-      "viewport": {"width": 1680, "height": 1050},
-      "webgl_vendor": "Intel Inc.",
-      "webgl_renderer": "Intel Iris OpenGL Engine",
-      "tags": ["macos", "integrated"],
-      "weight": 1
+      "user_agent_template": "Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/{chrome_major}.0.0.0 Safari/537.36",
+      "platform": "macOS", "platform_version": "14.0.0", "architecture": "x86", "bitness": "64", "mobile": false,
+      "accept_language": "en-US,en;q=0.9", "navigator_langs": ["en-US"], "locale": "en-US", "timezone": "America/New_York",
+      "viewport": {"width": 1680, "height": 1050}, "tags": ["macos", "integrated"], "weight": 1
    }
  ],
  "lane_profile_ids": {},
-  "default_region_by_engine": {
-    "yandex": "ru"
-  }
+  "default_region_by_engine": {"yandex": "ru"}
 }
--- a/core/browser_unit_test.go
+++ b/core/browser_unit_test.go
@@ -61,6 +61,61 @@ func TestResolveBrowserBinaryPathRejectsInvalidExplicit(t *testing.T) {
 	}
 }

+func TestBrowserLaunchLanguageIsProcessStable(t *testing.T) {
+	tests := []struct {
+		name string
+		opts BrowserOpts
+		want string
+	}{
+		{name: "default locale", opts: BrowserOpts{}, want: "en-US"},
+		{name: "request hint does not change process locale", opts: BrowserOpts{LanguageCode: "de"}, want: "en-US"},
+		{name: "regional hint does not change process locale", opts: BrowserOpts{LanguageCode: "en-GB"}, want: "en-US"},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			if got := browserLaunchLanguage(tt.opts); got != tt.want {
+				t.Fatalf("browserLaunchLanguage() = %q, want %q", got, tt.want)
+			}
+		})
+	}
+}
+
+func TestProfileNavigatorLanguagesStripsHeaderWeights(t *testing.T) {
+	profile := browserprofile.Profile{
+		AcceptLanguage: "en-US,en;q=0.9",
+		NavigatorLangs: []string{"en-US"},
+	}
+
+	got := profileNavigatorLanguages(profile)
+	want := []string{"en-US", "en"}
+	if len(got) != len(want) {
+		t.Fatalf("profileNavigatorLanguages() = %v, want %v", got, want)
+	}
+	for i := range want {
+		if got[i] != want[i] {
+			t.Fatalf("profileNavigatorLanguages() = %v, want %v", got, want)
+		}
+	}
+}
+
+func TestProfileNavigatorLanguagesForRuntime(t *testing.T) {
+	profile := browserprofile.Profile{
+		AcceptLanguage: "en-US,en;q=0.9",
+		NavigatorLangs: []string{"en-US"},
+	}
+
+	linuxHeadless := profileNavigatorLanguagesForRuntime(profile, "linux", true)
+	if len(linuxHeadless) != 1 || linuxHeadless[0] != "en-US" {
+		t.Fatalf("linux headless languages = %v, want [en-US]", linuxHeadless)
+	}
+
+	windowsHeadless := profileNavigatorLanguagesForRuntime(profile, "windows", true)
+	if len(windowsHeadless) != 2 || windowsHeadless[0] != "en-US" || windowsHeadless[1] != "en" {
+		t.Fatalf("windows headless languages = %v, want [en-US en]", windowsHeadless)
+	}
+}
+
 func TestMinPositiveDuration(t *testing.T) {
 	tests := []struct {
 		name string
@@ -95,6 +150,35 @@ func TestApplyProfileLanguageHintRewritesTimezone(t *testing.T) {
 	}
 }

+func TestRemoveChromeBrand(t *testing.T) {
+	profile := browserprofile.Profile{
+		UACHBrands: []browserprofile.BrandVersion{
+			{Brand: "Not_A Brand", Version: "24"},
+			{Brand: "Chromium", Version: "136"},
+			{Brand: "Google Chrome", Version: "136"},
+		},
+		UACHFullVerList: []browserprofile.BrandVersion{
+			{Brand: "Chromium", Version: "136.0.0.0"},
+			{Brand: "Google Chrome", Version: "136.0.0.0"},
+		},
+	}
+
+	got := removeChromeBrand(profile)
+	for _, brand := range got.UACHBrands {
+		if brand.Brand == "Google Chrome" {
+			t.Fatal("expected Google Chrome brand to be removed from UACHBrands")
+		}
+	}
+	for _, brand := range got.UACHFullVerList {
+		if brand.Brand == "Google Chrome" {
+			t.Fatal("expected Google Chrome brand to be removed from UACHFullVerList")
+		}
+	}
+	if len(got.UACHBrands) != 2 || len(got.UACHFullVerList) != 1 {
+		t.Fatalf("unexpected brand counts: brands=%d fullList=%d", len(got.UACHBrands), len(got.UACHFullVerList))
+	}
+}
+
 func TestApplyProfileLanguageHint(t *testing.T) {
 	base := browserprofile.Profile{
 		AcceptLanguage: "en-US,en;q=0.9",
--- a/core/common.go
+++ b/core/common.go
@@ -15,6 +15,14 @@ import (
 	"golang.org/x/time/rate"
 )

+// Extraction depth bounds for the unified extract=N query param. The default
+// is 1 (extract=true == extract=1 == "extract one result"); callers raise it up
+// to maxExtractTop. These mirror the CLI's --extract flag limits.
+const (
+	defaultExtractTop = 1
+	maxExtractTop     = 5
+)
+
 // ErrCaptcha is returned when the engine detects a captcha challenge page.
 // This error is treated as non-retryable by resilient search policies.
 var ErrCaptcha = errors.New("captcha detected")
@@ -50,7 +58,7 @@ var ErrEmptyResult = errors.New("empty_result")
 // ErrBlocked is returned when the search engine blocks the browser request.
 var ErrBlocked = errors.New("blocked")

-// ErrRateLimited is returned when the search engine returns an HTTP rate limit.
+// ErrRateLimited is returned when the search engine returns HTTP 429.
 var ErrRateLimited = errors.New("rate_limited")

 // IsProxyNetworkError reports whether err is a network-level error that
@@ -85,6 +93,7 @@ func classifyProxyNetworkError(err error) error {
 	if strings.Contains(msg, "proxyconnect") ||
 		strings.Contains(msg, "connection refused") ||
 		strings.Contains(msg, "connection reset") ||
+		strings.Contains(msg, "err_tunnel_connection_failed") ||
 		strings.Contains(msg, "no such host") ||
 		strings.Contains(msg, "network is unreachable") ||
 		strings.Contains(msg, "socks connect") {
@@ -385,33 +394,12 @@ func (searchQuery *Query) InitFromContext(reqCtx *fiber.Ctx) error {
 	if err != nil {
 		return errInvalidParam(fmt.Sprintf("features: %v", err))
 	}
-	searchQuery.Extract, err = strconv.ParseBool(reqCtx.Query("extract", "0"))
-	if err != nil {
-		return errInvalidParam(fmt.Sprintf("extract: %v", err))
-	}
-	searchQuery.ExtractTop = 3
-	if raw := strings.TrimSpace(reqCtx.Query("extract_top")); raw != "" {
-		extractTop, err := strconv.Atoi(raw)
-		if err != nil {
-			return errInvalidParam("extract_top must be an integer")
-		}
-		if extractTop < 1 {
-			extractTop = 1
-		}
-		if extractTop > 5 {
-			extractTop = 5
-		}
-		searchQuery.ExtractTop = extractTop
-	}
-	searchQuery.ExtractMode = strings.ToLower(strings.TrimSpace(reqCtx.Query("extract_mode", "auto")))
-	switch searchQuery.ExtractMode {
-	case "auto", "fast", "rendered":
-	default:
-		return errInvalidParam("extract_mode must be one of auto, fast, rendered")
-	}
-	searchQuery.ExtractMinRunes, err = parseNonNegativeIntQuery(reqCtx.Query("min_runes"), 0)
-	if err != nil {
-		return errInvalidParam("min_runes must be a non-negative integer")
+	// extract is a unified bool-or-int knob: extract=0/false disables, extract=N
+	// (or true/1) extracts the top N results. The tuning params extract_mode and
+	// min_runes also imply extraction (extract=0 still overrides them). The
+	// default depth is 1 — true == 1 == "extract one result".
+	if err := parseExtractParams(reqCtx, searchQuery); err != nil {
+		return err
 	}

 	searchQuery.ProxyOverride, err = NormalizeProxyRequestOverride(reqCtx.Get("X-Use-Proxy"))
@@ -437,6 +425,72 @@ func (searchQuery *Query) InitFromContext(reqCtx *fiber.Ctx) error {
 	return nil
 }

+// parseExtractParams reads the unified extract knob plus its tuning params onto
+// q. The extract param is bool-or-int:
+//
+//	extract=0 / extract=false  → extraction off
+//	extract=true / extract=1   → on, top 1
+//	extract=N (1..5)           → on, top N (clamped to maxExtractTop)
+//
+// extract_mode and min_runes tune how extraction runs and imply extraction when
+// present, unless extract is explicitly set (extract=0 wins over them). When
+// extraction is on but no depth is given, ExtractTop defaults to 1.
+func parseExtractParams(reqCtx *fiber.Ctx, q *Query) error {
+	q.ExtractTop = defaultExtractTop
+
+	// extract accepts both bool spellings (true/false/1/0) and an integer depth.
+	// Try bool first so legacy true/false keep working, then fall back to int.
+	extractExplicit := false
+	if raw := strings.TrimSpace(reqCtx.Query("extract")); raw != "" {
+		extractExplicit = true
+		if b, err := strconv.ParseBool(raw); err == nil {
+			q.Extract = b
+			if b {
+				q.ExtractTop = 1
+			}
+		} else if n, err := strconv.Atoi(raw); err == nil {
+			q.Extract = n > 0
+			if n > 0 {
+				q.ExtractTop = clampExtractTop(n)
+			}
+		} else {
+			return errInvalidParam("extract must be a boolean or an integer (0 disables, N extracts top N)")
+		}
+	}
+
+	q.ExtractMode = strings.ToLower(strings.TrimSpace(reqCtx.Query("extract_mode", "auto")))
+	switch q.ExtractMode {
+	case "auto", "fast", "rendered":
+	default:
+		return errInvalidParam("extract_mode must be one of auto, fast, rendered")
+	}
+	if !extractExplicit && strings.TrimSpace(reqCtx.Query("extract_mode")) != "" {
+		q.Extract = true
+	}
+
+	minRunes, err := parseNonNegativeIntQuery(reqCtx.Query("min_runes"), 0)
+	if err != nil {
+		return errInvalidParam("min_runes must be a non-negative integer")
+	}
+	q.ExtractMinRunes = minRunes
+	if !extractExplicit && strings.TrimSpace(reqCtx.Query("min_runes")) != "" {
+		q.Extract = true
+	}
+
+	return nil
+}
+
+// clampExtractTop bounds a requested extraction depth to [1, maxExtractTop].
+func clampExtractTop(n int) int {
+	if n < 1 {
+		return 1
+	}
+	if n > maxExtractTop {
+		return maxExtractTop
+	}
+	return n
+}
+
 // SearchEngineOptions controls engine pacing, selector waits, and captcha
 // handling behavior shared by browser and raw implementations.
 type SearchEngineOptions struct {
--- a/core/context.go
+++ b/core/context.go
@@ -39,12 +39,9 @@ func IsContextDone(err error) bool {

 // PrepareEngineContext applies request-scoped metadata expected by all engine
 // search implementations.
-func PrepareEngineContext(ctx context.Context, query Query, engineName string, minimalBrowserProfile bool) context.Context {
+func PrepareEngineContext(ctx context.Context, query Query, engineName string) context.Context {
 	ctx = WithEngine(EnsureContext(ctx), engineName)
 	ctx = WithProfileRegion(ctx, profileRegionHint(query))
-	if minimalBrowserProfile {
-		ctx = WithMinimalBrowserProfile(ctx)
-	}
 	return WithQueryHash(ctx, QueryHashFromQuery(query))
 }

--- a/core/feature_selectors.go
+++ b/core/feature_selectors.go
@@ -153,6 +153,17 @@ func AttachFeaturesToFirstResult(results []SearchResult, features []SerpFeature)
 	return results
 }

+// StripResultFeatures drops parser-attached features unless keep is true.
+func StripResultFeatures(results []SearchResult, keep bool) []SearchResult {
+	if keep {
+		return results
+	}
+	for i := range results {
+		results[i].Features = nil
+	}
+	return results
+}
+
 // DeduplicateSerpFeatures removes duplicate modules emitted by overlapping
 // selectors while preserving original order.
 func DeduplicateSerpFeatures(features []SerpFeature) []SerpFeature {
@@ -188,18 +199,20 @@ func selectedFeatureItems(container *goquery.Selection, selectors []string) []Fe
 	for _, selector := range selectors {
 		container.Find(selector).Each(func(_ int, item *goquery.Selection) {
 			text := cleanFeatureText(item.Text())
-			title := firstAttr(item, "data-q", "data-title", "aria-label", "title")
-			// Some modules (e.g. Google PAA) carry the question in an attribute
-			// and render the answer lazily, so the element text can be empty.
+			title := cleanFeatureText(firstAttr(item, "data-q", "data-title", "aria-label", "title"))
+			// Some modules keep the question in an attr and fill text lazily.
 			if text == "" {
-				text = cleanFeatureText(title)
+				text = title
 			}
 			if text == "" {
 				return
 			}
+			if title == "" {
+				title = text
+			}
 			link := firstAttr(item, "href", "data-url", "data-link")
 			items = append(items, FeatureItem{
-				Title: strings.TrimSpace(title),
+				Title: title,
 				Text:  text,
 				Link:  strings.TrimSpace(link),
 			})
@@ -251,8 +264,15 @@ func firstNonEmpty(values ...string) string {
 	return ""
 }

+var invisibleFormatChars = strings.NewReplacer(
+	"\u00AD", "",
+	"\u200B", "",
+	"\u2060", "",
+	"\uFEFF", "",
+)
+
 func cleanFeatureText(value string) string {
-	return strings.Join(strings.Fields(value), " ")
+	return strings.Join(strings.Fields(invisibleFormatChars.Replace(value)), " ")
 }

 func serpFeatureKey(feature SerpFeature) string {
--- a/core/http_client.go
+++ b/core/http_client.go
@@ -1,35 +1,97 @@
 package core

 import (
+	"bytes"
 	"context"
-	"crypto/tls"
 	"fmt"
+	"hash/fnv"
 	"io"
-	"net"
 	"net/http"
 	"net/url"
+	"strconv"
 	"strings"
+	"sync"
 	"time"

-	"github.com/corpix/uarand"
-	utls "github.com/refraction-networking/utls"
+	fhttp "github.com/bogdanfinn/fhttp"
+	tlsclient "github.com/bogdanfinn/tls-client"
+	"github.com/bogdanfinn/tls-client/profiles"
+	browserprofile "github.com/karust/openserp/core/browser"
 )

 const rawHTTPTimeout = 30 * time.Second
+const rawHTTPClientCacheMaxEntries = 64

-// SetAcceptLanguageHeader sets the Accept-Language header from a lang code.
-// No-op when the code has no language subtag.
-func SetAcceptLanguageHeader(req *http.Request, langCode string) {
-	if req == nil {
-		return
-	}
-	if value := BuildAcceptLanguageHeader(langCode); value != "" {
-		req.Header.Set("Accept-Language", value)
-	}
+// fallbackRawUserAgent guards against tls-client's "Go-http-client" UA leaking.
+const fallbackRawUserAgent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36"
+
+// rawChromeProfiles pairs each TLS fingerprint with its Chrome major so the UA
+// and Sec-CH-UA stay coherent. We round-robin the major here. Add presets as
+// tls-client ships them.
+var rawChromeProfiles = []struct {
+	major int
+	tls   profiles.ClientProfile
+}{
+	{133, profiles.Chrome_133},
+	{144, profiles.Chrome_144},
+	{146, profiles.Chrome_146},
 }

-// DrainAndCloseResponse drains unread bytes before closing so HTTP transports
-// can safely reuse connections when callers don't consume the full body.
+// pickRawChromeProfile hashes salt to a stable but varied fingerprint.
+func pickRawChromeProfile(salt string) (int, profiles.ClientProfile) {
+	h := fnv.New32a()
+	_, _ = h.Write([]byte(salt))
+	p := rawChromeProfiles[int(h.Sum32())%len(rawChromeProfiles)]
+	return p.major, p.tls
+}
+
+// rawHeaderOrder controls request header order; tls-client profiles do not.
+var rawHeaderOrder = []string{
+	"host",
+	"user-agent",
+	"accept",
+	"accept-language",
+	"accept-encoding",
+	"upgrade-insecure-requests",
+	"sec-ch-ua",
+	"sec-ch-ua-mobile",
+	"sec-ch-ua-platform",
+	"sec-fetch-site",
+	"sec-fetch-mode",
+	"sec-fetch-user",
+	"sec-fetch-dest",
+}
+
+var rawHTTPClientCache = struct {
+	sync.Mutex
+	clients map[rawHTTPClientKey]*rawHTTPClientEntry
+}{
+	clients: map[rawHTTPClientKey]*rawHTTPClientEntry{},
+}
+
+type rawHTTPClientKey struct {
+	proxyURL             string
+	profile              string
+	insecure             bool
+	guardPrivateNetworks bool
+}
+
+type rawHTTPClientEntry struct {
+	client   tlsclient.HttpClient
+	lastUsed time.Time
+}
+
+type rawRequestProfile struct {
+	id             string
+	userAgent      string
+	acceptLanguage string
+	secCHUA        string
+	platform       string
+	mobile         bool
+	tlsProfile     profiles.ClientProfile
+}
+
+// DrainAndCloseResponse drains then closes the body so the connection can be reused.
 func DrainAndCloseResponse(resp *http.Response) {
 	if resp == nil || resp.Body == nil {
 		return
@@ -39,28 +101,99 @@ func DrainAndCloseResponse(resp *http.Response) {
 	_ = resp.Body.Close()
 }

-// RawSearchRequest builds and executes a raw-mode SERP HTTP GET. It uses the
-// shared raw HTTP client (TLS fingerprinting, network usage tracking, proxy
-// support), randomizes the User-Agent, and applies the Accept-Language header
-// derived from the query locale. The caller owns the returned response and
-// must drain/close it (see DrainAndCloseResponse).
+// RawSearchRequest executes a raw-mode GET and returns a stdlib response.
 func RawSearchRequest(ctx context.Context, searchURL string, query Query) (*http.Response, error) {
+	profile := rawRequestProfileFor(ctx, query)
+	client, err := cachedRawHTTPClient(query, profile.cacheKey(), profile.tlsProfile)
+	if err != nil {
+		return nil, err
+	}
+	SetBrowserProfileID(ctx, profile.id)
+
+	// Guarded path validates every hop, including the first.
 	if query.GuardPrivateNetworks {
-		if err := ValidatePublicHTTPURL(ctx, searchURL); err != nil {
+		return doGuardedRawRequest(ctx, client, searchURL, profile, query)
+	}
+	return doRawRequest(ctx, client, searchURL, profile, query)
+}
+
+// doRawRequest issues one GET and converts the response at the boundary.
+func doRawRequest(ctx context.Context, client tlsclient.HttpClient, searchURL string, profile rawRequestProfile, query Query) (*http.Response, error) {
+	req, err := fhttp.NewRequestWithContext(ctx, fhttp.MethodGet, searchURL, nil)
+	if err != nil {
+		return nil, err
+	}
+	applyRawRequestHeaders(req, profile)
+	return execRawRequest(ctx, client, req, rawRequestUsesProxy(query))
+}
+
+// execRawRequest runs the request and converts proxy errors and the response.
+func execRawRequest(ctx context.Context, client tlsclient.HttpClient, req *fhttp.Request, proxied bool) (*http.Response, error) {
+	resp, err := client.Do(req)
+	if err != nil {
+		if proxied {
+			return nil, classifyProxyNetworkError(err)
+		}
+		return nil, err
+	}
+	return convertRawResponse(ctx, resp), nil
+}
+
+const maxGuardedRedirects = 10
+
+// doGuardedRawRequest validates every redirect hop before fetching it.
+func doGuardedRawRequest(ctx context.Context, client tlsclient.HttpClient, searchURL string, profile rawRequestProfile, query Query) (*http.Response, error) {
+	current := searchURL
+	for hop := 0; ; hop++ {
+		if err := ValidatePublicHTTPURL(ctx, current); err != nil {
 			return nil, err
 		}
+		resp, err := doRawRequest(ctx, client, current, profile, query)
+		if err != nil {
+			return nil, err
+		}
+		location, ok := redirectLocation(resp)
+		if !ok {
+			return resp, nil
+		}
+		if hop >= maxGuardedRedirects {
+			DrainAndCloseResponse(resp)
+			return nil, fmt.Errorf("%w: stopped after %d redirects", ErrEngineInternal, maxGuardedRedirects)
+		}
+		next, err := resolveRedirectURL(current, location)
+		if err != nil {
+			DrainAndCloseResponse(resp)
+			return nil, err
+		}
+		DrainAndCloseResponse(resp)
+		current = next
 	}
-	client, err := NewRawHTTPClient(query)
+}
+
+func redirectLocation(resp *http.Response) (string, bool) {
+	if resp == nil {
+		return "", false
+	}
+	switch resp.StatusCode {
+	case http.StatusMovedPermanently, http.StatusFound, http.StatusSeeOther,
+		http.StatusTemporaryRedirect, http.StatusPermanentRedirect:
+		location := strings.TrimSpace(resp.Header.Get("Location"))
+		return location, location != ""
+	default:
+		return "", false
+	}
+}
+
+func resolveRedirectURL(base, location string) (string, error) {
+	baseURL, err := url.Parse(base)
 	if err != nil {
-		return nil, err
+		return "", err
 	}
-	req, err := http.NewRequestWithContext(ctx, http.MethodGet, searchURL, nil)
+	locURL, err := url.Parse(location)
 	if err != nil {
-		return nil, err
+		return "", err
 	}
-	req.Header.Set("User-Agent", uarand.GetRandom())
-	SetAcceptLanguageHeader(req, query.LangCode)
-	return client.Do(req)
+	return baseURL.ResolveReference(locURL).String(), nil
 }

 func ReadRawSearchBody(resp *http.Response) ([]byte, error) {
@@ -91,140 +224,315 @@ func ClassifySearchHTTPStatus(status int) error {
 	return nil
 }

+// NewRawHTTPClient returns a stdlib client backed by tls-client.
 func NewRawHTTPClient(query Query) (*http.Client, error) {
-	transport, err := newRawTransport(query)
+	profile := rawRequestProfileFor(context.Background(), query)
+	client, err := cachedRawHTTPClient(query, profile.cacheKey(), profile.tlsProfile)
 	if err != nil {
 		return nil, err
 	}
-	roundTripper := http.RoundTripper(transport)
-	if transport.Proxy != nil {
-		roundTripper = proxyErrorTransport{base: roundTripper}
-	}

-	client := &http.Client{
-		Transport: roundTripper,
-		Timeout:   rawHTTPTimeout,
+	stdClient := &http.Client{
+		Transport: rawTLSRoundTripper{
+			client:               client,
+			proxied:              rawRequestUsesProxy(query),
+			guardPrivateNetworks: query.GuardPrivateNetworks,
+			profile:              profile,
+		},
+		Timeout: rawHTTPTimeout,
 	}
 	if query.GuardPrivateNetworks {
-		client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
+		stdClient.CheckRedirect = func(req *http.Request, via []*http.Request) error {
 			return ValidatePublicHTTPURL(req.Context(), req.URL.String())
 		}
 	}
-	return client, nil
+	return stdClient, nil
 }

-func newRawTransport(query Query) (*http.Transport, error) {
-	dialContext := dialNetworkUsageConn
-	if query.GuardPrivateNetworks {
-		dialContext = guardedDialNetworkUsageConn
-	}
-
-	transport := &http.Transport{
-		DialContext: dialContext,
-	}
-	if query.Insecure {
-		transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
-	}
-
+func cachedRawHTTPClient(query Query, profileKey string, tlsProfile profiles.ClientProfile) (tlsclient.HttpClient, error) {
 	proxyURL, err := NormalizeProxyURL(query.ProxyURL)
 	if err != nil {
 		return nil, err
 	}
-	if proxyURL != "" {
-		parsed, err := url.Parse(proxyURL)
-		if err != nil {
-			return nil, err
-		}

-		// Keep proxied requests on the standard transport path so SOCKS5/SOCKS5H
-		// resolution and routing are handled by the configured proxy correctly.
-		// The extract SSRF guard validates the target URL before the request and
-		// on redirects; the proxy address itself may legitimately be local.
-		transport.DialContext = dialNetworkUsageConn
-		transport.Proxy = http.ProxyURL(parsed)
-		return transport, nil
+	key := rawHTTPClientKey{
+		proxyURL:             proxyURL,
+		profile:              profileKey,
+		insecure:             query.Insecure,
+		guardPrivateNetworks: query.GuardPrivateNetworks,
 	}

-	transport.DialTLSContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
-		rawConn, err := dialContext(ctx, network, addr)
-		if err != nil {
-			return nil, err
-		}
+	rawHTTPClientCache.Lock()
+	defer rawHTTPClientCache.Unlock()

-		hostname := strings.Split(addr, ":")[0]
-		config := &utls.Config{
-			ServerName:         hostname,
-			InsecureSkipVerify: query.Insecure,
-			NextProtos:         []string{"http/1.1"},
-		}
-
-		uconn := utls.UClient(rawConn, config, utls.HelloChrome_Auto)
-		if err := uconn.BuildHandshakeState(); err != nil {
-			rawConn.Close()
-			return nil, err
-		}
-		forceHTTP1ALPN(uconn)
-		if err := uconn.Handshake(); err != nil {
-			rawConn.Close()
-			return nil, err
-		}
-
-		return uconn, nil
+	now := time.Now()
+	if entry := rawHTTPClientCache.clients[key]; entry != nil {
+		entry.lastUsed = now
+		return entry.client, nil
 	}

-	return transport, nil
+	client, err := newRawTLSClient(query, proxyURL, tlsProfile)
+	if err != nil {
+		return nil, err
+	}
+	rawHTTPClientCache.clients[key] = &rawHTTPClientEntry{
+		client:   client,
+		lastUsed: now,
+	}
+	evictRawHTTPClientCacheLocked()
+	return client, nil
 }

-func forceHTTP1ALPN(conn *utls.UConn) {
-	for _, ext := range conn.Extensions {
-		if alpn, ok := ext.(*utls.ALPNExtension); ok {
-			alpn.AlpnProtocols = []string{"http/1.1"}
+func rawRequestUsesProxy(query Query) bool {
+	return strings.TrimSpace(query.ProxyURL) != ""
+}
+
+func evictRawHTTPClientCacheLocked() {
+	for len(rawHTTPClientCache.clients) > rawHTTPClientCacheMaxEntries {
+		var (
+			oldestKey   rawHTTPClientKey
+			oldestEntry *rawHTTPClientEntry
+		)
+		for key, entry := range rawHTTPClientCache.clients {
+			if oldestEntry == nil || entry.lastUsed.Before(oldestEntry.lastUsed) {
+				oldestKey = key
+				oldestEntry = entry
+			}
+		}
+		if oldestEntry == nil {
 			return
 		}
+		delete(rawHTTPClientCache.clients, oldestKey)
+		oldestEntry.client.CloseIdleConnections()
 	}
-	conn.Extensions = append(conn.Extensions, &utls.ALPNExtension{AlpnProtocols: []string{"http/1.1"}})
 }

-func dialNetworkUsageConn(ctx context.Context, network, addr string) (net.Conn, error) {
-	dialer := &net.Dialer{}
-	conn, err := dialer.DialContext(ctx, network, addr)
+// newRawTLSClient builds a pooled Chrome-profile transport; proxyURL must be normalized.
+func newRawTLSClient(query Query, proxyURL string, tlsProfile profiles.ClientProfile) (tlsclient.HttpClient, error) {
+	options := []tlsclient.HttpClientOption{
+		tlsclient.WithClientProfile(tlsProfile),
+		tlsclient.WithTimeout(int(rawHTTPTimeout / time.Second)),
+		tlsclient.WithNotFollowRedirects(),
+	}
+	if query.Insecure {
+		options = append(options, tlsclient.WithInsecureSkipVerify())
+	}
+
+	if proxyURL != "" {
+		options = append(options, tlsclient.WithProxyUrl(proxyURL))
+	} else if query.GuardPrivateNetworks {
+		options = append(options, tlsclient.WithDialContext(GuardedDialContext))
+	}
+
+	return tlsclient.NewHttpClient(tlsclient.NewNoopLogger(), options...)
+}
+
+// convertRawResponse keeps fhttp from leaking past this file.
+func convertRawResponse(ctx context.Context, resp *fhttp.Response) *http.Response {
+	if resp == nil {
+		return nil
+	}
+	std := &http.Response{
+		Status:        resp.Status,
+		StatusCode:    resp.StatusCode,
+		Proto:         resp.Proto,
+		ProtoMajor:    resp.ProtoMajor,
+		ProtoMinor:    resp.ProtoMinor,
+		Header:        http.Header(resp.Header),
+		ContentLength: resp.ContentLength,
+		Body:          resp.Body,
+	}
+	if std.Body == nil {
+		std.Body = io.NopCloser(bytes.NewReader(nil))
+	}
+	std.Body = networkUsageReadCloser{ReadCloser: std.Body, ctx: ctx}
+	return std
+}
+
+type rawTLSRoundTripper struct {
+	client               tlsclient.HttpClient
+	proxied              bool
+	guardPrivateNetworks bool
+	profile              rawRequestProfile
+}
+
+func (rt rawTLSRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+	if rt.guardPrivateNetworks {
+		if err := ValidatePublicHTTPURL(req.Context(), req.URL.String()); err != nil {
+			return nil, err
+		}
+	}
+
+	freq, err := fhttp.NewRequestWithContext(req.Context(), req.Method, req.URL.String(), req.Body)
 	if err != nil {
 		return nil, err
 	}
-	return networkUsageConn{Conn: conn, ctx: ctx}, nil
-}
-
-func guardedDialNetworkUsageConn(ctx context.Context, network, addr string) (net.Conn, error) {
-	conn, err := GuardedDialContext(ctx, network, addr)
-	if err != nil {
-		return nil, err
+	for key, values := range req.Header {
+		freq.Header[key] = values
 	}
-	return networkUsageConn{Conn: conn, ctx: ctx}, nil
+	applyRawRequestHeaders(freq, rt.profile)
+	SetBrowserProfileID(req.Context(), rt.profile.id)
+	return execRawRequest(req.Context(), rt.client, freq, rt.proxied)
 }

-type networkUsageConn struct {
-	net.Conn
+func rawRequestProfileFor(ctx context.Context, query Query) rawRequestProfile {
+	engine := engineFromContext(ctx)
+	region := rawProfileRegion(ctx, query)
+	salt := rawProfileSalt(ctx, engine, region)
+
+	profile := browserprofile.Profile{}
+	if forcedID := forcedProfileIDFromContext(ctx); forcedID != "" {
+		if forced, ok := browserprofile.ProfileByID(forcedID); ok {
+			profile = forced
+		}
+	}
+	if strings.TrimSpace(profile.ID) == "" {
+		profile = browserprofile.SelectProfileForSession(engine, region, salt)
+	}
+	profile = applyProfileLanguageHint(profile, region)
+
+	major, tlsProfile := pickRawChromeProfile(salt + "\x00" + strings.TrimSpace(profile.ID))
+	profile = applyRawChromeMajor(profile, major)
+
+	userAgent := strings.TrimSpace(profile.UserAgent)
+	if userAgent == "" {
+		userAgent = fallbackRawUserAgent
+	}
+	acceptLanguage := strings.TrimSpace(profile.AcceptLanguage)
+	if acceptLanguage == "" {
+		acceptLanguage = BuildAcceptLanguageHeader(region)
+	}
+	if acceptLanguage == "" {
+		acceptLanguage = BuildAcceptLanguageHeader(query.LangCode)
+	}
+
+	return rawRequestProfile{
+		id:             strings.TrimSpace(profile.ID),
+		userAgent:      userAgent,
+		acceptLanguage: acceptLanguage,
+		secCHUA:        formatSecCHUA(profile.UACHBrands),
+		platform:       strings.TrimSpace(profile.Platform),
+		mobile:         profile.Mobile,
+		tlsProfile:     tlsProfile,
+	}
+}
+
+func applyRawChromeMajor(profile browserprofile.Profile, major int) browserprofile.Profile {
+	version := strconv.Itoa(major)
+	if template := strings.TrimSpace(profile.UserAgentTemplate); template != "" {
+		profile.UserAgent = strings.ReplaceAll(template, "{chrome_major}", version)
+	} else if extractChromeVersion(profile.UserAgent) == "" {
+		profile.UserAgent = fallbackRawUserAgent
+	} else {
+		profile.UserAgent = replaceChromeUserAgentVersion(profile.UserAgent, version+".0.0.0")
+	}
+	if len(profile.UACHBrands) == 0 {
+		profile.UACHBrands = rawUACHBrands(version, false)
+	} else {
+		profile.UACHBrands = patchBrandVersions(profile.UACHBrands, version, false)
+	}
+	if len(profile.UACHFullVerList) == 0 {
+		profile.UACHFullVerList = rawUACHBrands(version+".0.0.0", true)
+	} else {
+		profile.UACHFullVerList = patchBrandVersions(profile.UACHFullVerList, version+".0.0.0", true)
+	}
+	return profile
+}
+
+func rawUACHBrands(version string, full bool) []browserprofile.BrandVersion {
+	notABrandVersion := "24"
+	if full {
+		notABrandVersion = "24.0.0.0"
+	}
+	return []browserprofile.BrandVersion{
+		{Brand: "Not_A Brand", Version: notABrandVersion},
+		{Brand: "Chromium", Version: version},
+		{Brand: "Google Chrome", Version: version},
+	}
+}
+
+func rawProfileRegion(ctx context.Context, query Query) string {
+	if region := profileRegionFromContext(ctx); region != "" {
+		return region
+	}
+	if query.ProxyCountry != "" {
+		return query.ProxyCountry
+	}
+	return profileRegionHint(query)
+}
+
+func rawProfileSalt(ctx context.Context, engine, region string) string {
+	if laneKey := proxyLaneKeyFromContext(ctx); !laneKey.Empty() {
+		return laneKey.SessionID
+	}
+	return browserprofile.LaneKey(engine, region)
+}
+
+// cacheKey includes all headers that affect the pooled fingerprint.
+func (p rawRequestProfile) cacheKey() string {
+	return strings.Join([]string{p.id, p.userAgent, p.acceptLanguage, p.secCHUA, p.platform, fmt.Sprint(p.mobile)}, "\x00")
+}
+
+// applyRawRequestHeaders sets the Chrome identity headers and order; tls-client
+// owns Host and Accept-Encoding.
+func applyRawRequestHeaders(req *fhttp.Request, profile rawRequestProfile) {
+	if req == nil {
+		return
+	}
+	secCHUAMobile := "?0"
+	if profile.mobile {
+		secCHUAMobile = "?1"
+	}
+	platform := ""
+	if profile.platform != "" {
+		platform = quoteSecCHValue(profile.platform)
+	}
+	for _, h := range [][2]string{
+		{"User-Agent", profile.userAgent},
+		{"Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8"},
+		{"Accept-Language", profile.acceptLanguage},
+		{"Upgrade-Insecure-Requests", "1"},
+		{"Sec-CH-UA", profile.secCHUA},
+		{"Sec-CH-UA-Mobile", secCHUAMobile},
+		{"Sec-CH-UA-Platform", platform},
+		{"Sec-Fetch-Site", "none"},
+		{"Sec-Fetch-Mode", "navigate"},
+		{"Sec-Fetch-User", "?1"},
+		{"Sec-Fetch-Dest", "document"},
+	} {
+		if h[1] != "" {
+			req.Header.Set(h[0], h[1])
+		}
+	}
+	req.Header[fhttp.HeaderOrderKey] = rawHeaderOrder
+}
+
+func formatSecCHUA(brands []browserprofile.BrandVersion) string {
+	parts := make([]string, 0, len(brands))
+	for _, brand := range brands {
+		name := strings.TrimSpace(brand.Brand)
+		version := strings.TrimSpace(brand.Version)
+		if name == "" || version == "" {
+			continue
+		}
+		parts = append(parts, quoteSecCHValue(name)+`;v=`+quoteSecCHValue(version))
+	}
+	return strings.Join(parts, ", ")
+}
+
+func quoteSecCHValue(value string) string {
+	value = strings.ReplaceAll(value, `\`, `\\`)
+	value = strings.ReplaceAll(value, `"`, `\"`)
+	return `"` + value + `"`
+}
+
+type networkUsageReadCloser struct {
+	io.ReadCloser
 	ctx context.Context
 }

-func (c networkUsageConn) Read(p []byte) (int, error) {
-	n, err := c.Conn.Read(p)
-	AddNetworkBytes(c.ctx, int64(n))
+func (r networkUsageReadCloser) Read(p []byte) (int, error) {
+	n, err := r.ReadCloser.Read(p)
+	AddNetworkBytes(r.ctx, int64(n))
 	return n, err
 }
-
-type proxyErrorTransport struct {
-	base http.RoundTripper
-}
-
-func (t proxyErrorTransport) RoundTrip(req *http.Request) (*http.Response, error) {
-	resp, err := t.base.RoundTrip(req)
-	if err != nil {
-		return nil, classifyProxyNetworkError(err)
-	}
-	if resp != nil && resp.StatusCode == http.StatusProxyAuthRequired {
-		DrainAndCloseResponse(resp)
-		return nil, classifyProxyNetworkError(ErrProxyAuth)
-	}
-	return resp, nil
-}
--- a/core/http_client_test.go
+++ b/core/http_client_test.go
@@ -4,9 +4,13 @@ import (
 	"context"
 	"errors"
 	"io"
+	"net"
 	"net/http"
 	"net/http/httptest"
+	"strconv"
 	"strings"
+	"sync"
+	"sync/atomic"
 	"testing"
 )

@@ -106,11 +110,68 @@ func TestRawHTTPClientTracksNetworkBytes(t *testing.T) {
 	}
 }

-func TestRawHTTPClientTracksProxyErrorBytes(t *testing.T) {
-	proxyBody := "proxy auth required"
+func TestRawHTTPClientAppliesBrowserHeaderDefaults(t *testing.T) {
+	resetRawHTTPClientCache(t)
+
+	var userAgent, acceptLanguage, secCHUA string
+	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		userAgent = r.Header.Get("User-Agent")
+		acceptLanguage = r.Header.Get("Accept-Language")
+		secCHUA = r.Header.Get("Sec-CH-UA")
+		_, _ = w.Write([]byte("ok"))
+	}))
+	defer server.Close()
+
+	client, err := NewRawHTTPClient(Query{LangCode: "fr", Region: "FR"})
+	if err != nil {
+		t.Fatalf("new raw client: %v", err)
+	}
+
+	ctx := WithBrowserProfileUsage(context.Background())
+	req, err := http.NewRequestWithContext(ctx, http.MethodGet, server.URL, nil)
+	if err != nil {
+		t.Fatalf("new request: %v", err)
+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		t.Fatalf("do request: %v", err)
+	}
+	DrainAndCloseResponse(resp)
+
+	if userAgent == "" || strings.Contains(userAgent, "Go-http-client") {
+		t.Fatalf("unexpected User-Agent %q", userAgent)
+	}
+	if want := BuildAcceptLanguageHeader("fr-FR"); acceptLanguage != want {
+		t.Fatalf("Accept-Language = %q, want %q", acceptLanguage, want)
+	}
+	if secCHUA == "" {
+		t.Fatal("expected Sec-CH-UA to be set")
+	}
+	if ids := BrowserProfileIDsFromContext(ctx); len(ids) != 1 || ids[0] == "" {
+		t.Fatalf("expected one recorded profile id, got %v", ids)
+	}
+}
+
+func TestRawHTTPClientGuardRejectsInitialPrivateURLWithProxy(t *testing.T) {
+	client, err := NewRawHTTPClient(Query{
+		ProxyURL:             "http://127.0.0.1:1",
+		GuardPrivateNetworks: true,
+	})
+	if err != nil {
+		t.Fatalf("new raw client: %v", err)
+	}
+
+	resp, err := client.Get("http://127.0.0.1/")
+	DrainAndCloseResponse(resp)
+	if !errors.Is(err, ErrTargetNotAllowed) {
+		t.Fatalf("expected target guard error, got %v", err)
+	}
+}
+
+func TestRawHTTPClientProxyAuthError(t *testing.T) {
 	proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		w.WriteHeader(http.StatusProxyAuthRequired)
-		_, _ = w.Write([]byte(proxyBody))
+		_, _ = w.Write([]byte("proxy auth required"))
 	}))
 	defer proxy.Close()

@@ -129,7 +190,117 @@ func TestRawHTTPClientTracksProxyErrorBytes(t *testing.T) {
 	if !errors.Is(err, ErrProxyAuth) {
 		t.Fatalf("expected proxy auth error, got %v", err)
 	}
-	if got := NetworkBytesFromContext(ctx); got < int64(len(proxyBody)) {
-		t.Fatalf("expected tracked bytes >= proxy body length, got %d", got)
+}
+
+// TestRawSearchRequestReusesPooledClient checks that same-profile calls share
+// one connection and a stable Chrome UA. Bytes/headers are covered elsewhere.
+func TestRawSearchRequestReusesPooledClient(t *testing.T) {
+	resetRawHTTPClientCache(t)
+
+	var connCount atomic.Int32
+	var mu sync.Mutex
+	var userAgents []string
+
+	server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		mu.Lock()
+		userAgents = append(userAgents, r.Header.Get("User-Agent"))
+		mu.Unlock()
+		_, _ = w.Write([]byte("ok"))
+	}))
+	server.Config.ConnState = func(_ net.Conn, state http.ConnState) {
+		if state == http.StateNew {
+			connCount.Add(1)
+		}
+	}
+	server.Start()
+	defer server.Close()
+
+	query := Query{LangCode: "de", Region: "DE"}
+	for i := 0; i < 2; i++ {
+		ctx := WithEngine(WithBrowserProfileUsage(context.Background()), "google")
+		readRawSearchBodyForTest(t, ctx, server.URL, query)
+		if ids := BrowserProfileIDsFromContext(ctx); len(ids) != 1 || ids[0] == "" {
+			t.Fatalf("request %d recorded profile ids %v, want exactly one", i, ids)
+		}
+	}
+
+	if got := connCount.Load(); got != 1 {
+		t.Fatalf("expected one reused TCP connection, got %d", got)
+	}
+
+	mu.Lock()
+	defer mu.Unlock()
+	if len(userAgents) != 2 {
+		t.Fatalf("expected two captured User-Agents, got %d", len(userAgents))
+	}
+	if userAgents[0] == "" || strings.Contains(userAgents[0], "Go-http-client") || userAgents[0] != userAgents[1] {
+		t.Fatalf("expected stable Chrome User-Agent, got %q then %q", userAgents[0], userAgents[1])
+	}
+}
+
+// TestRawRequestProfilesRoundRobinCoherently checks rotation hits every Chrome
+// major and keeps UA major == Sec-CH-UA major == TLS fingerprint.
+func TestRawRequestProfilesRoundRobinCoherently(t *testing.T) {
+	tlsByMajor := map[int]string{}
+	for _, p := range rawChromeProfiles {
+		tlsByMajor[p.major] = p.tls.GetClientHelloStr()
+	}
+
+	seen := map[int]bool{}
+	for i := 0; i < 200; i++ {
+		ctx := WithProxyLaneKey(WithEngine(context.Background(), "google"),
+			ProxyLaneKey{Engine: "google", SessionID: "sid-" + strconv.Itoa(i)})
+
+		profile := rawRequestProfileFor(ctx, Query{Region: "US"})
+		major, err := strconv.Atoi(chromeMajorVersion(extractChromeVersion(profile.userAgent)))
+		if err != nil {
+			t.Fatalf("raw User-Agent has no Chrome major: %q", profile.userAgent)
+		}
+		wantTLS, ok := tlsByMajor[major]
+		if !ok {
+			t.Fatalf("UA major %d has no configured tls-client profile", major)
+		}
+		if profile.tlsProfile.GetClientHelloStr() != wantTLS {
+			t.Fatalf("major %d: TLS fingerprint does not match UA", major)
+		}
+		if !strings.Contains(profile.secCHUA, `;v="`+strconv.Itoa(major)+`"`) {
+			t.Fatalf("Sec-CH-UA %q does not match UA major %d", profile.secCHUA, major)
+		}
+		seen[major] = true
+	}
+
+	for _, p := range rawChromeProfiles {
+		if !seen[p.major] {
+			t.Fatalf("configured Chrome major %d never selected; seen=%v", p.major, seen)
+		}
+	}
+}
+
+func readRawSearchBodyForTest(t *testing.T, ctx context.Context, searchURL string, query Query) string {
+	t.Helper()
+
+	resp, err := RawSearchRequest(ctx, searchURL, query)
+	if err != nil {
+		t.Fatalf("raw search request: %v", err)
+	}
+	defer DrainAndCloseResponse(resp)
+
+	body, err := io.ReadAll(resp.Body)
+	if err != nil {
+		t.Fatalf("read raw search body: %v", err)
+	}
+	return string(body)
+}
+
+func resetRawHTTPClientCache(t *testing.T) {
+	t.Helper()
+
+	rawHTTPClientCache.Lock()
+	entries := rawHTTPClientCache.clients
+	rawHTTPClientCache.clients = map[rawHTTPClientKey]*rawHTTPClientEntry{}
+	rawHTTPClientCache.Unlock()
+
+	for _, entry := range entries {
+		entry.client.CloseIdleConnections()
 	}
 }
--- a/core/logger.go
+++ b/core/logger.go
@@ -251,7 +251,7 @@ func quoteIfNeeded(s string) string {
 	return s
 }

-func InitLogger(isVerbose, isDebug bool, format string) {
+func InitLogger(isVerbose, isDebug, isQuiet bool, format string) {
 	switch format {
 	case LogFormatText:
 		logrus.SetFormatter(&bracketFormatter{TimestampFormat: "2006-01-02 15:04:05"})
@@ -261,16 +261,22 @@ func InitLogger(isVerbose, isDebug bool, format string) {
 		})
 	}

-	if isDebug {
-		logrus.SetOutput(io.MultiWriter(os.Stdout))
+	// Logs go to stderr (+ optional file) so stdout carries only the payload.
+	switch {
+	case isDebug:
+		logrus.SetOutput(io.MultiWriter(os.Stderr))
 		logrus.SetReportCaller(true)
-	} else {
+	case isQuiet:
+		// One-shot CLI default: stderr only, no ./logs.txt in the user's CWD.
+		logrus.SetOutput(os.Stderr)
+		logrus.SetReportCaller(false)
+	default:
 		f, err := os.OpenFile("./logs.txt", os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
 		if err != nil {
 			fmt.Fprintf(os.Stderr, "Failed to open logs file ./logs.txt: %v\n", err)
-			logrus.SetOutput(io.MultiWriter(os.Stdout))
+			logrus.SetOutput(io.MultiWriter(os.Stderr))
 		} else {
-			logrus.SetOutput(io.MultiWriter(f, os.Stdout))
+			logrus.SetOutput(io.MultiWriter(f, os.Stderr))
 		}
 		logrus.SetReportCaller(false)
 	}
@@ -282,5 +288,9 @@ func InitLogger(isVerbose, isDebug bool, format string) {
 	if isDebug {
 		level = logrus.TraceLevel
 	}
+	if isQuiet && !isVerbose && !isDebug {
+		// Quiet keeps only warnings/errors on stderr.
+		level = logrus.WarnLevel
+	}
 	logrus.SetLevel(level)
 }
--- a/core/middleware.go
+++ b/core/middleware.go
@@ -30,7 +30,7 @@ type CORSConfig struct {
 const browserProfileIDHeader = "X-Browser-Profile-Id"
 const useProfileHeader = "X-Use-Profile"

-const exposedResponseHeaders = "X-Request-ID, X-Cache, X-Fallback-Engine, X-Proxy-Mode, X-Proxy-Tag, X-Proxy-Used, X-Network-Bytes, " + browserProfileIDHeader
+const exposedResponseHeaders = "X-Request-ID, X-Cache, X-Fallback-Engine, X-Proxy-Mode, X-Proxy-Tag, X-Proxy-Used, X-Proxy-Attempts, X-Network-Bytes, " + browserProfileIDHeader

 func DefaultCORSConfig() CORSConfig {
 	return CORSConfig{
@@ -65,10 +65,10 @@ func RequestContextMiddleware() fiber.Handler {

 // RequestTimeoutMiddleware bounds wall-clock time per request by attaching a
 // deadline to the user context, which fasthttp never cancels on client
-// disconnect. /mega/* (MegaTimeout) and /extract (batch budget) are exempt.
+// disconnect. /mega/* (MegaTimeout) and /extract* (batch budget) are exempt.
 func RequestTimeoutMiddleware(timeout time.Duration) fiber.Handler {
 	return func(c *fiber.Ctx) error {
-		if strings.HasPrefix(c.Path(), "/mega/") || c.Path() == "/extract" {
+		if strings.HasPrefix(c.Path(), "/mega/") || c.Path() == "/extract" || c.Path() == "/extract/batch" {
 			return c.Next()
 		}
 		ctx, cancel := context.WithTimeout(c.UserContext(), timeout)
--- a/core/middleware_test.go
+++ b/core/middleware_test.go
@@ -63,8 +63,8 @@ func TestCORSMiddleware_UsesConfiguredHeaders(t *testing.T) {
 	if got := resp.Header.Get("Access-Control-Max-Age"); got != "1200" {
 		t.Fatalf("unexpected max-age: %q", got)
 	}
-	if got := resp.Header.Get("Access-Control-Expose-Headers"); !strings.Contains(got, "X-Network-Bytes") || !strings.Contains(got, "X-Browser-Profile-Id") {
-		t.Fatalf("expected exposed headers to include network and browser profile headers, got %q", got)
+	if got := resp.Header.Get("Access-Control-Expose-Headers"); !strings.Contains(got, "X-Network-Bytes") || !strings.Contains(got, "X-Browser-Profile-Id") || !strings.Contains(got, "X-Proxy-Attempts") {
+		t.Fatalf("expected exposed headers to include network, browser profile, and proxy-attempts headers, got %q", got)
 	}
 }

--- a/core/page_helpers.go
+++ b/core/page_helpers.go
@@ -64,6 +64,56 @@ func WaitForElements(ctx context.Context, page *rod.Page, selectors []string, ti
 	return nil, "", ErrSearchTimeout
 }

+// DocSignals is the per-engine selector/text-marker config for
+// ClassifyChallengeDocument.
+type DocSignals struct {
+	CaptchaSelectors []string
+	CaptchaMarkers   []string
+	EmptySelectors   []string
+	EmptyMarkers     []string
+}
+
+// ClassifyChallengeDocument is the shared captcha/empty-result check behind
+// every engine's classify*Document. Selectors are checked before markers
+// since they're cheaper (no doc.Text() walk).
+func ClassifyChallengeDocument(doc *goquery.Document, s DocSignals) error {
+	if anySelectorMatches(doc, s.CaptchaSelectors) {
+		return ErrCaptcha
+	}
+	var text string
+	if len(s.CaptchaMarkers)+len(s.EmptyMarkers) > 0 {
+		text = strings.ToLower(doc.Text())
+	}
+	if anyMarkerMatches(text, s.CaptchaMarkers) {
+		return ErrCaptcha
+	}
+	if anySelectorMatches(doc, s.EmptySelectors) {
+		return ErrEmptyResult
+	}
+	if anyMarkerMatches(text, s.EmptyMarkers) {
+		return ErrEmptyResult
+	}
+	return nil
+}
+
+func anySelectorMatches(doc *goquery.Document, selectors []string) bool {
+	for _, selector := range selectors {
+		if doc.Find(selector).Length() > 0 {
+			return true
+		}
+	}
+	return false
+}
+
+func anyMarkerMatches(text string, markers []string) bool {
+	for _, marker := range markers {
+		if strings.Contains(text, marker) {
+			return true
+		}
+	}
+	return false
+}
+
 // HasAnySelector returns true if at least one of the supplied selectors
 // currently matches in the page DOM. It does not wait — pair with
 // WaitForElements when hydration may be in flight.
@@ -102,8 +152,49 @@ func HasAttribute(el *rod.Element, attr string) bool {
 	return err == nil && v != nil
 }

-// FirstNonEmptyText returns the trimmed text of the first selector under root
-// that yields non-empty content. Empty string if none match.
+// NormalizeWhitespace collapses runs of whitespace (newlines, source
+// indentation) into single spaces and trims the result.
+func NormalizeWhitespace(s string) string {
+	return strings.Join(strings.Fields(s), " ")
+}
+
+// ElementText returns el's visible text, falling back to textContent (for nodes
+// rod's Text() leaves empty), normalized. Empty string if el is nil or blank.
+func ElementText(el *rod.Element) string {
+	if el == nil {
+		return ""
+	}
+	if text, err := el.Text(); err == nil {
+		if normalized := NormalizeWhitespace(text); normalized != "" {
+			return normalized
+		}
+	}
+	if value, err := el.Property("textContent"); err == nil {
+		return NormalizeWhitespace(value.String())
+	}
+	return ""
+}
+
+// ElementAttribute returns the first non-empty value among attrs on el,
+// normalized. Empty string if el is nil or none are set.
+func ElementAttribute(el *rod.Element, attrs ...string) string {
+	if el == nil {
+		return ""
+	}
+	for _, attr := range attrs {
+		value, err := el.Attribute(attr)
+		if err != nil || value == nil {
+			continue
+		}
+		if normalized := NormalizeWhitespace(*value); normalized != "" {
+			return normalized
+		}
+	}
+	return ""
+}
+
+// FirstNonEmptyText returns the text (see ElementText) of the first selector
+// under root that yields non-empty content. Empty string if none match.
 func FirstNonEmptyText(root *rod.Element, selectors ...string) string {
 	if root == nil {
 		return ""
@@ -113,12 +204,8 @@ func FirstNonEmptyText(root *rod.Element, selectors ...string) string {
 		if err != nil {
 			continue
 		}
-		text, err := el.Text()
-		if err != nil {
-			continue
-		}
-		if trimmed := strings.TrimSpace(text); trimmed != "" {
-			return trimmed
+		if text := ElementText(el); text != "" {
+			return text
 		}
 	}
 	return ""
@@ -173,13 +260,46 @@ func FirstNonEmptyAttribute(root *rod.Element, attr string, selectors ...string)
 // (page.HTML -> goquery doc -> extract), so it lives here rather than being
 // copied per engine. Returns nil on any rendering/parse error.
 func FeaturesFromPage(page *rod.Page, extract func(*goquery.Document) []SerpFeature) []SerpFeature {
-	html, err := page.HTML()
-	if err != nil {
+	if page == nil {
 		return nil
 	}
-	doc, err := goquery.NewDocumentFromReader(strings.NewReader(html))
+	doc, err := DocumentFromPage(page)
 	if err != nil {
 		return nil
 	}
 	return extract(doc)
 }
+
+// DocumentFromPage snapshots a live rod page to a goquery document, so the
+// browser path can reuse the same parsing/classification rules as the raw
+// HTTP path instead of reimplementing them against the rod API.
+func DocumentFromPage(page *rod.Page) (*goquery.Document, error) {
+	html, err := page.HTML()
+	if err != nil {
+		return nil, err
+	}
+	return goquery.NewDocumentFromReader(strings.NewReader(html))
+}
+
+// ClassifyFromPage snapshots page and runs classify over it, returning nil if
+// the page can't be read (caller treats that as "not classified").
+func ClassifyFromPage(page *rod.Page, classify func(*goquery.Document) error) error {
+	doc, err := DocumentFromPage(page)
+	if err != nil {
+		return nil
+	}
+	return classify(doc)
+}
+
+const featureHydrationWait = 2000 * time.Millisecond
+
+// FeaturesFromPageWithWait lets async feature modules render before snapshotting.
+func FeaturesFromPageWithWait(ctx context.Context, page *rod.Page, extract func(*goquery.Document) []SerpFeature) []SerpFeature {
+	if page != nil {
+		_, _ = page.Eval(`() => window.scrollTo(0, document.body.scrollHeight)`)
+		if err := SleepContext(ctx, featureHydrationWait); err != nil {
+			return nil
+		}
+	}
+	return FeaturesFromPage(page, extract)
+}
--- a/core/profile_context.go
+++ b/core/profile_context.go
@@ -9,7 +9,6 @@ type profileContextKey string

 const profileRegionContextKey profileContextKey = "profile_region"
 const forcedProfileIDContextKey profileContextKey = "forced_profile_id"
-const minimalProfileContextKey profileContextKey = "minimal_profile"

 func WithProfileRegion(ctx context.Context, region string) context.Context {
 	region = strings.TrimSpace(region)
@@ -41,12 +40,3 @@ func forcedProfileIDFromContext(ctx context.Context) string {
 	value, _ := EnsureContext(ctx).Value(forcedProfileIDContextKey).(string)
 	return strings.TrimSpace(value)
 }
-
-func WithMinimalBrowserProfile(ctx context.Context) context.Context {
-	return context.WithValue(EnsureContext(ctx), minimalProfileContextKey, true)
-}
-
-func minimalBrowserProfileFromContext(ctx context.Context) bool {
-	value, _ := EnsureContext(ctx).Value(minimalProfileContextKey).(bool)
-	return value
-}
--- a/core/proxy.go
+++ b/core/proxy.go
@@ -24,6 +24,10 @@ const (
 	// ProxyPoolQuarantineDuration is how long an exhausted tag pool stays quarantined
 	// before a single probe proxy is re-enabled for recovery testing.
 	ProxyPoolQuarantineDuration = 5 * time.Minute
+	// ProxyChallengeCooldown is how long a captcha/blocked proxy is deprioritized
+	// in rotation. It does not degrade health, so the proxy is still served if
+	// it's the only one left.
+	ProxyChallengeCooldown = 2 * time.Minute
 )

 var supportedProxySchemes = map[string]struct{}{
@@ -99,6 +103,9 @@ type proxyState struct {
 	tags     []string
 	failures int
 	disabled bool
+	// challengedUntil deprioritizes (but does not disable) this proxy in
+	// rotation after a captcha/block. See ReportChallenged.
+	challengedUntil time.Time
 }

 type ProxyRegistry struct {
@@ -411,21 +418,29 @@ func (r *ProxyRegistry) NextByTagWithContext(ctx context.Context, tag string) st
 		}
 	}

+	now := time.Now()
 	start := r.nextByTag[tag]
-	for i := 0; i < len(urls); i++ {
-		idx := (start + i) % len(urls)
-		proxyURL := urls[idx]
-		state := r.states[proxyURL]
-		if state.disabled {
-			continue
-		}
+	// First pass skips challenged proxies; second pass relaxes that so a
+	// challenged-but-healthy proxy is still served rather than failing.
+	for _, skipChallenged := range []bool{true, false} {
+		for i := 0; i < len(urls); i++ {
+			idx := (start + i) % len(urls)
+			proxyURL := urls[idx]
+			state := r.states[proxyURL]
+			if state.disabled {
+				continue
+			}
+			if skipChallenged && now.Before(state.challengedUntil) {
+				continue
+			}

-		r.nextByTag[tag] = (idx + 1) % len(urls)
-		WithRequest(ctx).WithFields(logrus.Fields{
-			"proxy_tag": tag,
-			"proxy":     MaskProxyURL(proxyURL),
-		}).Debugf("Selected proxy for tag=%s: %s", tag, MaskProxyURL(proxyURL))
-		return proxyURL
+			r.nextByTag[tag] = (idx + 1) % len(urls)
+			WithRequest(ctx).WithFields(logrus.Fields{
+				"proxy_tag": tag,
+				"proxy":     MaskProxyURL(proxyURL),
+			}).Debugf("Selected proxy for tag=%s: %s", tag, MaskProxyURL(proxyURL))
+			return proxyURL
+		}
 	}

 	// All proxies are disabled and no probe could be selected.
@@ -498,22 +513,50 @@ func (r *ProxyRegistry) ReportSuccess(_ context.Context, proxyURL string) {
 	}
 }

-func (r *ProxyRegistry) HasHealthyProxyForTag(tag string) bool {
-	tag = normalizeTag(tag)
-	if tag == "" {
-		return false
+// ReportChallenged deprioritizes a captcha/blocked proxy for
+// ProxyChallengeCooldown without degrading its health (unlike ReportFailure, it
+// never disables the proxy or trips quarantine), so the next attempt prefers a
+// different IP.
+func (r *ProxyRegistry) ReportChallenged(ctx context.Context, proxyURL string) {
+	proxyURL, err := NormalizeProxyURL(proxyURL)
+	if err != nil || proxyURL == "" {
+		return
 	}

 	r.mu.Lock()
 	defer r.mu.Unlock()

-	for _, proxyURL := range r.tagIndex[tag] {
-		if state, ok := r.states[proxyURL]; ok && !state.disabled {
-			return true
-		}
+	state, ok := r.states[proxyURL]
+	if !ok {
+		return
+	}
+	state.challengedUntil = time.Now().Add(ProxyChallengeCooldown)
+	WithRequest(ctx).WithField("proxy", MaskProxyURL(proxyURL)).
+		Debugf("Deprioritized challenged proxy for %s: %s", ProxyChallengeCooldown, MaskProxyURL(proxyURL))
+}
+
+func (r *ProxyRegistry) HasHealthyProxyForTag(tag string) bool {
+	return r.HealthyCountForTag(tag) > 0
+}
+
+// HealthyCountForTag returns how many non-disabled proxies the tag pool holds
+// (a challenged proxy still counts — it's usable, just deprioritized).
+func (r *ProxyRegistry) HealthyCountForTag(tag string) int {
+	tag = normalizeTag(tag)
+	if tag == "" {
+		return 0
 	}

-	return false
+	r.mu.Lock()
+	defer r.mu.Unlock()
+
+	count := 0
+	for _, proxyURL := range r.tagIndex[tag] {
+		if state, ok := r.states[proxyURL]; ok && !state.disabled {
+			count++
+		}
+	}
+	return count
 }

 func (r *ProxyRegistry) BuildStats() ProxyStats {
--- a/core/proxy_lane.go
+++ b/core/proxy_lane.go
@@ -224,6 +224,9 @@ func ProxyLaneKeyForTenant(engine string, tenant string, q Query, proxyURL strin
 	if sessionID == "" {
 		sessionID = proxyLaneIDFromProxyURL(proxyURL)
 	}
+	if sessionID == "" {
+		sessionID = "direct"
+	}
 	return NormalizeProxyLaneKey(ProxyLaneKey{Tenant: tenant, Engine: engine, SessionID: sessionID})
 }

@@ -236,11 +239,11 @@ func proxyLaneIDFromProxyURL(raw string) string {
 	if err != nil {
 		return ""
 	}
-	username := ""
+	userInfo := ""
 	if parsed.User != nil {
-		username = parsed.User.Username()
+		userInfo = parsed.User.String()
 	}
-	sum := sha256.Sum256([]byte(parsed.Host + "|" + username))
+	sum := sha256.Sum256([]byte(parsed.Scheme + "|" + parsed.Host + "|" + userInfo))
 	return hex.EncodeToString(sum[:])[:16]
 }

--- a/core/proxy_lane_test.go
+++ b/core/proxy_lane_test.go
@@ -1,6 +1,7 @@
 package core

 import (
+	"strings"
 	"testing"
 	"time"

@@ -77,14 +78,17 @@ func TestLaneStoreEvictsLRU(t *testing.T) {
 	}
 }

-func TestProxyLaneKeyForOmitsPassword(t *testing.T) {
+func TestProxyLaneKeyForUsesCredentialIdentity(t *testing.T) {
 	a := ProxyLaneKeyForTenant("Google", "", Query{}, "http://user:pass-a@proxy.example:8080")
 	b := ProxyLaneKeyForTenant("google", "", Query{}, "http://user:pass-b@proxy.example:8080")
 	if a.Empty() || b.Empty() {
 		t.Fatalf("expected derived lane keys, got %#v %#v", a, b)
 	}
-	if a != b {
-		t.Fatalf("expected password changes not to affect lane key: %#v %#v", a, b)
+	if a == b {
+		t.Fatalf("expected password changes to produce separate lane keys: %#v", a)
+	}
+	if strings.Contains(a.ID(), "pass-a") || strings.Contains(b.ID(), "pass-b") {
+		t.Fatalf("lane key leaked password material: %#v %#v", a, b)
 	}
 }

@@ -99,3 +103,13 @@ func TestProxyLaneKeyIncludesTenant(t *testing.T) {
 		t.Fatalf("unexpected tenant lane id: %q", got)
 	}
 }
+
+func TestProxyLaneKeyUsesStableDirectSession(t *testing.T) {
+	key := ProxyLaneKeyForTenant("google", "tenant-a", Query{}, "")
+	if key.Empty() {
+		t.Fatal("expected a direct lane")
+	}
+	if key.SessionID != "direct" {
+		t.Fatalf("SessionID = %q, want direct", key.SessionID)
+	}
+}
--- a/core/proxy_rotation_test.go
+++ b/core/proxy_rotation_test.go
@@ -0,0 +1,175 @@
+package core
+
+import (
+	"context"
+	"sync"
+	"testing"
+
+	"golang.org/x/time/rate"
+)
+
+// captchaThenSuccessEngine returns ErrCaptcha for every proxy URL except the
+// one designated as healthy, where it succeeds. It records the proxy URL of
+// each attempt so the test can assert rotation happened.
+type captchaThenSuccessEngine struct {
+	name        string
+	goodProxy   string
+	mu          sync.Mutex
+	seenProxies []string
+}
+
+func (e *captchaThenSuccessEngine) Name() string                  { return e.name }
+func (e *captchaThenSuccessEngine) IsInitialized() bool           { return true }
+func (e *captchaThenSuccessEngine) GetRateLimiter() *rate.Limiter { return nil }
+
+func (e *captchaThenSuccessEngine) Search(ctx context.Context, q Query) ([]SearchResult, error) {
+	e.mu.Lock()
+	e.seenProxies = append(e.seenProxies, q.ProxyURL)
+	e.mu.Unlock()
+	if q.ProxyURL == e.goodProxy {
+		return []SearchResult{{Title: "ok", URL: "https://example.com", Rank: 1}}, nil
+	}
+	return nil, ErrCaptcha
+}
+
+func (e *captchaThenSuccessEngine) SearchImage(ctx context.Context, q Query) ([]SearchResult, error) {
+	return e.Search(ctx, q)
+}
+
+func (e *captchaThenSuccessEngine) attempts() int {
+	e.mu.Lock()
+	defer e.mu.Unlock()
+	return len(e.seenProxies)
+}
+
+func tagPoolSearcher(t *testing.T, engine SearchEngine, entries []ProxyEntryConfig) *ResilientSearcher {
+	t.Helper()
+	cfg := DefaultResilientConfig()
+	cfg.Retry.MaxRetries = 0
+	cfg.CircuitBreaker.FailureThreshold = 5
+	cfg.Proxy = ProxyConfig{
+		Runtime:        ProxyRuntimeBrowser,
+		Proxies:        ProxiesConfig{Entries: entries},
+		EnginePolicies: map[string]string{engine.Name(): "rot"},
+	}
+	return NewResilientSearcher([]SearchEngine{engine}, cfg)
+}
+
+func TestReportChallengedDeprioritizesWithoutDisabling(t *testing.T) {
+	registry, err := NewProxyRegistry([]ProxyEntryConfig{
+		{URL: "http://proxy1:8080", Tags: []string{"rot"}},
+		{URL: "http://proxy2:8080", Tags: []string{"rot"}},
+	}, 3)
+	if err != nil {
+		t.Fatalf("new proxy registry: %v", err)
+	}
+	ctx := context.Background()
+
+	// Challenge proxy1 from a fresh index; the next selection must skip it.
+	registry.ReportChallenged(ctx, "http://proxy1:8080")
+	if got := registry.NextByTag("rot"); got != "http://proxy2:8080" {
+		t.Fatalf("expected challenged proxy to be skipped, got %q", got)
+	}
+
+	// Health is untouched: both proxies still count as healthy.
+	if n := registry.HealthyCountForTag("rot"); n != 2 {
+		t.Fatalf("challenge must not degrade health, healthy=%d", n)
+	}
+
+	// When both are challenged, rotation still serves one (relaxed second pass).
+	registry.ReportChallenged(ctx, "http://proxy2:8080")
+	if got := registry.NextByTag("rot"); got == "" {
+		t.Fatal("expected a proxy even when all are challenged")
+	}
+}
+
+func TestSearchWithProtection_RotatesProxyOnCaptcha(t *testing.T) {
+	// Two proxies in the same tag pool; the second one is the one that works.
+	// NextByTag serves proxy1 first, so the first attempt gets a captcha and the
+	// retry should pick proxy2 and succeed.
+	good := "http://proxy2:8080"
+	engine := &captchaThenSuccessEngine{name: "google", goodProxy: good}
+	rs := tagPoolSearcher(t, engine, []ProxyEntryConfig{
+		{URL: "http://proxy1:8080", Tags: []string{"rot"}},
+		{URL: good, Tags: []string{"rot"}},
+	})
+
+	results, _, meta, err := rs.SearchPrimary(context.Background(), engine, Query{Text: "rotate"})
+	if err != nil {
+		t.Fatalf("expected success after rotation, got %v", err)
+	}
+	if len(results) != 1 {
+		t.Fatalf("expected 1 result, got %d", len(results))
+	}
+	if meta.Attempts != 2 {
+		t.Fatalf("expected 2 proxy attempts, got %d", meta.Attempts)
+	}
+	if got := engine.attempts(); got != 2 {
+		t.Fatalf("expected engine called twice, got %d", got)
+	}
+}
+
+func TestSearchWithProtection_NoRotationWithSingleProxy(t *testing.T) {
+	// Only one proxy in the pool: captcha must fail fast without a second
+	// attempt (HealthyCountForTag < 2).
+	engine := &captchaThenSuccessEngine{name: "google", goodProxy: "http://unused:8080"}
+	rs := tagPoolSearcher(t, engine, []ProxyEntryConfig{
+		{URL: "http://proxy1:8080", Tags: []string{"rot"}},
+	})
+
+	_, _, meta, err := rs.SearchPrimary(context.Background(), engine, Query{Text: "single"})
+	if err == nil {
+		t.Fatal("expected captcha failure with a single proxy")
+	}
+	if meta.Attempts != 1 {
+		t.Fatalf("expected exactly 1 attempt with single proxy, got %d", meta.Attempts)
+	}
+	if got := engine.attempts(); got != 1 {
+		t.Fatalf("expected engine called once, got %d", got)
+	}
+}
+
+func TestSearchWithProtection_GlobalProxyChallengeReportsNoRotation(t *testing.T) {
+	engine := &captchaThenSuccessEngine{name: "google", goodProxy: "http://unused:8080"}
+	cfg := DefaultResilientConfig()
+	cfg.Retry.MaxRetries = 0
+	cfg.CircuitBreaker.FailureThreshold = 5
+	cfg.Proxy = ProxyConfig{
+		Runtime: ProxyRuntimeBrowser,
+		Proxies: ProxiesConfig{
+			Global: "http://global-proxy:8080",
+		},
+	}
+	rs := NewResilientSearcher([]SearchEngine{engine}, cfg)
+
+	_, _, meta, err := rs.SearchPrimary(context.Background(), engine, Query{Text: "global"})
+	if err == nil {
+		t.Fatal("expected captcha failure with a global proxy")
+	}
+	if meta.Attempts != 1 {
+		t.Fatalf("expected exactly 1 attempt with global proxy, got %d", meta.Attempts)
+	}
+	if got := engine.attempts(); got != 1 {
+		t.Fatalf("expected engine called once, got %d", got)
+	}
+}
+
+func TestSearchWithProtection_DirectModeFailsFastOnCaptcha(t *testing.T) {
+	// Direct mode (no proxy config): captcha is non-retryable and rotation must
+	// not kick in.
+	engine := &captchaThenSuccessEngine{name: "google", goodProxy: "http://never:8080"}
+	cfg := DefaultResilientConfig()
+	cfg.Retry.MaxRetries = 0
+	rs := NewResilientSearcher([]SearchEngine{engine}, cfg)
+
+	_, _, meta, err := rs.SearchPrimary(context.Background(), engine, Query{Text: "direct"})
+	if err == nil {
+		t.Fatal("expected captcha failure in direct mode")
+	}
+	if meta.Attempts > 1 {
+		t.Fatalf("direct mode must not rotate proxies, attempts=%d", meta.Attempts)
+	}
+	if got := engine.attempts(); got != 1 {
+		t.Fatalf("expected engine called once in direct mode, got %d", got)
+	}
+}
--- a/core/proxy_test.go
+++ b/core/proxy_test.go
@@ -15,7 +15,6 @@ import (
 	"time"

 	socks5 "github.com/armon/go-socks5"
-	xcontext "golang.org/x/net/context"
 )

 type timeoutTestError struct{}
@@ -344,7 +343,7 @@ func TestNewRawHTTPClientSocks5hUsesProxyDNS(t *testing.T) {
 	}
 }

-func TestNewRawHTTPClientDirectTLSUsesHTTP1(t *testing.T) {
+func TestNewRawHTTPClientDirectTLSNegotiatesHTTP2(t *testing.T) {
 	server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		_, _ = w.Write([]byte(r.Proto))
 	}))
@@ -367,8 +366,8 @@ func TestNewRawHTTPClientDirectTLSUsesHTTP1(t *testing.T) {
 	if err != nil {
 		t.Fatalf("read response body: %v", err)
 	}
-	if string(body) != "HTTP/1.1" {
-		t.Fatalf("expected raw client to use HTTP/1.1, got %q", string(body))
+	if string(body) != "HTTP/2.0" {
+		t.Fatalf("expected raw client to negotiate HTTP/2, got %q", string(body))
 	}
 }

@@ -380,6 +379,7 @@ func TestClassifyProxyNetworkError(t *testing.T) {
 	}{
 		{name: "timeout", err: timeoutTestError{}, want: ErrTimeout},
 		{name: "connect", err: errors.New("proxyconnect tcp: connection refused"), want: ErrProxyConnect},
+		{name: "chrome tunnel failed", err: errors.New("navigation failed: net::ERR_TUNNEL_CONNECTION_FAILED"), want: ErrProxyConnect},
 		{name: "auth", err: errors.New("Proxy Authentication Required 407"), want: ErrProxyAuth},
 		{name: "parser", err: ErrParser, want: nil},
 	}
@@ -438,7 +438,7 @@ type staticResolver struct {
 	ip   net.IP
 }

-func (r staticResolver) Resolve(ctx xcontext.Context, name string) (xcontext.Context, net.IP, error) {
+func (r staticResolver) Resolve(ctx context.Context, name string) (context.Context, net.IP, error) {
 	if name == r.host {
 		return ctx, r.ip, nil
 	}
--- a/core/query_extract_param_test.go
+++ b/core/query_extract_param_test.go
@@ -0,0 +1,73 @@
+package core
+
+import (
+	"net/http"
+	"net/http/httptest"
+	"strconv"
+	"testing"
+
+	"github.com/gofiber/fiber/v2"
+)
+
+// TestInitFromContextExtractParams verifies how the unified extract knob and its
+// tuning params map onto Query.Extract / Query.ExtractTop. Key behaviors:
+//   - extract is bool-or-int: extract=0/false off, extract=true/1 → top 1,
+//     extract=N → top N (clamped to [1,5]).
+//   - extract_mode/min_runes imply extraction (top defaults to 1), but an
+//     explicit extract=0 still wins over them.
+func TestInitFromContextExtractParams(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		name        string
+		query       string
+		wantExtract bool
+		wantTop     int
+	}{
+		{"no params defaults off", "?text=q", false, 1},
+		{"extract=true means top 1", "?text=q&extract=true", true, 1},
+		{"extract=1 means top 1", "?text=q&extract=1", true, 1},
+		{"extract=3 means top 3", "?text=q&extract=3", true, 3},
+		{"extract=N clamps high", "?text=q&extract=99", true, 5},
+		{"extract=0 disables", "?text=q&extract=0", false, 1},
+		{"extract=false disables", "?text=q&extract=false", false, 1},
+		{"extract_mode implies extract", "?text=q&extract_mode=fast", true, 1},
+		{"min_runes implies extract", "?text=q&min_runes=200", true, 1},
+		{"explicit extract=0 overrides tuning", "?text=q&extract=0&extract_mode=fast", false, 1},
+	}
+
+	app := fiber.New()
+	app.Get("/probe", func(c *fiber.Ctx) error {
+		q := Query{}
+		if err := q.InitFromContext(c); err != nil {
+			return c.Status(http.StatusBadRequest).SendString(err.Error())
+		}
+		extract := "0"
+		if q.Extract {
+			extract = "1"
+		}
+		c.Set("X-Extract", extract)
+		c.Set("X-Extract-Top", strconv.Itoa(q.ExtractTop))
+		return c.SendStatus(http.StatusOK)
+	})
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			req := httptest.NewRequest(http.MethodGet, "/probe"+tt.query, nil)
+			resp, err := app.Test(req, -1)
+			if err != nil {
+				t.Fatalf("request failed: %v", err)
+			}
+			if resp.StatusCode != http.StatusOK {
+				t.Fatalf("unexpected status %d for %s", resp.StatusCode, tt.query)
+			}
+			gotExtract := resp.Header.Get("X-Extract") == "1"
+			if gotExtract != tt.wantExtract {
+				t.Errorf("%s: Extract = %v, want %v", tt.query, gotExtract, tt.wantExtract)
+			}
+			if got := resp.Header.Get("X-Extract-Top"); got != strconv.Itoa(tt.wantTop) {
+				t.Errorf("%s: ExtractTop = %s, want %d", tt.query, got, tt.wantTop)
+			}
+		})
+	}
+}
--- a/core/rank_state.go
+++ b/core/rank_state.go
@@ -0,0 +1,54 @@
+package core
+
+// RankState tracks organic, ad, and absolute ranks for mixed SERP rows.
+type RankState struct {
+	organicRank  int
+	adRank       int
+	absoluteRank int
+}
+
+// NewRankState seeds ranks for a 0-based page on a 10-results-per-page engine.
+func NewRankState(pageNum int) *RankState {
+	return NewRankStateAt(pageNum*10, pageNum*10+1)
+}
+
+// NewRankStateAt seeds ranks from explicit organic and absolute bases.
+func NewRankStateAt(organicBase, absoluteBase int) *RankState {
+	return &RankState{
+		organicRank:  organicBase,
+		adRank:       1,
+		absoluteRank: absoluteBase,
+	}
+}
+
+// Next reserves the rank pair for the next emitted row.
+func (r *RankState) Next(isAd bool) (rank, absoluteRank int) {
+	absoluteRank = r.absoluteRank
+	r.absoluteRank++
+	if isAd {
+		rank = r.adRank
+		r.adRank++
+		return rank, absoluteRank
+	}
+	r.organicRank++
+	return r.organicRank, absoluteRank
+}
+
+// SetSeparatedAdAbsoluteRanks gives separated ad/organic passes one mixed order.
+func SetSeparatedAdAbsoluteRanks(results []SearchResult, start int) {
+	adCount := 0
+	for i := range results {
+		if results[i].Ad {
+			adCount++
+			results[i].AbsoluteRank = start + results[i].Rank
+		}
+	}
+	organicAbsoluteRank := start + adCount + 1
+	for i := range results {
+		if results[i].Ad {
+			continue
+		}
+		results[i].AbsoluteRank = organicAbsoluteRank
+		organicAbsoluteRank++
+	}
+}
--- a/core/resilient.go
+++ b/core/resilient.go
@@ -28,6 +28,9 @@ type ProxyExecutionMeta struct {
 	Mode string `json:"mode"`
 	Tag  string `json:"tag,omitempty"`
 	Used string `json:"used"`
+	// Attempts is how many proxies were tried; >1 means a challenged proxy was
+	// rotated out in tag-pool mode.
+	Attempts int `json:"attempts,omitempty"`
 }

 type ResilientConfig struct {
@@ -171,56 +174,88 @@ func (rs *ResilientSearcher) searchWithProtection(ctx context.Context, engine Se
 	attemptMeta := rs.baseProxyMeta(policy)

 	startedAt := time.Now()
-	result := RetryableSearch(ctx, rs.retryCfg, engine.Name(), func(callCtx context.Context) ([]SearchResult, error) {
-		limiter := engine.GetRateLimiter()
-		if limiter != nil {
-			if err := limiter.Wait(callCtx); err != nil {
-				return nil, normalizeLimiterWaitErr(callCtx, err)
-			}
-		}

-		attemptQuery := q
-		proxyURL := ""
-		reportToRegistry := false
-		attemptMeta = rs.baseProxyMeta(policy)
-
-		switch policy.Mode {
-		case ProxyModeOff:
-			attemptQuery.ProxyURL = ""
-			attemptMeta.Used = "direct"
-		case ProxyModeRequestURL:
-			proxyURL = q.ProxyURL
-			attemptQuery.ProxyURL = proxyURL
-			attemptMeta.Used = MaskProxyURL(proxyURL)
-		case ProxyModeTagPool:
-			proxyURL = rs.selectProxyForQuery(policy, q, engineCtx)
-			if proxyURL == "" {
-				return nil, fmt.Errorf("%w: no healthy proxy available for tag %q", ErrProxyUnavailable, policy.Tag)
-			}
-			attemptQuery.ProxyURL = proxyURL
-			reportToRegistry = policy.Tag != ""
-			attemptMeta.Used = MaskProxyURL(proxyURL)
-		}
-
-		requestCtx := proxyRequestContext(callCtx, engine.Name(), attemptQuery)
-		results, err := invokeEngine(requestCtx, engine, attemptQuery, isImage)
-
-		if reportToRegistry {
-			rs.reportProxyAttempt(engineCtx, proxyURL, err)
-		}
-		if err != nil && errors.Is(err, ErrCaptcha) && rs.proxyCfg.Proxies.Lanes.DropCookiesOnChallenge {
-			// Recompute lane key only to gate the call: empty key means we have no
-			// session to drop cookies for. The dropper recomputes the key itself
-			// when it actually needs to mutate lane state.
-			if !ProxyLaneKeyForTenant(engine.Name(), TenantFromContext(callCtx), attemptQuery, attemptQuery.ProxyURL).Empty() {
-				if dropper, ok := engine.(proxyLaneCookieDropper); ok {
-					dropper.DropProxyLaneCookies(callCtx, attemptQuery)
+	// lastProxyURL is the unmasked proxy of the last attempt, for rotation below.
+	lastProxyURL := ""
+	runOnce := func() RetryResult {
+		return RetryableSearch(ctx, rs.retryCfg, engine.Name(), func(callCtx context.Context) ([]SearchResult, error) {
+			limiter := engine.GetRateLimiter()
+			if limiter != nil {
+				if err := limiter.Wait(callCtx); err != nil {
+					return nil, normalizeLimiterWaitErr(callCtx, err)
 				}
 			}
-		}

-		return results, err
-	})
+			attemptQuery := q
+			proxyURL := ""
+			reportToRegistry := false
+			attemptMeta = rs.baseProxyMeta(policy)
+
+			switch policy.Mode {
+			case ProxyModeOff:
+				attemptQuery.ProxyURL = ""
+				attemptMeta.Used = "direct"
+			case ProxyModeRequestURL:
+				proxyURL = q.ProxyURL
+				attemptQuery.ProxyURL = proxyURL
+				attemptMeta.Used = MaskProxyURL(proxyURL)
+			case ProxyModeTagPool:
+				proxyURL = rs.selectProxyForQuery(policy, q, engineCtx)
+				if proxyURL == "" {
+					return nil, fmt.Errorf("%w: no healthy proxy available for tag %q", ErrProxyUnavailable, policy.Tag)
+				}
+				attemptQuery.ProxyURL = proxyURL
+				reportToRegistry = policy.Tag != ""
+				attemptMeta.Used = MaskProxyURL(proxyURL)
+			}
+			lastProxyURL = proxyURL
+
+			requestCtx := proxyRequestContext(callCtx, engine.Name(), attemptQuery)
+			results, err := invokeEngine(requestCtx, engine, attemptQuery, isImage)
+
+			if reportToRegistry {
+				rs.reportProxyAttempt(engineCtx, proxyURL, err)
+			}
+			if err != nil && errors.Is(err, ErrCaptcha) && rs.proxyCfg.Proxies.Lanes.DropCookiesOnChallenge {
+				// Recompute lane key only to gate the call: empty key means we have no
+				// session to drop cookies for. The dropper recomputes the key itself
+				// when it actually needs to mutate lane state.
+				if !ProxyLaneKeyForTenant(engine.Name(), TenantFromContext(callCtx), attemptQuery, attemptQuery.ProxyURL).Empty() {
+					if dropper, ok := engine.(proxyLaneCookieDropper); ok {
+						dropper.DropProxyLaneCookies(callCtx, attemptQuery)
+					}
+				}
+			}
+
+			return results, err
+		})
+	}
+
+	result := runOnce()
+	attemptMeta.Attempts = 1
+
+	// On a captcha/block/rate-limit (non-retryable inside RetryableSearch), if
+	// the tag pool has another healthy proxy, deprioritize the burned one and
+	// retry once with the next. Tag-pool only - direct/request-url/global can't
+	// rotate.
+	canRotateChallengedProxy := result.Err != nil &&
+		policy.Mode == ProxyModeTagPool &&
+		policy.Tag != "" &&
+		rs.proxyRegistry != nil &&
+		IsProxyChallengeError(result.Err) &&
+		rs.proxyRegistry.HealthyCountForTag(policy.Tag) >= 2 &&
+		ctx.Err() == nil
+	if canRotateChallengedProxy {
+		rs.proxyRegistry.ReportChallenged(engineCtx, lastProxyURL)
+		WithRequestEngine(ctx, engine.Name()).WithError(result.Err).
+			Debug("Challenged proxy rotated out, retrying once with next proxy")
+		result = runOnce()
+		attemptMeta.Attempts = 2
+	} else if result.Err != nil && IsProxyChallengeError(result.Err) && strings.TrimSpace(lastProxyURL) != "" {
+		WithRequestEngine(ctx, engine.Name()).WithError(result.Err).
+			WithField("proxy", MaskProxyURL(lastProxyURL)).
+			Info("single proxy challenged and cannot rotate; configure proxies.entries with 2+ IPs")
+	}

 	if result.Err != nil {
 		if shouldRecordCircuitFailure(result.Err) {
@@ -233,6 +268,14 @@ func (rs *ResilientSearcher) searchWithProtection(ctx context.Context, engine Se
 	return result.Results, attemptMeta, nil
 }

+// IsProxyChallengeError reports whether err is a captcha/block/rate-limit — an
+// IP-reputation problem another proxy might dodge.
+func IsProxyChallengeError(err error) bool {
+	return errors.Is(err, ErrCaptcha) ||
+		errors.Is(err, ErrBlocked) ||
+		errors.Is(err, ErrRateLimited)
+}
+
 func shouldRecordCircuitFailure(err error) bool {
 	return err != nil &&
 		!IsContextDone(err) &&
--- a/core/response_builder.go
+++ b/core/response_builder.go
@@ -19,6 +19,17 @@ var imageDimensionPatterns = []*regexp.Regexp{
 	regexp.MustCompile(`(?i)\b(\d+)x(\d+)\b`),
 }

+var featureBaseURLs = map[string]string{
+	"google":     "https://www.google.com/",
+	"bing":       "https://www.bing.com/",
+	"duckduckgo": "https://duckduckgo.com/",
+	"duck":       "https://duckduckgo.com/",
+	"ddg":        "https://duckduckgo.com/",
+	"ecosia":     "https://www.ecosia.org/",
+	"yandex":     "https://www.yandex.com/",
+	"baidu":      "https://www.baidu.com/",
+}
+
 // EnrichContext carries request-scoped values needed to enrich a raw result.
 type EnrichContext struct {
 	Engine string
@@ -124,10 +135,16 @@ func EnrichSerpFeature(raw SerpFeature, engine string, sourceResultID string, ex
 		feature.SourceResultIDs = append(feature.SourceResultIDs, sourceResultID)
 	}
 	for i := range feature.Links {
-		feature.Links[i].URL = normalizeURL(feature.Links[i].URL)
+		feature.Links[i].Title = cleanFeatureText(feature.Links[i].Title)
+		feature.Links[i].URL = normalizeFeatureURL(feature.Links[i].URL, engine)
 	}
 	for i := range feature.Items {
-		feature.Items[i].Link = normalizeURL(feature.Items[i].Link)
+		feature.Items[i].Title = cleanFeatureText(feature.Items[i].Title)
+		feature.Items[i].Text = cleanFeatureText(feature.Items[i].Text)
+		if feature.Items[i].Title == "" {
+			feature.Items[i].Title = feature.Items[i].Text
+		}
+		feature.Items[i].Link = normalizeFeatureURL(feature.Items[i].Link, engine)
 	}
 	if feature.ID == "" {
 		feature.ID = buildFeatureID(feature)
@@ -138,6 +155,37 @@ func EnrichSerpFeature(raw SerpFeature, engine string, sourceResultID string, ex
 	return feature
 }

+func normalizeFeatureURL(raw string, engine string) string {
+	raw = strings.TrimSpace(invisibleFormatChars.Replace(raw))
+	if raw == "" {
+		return ""
+	}
+	if strings.HasPrefix(raw, "//") {
+		raw = "https:" + raw
+	}
+
+	parsed, err := url.Parse(raw)
+	if err != nil {
+		return raw
+	}
+	if parsed.IsAbs() {
+		if strings.EqualFold(parsed.Scheme, "http") || strings.EqualFold(parsed.Scheme, "https") {
+			return normalizeURL(raw)
+		}
+		return raw
+	}
+
+	base, ok := featureBaseURLs[strings.ToLower(strings.TrimSpace(engine))]
+	if !ok {
+		return normalizeURL(raw)
+	}
+	baseURL, err := url.Parse(base)
+	if err != nil {
+		return normalizeURL(raw)
+	}
+	return normalizeURL(baseURL.ResolveReference(parsed).String())
+}
+
 func buildFeatureID(feature SerpFeature) string {
 	primaryLink := ""
 	if len(feature.Links) > 0 {
@@ -444,7 +492,10 @@ func ValidateResultType(t ResultType) (ResultType, string) {
 	case ResultTypeOrganic, ResultTypeAd, ResultTypeFeaturedSnippet,
 		ResultTypeKnowledgePanel, ResultTypePeopleAlsoAsk, ResultTypeVideo,
 		ResultTypeImage, ResultTypeNews, ResultTypeShopping,
-		ResultTypeLocal, ResultTypeAnswerBox:
+		ResultTypeLocal, ResultTypeAnswerBox, ResultTypeAISummary,
+		ResultTypeRelatedQuestions, ResultTypeRelatedSearches,
+		ResultTypeSitelinks, ResultTypeVideos, ResultTypeImagesInline,
+		ResultTypeCalculator, ResultTypeWeather, ResultTypeDictionary:
 		return t, ""
 	}
 	return ResultTypeOrganic, fmt.Sprintf("unknown result type %q, defaulting to organic", t)
--- a/core/result_rank_test.go
+++ b/core/result_rank_test.go
@@ -101,6 +101,88 @@ func TestEnrichResultUsesExplicitResultType(t *testing.T) {
 	}
 }

+func TestValidateResultTypeAcceptsFullTaxonomy(t *testing.T) {
+	t.Parallel()
+
+	types := []ResultType{
+		ResultTypeOrganic,
+		ResultTypeAd,
+		ResultTypeFeaturedSnippet,
+		ResultTypeKnowledgePanel,
+		ResultTypePeopleAlsoAsk,
+		ResultTypeVideo,
+		ResultTypeImage,
+		ResultTypeNews,
+		ResultTypeShopping,
+		ResultTypeLocal,
+		ResultTypeAnswerBox,
+		ResultTypeAISummary,
+		ResultTypeRelatedQuestions,
+		ResultTypeRelatedSearches,
+		ResultTypeSitelinks,
+		ResultTypeVideos,
+		ResultTypeImagesInline,
+		ResultTypeCalculator,
+		ResultTypeWeather,
+		ResultTypeDictionary,
+	}
+
+	for _, typ := range types {
+		got, warning := ValidateResultType(typ)
+		if got != typ || warning != "" {
+			t.Fatalf("ValidateResultType(%q) = (%q, %q), want (%q, empty)", typ, got, warning, typ)
+		}
+	}
+}
+
+func TestRankStateInterleavesAdsAndSeedsPages(t *testing.T) {
+	t.Parallel()
+
+	// Page 1 (0-based): organic ranks continue at 11, ad ranks always restart at
+	// 1, and the absolute rank counts every emitted row regardless of kind.
+	rank := NewRankState(1)
+
+	steps := []struct {
+		isAd           bool
+		rank, absolute int
+	}{
+		{true, 1, 11},   // ad
+		{false, 11, 12}, // organic (seeded from page*10)
+		{false, 12, 13}, // organic
+		{true, 2, 14},   // ad interleaved after organics
+		{false, 13, 15}, // organic
+	}
+	for i, s := range steps {
+		gotRank, gotAbs := rank.Next(s.isAd)
+		if gotRank != s.rank || gotAbs != s.absolute {
+			t.Fatalf("step %d (ad=%v): got rank=%d absolute=%d, want rank=%d absolute=%d",
+				i, s.isAd, gotRank, gotAbs, s.rank, s.absolute)
+		}
+	}
+}
+
+func TestSetSeparatedAdAbsoluteRanks(t *testing.T) {
+	t.Parallel()
+
+	// Ecosia collects ads and organics in separate passes (each rank-1-based),
+	// then this assigns one mixed absolute order: ads first, then organics.
+	results := []SearchResult{
+		{Rank: 1, URL: "https://organic.example.com/one"},
+		{Rank: 2, URL: "https://organic.example.com/two"},
+		{Rank: 1, Ad: true, URL: "https://ads.example.com/one"},
+		{Rank: 2, Ad: true, URL: "https://ads.example.com/two"},
+	}
+
+	SetSeparatedAdAbsoluteRanks(results, 0)
+
+	want := []int{3, 4, 1, 2} // ads (passes 1,2) precede organics (3,4) in absolute order
+	for i, r := range results {
+		if r.AbsoluteRank != want[i] {
+			t.Fatalf("%s absolute rank = %d, want %d", r.URL, r.AbsoluteRank, want[i])
+		}
+	}
+}
+
 func TestEnvelopePaginationCountsOrganicResults(t *testing.T) {
 	t.Parallel()

--- a/core/retry.go
+++ b/core/retry.go
@@ -131,7 +131,7 @@ var nonRetryableSentinels = []struct {
 }{
 	{ErrCaptcha, "CAPTCHA detected"},
 	{ErrBlocked, "Blocked response detected"},
-	{ErrRateLimited, "Rate limited response detected"},
+	{ErrRateLimited, "HTTP 429 block detected"},
 	{ErrProxyUnavailable, "Proxy unavailable"},
 	{ErrParser, "Parser failure"},
 	{ErrEngineInternal, "Engine panic recovered"},
--- a/core/serp_features_test.go
+++ b/core/serp_features_test.go
@@ -5,6 +5,8 @@ import (
 	"strings"
 	"testing"
 	"time"
+
+	"github.com/PuerkitoBio/goquery"
 )

 func TestEnvelopeAlwaysIncludesSerpFeatures(t *testing.T) {
@@ -90,6 +92,95 @@ func TestAppendEnrichedSearchResultMirrorsExistingAnswerResultAsFeature(t *testi
 	}
 }

+func TestStripResultFeatures(t *testing.T) {
+	results := []SearchResult{
+		{
+			Rank:  1,
+			URL:   "https://example.com",
+			Title: "Example",
+			Features: []SerpFeature{{
+				Type: ResultTypeRelatedSearches,
+				Items: []FeatureItem{
+					{Text: "example search"},
+				},
+			}},
+		},
+		{
+			Rank:  2,
+			URL:   "https://example.org",
+			Title: "Example Org",
+		},
+	}
+
+	if kept := StripResultFeatures(results, true); len(kept[0].Features) != 1 {
+		t.Fatalf("keep=true must preserve features, got %#v", kept[0].Features)
+	}
+
+	stripped := StripResultFeatures(results, false)
+
+	if len(stripped) != 2 {
+		t.Fatalf("expected result count to be preserved, got %d", len(stripped))
+	}
+	for i, result := range stripped {
+		if len(result.Features) != 0 {
+			t.Fatalf("result %d kept features: %#v", i, result.Features)
+		}
+	}
+	if stripped[0].URL != "https://example.com" || stripped[1].Rank != 2 {
+		t.Fatalf("non-feature fields changed: %#v", stripped)
+	}
+}
+
+func TestFeatureItemsUseTextAsTitleAndStripInvisibleCharacters(t *testing.T) {
+	doc, err := goquery.NewDocumentFromReader(strings.NewReader(`
+<div class="related">
+  <a href="?q=python+coding">python coding&#8203;</a>
+</div>`))
+	if err != nil {
+		t.Fatalf("parse fixture: %v", err)
+	}
+
+	features := ExtractSerpFeaturesBySelectors(doc, []SerpFeatureSelector{{
+		Type:         ResultTypeRelatedSearches,
+		Title:        "Related searches",
+		Container:    []string{".related"},
+		ItemSelector: []string{"a"},
+		LinkSelector: []string{"a"},
+	}})
+
+	if len(features) != 1 || len(features[0].Items) != 1 {
+		t.Fatalf("expected one feature item, got %#v", features)
+	}
+	item := features[0].Items[0]
+	if item.Title != "python coding" || item.Text != "python coding" {
+		t.Fatalf("unexpected item text/title: %#v", item)
+	}
+}
+
+func TestEnrichSerpFeatureNormalizesRelativeLinks(t *testing.T) {
+	feature := EnrichSerpFeature(SerpFeature{
+		Type: ResultTypeRelatedSearches,
+		Items: []FeatureItem{{
+			Text: "python coding\u200b",
+			Link: "?q=python+coding&t=h",
+		}},
+		Links: []FeatureLink{{
+			Title: "baidu related",
+			URL:   "/s?wd=python",
+		}},
+	}, "duckduckgo", "", time.Unix(0, 0))
+
+	if feature.Items[0].Title != "python coding" || feature.Items[0].Text != "python coding" {
+		t.Fatalf("unexpected enriched item: %#v", feature.Items[0])
+	}
+	if got, want := feature.Items[0].Link, "https://duckduckgo.com/?q=python+coding&t=h"; got != want {
+		t.Fatalf("item link = %q, want %q", got, want)
+	}
+	if got, want := feature.Links[0].URL, "https://duckduckgo.com/s?wd=python"; got != want {
+		t.Fatalf("feature link = %q, want %q", got, want)
+	}
+}
+
 func TestRenderersIncludeSerpFeatures(t *testing.T) {
 	env := NewEnvelope(Query{Text: "openserp"}, "req-1", time.Unix(0, 0), []string{"google"})
 	AppendEnrichedSearchResult(env, SearchResult{
--- a/core/server.go
+++ b/core/server.go
@@ -195,18 +195,21 @@ func NewServerWithOptions(host string, port int, opts ServerOptions, searchEngin
 	for _, engine := range searchEngines {
 		locEngine := engine

-		endpointName := strings.ToLower(locEngine.Name())
-		if endpointName == "duckduckgo" {
-			endpointName = "duck"
+		endpointName := engineEndpointName(locEngine.Name())
+
+		searchHandler := func(c *fiber.Ctx) error {
+			return serv.handleDedicatedEndpoint(c, locEngine, false)
+		}
+		imageHandler := func(c *fiber.Ctx) error {
+			return serv.handleDedicatedEndpoint(c, locEngine, true)
 		}

-		serv.app.Get(fmt.Sprintf("/%s/search", endpointName), func(c *fiber.Ctx) error {
-			return serv.handleDedicatedEndpoint(c, locEngine, false)
-		})
-
-		serv.app.Get(fmt.Sprintf("/%s/image", endpointName), func(c *fiber.Ctx) error {
-			return serv.handleDedicatedEndpoint(c, locEngine, true)
-		})
+		serv.app.Get(fmt.Sprintf("/%s/search", endpointName), searchHandler)
+		serv.app.Get(fmt.Sprintf("/%s/image", endpointName), imageHandler)
+		if canonicalName := strings.ToLower(locEngine.Name()); canonicalName != endpointName {
+			serv.app.Get(fmt.Sprintf("/%s/search", canonicalName), searchHandler)
+			serv.app.Get(fmt.Sprintf("/%s/image", canonicalName), imageHandler)
+		}
 	}

 	for _, engine := range searchEngines {
@@ -215,15 +218,15 @@ func NewServerWithOptions(host string, port int, opts ServerOptions, searchEngin
 			continue
 		}
 		locParser := parser
-		parserEndpointName := strings.ToLower(parser.Name())
-		serv.app.Post(fmt.Sprintf("/%s/parse", parserEndpointName),
-			func(c *fiber.Ctx) error {
-				return serv.handleParseEndpoint(c, locParser)
-			})
-		if parserEndpointName == "duckduckgo" {
-			serv.app.Post("/duck/parse", func(c *fiber.Ctx) error {
-				return serv.handleParseEndpoint(c, locParser)
-			})
+		// Parse registers the canonical-name path, plus the endpoint slug when it
+		// differs (e.g. /duckduckgo/parse and /duck/parse both resolve).
+		parserName := strings.ToLower(parser.Name())
+		parseHandler := func(c *fiber.Ctx) error {
+			return serv.handleParseEndpoint(c, locParser)
+		}
+		serv.app.Post(fmt.Sprintf("/%s/parse", parserName), parseHandler)
+		if slug := engineEndpointName(parser.Name()); slug != parserName {
+			serv.app.Post(fmt.Sprintf("/%s/parse", slug), parseHandler)
 		}
 	}

@@ -232,6 +235,7 @@ func NewServerWithOptions(host string, port int, opts ServerOptions, searchEngin
 	serv.app.Get("/mega/engines", serv.handleListEngines)
 	serv.app.Get("/extract", serv.handleExtract)
 	serv.app.Post("/extract", serv.handleExtract)
+	serv.app.Post("/extract/batch", serv.handleBatchExtract)

 	return &serv
 }
@@ -402,6 +406,18 @@ func (s *Server) handleParseEndpoint(c *fiber.Ctx, parser HTMLParser) error {

 	results, err := parser.ParseHTML(bytes.NewReader(body))
 	if err != nil {
+		if isParseEndpointSearchError(err) {
+			spec := mapSearchError(err)
+			return &APIError{
+				HTTPStatus: spec.status,
+				ErrorCode:  spec.code,
+				Message:    spec.message,
+				Meta: map[string]any{
+					"engine":       parser.Name(),
+					"error_detail": err.Error(),
+				},
+			}
+		}
 		return &APIError{
 			HTTPStatus: fiber.StatusBadRequest,
 			ErrorCode:  "parser_failure",
@@ -421,6 +437,14 @@ func (s *Server) handleParseEndpoint(c *fiber.Ctx, parser HTMLParser) error {
 	return sendEnvelope(c, format, env)
 }

+func isParseEndpointSearchError(err error) bool {
+	return errors.Is(err, ErrCaptcha) ||
+		errors.Is(err, ErrBlocked) ||
+		errors.Is(err, ErrRateLimited) ||
+		errors.Is(err, ErrSearchTimeout) ||
+		errors.Is(err, ErrParser)
+}
+
 type searchErrorSpec struct {
 	status  int
 	code    string
@@ -434,7 +458,7 @@ func mapSearchError(err error) searchErrorSpec {
 	case errors.Is(err, ErrBlocked):
 		return searchErrorSpec{status: fiber.StatusForbidden, code: "blocked", message: "search engine blocked the request"}
 	case errors.Is(err, ErrRateLimited):
-		return searchErrorSpec{status: fiber.StatusTooManyRequests, code: "rate_limited", message: "search engine rate limited the request"}
+		return searchErrorSpec{status: fiber.StatusTooManyRequests, code: "blocked", message: "search engine blocked the request (HTTP 429)"}
 	case errors.Is(err, ErrSearchTimeout):
 		return searchErrorSpec{status: fiber.StatusGatewayTimeout, code: "search_timeout", message: ErrSearchTimeout.Error()}
 	case errors.Is(err, ErrProxyAuth):
@@ -464,6 +488,9 @@ func mapSearchError(err error) searchErrorSpec {
 func searchAPIError(err error, engineName string, q Query, proxyMeta ProxyExecutionMeta) *APIError {
 	spec := mapSearchError(err)
 	meta := searchErrorMeta(engineName, q, proxyMeta)
+	if errors.Is(err, ErrRateLimited) {
+		meta["upstream_status"] = fiber.StatusTooManyRequests
+	}
 	addErrorDetail(meta, err, spec.message, q)
 	return &APIError{
 		HTTPStatus: spec.status,
@@ -1012,6 +1039,9 @@ func (s *Server) handleMegaEndpoint(c *fiber.Ctx, action string) error {
 	if len(responded) == 0 {
 		err := fmt.Errorf("%w: %s", ErrAllEnginesFailed, strings.Join(enginesFailed, ","))
 		apiErr := searchAPIError(err, "mega", q, ProxyExecutionMeta{})
+		if detail := megaFailureMessage(engineErrors); detail != "" {
+			apiErr.Message = detail
+		}
 		apiErr.Meta["engine_errors"] = engineErrors
 		WithRequest(requestCtx).WithFields(logrus.Fields{
 			"action": action, "engines": engineNamesJoined,
@@ -1092,6 +1122,19 @@ func engineErrorNames(details []EngineErrorDetail) []string {
 	return names
 }

+func megaFailureMessage(details []EngineErrorDetail) string {
+	if len(details) == 1 {
+		detail := strings.TrimSpace(details[0].Message)
+		if detail == "" {
+			detail = details[0].Error
+		}
+		if detail != "" {
+			return fmt.Sprintf("all selected engines failed; %s: %s", details[0].Engine, detail)
+		}
+	}
+	return ""
+}
+
 func parseMegaRunConfig(c *fiber.Ctx) (megaRunConfig, error) {
 	cfg := megaRunConfig{
 		Mode:   megaModeBalanced,
@@ -1216,12 +1259,20 @@ func (s *Server) resolveEngines(ctx context.Context, enginesParam string) []Sear
 	return enginesToUse
 }

+func engineEndpointName(name string) string {
+	name = strings.ToLower(name)
+	if name == "duckduckgo" {
+		return "duck"
+	}
+	return name
+}
+
 func resolveEngineAlias(name string) string {
-	switch name {
+	switch strings.ToLower(name) {
 	case "duck", "ddg":
 		return "duckduckgo"
 	default:
-		return name
+		return strings.ToLower(name)
 	}
 }

@@ -1451,6 +1502,9 @@ func (s *Server) applyProxyHeaders(c *fiber.Ctx, meta ProxyExecutionMeta) {
 		c.Set("X-Proxy-Tag", tag)
 	}
 	c.Set("X-Proxy-Used", used)
+	if meta.Attempts > 1 {
+		c.Set("X-Proxy-Attempts", strconv.Itoa(meta.Attempts))
+	}
 }

 func setNetworkBytesHeader(c *fiber.Ctx, ctx context.Context) {
--- a/core/server_extract.go
+++ b/core/server_extract.go
@@ -7,6 +7,7 @@ import (
 	"fmt"
 	"io"
 	"net/http"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
@@ -20,9 +21,10 @@ type extractPayload struct {
 	Mode string `json:"mode"`
 	// Clean defaults to true (article-only). Pointer so we can tell "omitted"
 	// (use default) from an explicit false (full-page extraction).
-	Clean      *bool `json:"clean"`
-	UseLLMSTxt bool  `json:"use_llms_txt"`
-	MinRunes   int   `json:"min_runes"`
+	Clean      *bool  `json:"clean"`
+	UseLLMSTxt bool   `json:"use_llms_txt"`
+	MinRunes   int    `json:"min_runes"`
+	Lang       string `json:"lang"`
 }

 func (s *Server) handleExtract(c *fiber.Ctx) error {
@@ -57,11 +59,10 @@ func (s *Server) handleExtract(c *fiber.Ctx) error {
 	return sendExtractResult(c, format, result)
 }

-func (s *Server) extractRequestFromFiber(c *fiber.Ctx, cfg extractpkg.Config) (extractpkg.ExtractRequest, error) {
-	var body extractPayload
-	if len(c.Body()) > 0 {
-		_ = c.BodyParser(&body)
-	}
+// baseExtractRequest builds the URL-independent part of an extract request
+// from proxy headers, query params, and the parsed body. Shared by /extract
+// and /extract/batch so both accept the same knobs.
+func (s *Server) baseExtractRequest(c *fiber.Ctx, body extractPayload, cfg extractpkg.Config) (extractpkg.ExtractRequest, error) {
 	proxyOverride, err := NormalizeProxyRequestOverride(c.Get("X-Use-Proxy"))
 	if err != nil {
 		return extractpkg.ExtractRequest{}, errInvalidParam(fmt.Sprintf("X-Use-Proxy: %v", err))
@@ -78,7 +79,12 @@ func (s *Server) extractRequestFromFiber(c *fiber.Ctx, cfg extractpkg.Config) (e
 	if err := s.validateRequestProxyURL(&q); err != nil {
 		return extractpkg.ExtractRequest{}, err
 	}
-	mode := firstNonEmpty(c.Query("mode"), body.Mode, cfg.DefaultMode)
+	mode := extractpkg.Mode(strings.ToLower(firstNonEmpty(c.Query("mode"), body.Mode, cfg.DefaultMode)))
+	switch mode {
+	case extractpkg.ModeAuto, extractpkg.ModeFast, extractpkg.ModeRendered:
+	default:
+		return extractpkg.ExtractRequest{}, errInvalidParam("mode must be one of auto, fast, rendered")
+	}
 	// Default clean=true (article-only). FullPage is the inverse: full-readable-body
 	// extraction, opted in via clean=false on the query string or body.
 	bodyClean := true
@@ -90,15 +96,10 @@ func (s *Server) extractRequestFromFiber(c *fiber.Ctx, cfg extractpkg.Config) (e
 	if err != nil {
 		return extractpkg.ExtractRequest{}, errInvalidParam("min_runes must be a non-negative integer")
 	}
-	targetURL := extractpkg.NormalizeURL(strings.TrimSpace(firstNonEmpty(c.Query("url"), body.URL)))
-	if err := validateExtractTargetURL(c.UserContext(), targetURL, cfg.AllowPrivateNetworks); err != nil {
-		return extractpkg.ExtractRequest{}, errInvalidParam(err.Error())
-	}
 	return extractpkg.ExtractRequest{
-		URL:        targetURL,
-		Mode:       extractpkg.Mode(mode),
+		Mode:       mode,
 		ProxyURL:   proxyURL,
-		LangCode:   strings.TrimSpace(c.Query("lang")),
+		LangCode:   firstNonEmpty(body.Lang, c.Query("lang")),
 		Timeout:    cfg.Timeout,
 		MaxBytes:   cfg.MaxBytes,
 		FullPage:   !clean,
@@ -107,6 +108,22 @@ func (s *Server) extractRequestFromFiber(c *fiber.Ctx, cfg extractpkg.Config) (e
 	}, nil
 }

+func (s *Server) extractRequestFromFiber(c *fiber.Ctx, cfg extractpkg.Config) (extractpkg.ExtractRequest, error) {
+	var body extractPayload
+	if len(c.Body()) > 0 {
+		_ = c.BodyParser(&body)
+	}
+	req, err := s.baseExtractRequest(c, body, cfg)
+	if err != nil {
+		return extractpkg.ExtractRequest{}, err
+	}
+	req.URL = extractpkg.NormalizeURL(strings.TrimSpace(firstNonEmpty(c.Query("url"), body.URL)))
+	if err := validateExtractTargetURL(c.UserContext(), req.URL, cfg.AllowPrivateNetworks); err != nil {
+		return extractpkg.ExtractRequest{}, errInvalidParam(err.Error())
+	}
+	return req, nil
+}
+
 func (s *Server) newExtractor() extractpkg.Extractor {
 	return extractpkg.Extractor{
 		RawFetch:      s.rawExtractFetch,
@@ -116,14 +133,21 @@ func (s *Server) newExtractor() extractpkg.Extractor {
 }

 func (s *Server) rawExtractFetch(ctx context.Context, req extractpkg.ExtractRequest) (*extractpkg.FetchResponse, error) {
-	cfg := s.opts.Extract.Normalized()
+	return RawExtractFetch(ctx, req, s.opts.Extract, s.opts.FingerprintBrowserOpts.Insecure)
+}
+
+// RawExtractFetch performs the browserless extraction fetch: validate the
+// target, issue a guarded HTTP GET, classify the status, and return the body
+// capped to the byte budget. Shared by the HTTP server and the CLI.
+func RawExtractFetch(ctx context.Context, req extractpkg.ExtractRequest, cfg extractpkg.Config, insecure bool) (*extractpkg.FetchResponse, error) {
+	cfg = cfg.Normalized()
 	if err := validateExtractTargetURL(ctx, req.URL, cfg.AllowPrivateNetworks); err != nil {
 		return nil, err
 	}
 	resp, err := RawSearchRequest(ctx, req.URL, Query{
 		ProxyURL:             req.ProxyURL,
 		LangCode:             req.LangCode,
-		Insecure:             s.opts.FingerprintBrowserOpts.Insecure,
+		Insecure:             insecure,
 		GuardPrivateNetworks: !cfg.AllowPrivateNetworks,
 	})
 	if err != nil {
@@ -135,7 +159,7 @@ func (s *Server) rawExtractFetch(ctx context.Context, req extractpkg.ExtractRequ
 	}
 	limit := int64(req.MaxBytes)
 	if limit <= 0 {
-		limit = int64(s.opts.Extract.Normalized().MaxBytes)
+		limit = int64(cfg.MaxBytes)
 	}
 	body, err := io.ReadAll(io.LimitReader(resp.Body, limit+1))
 	if err != nil {
@@ -159,7 +183,15 @@ func (s *Server) renderedExtractFetch(ctx context.Context, req extractpkg.Extrac
 	if err != nil {
 		return nil, err
 	}
-	page, err := browser.Navigate(WithRequestProxyURL(ctx, req.ProxyURL), req.URL)
+	return RenderExtractHTML(WithRequestProxyURL(ctx, req.ProxyURL), browser, req)
+}
+
+// RenderExtractHTML navigates an already-resolved browser to the target,
+// returns its rendered HTML capped to the byte budget, and always closes the
+// page. Shared by the HTTP server's BrowserResolver path and the CLI's
+// one-shot browser. Callers own target validation and proxy gating.
+func RenderExtractHTML(ctx context.Context, browser *Browser, req extractpkg.ExtractRequest) (*extractpkg.FetchResponse, error) {
+	page, err := browser.Navigate(ctx, req.URL)
 	if err != nil {
 		return nil, err
 	}
@@ -220,7 +252,17 @@ func validateExtractTargetURL(ctx context.Context, rawURL string, allowPrivateNe
 }

 func (s *Server) enrichEnvelopeWithExtraction(ctx context.Context, env *Envelope, q Query, format string) {
-	cfg := s.opts.Extract.Normalized()
+	EnrichEnvelopeWithExtraction(ctx, env, q, format, s.newExtractor(), s.opts.Extract)
+}
+
+// EnrichEnvelopeWithExtraction fills env.Results[*].Extracted by running the
+// extractor over the top organic results, with candidate fill-in when a top
+// result fails. It is shared by the HTTP search handler and the CLI so both
+// apply the same depth bounds, batch deadline, and result selection. The
+// extractor and cfg are supplied by the caller (the server reuses its
+// long-lived browser pool; the CLI builds a one-shot browser).
+func EnrichEnvelopeWithExtraction(ctx context.Context, env *Envelope, q Query, format string, extractor extractpkg.Extractor, cfg extractpkg.Config) {
+	cfg = cfg.Normalized()
 	if env == nil || !q.Extract || !cfg.Enabled {
 		return
 	}
@@ -231,11 +273,7 @@ func (s *Server) enrichEnvelopeWithExtraction(ctx context.Context, env *Envelope
 	if format == "text" {
 		contentFormat = "text"
 	}
-	extractor := s.newExtractor()
-	limit := q.ExtractTop
-	if limit <= 0 || limit > 5 {
-		limit = 3
-	}
+	limit := clampExtractTop(q.ExtractTop)
 	if limit > len(env.Results) {
 		limit = len(env.Results)
 	}
@@ -245,7 +283,7 @@ func (s *Server) enrichEnvelopeWithExtraction(ctx context.Context, env *Envelope
 	}

 	// Per-fetch timeouts bound a single URL; this aggregate deadline bounds the
-	// whole batch so a few slow/hanging targets can't stretch the search request
+	// whole batch so a few slow/hanging targets can't stretch the request
 	// open-endedly. The ceiling is derived from the per-URL budget (see
 	// Config.BatchTimeout) rather than a separate knob. When it fires, in-flight
 	// fetches are cancelled and any not yet started record a timeout error instead
@@ -256,7 +294,7 @@ func (s *Server) enrichEnvelopeWithExtraction(ctx context.Context, env *Envelope
 	extractOne := func(idx int) {
 		// Skip the fetch entirely if the batch budget is already spent.
 		if err := ctx.Err(); err != nil {
-			env.Results[idx].Extracted = &ExtractedContent{Error: sanitizeExtractError(err)}
+			env.Results[idx].Extracted = &ExtractedContent{Error: SanitizeExtractError(err)}
 			return
 		}
 		req := extractpkg.ExtractRequest{
@@ -270,14 +308,14 @@ func (s *Server) enrichEnvelopeWithExtraction(ctx context.Context, env *Envelope
 		}
 		result, err := extractor.Extract(ctx, req)
 		if err != nil {
-			env.Results[idx].Extracted = &ExtractedContent{Error: sanitizeExtractError(err)}
+			env.Results[idx].Extracted = &ExtractedContent{Error: SanitizeExtractError(err)}
 			return
 		}
 		content := result.Markdown
 		if contentFormat == "text" {
 			content = result.Text
 		}
-		if !extractedContentLooksUseful(content) {
+		if !ExtractedContentLooksUseful(content) {
 			env.Results[idx].Extracted = &ExtractedContent{Error: "empty extracted content"}
 			return
 		}
@@ -320,7 +358,10 @@ func (s *Server) enrichEnvelopeWithExtraction(ctx context.Context, env *Envelope

 const minUsefulExtractRunes = 80

-func extractedContentLooksUseful(content string) bool {
+// ExtractedContentLooksUseful reports whether extracted page content is long
+// enough to keep, rather than an empty/boilerplate shell. Shared by the HTTP
+// server and the CLI so both apply the same threshold.
+func ExtractedContentLooksUseful(content string) bool {
 	return len([]rune(strings.TrimSpace(content))) >= minUsefulExtractRunes
 }

@@ -337,7 +378,7 @@ func extractedSuccessCount(results []Result) int {
 func extractedResultSucceeded(result Result) bool {
 	return result.Extracted != nil &&
 		result.Extracted.Error == "" &&
-		extractedContentLooksUseful(result.Extracted.Content)
+		ExtractedContentLooksUseful(result.Extracted.Content)
 }

 func sendExtractResult(c *fiber.Ctx, format string, result *extractpkg.ExtractResult) error {
@@ -379,7 +420,9 @@ func parseBoolDefault(raw string, fallback bool) bool {
 	return raw == "1" || strings.EqualFold(raw, "true") || strings.EqualFold(raw, "yes")
 }

-func sanitizeExtractError(err error) string {
+// SanitizeExtractError trims and length-bounds an extraction error for safe
+// inclusion in a response payload. Shared by the HTTP server and the CLI.
+func SanitizeExtractError(err error) string {
 	if err == nil {
 		return ""
 	}
@@ -392,3 +435,123 @@ func sanitizeExtractError(err error) string {
 	}
 	return msg
 }
+
+const maxBatchExtractURLs = 20
+
+type batchExtractPayload struct {
+	extractPayload
+	URLs []string `json:"urls"`
+}
+
+// batchExtractItem is one entry of the bare-array /extract/batch response.
+// The {page_content, metadata} shape is the Open WebUI ExternalWebLoader
+// contract - do not wrap it in the Envelope.
+type batchExtractItem struct {
+	PageContent string            `json:"page_content"`
+	Metadata    map[string]string `json:"metadata"`
+}
+
+func (s *Server) handleBatchExtract(c *fiber.Ctx) error {
+	requestCtx := withRequestUsage(c.UserContext(), "extract-batch")
+	c.SetUserContext(requestCtx)
+	defer setNetworkBytesHeader(c, requestCtx)
+	defer setBrowserProfileHeader(c, requestCtx)
+
+	cfg := s.opts.Extract.Normalized()
+	if !cfg.Enabled {
+		return &APIError{HTTPStatus: fiber.StatusNotFound, ErrorCode: "not_found", Message: "Extraction is disabled"}
+	}
+
+	if len(c.Body()) == 0 {
+		return errInvalidParam("request body is required")
+	}
+	var body batchExtractPayload
+	if err := c.BodyParser(&body); err != nil {
+		return errInvalidParam("invalid JSON body")
+	}
+	urls := dedupeBatchURLs(body.URLs)
+	if len(urls) == 0 {
+		return errInvalidParam("urls array is required and must contain at least one valid URL")
+	}
+	if len(urls) > maxBatchExtractURLs {
+		return errInvalidParam(fmt.Sprintf("urls array exceeds maximum of %d", maxBatchExtractURLs))
+	}
+	baseReq, err := s.baseExtractRequest(c, body.extractPayload, cfg)
+	if err != nil {
+		return err
+	}
+
+	// Target URLs are validated in the fetch path, inside the workers - a bad
+	// URL becomes an error item instead of failing the whole batch (Open WebUI
+	// drops every doc on a non-2xx). 400 is reserved for malformed requests.
+	extractor := s.newExtractor()
+	results := make([]batchExtractItem, len(urls))
+
+	// Bounded parallelism plus an aggregate deadline, same pattern as
+	// EnrichEnvelopeWithExtraction.
+	ctx, cancel := context.WithTimeout(requestCtx, cfg.BatchTimeout(len(urls)))
+	defer cancel()
+
+	sem := make(chan struct{}, cfg.MaxConcurrent)
+	var wg sync.WaitGroup
+	for i, u := range urls {
+		wg.Add(1)
+		sem <- struct{}{}
+		go func(idx int, url string) {
+			defer wg.Done()
+			defer func() { <-sem }()
+			results[idx] = batchExtractOne(ctx, extractor, baseReq, url)
+		}(i, u)
+	}
+	wg.Wait()
+
+	return c.JSON(results)
+}
+
+// dedupeBatchURLs normalizes, drops empties, and keeps first occurrence order.
+func dedupeBatchURLs(raw []string) []string {
+	seen := make(map[string]struct{}, len(raw))
+	var urls []string
+	for _, r := range raw {
+		u := extractpkg.NormalizeURL(strings.TrimSpace(r))
+		if u == "" {
+			continue
+		}
+		if _, dup := seen[u]; dup {
+			continue
+		}
+		seen[u] = struct{}{}
+		urls = append(urls, u)
+	}
+	return urls
+}
+
+// batchExtractOne extracts a single URL, folding failures into the item.
+func batchExtractOne(ctx context.Context, extractor extractpkg.Extractor, req extractpkg.ExtractRequest, url string) batchExtractItem {
+	fail := func(err error) batchExtractItem {
+		WithRequest(ctx).WithError(err).WithField("url", url).Warn("Batch extract failed")
+		return batchExtractItem{Metadata: map[string]string{"source": url, "error": SanitizeExtractError(err)}}
+	}
+	// Skip the fetch once the batch budget is spent.
+	if ctx.Err() != nil {
+		return fail(errors.New("batch timeout"))
+	}
+	req.URL = url
+	result, err := extractor.Extract(ctx, req)
+	if err != nil {
+		return fail(err)
+	}
+	return batchExtractItem{
+		PageContent: result.Markdown,
+		Metadata: map[string]string{
+			"source":      url,
+			"title":       result.Title,
+			"description": result.Description,
+			"lang":        result.Lang,
+			"canonical":   result.Canonical,
+			"mode_used":   result.Meta.ModeUsed,
+			"fetched_at":  result.Meta.FetchedAt,
+			"took_ms":     strconv.FormatInt(result.Meta.TookMs, 10),
+		},
+	}
+}
--- a/core/server_extract_test.go
+++ b/core/server_extract_test.go
@@ -2,9 +2,12 @@ package core

 import (
 	"context"
+	"encoding/json"
+	"fmt"
 	"net/http"
 	"net/http/httptest"
 	"strings"
+	"sync/atomic"
 	"testing"
 	"time"

@@ -109,3 +112,235 @@ func TestValidateExtractTargetURLNormalizesBarePublicIP(t *testing.T) {
 		t.Fatalf("expected bare public IP target to validate after scheme normalization: %v", err)
 	}
 }
+
+func TestBatchExtractSingleURL(t *testing.T) {
+	target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		_, _ = w.Write([]byte(`<html><body><article><h1>Test Page</h1><p>This is a test page with enough content to pass the minimum runes threshold for extraction in batch mode.</p></article></body></html>`))
+	}))
+	defer target.Close()
+
+	s := batchExtractTestServer(true)
+	resp := postBatchExtract(t, s, fmt.Sprintf(`{"urls":["%s"]}`, target.URL))
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
+	}
+	items := decodeBatchItems(t, resp)
+	if len(items) != 1 {
+		t.Fatalf("items count = %d, want 1", len(items))
+	}
+	if items[0].PageContent == "" {
+		t.Fatal("expected non-empty page_content")
+	}
+	if items[0].Metadata["title"] != "Test Page" {
+		t.Fatalf("metadata = %v, want title 'Test Page'", items[0].Metadata)
+	}
+}
+
+func TestBatchExtractHandlesMultipleURLs(t *testing.T) {
+	target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		_, _ = w.Write([]byte(`<html><body><article><h1>Multi</h1><p>Page with sufficient content for batch extraction test that verifies concurrent processing works correctly.</p></article></body></html>`))
+	}))
+	defer target.Close()
+
+	s := batchExtractTestServer(true)
+	resp := postBatchExtract(t, s, fmt.Sprintf(`{"urls":["%s/1","%s/2","%s/3"]}`, target.URL, target.URL, target.URL))
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
+	}
+	if items := decodeBatchItems(t, resp); len(items) != 3 {
+		t.Fatalf("items count = %d, want 3", len(items))
+	}
+}
+
+func batchExtractTestServer(allowPrivate bool) *Server {
+	opts := DefaultServerOptions()
+	opts.Extract = extractpkg.Config{
+		Enabled:              true,
+		DefaultMode:          string(extractpkg.ModeFast),
+		Timeout:              time.Second,
+		MaxBytes:             256 * 1024,
+		MaxConcurrent:        2,
+		AllowPrivateNetworks: allowPrivate,
+	}
+	return NewServerWithOptions("127.0.0.1", 0, opts)
+}
+
+func postBatchExtract(t *testing.T, s *Server, body string, header ...[2]string) *http.Response {
+	t.Helper()
+	req, err := http.NewRequest(http.MethodPost, "/extract/batch", strings.NewReader(body))
+	if err != nil {
+		t.Fatal(err)
+	}
+	req.Header.Set("Content-Type", "application/json")
+	for _, h := range header {
+		req.Header.Set(h[0], h[1])
+	}
+	resp, err := s.app.Test(req)
+	if err != nil {
+		t.Fatal(err)
+	}
+	return resp
+}
+
+func decodeBatchItems(t *testing.T, resp *http.Response) []batchExtractItem {
+	t.Helper()
+	var items []batchExtractItem
+	if err := json.NewDecoder(resp.Body).Decode(&items); err != nil {
+		t.Fatalf("decode response: %v", err)
+	}
+	return items
+}
+
+func TestBatchExtractReturnsPerURLErrors(t *testing.T) {
+	target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		_, _ = w.Write([]byte(`<html><body><article><h1>Good page</h1><p>Enough body text to extract something meaningful from this page in batch mode.</p></article></body></html>`))
+	}))
+	defer target.Close()
+
+	s := batchExtractTestServer(true)
+	resp := postBatchExtract(t, s, fmt.Sprintf(`{"urls":["%s","ftp://example.com/x"]}`, target.URL))
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
+	}
+	items := decodeBatchItems(t, resp)
+	if len(items) != 2 {
+		t.Fatalf("items count = %d, want 2", len(items))
+	}
+	if !strings.Contains(items[0].PageContent, "Good page") || items[0].Metadata["error"] != "" {
+		t.Fatalf("first item = %+v, want successful extraction", items[0])
+	}
+	if items[1].PageContent != "" || items[1].Metadata["error"] == "" {
+		t.Fatalf("second item = %+v, want error item", items[1])
+	}
+	if items[1].Metadata["source"] != "ftp://example.com/x" {
+		t.Fatalf("second item source = %q", items[1].Metadata["source"])
+	}
+}
+
+func TestBatchExtractKeepsPrivateNetworkGuardPerItem(t *testing.T) {
+	s := batchExtractTestServer(false)
+	resp := postBatchExtract(t, s, `{"urls":["http://169.254.169.254/latest/meta-data/"]}`)
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
+	}
+	items := decodeBatchItems(t, resp)
+	if len(items) != 1 {
+		t.Fatalf("items count = %d, want 1", len(items))
+	}
+	if items[0].PageContent != "" || !strings.Contains(items[0].Metadata["error"], "not allowed") {
+		t.Fatalf("item = %+v, want blocked-target error", items[0])
+	}
+}
+
+func TestBatchExtractDedupesURLs(t *testing.T) {
+	var hits int32
+	target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		atomic.AddInt32(&hits, 1)
+		_, _ = w.Write([]byte(`<html><body><article><h1>Dedup</h1><p>Some body text long enough for the extractor to produce markdown content here.</p></article></body></html>`))
+	}))
+	defer target.Close()
+
+	s := batchExtractTestServer(true)
+	body := fmt.Sprintf(`{"urls":["%s/a","%s/a","%s/b"]}`, target.URL, target.URL, target.URL)
+	resp := postBatchExtract(t, s, body)
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
+	}
+	if items := decodeBatchItems(t, resp); len(items) != 2 {
+		t.Fatalf("items count = %d, want 2", len(items))
+	}
+	if got := atomic.LoadInt32(&hits); got != 2 {
+		t.Fatalf("target hits = %d, want 2", got)
+	}
+}
+
+func TestBatchExtractDisabledReturns404(t *testing.T) {
+	opts := DefaultServerOptions()
+	opts.Extract = extractpkg.DefaultConfig()
+	opts.Extract.Enabled = false
+	s := NewServerWithOptions("127.0.0.1", 0, opts)
+
+	resp := postBatchExtract(t, s, `{"urls":["https://example.com"]}`)
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusNotFound {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusNotFound)
+	}
+}
+
+func TestBatchExtractRejectsInvalidMode(t *testing.T) {
+	s := batchExtractTestServer(true)
+	resp := postBatchExtract(t, s, `{"urls":["https://example.com"],"mode":"turbo"}`)
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusBadRequest)
+	}
+}
+
+func TestBatchExtractRejectsBadProxyHeader(t *testing.T) {
+	s := batchExtractTestServer(true)
+	resp := postBatchExtract(t, s, `{"urls":["https://example.com"]}`, [2]string{"X-Proxy-URL", "not-a-proxy"})
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusBadRequest)
+	}
+}
+
+func TestBatchExtractBodyLangReachesFetch(t *testing.T) {
+	var acceptLanguage string
+	target := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		acceptLanguage = r.Header.Get("Accept-Language")
+		_, _ = w.Write([]byte(`<html><body><article><h1>Lang</h1><p>Body text long enough to satisfy extraction thresholds for this test case.</p></article></body></html>`))
+	}))
+	defer target.Close()
+
+	s := batchExtractTestServer(true)
+	resp := postBatchExtract(t, s, fmt.Sprintf(`{"urls":["%s"],"lang":"de"}`, target.URL))
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusOK {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusOK)
+	}
+	if !strings.HasPrefix(acceptLanguage, "de") {
+		t.Fatalf("Accept-Language = %q, want de-prefixed", acceptLanguage)
+	}
+}
+
+func TestBatchExtractRejectsEmptyURLs(t *testing.T) {
+	s := batchExtractTestServer(true)
+	resp := postBatchExtract(t, s, `{"urls":[]}`)
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusBadRequest)
+	}
+}
+
+func TestBatchExtractRejectsURLsOverLimit(t *testing.T) {
+	// 21 URLs, limit is 20
+	urls := make([]string, 21)
+	for i := range urls {
+		urls[i] = fmt.Sprintf("https://example.com/%d", i)
+	}
+	body, _ := json.Marshal(map[string][]string{"urls": urls})
+
+	s := batchExtractTestServer(true)
+	resp := postBatchExtract(t, s, string(body))
+	defer resp.Body.Close()
+
+	if resp.StatusCode != http.StatusBadRequest {
+		t.Fatalf("status = %d, want %d", resp.StatusCode, http.StatusBadRequest)
+	}
+}
--- a/core/server_parse_test.go
+++ b/core/server_parse_test.go
@@ -87,6 +87,29 @@ func TestParseEndpointParserErrorReturns400(t *testing.T) {
 	}
 }

+func TestParseEndpointSearchErrorReturnsTypedError(t *testing.T) {
+	engine := &parserMock{
+		engineMock: engineMock{name: "google", initialized: true},
+		parseHTMLFn: func(_ io.Reader) ([]SearchResult, error) {
+			return nil, ErrCaptcha
+		},
+	}
+	srv := NewServerWithOptions("127.0.0.1", 7126, DefaultServerOptions(), engine)
+
+	resp := postHTML(t, srv, "/google/parse", "<html>captcha</html>")
+	if resp.StatusCode != http.StatusTooManyRequests {
+		t.Fatalf("expected 429, got %d", resp.StatusCode)
+	}
+
+	var body map[string]any
+	if err := json.NewDecoder(resp.Body).Decode(&body); err != nil {
+		t.Fatalf("decode error body: %v", err)
+	}
+	if body["error"] != "captcha_detected" {
+		t.Fatalf("expected captcha_detected, got %#v", body["error"])
+	}
+}
+
 func TestParseEndpointNotRegisteredForNonParserEngine(t *testing.T) {
 	// engineMock does NOT implement HTMLParser so no /mock/parse route is registered.
 	engine := &engineMock{name: "mock", initialized: true}
--- a/core/server_test.go
+++ b/core/server_test.go
@@ -97,6 +97,18 @@ func TestRequestIDHeaderIsEchoedWhenProvided(t *testing.T) {
 	}
 }

+func TestDuckDuckGoDedicatedAliasRoutes(t *testing.T) {
+	engine := &engineMock{name: "duckduckgo", initialized: true}
+	srv := NewServerWithOptions("127.0.0.1", 7119, DefaultServerOptions(), engine)
+
+	for _, path := range []string{"/duck/search?text=golang", "/duckduckgo/search?text=golang", "/duck/image?text=golang", "/duckduckgo/image?text=golang"} {
+		resp := request(t, srv, path)
+		if resp.StatusCode != http.StatusOK {
+			t.Fatalf("expected %s to return 200, got %d", path, resp.StatusCode)
+		}
+	}
+}
+
 func TestRequestIDHeaderIsGeneratedWhenMissing(t *testing.T) {
 	engine := &engineMock{name: "google", initialized: true}
 	srv := NewServerWithOptions("127.0.0.1", 7111, DefaultServerOptions(), engine)
@@ -1820,7 +1832,7 @@ func TestStableSearchErrorJSONWithProxyMeta(t *testing.T) {
 	}{
 		{name: "captcha", err: ErrCaptcha, wantStatus: http.StatusTooManyRequests, wantError: "captcha_detected"},
 		{name: "blocked", err: ErrBlocked, wantStatus: http.StatusForbidden, wantError: "blocked"},
-		{name: "rate limited", err: ErrRateLimited, wantStatus: http.StatusTooManyRequests, wantError: "rate_limited"},
+		{name: "rate limited", err: ErrRateLimited, wantStatus: http.StatusTooManyRequests, wantError: "blocked"},
 		{name: "search timeout", err: ErrSearchTimeout, wantStatus: http.StatusGatewayTimeout, wantError: "search_timeout"},
 		{name: "proxy connect", err: ErrProxyConnect, wantStatus: http.StatusServiceUnavailable, wantError: "proxy_connect"},
 		{name: "proxy auth", err: ErrProxyAuth, wantStatus: http.StatusServiceUnavailable, wantError: "proxy_auth"},
@@ -1879,6 +1891,15 @@ func TestStableSearchErrorJSONWithProxyMeta(t *testing.T) {
 			if payload.Error != tt.wantError {
 				t.Fatalf("expected error=%q, got %q", tt.wantError, payload.Error)
 			}
+			if errors.Is(tt.err, ErrRateLimited) {
+				wantMessage := "search engine blocked the request (HTTP 429)"
+				if payload.Message != wantMessage {
+					t.Fatalf("expected HTTP 429 block message, got %q", payload.Message)
+				}
+				if payload.Meta["upstream_status"] != float64(http.StatusTooManyRequests) {
+					t.Fatalf("expected upstream_status=429, got %#v", payload.Meta["upstream_status"])
+				}
+			}
 			if payload.Meta["proxy_used"] != "http://proxy.example:8080" {
 				t.Fatalf("expected masked proxy_used, got %#v", payload.Meta["proxy_used"])
 			}
@@ -1890,6 +1911,42 @@ func TestStableSearchErrorJSONWithProxyMeta(t *testing.T) {
 	}
 }

+func TestSearchErrorWithGlobalProxyIncludesProxyUsed(t *testing.T) {
+	engine := &engineMock{
+		name:        "google",
+		initialized: true,
+		searchFn: func(_ context.Context, _ Query) ([]SearchResult, error) {
+			return nil, ErrRateLimited
+		},
+	}
+
+	opts := DefaultServerOptions()
+	opts.Resilience.Retry.MaxRetries = 0
+	opts.Resilience.Proxy = ProxyConfig{
+		Runtime: ProxyRuntimeBrowser,
+		Proxies: ProxiesConfig{
+			Global: "http://global-proxy:8080",
+		},
+	}
+	srv := NewServerWithOptions("127.0.0.1", 7123, opts, engine)
+
+	resp := request(t, srv, "/google/search?text=golang")
+	if resp.StatusCode != http.StatusTooManyRequests {
+		t.Fatalf("expected 429, got %d", resp.StatusCode)
+	}
+
+	var payload JSONErrorResponse
+	if err := json.NewDecoder(resp.Body).Decode(&payload); err != nil {
+		t.Fatalf("decode error response: %v", err)
+	}
+	if payload.Error != "blocked" {
+		t.Fatalf("expected blocked, got %q", payload.Error)
+	}
+	if payload.Meta["proxy_used"] != "http://global-proxy:8080" {
+		t.Fatalf("expected masked global proxy, got %#v", payload.Meta["proxy_used"])
+	}
+}
+
 func TestSearchErrorIncludesSanitizedDetail(t *testing.T) {
 	engine := &engineMock{
 		name:        "google",
@@ -2462,6 +2519,36 @@ func TestMegaSearchAllFailuresReturnsDetails(t *testing.T) {
 	}
 }

+func TestMegaSearchSingleEngineFailureMessageIncludesCause(t *testing.T) {
+	google := &engineMock{
+		name:        "google",
+		initialized: true,
+		searchFn: func(_ context.Context, q Query) ([]SearchResult, error) {
+			return nil, fmt.Errorf("%w: selector timeout", ErrSearchTimeout)
+		},
+	}
+
+	opts := DefaultServerOptions()
+	opts.Resilience.Retry.MaxRetries = 0
+	srv := NewServerWithOptions("127.0.0.1", 7211, opts, google)
+
+	resp := request(t, srv, "/mega/search?text=golang&engines=google")
+	if resp.StatusCode != http.StatusBadGateway {
+		t.Fatalf("expected 502, got %d", resp.StatusCode)
+	}
+
+	var payload JSONErrorResponse
+	if err := json.NewDecoder(resp.Body).Decode(&payload); err != nil {
+		t.Fatalf("decode error response: %v", err)
+	}
+	if payload.Error != "all_engines_failed" {
+		t.Fatalf("expected all_engines_failed, got %q", payload.Error)
+	}
+	if !strings.Contains(payload.Message, "google:") || !strings.Contains(payload.Message, "selector timeout") {
+		t.Fatalf("expected message to include google failure cause, got %q", payload.Message)
+	}
+}
+
 func TestMegaSearchClustersGroupSameURL(t *testing.T) {
 	google := &engineMock{
 		name:        "google",
--- a/core/server_timeout_test.go
+++ b/core/server_timeout_test.go
@@ -12,8 +12,8 @@ import (
 )

 // FP-2: every endpoint that doesn't manage its own deadline budget must get
-// one from RequestTimeoutMiddleware; /mega/* (MegaTimeout) and /extract
-// (batch budget) keep theirs.
+// one from RequestTimeoutMiddleware; /mega/* (MegaTimeout) and /extract,
+// /extract/batch (batch budget) keep theirs.
 func TestRequestTimeoutMiddlewareSetsDeadlineExceptBudgetedPaths(t *testing.T) {
 	app := fiber.New()
 	app.Use(RequestTimeoutMiddleware(time.Minute))
@@ -32,12 +32,14 @@ func TestRequestTimeoutMiddlewareSetsDeadlineExceptBudgetedPaths(t *testing.T) {
 	app.Post("/google/parse", record("/google/parse"))
 	app.Get("/mega/search", record("/mega/search"))
 	app.Get("/extract", record("/extract"))
+	app.Post("/extract/batch", record("/extract/batch"))

 	for path, method := range map[string]string{
 		"/google/search": http.MethodGet,
 		"/google/parse":  http.MethodPost,
 		"/mega/search":   http.MethodGet,
 		"/extract":       http.MethodGet,
+		"/extract/batch": http.MethodPost,
 	} {
 		req := httptest.NewRequest(method, path, nil)
 		resp, err := app.Test(req, -1)
@@ -54,6 +56,7 @@ func TestRequestTimeoutMiddlewareSetsDeadlineExceptBudgetedPaths(t *testing.T) {
 		"/google/parse":  true,
 		"/mega/search":   false,
 		"/extract":       false,
+		"/extract/batch": false,
 	} {
 		if deadlines[path] != want {
 			t.Errorf("%s: deadline attached = %v, want %v", path, deadlines[path], want)
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -12,8 +12,18 @@ services:
    ports: 
      - 7000:7000
    command: serve -l
-    #volumes:
-    #  - ./config.yaml:/usr/src/app/config.yaml:ro
+    volumes:
+      - ./config.yaml:/usr/src/app/config.yaml:ro
+
+    # proxies:
+    #   entries:
+    #     - url: http://user:pass@res-proxy-1.example:8080
+    #       tags: [google]
+    #     - url: http://user:pass@res-proxy-2.example:8080
+    #       tags: [google]
+    # google:
+    #   proxy: google
+    
    environment:
      OPENSERP_SERVER_HOST: "0.0.0.0"
      OPENSERP_SERVER_PORT: 7000
--- a/docs/ADDING_AN_ENGINE.md
+++ b/docs/ADDING_AN_ENGINE.md
@@ -36,11 +36,16 @@ Raw engines should expose a `ParseHTML(io.Reader)` path so tests and

 Update:

- `cmd/serve.go` for server wiring.
- CLI search dispatch when the engine is CLI-visible.
+- `cmd/engines.go` with one `engineSpec` row. This central registry drives
+  server wiring, CLI dispatch, aliases, parser endpoints, and raw-mode support.
 - `config.yaml` with rate limits and optional proxy tag.
 - `README.md` and `docs/openapi.yaml` when public endpoints or parameters change.

+Set `rawSearchFn` only when the engine has raw HTTP support, and set
+`parseHTMLFn` when `POST /{engine}/parse` should be available. Touch
+`cmd/serve.go` or `cmd/root.go` only for new global behavior or flags, not for
+ordinary engine registration.
+
 ## 4. Add tests

 Required for the first PR:
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -2,12 +2,12 @@

 ## Overview

-OpenSERP is a Go API + CLI for search result extraction from Google, Yandex, Baidu, Bing, and DuckDuckGo.
+OpenSERP is a Go API + CLI for search result extraction from Google, Yandex, Baidu, Bing, DuckDuckGo, and Ecosia.

 Execution modes:

 - **Browser mode**: default path, headless Chromium via `go-rod`, supported by all engines.
- **Raw HTTP mode**: direct HTTP + `goquery`, currently supported by Google, Yandex, and Baidu.
+- **Raw HTTP mode**: direct HTTP + `goquery`, currently supported by Google, Yandex, Baidu, and Ecosia.

 Browser mode is the primary compatibility path.

@@ -54,6 +54,7 @@ openserp/
 ├── baidu/
 ├── bing/
 ├── duckduckgo/
+├── ecosia/
 └── testutil/
 ```

--- a/docs/demo.gif
+++ b/docs/demo.gif
--- a/docs/openapi.yaml
+++ b/docs/openapi.yaml
@@ -48,7 +48,6 @@ paths:
        - $ref: "#/components/parameters/FilterQuery"
        - $ref: "#/components/parameters/FeaturesQuery"
        - $ref: "#/components/parameters/ExtractQuery"
-        - $ref: "#/components/parameters/ExtractTopQuery"
        - $ref: "#/components/parameters/ExtractModeQuery"
        - $ref: "#/components/parameters/MinRunesQuery"
        - $ref: "#/components/parameters/FormatQuery"
@@ -330,7 +329,6 @@ paths:
        - $ref: "#/components/parameters/MegaDedupeQuery"
        - $ref: "#/components/parameters/MegaMergeQuery"
        - $ref: "#/components/parameters/ExtractQuery"
-        - $ref: "#/components/parameters/ExtractTopQuery"
        - $ref: "#/components/parameters/ExtractModeQuery"
        - $ref: "#/components/parameters/MinRunesQuery"
        - $ref: "#/components/parameters/FormatQuery"
@@ -538,6 +536,11 @@ paths:
                    extracted-text runes than this, escalate to a browser render.
                    `0` (default) uses the built-in floor. Ignored in `fast` and
                    `rendered` modes.
+                lang:
+                  type: string
+                  description: >
+                    Language hint sent as `Accept-Language` on the target fetch.
+                    The `lang` query parameter works as a fallback.
      responses:
        "200":
          description: Extracted URL content
@@ -549,6 +552,40 @@ paths:
          $ref: "#/components/responses/BadRequestError"
        "502":
          $ref: "#/components/responses/BadGatewayError"
+
+  /extract/batch:
+    post:
+      tags: [Search]
+      operationId: extractBatch
+      summary: Extract content from multiple URLs
+      description: >
+        Accepts up to 20 URLs (deduplicated) and returns a bare JSON array of
+        `{page_content, metadata}` items - the shape Open WebUI's external web
+        loader expects. A URL that fails validation or extraction yields an
+        item with `metadata.error` and an empty `page_content` instead of
+        failing the batch. `400` is reserved for malformed requests: empty or
+        oversized `urls`, unknown `mode`, invalid proxy headers.
+      parameters:
+        - $ref: "#/components/parameters/UseProxyHeader"
+        - $ref: "#/components/parameters/ProxyURLHeader"
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              $ref: "#/components/schemas/BatchExtractRequest"
+      responses:
+        "200":
+          description: Batch extraction results, one item per unique URL
+          content:
+            application/json:
+              schema:
+                type: array
+                items:
+                  $ref: "#/components/schemas/BatchExtractItem"
+        "400":
+          $ref: "#/components/responses/BadRequestError"
+
  /health:
    get:
      tags: [Health]
@@ -834,25 +871,27 @@ components:
      name: extract
      in: query
      required: false
-      description: Fetch and embed cleaned target-page content for top web results.
+      description: >
+        Fetch and embed cleaned target-page content for the top web results.
+        Accepts a boolean or an integer depth: `extract=0`/`false` disables
+        extraction; `extract=true`/`1` enriches the top result; `extract=N`
+        (1-5) enriches the top N results. The tuning params `extract_mode` and
+        `min_runes` imply `extract=true` (top 1) when present, unless
+        `extract=0` is set explicitly.
      schema:
-        type: boolean
+        oneOf:
+          - type: boolean
+          - type: integer
+            minimum: 0
+            maximum: 5
        default: false
-    ExtractTopQuery:
-      name: extract_top
-      in: query
-      required: false
-      description: Number of top organic results to enrich when `extract=true`.
-      schema:
-        type: integer
-        minimum: 1
-        maximum: 5
-        default: 3
    ExtractModeQuery:
      name: extract_mode
      in: query
      required: false
-      description: Extraction strategy for target pages.
+      description: >
+        Extraction strategy for target pages. Its presence implies
+        `extract=true` unless `extract=0` is set explicitly.
      schema:
        type: string
        enum: [auto, fast, rendered]
@@ -1094,14 +1133,15 @@ components:
                  engine: google
                  proxy_used: http://proxy.example:8080
                  proxy_session_id: sid-123
-            rateLimited:
+            upstreamBlocked:
              value:
-                error: rate_limited
+                error: blocked
                code: 429
-                message: "search engine rate limited the request"
+                message: "search engine blocked the request (HTTP 429)"
                meta:
                  engine: google
                  proxy_used: http://proxy.example:8080
+                  upstream_status: 429
    BadGatewayError:
      description: Engine internal failure, parser drift, or all-engine failure when fallback is enabled.
      content:
@@ -1710,13 +1750,14 @@ components:
          type: string
          description: >
            Stable machine-readable error class. Search-pipeline failures use the
-            following codes: `captcha_detected`, `blocked`, `rate_limited`,
+            following codes: `captcha_detected`, `blocked`,
            `search_timeout`, `proxy_connect`, `proxy_auth`, `proxy_timeout`,
            `proxy_unavailable`, `parser_failure`, `engine_internal`,
            `all_engines_failed`, `circuit_open`, `request_timeout`,
            `request_canceled`. Validation errors use `bad_request`. Other
-            generic codes (`not_found`, `service_unavailable`, `server_error`,
-            `client_error`, `error`) may appear for non-search routes.
+            generic codes (`not_found`, `rate_limited`, `service_unavailable`,
+            `server_error`, `client_error`, `error`) may appear for non-search
+            routes.
          enum:
            - bad_request
            - not_found
@@ -2035,3 +2076,55 @@ components:
            $ref: "#/components/schemas/MegaEngineInfo"
        total:
          type: integer
+
+    # ── Batch extract ─────────────────────────────────────────────────
+    BatchExtractRequest:
+      type: object
+      required: [urls]
+      properties:
+        urls:
+          type: array
+          items:
+            type: string
+          maxItems: 20
+          description: URLs to extract content from (max 20 after deduplication)
+        mode:
+          type: string
+          enum: [auto, fast, rendered]
+          default: auto
+          description: Extraction mode
+        clean:
+          type: boolean
+          default: true
+          description: >
+            Article-only extraction (default). Set `false` for
+            whole-readable-body extraction.
+        use_llms_txt:
+          type: boolean
+          default: false
+          description: >
+            Probe `/llms-full.txt` then `/llms.txt` for site-root URLs and use
+            that markdown instead of scraping HTML.
+        min_runes:
+          type: integer
+          minimum: 0
+          description: Auto-mode escalation floor, same semantics as `/extract`
+        lang:
+          type: string
+          description: >
+            Language hint sent as `Accept-Language` on target fetches. The
+            `lang` query parameter works as a fallback.
+
+    BatchExtractItem:
+      type: object
+      properties:
+        page_content:
+          type: string
+          description: Extracted markdown content, empty when the item failed
+        metadata:
+          type: object
+          additionalProperties:
+            type: string
+          description: >
+            Page metadata (source, title, lang, canonical, mode_used, etc.).
+            Failed items carry only `source` and `error`.
--- a/duckduckgo/parse_html.go
+++ b/duckduckgo/parse_html.go
@@ -1,6 +1,7 @@
 package duckduckgo

 import (
+	"errors"
 	"io"
 	"strings"

@@ -15,14 +16,27 @@ func ParseHTML(r io.Reader) ([]core.SearchResult, error) {
 	if err != nil {
 		return nil, err
 	}
+	pageStatus := classifyDDGDocument(doc)
+	if errors.Is(pageStatus, core.ErrEmptyResult) {
+		return []core.SearchResult{}, nil
+	}
+	if pageStatus != nil {
+		return nil, pageStatus
+	}
 	return parseDDGDocument(doc), nil
 }

+func classifyDDGDocument(doc *goquery.Document) error {
+	return core.ClassifyChallengeDocument(doc, core.DocSignals{
+		CaptchaSelectors: Selectors.CaptchaSelectors,
+		CaptchaMarkers:   Selectors.CaptchaMarkers,
+		EmptySelectors:   Selectors.NoResults,
+	})
+}
+
 func parseDDGDocument(doc *goquery.Document) []core.SearchResult {
 	var results []core.SearchResult
-	rank := 1
-	adRank := 1
-	absoluteRank := 1
+	rank := core.NewRankState(0)

 	resultSel := firstMatchingSelector(doc, Selectors.Results)
 	if resultSel == "" {
@@ -30,42 +44,49 @@ func parseDDGDocument(doc *goquery.Document) []core.SearchResult {
 	}

 	doc.Find(resultSel).Each(func(_ int, item *goquery.Selection) {
-		href := extractFirstAttr(item, Selectors.Link, "href")
+		href := ddgDocumentHref(item)
 		if href == "" || href == "#" || strings.HasPrefix(href, "javascript:") {
 			return
 		}

-		title := extractFirstText(item, Selectors.Title)
+		title := firstText(item, Selectors.Title...)
 		if title == "" {
 			return
 		}

-		desc := extractFirstText(item, Selectors.Desc)
+		desc := firstText(item, Selectors.Desc...)
+		isAd := ddgSelectionHasAdMarker(item)

-		isAd := duckduckgoSelectionHasAdMarker(item)
-
-		r := core.SearchResult{
-			Rank:         rank,
+		resultRank, absoluteRank := rank.Next(isAd)
+		results = append(results, core.SearchResult{
+			Rank:         resultRank,
 			AbsoluteRank: absoluteRank,
 			URL:          href,
 			Title:        title,
 			Description:  desc,
 			Ad:           isAd,
-		}
-		if !isAd {
-			rank++
-		} else {
-			r.Rank = adRank
-			adRank++
-		}
-		results = append(results, r)
-		absoluteRank++
+		})
 	})

 	return core.AttachFeaturesToFirstResult(core.DeduplicateResults(results), extractDDGFeatures(doc))
 }

-func duckduckgoSelectionHasAdMarker(item *goquery.Selection) bool {
+// firstText returns the normalized text of the first selector that matches a
+// non-empty element.
+func firstText(item *goquery.Selection, selectors ...string) string {
+	for _, sel := range selectors {
+		if tag := item.Find(sel).First(); tag.Length() > 0 {
+			if text := core.NormalizeWhitespace(tag.Text()); text != "" {
+				return text
+			}
+		}
+	}
+	return ""
+}
+
+// ddgSelectionHasAdMarker reports whether the row self-or-descendant matches any
+// DuckDuckGo ad-badge selector.
+func ddgSelectionHasAdMarker(item *goquery.Selection) bool {
 	for _, sel := range Selectors.AdBadge {
 		if item.Is(sel) || item.Find(sel).Length() > 0 {
 			return true
@@ -74,6 +95,22 @@ func duckduckgoSelectionHasAdMarker(item *goquery.Selection) bool {
 	return false
 }

+// ddgDocumentHref returns the first non-empty href among the link selectors,
+// trimmed. It skips a selector whose anchor has an absent/empty href and tries
+// the next (the pre-refactor extractFirstAttr behavior).
+func ddgDocumentHref(item *goquery.Selection) string {
+	for _, sel := range Selectors.Link {
+		tag := item.Find(sel).First()
+		if tag.Length() == 0 {
+			continue
+		}
+		if val, exists := tag.Attr("href"); exists && val != "" {
+			return strings.TrimSpace(val)
+		}
+	}
+	return ""
+}
+
 // firstMatchingSelector returns the first selector from the list that matches
 // at least one element in the document.
 func firstMatchingSelector(doc *goquery.Document, selectors []string) string {
@@ -84,34 +121,3 @@ func firstMatchingSelector(doc *goquery.Document, selectors []string) string {
 	}
 	return ""
 }
-
-// extractFirstAttr tries each selector in order and returns the named attribute
-// of the first match, or "".
-func extractFirstAttr(item *goquery.Selection, selectors []string, attr string) string {
-	for _, sel := range selectors {
-		tag := item.Find(sel).First()
-		if tag.Length() == 0 {
-			continue
-		}
-		val, exists := tag.Attr(attr)
-		if exists && val != "" {
-			return strings.TrimSpace(val)
-		}
-	}
-	return ""
-}
-
-// extractFirstText tries each selector in order and returns the trimmed text of
-// the first match, or "".
-func extractFirstText(item *goquery.Selection, selectors []string) string {
-	for _, sel := range selectors {
-		tag := item.Find(sel).First()
-		if tag.Length() == 0 {
-			continue
-		}
-		if text := strings.TrimSpace(tag.Text()); text != "" {
-			return text
-		}
-	}
-	return ""
-}
--- a/duckduckgo/parse_html_test.go
+++ b/duckduckgo/parse_html_test.go
@@ -2,9 +2,12 @@ package duckduckgo

 import (
 	"bytes"
+	"errors"
 	"os"
 	"strings"
 	"testing"
+
+	"github.com/karust/openserp/core"
 )

 func TestParseDDGHTML(t *testing.T) {
@@ -56,6 +59,29 @@ func TestParseDDGHTMLEmpty(t *testing.T) {
 	}
 }

+func TestParseDDGHTMLCaptcha(t *testing.T) {
+	t.Parallel()
+
+	html := `<html><body><form action="/anomaly.js"><input name="challenge"></form></body></html>`
+	results, err := ParseHTML(bytes.NewReader([]byte(html)))
+	if !errors.Is(err, core.ErrCaptcha) {
+		t.Fatalf("expected ErrCaptcha, got results=%d err=%v", len(results), err)
+	}
+}
+
+func TestParseDDGHTMLNoResults(t *testing.T) {
+	t.Parallel()
+
+	html := `<html><body><div data-testid="no-results">No results found.</div></body></html>`
+	results, err := ParseHTML(bytes.NewReader([]byte(html)))
+	if err != nil {
+		t.Fatalf("ParseHTML() error = %v", err)
+	}
+	if len(results) != 0 {
+		t.Fatalf("expected zero results, got %d", len(results))
+	}
+}
+
 func TestParseDDGHTMLAdsDoNotConsumeOrganicRank(t *testing.T) {
 	t.Parallel()

--- a/duckduckgo/search.go
+++ b/duckduckgo/search.go
@@ -7,6 +7,7 @@ import (
 	"time"

 	"github.com/go-rod/rod"
+	"github.com/go-rod/rod/lib/proto"
 	"github.com/karust/openserp/core"
 )

@@ -14,8 +15,7 @@ import (
 type DuckDuckGo struct {
 	core.Browser
 	core.SearchEngineOptions
-	pageSleep time.Duration // Sleep between pages
-	logger    *core.EngineLogger
+	logger *core.EngineLogger
 }

 // New creates a DuckDuckGo engine instance with browser/runtime options applied.
@@ -25,7 +25,6 @@ func New(browser core.Browser, opts core.SearchEngineOptions) *DuckDuckGo {
 	ddg.SearchEngineOptions = opts
 	ddg.logger = core.NewEngineLogger("DuckDuckGo")

-	ddg.pageSleep = time.Second * 1
 	return &ddg
 }

@@ -69,9 +68,7 @@ func (ddg *DuckDuckGo) isNoResults(page *rod.Page) bool {

 func (ddg *DuckDuckGo) parseResults(results rod.Elements, pageNum int) []core.SearchResult {
 	searchResults := []core.SearchResult{}
-	organicRank := pageNum * 10
-	adRank := 1
-	absoluteRank := pageNum*10 + 1
+	rank := core.NewRankState(pageNum)

 	for _, r := range results {
 		// Get URL - try multiple selectors
@@ -102,7 +99,7 @@ func (ddg *DuckDuckGo) parseResults(results rod.Elements, pageNum int) []core.Se
 		}

 		// Validate that we have a proper URL
-		hrefStr := linkText.String()
+		hrefStr := strings.TrimSpace(linkText.String())
 		if hrefStr == "" || hrefStr == "#" || hrefStr == "javascript:void(0)" {
 			ddg.logger.Debug("Invalid href: %s", hrefStr)
 			continue
@@ -116,15 +113,8 @@ func (ddg *DuckDuckGo) parseResults(results rod.Elements, pageNum int) []core.Se
 		desc := core.FirstNonEmptyText(r, Selectors.Desc...)

 		// Check if it's an ad
-		isAd := duckduckgoElementHasAdMarker(r)
-		resultRank := 0
-		if isAd {
-			resultRank = adRank
-			adRank++
-		} else {
-			organicRank++
-			resultRank = organicRank
-		}
+		isAd := ddgElementHasAdMarker(r)
+		resultRank, absoluteRank := rank.Next(isAd)

 		result := core.SearchResult{
 			Rank:         resultRank,
@@ -135,19 +125,17 @@ func (ddg *DuckDuckGo) parseResults(results rod.Elements, pageNum int) []core.Se
 			Ad:           isAd,
 		}
 		searchResults = append(searchResults, result)
-		absoluteRank++
 	}

 	return searchResults
 }

-func duckduckgoElementHasAdMarker(el *rod.Element) bool {
+func ddgElementHasAdMarker(el *rod.Element) bool {
 	if el == nil {
 		return false
 	}
 	for _, selector := range Selectors.AdBadge {
-		matches, err := el.Matches(selector)
-		if err == nil && matches {
+		if matches, err := el.Matches(selector); err == nil && matches {
 			return true
 		}
 		if adIndicator, err := el.Element(selector); err == nil && adIndicator != nil {
@@ -157,81 +145,108 @@ func duckduckgoElementHasAdMarker(el *rod.Element) bool {
 	return false
 }

+func windowOrganicResults(results []core.SearchResult, start, limit int) []core.SearchResult {
+	out := make([]core.SearchResult, 0, len(results))
+	skipped, kept := 0, 0
+	for _, result := range results {
+		if !result.Ad && skipped < start {
+			skipped++
+			continue
+		}
+		if !result.Ad && limit > 0 && kept >= limit {
+			continue
+		}
+		out = append(out, result)
+		if !result.Ad {
+			kept++
+		}
+	}
+	return out
+}
+
+func findMoreResultsButton(page *rod.Page) *rod.Element {
+	for _, selector := range Selectors.MoreResults {
+		has, button, err := page.Has(selector)
+		if err == nil && has && button != nil {
+			return button
+		}
+	}
+	return nil
+}
+
 // Search executes a DuckDuckGo web search and returns normalized search
 // results. It may return core.ErrCaptcha or core.ErrSearchTimeout.
 func (ddg *DuckDuckGo) Search(ctx context.Context, query core.Query) (results []core.SearchResult, err error) {
-	ctx = core.PrepareEngineContext(ctx, query, ddg.Name(), false)
+	ctx = core.PrepareEngineContext(ctx, query, ddg.Name())
 	scoped := *ddg
 	scoped.logger = ddg.logger.WithRequest(ctx)
 	ddg = &scoped

 	ddg.logger.Debug("Starting search, query: %+v", query)
-
-	allResults := []core.SearchResult{}
-	var pageFeatures []core.SerpFeature
-	searchPage := 0
-
-	// fetchPage loads one SERP page and appends parsed results.
-	// Returns (done, error): done=true ends the outer loop without error.
-	fetchPage := func() (bool, error) {
-		url, err := BuildURL(query, searchPage)
-		if err != nil {
-			return false, err
-		}
-
-		page, err := ddg.Navigate(ctx, url)
-		if err != nil {
-			return false, err
-		}
-		defer core.DeferClosePage(ctx, page, &ddg.Browser)()
-
-		elements, selector, err := core.WaitForElements(ctx, page, Selectors.Results, ddg.GetSelectorTimeout())
-		if err != nil {
-			if ddg.isNoResults(page) {
-				ddg.logger.Warn("No results found")
-				return true, nil
-			}
-			if ddg.isCaptcha(page) {
-				ddg.logger.Error("Captcha detected: %s", url)
-				return false, core.ErrCaptcha
-			}
-			ddg.logger.Error("Cannot parse search results: %s", err)
-			return false, core.ErrSearchTimeout
-		}
-		ddg.logger.Debug("Found results with selector: %s", selector)
-
-		r := ddg.parseResults(elements, searchPage)
-		if len(r) == 0 {
-			ddg.logger.Debug("No valid results found on page %d", searchPage)
-			return false, core.ErrSearchTimeout
-		}
-
-		if query.Features && searchPage == 0 {
-			pageFeatures = extractDDGFeaturesFromPage(page)
-		}
-		allResults = append(allResults, r...)
-		return false, nil
+	if query.Start < 0 {
+		return nil, fmt.Errorf("incorrect start provided")
 	}

-	for core.ShouldFetchResultPage(core.CountOrganicResults(allResults), query.Limit, searchPage) {
-		done, err := fetchPage()
-		if err != nil {
-			return nil, err
+	url, err := BuildURL(query, 0)
+	if err != nil {
+		return nil, err
+	}
+	page, err := ddg.Navigate(ctx, url)
+	if err != nil {
+		return nil, err
+	}
+	defer core.DeferClosePage(ctx, page, &ddg.Browser)()
+
+	elements, selector, err := core.WaitForElements(ctx, page, Selectors.Results, ddg.GetSelectorTimeout())
+	if err != nil {
+		if ddg.isNoResults(page) {
+			ddg.logger.Warn("No results found")
+			return []core.SearchResult{}, nil
 		}
-		searchPage++
-		if done || !core.ShouldFetchResultPage(core.CountOrganicResults(allResults), query.Limit, searchPage) {
+		if ddg.isCaptcha(page) {
+			ddg.logger.Error("Captcha detected: %s", url)
+			return nil, core.ErrCaptcha
+		}
+		ddg.logger.Error("Cannot parse search results: %s", err)
+		return nil, core.ErrSearchTimeout
+	}
+	ddg.logger.Debug("Found results with selector: %s", selector)
+
+	allResults := ddg.parseResults(elements, 0)
+	if len(allResults) == 0 {
+		return nil, core.ErrSearchTimeout
+	}
+	var pageFeatures []core.SerpFeature
+	if query.Features {
+		pageFeatures = extractDDGFeaturesFromPage(page)
+	}
+
+	wantOrganic := query.Start + query.Limit
+	for core.CountOrganicResults(allResults) < wantOrganic {
+		button := findMoreResultsButton(page)
+		if button == nil {
 			break
 		}
-		if err := core.SleepContext(ctx, ddg.pageSleep); err != nil {
-			return nil, err
+		before := len(elements)
+		if err := button.Click(proto.InputMouseButtonLeft, 1); err != nil {
+			ddg.logger.Debug("More results click failed: %s", err)
+			break
 		}
+		if err := page.Timeout(ddg.GetSelectorTimeout()).WaitElementsMoreThan(selector, before); err != nil {
+			if ctx.Err() != nil {
+				return nil, ctx.Err()
+			}
+			ddg.logger.Debug("No additional results loaded")
+			break
+		}
+		elements, err = page.Elements(selector)
+		if err != nil || len(elements) <= before {
+			break
+		}
+		allResults = core.DeduplicateResults(append(allResults, ddg.parseResults(elements, 0)...))
 	}

-	// Deduplicate results
-	deduped := core.DeduplicateResults(allResults)
-
-	// Trim to exact limit if necessary
-	deduped = core.LimitOrganicResults(deduped, query.Limit)
+	deduped := windowOrganicResults(core.DeduplicateResults(allResults), query.Start, query.Limit)

 	ddg.logger.Info("Search completed: %d results", len(deduped))
 	return core.AttachFeaturesToFirstResult(deduped, pageFeatures), nil
@@ -240,7 +255,7 @@ func (ddg *DuckDuckGo) Search(ctx context.Context, query core.Query) (results []
 // SearchImage executes a DuckDuckGo image search and returns normalized image
 // results. It may return core.ErrCaptcha or core.ErrSearchTimeout.
 func (ddg *DuckDuckGo) SearchImage(ctx context.Context, query core.Query) ([]core.SearchResult, error) {
-	ctx = core.PrepareEngineContext(ctx, query, ddg.Name(), false)
+	ctx = core.PrepareEngineContext(ctx, query, ddg.Name())
 	scoped := *ddg
 	scoped.logger = ddg.logger.WithRequest(ctx)
 	ddg = &scoped
--- a/duckduckgo/search_test.go
+++ b/duckduckgo/search_test.go
@@ -1,10 +1,14 @@
 package duckduckgo

 import (
+	"context"
 	"net/url"
+	"os"
 	"testing"

 	"github.com/karust/openserp/core"
+	"github.com/karust/openserp/testutil"
+	"github.com/karust/openserp/testutil/ithelper"
 )

 func TestBuildURL(t *testing.T) {
@@ -257,67 +261,51 @@ func TestDuckDuckGoLanguageMapping(t *testing.T) {
 	}
 }

-func TestShouldFetchDuckDuckGoPage(t *testing.T) {
-	results := func(organic, ads int) []core.SearchResult {
-		out := make([]core.SearchResult, 0, organic+ads)
-		for i := 0; i < ads; i++ {
-			out = append(out, core.SearchResult{URL: "https://ad.example/" + string(rune('a'+i)), Ad: true})
-		}
-		for i := 0; i < organic; i++ {
-			out = append(out, core.SearchResult{URL: "https://example.com/" + string(rune('a'+i))})
-		}
-		return out
+func TestWindowOrganicResults(t *testing.T) {
+	results := []core.SearchResult{
+		{URL: "https://ad.example", Ad: true},
+		{URL: "https://example.com/1", Rank: 1},
+		{URL: "https://example.com/2", Rank: 2},
+		{URL: "https://example.com/3", Rank: 3},
+		{URL: "https://example.com/4", Rank: 4},
 	}

-	tests := []struct {
-		name         string
-		results      []core.SearchResult
-		limit        int
-		pagesFetched int
-		want         bool
-	}{
-		{
-			name:         "first page is always fetched",
-			limit:        10,
-			pagesFetched: 0,
-			want:         true,
-		},
-		{
-			name:         "default limit does not chase a short first page",
-			results:      results(8, 2),
-			limit:        10,
-			pagesFetched: 1,
-			want:         false,
-		},
-		{
-			name:         "explicit larger limit can paginate",
-			results:      results(8, 0),
-			limit:        11,
-			pagesFetched: 1,
-			want:         true,
-		},
-		{
-			name:         "satisfied larger limit stops",
-			results:      results(11, 0),
-			limit:        11,
-			pagesFetched: 1,
-			want:         false,
-		},
-		{
-			name:         "unset internal query stops after first page",
-			results:      results(8, 0),
-			limit:        0,
-			pagesFetched: 1,
-			want:         false,
-		},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			got := core.ShouldFetchResultPage(core.CountOrganicResults(tt.results), tt.limit, tt.pagesFetched)
-			if got != tt.want {
-				t.Fatalf("ShouldFetchResultPage() = %t, want %t", got, tt.want)
-			}
-		})
+	got := windowOrganicResults(results, 1, 2)
+	if len(got) != 3 || !got[0].Ad || got[1].Rank != 2 || got[2].Rank != 3 {
+		t.Fatalf("unexpected result window: %#v", got)
+	}
+}
+
+func TestFindMoreResultsButton(t *testing.T) {
+	testutil.RequireIntegration(t)
+
+	fixture, err := os.ReadFile("testdata/search_results.html")
+	if err != nil {
+		t.Fatalf("read fixture: %v", err)
+	}
+
+	browser := ithelper.CreateBrowser(t)
+	page, err := browser.Navigate(context.Background(), "about:blank")
+	if err != nil {
+		t.Fatalf("navigate: %v", err)
+	}
+	defer core.DeferClosePage(context.Background(), page, browser)()
+
+	cases := []struct {
+		html    string
+		wantHit bool
+	}{
+		{string(fixture), true},                               // real DDG markup
+		{`<button id="js-more-results-btn">x</button>`, true}, // renamed id, substring fallback
+		{`<div>no button</div>`, false},
+	}
+
+	for _, tc := range cases {
+		if err := page.SetDocumentContent(tc.html); err != nil {
+			t.Fatalf("set content: %v", err)
+		}
+		if hit := findMoreResultsButton(page) != nil; hit != tc.wantHit {
+			t.Fatalf("hit=%v want=%v", hit, tc.wantHit)
+		}
 	}
 }
--- a/duckduckgo/selectors.go
+++ b/duckduckgo/selectors.go
@@ -14,6 +14,7 @@ var Selectors = struct {
 	Desc             []string
 	Link             []string
 	AdBadge          []string
+	MoreResults      []string
 	ImageResult      []string
 	ImageImg         []string
 	ImageTitle       []string
@@ -83,6 +84,10 @@ var Selectors = struct {
 		".ad-badge",
 		".result--ad",
 	},
+	MoreResults: []string{
+		"#more-results",
+		"[id*='more-result']",
+	},
 	ImageResult: []string{
 		"figure[data-testid='image-result']",
 		"figure",
--- a/duckduckgo/testdata/search_results.html
+++ b/duckduckgo/testdata/search_results.html
--- a/duckduckgo/url.go
+++ b/duckduckgo/url.go
@@ -79,6 +79,61 @@ func duckDuckGoKL(langCode, region string) string {
 	return ddgKLByLocale[locale.Language]
 }

+// addQueryText appends the "q" param, combining text with site/filetype
+// operators. Returns an error when the resulting query is empty.
+func addQueryText(params url.Values, q core.Query) error {
+	if q.Text == "" && q.Site == "" && q.Filetype == "" {
+		return errors.New("empty query built")
+	}
+	text := q.Text
+	if q.Site != "" {
+		text += " site:" + q.Site
+	}
+	if q.Filetype != "" {
+		text += " filetype:" + q.Filetype
+	}
+	params.Add("q", text)
+	return nil
+}
+
+// addDateRange appends the "df" param, converting the YYYYMMDD..YYYYMMDD
+// interval to DuckDuckGo's YYYY-MM-DD..YYYY-MM-DD form. No-op when unset.
+func addDateRange(params url.Values, interval string) error {
+	if interval == "" {
+		return nil
+	}
+	intervals := strings.Split(interval, "..")
+	if len(intervals) != 2 {
+		return errors.New("incorrect date interval provided")
+	}
+	start, err := time.Parse("20060102", intervals[0])
+	if err != nil {
+		return errors.New("invalid start date format, expected YYYYMMDD")
+	}
+	end, err := time.Parse("20060102", intervals[1])
+	if err != nil {
+		return errors.New("invalid end date format, expected YYYYMMDD")
+	}
+	params.Add("df", start.Format("2006-01-02")+".."+end.Format("2006-01-02"))
+	return nil
+}
+
+// buildParams assembles the query, date range, and locale params shared by web
+// and image search.
+func buildParams(q core.Query) (url.Values, error) {
+	params := url.Values{}
+	if err := addQueryText(params, q); err != nil {
+		return nil, err
+	}
+	if err := addDateRange(params, q.DateInterval); err != nil {
+		return nil, err
+	}
+	if kl := duckDuckGoKL(q.LangCode, q.Region); kl != "" {
+		params.Add("kl", kl)
+	}
+	return params, nil
+}
+
 // BuildURL builds a DuckDuckGo web search URL for the provided query and page
 // index. It returns an error when query text or date parameters are invalid.
 func BuildURL(q core.Query, page int) (string, error) {
@@ -86,66 +141,17 @@ func BuildURL(q core.Query, page int) (string, error) {
 	if err != nil {
 		return "", err
 	}
-
-	base.Path += ""
-	params := url.Values{}
-
-	// Set request text
-	if q.Text != "" || q.Site != "" || q.Filetype != "" {
-		text := q.Text
-		if q.Site != "" {
-			text += " site:" + q.Site
-		}
-		if q.Filetype != "" {
-			text += " filetype:" + q.Filetype
-		}
-
-		params.Add("q", text)
+	params, err := buildParams(q)
+	if err != nil {
+		return "", err
 	}

-	if len(params.Get("q")) == 0 {
-		return "", errors.New("empty query built")
-	}
-
-	// Set search date range
-	if q.DateInterval != "" {
-		intervals := strings.Split(q.DateInterval, "..")
-		if len(intervals) != 2 {
-			return "", errors.New("incorrect date interval provided")
-		}
-
-		// Convert from YYYYMMDD to YYYY-MM-DD format (DuckDuckGo requirement)
-		startDate, err := time.Parse("20060102", intervals[0])
-		if err != nil {
-			return "", errors.New("invalid start date format, expected YYYYMMDD")
-		}
-
-		endDate, err := time.Parse("20060102", intervals[1])
-		if err != nil {
-			return "", errors.New("invalid end date format, expected YYYYMMDD")
-		}
-
-		// DuckDuckGo uses YYYY-MM-DD..YYYY-MM-DD format
-		dateRange := fmt.Sprintf("%s..%s",
-			startDate.Format("2006-01-02"),
-			endDate.Format("2006-01-02"))
-		params.Add("df", dateRange)
-	}
-
-	if kl := duckDuckGoKL(q.LangCode, q.Region); kl != "" {
-		params.Add("kl", kl)
-	}
-
-	// DuckDuckGo specific parameters
 	params.Add("t", "h")    // HTML format
 	params.Add("ia", "web") // Web search

-	// Add pagination parameter if not on first page
+	// Pagination uses 's' (start offset); ~25 results per page.
 	if page > 0 {
-		// DuckDuckGo uses 's' parameter for pagination (start offset)
-		// Each page has approximately 25-30 results, but we'll use conservative estimate
-		offset := page * 25
-		params.Add("s", fmt.Sprintf("%d", offset))
+		params.Add("s", fmt.Sprintf("%d", page*25))
 	}

 	base.RawQuery = params.Encode()
@@ -159,58 +165,15 @@ func BuildImageURL(q core.Query) (string, error) {
 	if err != nil {
 		return "", err
 	}
+	params, err := buildParams(q)
+	if err != nil {
+		return "", err
+	}

-	base.Path += ""
-	params := url.Values{}
 	params.Add("t", "h")        // HTML format
 	params.Add("iax", "images") // Image search
 	params.Add("ia", "images")

-	// Set request text
-	if q.Text != "" || q.Site != "" || q.Filetype != "" {
-		text := q.Text
-		if q.Site != "" {
-			text += " site:" + q.Site
-		}
-		if q.Filetype != "" {
-			text += " filetype:" + q.Filetype
-		}
-
-		params.Add("q", text)
-	}
-
-	if len(params.Get("q")) == 0 {
-		return "", errors.New("empty query built")
-	}
-
-	// Set search date range
-	if q.DateInterval != "" {
-		intervals := strings.Split(q.DateInterval, "..")
-		if len(intervals) != 2 {
-			return "", errors.New("incorrect date interval provided")
-		}
-
-		// Convert from YYYYMMDD to YYYY-MM-DD format (DuckDuckGo requirement)
-		startDate, err := time.Parse("20060102", intervals[0])
-		if err != nil {
-			return "", errors.New("invalid start date format, expected YYYYMMDD")
-		}
-
-		endDate, err := time.Parse("20060102", intervals[1])
-		if err != nil {
-			return "", errors.New("invalid end date format, expected YYYYMMDD")
-		}
-
-		// DuckDuckGo uses YYYY-MM-DD..YYYY-MM-DD format
-		dateRange := fmt.Sprintf("%s..%s",
-			startDate.Format("2006-01-02"),
-			endDate.Format("2006-01-02"))
-		params.Add("df", dateRange)
-	}
-
-	if kl := duckDuckGoKL(q.LangCode, q.Region); kl != "" {
-		params.Add("kl", kl)
-	}
 	base.RawQuery = params.Encode()
 	return base.String(), nil
 }
--- a/ecosia/captcha_selector_test.go
+++ b/ecosia/captcha_selector_test.go
@@ -0,0 +1,54 @@
+package ecosia
+
+import (
+	"testing"
+
+	"github.com/PuerkitoBio/goquery"
+	"github.com/karust/openserp/testutil"
+)
+
+// TestEcosiaPageTypeSelectors verifies that the selectors defined in selectors.go
+// match (or don't match) real fixture HTML without needing a browser.
+func TestEcosiaPageTypeSelectors(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		fixture  string
+		selector string
+		wantHit  bool
+	}{
+		{"search_captcha.html", Selectors.Captcha, true},
+		{"search_captcha.html", Selectors.Mainline, false},
+
+		{"search_results.html", Selectors.Mainline, true},
+		{"search_results.html", Selectors.Captcha, false},
+
+		{"search_no_results.html", Selectors.Captcha, false},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.fixture+"/"+tt.selector, func(t *testing.T) {
+			t.Parallel()
+			assertSelector(t, tt.fixture, tt.selector, tt.wantHit)
+		})
+	}
+}
+
+func assertSelector(t *testing.T, fixture, selector string, wantHit bool) {
+	t.Helper()
+
+	resp := testutil.ResponseFromFixture(t, fixture)
+	doc, err := goquery.NewDocumentFromReader(resp.Body)
+	if err != nil {
+		t.Fatalf("parse fixture: %v", err)
+	}
+
+	got := doc.Find(selector).Length() > 0
+	if got != wantHit {
+		if wantHit {
+			t.Fatalf("selector %q not found in %s — update selectors.go", selector, fixture)
+		} else {
+			t.Fatalf("selector %q unexpectedly present in %s", selector, fixture)
+		}
+	}
+}
--- a/ecosia/features.go
+++ b/ecosia/features.go
@@ -7,7 +7,7 @@ import (
 )

 func extractEcosiaFeatures(doc *goquery.Document) []core.SerpFeature {
-	features := core.ExtractSerpFeaturesBySelectors(doc, []core.SerpFeatureSelector{
+	return core.ExtractSerpFeaturesBySelectors(doc, []core.SerpFeatureSelector{
 		{
 			Type:          core.ResultTypeAnswerBox,
 			Title:         "Answer",
@@ -27,7 +27,6 @@ func extractEcosiaFeatures(doc *goquery.Document) []core.SerpFeature {
 			Confidence:   0.8,
 		},
 	})
-	return core.DeduplicateSerpFeatures(features)
 }

 func extractEcosiaFeaturesFromPage(page *rod.Page) []core.SerpFeature {
--- a/ecosia/parse_html.go
+++ b/ecosia/parse_html.go
@@ -1,6 +1,8 @@
 package ecosia

 import (
+	"errors"
+	"fmt"
 	"io"
 	"strings"

@@ -15,140 +17,116 @@ func ParseHTML(r io.Reader) ([]core.SearchResult, error) {
 	if err != nil {
 		return nil, err
 	}
+	pageStatus := classifyEcosiaDocument(doc)
+	if errors.Is(pageStatus, core.ErrEmptyResult) {
+		return []core.SearchResult{}, nil
+	}
+	if pageStatus != nil {
+		return nil, pageStatus
+	}
 	return parseEcosiaDocument(doc), nil
 }

+func classifyEcosiaDocument(doc *goquery.Document) error {
+	if isCaptchaDoc(doc) {
+		return core.ErrCaptcha
+	}
+	if doc.Find(Selectors.NoResults).Length() > 0 ||
+		(doc.Find(Selectors.Mainline).Length() > 0 &&
+			doc.Find(Selectors.Result).Length() == 0 &&
+			doc.Find(Selectors.Ad).Length() == 0) {
+		return core.ErrEmptyResult
+	}
+	return nil
+}
+
 func parseEcosiaDocument(doc *goquery.Document) []core.SearchResult {
 	var results []core.SearchResult
 	rank := 1
 	adRank := 1

 	doc.Find(Selectors.Result).Each(func(_ int, item *goquery.Selection) {
-		res, ok := parseEcosiaItem(item, rank, false)
-		if !ok {
-			return
+		if res, ok := parseEcosiaSelectionRow(item, rank, false); ok {
+			results = append(results, res)
+			rank++
 		}
-		results = append(results, res)
-		rank++
 	})

 	doc.Find(Selectors.Ad).Each(func(_ int, item *goquery.Selection) {
-		res, ok := parseEcosiaItem(item, adRank, true)
-		if !ok {
-			return
+		if res, ok := parseEcosiaSelectionRow(item, adRank, true); ok {
+			results = append(results, res)
+			adRank++
 		}
-		results = append(results, res)
-		adRank++
 	})

-	setSeparatedAdAbsoluteRanks(results, 0)
+	core.SetSeparatedAdAbsoluteRanks(results, 0)
 	return core.AttachFeaturesToFirstResult(core.DeduplicateResults(results), extractEcosiaFeatures(doc))
 }

-func parseEcosiaItem(item *goquery.Selection, rank int, ad bool) (core.SearchResult, bool) {
-	linkTag := item.Find(Selectors.ResultLink).First()
-	if linkTag.Length() == 0 {
-		linkTag = item.Find("a[href]").First()
-	}
-	if linkTag.Length() == 0 {
-		return core.SearchResult{}, false
+// parseEcosiaSelectionRow extracts a web row from a goquery selection.
+func parseEcosiaSelectionRow(item *goquery.Selection, rank int, ad bool) (core.SearchResult, bool) {
+	link := item.Find(Selectors.ResultLink).First()
+	if link.Length() == 0 {
+		// Fall back to the first anchor when the test-id selector is absent.
+		link = item.Find("a[href]").First()
 	}
+	href, _ := link.Attr("href")

-	href, exists := linkTag.Attr("href")
-	if !exists {
-		return core.SearchResult{}, false
+	title := selectionText(item, Selectors.Title)
+	if title == "" {
+		title = selectionText(item, "h2, h3")
 	}
+	desc := selectionText(item, Selectors.Desc)
+	return assembleEcosiaRow(href, title, desc, rank, ad)
+}
+
+// assembleEcosiaRow validates an already-extracted web row and builds the
+// result. Shared by the rod (browser) and goquery (raw / parse) parsers.
+func assembleEcosiaRow(href, title, desc string, rank int, ad bool) (core.SearchResult, bool) {
 	href = strings.TrimSpace(href)
 	if href == "" || strings.HasPrefix(href, "javascript:") {
 		return core.SearchResult{}, false
 	}
+	return core.SearchResult{
+		Rank:        rank,
+		URL:         href,
+		Title:       strings.TrimSpace(title),
+		Description: strings.TrimSpace(desc),
+		Ad:          ad,
+	}, true
+}

-	title := ""
-	if t := item.Find(Selectors.Title).First(); t.Length() > 0 {
-		title = strings.TrimSpace(t.Text())
+// assembleEcosiaImageRow validates an already-extracted image card and builds
+// the result (formatting the source/dimensions description). Shared by both
+// parser backends.
+func assembleEcosiaImageRow(href, title, source, dims string, rank int) (core.SearchResult, bool) {
+	href = strings.TrimSpace(href)
+	if href == "" {
+		return core.SearchResult{}, false
 	}
-	if title == "" {
-		if t := item.Find("h2, h3").First(); t.Length() > 0 {
-			title = strings.TrimSpace(t.Text())
+	desc := source
+	if dims != "" {
+		if source != "" {
+			desc = fmt.Sprintf("%s (%s)", source, dims)
+		} else {
+			desc = dims
 		}
 	}
-
-	desc := ""
-	if d := item.Find(Selectors.Desc).First(); d.Length() > 0 {
-		desc = strings.TrimSpace(d.Text())
-	}
-
 	return core.SearchResult{
 		Rank:        rank,
 		URL:         href,
 		Title:       title,
 		Description: desc,
-		Ad:          ad,
 	}, true
 }

-func setSeparatedAdAbsoluteRanks(results []core.SearchResult, start int) {
-	adCount := 0
-	for i := range results {
-		if results[i].Ad {
-			adCount++
-			results[i].AbsoluteRank = start + results[i].Rank
+// selectionText returns the trimmed text of the first selector that matches,
+// the goquery counterpart of the rod element's selector-fallback text lookup.
+func selectionText(item *goquery.Selection, selectors ...string) string {
+	for _, selector := range selectors {
+		if tag := item.Find(selector).First(); tag.Length() > 0 {
+			return strings.TrimSpace(tag.Text())
 		}
 	}
-	organicAbsoluteRank := start + adCount + 1
-	for i := range results {
-		if results[i].Ad {
-			continue
-		}
-		results[i].AbsoluteRank = organicAbsoluteRank
-		organicAbsoluteRank++
-	}
+	return ""
 }
-
-// // parseEcosiaImageItem extracts a single image card from a goquery Selection.
-// func parseEcosiaImageItem(item *goquery.Selection, rank int) (core.SearchResult, bool) {
-// 	linkTag := item.Find(Selectors.ImageLink).First()
-// 	if linkTag.Length() == 0 {
-// 		return core.SearchResult{}, false
-// 	}
-// 	href, exists := linkTag.Attr("href")
-// 	if !exists {
-// 		return core.SearchResult{}, false
-// 	}
-// 	imgURL := strings.TrimSpace(href)
-// 	if imgURL == "" {
-// 		return core.SearchResult{}, false
-// 	}
-
-// 	title := ""
-// 	if img := linkTag.Find("img").First(); img.Length() > 0 {
-// 		if alt, err := img.Attr("alt"); err {
-// 			title = strings.TrimSpace(alt)
-// 		}
-// 	}
-
-// 	source := ""
-// 	if s := item.Find(Selectors.ImageSource).First(); s.Length() > 0 {
-// 		source = strings.TrimSpace(s.Text())
-// 	}
-// 	dims := ""
-// 	if d := item.Find(Selectors.ImageDims).First(); d.Length() > 0 {
-// 		dims = strings.TrimSpace(d.Text())
-// 	}
-
-// 	desc := source
-// 	if dims != "" {
-// 		if source != "" {
-// 			desc = fmt.Sprintf("%s (%s)", source, dims)
-// 		} else {
-// 			desc = dims
-// 		}
-// 	}
-
-// 	return core.SearchResult{
-// 		Rank:        rank,
-// 		URL:         imgURL,
-// 		Title:       title,
-// 		Description: desc,
-// 	}, true
-// }
--- a/ecosia/search.go
+++ b/ecosia/search.go
@@ -9,7 +9,6 @@ package ecosia
 import (
 	"context"
 	"errors"
-	"fmt"
 	"strings"
 	"time"

@@ -35,13 +34,17 @@ func startPage(start int) (pageNum, startRank int, err error) {
 	return pageNum, startRank, nil
 }

-// Cloudflare interstitial markers. URL/title are CF defaults.
+// Cloudflare interstitial markers. URL/title are CF defaults; cfBodyMarkers are
+// challenge-page phrases absent from a real SERP, shared by the browser
+// (isCaptcha) and raw (isCaptchaDoc) paths. Bare "captcha" is omitted — too
+// broad (appears in SERP snippets) and not on the interstitial anyway.
 const (
-	cfURLPath    = "cdn-cgi"
-	cfPageTitle  = "just a moment"
-	cfBodyMarker = "not a bot"
+	cfURLPath   = "cdn-cgi"
+	cfPageTitle = "just a moment"
 )

+var cfBodyMarkers = []string{"not a robot", "not a bot", "unusual traffic"}
+
 // Ecosia implements core.SearchEngine for Ecosia SERP pages. Additional
 // documentation at https://support.ecosia.org/article/447-search-features.
 type Ecosia struct {
@@ -65,10 +68,11 @@ func New(browser core.Browser, opts core.SearchEngineOptions) *Ecosia {
 func (e *Ecosia) Name() string { return "ecosia" }

 // isCaptcha reports whether the current page is a Cloudflare interstitial.
-// Checks the URL, title, then body text (cheapest first).
+// Checks the URL and title first - signals only the live page carries, not a
+// bare HTML snapshot - then falls back to the shared body-marker/selector
+// check (isCaptchaDoc) used by the raw HTML path, so both paths agree.
 func (e *Ecosia) isCaptcha(page *rod.Page) bool {
-	info, err := page.Info()
-	if err == nil {
+	if info, err := page.Info(); err == nil {
 		if strings.Contains(strings.ToLower(info.URL), cfURLPath) {
 			return true
 		}
@@ -76,11 +80,7 @@ func (e *Ecosia) isCaptcha(page *rod.Page) bool {
 			return true
 		}
 	}
-	html, err := page.Timeout(e.GetSelectorTimeout()).HTML()
-	if err != nil {
-		return false
-	}
-	return strings.Contains(strings.ToLower(html), cfBodyMarker)
+	return errors.Is(core.ClassifyFromPage(page, classifyEcosiaDocument), core.ErrCaptcha)
 }

 func (e *Ecosia) parseResult(elem *rod.Element, rank int, ad bool) (core.SearchResult, bool) {
@@ -96,36 +96,16 @@ func (e *Ecosia) parseResult(elem *rod.Element, rank int, ad bool) (core.SearchR
 	if err != nil {
 		return core.SearchResult{}, false
 	}
-	hrefStr := strings.TrimSpace(href.String())
-	if hrefStr == "" || strings.HasPrefix(hrefStr, "javascript:") {
-		return core.SearchResult{}, false
-	}

-	title := ""
-	if t, err := elem.Element(Selectors.Title); err == nil {
-		title, _ = t.Text()
-	} else if t, err := elem.Element("h2, h3"); err == nil {
-		title, _ = t.Text()
-	}
-
-	desc := ""
-	if d, err := elem.Element(Selectors.Desc); err == nil {
-		desc, _ = d.Text()
-	}
-
-	return core.SearchResult{
-		Rank:        rank,
-		URL:         hrefStr,
-		Title:       strings.TrimSpace(title),
-		Description: strings.TrimSpace(desc),
-		Ad:          ad,
-	}, true
+	title := core.FirstNonEmptyText(elem, Selectors.Title, "h2, h3")
+	desc := core.FirstNonEmptyText(elem, Selectors.Desc)
+	return assembleEcosiaRow(href.String(), title, desc, rank, ad)
 }

 // Search executes an Ecosia web search and returns normalized search results.
 // It may return core.ErrCaptcha or core.ErrSearchTimeout.
 func (e *Ecosia) Search(ctx context.Context, query core.Query) (results []core.SearchResult, err error) {
-	ctx = core.PrepareEngineContext(ctx, query, e.Name(), false)
+	ctx = core.PrepareEngineContext(ctx, query, e.Name())
 	scoped := *e
 	scoped.logger = e.logger.WithRequest(ctx)
 	e = &scoped
@@ -210,7 +190,7 @@ func (e *Ecosia) Search(ctx context.Context, query core.Query) (results []core.S
 		}
 	}

-	setSeparatedAdAbsoluteRanks(all, query.Start)
+	core.SetSeparatedAdAbsoluteRanks(all, query.Start)
 	deduped := core.DeduplicateResults(all)
 	if query.Limit > 0 {
 		deduped = core.LimitOrganicResults(deduped, query.Limit)
@@ -230,10 +210,6 @@ func (e *Ecosia) parseImageResult(el *rod.Element, rank int) (core.SearchResult,
 	if err != nil {
 		return core.SearchResult{}, false
 	}
-	imgURL := strings.TrimSpace(href.String())
-	if imgURL == "" {
-		return core.SearchResult{}, false
-	}

 	title := ""
 	if img, err := link.Element("img"); err == nil {
@@ -241,33 +217,19 @@ func (e *Ecosia) parseImageResult(el *rod.Element, rank int) (core.SearchResult,
 			title = strings.TrimSpace(*alt)
 		}
 	}
+	source := elementText(el, Selectors.ImageSource)
+	dims := elementText(el, Selectors.ImageDims)
+	return assembleEcosiaImageRow(href.String(), title, source, dims, rank)
+}

-	source := ""
-	if s, err := el.Element(Selectors.ImageSource); err == nil {
-		source, _ = s.Text()
-		source = strings.TrimSpace(source)
+// elementText returns the trimmed text of the first descendant matching
+// selector, or "" if absent.
+func elementText(el *rod.Element, selector string) string {
+	if e, err := el.Element(selector); err == nil {
+		text, _ := e.Text()
+		return strings.TrimSpace(text)
 	}
-	dims := ""
-	if d, err := el.Element(Selectors.ImageDims); err == nil {
-		dims, _ = d.Text()
-		dims = strings.TrimSpace(dims)
-	}
-
-	desc := source
-	if dims != "" {
-		if source != "" {
-			desc = fmt.Sprintf("%s (%s)", source, dims)
-		} else {
-			desc = dims
-		}
-	}
-
-	return core.SearchResult{
-		Rank:        rank,
-		URL:         imgURL,
-		Title:       title,
-		Description: desc,
-	}, true
+	return ""
 }

 // SearchImage executes an Ecosia image search and returns normalized image
@@ -275,7 +237,7 @@ func (e *Ecosia) parseImageResult(el *rod.Element, rank int) (core.SearchResult,
 // query.Start is ignored: per-page card count varies, so callers should
 // drive depth through query.Limit alone.
 func (e *Ecosia) SearchImage(ctx context.Context, query core.Query) (results []core.SearchResult, err error) {
-	ctx = core.PrepareEngineContext(ctx, query, e.Name(), false)
+	ctx = core.PrepareEngineContext(ctx, query, e.Name())
 	scoped := *e
 	scoped.logger = e.logger.WithRequest(ctx)
 	e = &scoped
--- a/ecosia/search_raw.go
+++ b/ecosia/search_raw.go
@@ -12,21 +12,23 @@ import (
 	"github.com/karust/openserp/core"
 )

+// isCaptchaDoc reports whether a parsed Ecosia document is a Cloudflare
+// challenge rather than a SERP. Prefers the hidden Turnstile input, then falls
+// back to the shared challenge-page body phrases (see cfBodyMarkers).
+func isCaptchaDoc(doc *goquery.Document) bool {
+	err := core.ClassifyChallengeDocument(doc, core.DocSignals{
+		CaptchaSelectors: []string{Selectors.Captcha},
+		CaptchaMarkers:   cfBodyMarkers,
+	})
+	return errors.Is(err, core.ErrCaptcha)
+}
+
 func classifyEcosiaRawHTML(body []byte) error {
 	doc, err := goquery.NewDocumentFromReader(bytes.NewReader(body))
 	if err != nil {
 		return err
 	}
-	if strings.Contains(strings.ToLower(doc.Text()), "captcha") {
-		return core.ErrCaptcha
-	}
-	if doc.Find("[data-test-id='web-no-results']").Length() > 0 ||
-		(doc.Find(Selectors.Mainline).Length() > 0 &&
-			doc.Find(Selectors.Result).Length() == 0 &&
-			doc.Find(Selectors.Ad).Length() == 0) {
-		return core.ErrEmptyResult
-	}
-	return nil
+	return classifyEcosiaDocument(doc)
 }

 // imageResultParser parses an Ecosia image SERP HTML response into search
@@ -41,37 +43,21 @@ func imageResultParser(response *http.Response) ([]core.SearchResult, error) {
 		rank    = 1
 	)
 	doc.Find(Selectors.ImageResult).Each(func(_ int, s *goquery.Selection) {
-		href, ok := s.Find(Selectors.ImageLink).Attr("href")
-		if !ok || strings.TrimSpace(href) == "" {
-			return
+		link := s.Find(Selectors.ImageLink)
+		href, _ := link.Attr("href")
+		title, _ := link.Find("img").Attr("alt")
+		source := strings.TrimSpace(s.Find(Selectors.ImageSource).Text())
+		dims := strings.TrimSpace(s.Find(Selectors.ImageDims).Text())
+		if res, ok := assembleEcosiaImageRow(href, strings.TrimSpace(title), source, dims, rank); ok {
+			results = append(results, res)
+			rank++
 		}
-		title, _ := s.Find(Selectors.ImageLink).Find("img").Attr("alt")
-		title = strings.TrimSpace(title)
-		var (
-			source = strings.TrimSpace(s.Find(Selectors.ImageSource).Text())
-			dims   = strings.TrimSpace(s.Find(Selectors.ImageDims).Text())
-			desc   = source
-		)
-		if dims != "" {
-			if source != "" {
-				desc = source + " (" + dims + ")"
-			} else {
-				desc = dims
-			}
-		}
-		results = append(results, core.SearchResult{
-			Rank:        rank,
-			URL:         href,
-			Title:       title,
-			Description: desc,
-		})
-		rank++
 	})
 	return results, nil
 }

 func Search(ctx context.Context, query core.Query) (results []core.SearchResult, err error) {
-	ctx = core.PrepareEngineContext(ctx, query, "ecosia", false)
+	ctx = core.PrepareEngineContext(ctx, query, "ecosia")

 	pageNum, startRank, err := startPage(query.Start)
 	if err != nil {
@@ -124,11 +110,12 @@ func Search(ctx context.Context, query core.Query) (results []core.SearchResult,
 		parsedResults[i].Rank = startRank + organicIdx
 		organicIdx++
 	}
-	setSeparatedAdAbsoluteRanks(parsedResults, pageNum*10)
+	core.SetSeparatedAdAbsoluteRanks(parsedResults, pageNum*10)

 	core.WithRequest(ctx).WithField("results_count", len(parsedResults)).Debug(
 		fmt.Sprintf("Ecosia Raw results : %v", parsedResults),
 	)

-	return core.DeduplicateResults(parsedResults), nil
+	deduped := core.StripResultFeatures(core.DeduplicateResults(parsedResults), query.Features)
+	return deduped, nil
 }
--- a/ecosia/search_raw_test.go
+++ b/ecosia/search_raw_test.go
@@ -44,13 +44,66 @@ func TestEcosiaImageResultParser(t *testing.T) {
 func TestEcosiaClassifyRawHTML(t *testing.T) {
 	t.Parallel()

-	body, err := io.ReadAll(testutil.ResponseFromFixture(t, "search_no_results.html").Body)
-	if err != nil {
-		t.Fatalf("read fixture body: %v", err)
+	tests := []struct {
+		fixture string
+		want    error
+	}{
+		{"search_no_results.html", core.ErrEmptyResult},
+		{"search_captcha.html", core.ErrCaptcha},
+		{"search_results.html", nil},
 	}

-	err = classifyEcosiaRawHTML(body)
-	if !errors.Is(err, core.ErrEmptyResult) {
-		t.Fatalf("expected %v for search_no_results.html, got %v", core.ErrEmptyResult, err)
+	for _, tt := range tests {
+		t.Run(tt.fixture, func(t *testing.T) {
+			t.Parallel()
+
+			body, err := io.ReadAll(testutil.ResponseFromFixture(t, tt.fixture).Body)
+			if err != nil {
+				t.Fatalf("read fixture body: %v", err)
+			}
+
+			got := classifyEcosiaRawHTML(body)
+			if tt.want == nil {
+				if got != nil {
+					t.Fatalf("expected nil for %s, got %v", tt.fixture, got)
+				}
+				return
+			}
+			if !errors.Is(got, tt.want) {
+				t.Fatalf("expected %v for %s, got %v", tt.want, tt.fixture, got)
+			}
+		})
+	}
+}
+
+func TestEcosiaParseHTMLClassifiesCaptchaAndNoResults(t *testing.T) {
+	t.Parallel()
+
+	tests := []struct {
+		fixture string
+		wantErr error
+	}{
+		{"search_captcha.html", core.ErrCaptcha},
+		{"search_no_results.html", nil},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.fixture, func(t *testing.T) {
+			t.Parallel()
+
+			results, err := ParseHTML(testutil.ResponseFromFixture(t, tt.fixture).Body)
+			if tt.wantErr != nil {
+				if !errors.Is(err, tt.wantErr) {
+					t.Fatalf("expected %v for %s, got %v", tt.wantErr, tt.fixture, err)
+				}
+				return
+			}
+			if err != nil {
+				t.Fatalf("ParseHTML() error = %v", err)
+			}
+			if len(results) != 0 {
+				t.Fatalf("expected zero results for %s, got %d", tt.fixture, len(results))
+			}
+		})
 	}
 }
--- a/ecosia/selectors.go
+++ b/ecosia/selectors.go
@@ -2,6 +2,8 @@ package ecosia

 // Selectors is the single source of truth for Ecosia SERP CSS selectors.
 var Selectors = struct {
+	Captcha     string
+	NoResults   string
 	Mainline    string
 	Result      string
 	Ad          string
@@ -13,6 +15,11 @@ var Selectors = struct {
 	ImageSource string
 	ImageDims   string
 }{
+	// Captcha matches Ecosia's Cloudflare Turnstile interstitial. The hidden
+	// cf-turnstile-response input is present on every challenge page and never
+	// on a real SERP, so it's a precise marker for the raw (browserless) path.
+	Captcha:     "input[name='cf-turnstile-response']",
+	NoResults:   "[data-test-id='web-no-results']",
 	Mainline:    "[data-test-id='mainline']",
 	Result:      "[data-test-id='mainline-result-web']",
 	Ad:          "[data-test-id='mainline-result-ad']",
--- a/ecosia/testdata/search_captcha.html
+++ b/ecosia/testdata/search_captcha.html
@@ -0,0 +1 @@
+<html xmlns="http://test.test" dir="ltr"><head><title>Just a moment...</title></head><body><main><h1>Confirm you’re not a robot</h1><p>Our system has detected unusual traffic from your network. Please solve the challenge below to show you’re not a robot.</p><div><div class="main-wrapper" role="main"><div class="main-content"><h2 id="FbRG5"></h2><p id="jpqrL2"></p><div id="BbLB6" style=""><div><div><input type="hidden" name="cf-turnstile-response" id="id_1"/></div></div></div><div id="TQpKs1" style=""><div>Verification successful. Waiting for www.ecosia.org to respond</div></div><div id="ROlTq4" class="spacer loading-verifying" style=""><div class="lds-ring"><div></div><div></div><div></div><div></div></div></div></div></div></div></main></body></html>
--- a/examples/content/js-search-with-extract/README.md
+++ b/examples/content/js-search-with-extract/README.md
@@ -8,4 +8,4 @@ npm install
 node index.js
 ```

-Edit the `text`, `extractTop`, or `extractMode` in [index.js](index.js) to tune it.
+Edit the `text`, `extract` depth, or `extractMode` in [index.js](index.js) to tune it.
--- a/examples/content/js-search-with-extract/index.js
+++ b/examples/content/js-search-with-extract/index.js
@@ -4,14 +4,13 @@ import { OpenSERP } from "@openserp/sdk";
 //   const client = new OpenSERP({ apiKey: "<YOUR_API_TOKEN>", timeoutMs: 60_000 });
 const client = new OpenSERP({ baseUrl: "http://localhost:7000", timeoutMs: 60_000 });

-// `extract: true` fetches the top pages and returns their cleaned content
-// alongside each result, so you get the page text in a single request.
-// `extractTop` (max 5) controls how many results are enriched.
+// `extract: N` fetches the top N pages (max 5) and returns their cleaned
+// content alongside each result, so you get the page text in a single request.
+// `extract: true` is shorthand for the top result.
 const { results } = await client.search({
  engine: "ecosia",
  text: "what is a serp api",
-  extract: true,
-  extractTop: 2,
+  extract: 2,
  extractMode: "auto",
 });

--- a/go.mod
+++ b/go.mod
@@ -1,6 +1,6 @@
 module github.com/karust/openserp

-go 1.24
+go 1.24.1

 toolchain go1.24.6

@@ -9,19 +9,19 @@ require (
 	github.com/JohannesKaufmann/html-to-markdown/v2 v2.4.0
 	github.com/PuerkitoBio/goquery v1.10.3
 	github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5
-	github.com/corpix/uarand v0.2.0
+	github.com/bogdanfinn/fhttp v0.6.8
+	github.com/bogdanfinn/tls-client v1.15.1
 	github.com/go-rod/rod v0.116.2
-	github.com/gofiber/fiber/v2 v2.52.9
+	github.com/gofiber/fiber/v2 v2.52.13
 	github.com/google/uuid v1.6.0
 	github.com/markusmobius/go-trafilatura v1.12.2
-	github.com/refraction-networking/utls v1.8.0
-	github.com/sirupsen/logrus v1.9.3
+	github.com/sirupsen/logrus v1.9.4
 	github.com/spf13/cobra v1.9.1
-	github.com/spf13/pflag v1.0.7
+	github.com/spf13/pflag v1.0.10
 	github.com/spf13/viper v1.20.1
 	github.com/ysmood/gson v0.7.3
-	golang.org/x/net v0.43.0
-	golang.org/x/time v0.12.0
+	golang.org/x/net v0.50.0
+	golang.org/x/time v0.14.0
 	gopkg.in/yaml.v3 v3.0.1
 )

@@ -31,6 +31,11 @@ require (
 	github.com/andybalholm/brotli v1.2.0 // indirect
 	github.com/andybalholm/cascadia v1.3.3 // indirect
 	github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de // indirect
+	github.com/bdandy/go-errors v1.2.2 // indirect
+	github.com/bdandy/go-socks4 v1.2.3 // indirect
+	github.com/bogdanfinn/quic-go-utls v1.0.9-utls // indirect
+	github.com/bogdanfinn/utls v1.7.7-barnius // indirect
+	github.com/bogdanfinn/websocket v1.5.5-barnius // indirect
 	github.com/elliotchance/pie/v2 v2.9.0 // indirect
 	github.com/forPelevin/gomoji v1.2.0 // indirect
 	github.com/fsnotify/fsnotify v1.9.0 // indirect
@@ -42,7 +47,7 @@ require (
 	github.com/hablullah/go-juliandays v1.0.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jalaali/go-jalaali v0.0.0-20210801064154-80525e88d958 // indirect
-	github.com/klauspost/compress v1.18.0 // indirect
+	github.com/klauspost/compress v1.18.2 // indirect
 	github.com/markusmobius/go-dateparser v1.2.3 // indirect
 	github.com/markusmobius/go-domdistiller v0.0.0-20240926050704-25b8d046ffb4 // indirect
 	github.com/markusmobius/go-htmldate v1.9.1 // indirect
@@ -50,14 +55,15 @@ require (
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/mattn/go-runewidth v0.0.16 // indirect
 	github.com/pelletier/go-toml/v2 v2.2.4 // indirect
+	github.com/quic-go/qpack v0.6.0 // indirect
 	github.com/rivo/uniseg v0.4.7 // indirect
-	github.com/rogpeppe/go-internal v1.10.0 // indirect
 	github.com/rs/zerolog v1.33.0 // indirect
 	github.com/sagikazarmark/locafero v0.10.0 // indirect
 	github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
 	github.com/spf13/afero v1.14.0 // indirect
 	github.com/spf13/cast v1.9.2 // indirect
 	github.com/subosito/gotenv v1.6.0 // indirect
+	github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 // indirect
 	github.com/tetratelabs/wazero v1.8.1 // indirect
 	github.com/valyala/bytebufferpool v1.0.0 // indirect
 	github.com/valyala/fasthttp v1.65.0 // indirect
@@ -68,9 +74,8 @@ require (
 	github.com/ysmood/goob v0.4.0 // indirect
 	github.com/ysmood/got v0.41.0 // indirect
 	github.com/ysmood/leakless v0.9.0 // indirect
-	golang.org/x/crypto v0.41.0 // indirect
+	golang.org/x/crypto v0.48.0 // indirect
 	golang.org/x/exp v0.0.0-20241009180824-f66d83c29e7c // indirect
-	golang.org/x/sys v0.35.0 // indirect
-	golang.org/x/text v0.28.0 // indirect
-	gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
+	golang.org/x/sys v0.41.0 // indirect
+	golang.org/x/text v0.34.0 // indirect
 )
--- a/go.sum
+++ b/go.sum
@@ -16,9 +16,21 @@ github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de h1:FxWPpzIjnTlhP
 github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de/go.mod h1:DCaWoUhZrYW9p1lxo/cm8EmUOOzAPSEZNGF2DK1dJgw=
 github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 h1:0CwZNZbxp69SHPdPJAN/hZIm0C4OItdklCFmMRWYpio=
 github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkYZB8zMSxRWpUBQtwG5a7fFgvEO+odwuTv2gs=
+github.com/bdandy/go-errors v1.2.2 h1:WdFv/oukjTJCLa79UfkGmwX7ZxONAihKu4V0mLIs11Q=
+github.com/bdandy/go-errors v1.2.2/go.mod h1:NkYHl4Fey9oRRdbB1CoC6e84tuqQHiqrOcZpqFEkBxM=
+github.com/bdandy/go-socks4 v1.2.3 h1:Q6Y2heY1GRjCtHbmlKfnwrKVU/k81LS8mRGLRlmDlic=
+github.com/bdandy/go-socks4 v1.2.3/go.mod h1:98kiVFgpdogR8aIGLWLvjDVZ8XcKPsSI/ypGrO+bqHI=
+github.com/bogdanfinn/fhttp v0.6.8 h1:LiQyHOY3i0QoxxNB7nq27/nGNNbtPj0fuBPozhR7Ws4=
+github.com/bogdanfinn/fhttp v0.6.8/go.mod h1:A+EKDzMx2hb4IUbMx4TlkoHnaJEiLl8r/1Ss1Y+5e5M=
+github.com/bogdanfinn/quic-go-utls v1.0.9-utls h1:tV6eDEiRbRCcepALSzxR94JUVD3N3ACIiRLgyc2Ep8s=
+github.com/bogdanfinn/quic-go-utls v1.0.9-utls/go.mod h1:aHph9B9H9yPOt5xnhWKSOum27DJAqpiHzwX+gjvaXcg=
+github.com/bogdanfinn/tls-client v1.15.1 h1:KiFAlED55DJ8Fcocn+/1nX6PrDFcttIHAf/GDkV6KN8=
+github.com/bogdanfinn/tls-client v1.15.1/go.mod h1:LsU6mXVn8MOFDwTkyRfI7V1BZM1p0wf2ZfZsICW/1fM=
+github.com/bogdanfinn/utls v1.7.7-barnius h1:OuJ497cc7F3yKNVHRsYPQdGggmk5x6+V5ZlrCR7fOLU=
+github.com/bogdanfinn/utls v1.7.7-barnius/go.mod h1:aAK1VZQlpKZClF1WEQeq6kyclbkPq4hz6xTbB5xSlmg=
+github.com/bogdanfinn/websocket v1.5.5-barnius h1:bY+qnxpai1qe7Jmjx+Sds/cmOSpuuLoR8x61rWltjOI=
+github.com/bogdanfinn/websocket v1.5.5-barnius/go.mod h1:gvvEw6pTKHb7yOiFvIfAFTStQWyrm25BMVCTj5wRSsI=
 github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
-github.com/corpix/uarand v0.2.0 h1:U98xXwud/AVuCpkpgfPF7J5TQgr7R5tqT8VZP5KWbzE=
-github.com/corpix/uarand v0.2.0/go.mod h1:/3Z1QIqWkDIhf6XWn/08/uMHoQ8JUoTIKc2iPchBOmM=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
@@ -40,8 +52,8 @@ github.com/go-shiori/go-readability v0.0.0-20241012063810-92284fa8a71f/go.mod h1
 github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
 github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
 github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
-github.com/gofiber/fiber/v2 v2.52.9 h1:YjKl5DOiyP3j0mO61u3NTmK7or8GzzWzCFzkboyP5cw=
-github.com/gofiber/fiber/v2 v2.52.9/go.mod h1:YEcBbO/FB+5M1IZNBP9FO3J9281zgPAreiI1oqg8nDw=
+github.com/gofiber/fiber/v2 v2.52.13 h1:TOKP64iqC9b5P49VrBW5tHhUOvDyrtJ0xePEfzJbCbk=
+github.com/gofiber/fiber/v2 v2.52.13/go.mod h1:YEcBbO/FB+5M1IZNBP9FO3J9281zgPAreiI1oqg8nDw=
 github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f h1:3BSP1Tbs2djlpprl7wCLuiqMaUh5SJkkzI2gDs+FgLs=
 github.com/gogs/chardet v0.0.0-20211120154057-b7413eaefb8f/go.mod h1:Pcatq5tYkCW2Q6yrR2VRHlbHpZ/R4/7qyL1TCF7vl14=
 github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
@@ -56,13 +68,10 @@ github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
 github.com/jalaali/go-jalaali v0.0.0-20210801064154-80525e88d958 h1:qxLoi6CAcXVzjfvu+KXIXJOAsQB62LXjsfbOaErsVzE=
 github.com/jalaali/go-jalaali v0.0.0-20210801064154-80525e88d958/go.mod h1:Wqfu7mjUHj9WDzSSPI5KfBclTTEnLveRUFr/ujWnTgE=
-github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
-github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
-github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
+github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
+github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
 github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
 github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
-github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
-github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
 github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
 github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
 github.com/magefile/mage v1.15.1-0.20230912152418-9f54e0f83e2a h1:tdPcGgyiH0K+SbsJBBm2oPyEIOTAvLBwD9TuUwVtZho=
@@ -90,8 +99,8 @@ github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/refraction-networking/utls v1.8.0 h1:L38krhiTAyj9EeiQQa2sg+hYb4qwLCqdMcpZrRfbONE=
-github.com/refraction-networking/utls v1.8.0/go.mod h1:jkSOEkLqn+S/jtpEHPOsVv/4V4EVnelwbMQl4vCWXAM=
+github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8=
+github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII=
 github.com/rivo/uniseg v0.1.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
@@ -109,8 +118,8 @@ github.com/sebdah/goldie/v2 v2.7.1 h1:PkBHymaYdtvEkZV7TmyqKxdmn5/Vcj+8TpATWZjnG5
 github.com/sebdah/goldie/v2 v2.7.1/go.mod h1:oZ9fp0+se1eapSRjfYbsV/0Hqhbuu3bJVvKI/NNtssI=
 github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw=
 github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
-github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
-github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
+github.com/sirupsen/logrus v1.9.4 h1:TsZE7l11zFCLZnZ+teH4Umoq5BhEIfIzfRDZ1Uzql2w=
+github.com/sirupsen/logrus v1.9.4/go.mod h1:ftWc9WdOfJ0a92nsE2jF5u5ZwH8Bv2zdeOC42RjbV2g=
 github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
 github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8/go.mod h1:3n1Cwaq1E1/1lhQhtRK2ts/ZwZEhjcQeJQ1RuC6Q/8U=
 github.com/spf13/afero v1.14.0 h1:9tH6MapGnn/j0eb0yIXiLjERO8RB6xIVZRDCX7PtqWA=
@@ -120,16 +129,18 @@ github.com/spf13/cast v1.9.2/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqe
 github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
 github.com/spf13/cobra v1.9.1/go.mod h1:nDyEzZ8ogv936Cinf6g1RU9MRY64Ir93oCnqb9wxYW0=
 github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
-github.com/spf13/pflag v1.0.7 h1:vN6T9TfwStFPFM5XzjsvmzZkLuaLX+HS+0SeFLRgU6M=
-github.com/spf13/pflag v1.0.7/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
+github.com/spf13/pflag v1.0.10 h1:4EBh2KAYBwaONj6b2Ye1GiHfwjqyROoF4RwYO+vPwFk=
+github.com/spf13/pflag v1.0.10/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/spf13/viper v1.20.1 h1:ZMi+z/lvLyPSCoNtFCpqjy0S4kPbirhpTMwl8BkW9X4=
 github.com/spf13/viper v1.20.1/go.mod h1:P9Mdzt1zoHIG8m2eZQinpiBjo6kCmZSKBClNNqjJvu4=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
-github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
-github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
+github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
 github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
 github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
+github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5 h1:YqAladjX7xpA6BM04leXMWAEjS0mTZ5kUU9KRBriQJc=
+github.com/tam7t/hpkp v0.0.0-20160821193359-2b70b4024ed5/go.mod h1:2JjD2zLQYH5HO74y5+aE3remJQvl6q4Sn6aWA2wD1Ng=
 github.com/tetratelabs/wazero v1.8.1 h1:NrcgVbWfkWvVc4UtT4LRLDf91PsOzDzefMdwhLfA550=
 github.com/tetratelabs/wazero v1.8.1/go.mod h1:yAI0XTsMBhREkM/YDAK/zNou3GoiAce1P6+rp/wQhjs=
 github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
@@ -163,14 +174,16 @@ github.com/ysmood/leakless v0.9.0/go.mod h1:R8iAXPRaG97QJwqxs74RdwzcRHT1SWCGTNqY
 github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
 github.com/yuin/goldmark v1.7.13 h1:GPddIs617DnBLFFVJFgpo1aBfe/4xcvMc3SB5t/D0pA=
 github.com/yuin/goldmark v1.7.13/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
+go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko=
+go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
 golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
 golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
 golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
-golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
-golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
+golang.org/x/crypto v0.48.0 h1:/VRzVqiRSggnhY7gNRxPauEQ5Drw9haKdM0jqfcCFts=
+golang.org/x/crypto v0.48.0/go.mod h1:r0kV5h3qnFPlQnBSrULhlsRfryS2pmewsg+XfMgkVos=
 golang.org/x/exp v0.0.0-20241009180824-f66d83c29e7c h1:7dEasQXItcW1xKJ2+gg5VOiBnqWrJc+rq0DPKyvvdbY=
 golang.org/x/exp v0.0.0-20241009180824-f66d83c29e7c/go.mod h1:NQtJDoLvd6faHhE7m4T/1IY708gDefGGjR/iUW8yQQ8=
 golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
@@ -180,6 +193,7 @@ golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
 golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
+golang.org/x/net v0.0.0-20211104170005-ce137452f963/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
 golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
@@ -187,8 +201,8 @@ golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
 golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
 golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
 golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
-golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
-golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
+golang.org/x/net v0.50.0 h1:ucWh9eiCGyDR3vtzso0WMQinm2Dnt8cFMuQa9K33J60=
+golang.org/x/net v0.50.0/go.mod h1:UgoSli3F/pBgdJBHCTc+tp3gmrU4XswgGRgtnwWTfyM=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
@@ -198,9 +212,9 @@ golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
 golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -210,8 +224,8 @@ golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
 golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
 golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
-golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
+golang.org/x/sys v0.41.0 h1:Ivj+2Cp/ylzLiEU89QhWblYnOE9zerudt9Ftecq2C6k=
+golang.org/x/sys v0.41.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
 golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
@@ -223,6 +237,7 @@ golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
 golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
@@ -230,10 +245,10 @@ golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
 golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
 golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
 golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
-golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
-golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
-golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE=
-golang.org/x/time v0.12.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg=
+golang.org/x/text v0.34.0 h1:oL/Qq0Kdaqxa1KbNeMKwQq0reLCCaFtqu2eNuSeNHbk=
+golang.org/x/text v0.34.0/go.mod h1:homfLqTYRFyVYemLBFl5GgL/DWEiH5wcsQ5gSh1yziA=
+golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
+golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
 golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
--- a/google/captcha_selector_test.go
+++ b/google/captcha_selector_test.go
@@ -18,13 +18,29 @@ func TestGooglePageTypeSelectors(t *testing.T) {
 		wantHit  bool
 	}{
 		{"search_captcha.html", Selectors.Captcha, true},
+		{"search_captcha.html", Selectors.CaptchaPage, true},
 		{"search_captcha.html", Selectors.ResultStats, false},

+		{"search_captcha_new.html", Selectors.Captcha, true},
+		{"search_captcha_new.html", Selectors.CaptchaPage, true},
+		{"search_captcha_new.html", Selectors.ResultStats, false},
+
+		{"search_sorry.html", Selectors.CaptchaPage, true},
+		{"search_sorry.html", Selectors.ResultStats, false},
+
+		{"search_soft_block.html", Selectors.SoftBlock, true},
+		{"search_soft_block.html", Selectors.Captcha, false},
+		{"search_soft_block.html", Selectors.CaptchaPage, false},
+		{"search_soft_block.html", Selectors.ResultStats, false},
+
 		{"search_results.html", Selectors.ResultStats, true},
 		{"search_results.html", Selectors.Captcha, false},
+		{"search_results.html", Selectors.CaptchaPage, false},

 		{"search_no_results.html", Selectors.ResultStats, true},
 		{"search_no_results.html", Selectors.Captcha, false},
+		{"search_no_results.html", Selectors.CaptchaPage, false},
+		{"search_no_results.html", Selectors.NoResults, true},
 	}

 	for _, tt := range tests {
@@ -47,7 +63,7 @@ func assertSelector(t *testing.T, fixture, selector string, wantHit bool) {
 	got := doc.Find(selector).Length() > 0
 	if got != wantHit {
 		if wantHit {
-			t.Fatalf("selector %q not found in %s — update selectors.go", selector, fixture)
+			t.Fatalf("selector %q not found in %s - update selectors.go", selector, fixture)
 		} else {
 			t.Fatalf("selector %q unexpectedly present in %s", selector, fixture)
 		}
--- a/google/features.go
+++ b/google/features.go
@@ -1,6 +1,7 @@
 package google

 import (
+	"context"
 	"strings"

 	"github.com/PuerkitoBio/goquery"
@@ -11,30 +12,20 @@ import (
 func extractGoogleFeatures(doc *goquery.Document) []core.SerpFeature {
 	features := core.ExtractSerpFeaturesBySelectors(doc, []core.SerpFeatureSelector{
 		{
-			// Google's AI Overview prose renders into the main-col streaming
-			// container; the text is fragmented across span[data-subtree]/<strong>/
-			// <code> nodes, so the text selector takes the container's whole
-			// collapsed text to reconstruct the answer. The older data-mcpr/
-			// data-rsoextract containers are kept as fallbacks for other layouts.
+			// aimc carries the real AI Overview; mfc can be a placeholder.
 			Type:          core.ResultTypeAISummary,
 			Title:         "AI Overview",
-			Container:     []string{"div[data-container-id='main-col'][data-sfc-root='c']", "div[data-mcpr]", "div[aria-label*='AI Overview']", "div[data-rsoextract]"},
+			Container:     []string{"div[data-mcpr]:has(div[data-subtree='aimc'])", "div[data-container-id='main-col'][data-sfc-root='c']", "div[data-subtree='aifb']", "div[data-mcpr]", "div[aria-label*='AI Overview']", "div[jsname][data-rl]", "div[data-rsoextract]"},
 			TitleSelector: []string{"[role='heading']", "h2", "h3"},
-			TextSelector:  []string{"div[data-streaming-container]", "div[data-sncf='1']", "[data-attrid*='description']"},
-			LinkSelector:  []string{"a[href^='http']"},
+			TextSelector:  []string{"div[data-subtree='aimc']", "div[data-streaming-container]", "div[data-sncf='1']", "[data-attrid*='description']", "div[data-subtree='aifb']"},
+			LinkSelector:  []string{"div[data-subtree='aimc'] a[href^='http']", "a[href^='http']"},
 			Position:      1,
 			Confidence:    0.75,
-			// Emit a single AI Overview: the main-col container yields the prose;
-			// the data-mcpr fallback otherwise also matches and sweeps embedded CSS.
-			SingleMatch: true,
+			SingleMatch:   true,
 		},
 		{
-			Type:  core.ResultTypePeopleAlsoAsk,
-			Title: "People also ask",
-			// div[data-initq] is the single outer PAA module. jsname='yEVEwb'
-			// also matches inner expandable sub-panels (one per question), so
-			// using it as a container fragments the module into N features;
-			// keep it only as a fallback when data-initq is absent.
+			Type:         core.ResultTypePeopleAlsoAsk,
+			Title:        "People also ask",
 			Container:    []string{"div[data-initq]", "div[jsname='yEVEwb']"},
 			ItemSelector: []string{"div.related-question-pair[data-q]", "div[data-q]"},
 			LinkSelector: []string{"a[href^='http']"},
@@ -43,27 +34,23 @@ func extractGoogleFeatures(doc *goquery.Document) []core.SerpFeature {
 			SingleMatch:  true,
 		},
 		{
-			Type:  core.ResultTypeRelatedSearches,
-			Title: "Related searches",
-			// Scope to the dedicated related-search footer modules only. The
-			// main #rso results container also carries data-async-context, so a
-			// query:-prefix match there yields navigation chips, not searches.
+			Type:         core.ResultTypeRelatedSearches,
+			Title:        "Related searches",
 			Container:    []string{"div[jsname='yEVEwb'][role='navigation']", "div[data-abe='1']"},
 			ItemSelector: []string{"a[href*='/search?']"},
 			LinkSelector: []string{"a[href*='/search?']"},
 			Confidence:   0.6,
 		},
 	})
-	features = filterGooglePlaceholders(features)
-	return core.DeduplicateSerpFeatures(features)
+	return filterGooglePlaceholders(features)
 }

-// googlePlaceholderText flags AI-overview text that Google renders when no
-// summary exists ("An AI Overview is not available...") and bare expander
-// labels ("Show more") so we don't emit empty/false-positive features.
+// googlePlaceholderText catches empty AI Overview shells.
 var googlePlaceholderText = []string{
 	"ai overview is not available",
 	"an ai overview is not available for this search",
+	// Localized "An AI Overview is not available for this query" (ru).
+	"обзор от ии недоступен",
 }

 func filterGooglePlaceholders(features []core.SerpFeature) []core.SerpFeature {
@@ -82,8 +69,6 @@ func isGooglePlaceholder(feature core.SerpFeature) bool {
 	if text == "" || text == "show more" || text == "show less" {
 		return true
 	}
-	// A fallback container can wrap an inline <style> block whose collapsed text
-	// is CSS, not prose. Reject text that is clearly a stylesheet.
 	if looksLikeCSS(text) {
 		return true
 	}
@@ -95,23 +80,17 @@ func isGooglePlaceholder(feature core.SerpFeature) bool {
 	return false
 }

-// looksLikeCSS reports whether collapsed text is dominated by stylesheet syntax.
-// It keys off CSS-specific tokens (@keyframes/@media and "selector {" rule
-// blocks) rather than a raw brace count, so prose containing code samples with
-// braces is not misclassified.
+// looksLikeCSS catches fallback containers that swept inline styles.
 func looksLikeCSS(text string) bool {
 	if strings.Contains(text, "@keyframes") || strings.Contains(text, "@media") {
 		return true
 	}
-	// CSS rule blocks look like "} .cls {" / "} #id {"; prose almost never does.
 	if strings.Contains(text, "} .") || strings.Contains(text, "} #") {
 		return true
 	}
-	// A dense run of "prop: value;" declarations inside a "{ ... }" block is the
-	// other tell for inlined stylesheet text.
 	return strings.Contains(text, "{ ") && strings.Count(text, ": ") > 20 && strings.Count(text, ";") > 20
 }

-func extractGoogleFeaturesFromPage(page *rod.Page) []core.SerpFeature {
-	return core.FeaturesFromPage(page, extractGoogleFeatures)
+func extractGoogleFeaturesFromPage(ctx context.Context, page *rod.Page) []core.SerpFeature {
+	return core.FeaturesFromPageWithWait(ctx, page, extractGoogleFeatures)
 }
--- a/google/parse_html_test.go
+++ b/google/parse_html_test.go
@@ -2,11 +2,13 @@ package google

 import (
 	"bytes"
+	"errors"
 	"os"
 	"strings"
 	"testing"

 	"github.com/PuerkitoBio/goquery"
+	"github.com/karust/openserp/core"
 )

 func TestParseHTML(t *testing.T) {
@@ -126,6 +128,40 @@ func TestParseHTMLNoResults(t *testing.T) {
 	}
 }

+func TestParseHTMLCaptcha(t *testing.T) {
+	t.Parallel()
+
+	for _, fixture := range []string{"search_captcha.html", "search_captcha_new.html"} {
+		t.Run(fixture, func(t *testing.T) {
+			t.Parallel()
+
+			data, err := os.ReadFile("testdata/" + fixture)
+			if err != nil {
+				t.Fatalf("read fixture: %v", err)
+			}
+
+			results, err := ParseHTML(bytes.NewReader(data))
+			if !errors.Is(err, core.ErrCaptcha) {
+				t.Fatalf("expected ErrCaptcha, got results=%d err=%v", len(results), err)
+			}
+		})
+	}
+}
+
+func TestParseHTMLSoftBlock(t *testing.T) {
+	t.Parallel()
+
+	data, err := os.ReadFile("testdata/search_soft_block.html")
+	if err != nil {
+		t.Fatalf("read fixture: %v", err)
+	}
+
+	results, err := ParseHTML(bytes.NewReader(data))
+	if !errors.Is(err, core.ErrBlocked) {
+		t.Fatalf("expected ErrBlocked, got results=%d err=%v", len(results), err)
+	}
+}
+
 func TestParseHTMLAdsDoNotConsumeOrganicRank(t *testing.T) {
 	t.Parallel()

--- a/google/search.go
+++ b/google/search.go
@@ -42,6 +42,12 @@ func (gogl *Google) getTotalResults(page *rod.Page) (int, error) {
 		return 0, core.ErrParser
 	}

+	// Stats div is absent on many locales; probe first so .Search doesn't block
+	// the full selector timeout.
+	if has, _, err := page.Has(Selectors.ResultStats); err != nil || !has {
+		return 0, nil
+	}
+
 	resultsStats, err := page.Timeout(gogl.GetSelectorTimeout()).Search(Selectors.ResultStats)
 	if err != nil {
 		return 0, errors.New("Result stats not found: " + err.Error())
@@ -104,37 +110,50 @@ func (gogl *Google) solveCaptcha(page *rod.Page, sitekey, datas, proxyURL string
 	return true
 }

-func (gogl *Google) checkCaptcha(page *rod.Page, queryProxyURL string) bool {
-	has, _, _ := page.Has(Selectors.Captcha)
-	if !has {
+// classifyPage runs the same captcha/soft-block/no-results rules the raw HTML
+// path uses (search_raw.go), against a snapshot of the live page, so both
+// paths can't drift apart. On ErrCaptcha it then tries to solve, since that
+// needs the live page's captcha element attributes.
+func (gogl *Google) classifyPage(page *rod.Page, queryProxyURL string) error {
+	err := core.ClassifyFromPage(page, classifyGoogleDocument)
+	if page != nil {
+		if info, infoErr := page.Info(); infoErr == nil && isGoogleSorryURL(info.URL) {
+			err = core.ErrCaptcha
+		}
+	}
+	if err == nil {
+		return nil
+	}
+	if errors.Is(err, core.ErrCaptcha) && gogl.solveCaptchaOnPage(page, queryProxyURL) {
+		return nil
+	}
+	return err
+}
+
+func (gogl *Google) solveCaptchaOnPage(page *rod.Page, queryProxyURL string) bool {
+	if !gogl.IsSolveCaptcha || !gogl.CaptchaSolverEnabled {
+		return false
+	}
+	captchaDiv, err := page.Element(Selectors.Captcha)
+	if err != nil {
+		return false
+	}
+	sitekey, err := captchaDiv.Attribute("data-sitekey")
+	if err != nil || sitekey == nil {
+		gogl.logger.Error("Cannot get captcha sitekey: %v", err)
+		return false
+	}
+	dataS, err := captchaDiv.Attribute("data-s")
+	if err != nil || dataS == nil {
+		gogl.logger.Error("Cannot get captcha datas: %v", err)
 		return false
 	}

-	captchaDiv, err := page.Element(Selectors.Captcha)
-	if err != nil {
-		return true
+	proxyURL := queryProxyURL
+	if strings.TrimSpace(proxyURL) == "" {
+		proxyURL = gogl.ProxyURL
 	}
-
-	sitekey, err := captchaDiv.Attribute("data-sitekey")
-	if err != nil {
-		gogl.logger.Error("Cannot get captcha sitekey: %s", err)
-		return true
-	}
-
-	dataS, err := captchaDiv.Attribute("data-s")
-	if err != nil {
-		gogl.logger.Error("Cannot get captcha datas: %s", err)
-		return true
-	}
-
-	if gogl.IsSolveCaptcha && gogl.CaptchaSolverEnabled {
-		proxyURL := queryProxyURL
-		if strings.TrimSpace(proxyURL) == "" {
-			proxyURL = gogl.ProxyURL
-		}
-		return !gogl.solveCaptcha(page, *sitekey, *dataS, proxyURL)
-	}
-	return true
+	return gogl.solveCaptcha(page, *sitekey, *dataS, proxyURL)
 }

 func (gogl *Google) preparePage(page *rod.Page) {
@@ -147,7 +166,33 @@ func (gogl *Google) preparePage(page *rod.Page) {
 	}
 }

+// waitAnswersExpanded polls until the first PAA entry has expanded to a
+// title+body (≥2 text lines) or maxWait elapses, replacing a flat 2s sleep.
+func (gogl *Google) waitAnswersExpanded(ctx context.Context, answers rod.Elements, maxWait time.Duration) error {
+	if len(answers) == 0 || maxWait <= 0 {
+		return nil
+	}
+	deadline := time.Now().Add(maxWait)
+	for {
+		text, err := answers[0].Text()
+		if err == nil && len(strings.Split(text, "\n")) >= 2 {
+			return nil
+		}
+		if !time.Now().Before(deadline) {
+			return nil
+		}
+		if err := core.SleepContext(ctx, 100*time.Millisecond); err != nil {
+			return err
+		}
+	}
+}
+
 func (gogl *Google) acceptCookies(page *rod.Page) {
+	// Probe with Has first so a banner-less SERP doesn't block on .Search's full
+	// timeout (AGENTS.md: use Has for existence).
+	if has, _, err := page.Has(Selectors.CookieBtn); err != nil || !has {
+		return
+	}
 	diaglogBtns, err := page.Timeout(gogl.Timeout / 10).Search(Selectors.CookieBtn)
 	if err != nil {
 		gogl.logger.Debug("Cookie consent not found: %s", err)
@@ -183,7 +228,7 @@ func googleElementHasAdMarker(el *rod.Element) bool {
 // Search executes a Google web search and returns normalized search results.
 // It may return core.ErrCaptcha or core.ErrSearchTimeout.
 func (gogl *Google) Search(ctx context.Context, query core.Query) (results []core.SearchResult, err error) {
-	ctx = core.PrepareEngineContext(ctx, query, gogl.Name(), true)
+	ctx = core.PrepareEngineContext(ctx, query, gogl.Name())
 	scoped := *gogl
 	scoped.logger = gogl.logger.WithRequest(ctx)
 	gogl = &scoped
@@ -204,10 +249,9 @@ func (gogl *Google) Search(ctx context.Context, query core.Query) (results []cor
 	defer gogl.close(ctx, page)
 	gogl.preparePage(page)

-	// Check first if there captcha
-	if gogl.checkCaptcha(page, query.ProxyURL) {
-		gogl.logger.Error("Captcha detected: %s", url)
-		return nil, core.ErrCaptcha
+	if err := gogl.classifyPage(page, query.ProxyURL); err != nil && !errors.Is(err, core.ErrEmptyResult) {
+		gogl.logger.Error("Page classified as %v: %s", err, url)
+		return nil, err
 	}

 	// Accept cookie consent so Google renders its SERP feature modules
@@ -220,17 +264,16 @@ func (gogl *Google) Search(ctx context.Context, query core.Query) (results []cor
 	// different SERP markup for the same query.
 	searchResultElems, matchedSelector, err := core.WaitForElements(ctx, page, searchResultSelectors(), gogl.GetSelectorTimeout())
 	if err != nil {
-		if gogl.checkCaptcha(page, query.ProxyURL) {
-			gogl.logger.Error("Captcha detected: %s", url)
-			return nil, core.ErrCaptcha
+		if pageErr := gogl.classifyPage(page, query.ProxyURL); pageErr != nil {
+			if errors.Is(pageErr, core.ErrEmptyResult) {
+				return nil, nil
+			}
+			gogl.logger.Error("Page classified as %v: %s", pageErr, url)
+			return nil, pageErr
 		}
 		if core.IsContextDone(err) {
 			return nil, err
 		}
-		// Keep empty-SERP behavior for selector timeout only.
-		if errors.Is(err, core.ErrSearchTimeout) {
-			return nil, nil
-		}
 		return nil, core.ErrSearchTimeout
 	}
 	gogl.logger.Debug("Search result selector matched: %s (%d elements)", matchedSelector, len(searchResultElems))
@@ -241,9 +284,7 @@ func (gogl *Google) Search(ctx context.Context, query core.Query) (results []cor
 	}
 	gogl.logger.Info("Found %d total results", totalResults)

-	rank := query.Start
-	adRank := 1
-	absoluteRank := query.Start + 1
+	rank := core.NewRankStateAt(query.Start, query.Start+1)
 	// When matched by the canonical organic selector (div.tF2Cxc) every element
 	// is already an organic result, but the wrapper itself often lacks data-ved
 	// (it sits on the outer .g/data-hveid container). Only require data-ved when
@@ -299,10 +340,8 @@ func (gogl *Google) Search(ctx context.Context, query core.Query) (results []cor
 			} else {
 				srchRes.Description = strings.TrimSpace(text)
 			}
-			srchRes.Rank = adRank
-			srchRes.AbsoluteRank = absoluteRank
-			adRank++
-			absoluteRank++
+			srchRes.Rank, srchRes.AbsoluteRank = rank.Next(true)
+			searchResults = append(searchResults, srchRes)

 		} else if isAnswerBox {
 			// 2. Parse answer boxes
@@ -331,7 +370,8 @@ func (gogl *Google) Search(ctx context.Context, query core.Query) (results []cor
 				}

 			}
-			if err := core.SleepContext(ctx, 2*time.Second); err != nil {
+			// Poll for expansion (usually 200-400ms) instead of a flat 2s sleep.
+			if err := gogl.waitAnswersExpanded(ctx, answers, 2*time.Second); err != nil {
 				return nil, err
 			}

@@ -420,34 +460,22 @@ func (gogl *Google) Search(ctx context.Context, query core.Query) (results []cor
 			}
 			srchRes.Description = desc

-			rank += 1
-			srchRes.Rank = rank
-			srchRes.AbsoluteRank = absoluteRank
-			absoluteRank++
+			srchRes.Rank, srchRes.AbsoluteRank = rank.Next(false)
 			searchResults = append(searchResults, srchRes)
 			continue

 		} else {
 			continue
 		}
-
-		if srchRes.Ad && srchRes.Rank == 0 {
-			srchRes.Rank = adRank
-			adRank++
-		} else if !srchRes.Ad {
-			srchRes.Rank = rank
-		}
-		if srchRes.AbsoluteRank == 0 {
-			srchRes.AbsoluteRank = absoluteRank
-			absoluteRank++
-		}
-		searchResults = append(searchResults, srchRes)
 	}

 	deduped := core.DeduplicateResults(searchResults)
 	if len(deduped) == 0 {
-		if gogl.checkCaptcha(page, query.ProxyURL) {
-			return nil, core.ErrCaptcha
+		if pageErr := gogl.classifyPage(page, query.ProxyURL); pageErr != nil {
+			if errors.Is(pageErr, core.ErrEmptyResult) {
+				return nil, nil
+			}
+			return nil, pageErr
 		}
 		// Result candidates were found by Selectors.Results but none parsed
 		// into usable rows: treat as a genuine no-results SERP rather than a
@@ -458,7 +486,7 @@ func (gogl *Google) Search(ctx context.Context, query core.Query) (results []cor
 		return nil, core.ErrSearchTimeout
 	}
 	if query.Features {
-		deduped = core.AttachFeaturesToFirstResult(deduped, extractGoogleFeaturesFromPage(page))
+		deduped = core.AttachFeaturesToFirstResult(deduped, extractGoogleFeaturesFromPage(ctx, page))
 	}
 	return deduped, nil
 }
@@ -466,7 +494,7 @@ func (gogl *Google) Search(ctx context.Context, query core.Query) (results []cor
 // SearchImage executes a Google image search and returns normalized image
 // results. It may return core.ErrCaptcha or core.ErrSearchTimeout.
 func (gogl *Google) SearchImage(ctx context.Context, query core.Query) ([]core.SearchResult, error) {
-	ctx = core.PrepareEngineContext(ctx, query, gogl.Name(), true)
+	ctx = core.PrepareEngineContext(ctx, query, gogl.Name())
 	scoped := *gogl
 	scoped.logger = gogl.logger.WithRequest(ctx)
 	gogl = &scoped
@@ -504,7 +532,7 @@ func (gogl *Google) SearchImage(ctx context.Context, query core.Query) ([]core.S

 		resultElements, _, err := core.WaitForElements(ctx, page, []string{Selectors.ImageResults}, gogl.GetSelectorTimeout())
 		if err != nil {
-			if gogl.checkCaptcha(page, query.ProxyURL) {
+			if pageErr := gogl.classifyPage(page, query.ProxyURL); errors.Is(pageErr, core.ErrCaptcha) {
 				gogl.logger.Error("Captcha detected: %s", url)
 				return *core.ConvertSearchResultsMap(searchResultsMap), core.ErrCaptcha
 			}
--- a/google/search_raw.go
+++ b/google/search_raw.go
@@ -6,6 +6,7 @@ import (
 	"errors"
 	"fmt"
 	"io"
+	"regexp"
 	"strings"

 	"github.com/PuerkitoBio/goquery"
@@ -13,6 +14,10 @@ import (
 	"github.com/sirupsen/logrus"
 )

+// Matches a literal zero result count, e.g. "about 0 results". Requires "0"
+// as its own word so it doesn't match large counts like "10,500,000 results".
+var zeroResultsPattern = regexp.MustCompile(`\b0 results\b`)
+
 // ParseHTML parses a Google SERP HTML document and returns search results.
 // It is the pure parser used by both raw HTTP search and parse endpoints.
 func ParseHTML(r io.Reader) ([]core.SearchResult, error) {
@@ -20,14 +25,19 @@ func ParseHTML(r io.Reader) ([]core.SearchResult, error) {
 	if err != nil {
 		return nil, err
 	}
+	pageStatus := classifyGoogleDocument(doc)
+	if errors.Is(pageStatus, core.ErrEmptyResult) {
+		return []core.SearchResult{}, nil
+	}
+	if pageStatus != nil {
+		return nil, pageStatus
+	}
 	return parseGoogleDocument(doc), nil
 }

 func parseGoogleDocument(doc *goquery.Document) []core.SearchResult {
 	results := []core.SearchResult{}
-	rank := 1
-	adRank := 1
-	absoluteRank := 1
+	rank := core.NewRankState(0)

 	// Prefer the canonical organic result block (div.tF2Cxc, innermost). Fall
 	// back to the broad attribute selector only when no tF2Cxc blocks exist, so
@@ -81,14 +91,7 @@ func parseGoogleDocument(doc *goquery.Document) []core.SearchResult {
 		desc := descTag.Text()

 		if link != "" && link != "#" {
-			resultRank := rank
-			if isAd {
-				resultRank = adRank
-				adRank++
-			} else {
-				rank++
-			}
-
+			resultRank, absoluteRank := rank.Next(isAd)
 			result := core.SearchResult{
 				Rank:         resultRank,
 				AbsoluteRank: absoluteRank,
@@ -99,7 +102,6 @@ func parseGoogleDocument(doc *goquery.Document) []core.SearchResult {
 			}

 			results = append(results, result)
-			absoluteRank++
 		}
 	}

@@ -114,20 +116,64 @@ func classifyGoogleRawHTML(body []byte) error {
 	if err != nil {
 		return err
 	}
-	if doc.Find(Selectors.Captcha).Length() > 0 {
+	return classifyGoogleDocument(doc)
+}
+
+func isGoogleSorryURL(rawURL string) bool {
+	return strings.Contains(strings.ToLower(rawURL), "/sorry/")
+}
+
+func classifyGoogleDocument(doc *goquery.Document) error {
+	if isGoogleCaptchaDocument(doc) {
 		return core.ErrCaptcha
 	}
-
-	text := strings.ToLower(doc.Text())
-	if strings.Contains(text, "did not match any documents") ||
-		strings.Contains(text, "about 0 results") {
+	if isGoogleSoftBlockDocument(doc) {
+		return core.ErrBlocked
+	}
+	if isGoogleNoResultsDocument(doc) {
 		return core.ErrEmptyResult
 	}
 	return nil
 }

+func isGoogleCaptchaDocument(doc *goquery.Document) bool {
+	err := core.ClassifyChallengeDocument(doc, core.DocSignals{
+		CaptchaSelectors: []string{Selectors.CaptchaPage},
+		CaptchaMarkers:   Selectors.CaptchaMarkers,
+	})
+	return errors.Is(err, core.ErrCaptcha)
+}
+
+// isGoogleSoftBlockDocument reports the JS-retry interstitial Google serves
+// instead of a SERP.
+func isGoogleSoftBlockDocument(doc *goquery.Document) bool {
+	if doc.Find(Selectors.Results).Length() > 0 || doc.Find(Selectors.ResultsBroad).Length() > 0 {
+		return false
+	}
+	return strings.Contains(strings.ToLower(doc.Find(Selectors.SoftBlock).Text()), "/httpservice/retry/enablejs")
+}
+
+func isGoogleNoResultsDocument(doc *goquery.Document) bool {
+	stats := strings.ToLower(doc.Find(Selectors.ResultStats).Text())
+	if isZeroResultStats(stats) {
+		return true
+	}
+	text := strings.ToLower(doc.Find(Selectors.NoResults).Text())
+	if text == "" {
+		text = strings.ToLower(doc.Text())
+	}
+	return strings.Contains(text, "did not match any documents") || isZeroResultStats(text)
+}
+
+// isZeroResultStats reports whether s states a literal zero result count,
+// e.g. "about 0 results". A plain substring check on "0 results" also matches
+// large counts like "10,500,000 results", so require "0" as its own word.
+func isZeroResultStats(s string) bool {
+	return zeroResultsPattern.MatchString(s)
+}
+
 func Search(ctx context.Context, query core.Query) (results []core.SearchResult, err error) {
-	ctx = core.PrepareEngineContext(ctx, query, "google", false)
+	ctx = core.PrepareEngineContext(ctx, query, "google")

 	googleURL, err := BuildURL(query)
 	if err != nil {
@@ -143,6 +189,9 @@ func Search(ctx context.Context, query core.Query) (results []core.SearchResult,
 	core.WithRequest(ctx).WithField("status_code", res.StatusCode).Debug(
 		fmt.Sprintf("Google Raw response: code=%d", res.StatusCode),
 	)
+	if res.Request != nil && res.Request.URL != nil && isGoogleSorryURL(res.Request.URL.String()) {
+		return nil, core.ErrCaptcha
+	}

 	body, err := core.ReadRawSearchBody(res)
 	if err != nil {
@@ -183,5 +232,5 @@ func Search(ctx context.Context, query core.Query) (results []core.SearchResult,
 		fmt.Sprintf("Google Raw results : %v", parsedResults),
 	)

-	return parsedResults, nil
+	return core.StripResultFeatures(parsedResults, query.Features), nil
 }
--- a/google/search_raw_test.go
+++ b/google/search_raw_test.go
@@ -18,6 +18,7 @@ func TestGoogleParseHTMLFixtures(t *testing.T) {
 		minResultCount int
 		maxResultCount int
 		wantZero       bool
+		wantErr        error
 	}{
 		{
 			name:           "search results",
@@ -31,9 +32,19 @@ func TestGoogleParseHTMLFixtures(t *testing.T) {
 			wantZero: true,
 		},
 		{
-			name:     "captcha page",
-			fixture:  "search_captcha.html",
-			wantZero: true,
+			name:    "captcha page",
+			fixture: "search_captcha.html",
+			wantErr: core.ErrCaptcha,
+		},
+		{
+			name:    "new captcha page",
+			fixture: "search_captcha_new.html",
+			wantErr: core.ErrCaptcha,
+		},
+		{
+			name:    "soft block page",
+			fixture: "search_soft_block.html",
+			wantErr: core.ErrBlocked,
 		},
 	}

@@ -42,6 +53,12 @@ func TestGoogleParseHTMLFixtures(t *testing.T) {
 			t.Parallel()

 			results, err := ParseHTML(testutil.ResponseFromFixture(t, tt.fixture).Body)
+			if tt.wantErr != nil {
+				if !errors.Is(err, tt.wantErr) {
+					t.Fatalf("expected %v for %s, got %v", tt.wantErr, tt.fixture, err)
+				}
+				return
+			}
 			if err != nil {
 				t.Fatalf("ParseHTML() error = %v", err)
 			}
@@ -88,6 +105,9 @@ func TestGoogleClassifyRawHTML(t *testing.T) {
 	}{
 		{name: "no results", fixture: "search_no_results.html", want: core.ErrEmptyResult},
 		{name: "captcha page", fixture: "search_captcha.html", want: core.ErrCaptcha},
+		{name: "new captcha page", fixture: "search_captcha_new.html", want: core.ErrCaptcha},
+		{name: "sorry challenge page", fixture: "search_sorry.html", want: core.ErrCaptcha},
+		{name: "soft block page", fixture: "search_soft_block.html", want: core.ErrBlocked},
 	}

 	for _, tt := range tests {
--- a/google/selectors.go
+++ b/google/selectors.go
@@ -4,19 +4,23 @@ package google
 // Both the browser parser (search.go, rod) and HTML parser (search_raw.go,
 // goquery) reference these. When Google changes their DOM, edit here only.
 var Selectors = struct {
-	Captcha      string
-	ResultStats  string
-	CookieBtn    string
-	Results      string
-	ResultsBroad string
-	Ad           string
-	Link         string
-	Title        string
-	DescPrimary  string
-	DescFallback string
-	DescAny      string
-	AnswerBox    string
-	AnswerItem   string
+	Captcha        string
+	CaptchaPage    string
+	CaptchaMarkers []string
+	SoftBlock      string
+	NoResults      string
+	ResultStats    string
+	CookieBtn      string
+	Results        string
+	ResultsBroad   string
+	Ad             string
+	Link           string
+	Title          string
+	DescPrimary    string
+	DescFallback   string
+	DescAny        string
+	AnswerBox      string
+	AnswerItem     string

 	// Image search.
 	ImageResults      string
@@ -24,7 +28,19 @@ var Selectors = struct {
 	ImageLinkFallback string
 	ImageTitle        []string
 }{
-	Captcha:     "div[data-sitekey]",
+	Captcha:     "[data-sitekey]",
+	CaptchaPage: "form#captcha-form, form[action*='/sorry/'], body[onload*='captcha'], [data-sitekey], .g-recaptcha, script[src*='recaptcha']",
+	// CaptchaMarkers is the page-text fallback for captcha variants whose
+	// markup doesn't match CaptchaPage.
+	CaptchaMarkers: []string{
+		"detected unusual traffic",
+		"unusual traffic from your computer network",
+		"before you continue",
+		"not a robot",
+		"solve the captcha",
+	},
+	SoftBlock:   "noscript",
+	NoResults:   "#botstuff, #topstuff, .mnr-c",
 	ResultStats: "div#result-stats",
 	CookieBtn:   "div[role='dialog'][aria-modal] button",
 	// Results targets the canonical organic result block. div.tF2Cxc is the
--- a/google/serp_features_test.go
+++ b/google/serp_features_test.go
@@ -9,12 +9,7 @@ import (
 	"github.com/karust/openserp/core"
 )

-// TestParseHTMLFixtureExtractsRealFeatures guards against selectors drifting
-// away from the real-SERP fixture (a JavaScript "fetch API" query). The fixture
-// renders Google's AI Overview into the main-col streaming container, so the
-// raw parser must emit exactly one ai_summary carrying the full multi-paragraph
-// answer (not just the heading) plus its cited source links. The data-mcpr
-// fallback container must not also fire and leak inline CSS as a second summary.
+// TestParseHTMLFixtureExtractsRealFeatures guards the live AI Overview fixture.
 func TestParseHTMLFixtureExtractsRealFeatures(t *testing.T) {
 	t.Parallel()
 	f, err := os.Open("testdata/search_results.html")
@@ -27,26 +22,25 @@ func TestParseHTMLFixtureExtractsRealFeatures(t *testing.T) {
 	if err != nil {
 		t.Fatalf("ParseHTML() error = %v", err)
 	}
-	assertFeatureType(t, results, core.ResultTypeAISummary)

-	summaries := 0
+	var summaries int
 	for _, r := range results {
 		for _, ft := range r.Features {
 			if ft.Type != core.ResultTypeAISummary {
 				continue
 			}
 			summaries++
-			// The full answer body must be captured, not just the heading.
 			if len(ft.Text) < 200 {
 				t.Fatalf("ai_summary text looks truncated (%d chars): %q", len(ft.Text), ft.Text)
 			}
-			// A fallback container that swept a <style> block would surface CSS
-			// rule syntax rather than prose.
+			if strings.Contains(strings.ToLower(ft.Text), "недоступен") {
+				t.Fatalf("ai_summary captured the 'not available' placeholder: %q", ft.Text)
+			}
 			if strings.Contains(ft.Text, "@keyframes") || strings.Contains(ft.Text, "} .") {
 				t.Fatalf("ai_summary text contains CSS, not prose: %q", ft.Text)
 			}
 			if len(ft.Links) == 0 {
-				t.Fatal("expected the ai_summary to carry cited source links")
+				t.Fatal("expected ai_summary to carry cited source links")
 			}
 		}
 	}
--- a/google/testdata/search_captcha_new.html
+++ b/google/testdata/search_captcha_new.html
@@ -0,0 +1 @@
+<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"><meta name="viewport" content="initial-scale=1"><title>https://www.google.com/search?ie=UTF-8&amp;oq=weather&amp;pws=0&amp;q=weather&amp;sourceid=chrome</title></head><body style="font-family:arial,sans-serif;background-color:#fff;color:#000;padding:20px;font-size:18px;overscroll-behavior:contain" onload='e=document.getElementById("captcha"),e&&e.focus(),solveSimpleChallenge&&solveSimpleChallenge(0,0)'><div style="max-width:400px"><hr noshade="" size="1" style="color:#ccc;background-color:#ccc"><br><form id="captcha-form" action="index" method="post"><noscript><div style="font-size:13px">In order to continue, please enable javascript on your web browser.</div></noscript><script src="https://www.google.com/recaptcha/enterprise.js" async="" defer=""></script><script>var submitCallback=function(t){document.getElementById("captcha-form").submit()}</script><div id="recaptcha" class="g-recaptcha" data-sitekey="6LfwuyUTAAAAAOAmoS0fdqijC2PbbdH4kjq62Y1b" data-callback="submitCallback" data-s="_4UNXgVVh_xQKhxMBcOJ5Moeza-u52wxA1tuaG87XZwVDXRblsUTZ3nfrOeMsYYu1164fEiA0uhtkqHu7LXSgcBhNzeomoTeq5ztZoh5pdLLdqSzx75mljudOAJV7yfYiM4zWoGy4zg8xWWKZ6Rx5GRjBN_UH-y5crgkZlmPGAZoXiMo4-3lv1oR58YLrbUq8upst9DSbArZ9i31BwQ_fEWv-J17uDF8wzT_kEVPJxltBQQzNrzBs1eEBWS42p6BcIYzLAvjMNNqh_hPHbI7EDxCyhJqFz0"></div><input type="hidden" name="q" value="EgRr25U6GIS7kNIGIjCoB1yvjD_tndmRVr8xZHpYZiL5ETXEMM63dlqgULOz5rXS8U7M5LO3Aj1QSvFyO_syAnJSWgFD"><input type="hidden" name="continue" value="https://www.google.com/search?ie=UTF-8&amp;oq=weather&amp;pws=0&amp;q=weather&amp;sourceid=chrome"></form><hr noshade="" size="1" style="color:#ccc;background-color:#ccc"><div style="font-size:13px"><b>About this page</b><br><br>Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot.<a href="#" onclick='document.getElementById("infoDiv").style.display="block"'>Why did this happen?</a><br><br><div id="infoDiv" style="display:none;background-color:#eee;padding:10px;margin:0 0 15px 0;line-height:1.4em">This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the<a href="//www.google.com/policies/terms/">Terms of Service</a>. The block will expire shortly after those requests stop. In the meantime, solving the above CAPTCHA will let you continue to use our services.<br><br>This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests. If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible.<a href="//support.google.com/websearch/answer/86640">Learn more</a><br><br>Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly.</div><br><div style="font-size:13px;line-break:anywhere">IP address: 107.219.149.58<br>Time: 2026-06-30T19:48:20Z<br>URL: https://www.google.com/search?ie=UTF-8&amp;oq=weather&amp;pws=0&amp;q=weather&amp;sourceid=chrome<br></div></div></div></body></html>
--- a/google/testdata/search_soft_block.html
+++ b/google/testdata/search_soft_block.html
@@ -0,0 +1 @@
+<html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><title>weather - Google Search</title></head><body><noscript><iframe src="/httpservice/retry/enablejs?sei=abc123" frameborder="0" height="100%" width="100%"></iframe>Click here if you're not redirected within a few seconds.</noscript><div id="main"></div></body></html>
--- a/google/testdata/search_sorry.html
+++ b/google/testdata/search_sorry.html
@@ -0,0 +1,15 @@
+<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"><meta name="viewport" content="initial-scale=1"><title>https://www.google.com/sorry/index</title></head>
+<body style="font-family: arial, sans-serif; background-color: #fff; color: #000; padding:20px; font-size:18px; overscroll-behavior:contain;" onload="e=document.getElementById('captcha');if(e){e.focus();} if(solveSimpleChallenge) {solveSimpleChallenge(0,0);}">
+<div style="max-width:400px;">
+<hr noshade="" size="1" style="color:#ccc; background-color:#ccc;"><br>
+<div style="font-size:13px;">
+Our systems have detected unusual traffic from your computer network.  Please try your request again later.  <a href="#" onclick="document.getElementById('infoDiv0').style.display='block';">Why did this happen?</a><br><br>
+<div id="infoDiv0" style="display:none; background-color:#eee; padding:10px; margin:0 0 15px 0; line-height:1.4em;">
+This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the <a href="//www.google.com/policies/terms/">Terms of Service</a>. The block will expire shortly after those requests stop.<br><br>This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests.  If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible.  <a href="//support.google.com/websearch/answer/86640">Learn more</a><br><br>Sometimes you may see this page if you are using advanced terms that robots are known to use, or sending requests very quickly.
+</div>
+<br><div style="font-size:13px; line-break: anywhere;">
+IP address: 212.192.4.209<br>Time: 2026-07-01T20:58:45Z<br>URL: https://www.google.com/sorry/index<br>
+</div>
+</div>
+</div>
+</body></html>
--- a/google/url.go
+++ b/google/url.go
@@ -262,7 +262,7 @@ func BuildURL(q core.Query) (string, error) {
 	if q.DateInterval != "" {
 		intervals := strings.Split(q.DateInterval, "..")
 		if len(intervals) != 2 {
-			return "", errors.New("incorrect data interval provided")
+			return "", errors.New("incorrect date interval provided")
 		}

 		dataParam := fmt.Sprintf("cdr:1,cd_min:%s,cd_max:%s", intervals[0], intervals[1])
@@ -343,7 +343,7 @@ func BuildImageURL(q core.Query) (string, error) {
 	if q.DateInterval != "" {
 		intervals := strings.Split(q.DateInterval, "..")
 		if len(intervals) != 2 {
-			return "", errors.New("incorrect data interval provided")
+			return "", errors.New("incorrect date interval provided")
 		}

 		dataParam := fmt.Sprintf("cdr:1,cd_min:%s,cd_max:%s", intervals[0], intervals[1])
--- a/main.go
+++ b/main.go
@@ -11,7 +11,7 @@ func main() {
 	defer recoverPanic()

 	if err := cmd.RootCmd.Execute(); err != nil {
-		logrus.Info(err)
+		// Cobra already prints the error to stderr; just set the exit code.
 		os.Exit(1)
 	}
 }
--- a/yandex/features.go
+++ b/yandex/features.go
@@ -7,7 +7,7 @@ import (
 )

 func extractYandexFeatures(doc *goquery.Document) []core.SerpFeature {
-	features := core.ExtractSerpFeaturesBySelectors(doc, []core.SerpFeatureSelector{
+	return core.ExtractSerpFeaturesBySelectors(doc, []core.SerpFeatureSelector{
 		{
 			// Neuro/AI answer card (data-fast-name='neuro_answer'). Most of the
 			// answer body renders client-side from a data-state JSON blob, so the
@@ -42,7 +42,6 @@ func extractYandexFeatures(doc *goquery.Document) []core.SerpFeature {
 			Confidence:   0.7,
 		},
 	})
-	return core.DeduplicateSerpFeatures(features)
 }

 func extractYandexFeaturesFromPage(page *rod.Page) []core.SerpFeature {
--- a/yandex/parse_html.go
+++ b/yandex/parse_html.go
@@ -1,6 +1,7 @@
 package yandex

 import (
+	"errors"
 	"io"
 	"net/url"
 	"strings"
@@ -16,14 +17,26 @@ func ParseHTML(r io.Reader) ([]core.SearchResult, error) {
 	if err != nil {
 		return nil, err
 	}
+	pageStatus := classifyYandexDocument(doc)
+	if errors.Is(pageStatus, core.ErrEmptyResult) {
+		return []core.SearchResult{}, nil
+	}
+	if pageStatus != nil {
+		return nil, pageStatus
+	}
 	return parseYandexDocument(doc), nil
 }

+func classifyYandexDocument(doc *goquery.Document) error {
+	return core.ClassifyChallengeDocument(doc, core.DocSignals{
+		CaptchaSelectors: []string{Selectors.Captcha},
+		EmptySelectors:   []string{Selectors.NoResults},
+	})
+}
+
 func parseYandexDocument(doc *goquery.Document) []core.SearchResult {
 	var results []core.SearchResult
-	rank := 1
-	adRank := 1
-	absoluteRank := 1
+	rank := core.NewRankState(0)

 	doc.Find(Selectors.Results).Each(func(_ int, item *goquery.Selection) {
 		// The neuro/AI answer renders as a serp-item li too, so it would be
@@ -33,73 +46,79 @@ func parseYandexDocument(doc *goquery.Document) []core.SearchResult {
 			return
 		}

-		// Skip blocks without a result heading (filters out non-organic blocks
-		// that share the result-row container).
-		titleTag := item.Find(Selectors.Title).First()
-		if titleTag.Length() == 0 {
-			return
-		}
-
-		// Prefer the canonical organic-title link, then the closest <a> wrapping
-		// the title, then any <a> in the block.
-		linkTag := item.Find(Selectors.LinkPrimary).First()
-		if linkTag.Length() == 0 {
-			linkTag = titleTag.Closest("a")
-		}
-		if linkTag.Length() == 0 {
-			linkTag = item.Find(Selectors.Link).First()
-		}
-		if linkTag.Length() == 0 {
-			return
-		}
-
-		href, exists := linkTag.Attr("href")
-		if !exists {
-			return
-		}
-		href = strings.TrimSpace(href)
-		if href == "" || href == "#" || strings.HasPrefix(href, "javascript:") {
+		href, ok := yandexDocumentHref(item)
+		if !ok {
 			return
 		}
+		title := core.NormalizeWhitespace(item.Find(Selectors.Title).First().Text())
+		desc := firstNonEmptyText(item, Selectors.Desc, Selectors.DescFallback)
 		isAd := yandexSelectionHasAdMarker(item) || yandexURLLooksAd(href)

-		title := strings.TrimSpace(titleTag.Text())
-		if title == "" {
-			return
+		if res, ok := assembleYandexRow(href, title, desc, isAd, rank); ok {
+			results = append(results, res)
 		}
-
-		desc := ""
-		if descTag := item.Find(Selectors.Desc).First(); descTag.Length() > 0 {
-			desc = strings.TrimSpace(descTag.Text())
-		}
-		if desc == "" {
-			if descTag := item.Find(Selectors.DescFallback).First(); descTag.Length() > 0 {
-				desc = strings.TrimSpace(descTag.Text())
-			}
-		}
-
-		resultRank := rank
-		if isAd {
-			resultRank = adRank
-			adRank++
-		} else {
-			rank++
-		}
-
-		results = append(results, core.SearchResult{
-			Rank:         resultRank,
-			AbsoluteRank: absoluteRank,
-			URL:          href,
-			Title:        title,
-			Description:  desc,
-			Ad:           isAd,
-		})
-		absoluteRank++
 	})

 	return core.AttachFeaturesToFirstResult(core.DeduplicateResults(results), extractYandexFeatures(doc))
 }

+// assembleYandexRow validates an already-extracted Yandex row and assigns ranks.
+// Shared by the rod (browser) and goquery (raw / parse) parsers, which differ
+// only in how they pull title/href/desc out of the DOM.
+func assembleYandexRow(href, title, desc string, isAd bool, rank *core.RankState) (core.SearchResult, bool) {
+	href = strings.TrimSpace(href)
+	if href == "" || href == "#" || strings.HasPrefix(href, "javascript:") {
+		return core.SearchResult{}, false
+	}
+	if title == "" {
+		return core.SearchResult{}, false
+	}
+
+	resultRank, absoluteRank := rank.Next(isAd)
+	return core.SearchResult{
+		Rank:         resultRank,
+		AbsoluteRank: absoluteRank,
+		URL:          href,
+		Title:        title,
+		Description:  desc,
+		Ad:           isAd,
+	}, true
+}
+
+// yandexDocumentHref resolves a result row's link href in the goquery path:
+// the canonical organic-title link, then the <a> wrapping the title, then any
+// <a> in the block. ok=false when no anchor with an href attribute is found.
+func yandexDocumentHref(item *goquery.Selection) (string, bool) {
+	linkTag := item.Find(Selectors.LinkPrimary).First()
+	if linkTag.Length() == 0 {
+		linkTag = item.Find(Selectors.Title).First().Closest("a")
+	}
+	if linkTag.Length() == 0 {
+		linkTag = item.Find(Selectors.Link).First()
+	}
+	if linkTag.Length() == 0 {
+		return "", false
+	}
+	href, exists := linkTag.Attr("href")
+	if !exists {
+		return "", false
+	}
+	return href, true
+}
+
+// firstNonEmptyText returns the normalized text of the first selector that
+// matches a non-empty element, the goquery counterpart of core.FirstNonEmptyText.
+func firstNonEmptyText(item *goquery.Selection, selectors ...string) string {
+	for _, selector := range selectors {
+		if tag := item.Find(selector).First(); tag.Length() > 0 {
+			if text := core.NormalizeWhitespace(tag.Text()); text != "" {
+				return text
+			}
+		}
+	}
+	return ""
+}
+
 // isYandexNeuroAnswer reports whether a serp-item is the AI/neuro answer card.
 func isYandexNeuroAnswer(item *goquery.Selection) bool {
 	if name, ok := item.Attr("data-fast-name"); ok && name == "neuro_answer" {
--- a/Show More
+++ b/Show More
Author	SHA1	Message	Date
Rustem Kamalov	29c7b0fbe0	expose proxy-attempts + use-profile CORS headers; simplify DDG url builders	2026-07-22 15:17:16 +03:00
Parash Subedi	5d5d135fbf	feat: update DuckDuckGo search logic, use "more results" button for next page results (#41 ) * feat: implement windowOrganicResults function and update DuckDuckGo search logic * Add fallback ddg "more results" selector. Update test HTML + add test --------- Co-authored-by: Rustem Kamalov <karust13@gmail.com>	2026-07-22 00:11:20 +03:00
Rustem Kamalov	2a57bd4248	fix: /extract/batch per-URL errors, param parity, proxy headers. Update readme	2026-07-19 04:18:50 +03:00
Rustem Kamalov	01223e35d3	Merge pull request #39 from AIWintermuteAI/main Add `/extract/batch` path for batch URL extraction	2026-07-18 21:44:46 +03:00
Rustem Kamalov	00edd86c1c	fix: openapi yaml parse error + /extract/batch timeout exemption	2026-07-18 21:31:46 +03:00
AIWintermuteAI	e430584324	refactor: use errInvalidParam helper and add per-URL took_ms to batch extract	2026-07-18 09:56:20 +02:00
AIWintermuteAI	aec0d9c42a	cleanup	2026-07-18 09:49:38 +02:00
AIWintermuteAI	58fdf2b7a6	docs: add /extract/batch endpoint to OpenAPI spec	2026-07-18 00:19:15 +02:00
AIWintermuteAI	c3934bb75b	feat: add POST /extract/batch endpoint for WebUI integration	2026-07-18 00:18:05 +02:00
Rustem Kamalov	e7827cc03d	refactor(browser): native CDP fingerprinting, drop patch.js + localized profiles. Update docker chrome	2026-07-13 06:07:37 +03:00
Rustem Kamalov	cf21bf5150	fix google: detect /sorry/ block pages; scope proxy sessions to credentials. Update default config	2026-07-09 19:30:21 +03:00
Rustem Kamalov	9f68086d47	refactor(engines): unify captcha/block/no-results detection across raw and browser paths; Fix google proxy block diagnostics	2026-07-02 00:47:21 +03:00
Rustem Kamalov	3d5eddfeed	fix(engines): better classify captcha/blocked/no-results pages instead of returning empty results, also enable it to `<engine>/parse` endpoints. Enable `allow_request_proxy_url` by default	2026-07-01 01:38:03 +03:00
Rustem Kamalov	e7af939599	fix: strip features when disabled and normalize feature output. Improve google+bing SERP features extraction	2026-06-29 10:36:46 +03:00
Rustem Kamalov	15bab0295f	refactor: extract shared RankState, unify rod/goquery parse paths	2026-06-27 18:04:25 +03:00
Rustem Kamalov	4a81ab774c	core/http: adopt tls-client for raw+extract fingerprinting. Update docs	2026-06-27 06:10:11 +03:00
Rustem Kamalov	ba27fcf1f4	remove dead code, pin golangci linters	2026-06-25 17:06:01 +03:00
Rustem Kamalov	9cc69da758	feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs - Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json\|text\|markdown\|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution) - Unify engines behind a single engineSpec registry (CLI + serve share it) - Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers - Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks - Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable	2026-06-16 03:51:37 +03:00
				`@@ -0,0 +1 @@`
				<html xmlns="http://test.test" dir="ltr"><head><title>Just a moment...</title></head><body><main><h1>Confirm you’re not a robot</h1><p>Our system has detected unusual traffic from your network. Please solve the challenge below to show you’re not a robot.</p><div><div class="main-wrapper" role="main"><div class="main-content"><h2 id="FbRG5"></h2><p id="jpqrL2"></p><div id="BbLB6" style=""><div><div><input type="hidden" name="cf-turnstile-response" id="id_1"/></div></div></div><div id="TQpKs1" style=""><div>Verification successful. Waiting for www.ecosia.org to respond</div></div><div id="ROlTq4" class="spacer loading-verifying" style=""><div class="lds-ring"><div></div><div></div><div></div><div></div></div></div></div></div></div></main></body></html>
				`@@ -0,0 +1 @@`
				<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"><meta name="viewport" content="initial-scale=1"><title>https://www.google.com/search?ie=UTF-8&oq=weather&pws=0&q=weather&sourceid=chrome</title></head><body style="font-family:arial,sans-serif;background-color:#fff;color:#000;padding:20px;font-size:18px;overscroll-behavior:contain" onload='e=document.getElementById("captcha"),e&&e.focus(),solveSimpleChallenge&&solveSimpleChallenge(0,0)'><div style="max-width:400px"><hr noshade="" size="1" style="color:#ccc;background-color:#ccc"><br><form id="captcha-form" action="index" method="post"><noscript><div style="font-size:13px">In order to continue, please enable javascript on your web browser.</div></noscript><script src="https://www.google.com/recaptcha/enterprise.js" async="" defer=""></script><script>var submitCallback=function(t){document.getElementById("captcha-form").submit()}</script><div id="recaptcha" class="g-recaptcha" data-sitekey="6LfwuyUTAAAAAOAmoS0fdqijC2PbbdH4kjq62Y1b" data-callback="submitCallback" data-s="_4UNXgVVh_xQKhxMBcOJ5Moeza-u52wxA1tuaG87XZwVDXRblsUTZ3nfrOeMsYYu1164fEiA0uhtkqHu7LXSgcBhNzeomoTeq5ztZoh5pdLLdqSzx75mljudOAJV7yfYiM4zWoGy4zg8xWWKZ6Rx5GRjBN_UH-y5crgkZlmPGAZoXiMo4-3lv1oR58YLrbUq8upst9DSbArZ9i31BwQ_fEWv-J17uDF8wzT_kEVPJxltBQQzNrzBs1eEBWS42p6BcIYzLAvjMNNqh_hPHbI7EDxCyhJqFz0"></div><input type="hidden" name="q" value="EgRr25U6GIS7kNIGIjCoB1yvjD_tndmRVr8xZHpYZiL5ETXEMM63dlqgULOz5rXS8U7M5LO3Aj1QSvFyO_syAnJSWgFD"><input type="hidden" name="continue" value="https://www.google.com/search?ie=UTF-8&oq=weather&pws=0&q=weather&sourceid=chrome"></form><hr noshade="" size="1" style="color:#ccc;background-color:#ccc"><div style="font-size:13px"><b>About this page</b><br><br>Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot.<a href="#" onclick='document.getElementById("infoDiv").style.display="block"'>Why did this happen?</a><br><br><div id="infoDiv" style="display:none;background-color:#eee;padding:10px;margin:0 0 15px 0;line-height:1.4em">This page appears when Google automatically detects requests coming from your computer network which appear to be in violation of the<a href="//www.google.com/policies/terms/">Terms of Service</a>. The block will expire shortly after those requests stop. In the meantime, solving the above CAPTCHA will let you continue to use our services.<br><br>This traffic may have been sent by malicious software, a browser plug-in, or a script that sends automated requests. If you share your network connection, ask your administrator for help — a different computer using the same IP address may be responsible.<a href="//support.google.com/websearch/answer/86640">Learn more</a><br><br>Sometimes you may be asked to solve the CAPTCHA if you are using advanced terms that robots are known to use, or sending requests very quickly.</div><br><div style="font-size:13px;line-break:anywhere">IP address: 107.219.149.58<br>Time: 2026-06-30T19:48:20Z<br>URL: https://www.google.com/search?ie=UTF-8&oq=weather&pws=0&q=weather&sourceid=chrome<br></div></div></div></body></html>
				`@@ -0,0 +1 @@`
				`<html><head><meta http-equiv="content-type" content="text/html; charset=UTF-8"><title>weather - Google Search</title></head><body><noscript><iframe src="/httpservice/retry/enablejs?sei=abc123" frameborder="0" height="100%" width="100%"></iframe>Click here if you're not redirected within a few seconds.</noscript><div id="main"></div></body></html>`