Rustem Kamalov
cf21bf5150
fix google: detect /sorry/ block pages; scope proxy sessions to credentials. Update default config
2026-07-09 19:30:21 +03:00
Rustem Kamalov
9cc69da758
feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs
...
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
2026-06-16 03:51:37 +03:00
Rustem Kamalov
04db5347a8
fix(core): harden extract and stabilize request protection
...
- breaker: stop counting client cancellations/deadlines (incl. bare
rate-limiter wait errors) and circuit-open as engine failures
- rate limiting: cache limiters in SearchEngineOptions and rawEngine so
pacing applies on raw/library paths; pool wrapper delegates
- /extract SSRF guard: public-IP-only policy with dial-time IP pinning,
redirect re-validation, rendered-mode preflight, http/https allow-list,
ErrTargetNotAllowed -> HTTP 400, extract.allow_private_networks
escape hatch (default off)
- browser: no rod Must* on the request path; CLI parses block_resources
without panicking
2026-06-12 18:09:40 +03:00
Rustem Kamalov
e8fb0c24fa
feat(proxy): support X-Proxy-URL via per-process browser pool
2026-04-28 03:46:08 +03:00
Rustem Kamalov
e4a84d8914
feat(browser): make configurable resource/tracker blocking
2026-04-24 04:02:23 +03:00
Rustem Kamalov
08edec5779
fix(captcha): exit non-zero on misconfig, tighten solver gate tests
2026-04-20 20:12:19 +03:00
Rustem Kamalov
3daa48e6a3
Proxy policy wiring and fail-closed execution
...
- replace proxy wiring with global and per-engine tag policies
- split stats endpoints and lock fail-closed proxy behavior with tests
2026-04-01 00:54:21 +03:00
