16 Commits

Author SHA1 Message Date
Rustem Kamalov
9f68086d47 refactor(engines): unify captcha/block/no-results detection across raw and browser paths; Fix google proxy block diagnostics 2026-07-02 00:47:21 +03:00
Rustem Kamalov
9cc69da758 feat(cli): clean search, add extract/format flags; harden engines & proxy rotation; fix bugs; update docs
- Add structured `search [engine] [query]` CLI: --limit/--lang/--region/--site/--file, --format (json|text|markdown|ndjson), --extract N, --search-timeout; Envelope and route logs to stderr with a --quiet default (fixes stdout pollution)
- Unify engines behind a single engineSpec registry (CLI + serve share it)
- Unify the extract knob to bool-or-int `extract=N` (drop extract_top); CLI and HTTP share core batch extraction, raw/rendered fetch, and clamp helpers
- Engines: Ecosia CF captcha detection (raw + browser), Yandex progressive-result wait, Google PAA poll + Has() existence probes, Bing title/desc attribute fallbacks
- Proxy: rotate challenged proxies out of the tag pool for one retry (X-Proxy-Attempts); browser health-ping skip window; opt-in WaitStable
2026-06-16 03:51:37 +03:00
Rustem Kamalov
04db5347a8 fix(core): harden extract and stabilize request protection
- breaker: stop counting client cancellations/deadlines (incl. bare
  rate-limiter wait errors) and circuit-open as engine failures
- rate limiting: cache limiters in SearchEngineOptions and rawEngine so
  pacing applies on raw/library paths; pool wrapper delegates
- /extract SSRF guard: public-IP-only policy with dial-time IP pinning,
  redirect re-validation, rendered-mode preflight, http/https allow-list,
  ErrTargetNotAllowed -> HTTP 400, extract.allow_private_networks
  escape hatch (default off)
- browser: no rod Must* on the request path; CLI parses block_resources
  without panicking
2026-06-12 18:09:40 +03:00
Rustem Kamalov
d20a7c53cd fix: centralize engine panic recovery in resilient layer
A rod panic during SearchImage propagated uncaught into fasthttp and killed the whole process: per-engine recover blocks only covered Search (5 of 6 engines had no recovery on SearchImage), and there was no Fiber
recover middleware.

Regression test: a panicking SearchImage returns 502 engine_internal and the server keeps serving.
2026-06-12 04:24:06 +03:00
Rustem Kamalov
00630a5d31 refactor: centralize engine prelude, page cleanup, and rate limiter 2026-05-11 11:35:09 +03:00
Rustem Kamalov
5d48690eb3 feat(mega): add mode fast/any/balanced with CB latency-based fast path 2026-05-02 16:30:56 +03:00
Rustem Kamalov
f9890ae10f feat: add network bytes telemetry, mega timeout, and per-engine error details 2026-04-29 02:45:16 +03:00
Rustem Kamalov
e8fb0c24fa feat(proxy): support X-Proxy-URL via per-process browser pool 2026-04-28 03:46:08 +03:00
Rustem Kamalov
16d3a844ad feat: v1 response envelope, normalized Result struct, clusters, and enrichment
- All search endpoints now return a JSON envelope with query echo,
  meta (request_id, took_ms, engines_responded, engines_failed, version),
  results array, and pagination fields
- SearchResult renamed to RawResult internally; new Result struct adds
  id, type, display_url, domain, favicon, is_ad, position, engine_meta,
  domain_info, classification
- Result IDs are stable SHA-256 hashes of engine+normalized_url
- Bing redirect URLs (bing.com/ck/a) are unwrapped before hashing
- ResultType enum defined: organic, ad, answer_box, featured_snippet, etc.
- /mega/search response includes clusters array grouping same-URL results
  across engines with score, best_rank, and occurrences
- engines_failed field surfaces partial failures in megasearch
- Lightweight domain enrichment: TLD/SLD, is_gov/edu/mil/news/forum/
  marketplace/social flags, content_type and source_hint classification
- SearchAllParallel/SearchAllImageParallel now return (results, responded, failed)
- Image endpoints return ImageEnvelope with dedicated ImageResult shape
2026-04-26 02:57:59 +03:00
Rustem Kamalov
0ccb13c36e fix(proxy): isolate proxy health from captcha/parser errors, replace re-enable thrash with quarantine 2026-04-24 04:33:41 +03:00
Rustem Kamalov
0d4e2da960 Improve logging: add structured logs, enrich logs with useful info. 2026-04-22 03:39:30 +03:00
Rustem Kamalov
e89b1a3a50 feat: propagate request context across search, retry, limiter, and browser navigation 2026-04-20 05:53:13 +03:00
Rustem Kamalov
3c07c7789a Runtime proxy override 2026-04-01 17:38:13 +03:00
Rustem Kamalov
3daa48e6a3 Proxy policy wiring and fail-closed execution
- replace proxy wiring with global and per-engine tag policies
- split stats endpoints and lock fail-closed proxy behavior with tests
2026-04-01 00:54:21 +03:00
Rustem Kamalov
8bec5578c0 Implement proxy support with configuration, pooling, and integration tests
- Added ProxyConfig and ProxyPool structures to manage proxy settings and rotation.
- Implemented normalization functions for proxy URLs and configurations.
- Created a new ProxyPool that supports failure tracking and round-robin selection.
- Integrated proxy handling into the ResilientSearcher for search queries.
- Added integration tests for various proxy scenarios including SOCKS5 and HTTP proxies.
- Updated server and resilience stats to include proxy information.
- Refactored search functions to utilize the new proxy client.
- Removed direct proxy handling from search_raw.go and yandex/search_raw.go, using the new core proxy client instead.
- Added unit tests for proxy configuration and pool behavior.
2026-03-30 22:46:03 +03:00
Rustem Kamalov
f219c78d84 Add retry/circuit breaker, configurable fallback and CORS.
Based in part on work from PR #21 by @Sai-Prashanth123, adapted and integrated with project-specific fixes.
2026-03-25 23:31:41 +03:00