diff --git a/README.md b/README.md index 9c2083e..3083aec 100644 --- a/README.md +++ b/README.md @@ -6,18 +6,18 @@ [![Go Reference](https://pkg.go.dev/badge/github/karust/openserp?style=for-the-badge)](https://pkg.go.dev/github.com/karust/openserp) [![release](https://img.shields.io/github/release/karust/openserp)](https://github.com/karust/openserp/releases) - + -**OpenSERP** provides free API and CLI access to multiple search engines including **[Google, Yandex, Baidu, Bing, DuckDuckGo]**. Get comprehensive search results without expensive API subscriptions! +**OpenSERP** provides free API and CLI access to multiple search engines including **Google, Yandex, Baidu, Bing, and DuckDuckGo**. Get comprehensive search results without expensive API subscriptions! ## Features -- 🔍 **Multi-Engine Support**: Google, Yandex, Baidu, Bing, DuckDuckGo... -- 🌐 **Megasearch**: Aggregate results from multiple engines simultaneously -- 🖼 **Images**: Image search is also available! -- 🎯 **Advanced Filtering**: Language, date range, file type, site-specific searches -- 🌍 **Proxy Support**: HTTP/SOCKS5 proxy support -- 🐳 **Docker Ready**: Easy deployment with Docker +- 🔍 **Multi-engine** - search with dedicated endpoints for each engine +- 🌐 **Megasearch** - cross-engine aggregation with deduplication +- 🖼 **Images** - image search is also available +- 🎯 **Advanced filters** - language, date range, file type, and site queries +- 🌍 **Configurable** - proxy, cache, and resilient mode support +- 🐳 **Docker-ready** - local and container deployment ## Quick Start⚡️ @@ -31,36 +31,36 @@ docker run -p 127.0.0.1:7000:7000 -it karust/openserp serve -a 0.0.0.0 -p 7000 docker compose up --build ``` -### From Source +### From source ```bash -# Clone and build git clone https://github.com/karust/openserp.git cd openserp go build -o openserp . - -# Run the server ./openserp serve ``` -## 🌐 Megasearch & Megaimage - Search Everything at Once! +## 🌐 Megasearch & Megaimage -**Megasearch** aggregates results from multiple engines simultaneously with automatic deduplication. **Megaimage** does the same for image searches! - -### Megasearch (Web Results) +Search all engines at once: ```bash -# Search ALL engines at once -curl "http://localhost:7000/mega/search?text=golang&limit=10" - -# Pick specific engines -curl "http://localhost:7000/mega/search?text=golang&engines=duckduckgo,bing&limit=15" - -# Advanced filtering -curl "http://localhost:7000/mega/search?text=Donald+Trump&engines=duckduckgo,bing&limit=20&date=20251005..20251005&lang=EN" +curl "http://127.0.0.1:7000/mega/search?text=golang&limit=10" ``` -- API response example: +Search only selected engines: + +```bash +curl "http://127.0.0.1:7000/mega/search?text=golang&engines=duckduckgo,bing&limit=15" +``` + +Advanced filtering: + +```bash +curl "http://127.0.0.1:7000/mega/search?text=Donald+Trump&engines=duckduckgo,bing&limit=20&date=20251005..20251005&lang=EN" +``` + +API response example: ```json [ @@ -68,48 +68,46 @@ curl "http://localhost:7000/mega/search?text=Donald+Trump&engines=duckduckgo,bin "rank": 1, "url": "https://en.wikipedia.org/wiki/Golden_Retriever", "title": "Golden Retriever - Wikipedia", - "description": "The Golden Retriever is a Scottish breed of retriever dog of medium size. It is characterised by a gentle and affectionate nature and a striking golden coat. It is a working dog, and registration is subject to successful completion of a working trial. [2] It is commonly kept as a companion dog and is among the most frequently registered breeds in several Western countries; some may compete in ...", + "description": "The Golden Retriever is a Scottish breed of retriever dog of medium size. It is characterised by a gentle and affectionate nature and a striking golden coat.", "ad": false, "engine": "duckduckgo" }, { "rank": 2, - "url": "https://www.bing.com/ck/a?!&&p=6f15ac4589858d0a104cd6f55cc8e91e8d8d6da91f905b626921f67f2323a467JmltdHM9MTc1OTE5MDQwMA&ptn=3&ver=2&hsh=4&fclid=2357c2f4-6131-68de-359f-d48c607c691d&u=a1aHR0cHM6Ly93d3cuZ29sZGVucmV0cmlldmVyZm9ydW0uY29tL3RocmVhZHMvdW5kZXJzdGFuZGluZy13aHktZ29sZGVuLXJldHJpZXZlciVFMiU4MCU5OXMtbGlmZXNwYW4taGFsdmVkLWluLXRoZS1sYXN0LTM1LXllYXJzLjM1NzMyMi8&ntb=1", - "title": "Golden Retriever Dog Forums\nhttps://www.goldenretrieverforum.com › threads › understanding-why-g...", - "description": "Oct 20, 2024 · Back in the 1970s, Golden Retrievers routinely lived until 16 and 17 years old, they are now living until 9 or 10 years old. Golden Retrievers seem to be dying mostly of bone ...", + "url": "https://www.bing.com/ck/a?!&&p=6f15ac4589858d0a104cd6f55cc8", + "title": "Golden Retriever Dog Forums", + "description": "Oct 20, 2024 · Back in the 1970s, Golden Retrievers routinely lived until 16 and 17 years old, they are now...", "ad": false, "engine": "bing" }, { "rank": 3, - "url": "http://www.baidu.com/link?url=2544q3ugc68j0scVxdpWCSX-gl2AmuCy1l7uRR3loIfS1hmJWMiJKW4MDGWoZrLE7X-ybu1L7T8PspoL7iy_dK", - "title": "golden retrievers是什么意思_golden retrievers怎么读_解释_用法...", - "description": "\n\n2025年9月21日golden retrievers 读音:美英 golden retrievers基本解释 金毛猎犬 分词解释 golden金(黄)色的 retrievers寻猎物犬( retriever的名词复数 ) 词组短语 golden retrieversfor sale出售金毛寻回犬 golden retrieversnear me我附近的金毛寻回犬 golden retrieverspuppies金毛寻回犬幼犬...\ndanci.gei6.com/golden...retrievers...", + "url": "http://www.baidu.com/link?url==2544q3ugc68j0scVxdpWCSX-gl2AmuCy1l7uRR3loIfS1", + "title": "golden retrievers是什么意思", + "description": "2025年9月21日golden retrievers 读音:美英 golden retrievers基本解释 金毛猎犬 分词解释 golden金(黄)色的...", "ad": false, "engine": "baidu" } ] ``` -### Megaimage (Image Results) +Image search: ```bash -# Search images across ALL engines -curl "http://localhost:7000/mega/image?text=golang logo&limit=20" +curl "http://127.0.0.1:7000/mega/image?text=golang logo&limit=20" ``` -### Available Engines +List available engines: ```bash -# Check which engines are available -curl "http://localhost:7000/mega/engines" +curl "http://127.0.0.1:7000/mega/engines" ``` **Available engines:** `google`, `yandex`, `baidu`, `bing`, `duckduckgo` ## 🔍 Individual Engine APIs -### Search Parameters +Common query parameters: | Parameter | Description | Example | | --------- | -------------------- | --------------------------------- | @@ -120,85 +118,62 @@ curl "http://localhost:7000/mega/engines" | `site` | Site-specific search | `github.com`, `stackoverflow.com` | | `limit` | Number of results | `10`, `25`, `50` | -### Engine-Specific Parameters +Engine-specific parameters: -| Parameter | Supported engines | Notes | -| --------- | ----------------------------------- | ---------------------------------------------------------------------- | -| `start` | `google`, `bing`, `yandex`, `baidu` | Web search pagination offset. | -| `filter` | `google` | Duplicate filter (`true` => hide similar, `false` => include similar). | -| `answers` | `google` | Include Google answer boxes in output (negative ranks). | +| Parameter | Supported engines | Notes | +| --------- | ----------------------------------- | ------------------------------------------------------------------ | +| `start` | `google`, `bing`, `yandex`, `baidu` | Web search pagination offset. | +| `filter` | `google` | Duplicate filter (`true` hides similar, `false` includes similar). | +| `answers` | `google` | Include Google answer boxes in output with negative ranks. | -### Individual Engine Examples +Examples: ```bash -# DuckDuckGo search -curl "http://localhost:7000/duck/search?text=golang&limit=7" - -# Google search -curl "http://localhost:7000/google/search?text=golang&lang=EN&limit=10" - -# Google search (pagination + include similar/hidden results) -curl "http://localhost:7000/google/search?text=golang&lang=EN&limit=10&start=10&filter=false" - -# Bing search (offset 20 => around page 3 for 10 results/page) -curl "http://localhost:7000/bing/search?text=golang&limit=10&start=20" - -# Yandex search (offset 10 => second page) -curl "http://localhost:7000/yandex/search?text=golang&limit=10&start=10" +curl "http://127.0.0.1:7000/duck/search?text=golang&limit=7" +curl "http://127.0.0.1:7000/google/search?text=golang&lang=EN&limit=10" +curl "http://127.0.0.1:7000/bing/search?text=golang&limit=10&start=20" +curl "http://127.0.0.1:7000/yandex/search?text=golang&limit=10&start=10" +curl "http://127.0.0.1:7000/bing/image?text=golang&limit=20" ``` -### Image Search +## 🌍 Proxy Support + +OpenSERP supports HTTP and SOCKS5 proxies. + +Simple global proxy: ```bash -# Bing Images -curl "http://localhost:7000/bing/image?text=golang&limit=20" - -# Baidu Images -curl "http://localhost:7000/baidu/image?text=golang&limit=15" -``` - -## 🌐 Proxy Support - -OpenSERP supports HTTP and SOCKS5 proxies with authentication: - -```bash -# SOCKS5 proxy ./openserp serve --proxy socks5://127.0.0.1:1080 - -# HTTP proxy with authentication ./openserp search bing "query" --proxy http://user:pass@127.0.0.1:8080 ``` -## Health Check +Advanced proxy configuration is available in [config.yaml](./config.yaml). +You can enable tagged proxy pools and per-request override via `X-Use-Proxy: ` or `X-Use-Proxy: direct`. + +## Health & Stats ```bash curl -i "http://127.0.0.1:7000/health" +curl "http://127.0.0.1:7000/stats" +curl "http://127.0.0.1:7000/stats/cache" +curl "http://127.0.0.1:7000/stats/proxy" +curl "http://127.0.0.1:7000/stats/cb" ``` -Response includes: - -- `status`: `healthy`, `degraded`, or `unhealthy` -- `uptime` -- per-engine readiness in `engines` -- basic runtime stats in `system` - -HTTP status behavior: - -- `200` for `healthy` and `degraded` -- `503` for `unhealthy` +Useful response headers in server mode: `X-Cache`, `X-Fallback-Engine`,`X-Proxy-Mode`, `X-Proxy-Tag`, `X-Proxy-Used` ## License -This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details. +This project is licensed under the MIT License. See [LICENSE](LICENSE). ## 🤝 Contributing -Contributions are welcome! Please feel free to submit a Pull Request. +Contributions are welcome. Please feel free to submit a pull request. ## 👾 Issues & Support -If you encounter any issues or have questions: +If you encounter issues or have questions: - Open an issue on GitHub -- Check existing issues for solutions -- Review the documentation above +- Check existing issues for similar reports +- Review the documentation and example config diff --git a/config.yaml b/config.yaml index 95e2edd..88d43fa 100644 --- a/config.yaml +++ b/config.yaml @@ -45,7 +45,7 @@ cors: enabled: true allow_origins: "*" allow_methods: "GET, POST, OPTIONS" - allow_headers: "Origin, Content-Type, Accept, Authorization" + allow_headers: "Origin, Content-Type, Accept, Authorization, X-Use-Proxy" max_age: 86400 # 2captcha: