27 Commits

Author SHA1 Message Date
Rustem Kamalov
b4ef23a913 Add megasearch API response example 2025-10-01 04:56:40 +03:00
Rustem Kamalov
0a347edc77 New megasearch and megaimage functionality. Update readme 2025-10-01 04:34:41 +03:00
Rustem Kamalov
de0a6a80d4 DuckDuck fixes. Non-stealth antibot fixes 2025-10-01 04:30:58 +03:00
Rustem Kamalov
74fc15a6e2 Add DuckDuckGo. Add custom logger. Fix arg priority and other issues 2025-09-30 05:27:54 +03:00
Rustem Kamalov
f8815b4560 Add Bing search+images 2025-09-19 02:11:55 +03:00
Rustem Kamalov
49e9837878 Fix build issues 2025-08-29 03:44:38 +03:00
Rustem Kamalov
fb8e8cd6fe Merge pull request #12 from PACHAKUTlQ/main
Bug fixes and TLS fingerprint spoofing
2025-08-29 03:35:13 +03:00
Pachakutiq
f7654a52f0 chore: Update dependencies 2025-08-28 17:20:28 +08:00
Pachakutiq
8fdefb6ed5 fix(google)!: Fix missing datas 2025-08-28 16:44:22 +08:00
Pachakutiq
81fc057c83 feat(raw search)!: Use utls to mimic chrome TLS clienthello fingerprint 2025-08-28 16:43:39 +08:00
Pachakutiq
c35c5f134d chore: Add utls as dependency 2025-08-28 16:42:18 +08:00
Pachakutiq
83385711fe chore: Update dependency versions 2025-08-28 16:41:58 +08:00
Pachakutiq
c980ca4dc4 fix!: Allow insecure TLS connections 2025-08-28 16:35:53 +08:00
Rustem Kamalov
d8c3700252 Stealth browser as option. Update default config 2025-08-18 22:49:45 +03:00
Rustem Kamalov
7f0f32ac4e Handle failed navigation 2025-07-03 02:29:12 +03:00
Rustem Kamalov
ba94c69dd0 Merge pull request #10 from PACHAKUTlQ/main
Support http and socks5 proxy and fix failure due to updates of DOM on Google SERP
2025-07-03 02:09:18 +03:00
Pachakutiq
c387268d20 chore: Exclude .aider* in .gitignore 2025-05-27 15:05:53 +08:00
Pachakutiq
2f8ee2164d fix: Fix serve mode not supporting raw mode 2025-04-10 18:07:57 +08:00
Pachakutiq
6691ebaadd fix: Remove duplicated results for all search engines and both modes 2025-04-10 16:59:31 +08:00
Pachakutiq
8099dae3ce fix: Remove duplicated results in google/search.go 2025-04-10 15:35:54 +08:00
Pachakutiq
2f77dd6fe5 doc: Add proxy guide in README.md 2025-04-09 02:55:59 +08:00
Pachakutiq
90f27c2974 feat: Support short flag -x for proxy 2025-04-09 02:54:42 +08:00
Pachakutiq
ec03439b2f style: Format README.md using prettier 2025-04-09 02:11:22 +08:00
Pachakutiq
5ad89b2ea8 feat: Support proxy for serve 2025-04-09 02:08:01 +08:00
Pachakutiq
b2ffcf8477 feat: Support http and socks5 proxy (support authentication) for chrome driver mode 2025-04-08 23:51:44 +08:00
Pachakutiq
21f12c5996 fix: Google updated DOM of search result page 2025-04-08 18:48:07 +08:00
Pachakutiq
84e2d46c4b feat: Support http and socks5 proxy with auth for raw search 2025-04-08 16:06:41 +08:00
30 changed files with 2611 additions and 812 deletions

1
.gitignore vendored
View File

@@ -24,3 +24,4 @@ logrus.txt
logs.txt
.release
core/test/
.aider*

218
README.md
View File

@@ -1,91 +1,173 @@
# OpenSERP (Search Engine Results Page)
![OpenSERP](/logo.png)
![OpenSERP](/logo.svg)
[![Go Report Card](https://goreportcard.com/badge/github.com/karust/openserp)](https://goreportcard.com/report/github.com/karust/openserp)
[![Go Reference](https://pkg.go.dev/badge/github.com/karust/openserp.svg)](https://pkg.go.dev/github.com/karust/openserp)
[![release](https://img.shields.io/github/release-pre/karust/openserp.svg)](https://github.com/karust/openserp/releases)
<!-- ![Docker Image Size (tag)](https://img.shields.io/docker/image-size/karust/openserp/latest) -->
API access for search engines results if available isn't free.
Using OpenSERP, you can get search results from **Google**, **Yandex**, **Baidu** via API or CLI!
**OpenSERP** provides free API access to multiple search engines including **[Google, Yandex, Baidu, Bing, DuckDuckGo]**. Get comprehensive search results without expensive API subscriptions!
See [Docker](#docker) and [CLI](#cli) usage examples below ([search](#search), [images](#images)).
## Features
- 🔍 **Multi-Engine Support**: Google, Yandex, Baidu, Bing, DuckDuckGo...
- 🌐 **Megasearch**: Aggregate results from multiple engines simultaneously
- 🎯 **Advanced Filtering**: Language, date range, file type, site-specific searches
- 🌍 **Proxy Support**: HTTP/SOCKS5 proxy support
- 🐳 **Docker Ready**: Easy deployment with Docker
## Quick Start⚡
### Docker (Recommended)
## Docker usage <a name="docker"></a> 🐳
* Run API server:
```bash
# Use prebuilt image
# Run the API server via prebuilt image
docker run -p 127.0.0.1:7000:7000 -it karust/openserp serve -a 0.0.0.0 -p 7000
# Or build one and run using docker-compose.yaml
docker-compose up --build
# Or use docker-compose
docker compose up --build
```
### Request parameters
| Param | Description |
|-------|--------------------------------------------------------------|
| text | Text to search |
| lang | Search pages in selected language (`EN`, `DE`, `RU`...) |
| date | Date in `YYYYMMDD..YYYYMMDD` format (e.g. 20181010..20231010) |
| file | File extension to search (e.g. `PDF`, `DOC`) |
| site | Search within a specific website |
| limit | Limit the number of results
| answers | Include google answers as negative rank indexes (e.g. `true`, `false`)
### From Source
### **Search**
### *Example request*
Get 20 **Google** results for `hello world`, only in English:
```
GET http:/127.0.0.1:7000/google/search?lang=EN&limit=20&text=hello world
```
You can replace `google` to `yandex` or `baidu` in query to change search engine.
|
### *Example response*
```JSON
[
{
"rank": 1,
"url": "https://en.wikipedia.org/wiki/%22Hello,_World!%22_program",
"title": "\"Hello, World!\" program",
"description": "A \"Hello, World!\" program is generally a computer program that ignores any input, and outputs or displays a message similar to \"Hello, World!\".",
"ad": false
},
]
```
### **Images** **[WIP]**
### *Example request*
Get 100 **Google** results for `golden puppy`:
```
GET http://127.0.0.1:7000/google/image?text=golden puppy&limit=100
```
## CLI <a name="cli"></a> ⌨️
* Use `-h` flag to see commands.
* You can use `serve` command to serve API:
```bash
openserp serve
# Clone and build
git clone https://github.com/karust/openserp.git
cd openserp
go build -o openserp .
# Run the server
./openserp serve
```
* Or print results in CLI using `search` command:
## 🌐 Megasearch & Megaimage - Search Everything at Once!
**Megasearch** aggregates results from multiple engines simultaneously with automatic deduplication. **Megaimage** does the same for image searches!
### Megasearch (Web Results)
```bash
openserp search google "how to get banned from google fast" # Change `google` to `yandex` or `baidu`
# Search ALL engines at once
curl "http://localhost:7000/mega/search?text=golang&limit=10"
# Pick specific engines
curl "http://localhost:7000/mega/search?text=golang&engines=google,bing&limit=5"
# Advanced filtering
curl "http://localhost:7000/mega/search?text=golang&lang=EN&site=github.com&limit=8"
```
As a result you should get JSON output containting search results:
- API response example:
```json
[
{
"rank": 1,
"url": "https://www.cyberoptik.net/blog/6-sure-fire-ways-to-get-banned-from-google/",
"title": "11 Sure-Fire Ways to Get Banned From Google | CyberOptik",
"description": "How To Get Banned From Google · 1. Cloaking: The Art of Deception · 2. Plagiarism: Because Originality is Overrated · 3. Keyword Stuffing: More is Always Better · 4 ...",
"ad": false
},
{
"rank": 1,
"url": "https://en.wikipedia.org/wiki/Golden_Retriever",
"title": "Golden Retriever - Wikipedia",
"description": "The Golden Retriever is a Scottish breed of retriever dog of medium size. It is characterised by a gentle and affectionate nature and a striking golden coat. It is a working dog, and registration is subject to successful completion of a working trial. [2] It is commonly kept as a companion dog and is among the most frequently registered breeds in several Western countries; some may compete in ...",
"ad": false,
"engine": "duckduckgo"
},
{
"rank": 2,
"url": "https://www.bing.com/ck/a?!&&p=6f15ac4589858d0a104cd6f55cc8e91e8d8d6da91f905b626921f67f2323a467JmltdHM9MTc1OTE5MDQwMA&ptn=3&ver=2&hsh=4&fclid=2357c2f4-6131-68de-359f-d48c607c691d&u=a1aHR0cHM6Ly93d3cuZ29sZGVucmV0cmlldmVyZm9ydW0uY29tL3RocmVhZHMvdW5kZXJzdGFuZGluZy13aHktZ29sZGVuLXJldHJpZXZlciVFMiU4MCU5OXMtbGlmZXNwYW4taGFsdmVkLWluLXRoZS1sYXN0LTM1LXllYXJzLjM1NzMyMi8&ntb=1",
"title": "Golden Retriever Dog Forums\nhttps://www.goldenretrieverforum.com threads understanding-why-g…",
"description": "Oct 20, 2024 · Back in the 1970s, Golden Retrievers routinely lived until 16 and 17 years old, they are now living until 9 or 10 years old. Golden Retrievers seem to be dying mostly of bone …",
"ad": false,
"engine": "bing"
},
{
"rank": 3,
"url": "http://www.baidu.com/link?url=2544q3ugc68j0scVxdpWCSX-gl2AmuCy1l7uRR3loIfS1hmJWMiJKW4MDGWoZrLE7X-ybu1L7T8PspoL7iy_dK",
"title": "golden retrievers是什么意思_golden retrievers怎么读_解释_用法...",
"description": "\n\n2025年9月21日golden retrievers 读音:美英 golden retrievers基本解释 金毛猎犬 分词解释 golden金(黄)色的 retrievers寻猎物犬( retriever的名词复数 ) 词组短语 golden retrieversfor sale出售金毛寻回犬 golden retrieversnear me我附近的金毛寻回犬 golden retrieverspuppies金毛寻回犬幼犬...\ndanci.gei6.com/golden...retrievers...",
"ad": false,
"engine": "baidu"
}
]
```
```
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details
### Megaimage (Image Results)
## Bugs + Questions 👾
If you have some issues/bugs/questions, feel free to open an issue.
```bash
# Search images across ALL engines
curl "http://localhost:7000/mega/image?text=golang logo&limit=20"
# Pick specific engines for images
curl "http://localhost:7000/mega/image?text=golang&engines=google,bing&limit=15"
# Language-specific image search
curl "http://localhost:7000/mega/image?text=golang&lang=EN&limit=10"
```
### Available Engines
```bash
# Check which engines are available
curl "http://localhost:7000/mega/engines"
```
**Available engines:** `google`, `yandex`, `baidu`, `bing`, `duckduckgo`
## 🔍 Individual Engine APIs
### Search Parameters
| Parameter | Description | Example |
| --------- | -------------------- | --------------------------------- |
| `text` | Search query | `golang programming` |
| `lang` | Language code | `EN`, `DE`, `RU`, `ES` |
| `date` | Date range | `20230101..20231231` |
| `file` | File extension | `PDF`, `DOC`, `XLS` |
| `site` | Site-specific search | `github.com`, `stackoverflow.com` |
| `limit` | Number of results | `10`, `25`, `50` |
| `answers` | Include Q&A results | `true`, `false` |
### Individual Engine Examples
```bash
# DuckDuckGo search
curl "http://localhost:7000/duck/search?text=golang&limit=7"
# Google search
curl "http://localhost:7000/google/search?text=golang&lang=EN&limit=10"
```
### Image Search
```bash
# Bing Images
curl "http://localhost:7000/bing/image?text=golang&limit=20"
# Baidu Images
curl "http://localhost:7000/baidu/image?text=golang&limit=15"
```
## 🌐 Proxy Support
OpenSERP supports HTTP and SOCKS5 proxies with authentication:
```bash
# SOCKS5 proxy
./openserp serve --proxy socks5://127.0.0.1:1080
# HTTP proxy with authentication
./openserp search bing "query" --proxy http://user:pass@127.0.0.1:8080
```
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
## 🤝 Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## 👾 Issues & Support
If you encounter any issues or have questions:
- Open an issue on GitHub
- Check existing issues for solutions
- Review the documentation above

View File

@@ -8,7 +8,6 @@ import (
"github.com/go-rod/rod"
"github.com/karust/openserp/core"
"github.com/sirupsen/logrus"
"golang.org/x/time/rate"
)
@@ -35,12 +34,14 @@ type imageDataJson struct {
type Baidu struct {
core.Browser
core.SearchEngineOptions
logger *core.EngineLogger
}
func New(browser core.Browser, opts core.SearchEngineOptions) *Baidu {
baid := Baidu{Browser: browser}
opts.Init()
baid.SearchEngineOptions = opts
baid.logger = core.NewEngineLogger("Baidu")
return &baid
}
@@ -64,7 +65,7 @@ func (baid *Baidu) isTimeout(page *rod.Page) bool {
}
func (baid *Baidu) Search(query core.Query) ([]core.SearchResult, error) {
logrus.Tracef("Start Baidu search, query: %+v", query)
baid.logger.Debug("Starting search, query: %+v", query)
searchResults := []core.SearchResult{}
@@ -74,12 +75,15 @@ func (baid *Baidu) Search(query core.Query) ([]core.SearchResult, error) {
return nil, err
}
page := baid.Navigate(url)
page, err := baid.Navigate(url)
if err != nil {
return nil, err
}
results, err := page.Timeout(baid.Timeout).Search("div.c-container.new-pmd")
if err != nil {
defer page.Close()
logrus.Errorf("Cannot parse search results: %s", err)
baid.logger.Error("Cannot parse search results: %s", err)
return nil, core.ErrSearchTimeout
}
@@ -88,10 +92,10 @@ func (baid *Baidu) Search(query core.Query) ([]core.SearchResult, error) {
defer page.Close()
if baid.isCaptcha(page) {
logrus.Errorf("Baidu captcha occurred during: %s", url)
baid.logger.Error("Captcha detected: %s", url)
return nil, core.ErrCaptcha
} else if baid.isTimeout(page) {
logrus.Errorf("Baidu timeout occurred during: %s", url)
baid.logger.Error("Timeout occurred: %s", url)
return nil, core.ErrCaptcha
}
return nil, nil
@@ -110,13 +114,13 @@ func (baid *Baidu) Search(query core.Query) ([]core.SearchResult, error) {
}
linkText, err := link.Property("href")
if err != nil {
logrus.Error("No `href` tag found")
baid.logger.Error("Missing href tag")
}
// Get title
title, err := link.Text()
if err != nil {
logrus.Error("Cannot extract text from title")
baid.logger.Error("Failed to extract title")
title = "No title"
}
@@ -131,18 +135,18 @@ func (baid *Baidu) Search(query core.Query) ([]core.SearchResult, error) {
searchResults = append(searchResults, gR)
}
if !baid.LeavePageOpen {
if !baid.Browser.LeavePageOpen {
err = page.Close()
if err != nil {
logrus.Error(err)
baid.logger.Error("Page close error: %v", err)
}
}
return searchResults, nil
return core.DeduplicateResults(searchResults), nil
}
func (baid *Baidu) SearchImage(query core.Query) ([]core.SearchResult, error) {
logrus.Tracef("Start Baidu Image search, query: %+v", query)
baid.logger.Debug("Starting image search, query: %+v", query)
searchResults := []core.SearchResult{}
searchPage := 0
@@ -154,8 +158,12 @@ func (baid *Baidu) SearchImage(query core.Query) ([]core.SearchResult, error) {
}
// Get anti-crawler cookies first, then reload page
page := baid.Navigate(url)
if !baid.LeavePageOpen {
page, err := baid.Navigate(url)
if err != nil {
return nil, err
}
if !baid.Browser.LeavePageOpen {
defer page.Close()
}
page.Reload()
@@ -164,7 +172,7 @@ func (baid *Baidu) SearchImage(query core.Query) ([]core.SearchResult, error) {
result, err := page.Timeout(baid.Timeout).Search("body > pre")
if err != nil {
defer page.Close()
logrus.Errorf("Cannot parse search results: %s", err)
baid.logger.Error("Cannot parse search results: %s", err)
return nil, core.ErrSearchTimeout
}
@@ -173,10 +181,10 @@ func (baid *Baidu) SearchImage(query core.Query) ([]core.SearchResult, error) {
defer page.Close()
if baid.isCaptcha(page) {
logrus.Errorf("Baidu captcha occurred during: %s", url)
baid.logger.Error("Captcha detected: %s", url)
return nil, core.ErrCaptcha
} else if baid.isTimeout(page) {
logrus.Errorf("Baidu timeout occurred during: %s", url)
baid.logger.Error("Timeout occurred: %s", url)
return nil, core.ErrCaptcha
}
return nil, nil
@@ -200,7 +208,7 @@ func (baid *Baidu) SearchImage(query core.Query) ([]core.SearchResult, error) {
err = json.Unmarshal([]byte(fixedJson), &data)
if err != nil {
logrus.Errorf("Cannot unmarshal JSON: %v\nData: %v", err, jsonText)
baid.logger.Error("Failed to unmarshal JSON: %v", err)
return nil, err
}
@@ -226,10 +234,10 @@ func (baid *Baidu) SearchImage(query core.Query) ([]core.SearchResult, error) {
searchPage += 1
if !baid.LeavePageOpen {
if !baid.Browser.LeavePageOpen {
page.Close()
}
}
return searchResults, nil
return core.DeduplicateResults(searchResults), nil
}

View File

@@ -1,18 +1,67 @@
package baidu
import (
"context"
"crypto/tls"
"fmt"
"net"
"net/http"
"net/url"
"strings"
"time"
"github.com/PuerkitoBio/goquery"
"github.com/corpix/uarand"
"github.com/karust/openserp/core"
"github.com/sirupsen/logrus"
utls "github.com/refraction-networking/utls"
)
func baiduRequest(searchURL string) (*http.Response, error) {
baseClient := &http.Client{}
func baiduRequest(searchURL string, query core.Query) (*http.Response, error) {
// Create HTTP transport with proxy
transport := &http.Transport{}
if query.ProxyURL != "" {
proxyUrl, err := url.Parse(query.ProxyURL)
if err != nil {
return nil, err
}
transport.Proxy = http.ProxyURL(proxyUrl)
}
// Set insecure TLS
if query.Insecure {
transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
}
transport.DialTLSContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
dialer := &net.Dialer{}
rawConn, err := dialer.DialContext(ctx, network, addr)
if err != nil {
return nil, err
}
hostname := strings.Split(addr, ":")[0]
config := &utls.Config{
ServerName: hostname,
InsecureSkipVerify: query.Insecure,
}
uconn := utls.UClient(rawConn, config, utls.HelloChrome_Auto)
if err := uconn.Handshake(); err != nil {
rawConn.Close()
return nil, err
}
return uconn, nil
}
baseClient := &http.Client{
Transport: transport,
Timeout: time.Second * 10,
}
req, err := http.NewRequest("GET", searchURL, nil)
if err != nil {
return nil, err
@@ -27,7 +76,7 @@ func baiduRequest(searchURL string) (*http.Response, error) {
}
func baiduResultParser(response *http.Response) ([]core.SearchResult, error) {
doc, err := goquery.NewDocumentFromResponse(response)
doc, err := goquery.NewDocumentFromReader(response.Body)
if err != nil {
return nil, err
}
@@ -79,7 +128,7 @@ func Search(query core.Query) ([]core.SearchResult, error) {
}
logrus.Debugf("Baidu URL built: %s", googleURL)
res, err := baiduRequest(googleURL)
res, err := baiduRequest(googleURL, query)
if err != nil {
return nil, err
}
@@ -91,5 +140,5 @@ func Search(query core.Query) ([]core.SearchResult, error) {
}
logrus.Debugf("Baidu Raw results : %v", results)
return results, nil
return core.DeduplicateResults(results), nil
}

View File

@@ -13,7 +13,7 @@ import (
)
func dateToTimestamp(date string) (int64, error) {
layout := "20060201"
layout := "20060102"
t, err := time.Parse(layout, date)
if err != nil {
return 0, err
@@ -55,12 +55,12 @@ func BuildURL(q core.Query) (string, error) {
if q.LangCode != "" {
//params.Add("rqlang", q.LangCode)
logrus.Warn("Baidu's Language specific search not supported yet")
logrus.Warn("Language search not supported")
}
if q.Filetype != "" {
//params.Add("ft", q.Filetype)
logrus.Warn("Baidu's File search not supported yet")
logrus.Warn("File search not supported")
}
if q.Limit != 0 {

161
bing/bing_test.go Normal file
View File

@@ -0,0 +1,161 @@
package bing
import (
"net/url"
"strings"
"testing"
"time"
"github.com/karust/openserp/core"
)
var browser *core.Browser
func init() {
opts := core.BrowserOpts{IsHeadless: false, IsLeakless: false, UseStealth: true, Timeout: time.Second * 5, LeavePageOpen: true}
browser, _ = core.NewBrowser(opts)
}
func TestSearchBing(t *testing.T) {
bing := New(*browser, core.SearchEngineOptions{})
query := core.Query{Text: "golang programming", Limit: 10}
results, err := bing.Search(query)
if err != nil {
t.Fatalf("Cannot [SearchBing]: %s", err)
}
if len(results) == 0 {
t.Fatalf("[SearchBing] returned empty result")
}
// Check that we have some basic fields populated
firstResult := results[0]
if firstResult.Title == "" {
t.Errorf("First result missing title: %+v", firstResult)
}
if firstResult.URL == "" {
t.Errorf("First result missing URL: %+v", firstResult)
}
if firstResult.Rank == 0 {
t.Errorf("First result missing rank: %+v", firstResult)
}
}
func TestBuildImageURL(t *testing.T) {
tests := []struct {
name string
query core.Query
wantErr bool
wantCont string
}{
{
name: "basic image query",
query: core.Query{Text: "test"},
wantErr: false,
wantCont: "q=test",
},
{
name: "image query with site",
query: core.Query{Text: "cats", Site: "example.com"},
wantErr: false,
wantCont: "q=cats+site%3Aexample.com",
},
{
name: "image query with filetype",
query: core.Query{Text: "dogs", Filetype: "png"},
wantErr: false,
wantCont: "q=dogs+filetype%3Apng",
},
{
name: "empty query",
query: core.Query{Text: ""},
wantErr: true,
wantCont: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := BuildImageURL(tt.query)
if (err != nil) != tt.wantErr {
t.Errorf("BuildImageURL() error = %v, wantErr %v", err, tt.wantErr)
return
}
if !tt.wantErr && got != "" {
if !strings.Contains(got, tt.wantCont) {
t.Errorf("BuildImageURL() = %v, should contain %v", got, tt.wantCont)
}
// Test that URL is valid
_, err := url.Parse(got)
if err != nil {
t.Errorf("BuildImageURL() returned invalid URL: %v", err)
}
// Should be a Bing images URL
if !strings.Contains(got, "bing.com/images/search") {
t.Errorf("BuildImageURL() should return Bing images URL, got: %v", got)
}
}
})
}
}
func TestBingImageSearch(t *testing.T) {
bing := New(*browser, core.SearchEngineOptions{RateTime: 5})
query := core.Query{
Text: "golden puppy",
Limit: 25,
Filetype: "jpg",
}
results, err := bing.SearchImage(query)
if err != nil {
t.Fatalf("Cannot search Bing images: %s", err)
}
if len(results) == 0 {
t.Fatalf("Bing image search returned empty result")
}
// Check that we have image results with proper fields
firstResult := results[0]
if firstResult.URL == "" {
t.Errorf("First result missing image URL: %+v", firstResult)
}
if firstResult.Title == "" {
t.Errorf("First result missing title: %+v", firstResult)
}
// Check that we have either image URL or source URL
hasImageURL := firstResult.URL != ""
hasSourceURL := firstResult.URL != ""
if !hasImageURL && !hasSourceURL {
t.Errorf("First result should have either image URL or source URL: %+v", firstResult)
}
// For image results, URL should typically point to an image file
if hasImageURL {
// Check if it looks like an image URL (common extensions)
imageExtensions := []string{".jpg", ".jpeg", ".png", ".gif", ".webp", ".bmp"}
hasImageExtension := false
for _, ext := range imageExtensions {
if strings.Contains(strings.ToLower(firstResult.URL), ext) {
hasImageExtension = true
break
}
}
if !hasImageExtension {
t.Logf("Image URL doesn't have common extension (might be valid): %s", firstResult.URL)
}
}
t.Logf("Found %d image results", len(results))
t.Logf("First result - Title: %s", firstResult.Title)
t.Logf("First result - Image URL: %s", firstResult.URL)
t.Logf("First result - Source URL: %s", firstResult.URL)
}

323
bing/search.go Normal file
View File

@@ -0,0 +1,323 @@
package bing
import (
"encoding/json"
"errors"
"fmt"
"strings"
"time"
"github.com/go-rod/rod"
"github.com/go-rod/rod/lib/proto"
"github.com/karust/openserp/core"
"golang.org/x/time/rate"
)
type Bing struct {
core.Browser
core.SearchEngineOptions
logger *core.EngineLogger
}
func New(browser core.Browser, opts core.SearchEngineOptions) *Bing {
bing := Bing{Browser: browser}
opts.Init()
bing.SearchEngineOptions = opts
bing.logger = core.NewEngineLogger("Bing")
return &bing
}
func (bing *Bing) Name() string {
return "bing"
}
func (bing *Bing) GetRateLimiter() *rate.Limiter {
ratelimit := rate.Every(bing.GetRatelimit())
return rate.NewLimiter(ratelimit, bing.RateBurst)
}
func (bing *Bing) getTotalResults(page *rod.Page) (int, error) {
results, err := page.Timeout(bing.GetSelectorTimeout()).Elements("li.b_algo")
if err != nil {
return 0, errors.New("Cannot find result elements: " + err.Error())
}
return len(results), nil
}
func (bing *Bing) checkCaptcha(page *rod.Page) bool {
if page == nil {
return false
}
if info, err := page.Info(); err == nil {
url := strings.ToLower(info.URL)
if strings.Contains(url, "turing") || strings.Contains(url, "captcha") {
return true
}
}
timeout := bing.GetSelectorTimeout() / 2
if timeout <= 0 {
timeout = time.Second * 2
}
selectors := []string{
"div.captcha",
"div.captcha_header",
}
for _, selector := range selectors {
has, err, _ := page.Timeout(timeout).Has(selector)
if err == nil && has {
bing.logger.Debug("Captcha detected: %s", selector)
return true
}
}
return false
}
func (bing *Bing) acceptCookies(page *rod.Page) {
consentBtn, err := page.Timeout(bing.Timeout / 10).Element("button#bnp_btn_accept")
if err != nil {
return
}
consentBtn.Click(proto.InputMouseButtonLeft, 1)
time.Sleep(time.Millisecond * 500)
}
func (bing *Bing) close(page *rod.Page) {
if !bing.Browser.LeavePageOpen {
if page != nil {
err := page.Close()
if err != nil {
bing.logger.Debug("Page close error: %v", err)
}
}
}
}
func (bing *Bing) Search(query core.Query) ([]core.SearchResult, error) {
bing.logger.Debug("Starting search, query: %+v", query)
searchResults := []core.SearchResult{}
url, err := BuildURL(query)
if err != nil {
return nil, err
}
page, err := bing.Navigate(url)
if err != nil {
return nil, err
}
defer bing.close(page)
page.WaitLoad()
if bing.checkCaptcha(page) {
bing.logger.Error("Captcha detected: %s", url)
return nil, core.ErrCaptcha
}
bing.acceptCookies(page)
page.WaitLoad()
organicElements, err := page.Timeout(bing.Timeout).Elements("li.b_algo")
if err != nil {
bing.logger.Error("Cannot parse organic results: %s", err)
return nil, core.ErrSearchTimeout
}
adElements, err := page.Timeout(bing.Timeout).Elements("li.b_ad")
if err != nil {
bing.logger.Debug("No ads found")
}
totalResults, err := bing.getTotalResults(page)
if err != nil {
bing.logger.Debug("Failed to get total results: %v", err)
}
bing.logger.Info("Found %d results (%d ads)", totalResults, len(adElements))
rank := 0
for _, result := range organicElements {
srchRes := core.SearchResult{}
titleElem, err := result.Element("a")
if err != nil {
bing.logger.Debug("Missing title")
continue
}
srchRes.Title, _ = titleElem.Text()
href, err := titleElem.Property("href")
if err != nil {
bing.logger.Debug("Missing URL")
continue
}
srchRes.URL = href.String()
var desc string
if descElem, err := result.Element("div.b_caption p"); err == nil {
desc, _ = descElem.Text()
} else if descElem, err := result.Element("div.b_caption div"); err == nil {
desc, _ = descElem.Text()
} else if descElem, err := result.Element("p"); err == nil {
desc, _ = descElem.Text()
} else {
fullText, _ := result.Text()
desc = strings.TrimSpace(strings.Replace(fullText, srchRes.Title, "", 1))
}
srchRes.Description = desc
rank++
srchRes.Rank = rank
srchRes.Ad = false
searchResults = append(searchResults, srchRes)
}
for _, adResult := range adElements {
srchRes := core.SearchResult{Ad: true}
titleElem, err := adResult.Element("h2 a")
if err != nil {
bing.logger.Debug("Ad missing title")
continue
}
srchRes.Title, _ = titleElem.Text()
href, err := titleElem.Property("href")
if err != nil {
bing.logger.Debug("Ad missing URL")
continue
}
srchRes.URL = href.String()
if descElem, err := adResult.Element("p"); err == nil {
srchRes.Description, _ = descElem.Text()
}
// Mark ads with negative rank
srchRes.Rank = -1
searchResults = append(searchResults, srchRes)
}
return core.DeduplicateResults(searchResults), nil
}
// BingImageData represents the JSON structure in the m attribute of image elements
type BingImageData struct {
T string `json:"t"` // Title
Desc string `json:"desc"` // Description
IMGURL string `json:"imgurl"` // Original image URL
W int `json:"w"` // Width
H int `json:"h"` // Height
PURL string `json:"purl"` // Page URL
TURL string `json:"turl"` // Thumbnail URL
MURL string `json:"murl"` // Image URL
}
// SearchImage performs Bing image search and returns results
func (bing *Bing) SearchImage(query core.Query) ([]core.SearchResult, error) {
bing.logger.Debug("Starting image search, query: %+v", query)
searchResults := []core.SearchResult{}
// Build Bing image search URL
url, err := BuildImageURL(query)
if err != nil {
return nil, err
}
page, err := bing.Navigate(url)
if err != nil {
return nil, err
}
defer bing.close(page)
page.WaitLoad()
// Check for captcha
if bing.checkCaptcha(page) {
bing.logger.Error("Captcha detected during image search: %s", url)
return nil, core.ErrCaptcha
}
// Accept cookies if present
bing.acceptCookies(page)
// Wait for image results to load
page.WaitLoad()
time.Sleep(time.Second * 2)
// Find all image result containers using CSS selector
imageContainers, err := page.Timeout(bing.Timeout).Elements("div.iuscp, div.isv")
if err != nil {
bing.logger.Error("Cannot parse image results: %s", err)
return nil, core.ErrSearchTimeout
}
if len(imageContainers) == 0 {
return nil, errors.New("no image results found")
}
bing.logger.Info("Found %d image elements", len(imageContainers))
rank := 0
for _, c := range imageContainers {
srchRes := core.SearchResult{}
// Get the <a> element inside the div
linkElem, err := c.Element("a")
if err != nil {
bing.logger.Debug("Missing <a> element")
continue
}
// Extract image metadata from m attribute (contains JSON)
mAttr, err := linkElem.Attribute("m")
if err != nil || mAttr == nil {
bing.logger.Debug("Missing m attribute")
continue
}
// Ensure we have valid JSON data to unmarshal
jsonData := []byte(*mAttr)
if len(jsonData) == 0 {
bing.logger.Debug("Empty JSON data")
continue
}
// Initialize imgData properly
var imgData BingImageData
err = json.Unmarshal(jsonData, &imgData)
if err != nil {
bing.logger.Debug("Failed to parse JSON: %s", err)
continue
}
// Extract information from the parsed data
srchRes.Title = imgData.T
srchRes.URL = imgData.IMGURL
srchRes.Description = imgData.Desc
// Add dimensions to description if available
if imgData.W > 0 && imgData.H > 0 {
srchRes.Description += fmt.Sprintf(" (%dx%d)", imgData.W, imgData.H)
}
// Get the page URL
if imgData.MURL != "" {
srchRes.URL = imgData.MURL
}
rank++
srchRes.Rank = rank
searchResults = append(searchResults, srchRes)
}
return searchResults, nil
}

122
bing/url.go Normal file
View File

@@ -0,0 +1,122 @@
package bing
import (
"errors"
"fmt"
"net/url"
"strconv"
"strings"
"time"
"github.com/karust/openserp/core"
"github.com/sirupsen/logrus"
)
func BuildURL(q core.Query) (string, error) {
base, err := url.Parse("https://www.bing.com")
if err != nil {
return "", err
}
base.Path += "search"
params := url.Values{}
// Set search query text with operators
if q.Text != "" || q.Site != "" || q.Filetype != "" {
text := q.Text
if q.Site != "" {
text += " site:" + q.Site
}
if q.Filetype != "" {
text += " filetype:" + q.Filetype
}
logrus.Tracef("Query text: %s", text)
params.Add("q", text)
}
if len(params.Get("q")) == 0 {
return "", errors.New("empty query built")
}
if q.LangCode != "" {
params.Add("setlang", strings.ToLower(q.LangCode))
}
// Set result offset (pagination) - Bing uses "first" parameter
if q.Limit > 0 {
params.Add("count", strconv.Itoa(q.Limit))
}
// Set search date range - Bing supports date filtering via query text
if q.DateInterval != "" {
intervals := strings.Split(q.DateInterval, "..")
if len(intervals) != 2 {
return "", errors.New("incorrect date interval provided, expected format: YYYYMMDD..YYYYMMDD")
}
// Convert YYYYMMDD to YYYY-MM-DD format for Bing
startDate, err := time.Parse("20060102", intervals[0])
if err != nil {
return "", errors.New("invalid start date format, expected YYYYMMDD")
}
endDate, err := time.Parse("20060102", intervals[1])
if err != nil {
return "", errors.New("invalid end date format, expected YYYYMMDD")
}
// Add date range to the search query text (Bing supports this format)
dateRange := fmt.Sprintf(" after:%s before:%s",
startDate.Format("2006-01-02"),
endDate.Format("2006-01-02"))
// Update the query text to include date range
currentQuery := params.Get("q")
params.Set("q", currentQuery+dateRange)
}
// Bing-specific parameters for consistent results
params.Add("form", "QBLH") // Standard search form
params.Add("qs", "HS") // Query suggestions
params.Add("sp", "-1") // Search provider
params.Add("pq", params.Get("q")) // Previous query
base.RawQuery = params.Encode()
return base.String(), nil
}
func BuildImageURL(q core.Query) (string, error) {
base, err := url.Parse("https://www.bing.com")
if err != nil {
return "", err
}
base.Path += "images/search"
params := url.Values{}
if q.Text != "" || q.Site != "" {
text := q.Text
if q.Site != "" {
text += " site:" + q.Site
}
params.Add("q", text)
}
if len(params.Get("q")) == 0 {
return "", errors.New("empty query built")
}
// Add common parameters
if q.LangCode != "" {
params.Add("setlang", strings.ToLower(q.LangCode))
}
// Image-specific parameters
params.Add("form", "HDRSC2")
params.Add("first", "1")
params.Add("scenario", "ImageBasicHover")
base.RawQuery = params.Encode()
return base.String(), nil
}

View File

@@ -2,6 +2,7 @@ package cmd
import (
"fmt"
"strconv"
"strings"
"github.com/karust/openserp/core"
@@ -12,17 +13,19 @@ import (
)
const (
version = "0.3"
version = "0.5.3"
defaultConfigFilename = "config"
envPrefix = "OPENSERP"
)
type Config struct {
App AppConfig `mapstructure:"app"`
Config2Capcha Config2Captcha `mapstructure:"2captcha"`
GoogleConfig core.SearchEngineOptions `mapstructure:"google"`
YandexConfig core.SearchEngineOptions `mapstructure:"yandex"`
BaiduConfig core.SearchEngineOptions `mapstructure:"baidu"`
App AppConfig `mapstructure:"app"`
Config2Capcha Config2Captcha `mapstructure:"2captcha"`
GoogleConfig core.SearchEngineOptions `mapstructure:"google"`
YandexConfig core.SearchEngineOptions `mapstructure:"yandex"`
BaiduConfig core.SearchEngineOptions `mapstructure:"baidu"`
BingConfig core.SearchEngineOptions `mapstructure:"bing"`
DuckDuckGoConfig core.SearchEngineOptions `mapstructure:"duckduckgo"`
}
type Config2Captcha struct {
@@ -40,10 +43,20 @@ type AppConfig struct {
IsDebug bool `mapstructure:"debug"`
IsVerbose bool `mapstructure:"verbose"`
IsRawRequests bool `mapstructure:"raw_requests"`
ProxyURL string `mapstructure:"proxy"`
Insecure bool `mapstructure:"insecure"`
IsStealth bool `mapstructure:"stealth"`
}
var config = Config{}
var flagToConfigKey = map[string]string{
"config": "app.config_path",
"leave": "app.leave_head",
"raw": "app.raw_requests",
"2captcha_key": "2captcha.apikey",
}
var RootCmd = &cobra.Command{
Use: "openserp",
Short: "Open SERP",
@@ -72,15 +85,39 @@ var RootCmd = &cobra.Command{
// Bind each cobra flag to its associated viper configuration (config file and environment variable)
func bindFlags(cmd *cobra.Command, vpr *viper.Viper) {
cmd.Flags().VisitAll(func(flg *pflag.Flag) {
configName := "app." + flg.Name
configName, ok := flagToConfigKey[flg.Name]
if !ok {
configName = "app." + flg.Name
}
if err := vpr.BindPFlag(configName, flg); err != nil {
logrus.Errorf("Unable to bind flag %s: %v", flg.Name, err)
}
// Apply viper config value to the flag if viper has a value
if flg.Changed {
vpr.Set(configName, flg.Value)
val, err := parseFlagValue(flg)
if err != nil {
logrus.Errorf("Unable to parse flag %s: %v", flg.Name, err)
return
}
vpr.Set(configName, val)
}
})
}
func parseFlagValue(flg *pflag.Flag) (interface{}, error) {
switch flg.Value.Type() {
case "string":
return flg.Value.String(), nil
case "bool":
return strconv.ParseBool(flg.Value.String())
case "int":
return strconv.Atoi(flg.Value.String())
default:
return flg.Value.String(), nil
}
}
// Initialize Viper
func initializeConfig(cmd *cobra.Command) error {
v := viper.New()
@@ -89,14 +126,14 @@ func initializeConfig(cmd *cobra.Command) error {
v.SetConfigName(defaultConfigFilename)
v.AddConfigPath(".")
// 1. Config. Return an error if we cannot parse the config file.
// 1. Config file (lowest priority). Return an error if we cannot parse the config file.
err := v.ReadInConfig()
if err != nil {
err = fmt.Errorf("cannot read config: %v", err)
logrus.Warn(err)
}
// 2. Env. Bind environment variables to their equivalent keys with underscores
// 2. Environment variables (medium priority). Bind environment variables to their equivalent keys with underscores
for _, key := range v.AllKeys() {
envKey := envPrefix + "_" + strings.ToUpper(strings.ReplaceAll(key, ".", "_"))
err := v.BindEnv(key, envKey)
@@ -105,7 +142,7 @@ func initializeConfig(cmd *cobra.Command) error {
}
}
// 3. Cmd flags. Bind the current command's flags to viper
// 3. Command flags (highest priority). Bind the current command's flags to viper
bindFlags(cmd, v)
// Dump Viper values to config struct
@@ -133,4 +170,7 @@ func init() {
RootCmd.PersistentFlags().BoolVarP(&config.App.IsRawRequests, "raw", "r", false, "Disable browser usage, use HTTP requests")
RootCmd.PersistentFlags().BoolVarP(&config.App.IsLeaveHead, "leave", "", false, "Leave browser and tabs opened after search is made")
RootCmd.PersistentFlags().StringVarP(&config.Config2Capcha.ApiKey, "2captcha_key", "", "", "2 captcha api key")
RootCmd.PersistentFlags().StringVarP(&config.App.ProxyURL, "proxy", "x", "", "HTTP or Socks5 proxy URL (e.g. http://user:pass@127.0.0.1:8080)")
RootCmd.PersistentFlags().BoolVarP(&config.App.IsStealth, "stealth", "s", false, "Use stealth browser plugin")
RootCmd.PersistentFlags().BoolVarP(&config.App.Insecure, "insecure", "k", false, "Allow insecure TLS connections")
}

View File

@@ -7,7 +7,9 @@ import (
"time"
"github.com/karust/openserp/baidu"
"github.com/karust/openserp/bing"
"github.com/karust/openserp/core"
"github.com/karust/openserp/duckduckgo"
"github.com/karust/openserp/google"
"github.com/karust/openserp/yandex"
"github.com/sirupsen/logrus"
@@ -17,7 +19,7 @@ import (
var searchCMD = &cobra.Command{
Use: "search",
Aliases: []string{"find"},
Short: "Search results using chosen web search engine (google, yandex, baidu)",
Short: "Search results using chosen web search engine (google, yandex, baidu, bing, duckduckgo)",
Args: cobra.MatchAll(cobra.OnlyValidArgs, cobra.ExactArgs(2)),
Run: search,
}
@@ -26,22 +28,30 @@ func search(cmd *cobra.Command, args []string) {
var err error
engineType := args[0]
query := core.Query{
Text: args[1],
Limit: 10,
Text: args[1],
Limit: 10,
ProxyURL: config.App.ProxyURL,
Insecure: config.App.Insecure,
}
results := []core.SearchResult{}
logrus.Infof("Starting SERP search request using %s engine for query: %s", engineType, query.Text)
var results []core.SearchResult
if config.App.IsRawRequests {
logrus.Infof("Using raw requests mode for %s search", engineType)
results, err = searchRaw(engineType, query)
} else {
logrus.Infof("Using browser mode for %s search", engineType)
results, err = searchBrowser(engineType, query)
}
if err != nil {
logrus.Error(err)
logrus.Errorf("Error during %s search: %s", engineType, err)
return
}
logrus.Infof("Successfully completed SERP search using %s engine, returned %d results", engineType, len(results))
b, err := json.MarshalIndent(results, "", " ")
if err != nil {
logrus.Error(err)
@@ -59,6 +69,9 @@ func searchBrowser(engineType string, query core.Query) ([]core.SearchResult, er
Timeout: time.Second * time.Duration(config.App.Timeout),
LeavePageOpen: config.App.IsLeaveHead,
CaptchaSolverApiKey: config.Config2Capcha.ApiKey,
ProxyURL: config.App.ProxyURL,
Insecure: config.App.Insecure,
UseStealth: config.App.IsStealth,
}
if config.App.IsDebug {
@@ -77,6 +90,10 @@ func searchBrowser(engineType string, query core.Query) ([]core.SearchResult, er
engine = google.New(*browser, config.GoogleConfig)
case "baidu":
engine = baidu.New(*browser, config.BaiduConfig)
case "bing":
engine = bing.New(*browser, config.BingConfig)
case "duck":
engine = duckduckgo.New(*browser, config.DuckDuckGoConfig)
default:
logrus.Infof("No `%s` search engine found", engineType)
}
@@ -94,6 +111,12 @@ func searchRaw(engineType string, query core.Query) ([]core.SearchResult, error)
return google.Search(query)
case "baidu":
return baidu.Search(query)
case "bing":
logrus.Warn("Bing does not support raw HTTP requests mode. Please use browser mode instead.")
return nil, fmt.Errorf("bing does not support raw requests mode")
case "duck":
logrus.Warn("DuckDuckGo does not support raw HTTP requests mode. Please use browser mode instead.")
return nil, fmt.Errorf("duckduckgo does not support raw requests mode")
default:
logrus.Infof("No `%s` search engine found", engineType)
}

View File

@@ -1,16 +1,59 @@
package cmd
import (
"fmt"
"time"
"github.com/karust/openserp/baidu"
"github.com/karust/openserp/bing"
"github.com/karust/openserp/core"
"github.com/karust/openserp/duckduckgo"
"github.com/karust/openserp/google"
"github.com/karust/openserp/yandex"
"github.com/sirupsen/logrus"
"github.com/spf13/cobra"
"golang.org/x/time/rate"
)
// rawEngine implements SearchEngine interface for raw HTTP requests
type rawEngine struct {
name string
}
func (r *rawEngine) Search(q core.Query) ([]core.SearchResult, error) {
// Inject proxy settings from config
q.ProxyURL = config.App.ProxyURL
q.Insecure = config.App.Insecure
switch r.name {
case "google":
return google.Search(q)
case "yandex":
return yandex.Search(q)
case "baidu":
return baidu.Search(q)
default:
return nil, fmt.Errorf("unsupported engine: %s", r.name)
}
}
func (r *rawEngine) SearchImage(q core.Query) ([]core.SearchResult, error) {
return nil, fmt.Errorf("image search is not supported in raw mode for %s", r.name)
}
func (r *rawEngine) Name() string {
return r.name
}
func (r *rawEngine) IsInitialized() bool {
return true
}
func (r *rawEngine) GetRateLimiter() *rate.Limiter {
// Use default rate limiter for raw requests
return rate.NewLimiter(rate.Every(time.Second), 5)
}
var serveCMD = &cobra.Command{
Use: "serve",
Aliases: []string{"listen"},
@@ -20,12 +63,26 @@ var serveCMD = &cobra.Command{
}
func serve(cmd *cobra.Command, args []string) {
if config.App.IsRawRequests {
logrus.Warn("Browserless results are very inconsistent or may not even work!")
serv := core.NewServer(config.App.Host, config.App.Port,
&rawEngine{name: "google"},
&rawEngine{name: "yandex"},
&rawEngine{name: "baidu"},
)
serv.Listen()
return
}
opts := core.BrowserOpts{
IsHeadless: !config.App.IsBrowserHead, // Disable headless if browser head mode is set
IsLeakless: config.App.IsLeakless,
Timeout: time.Second * time.Duration(config.App.Timeout),
LeavePageOpen: config.App.IsLeaveHead,
CaptchaSolverApiKey: config.Config2Capcha.ApiKey,
ProxyURL: config.App.ProxyURL,
Insecure: config.App.Insecure,
UseStealth: config.App.IsStealth,
}
if config.App.IsDebug {
@@ -35,14 +92,21 @@ func serve(cmd *cobra.Command, args []string) {
browser, err := core.NewBrowser(opts)
if err != nil {
logrus.Error(err)
return
}
yand := yandex.New(*browser, config.YandexConfig)
gogl := google.New(*browser, config.GoogleConfig)
baidu := baidu.New(*browser, config.BaiduConfig)
bing := bing.New(*browser, config.BingConfig)
ddg := duckduckgo.New(*browser, config.DuckDuckGoConfig)
serv := core.NewServer(config.App.Host, config.App.Port, gogl, yand, baidu)
serv.Listen()
serv := core.NewServer(config.App.Host, config.App.Port, gogl, yand, baidu, bing, ddg)
err = serv.Listen()
if err != nil {
logrus.Error(err)
}
}
func init() {

View File

@@ -7,6 +7,8 @@ app:
head: false
leakless: false
leave_head: false
stealth: false
insecure: true
2captcha:
apikey: "123123123123123"
@@ -23,3 +25,11 @@ yandex:
baidu:
rate_requests: 4
rate_burst: 2
bing:
rate_requests: 4
rate_burst: 2
duckduckgo:
rate_requests: 4
rate_burst: 2

View File

@@ -1,11 +1,15 @@
package core
import (
"fmt"
"net/url"
"strings"
"time"
"github.com/go-rod/rod"
"github.com/go-rod/rod/lib/devices"
"github.com/go-rod/rod/lib/launcher"
"github.com/go-rod/rod/lib/proto"
"github.com/go-rod/stealth"
"github.com/sirupsen/logrus"
)
@@ -19,6 +23,9 @@ type BrowserOpts struct {
LeavePageOpen bool // Leave pages and browser open
WaitLoadTime time.Duration // Time to wait till page loads
CaptchaSolverApiKey string // 2Captcha api key
ProxyURL string // Proxy URL
Insecure bool // Allow insecure TLS connections
UseStealth bool // Use go-rod stealth plugin
}
@@ -47,9 +54,34 @@ func NewBrowser(opts BrowserOpts) (*Browser, error) {
path, has := launcher.LookPath()
logrus.Debug("Browser found: ", has)
// Create launcher
l := launcher.New().Bin(path).Leakless(opts.IsLeakless).Headless(opts.IsHeadless).Set("disable-blink-features", "AutomationControlled").
Delete("enable-automation")
// Configure proxy if specified
if opts.ProxyURL != "" {
proxyUrl, err := url.Parse(opts.ProxyURL)
if err != nil {
return nil, fmt.Errorf("invalid proxy URL: %v", err)
}
// Make sure the proxy URL includes the scheme when passed to launcher
// This ensures proper handling of SOCKS5 proxies
proxyStr := proxyUrl.String()
logrus.Debugf("Setting up proxy: %s", proxyStr)
l = l.Proxy(proxyStr)
// Check if proxy has auth credentials
if proxyUrl.User != nil {
username := proxyUrl.User.Username()
logrus.Debugf("Using proxy authentication: %s:****", username)
// We'll handle auth in the Navigate method
}
}
var err error
b := Browser{BrowserOpts: opts}
b.browserAddr, err = launcher.New().Bin(path).Leakless(opts.IsLeakless).Headless(opts.IsHeadless).Launch()
b.browserAddr, err = l.Launch()
if opts.CaptchaSolverApiKey != "" {
b.CaptchaSolver = NewSolver(opts.CaptchaSolverApiKey)
@@ -69,31 +101,79 @@ func (b *Browser) IsInitialized() bool {
}
// Open URL
func (b *Browser) Navigate(URL string) *rod.Page {
func (b *Browser) Navigate(URL string) (*rod.Page, error) {
logrus.Debug("Navigate to: ", URL)
b.browser = rod.New().ControlURL(b.browserAddr)
b.browser.MustConnect()
b.browser.SetCookies(nil)
//page := b.browser.MustPage(URL)
page := stealth.MustPage(b.browser)
page.MustEmulate(devices.Device{
//UserAgent: uarand.GetRandom(),
AcceptLanguage: b.LanguageCode,
})
page.MustNavigate(URL)
// Handle proxy authentication before any navigations
if b.ProxyURL != "" {
proxyUrl, _ := url.Parse(b.ProxyURL)
// Always ignore certificate errors when using proxies
// This fixes the ERR_CERT_AUTHORITY_INVALID error for SOCKS5 proxies
b.browser.MustIgnoreCertErrors(true)
if proxyUrl.User != nil {
username := proxyUrl.User.Username()
password, _ := proxyUrl.User.Password()
// Launch auth handler before any navigation occurs
go b.browser.MustHandleAuth(username, password)()
}
} else if b.Insecure {
// Still respect the insecure flag if no proxy is used
b.browser.MustIgnoreCertErrors(true)
}
ua := strings.ReplaceAll(b.browser.MustVersion().UserAgent, "HeadlessChrome/", "Chrome/")
var page *rod.Page
if b.UseStealth {
page = stealth.MustPage(b.browser)
page.MustEmulate(devices.Device{
AcceptLanguage: b.LanguageCode,
UserAgent: ua,
})
} else {
page = b.browser.MustPage("about:blank")
page.MustEmulate(devices.Device{
AcceptLanguage: b.LanguageCode,
UserAgent: ua,
})
proto.EmulationSetDeviceMetricsOverride{
Width: 1920,
Height: 1080,
DeviceScaleFactor: 1,
Mobile: false,
ScreenWidth: &[]int{1920}[0],
ScreenHeight: &[]int{1080}[0],
}.Call(page)
}
//EnableCustomStealth(page)
err := page.Navigate(URL)
if err != nil {
return nil, err
}
// Avoid panics from MustWaitLoad when the target navigates/closes mid-wait
if werr := page.WaitLoad(); werr != nil {
logrus.Debugf("WaitLoad returned early: %v", werr)
}
wait := page.MustWaitRequestIdle()
// may cause bugs with google
if b.WaitRequests {
wait()
}
// Wait till page loads
//time.Sleep(b.WaitLoadTime)
return page
time.Sleep(2 * time.Second)
return page, nil
}
func (b *Browser) Close() error {

View File

@@ -38,13 +38,12 @@ func TestBot(t *testing.T) {
t.Fatalf("Error failed initializing browser: %s", err)
}
page := browser.Navigate("https://bot.sannysoft.com")
page, _ := browser.Navigate("https://bot.sannysoft.com")
page.MustScreenshotFullPage("./test/screenshot_bot.png")
page = browser.Navigate("https://www.whatismybrowser.com/")
page, _ = browser.Navigate("https://www.whatismybrowser.com/")
page.MustScreenshotFullPage("./test/screenshot_browser.png")
page = browser.Navigate("https://abrahamjuliot.github.io/creepjs/")
page, _ = browser.Navigate("https://abrahamjuliot.github.io/creepjs/")
page.MustScreenshotFullPage("./test/screenshot_creep.png")
}

View File

@@ -14,7 +14,7 @@ func NewSolver(apikey string) *CaptchaSolver {
return &cs
}
func (cs *CaptchaSolver) SolveReCaptcha2(sitekey, pageUrl, dataS string) (string, error) {
func (cs *CaptchaSolver) SolveReCaptcha2(sitekey, pageUrl, dataS string) (string, string, error) {
cap := api2captcha.ReCaptcha{
SiteKey: sitekey,
Url: pageUrl,

View File

@@ -13,7 +13,7 @@ func Test2Captcha(t *testing.T) {
sitekey := "6LfwuyUTAAAAAOAmoS0fdqijC2PbbdH4kjq62Y1b"
url := "https://www.google.com/sorry/index?continue=https://www.google.de/search%3Fhl%3DDE%26lr%3Dlang_de%26nfpr%3D1%26num%3D500%26pws%3D0%26q%3Dwhere%2Bwhy%2Beach&hl=DE&q=EgRegw55GObHiq4GIjDqmzFKayGXrS2-s9ooWfcskhpK8-6tIjWSaSvhxd3f5eAyUXj7lYq2DYLDXB8ASz0yAXJaAUM"
datas := "Ghk0n7ZQNDS0c7ES53eef_YBfSdfeXnyRD0p2OR0R4Dg91CUXKS_hio5Do6TpJ8sHhhOat_NymTASZGe1gqAjP7w9dSvhvRT7QXsrdziO3JPngLDSRzDdjT42GDcSbO0kzInlDPxe1yy2t4yifo9xHpMnlZU7pTVNTQUIXqOMLHAR-iERi6aoSQDQ4d-88-jW3LEinquxEut0OhHG2l2stwG9AnCmNvCsUNJda-H24saFlOh5csK9KNXeeQmpr6at52_skMIMiLXSlY56vYFVCRMkXLQdAM"
resp, err := solver.SolveReCaptcha2(sitekey, url, datas)
resp, _, err := solver.SolveReCaptcha2(sitekey, url, datas)
if err != nil || resp == "" {
t.Fatalf("Failed to solve recaptchaV2: %s", err)
}

View File

@@ -20,6 +20,26 @@ type SearchResult struct {
Ad bool `json:"ad"`
}
func DeduplicateResults(results []SearchResult) []SearchResult {
unique := make(map[string]bool)
var deduped []SearchResult
for _, result := range results {
if result.URL == "" {
continue
}
if !unique[result.URL] {
unique[result.URL] = true
deduped = append(deduped, result)
}
}
sort.Slice(deduped, func(i, j int) bool {
return deduped[i].Rank < deduped[j].Rank
})
return deduped
}
func ConvertSearchResultsMap(searchResultsMap map[string]SearchResult) *[]SearchResult {
searchResults := []SearchResult{}
@@ -41,6 +61,8 @@ type Query struct {
Site string // Search site
Limit int // Limit the number of results
Answers bool // Include question and answers from SERP page to results with negative indexes
ProxyURL string // Proxy URL for raw requests
Insecure bool // Allow insecure TLS connections
}
func (q Query) IsEmpty() bool {

View File

@@ -14,20 +14,93 @@ type customFormatter struct {
}
func (f *customFormatter) Format(entry *logrus.Entry) ([]byte, error) {
// var levelColor int
// switch entry.Level {
// case logrus.DebugLevel, logrus.TraceLevel:
// levelColor = 42 // green highlight
// case logrus.WarnLevel:
// levelColor = 33 // yellow
// case logrus.ErrorLevel, logrus.FatalLevel, logrus.PanicLevel:
// levelColor = 41 // red highlight
// default:
// levelColor = 36 // blue
// }
//[]byte(fmt.Sprintf("[%s] - \x1b[%dm%s\x1b[0m - %s\n", entry.Time.Format(f.TimestampFormat), levelColor, strings.ToUpper(entry.Level.String()), entry.Message))
message := entry.Message
return []byte(fmt.Sprintf("[%s][%s] \t%s\n", entry.Time.Format(f.TimestampFormat), strings.ToUpper(entry.Level.String()), entry.Message)), nil
// Check if engine name is provided as a field
engineName := ""
if engine, exists := entry.Data["engine"]; exists {
if engineStr, ok := engine.(string); ok {
engineName = engineStr
}
}
// Format: [timestamp][level][engine] message
if engineName != "" {
return []byte(fmt.Sprintf("[%s][%s][%s] %s\n",
entry.Time.Format(f.TimestampFormat),
strings.ToUpper(entry.Level.String()),
engineName,
message)), nil
}
// Format: [timestamp][level] message (no engine)
return []byte(fmt.Sprintf("[%s][%s] %s\n",
entry.Time.Format(f.TimestampFormat),
strings.ToUpper(entry.Level.String()),
message)), nil
}
// EngineLogger provides simplified logging for search engines
type EngineLogger struct {
engine string
logger *logrus.Entry
}
// NewEngineLogger creates a new logger for a specific search engine
func NewEngineLogger(engine string) *EngineLogger {
return &EngineLogger{
engine: engine,
logger: logrus.WithField("engine", engine),
}
}
// Debug logs a debug message
func (el *EngineLogger) Debug(message string, args ...interface{}) {
el.logger.Debugf(message, args...)
}
// Info logs an info message
func (el *EngineLogger) Info(message string, args ...interface{}) {
el.logger.Infof(message, args...)
}
// Warn logs a warning message
func (el *EngineLogger) Warn(message string, args ...interface{}) {
el.logger.Warnf(message, args...)
}
// Error logs an error message
func (el *EngineLogger) Error(message string, args ...interface{}) {
el.logger.Errorf(message, args...)
}
// Fatal logs a fatal message
func (el *EngineLogger) Fatal(message string, args ...interface{}) {
el.logger.Fatalf(message, args...)
}
// Panic logs a panic message
func (el *EngineLogger) Panic(message string, args ...interface{}) {
el.logger.Panicf(message, args...)
}
// LogWithEngine logs a message with engine information (deprecated - use EngineLogger instead)
func LogWithEngine(level logrus.Level, engine, message string, args ...interface{}) {
entry := logrus.WithField("engine", engine)
switch level {
case logrus.DebugLevel:
entry.Debugf(message, args...)
case logrus.InfoLevel:
entry.Infof(message, args...)
case logrus.WarnLevel:
entry.Warnf(message, args...)
case logrus.ErrorLevel:
entry.Errorf(message, args...)
case logrus.FatalLevel:
entry.Fatalf(message, args...)
case logrus.PanicLevel:
entry.Panicf(message, args...)
}
}
func InitLogger(isVerbose, isDebug bool) {

View File

@@ -3,7 +3,9 @@ package core
import (
"context"
"fmt"
"sort"
"strings"
"sync"
"github.com/gofiber/fiber/v2"
"github.com/sirupsen/logrus"
@@ -19,22 +21,30 @@ type SearchEngine interface {
}
type Server struct {
app *fiber.App
addr string
app *fiber.App
addr string
searchEngines []SearchEngine
}
func NewServer(host string, port int, searchEngines ...SearchEngine) *Server {
addr := fmt.Sprintf("%s:%d", host, port)
serv := Server{
app: fiber.New(),
addr: addr,
app: fiber.New(),
addr: addr,
searchEngines: searchEngines,
}
for _, engine := range searchEngines {
locEngine := engine
limiter := engine.GetRateLimiter()
serv.app.Get(fmt.Sprintf("/%s/search", strings.ToLower(locEngine.Name())), func(c *fiber.Ctx) error {
// Custom endpoint mapping for DuckDuckGo
endpointName := strings.ToLower(locEngine.Name())
if endpointName == "duckduckgo" {
endpointName = "duck"
}
serv.app.Get(fmt.Sprintf("/%s/search", endpointName), func(c *fiber.Ctx) error {
q := Query{}
err := q.InitFromContext(c)
if err != nil {
@@ -42,6 +52,8 @@ func NewServer(host string, port int, searchEngines ...SearchEngine) *Server {
return err
}
logrus.Infof("Starting SERP search request using %s engine for query: %s", locEngine.Name(), q.Text)
err = limiter.Wait(context.Background())
if err != nil {
logrus.Errorf("Ratelimiter error during %s query: %s", locEngine.Name(), err)
@@ -60,10 +72,11 @@ func NewServer(host string, port int, searchEngines ...SearchEngine) *Server {
return fiber.NewError(fiber.StatusServiceUnavailable, err.Error())
}
logrus.Infof("Successfully completed SERP search using %s engine, returned %d results", locEngine.Name(), len(res))
return c.JSON(res)
})
serv.app.Get(fmt.Sprintf("/%s/image", strings.ToLower(locEngine.Name())), func(c *fiber.Ctx) error {
serv.app.Get(fmt.Sprintf("/%s/image", endpointName), func(c *fiber.Ctx) error {
q := Query{}
err := q.InitFromContext(c)
if err != nil {
@@ -71,6 +84,8 @@ func NewServer(host string, port int, searchEngines ...SearchEngine) *Server {
return err
}
logrus.Infof("Starting SERP image search request using %s engine for query: %s", locEngine.Name(), q.Text)
err = limiter.Wait(context.Background())
if err != nil {
logrus.Errorf("Ratelimiter error during %s query: %s", locEngine.Name(), err)
@@ -79,6 +94,7 @@ func NewServer(host string, port int, searchEngines ...SearchEngine) *Server {
res, err := locEngine.SearchImage(q)
if err != nil && len(res) > 0 {
logrus.Warnf("Partial results returned from %s image search despite error: %s", locEngine.Name(), err)
c.Status(503)
return c.JSON(res)
}
@@ -91,17 +107,269 @@ func NewServer(host string, port int, searchEngines ...SearchEngine) *Server {
err = fmt.Errorf("%s", err)
}
logrus.Errorf("Error during %s search: %s", locEngine.Name(), err)
logrus.Errorf("Error during %s image search: %s", locEngine.Name(), err)
return fiber.NewError(fiber.StatusServiceUnavailable, err.Error())
}
logrus.Infof("Successfully completed SERP image search using [%s], returned %d results", locEngine.Name(), len(res))
return c.JSON(res)
})
}
// Add megasearch endpoint
serv.app.Get("/mega/search", serv.handleMegaSearch)
// Add megasearch image endpoint
serv.app.Get("/mega/image", serv.handleMegaImage)
// Add endpoint to list available engines
serv.app.Get("/mega/engines", serv.handleListEngines)
return &serv
}
// MegaSearchResult represents a search result with engine information
type MegaSearchResult struct {
SearchResult
Engine string `json:"engine"`
}
// handleMegaSearch handles the /megasearch endpoint
func (s *Server) handleMegaSearch(c *fiber.Ctx) error {
q := Query{}
err := q.InitFromContext(c)
if err != nil {
logrus.Errorf("Error while setting megasearch query: %s", err)
return err
}
// Get engines parameter to filter which engines to use
enginesParam := c.Query("engines", "")
var enginesToUse []SearchEngine
if enginesParam != "" {
// Parse comma-separated list of engines
engineNames := strings.Split(enginesParam, ",")
for _, engineName := range engineNames {
engineName = strings.TrimSpace(strings.ToLower(engineName))
for _, engine := range s.searchEngines {
if strings.ToLower(engine.Name()) == engineName {
enginesToUse = append(enginesToUse, engine)
break
}
}
}
} else {
// Use all engines if no specific engines specified
enginesToUse = s.searchEngines
}
if len(enginesToUse) == 0 {
return fiber.NewError(fiber.StatusBadRequest, "No valid search engines specified")
}
// Log which engines will be used
engineNames := make([]string, len(enginesToUse))
for i, engine := range enginesToUse {
engineNames[i] = engine.Name()
}
logrus.Infof("Starting SERP megasearch request using engines: %s for query: %s", strings.Join(engineNames, ", "), q.Text)
// Execute searches in parallel across selected engines
results := s.searchSelectedEngines(q, enginesToUse)
// Deduplicate results while preserving engine information
dedupedResults := s.deduplicateMegaResults(results)
logrus.Infof("Successfully completed SERP megasearch using %d engines, returned %d deduplicated results", len(enginesToUse), len(dedupedResults))
return c.JSON(dedupedResults)
}
// handleMegaImage handles the /mega/image endpoint
func (s *Server) handleMegaImage(c *fiber.Ctx) error {
q := Query{}
err := q.InitFromContext(c)
if err != nil {
logrus.Errorf("Error while setting megasearch image query: %s", err)
return err
}
// Get engines parameter to filter which engines to use
enginesParam := c.Query("engines", "")
var enginesToUse []SearchEngine
if enginesParam != "" {
// Parse comma-separated list of engines
engineNames := strings.Split(enginesParam, ",")
for _, engineName := range engineNames {
engineName = strings.TrimSpace(strings.ToLower(engineName))
for _, engine := range s.searchEngines {
if strings.ToLower(engine.Name()) == engineName {
enginesToUse = append(enginesToUse, engine)
break
}
}
}
} else {
// Use all engines if no specific engines specified
enginesToUse = s.searchEngines
}
if len(enginesToUse) == 0 {
return fiber.NewError(fiber.StatusBadRequest, "No valid search engines specified")
}
// Log which engines will be used
engineNames := make([]string, len(enginesToUse))
for i, engine := range enginesToUse {
engineNames[i] = engine.Name()
}
logrus.Infof("Starting SERP megasearch image request using engines: %s for query: %s", strings.Join(engineNames, ", "), q.Text)
// Execute image searches in parallel across selected engines
results := s.searchSelectedEnginesImage(q, enginesToUse)
// Deduplicate results while preserving engine information
dedupedResults := s.deduplicateMegaResults(results)
logrus.Infof("Successfully completed SERP megasearch image using %d engines, returned %d deduplicated results", len(enginesToUse), len(dedupedResults))
return c.JSON(dedupedResults)
}
// handleListEngines lists all available search engines
func (s *Server) handleListEngines(c *fiber.Ctx) error {
var engines []map[string]interface{}
for _, engine := range s.searchEngines {
engines = append(engines, map[string]interface{}{
"name": engine.Name(),
"initialized": engine.IsInitialized(),
})
}
return c.JSON(map[string]interface{}{
"engines": engines,
"total": len(engines),
})
}
// searchSelectedEngines performs parallel searches across selected engines
func (s *Server) searchSelectedEngines(q Query, engines []SearchEngine) []MegaSearchResult {
var wg sync.WaitGroup
var mu sync.Mutex
var allResults []MegaSearchResult
for _, engine := range engines {
wg.Add(1)
go func(eng SearchEngine) {
defer wg.Done()
// Apply rate limiting
limiter := eng.GetRateLimiter()
if limiter != nil {
err := limiter.Wait(context.Background())
if err != nil {
logrus.Errorf("Ratelimiter error during %s megasearch: %s", eng.Name(), err)
}
}
// Perform search
results, err := eng.Search(q)
if err != nil {
logrus.Errorf("Error during %s megasearch: %s", eng.Name(), err)
return
}
// Convert to MegaSearchResult with engine info
mu.Lock()
for _, result := range results {
megaResult := MegaSearchResult{
SearchResult: result,
Engine: eng.Name(),
}
allResults = append(allResults, megaResult)
}
mu.Unlock()
}(engine)
}
wg.Wait()
return allResults
}
// searchSelectedEnginesImage performs parallel image searches across selected engines
func (s *Server) searchSelectedEnginesImage(q Query, engines []SearchEngine) []MegaSearchResult {
var wg sync.WaitGroup
var mu sync.Mutex
var allResults []MegaSearchResult
for _, engine := range engines {
wg.Add(1)
go func(eng SearchEngine) {
defer wg.Done()
// Apply rate limiting
limiter := eng.GetRateLimiter()
if limiter != nil {
err := limiter.Wait(context.Background())
if err != nil {
logrus.Errorf("Ratelimiter error during %s megasearch image: %s", eng.Name(), err)
}
}
// Perform image search
results, err := eng.SearchImage(q)
if err != nil {
logrus.Errorf("Error during %s megasearch image: %s", eng.Name(), err)
return
}
// Convert to MegaSearchResult with engine info
mu.Lock()
for _, result := range results {
megaResult := MegaSearchResult{
SearchResult: result,
Engine: eng.Name(),
}
allResults = append(allResults, megaResult)
}
mu.Unlock()
}(engine)
}
wg.Wait()
return allResults
}
// deduplicateMegaResults deduplicates results while preserving engine information
func (s *Server) deduplicateMegaResults(results []MegaSearchResult) []MegaSearchResult {
urlMap := make(map[string]MegaSearchResult)
// Process results and keep the first occurrence of each URL
for _, result := range results {
if result.URL == "" {
continue
}
if _, exists := urlMap[result.URL]; !exists {
urlMap[result.URL] = result
}
}
// Convert map back to slice and sort by rank
var deduped []MegaSearchResult
for _, result := range urlMap {
deduped = append(deduped, result)
}
// Sort by rank
sort.Slice(deduped, func(i, j int) bool {
return deduped[i].Rank < deduped[j].Rank
})
return deduped
}
func (s *Server) Listen() error {
return s.app.Listen(s.addr)
}

420
duckduckgo/search.go Normal file
View File

@@ -0,0 +1,420 @@
package duckduckgo
import (
"fmt"
"strings"
"time"
"github.com/go-rod/rod"
"github.com/karust/openserp/core"
"golang.org/x/time/rate"
)
type DuckDuckGo struct {
core.Browser
core.SearchEngineOptions
pageSleep time.Duration // Sleep between pages
logger *core.EngineLogger
}
func New(browser core.Browser, opts core.SearchEngineOptions) *DuckDuckGo {
ddg := DuckDuckGo{Browser: browser}
opts.Init()
ddg.SearchEngineOptions = opts
ddg.logger = core.NewEngineLogger("DuckDuckGo")
ddg.pageSleep = time.Second * 1
return &ddg
}
func (ddg *DuckDuckGo) Name() string {
return "duckduckgo"
}
func (ddg *DuckDuckGo) GetRateLimiter() *rate.Limiter {
ratelimit := rate.Every(ddg.GetRatelimit())
return rate.NewLimiter(ratelimit, ddg.RateBurst)
}
func (ddg *DuckDuckGo) isCaptcha(page *rod.Page) bool {
html, err := page.Timeout(ddg.GetSelectorTimeout()).HTML()
if err != nil {
return false
}
return strings.Contains(html, "bots user")
}
// Check if no results are found
func (ddg *DuckDuckGo) isNoResults(page *rod.Page) bool {
_, err := page.Timeout(ddg.GetSelectorTimeout()).Search("div[class*='no-results']")
return err == nil
}
func (ddg *DuckDuckGo) parseResults(results rod.Elements, pageNum int) []core.SearchResult {
searchResults := []core.SearchResult{}
for i, r := range results {
// Get URL - try multiple selectors
var link *rod.Element
var err error
linkSelectors := []string{
"a[data-testid='result-title-a']",
"a.result__a",
"a.result__url",
"h2 a",
"h3 a",
"a[href]",
"a",
}
for _, selector := range linkSelectors {
link, err = r.Element(selector)
if err == nil {
break
}
}
if err != nil {
ddg.logger.Debug("Missing link")
continue
}
linkText, err := link.Property("href")
if err != nil {
ddg.logger.Debug("Missing href")
continue
}
// Validate that we have a proper URL
hrefStr := linkText.String()
if hrefStr == "" || hrefStr == "#" || hrefStr == "javascript:void(0)" {
ddg.logger.Debug("Invalid href: %s", hrefStr)
continue
}
// Get title - try multiple selectors
var titleTag *rod.Element
titleSelectors := []string{
"h2",
".result__title",
".result__a",
"span",
"div",
}
for _, selector := range titleSelectors {
titleTag, err = r.Element(selector)
if err == nil {
break
}
}
title := "No title"
if titleTag != nil {
title, _ = titleTag.Text()
}
// Get description - try multiple selectors
desc := ""
descSelectors := []string{
"div[data-result='snippet']",
".result__snippet",
".result__body",
"span[class*='snippet']",
"div[class*='snippet']",
"p",
}
for _, selector := range descSelectors {
descTag, err := r.Element(selector)
if err == nil {
desc = descTag.MustText()
break
}
}
// Check if it's an ad
isAd := false
adSelectors := []string{
"[data-testid='ad-badge']",
".ad-badge",
".result--ad",
}
for _, selector := range adSelectors {
adIndicator, err := r.Element(selector)
if err == nil && adIndicator != nil {
isAd = true
break
}
}
result := core.SearchResult{
Rank: (pageNum * 10) + (i + 1),
URL: hrefStr,
Title: title,
Description: desc,
Ad: isAd,
}
searchResults = append(searchResults, result)
}
return searchResults
}
func (ddg *DuckDuckGo) Search(query core.Query) ([]core.SearchResult, error) {
ddg.logger.Debug("Starting search, query: %+v", query)
allResults := []core.SearchResult{}
searchPage := 0
for len(allResults) < query.Limit {
url, err := BuildURL(query)
if err != nil {
return nil, err
}
page, err := ddg.Navigate(url)
if err != nil {
return nil, err
}
// Get all search results in page - try multiple selectors
var searchRes *rod.SearchResult
var searchErr error
// Try different selectors for DuckDuckGo results
selectors := []string{
"article[data-testid='result']",
"div[data-testid='result']",
"div.result",
"div.web-result",
".result",
"[data-testid='result']",
}
for _, selector := range selectors {
searchRes, searchErr = page.Timeout(ddg.GetSelectorTimeout()).Search(selector)
if searchErr == nil && searchRes != nil {
ddg.logger.Debug("Found results with selector: %s", selector)
break
}
}
page.MustScreenshotFullPage("./duck.png")
if searchErr != nil {
defer page.Close()
ddg.logger.Error("Cannot parse search results: %s", searchErr)
return nil, core.ErrSearchTimeout
}
// Check why no results, maybe captcha?
if searchRes == nil {
defer page.Close()
if ddg.isNoResults(page) {
ddg.logger.Warn("No results found")
} else if ddg.isCaptcha(page) {
ddg.logger.Error("Captcha detected: %s", url)
return nil, core.ErrCaptcha
}
break
}
elements, err := searchRes.All()
if err != nil {
ddg.logger.Error("Cannot get search elements: %s", err)
break
}
r := ddg.parseResults(elements, searchPage)
if len(r) == 0 {
ddg.logger.Debug("No valid results found on page %d", searchPage)
return nil, core.ErrSearchTimeout
}
allResults = append(allResults, r...)
searchPage++
if !ddg.Browser.LeavePageOpen {
// Close tab before opening new one during the cycle
err = page.Close()
if err != nil {
ddg.logger.Debug("Page close error: %v", err)
}
}
time.Sleep(ddg.pageSleep)
}
ddg.logger.Info("Search completed: %d results", len(allResults))
return core.DeduplicateResults(allResults), nil
}
func (ddg *DuckDuckGo) SearchImage(query core.Query) ([]core.SearchResult, error) {
ddg.logger.Debug("Starting image search, query: %+v", query)
searchResults := []core.SearchResult{}
url, err := BuildImageURL(query)
if err != nil {
return nil, err
}
page, err := ddg.Navigate(url)
if err != nil {
return nil, err
}
if !ddg.Browser.LeavePageOpen {
defer page.Close()
}
// Wait for page to load
page.WaitLoad()
time.Sleep(time.Second * 2) // Give time for images to load
// Try multiple selectors for DuckDuckGo image results
var searchRes *rod.SearchResult
var searchErr error
selectors := []string{
"figure",
// "figure.nsogf_Hpj9UUxfhcwQd5",
// "div[data-testid='result']",
// "div.tile--img",
// "div.tile.tile--img",
// "div.js-images-show-more",
// "div.img-result",
}
ddg.logger.Debug("Trying selectors: %v", selectors)
for _, selector := range selectors {
searchRes, searchErr = page.Timeout(ddg.GetSelectorTimeout()).Search(selector)
if searchErr == nil && searchRes != nil {
ddg.logger.Debug("Found image results with selector: %s", selector)
break
} else {
ddg.logger.Debug("Selector '%s' not found: %v", selector, searchErr)
}
}
if searchErr != nil {
ddg.logger.Error("Cannot find image results: %s", searchErr)
return searchResults, core.ErrSearchTimeout
}
// Check why no results
if searchRes == nil {
if ddg.isCaptcha(page) {
ddg.logger.Error("Captcha detected: %s", url)
return searchResults, core.ErrCaptcha
} else if ddg.isNoResults(page) {
ddg.logger.Warn("No image results found")
}
return searchResults, core.ErrSearchTimeout
}
elements, err := searchRes.All()
if err != nil {
ddg.logger.Error("Cannot get search elements: %s", err)
return searchResults, err
}
ddg.logger.Info("Found %d image elements", len(elements))
for i, r := range elements {
// Get image URL - try multiple selectors
var imgTag *rod.Element
var imgErr error
imgSelectors := []string{
"img",
"div.SZ76bwIlqO8BBoqOLqYV img",
"img[src*='duckduckgo.com']",
}
for _, selector := range imgSelectors {
imgTag, imgErr = r.Element(selector)
if imgErr == nil {
break
}
}
if imgErr != nil {
ddg.logger.Debug("Missing img tag for element %d", i)
continue
}
imgSrc, err := imgTag.Property("src")
if err != nil {
ddg.logger.Debug("Missing src property for image %d", i)
continue
}
// Get title - try multiple selectors based on the HTML structure
var titleTag *rod.Element
var titleErr error
titleSelectors := []string{
"figcaption a p span",
"figcaption span",
"figcaption p span",
"span.EKtkFWMYpwzMKOYr0GYm",
"h3",
"span",
"p",
}
for _, selector := range titleSelectors {
titleTag, titleErr = r.Element(selector)
if titleErr == nil {
break
}
}
title := "No title"
if titleTag != nil {
title, _ = titleTag.Text()
}
// Get source page URL - try multiple selectors
var linkTag *rod.Element
var linkErr error
linkSelectors := []string{
"figcaption a",
"a",
}
for _, selector := range linkSelectors {
linkTag, linkErr = r.Element(selector)
if linkErr == nil {
break
}
}
sourceURL := ""
if linkTag != nil {
href, err := linkTag.Property("href")
if err == nil {
sourceURL = href.String()
}
}
result := core.SearchResult{
Rank: i + 1,
URL: imgSrc.String(),
Title: title,
Description: fmt.Sprintf("Source: %s", sourceURL),
}
searchResults = append(searchResults, result)
}
ddg.logger.Info("Parsed %d image results", len(searchResults))
return core.DeduplicateResults(searchResults), nil
}

103
duckduckgo/search_test.go Normal file
View File

@@ -0,0 +1,103 @@
package duckduckgo
import (
"testing"
"github.com/karust/openserp/core"
)
func TestBuildURL(t *testing.T) {
tests := []struct {
name string
query core.Query
expected string
wantErr bool
}{
{
name: "Basic search",
query: core.Query{
Text: "golang programming",
},
expected: "https://duckduckgo.com/?q=golang+programming&t=h&ia=web",
wantErr: false,
},
{
name: "Search with site filter",
query: core.Query{
Text: "golang",
Site: "github.com",
},
expected: "https://duckduckgo.com/?q=golang+site%3Agithub.com&t=h&ia=web",
wantErr: false,
},
{
name: "Search with filetype",
query: core.Query{
Text: "documentation",
Filetype: "pdf",
},
expected: "https://duckduckgo.com/?q=documentation+filetype%3Apdf&t=h&ia=web",
wantErr: false,
},
{
name: "Empty query",
query: core.Query{
Text: "",
},
expected: "",
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := BuildURL(tt.query)
if (err != nil) != tt.wantErr {
t.Errorf("BuildURL() error = %v, wantErr %v", err, tt.wantErr)
return
}
if got != tt.expected {
t.Errorf("BuildURL() = %v, want %v", got, tt.expected)
}
})
}
}
func TestBuildImageURL(t *testing.T) {
tests := []struct {
name string
query core.Query
expected string
wantErr bool
}{
{
name: "Basic image search",
query: core.Query{
Text: "golang logo",
},
expected: "https://duckduckgo.com/?q=golang+logo&t=h&iax=images&ia=images",
wantErr: false,
},
{
name: "Empty query",
query: core.Query{
Text: "",
},
expected: "",
wantErr: true,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := BuildImageURL(tt.query)
if (err != nil) != tt.wantErr {
t.Errorf("BuildImageURL() error = %v, wantErr %v", err, tt.wantErr)
return
}
if got != tt.expected {
t.Errorf("BuildImageURL() = %v, want %v", got, tt.expected)
}
})
}
}

99
duckduckgo/url.go Normal file
View File

@@ -0,0 +1,99 @@
package duckduckgo
import (
"errors"
"fmt"
"net/url"
"strings"
"github.com/karust/openserp/core"
)
const baseURL = "https://duckduckgo.com"
func BuildURL(q core.Query) (string, error) {
base, err := url.Parse(baseURL)
if err != nil {
return "", err
}
base.Path += ""
params := url.Values{}
// Set request text
if q.Text != "" || q.Site != "" || q.Filetype != "" {
text := q.Text
if q.Site != "" {
text += " site:" + q.Site
}
if q.Filetype != "" {
text += " filetype:" + q.Filetype
}
params.Add("q", text)
}
if len(params.Get("q")) == 0 {
return "", errors.New("empty query built")
}
// Set search date range
if q.DateInterval != "" {
intervals := strings.Split(q.DateInterval, "..")
if len(intervals) != 2 {
return "", errors.New("incorrect data interval provided")
}
// DuckDuckGo uses different date format
params.Add("df", fmt.Sprintf("%s..%s", intervals[0], intervals[1]))
}
// Set language
if q.LangCode != "" {
params.Add("kl", strings.ToLower(q.LangCode))
}
// DuckDuckGo specific parameters
params.Add("t", "h") // HTML format
params.Add("ia", "web") // Web search
base.RawQuery = params.Encode()
return base.String(), nil
}
func BuildImageURL(q core.Query) (string, error) {
base, err := url.Parse(baseURL)
if err != nil {
return "", err
}
base.Path += ""
params := url.Values{}
params.Add("t", "h") // HTML format
params.Add("iax", "images") // Image search
params.Add("ia", "images")
// Set request text
if q.Text != "" || q.Site != "" || q.Filetype != "" {
text := q.Text
if q.Site != "" {
text += " site:" + q.Site
}
if q.Filetype != "" {
text += " filetype:" + q.Filetype
}
params.Add("q", text)
}
if len(params.Get("q")) == 0 {
return "", errors.New("empty query built")
}
// Set language
if q.LangCode != "" {
params.Add("kl", strings.ToLower(q.LangCode))
}
base.RawQuery = params.Encode()
return base.String(), nil
}

75
go.mod
View File

@@ -1,57 +1,54 @@
module github.com/karust/openserp
go 1.20
go 1.24
toolchain go1.24.6
require (
github.com/2captcha/2captcha-go v1.1.4
github.com/PuerkitoBio/goquery v1.8.1
github.com/2captcha/2captcha-go v1.1.10
github.com/PuerkitoBio/goquery v1.10.3
github.com/corpix/uarand v0.2.0
github.com/go-rod/rod v0.113.3
github.com/go-rod/rod v0.116.2
github.com/go-rod/stealth v0.4.9
github.com/gofiber/fiber/v2 v2.47.0
github.com/gofiber/fiber/v2 v2.52.9
github.com/refraction-networking/utls v1.8.0
github.com/sirupsen/logrus v1.9.3
github.com/spf13/cobra v1.7.0
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.16.0
golang.org/x/time v0.3.0
github.com/spf13/cobra v1.9.1
github.com/spf13/pflag v1.0.7
github.com/spf13/viper v1.20.1
golang.org/x/time v0.12.0
)
require (
github.com/andybalholm/brotli v1.0.5 // indirect
github.com/andybalholm/cascadia v1.3.2 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/hashicorp/hcl v1.0.0 // indirect
github.com/andybalholm/brotli v1.2.0 // indirect
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/fsnotify/fsnotify v1.9.0 // indirect
github.com/go-viper/mapstructure/v2 v2.4.0 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/klauspost/compress v1.16.6 // indirect
github.com/magiconair/properties v1.8.7 // indirect
github.com/mattn/go-colorable v0.1.13 // indirect
github.com/mattn/go-isatty v0.0.19 // indirect
github.com/mattn/go-runewidth v0.0.14 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/pelletier/go-toml/v2 v2.0.8 // indirect
github.com/philhofer/fwd v1.1.2 // indirect
github.com/rivo/uniseg v0.4.4 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-runewidth v0.0.16 // indirect
github.com/pelletier/go-toml/v2 v2.2.4 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/rogpeppe/go-internal v1.10.0 // indirect
github.com/savsgio/dictpool v0.0.0-20221023140959-7bf2e61cea94 // indirect
github.com/savsgio/gotils v0.0.0-20230208104028-c358bd845dee // indirect
github.com/spf13/afero v1.9.5 // indirect
github.com/spf13/cast v1.5.1 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
github.com/subosito/gotenv v1.4.2 // indirect
github.com/tinylib/msgp v1.1.8 // indirect
github.com/sagikazarmark/locafero v0.10.0 // indirect
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 // indirect
github.com/spf13/afero v1.14.0 // indirect
github.com/spf13/cast v1.9.2 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/valyala/fasthttp v1.48.0 // indirect
github.com/valyala/tcplisten v1.0.0 // indirect
github.com/ysmood/fetchup v0.2.3 // indirect
github.com/valyala/fasthttp v1.65.0 // indirect
github.com/ysmood/fetchup v0.3.0 // indirect
github.com/ysmood/goob v0.4.0 // indirect
github.com/ysmood/got v0.34.1 // indirect
github.com/ysmood/got v0.41.0 // indirect
github.com/ysmood/gson v0.7.3 // indirect
github.com/ysmood/leakless v0.8.0 // indirect
golang.org/x/net v0.11.0 // indirect
golang.org/x/sys v0.9.0 // indirect
golang.org/x/text v0.10.0 // indirect
github.com/ysmood/leakless v0.9.0 // indirect
golang.org/x/crypto v0.41.0 // indirect
golang.org/x/net v0.43.0 // indirect
golang.org/x/sys v0.35.0 // indirect
golang.org/x/text v0.28.0 // indirect
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

628
go.sum
View File

@@ -1,591 +1,189 @@
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
cloud.google.com/go v0.38.0/go.mod h1:990N+gfupTy94rShfmMCWGDn0LpTmnzTp2qbd1dvSRU=
cloud.google.com/go v0.44.1/go.mod h1:iSa0KzasP4Uvy3f1mN/7PiObzGgflwredwwASm/v6AU=
cloud.google.com/go v0.44.2/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.44.3/go.mod h1:60680Gw3Yr4ikxnPRS/oxxkBccT6SA1yMk63TGekxKY=
cloud.google.com/go v0.45.1/go.mod h1:RpBamKRgapWJb87xiFSdk4g1CME7QZg3uwTez+TSTjc=
cloud.google.com/go v0.46.3/go.mod h1:a6bKKbmY7er1mI7TEI4lsAkts/mkhTSZK8w33B4RAg0=
cloud.google.com/go v0.50.0/go.mod h1:r9sluTvynVuxRIOHXQEHMFffphuXHOMZMycpNR5e6To=
cloud.google.com/go v0.52.0/go.mod h1:pXajvRH/6o3+F9jDHZWQ5PbGhn+o8w9qiu/CffaVdO4=
cloud.google.com/go v0.53.0/go.mod h1:fp/UouUEsRkN6ryDKNW/Upv/JBKnv6WDthjR6+vze6M=
cloud.google.com/go v0.54.0/go.mod h1:1rq2OEkV3YMf6n/9ZvGWI3GWw0VoqH/1x2nd8Is/bPc=
cloud.google.com/go v0.56.0/go.mod h1:jr7tqZxxKOVYizybht9+26Z/gUq7tiRzu+ACVAMbKVk=
cloud.google.com/go v0.57.0/go.mod h1:oXiQ6Rzq3RAkkY7N6t3TcE6jE+CIBBbA36lwQ1JyzZs=
cloud.google.com/go v0.62.0/go.mod h1:jmCYTdRCQuc1PHIIJ/maLInMho30T/Y0M4hTdTShOYc=
cloud.google.com/go v0.65.0/go.mod h1:O5N8zS7uWy9vkA9vayVHs65eM1ubvY4h553ofrNHObY=
cloud.google.com/go v0.72.0/go.mod h1:M+5Vjvlc2wnp6tjzE102Dw08nGShTscUx2nZMufOKPI=
cloud.google.com/go v0.74.0/go.mod h1:VV1xSbzvo+9QJOxLDaJfTjx5e+MePCpCWwvftOeQmWk=
cloud.google.com/go v0.75.0/go.mod h1:VGuuCn7PG0dwsd5XPVm2Mm3wlh3EL55/79EKB6hlPTY=
cloud.google.com/go/bigquery v1.0.1/go.mod h1:i/xbL2UlR5RvWAURpBYZTtm/cXjCha9lbfbpx4poX+o=
cloud.google.com/go/bigquery v1.3.0/go.mod h1:PjpwJnslEMmckchkHFfq+HTD2DmtT67aNFKH1/VBDHE=
cloud.google.com/go/bigquery v1.4.0/go.mod h1:S8dzgnTigyfTmLBfrtrhyYhwRxG72rYxvftPBK2Dvzc=
cloud.google.com/go/bigquery v1.5.0/go.mod h1:snEHRnqQbz117VIFhE8bmtwIDY80NLUZUMb4Nv6dBIg=
cloud.google.com/go/bigquery v1.7.0/go.mod h1://okPTzCYNXSlb24MZs83e2Do+h+VXtc4gLoIoXIAPc=
cloud.google.com/go/bigquery v1.8.0/go.mod h1:J5hqkt3O0uAFnINi6JXValWIb1v0goeZM77hZzJN/fQ=
cloud.google.com/go/datastore v1.0.0/go.mod h1:LXYbyblFSglQ5pkeyhO+Qmw7ukd3C+pD7TKLgZqpHYE=
cloud.google.com/go/datastore v1.1.0/go.mod h1:umbIZjpQpHh4hmRpGhH4tLFup+FVzqBi1b3c64qFpCk=
cloud.google.com/go/pubsub v1.0.1/go.mod h1:R0Gpsv3s54REJCy4fxDixWD93lHJMoZTyQ2kNxGRt3I=
cloud.google.com/go/pubsub v1.1.0/go.mod h1:EwwdRX2sKPjnvnqCa270oGRyludottCI76h+R3AArQw=
cloud.google.com/go/pubsub v1.2.0/go.mod h1:jhfEVHT8odbXTkndysNHCcx0awwzvfOlguIAii9o8iA=
cloud.google.com/go/pubsub v1.3.1/go.mod h1:i+ucay31+CNRpDW4Lu78I4xXG+O1r/MAHgjpRVR+TSU=
cloud.google.com/go/storage v1.0.0/go.mod h1:IhtSnM/ZTZV8YYJWCY8RULGVqBDmpoyjwiyrjsg+URw=
cloud.google.com/go/storage v1.5.0/go.mod h1:tpKbwo567HUNpVclU5sGELwQWBDZ8gh0ZeosJ0Rtdos=
cloud.google.com/go/storage v1.6.0/go.mod h1:N7U0C8pVQ/+NIKOBQyamJIeKQKkZ+mxpohlUTyfDhBk=
cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RXyy7KQOVs=
cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
cloud.google.com/go/storage v1.14.0/go.mod h1:GrKmX003DSIwi9o29oFT7YDnHYwZoctc3fOKtUw0Xmo=
dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
github.com/2captcha/2captcha-go v1.1.4 h1:Fm62VPvVhEHYQ8AI+/uquiTg41ml9f8ASjUkVuBvHcE=
github.com/2captcha/2captcha-go v1.1.4/go.mod h1:hYOq+KVOq/0zAG6OTYW7Y313qDkHv58CcaOyjdBQSco=
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
github.com/PuerkitoBio/goquery v1.8.1 h1:uQxhNlArOIdbrH1tr0UXwdVFgDcZDrZVdcpygAcwmWM=
github.com/PuerkitoBio/goquery v1.8.1/go.mod h1:Q8ICL1kNUJ2sXGoAhPGUdYDJvgQgHzJsnnd3H7Ho5jQ=
github.com/andybalholm/brotli v1.0.5 h1:8uQZIdzKmjc/iuPu7O2ioW48L81FgatrcpfFmiq/cCs=
github.com/andybalholm/brotli v1.0.5/go.mod h1:fO7iG3H7G2nSZ7m0zPUDn85XEX2GTukHGRSepvi9Eig=
github.com/andybalholm/cascadia v1.3.1/go.mod h1:R4bJ1UQfqADjvDa4P6HZHLh/3OxWWEqc0Sk8XGwHqvA=
github.com/andybalholm/cascadia v1.3.2 h1:3Xi6Dw5lHF15JtdcmAHD3i1+T8plmv7BQ/nsViSLyss=
github.com/andybalholm/cascadia v1.3.2/go.mod h1:7gtRlve5FxPPgIgX36uWBX58OdBsSS6lUvCFb+h7KvU=
github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
github.com/chzyer/logex v1.1.10/go.mod h1:+Ywpsq7O8HXn0nuIou7OrIPyXbp3wmkHB+jjWRnGsAI=
github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5PlCu98SY8svDHJxuZscDgtXS6KTTbou5AhLI=
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/udpa/go v0.0.0-20200629203442-efcf912fb354/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/2captcha/2captcha-go v1.1.10 h1:U3Y7VLwR9z5XpCMijB+FkhHRt6QikKnHDEy1uLQVCD8=
github.com/2captcha/2captcha-go v1.1.10/go.mod h1:TsupeToBP0BPHfZOQpNDb61NKqtbBJ2ddptqSK2p9/M=
github.com/PuerkitoBio/goquery v1.10.3 h1:pFYcNSqHxBD06Fpj/KsbStFRsgRATgnf3LeXiUkhzPo=
github.com/PuerkitoBio/goquery v1.10.3/go.mod h1:tMUX0zDMHXYlAQk6p35XxQMqMweEKB7iK7iLNd4RH4Y=
github.com/andybalholm/brotli v1.2.0 h1:ukwgCxwYrmACq68yiUqwIWnGY0cTPox/M94sVwToPjQ=
github.com/andybalholm/brotli v1.2.0/go.mod h1:rzTDkvFWvIrjDXZHkuS16NPggd91W3kUSvPlQ1pLaKY=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/corpix/uarand v0.2.0 h1:U98xXwud/AVuCpkpgfPF7J5TQgr7R5tqT8VZP5KWbzE=
github.com/corpix/uarand v0.2.0/go.mod h1:/3Z1QIqWkDIhf6XWn/08/uMHoQ8JUoTIKc2iPchBOmM=
github.com/cpuguy83/go-md2man/v2 v2.0.2/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.9.7/go.mod h1:cwu0lG7PUMfa9snN8LXBig5ynNVH9qI8YYLbd1fK2po=
github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
github.com/frankban/quicktest v1.14.4 h1:g2rn0vABPOOXmZUj+vbmUp0lPoXEMuhTpIluN0XL9UY=
github.com/fsnotify/fsnotify v1.6.0 h1:n+5WquG0fcWoWp6xPWfHdbskMCQaFnG6PfBrh1Ky4HY=
github.com/fsnotify/fsnotify v1.6.0/go.mod h1:sl3t1tCWJFWoRz9R8WJCbQihKKwmorjAbSClcnxKAGw=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/frankban/quicktest v1.14.6 h1:7Xjx+VpznH+oBnejlPUj8oUpdxnVs4f8XU8WnHkI4W8=
github.com/frankban/quicktest v1.14.6/go.mod h1:4ptaffx2x8+WTWXmUCuVU6aPUX1/Mz7zb5vbUoiM6w0=
github.com/fsnotify/fsnotify v1.9.0 h1:2Ml+OJNzbYCTzsxtv8vKSFD9PbJjmhYF14k/jKC7S9k=
github.com/fsnotify/fsnotify v1.9.0/go.mod h1:8jBTzvmWwFyi3Pb8djgCCO5IBqzKJ/Jwo8TRcHyHii0=
github.com/go-rod/rod v0.113.0/go.mod h1:aiedSEFg5DwG/fnNbUOTPMTTWX3MRj6vIs/a684Mthw=
github.com/go-rod/rod v0.113.3 h1:oLiKZW721CCMwA5g7977cWfcAKQ+FuosP47Zf1QiDrA=
github.com/go-rod/rod v0.113.3/go.mod h1:aiedSEFg5DwG/fnNbUOTPMTTWX3MRj6vIs/a684Mthw=
github.com/go-rod/rod v0.116.2 h1:A5t2Ky2A+5eD/ZJQr1EfsQSe5rms5Xof/qj296e+ZqA=
github.com/go-rod/rod v0.116.2/go.mod h1:H+CMO9SCNc2TJ2WfrG+pKhITz57uGNYU43qYHh438Mg=
github.com/go-rod/stealth v0.4.9 h1:X2PmQk4DUF2wzw6GOsWjW/glb8K5ebnftbEvLh7MlZ4=
github.com/go-rod/stealth v0.4.9/go.mod h1:eAzyvw8c0iAd5nJJsSWeh0fQ5z94vCIfdi1hUmYDimc=
github.com/gofiber/fiber/v2 v2.47.0 h1:EN5lHVCc+Pyqh5OEsk8fzRiifgwpbrP0rulQ4iNf3fs=
github.com/gofiber/fiber/v2 v2.47.0/go.mod h1:mbFMVN1lQuzziTkkakgtKKdjfsXSw9BKR5lmcNksUoU=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20191227052852-215e87163ea7/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
github.com/golang/mock v1.3.1/go.mod h1:sBzyDLLjw3U8JLTeZvSv8jJB+tU5PVekmnlKIyFUx0Y=
github.com/golang/mock v1.4.0/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.1/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.3/go.mod h1:UOMv5ysSaYNkG+OFQykRIcU/QvvxJf3p21QfJ2Bt3cw=
github.com/golang/mock v1.4.4/go.mod h1:l3mdAwkq5BuhzHwde/uurv3sEJeZMXNpwsxVWU71h+4=
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.4/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw=
github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/golang/protobuf v1.4.3/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.4.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.1/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.2/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
github.com/google/martian/v3 v3.1.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20190515194954-54271f7e092f/go.mod h1:zfwlbNMJ+OItoe0UupaVj+oy1omPYYDuagoSzA8v9mc=
github.com/google/pprof v0.0.0-20191218002539-d4f498aebedc/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200708004538-1a94d8640e99/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20201023163331-3e6fc7fc9c4c/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20201203190320-1bf35d6f28c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/pprof v0.0.0-20201218002935-b9804c9f04c2/go.mod h1:kpwsk12EmLew5upagYY7GY0pfYCcupk39gWOCRROcvE=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.3.0 h1:t6JiXgmwXMjEs8VusXIJk2BXHsn+wx8BZdTaoZ5fu7I=
github.com/google/uuid v1.3.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/googleapis/gax-go/v2 v2.0.4/go.mod h1:0Wqv26UfaUD9n4G6kQubkQ+KchISgw+vpHVxEJEs9eg=
github.com/googleapis/gax-go/v2 v2.0.5/go.mod h1:DWXyrwAJ9X0FpwwEdw+IPEYBICEFu5mhpdKc/us6bOk=
github.com/googleapis/google-cloud-go-testing v0.0.0-20200911160855-bcd43fbb19e8/go.mod h1:dvDLG8qkwmyD9a/MJJN3XJcT3xFxOKAvTZGvuZmac9g=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/ianlancetaylor/demangle v0.0.0-20181102032728-5e5cf60278f6/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/ianlancetaylor/demangle v0.0.0-20200824232613-28f6c0f3b639/go.mod h1:aSSvb/t6k1mPoxDqO4vJh6VOCGPwU4O0C2/Eqndh1Sc=
github.com/go-viper/mapstructure/v2 v2.4.0 h1:EBsztssimR/CONLSZZ04E8qAkxNYq4Qp9LvH92wZUgs=
github.com/go-viper/mapstructure/v2 v2.4.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
github.com/gofiber/fiber/v2 v2.52.9 h1:YjKl5DOiyP3j0mO61u3NTmK7or8GzzWzCFzkboyP5cw=
github.com/gofiber/fiber/v2 v2.52.9/go.mod h1:YEcBbO/FB+5M1IZNBP9FO3J9281zgPAreiI1oqg8nDw=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
github.com/jstemmer/go-junit-report v0.9.1/go.mod h1:Brl9GWCQeLvo8nXZwPNNblvFj/XSXhF0NWZEnDohbsk=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.16.6 h1:91SKEy4K37vkp255cJ8QesJhjyRO0hn9i9G0GoUwLsk=
github.com/klauspost/compress v1.16.6/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/kr/fs v0.1.0/go.mod h1:FFnZGqtBN9Gxj7eW1uZ42v5BccTP0vu6NEaFoC2HwRg=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/kr/pretty v0.2.1/go.mod h1:ipq/a2n7PKx3OHsz4KJII5eveXtPO4qwEXGdVfWzfnI=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0VQdvPDY=
github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
github.com/mattn/go-colorable v0.1.13 h1:fFA4WZxdEF4tXPZVKMLwD8oUnCTTo08duU7wxecdEvA=
github.com/mattn/go-colorable v0.1.13/go.mod h1:7S9/ev0klgBDR4GtXTXX8a3vIGJpMovkB8vQcUbaXHg=
github.com/mattn/go-isatty v0.0.16/go.mod h1:kYGgaQfpe5nmfYZH+SKPsOc2e4SrIfOl2e/yFXSvRLM=
github.com/mattn/go-isatty v0.0.19 h1:JITubQf0MOLdlGRuRq+jtsDlekdYPia9ZFsB8h/APPA=
github.com/mattn/go-isatty v0.0.19/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-runewidth v0.0.14 h1:+xnbZSEeDbOIg5/mE6JF0w6n9duR1l3/WmbinWVwUuU=
github.com/mattn/go-runewidth v0.0.14/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/pelletier/go-toml/v2 v2.0.8 h1:0ctb6s9mE31h0/lhu+J6OPmVeDxJn+kYnJc2jZR9tGQ=
github.com/pelletier/go-toml/v2 v2.0.8/go.mod h1:vuYfssBdrU2XDZ9bYydBu6t+6a6PYNcZljzZR9VXg+4=
github.com/philhofer/fwd v1.1.1/go.mod h1:gk3iGcWd9+svBvR0sR+KPcfE+RNWozjowpeBVG3ZVNU=
github.com/philhofer/fwd v1.1.2 h1:bnDivRJ1EWPjUIRXV5KfORO897HTbpFAQddBdE8t7Gw=
github.com/philhofer/fwd v1.1.2/go.mod h1:qkPdfjR2SIEbspLqpe1tO4n5yICnr2DY7mqEx2tUTP0=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/sftp v1.13.1/go.mod h1:3HaPG6Dq1ILlpPZRO0HVMrsydcdLt6HRDccSgb87qRg=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-runewidth v0.0.16 h1:E5ScNMtiwvlvB5paMFdw9p4kSQzbXFikJ5SQO6TULQc=
github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
github.com/pelletier/go-toml/v2 v2.2.4 h1:mye9XuhQ6gvn5h28+VilKrrPoQVanw5PMw/TB0t5Ec4=
github.com/pelletier/go-toml/v2 v2.2.4/go.mod h1:2gIqNv+qfxSVS7cM2xJQKtLSTLUE9V8t9Stt+h56mCY=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
github.com/refraction-networking/utls v1.8.0 h1:L38krhiTAyj9EeiQQa2sg+hYb4qwLCqdMcpZrRfbONE=
github.com/refraction-networking/utls v1.8.0/go.mod h1:jkSOEkLqn+S/jtpEHPOsVv/4V4EVnelwbMQl4vCWXAM=
github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
github.com/rivo/uniseg v0.4.4 h1:8TfxU8dW6PdqD27gjM8MVNuicgxIjxpm4K7x4jp8sis=
github.com/rivo/uniseg v0.4.4/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/savsgio/dictpool v0.0.0-20221023140959-7bf2e61cea94 h1:rmMl4fXJhKMNWl+K+r/fq4FbbKI+Ia2m9hYBLm2h4G4=
github.com/savsgio/dictpool v0.0.0-20221023140959-7bf2e61cea94/go.mod h1:90zrgN3D/WJsDd1iXHT96alCoN2KJo6/4x1DZC3wZs8=
github.com/savsgio/gotils v0.0.0-20220530130905-52f3993e8d6d/go.mod h1:Gy+0tqhJvgGlqnTF8CVGP0AaGRjwBtXs/a5PA0Y3+A4=
github.com/savsgio/gotils v0.0.0-20230208104028-c358bd845dee h1:8Iv5m6xEo1NR1AvpV+7XmhI4r39LGNzwUL4YpMuL5vk=
github.com/savsgio/gotils v0.0.0-20230208104028-c358bd845dee/go.mod h1:qwtSXrKuJh/zsFQ12yEE89xfCrGKK63Rr7ctU/uCo4g=
github.com/sagikazarmark/locafero v0.10.0 h1:FM8Cv6j2KqIhM2ZK7HZjm4mpj9NBktLgowT1aN9q5Cc=
github.com/sagikazarmark/locafero v0.10.0/go.mod h1:Ieo3EUsjifvQu4NZwV5sPd4dwvu0OCgEQV7vjc9yDjw=
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
github.com/spf13/afero v1.9.5 h1:stMpOSZFs//0Lv29HduCmli3GUfpFoF3Y1Q/aXj/wVM=
github.com/spf13/afero v1.9.5/go.mod h1:UBogFpq8E9Hx+xc5CNTTEpTnuHVmXDwZcZcE1eb/UhQ=
github.com/spf13/cast v1.5.1 h1:R+kOtfhWQE6TVQzY+4D7wJLBgkdVasCEFxSUBYBYIlA=
github.com/spf13/cast v1.5.1/go.mod h1:b9PdjNptOpzXr7Rq1q9gJML/2cdGQAo69NKzQ10KN48=
github.com/spf13/cobra v1.7.0 h1:hyqWnYt1ZQShIddO5kBpj3vu05/++x6tJ6dg8EC572I=
github.com/spf13/cobra v1.7.0/go.mod h1:uLxZILRyS/50WlhOIKD7W6V5bgeIt+4sICxh6uRMrb0=
github.com/spf13/jwalterweatherman v1.1.0 h1:ue6voC5bR5F8YxI5S67j9i582FU4Qvo2bmqnqMYADFk=
github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo=
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.16.0 h1:rGGH0XDZhdUOryiDWjmIvUSWpbNqisK8Wk0Vyefw8hc=
github.com/spf13/viper v1.16.0/go.mod h1:yg78JgCJcbrQOvV9YLXgkLaZqUidkY9K+Dd1FofRzQg=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8 h1:+jumHNA0Wrelhe64i8F6HNlS8pkoyMv5sreGx2Ry5Rw=
github.com/sourcegraph/conc v0.3.1-0.20240121214520-5f936abd7ae8/go.mod h1:3n1Cwaq1E1/1lhQhtRK2ts/ZwZEhjcQeJQ1RuC6Q/8U=
github.com/spf13/afero v1.14.0 h1:9tH6MapGnn/j0eb0yIXiLjERO8RB6xIVZRDCX7PtqWA=
github.com/spf13/afero v1.14.0/go.mod h1:acJQ8t0ohCGuMN3O+Pv0V0hgMxNYDlvdk+VTfyZmbYo=
github.com/spf13/cast v1.9.2 h1:SsGfm7M8QOFtEzumm7UZrZdLLquNdzFYfIbEXntcFbE=
github.com/spf13/cast v1.9.2/go.mod h1:jNfB8QC9IA6ZuY2ZjDp0KtFO2LZZlg4S/7bzP6qqeHo=
github.com/spf13/cobra v1.9.1 h1:CXSaggrXdbHK9CF+8ywj8Amf7PBRmPCOJugH954Nnlo=
github.com/spf13/cobra v1.9.1/go.mod h1:nDyEzZ8ogv936Cinf6g1RU9MRY64Ir93oCnqb9wxYW0=
github.com/spf13/pflag v1.0.6/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/pflag v1.0.7 h1:vN6T9TfwStFPFM5XzjsvmzZkLuaLX+HS+0SeFLRgU6M=
github.com/spf13/pflag v1.0.7/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/spf13/viper v1.20.1 h1:ZMi+z/lvLyPSCoNtFCpqjy0S4kPbirhpTMwl8BkW9X4=
github.com/spf13/viper v1.20.1/go.mod h1:P9Mdzt1zoHIG8m2eZQinpiBjo6kCmZSKBClNNqjJvu4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.3 h1:RP3t2pwF7cMEbC1dqtB6poj3niw/9gnV4Cjg5oW5gtY=
github.com/stretchr/testify v1.8.3/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/subosito/gotenv v1.4.2 h1:X1TuBLAMDFbaTAChgCBLu3DU3UPyELpnF2jjJ2cz/S8=
github.com/subosito/gotenv v1.4.2/go.mod h1:ayKnFf/c6rvx/2iiLrJUk1e6plDbT3edrFNGqEflhK0=
github.com/tinylib/msgp v1.1.6/go.mod h1:75BAfg2hauQhs3qedfdDZmWAPcFMAvJE5b9rGOMufyw=
github.com/tinylib/msgp v1.1.8 h1:FCXC1xanKO4I8plpHGH2P7koL/RzZs12l/+r7vakfm0=
github.com/tinylib/msgp v1.1.8/go.mod h1:qkpG+2ldGg4xRFmx+jfTvZPxfGFhi64BcnL9vkCm/Tw=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/subosito/gotenv v1.6.0 h1:9NlTDc1FTs4qu0DDq7AEtTPNw6SVm7uBMsUCUjABIf8=
github.com/subosito/gotenv v1.6.0/go.mod h1:Dk4QP5c2W3ibzajGcXpNraDfq2IrhjMIvMSWPKKo0FU=
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
github.com/valyala/fasthttp v1.48.0 h1:oJWvHb9BIZToTQS3MuQ2R3bJZiNSa2KiNdeI8A+79Tc=
github.com/valyala/fasthttp v1.48.0/go.mod h1:k2zXd82h/7UZc3VOdJ2WaUqt1uZ/XpXAfE9i+HBC3lA=
github.com/valyala/tcplisten v1.0.0 h1:rBHj/Xf+E1tRGZyWIWwJDiRY0zc1Js+CV5DqwacVSA8=
github.com/valyala/tcplisten v1.0.0/go.mod h1:T0xQ8SeCZGxckz9qRXTfG43PvQ/mcWh7FwZEA7Ioqkc=
github.com/ysmood/fetchup v0.2.3 h1:ulX+SonA0Vma5zUFXtv52Kzip/xe7aj4vqT5AJwQ+ZQ=
github.com/valyala/fasthttp v1.65.0 h1:j/u3uzFEGFfRxw79iYzJN+TteTJwbYkru9uDp3d0Yf8=
github.com/valyala/fasthttp v1.65.0/go.mod h1:P/93/YkKPMsKSnATEeELUCkG8a7Y+k99uxNHVbKINr4=
github.com/xyproto/randomstring v1.0.5 h1:YtlWPoRdgMu3NZtP45drfy1GKoojuR7hmRcnhZqKjWU=
github.com/xyproto/randomstring v1.0.5/go.mod h1:rgmS5DeNXLivK7YprL0pY+lTuhNQW3iGxZ18UQApw/E=
github.com/ysmood/fetchup v0.2.3/go.mod h1:xhibcRKziSvol0H1/pj33dnKrYyI2ebIvz5cOOkYGns=
github.com/ysmood/fetchup v0.3.0 h1:UhYz9xnLEVn2ukSuK3KCgcznWpHMdrmbsPpllcylyu8=
github.com/ysmood/fetchup v0.3.0/go.mod h1:hbysoq65PXL0NQeNzUczNYIKpwpkwFL4LXMDEvIQq9A=
github.com/ysmood/goob v0.4.0 h1:HsxXhyLBeGzWXnqVKtmT9qM7EuVs/XOgkX7T6r1o1AQ=
github.com/ysmood/goob v0.4.0/go.mod h1:u6yx7ZhS4Exf2MwciFr6nIM8knHQIE22lFpWHnfql18=
github.com/ysmood/gop v0.0.2 h1:VuWweTmXK+zedLqYufJdh3PlxDNBOfFHjIZlPT2T5nw=
github.com/ysmood/gop v0.0.2/go.mod h1:rr5z2z27oGEbyB787hpEcx4ab8cCiPnKxn0SUHt6xzk=
github.com/ysmood/got v0.34.1 h1:IrV2uWLs45VXNvZqhJ6g2nIhY+pgIG1CUoOcqfXFl1s=
github.com/ysmood/gop v0.2.0 h1:+tFrG0TWPxT6p9ZaZs+VY+opCvHU8/3Fk6BaNv6kqKg=
github.com/ysmood/gop v0.2.0/go.mod h1:rr5z2z27oGEbyB787hpEcx4ab8cCiPnKxn0SUHt6xzk=
github.com/ysmood/got v0.34.1/go.mod h1:yddyjq/PmAf08RMLSwDjPyCvHvYed+WjHnQxpH851LM=
github.com/ysmood/got v0.41.0 h1:XiFH311ltTSGyxjeKcNvy7dzbJjjTzn6DBgK313JHBs=
github.com/ysmood/got v0.41.0/go.mod h1:W7DdpuX6skL3NszLmAsC5hT7JAhuLZhByVzHTq874Qg=
github.com/ysmood/gotrace v0.6.0 h1:SyI1d4jclswLhg7SWTL6os3L1WOKeNn/ZtzVQF8QmdY=
github.com/ysmood/gotrace v0.6.0/go.mod h1:TzhIG7nHDry5//eYZDYcTzuJLYQIkykJzCRIo4/dzQM=
github.com/ysmood/gson v0.7.3 h1:QFkWbTH8MxyUTKPkVWAENJhxqdBa4lYTQWqZCiLG6kE=
github.com/ysmood/gson v0.7.3/go.mod h1:3Kzs5zDl21g5F/BlLTNcuAGAYLKt2lV5G8D1zF3RNmg=
github.com/ysmood/leakless v0.8.0 h1:BzLrVoiwxikpgEQR0Lk8NyBN5Cit2b1z+u0mgL4ZJak=
github.com/ysmood/leakless v0.8.0/go.mod h1:R8iAXPRaG97QJwqxs74RdwzcRHT1SWCGTNqY8q0JvMQ=
github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/ysmood/leakless v0.9.0 h1:qxCG5VirSBvmi3uynXFkcnLMzkphdh3xx5FtrORwDCU=
github.com/ysmood/leakless v0.9.0/go.mod h1:R8iAXPRaG97QJwqxs74RdwzcRHT1SWCGTNqY8q0JvMQ=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.3/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.4/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
go.opencensus.io v0.22.5/go.mod h1:5pWMHQbX5EPX2/62yrJeAkowc+lfs/XD7Uxpq3pI6kk=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220722155217-630584e8d5aa/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=
golang.org/x/exp v0.0.0-20191129062945-2f5052295587/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20191227195350-da58074b4299/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200119233911-0405dc783f0a/go.mod h1:2RIsYlXP63K8oxa1u096TMicItID8zy7Y6sNkU49FU4=
golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM=
golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU=
golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190409202823-959b441ac422/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190909230951-414d861bb4ac/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
golang.org/x/lint v0.0.0-20191125180803-fdd1cda4f05f/go.mod h1:5qLYkcX4OjUUV8bRuDixDT3tpyyb+LUpUlRWLxfhWrs=
golang.org/x/lint v0.0.0-20200130185559-910be7a94367/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY=
golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE=
golang.org/x/mobile v0.0.0-20190719004257-d2bd2a29d028/go.mod h1:E/iHnbuqvinMTCcRqshq8CkpyQDoeVncDDYHnLhea+o=
golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
golang.org/x/mod v0.1.0/go.mod h1:0QHyrYULN0/3qlju5TqG8bIK38QM8yzMo5ekMj3DlcY=
golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.1.1-0.20191107180719-034126e5016b/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.7.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190108225652-1e06a53dbb7e/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190501004415-9ce7a6920f09/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190503192946-f4e77d36d62c/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190628185345-da137c7871d7/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20190724013045-ca1201d0de80/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200114155413-6afb5195e5aa/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200222125558-5a598a2470a0/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200301022130-244492dfa37a/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20200324143707-d3edc9973b7e/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200501053045-e0ff5e5a1de5/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200506145744-7e3656a0809f/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200513185701-a91f0712d120/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200520182314-0ba52f642ac2/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
golang.org/x/net v0.0.0-20200625001655-4c5254603344/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200707034311-ab3426394381/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20200822124328-c89045814202/go.mod h1:/O7V0waA8r7cgGh81Ro3o1hOxt32SMVPicZroKQ2sZA=
golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201031054903-ff519b6c9102/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
golang.org/x/net v0.0.0-20201209123823-ac852fbbde11/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210916014120-12bc252f5db8/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.3.0/go.mod h1:MBQ8lrhLObU/6UmLb4fmbmk5OcyYmqtbGd/9yIeKjEE=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.7.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.9.0/go.mod h1:d48xBJpPfHeWQsugry2m+kC02ZBRGRgulfHnEXEuWns=
golang.org/x/net v0.11.0 h1:Gi2tvZIJyBtO9SDr1q9h5hEQCp/4L2RQ+ar0qjx2oNU=
golang.org/x/net v0.11.0/go.mod h1:2L/ixqYpgIVXmeoSA/4Lu7BzTG4KIyPIryS4IsOd1oQ=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20200902213428-5d25da1a8d43/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20201109201403-9fd604954f58/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20201208152858-08078c50e5b5/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/oauth2 v0.0.0-20210218202405-ba52d332ba99/go.mod h1:KelEdhl1UZF7XfJ4dDtk6s++YSgaE7mD/BuKKDLBl4A=
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200317015054-43a5402ce75a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20200625203802-6e8e738ad208/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190507160741-ecd444e8653b/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190606165138-5da285871e9c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190624142023-c5567b49c5d0/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20190726091711-fc99dfbffb4e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191001151750-bb3f8db39f24/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20191228213918-04cbcbbfeed8/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200113162924-86b910548bc1/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200122134326-e047566fdf82/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200212091648-12a6c2dcc1e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200302150141-5c8b2ff67527/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200331124033-c3d80250170d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200501052902-10377860bb8e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200511232937-7e40ca221e25/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200515095857-1151b9dac4a9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200523222454-059865788121/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200803210538-64077c9b5642/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200905004654-be1d3432aa8f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20201201145000-ef89a241ccb3/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210104204734-6f8348627aad/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210225134936-a50acf3fe073/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220908164124-27713097b956/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.3.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.7.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.9.0 h1:KS/R3tvhPqvJvwcKfnBHJwwthS11LRhmM5D59eEXa0s=
golang.org/x/sys v0.9.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.3.0/go.mod h1:q750SLmJuPmVoN1blW3UFBPREJfb1KmY3vwxfr+nFDA=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.7.0/go.mod h1:P32HKFT3hSsZrRxla30E9HqToFYAQPCMs/zFMBUFqPY=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.5.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.10.0 h1:UpjohKhiEgNc0CSauXmwYftY1+LlaC75SJwh0SgCX58=
golang.org/x/text v0.10.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
golang.org/x/time v0.12.0 h1:ScB/8o8olJvc+CQPWrK3fPZNfh7qgwCrY0zJmoEQLSE=
golang.org/x/time v0.12.0/go.mod h1:CDIdPxbZBQxdj6cxyCIdrNogrJKMJ7pr37NYpMcMDSg=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
golang.org/x/tools v0.0.0-20190425150028-36563e24a262/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190506145303-2d16b83fe98c/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
golang.org/x/tools v0.0.0-20190606124116-d0a3d012864b/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190628153133-6cdbf07be9d0/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
golang.org/x/tools v0.0.0-20190816200558-6889da9d5479/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20190911174233-4f2ddba30aff/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191012152004-8de300cfc20a/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191113191852-77e3bb0ad9e7/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191115202509-3a792d9c32b2/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191125144606-a911d9008d1f/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191216173652-a0e659d51361/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20191227053925-7b8e75db28f4/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200117161641-43d50277825c/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200122220014-bf1340f18c4a/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200204074204-1cc6d1ef6c74/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200207183749-b753a1ba74fa/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200212150539-ea181f53ac56/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200224181240-023911ca70b2/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200227222343-706bc42d1f0d/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200304193943-95d2e580d8eb/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw=
golang.org/x/tools v0.0.0-20200312045724-11d5b4c81c7d/go.mod h1:o4KQGtdN14AW+yjsvvwRTJJuXz8XRtIHtEnmAXLyFUw=
golang.org/x/tools v0.0.0-20200331025713-a30bf2db82d4/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8=
golang.org/x/tools v0.0.0-20200501065659-ab2804fb9c9d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200512131952-2bc93b1c0c88/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200515010526-7d3b6ebf133d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200618134242-20370b0cb4b2/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200729194436-6467de6f59a7/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200804011535-6c149bb5ef0d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200825202427-b303f430e36d/go.mod h1:njjCfa9FT2d7l9Bc6FUM5FLjQPp3cFF28FI3qnDFljA=
golang.org/x/tools v0.0.0-20200904185747-39188db58858/go.mod h1:Cj7w3i3Rnn0Xh82ur9kSqwfTHTeVxaDqrfMjpcNT6bE=
golang.org/x/tools v0.0.0-20201022035929-9cf592e881e9/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20201110124207-079ba7bd75cd/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20201201161351-ac6f37ff4c2a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20201208233053-a543418bbed2/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20210105154028-b0ab187a4818/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.0.0-20210108195828-e2f9c7f1fc8e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
golang.org/x/tools v0.1.0/go.mod h1:xkSsbof2nBLbhDlRMhhhyNLN/zl3eTqcnHD5viDpcZ0=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.4.0/go.mod h1:UE5sM2OK9E/d67R0ANs2xJizIymRP5gJU295PvKXxjQ=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.9.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
google.golang.org/api v0.13.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.14.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.15.0/go.mod h1:iLdEw5Ide6rF15KTC1Kkl0iskquN2gFfn9o9XIsbkAI=
google.golang.org/api v0.17.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.18.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.19.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.20.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.22.0/go.mod h1:BwFmGc8tA3vsd7r/7kR8DY7iEEGSU04BFxCo5jP/sfE=
google.golang.org/api v0.24.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.28.0/go.mod h1:lIXQywCXRcnZPGlsd8NbLnOjtAoL6em04bJ9+z0MncE=
google.golang.org/api v0.29.0/go.mod h1:Lcubydp8VUV7KeIHD9z2Bys/sm/vGKnG1UHuDBSrHWM=
google.golang.org/api v0.30.0/go.mod h1:QGmEvQ87FHZNiUVJkT14jQNYJ4ZJjdRF23ZXz5138Fc=
google.golang.org/api v0.35.0/go.mod h1:/XrVsuzM0rZmrsbjJutiuftIzeuTQcEeaYcSk/mQ1dg=
google.golang.org/api v0.36.0/go.mod h1:+z5ficQTmoYpPn8LCUNVpK5I7hwkpjbcgqA7I34qYtE=
google.golang.org/api v0.40.0/go.mod h1:fYKFpnQN0DsDSKRVRcQSDQNtqWPfM9i+zNPxepjRCQ8=
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.5.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
google.golang.org/appengine v1.6.1/go.mod h1:i06prIuMbXzDqacNJfV5OdTW448YApPu5ww/cMBSeb0=
google.golang.org/appengine v1.6.5/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.6/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/appengine v1.6.7/go.mod h1:8WjMMxjGQR8xUklV/ARdw2HLXBOI7O7uCIDZVag1xfc=
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190502173448-54afdca5d873/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE=
google.golang.org/genproto v0.0.0-20190801165951-fa694d86fc64/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
google.golang.org/genproto v0.0.0-20190911173649-1774047e7e51/go.mod h1:IbNlFCBrqXvoKpeg0TB2l7cyZUmoaFKYIwrEpbDKLA8=
google.golang.org/genproto v0.0.0-20191108220845-16a3f7862a1a/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191115194625-c23dd37a84c9/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191216164720-4f79533eabd1/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20191230161307-f3c370f40bfb/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200115191322-ca5a22157cba/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200122232147-0452cf42e150/go.mod h1:n3cpQtvxv34hfy77yVDNjmbRyujviMdxYliBSkLhpCc=
google.golang.org/genproto v0.0.0-20200204135345-fa8e72b47b90/go.mod h1:GmwEX6Z4W5gMy59cAlVYjN9JhxgbQH6Gn+gFDQe2lzA=
google.golang.org/genproto v0.0.0-20200212174721-66ed5ce911ce/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200224152610-e50cd9704f63/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200228133532-8c2c7df3a383/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200305110556-506484158171/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200312145019-da6875a35672/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200331122359-1ee6d9798940/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200430143042-b979b6f78d84/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200511104702-f5ebc3bea380/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20200515170657-fc4c6c6a6587/go.mod h1:YsZOwe1myG/8QRHRsmBRE1LrgQY60beZKjly0O1fX9U=
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20200618031413-b414f8b61790/go.mod h1:jDfRM7FcilCzHH/e9qn6dsT145K34l5v+OpcnNgKAAA=
google.golang.org/genproto v0.0.0-20200729003335-053ba62fc06f/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200804131852-c06518451d9c/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200825200019-8632dd797987/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20200904004341-0bd0a958aa1d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201109203340-2640f1f9cdfb/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201201144952-b05cb90ed32e/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201210142538-e3217bee35cc/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20201214200347-8c77b98c765d/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210108203827-ffc7fda8c3d7/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/genproto v0.0.0-20210226172003-ab064af71705/go.mod h1:FWY/as6DDZQgahTzZj3fqbO1CbirC29ZNUFHwi0/+no=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38=
google.golang.org/grpc v1.21.1/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.26.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.27.1/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60=
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
google.golang.org/grpc v1.30.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.0/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.31.1/go.mod h1:N36X2cJ7JwdamYAgDz+s+rVMFjt3numwzf/HckM8pak=
google.golang.org/grpc v1.33.2/go.mod h1:JMHMWHQWaTccqQQlmk3MJZS+GWXOdAesneDmEnv2fbc=
google.golang.org/grpc v1.34.0/go.mod h1:WotjhfgOW/POjDeRt8vscBtXq+2VjORFy659qA51WJ8=
google.golang.org/grpc v1.35.0/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAGRRjU=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4=
google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190418001031-e561f6794a2a/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
honnef.co/go/tools v0.0.1-2020.1.3/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
honnef.co/go/tools v0.0.1-2020.1.4/go.mod h1:X/FiERA/W4tHapMX5mGpAtMSVEeEUOyHaw9vFzvIQ3k=
rsc.io/binaryregexp v0.2.0/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8=
rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0=
rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA=

View File

@@ -11,7 +11,6 @@ import (
"github.com/go-rod/rod"
"github.com/go-rod/rod/lib/proto"
"github.com/karust/openserp/core"
"github.com/sirupsen/logrus"
"golang.org/x/time/rate"
)
@@ -19,13 +18,15 @@ type Google struct {
core.Browser
core.SearchEngineOptions
rgxpGetDigits *regexp.Regexp
logger *core.EngineLogger
}
func New(browser core.Browser, opts core.SearchEngineOptions) *Google {
gogl := Google{Browser: browser}
opts.Init()
gogl.SearchEngineOptions = opts
gogl.rgxpGetDigits = regexp.MustCompile("\\d")
gogl.logger = core.NewEngineLogger("Google")
gogl.rgxpGetDigits = regexp.MustCompile(`\d`)
return &gogl
}
@@ -68,18 +69,18 @@ func (gogl *Google) getTotalResults(page *rod.Page) (int, error) {
}
func (gogl *Google) solveCaptcha(page *rod.Page, sitekey, datas string) bool {
logrus.Debugf("Solve google Captcha: sitekey=%s, datas=%s, url=%s", sitekey, datas, page.MustInfo().URL)
gogl.logger.Debug("Solve captcha: sitekey=%s", sitekey)
resp, err := gogl.CaptchaSolver.SolveReCaptcha2(sitekey, page.MustInfo().URL, datas)
resp, _, err := gogl.CaptchaSolver.SolveReCaptcha2(sitekey, page.MustInfo().URL, datas)
if err != nil {
logrus.Errorf("Error solving google captcha: %s", err)
gogl.logger.Error("Captcha solve failed: %s", err)
return false
}
logrus.Debug("Resp:", resp)
gogl.logger.Debug("Captcha response received")
_, err = page.Eval(fmt.Sprintf(`;(() => { document.getElementById("g-recaptcha-response").innerHTML="%s"; submitCallback(); })();`, resp))
if err != nil {
logrus.Errorf("Error setting captcha response: %s", err)
gogl.logger.Error("Failed to set captcha response: %s", err)
return false
}
@@ -94,13 +95,13 @@ func (gogl *Google) checkCaptcha(page *rod.Page) bool {
sitekey, err := captchaDiv.First.Attribute("data-sitekey")
if err != nil {
logrus.Errorf("Cannot get Google captcha sitekey: %s", err)
gogl.logger.Error("Cannot get captcha sitekey: %s", err)
return false
}
dataS, err := captchaDiv.First.Attribute("data-s")
if err != nil {
logrus.Errorf("Cannot get Google captcha datas: %s", err)
gogl.logger.Error("Cannot get captcha datas: %s", err)
return false
}
@@ -114,26 +115,27 @@ func (gogl *Google) preparePage(page *rod.Page) {
// Remove "similar queries" lists
_, err := page.Eval(";(() => { document.querySelectorAll(`div[data-initq]`).forEach( el => el.remove()); })();")
if err != nil {
logrus.Errorf("Error preparing the page: %s", err)
gogl.logger.Error("Page preparation failed: %s", err)
}
}
func (gogl *Google) acceptCookies(page *rod.Page) {
diaglogBtns, err := page.Timeout(gogl.Timeout / 10).Search("div[role='dialog'][aria-modal] button")
if err != nil {
logrus.Errorf("Cannot find cookie consent: %s", err)
gogl.logger.Debug("Cookie consent not found: %s", err)
return
}
btnElms, err := diaglogBtns.All()
if err != nil {
logrus.Errorf("Cannot get cookie consent buttons: %s", err)
gogl.logger.Debug("Cannot get cookie consent buttons: %s", err)
return
}
btnElms[3].Click(proto.InputMouseButtonLeft, 1)
}
func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
logrus.Tracef("Start Google search, query: %+v", query)
gogl.logger.Debug("Starting search, query: %+v", query)
searchResults := []core.SearchResult{}
@@ -142,13 +144,17 @@ func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
if err != nil {
return nil, err
}
page := gogl.Navigate(url)
page, err := gogl.Navigate(url)
if err != nil {
return nil, err
}
defer gogl.close(page)
gogl.preparePage(page)
// Check first if there captcha
if gogl.checkCaptcha(page) {
logrus.Errorf("Google captcha occurred during: %s", url)
gogl.logger.Error("Captcha detected: %s", url)
return nil, core.ErrCaptcha
}
@@ -157,10 +163,10 @@ func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
gogl.acceptCookies(page)
}
// Find all results
results, err := page.Timeout(gogl.Timeout).Search("div[data-hveid]")
// Find all results using stable attributes
results, err := page.Timeout(gogl.Timeout).Search("div[data-hveid][data-ved]")
if err != nil {
logrus.Errorf("Cannot parse search results: %s", err)
gogl.logger.Error("Cannot parse search results: %s", err)
return nil, core.ErrSearchTimeout
}
@@ -170,9 +176,9 @@ func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
totalResults, err := gogl.getTotalResults(page)
if err != nil {
logrus.Errorf("Error capturing total results: %v", err)
gogl.logger.Debug("Failed to get total results: %v", err)
}
logrus.Infof("%d SERP results", totalResults)
gogl.logger.Info("Found %d total results", totalResults)
searchResultElems, err := results.All()
if err != nil {
@@ -193,14 +199,14 @@ func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
// Get URL
link, err := resEl.Element("a")
if err != nil {
logrus.Debug("No link found")
gogl.logger.Debug("Missing link")
continue
}
link.MoveMouseOut()
href, err := link.Property("href")
if err != nil {
logrus.Debug("No `href` tag found")
gogl.logger.Debug("Missing href")
continue
}
srchRes.URL = href.String()
@@ -218,17 +224,17 @@ func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
// 2. Parse answer boxes
answerEls, err := resEl.Page().Search("div[data-hveid][data-ulkwtsb] div[data-q]")
if err != nil {
logrus.Debugf("Error while parsing answer box 1: %s", err.Error())
gogl.logger.Debug("Answer parsing failed: %s", err.Error())
continue
}
answers, err := answerEls.All()
if err != nil {
logrus.Debugf("Error while parsing answer box 2: %s", err.Error())
gogl.logger.Debug("Answer parsing failed: %s", err.Error())
continue
}
logrus.Infof("%d answers found", len(answers))
gogl.logger.Info("Found %d answers", len(answers))
// Unvail answer contents
for _, answ := range answers {
@@ -241,21 +247,21 @@ func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
for i, answ := range answers {
answerText := strings.Split(answ.MustText(), "\n")
if len(answerText) < 2 {
logrus.Debugf("Short answer text: %s", answerText)
gogl.logger.Debug("Short answer text: %s", answerText)
continue
}
// Get URL
link, err := answ.Element("a")
if err != nil {
logrus.Debug("No answer link found")
gogl.logger.Debug("Missing answer link")
continue
}
link.MoveMouseOut()
href, err := link.Property("href")
if err != nil {
logrus.Debug("No answer `href` tag found")
gogl.logger.Debug("Missing answer href")
continue
}
srchRes.URL = href.String()
@@ -265,35 +271,56 @@ func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
searchResults = append(searchResults, srchRes)
}
continue
} else if strings.Contains(attrs, "data-ved") && strings.Contains(attrs, "lang") {
// 3. Parse regular search results
// Get URL
link, err := resEl.Element("a")
} else if strings.Contains(attrs, "data-ved") {
// Parse regular search results
// Get title from h3
titleTag, err := resEl.Element("h3")
if err != nil {
continue
}
href, err := link.Property("href")
if err != nil {
logrus.Debug("No `href` tag found")
srchRes.Title, _ = titleTag.Text()
// Get URL from parent link of h3
link, err := titleTag.Parent()
if err == nil && link.MustMatches("a") {
href, _ := link.Property("href")
srchRes.URL = href.String()
}
srchRes.URL = href.String()
// Skip if URL is empty
if srchRes.URL == "" {
continue
}
// Get description using multiple fallback strategies
desc := ""
if descTag, err := resEl.Element("div[data-sncf='1'] div"); err == nil {
desc = descTag.MustText()
} else if descTag, err := resEl.Element("div.VwiC3b"); err == nil {
desc = descTag.MustText()
} else {
// Structural fallback
parent, err := titleTag.Parent()
if err == nil {
parent, err = parent.Parent()
if err == nil {
parent, err = parent.Parent()
if err == nil {
if descTag, err := parent.Next(); err == nil {
if descDiv, err := descTag.Element("div"); err == nil {
desc = descDiv.MustText()
}
}
}
}
}
}
srchRes.Description = desc
rank += 1
// Get title
titleTag, err := link.Element("h3")
if err != nil {
continue
}
srchRes.Title, err = titleTag.Text()
if err != nil {
logrus.Debug("Cannot extract text from title")
}
// Get description
text := resEl.MustText()
textSliced := strings.Split(text, "\n")
srchRes.Description = strings.Join(textSliced[:], "\n")
srchRes.Rank = rank
searchResults = append(searchResults, srchRes)
continue
} else {
//fmt.Println(i, attrs)
@@ -304,11 +331,11 @@ func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
searchResults = append(searchResults, srchRes)
}
return searchResults, nil
return core.DeduplicateResults(searchResults), nil
}
func (gogl *Google) SearchImage(query core.Query) ([]core.SearchResult, error) {
logrus.Tracef("Start Google Image search, query: %+v", query)
gogl.logger.Debug("Starting image search, query: %+v", query)
searchResultsMap := map[string]core.SearchResult{}
url, err := BuildImageURL(query)
@@ -316,7 +343,11 @@ func (gogl *Google) SearchImage(query core.Query) ([]core.SearchResult, error) {
return nil, err
}
page := gogl.Navigate(url)
page, err := gogl.Navigate(url)
if err != nil {
return nil, err
}
defer gogl.close(page)
for len(searchResultsMap) < query.Limit {
@@ -324,16 +355,16 @@ func (gogl *Google) SearchImage(query core.Query) ([]core.SearchResult, error) {
page.Mouse.Scroll(0, 1000000, 1)
page.WaitLoad()
results, err := page.Timeout(gogl.Timeout).Search("div[data-hveid][data-ved][jsaction][jsdata]")
results, err := page.Timeout(gogl.Timeout).Search("div[data-hveid][data-ved][jsaction]")
if err != nil {
logrus.Errorf("Cannot parse search results: %s", err)
gogl.logger.Error("Cannot parse search results: %s", err)
return *core.ConvertSearchResultsMap(searchResultsMap), core.ErrSearchTimeout
}
// Check why no results
if results == nil {
if gogl.checkCaptcha(page) {
logrus.Errorf("Google captcha occurred during: %s", url)
gogl.logger.Error("Captcha detected: %s", url)
return *core.ConvertSearchResultsMap(searchResultsMap), core.ErrCaptcha
}
return *core.ConvertSearchResultsMap(searchResultsMap), err
@@ -352,13 +383,13 @@ func (gogl *Google) SearchImage(query core.Query) ([]core.SearchResult, error) {
// TODO: parse AF_initDataCallback to optimize instead of this?
err := r.Click(proto.InputMouseButtonRight, 1)
if err != nil {
logrus.Error("Error clicking")
gogl.logger.Error("Click failed")
continue
}
dataVed, err := r.Attribute("data-ved")
if err != nil {
logrus.Error("Cannot find `data-ved` attr")
gogl.logger.Error("Missing data-ved attribute")
continue
}
@@ -375,25 +406,25 @@ func (gogl *Google) SearchImage(query core.Query) ([]core.SearchResult, error) {
linkText, err := link.Property("href")
if err != nil {
logrus.Error("No `href` tag found")
gogl.logger.Error("Missing href")
}
imgSrc, err := parseSourceImageURL(linkText.String())
if err != nil {
logrus.Errorf("Cannot parse image href: %v", err)
gogl.logger.Error("Failed to parse image URL: %v", err)
continue
}
// Get title
titleTag, err := r.Element("h3")
if err != nil {
logrus.Error("No `h3` tag found")
gogl.logger.Error("Missing h3 tag")
continue
}
title, err := titleTag.Text()
if err != nil {
logrus.Error("Cannot extract text from title")
gogl.logger.Error("Failed to extract title")
title = "No title"
}
@@ -416,7 +447,7 @@ func (gogl *Google) close(page *rod.Page) {
if !gogl.Browser.LeavePageOpen {
err := page.Close()
if err != nil {
logrus.Error(err)
gogl.logger.Debug("Page close error: %v", err)
}
}
}

View File

@@ -1,17 +1,66 @@
package google
import (
"context"
"crypto/tls"
"net"
"net/http"
"net/url"
"strings"
"time"
"github.com/PuerkitoBio/goquery"
"github.com/corpix/uarand"
"github.com/karust/openserp/core"
"github.com/sirupsen/logrus"
utls "github.com/refraction-networking/utls"
)
func googleRequest(searchURL string) (*http.Response, error) {
baseClient := &http.Client{}
func googleRequest(searchURL string, query core.Query) (*http.Response, error) {
// Create HTTP transport with proxy
transport := &http.Transport{}
if query.ProxyURL != "" {
proxyUrl, err := url.Parse(query.ProxyURL)
if err != nil {
return nil, err
}
transport.Proxy = http.ProxyURL(proxyUrl)
}
// Set insecure TLS
if query.Insecure {
transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
}
transport.DialTLSContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
dialer := &net.Dialer{}
rawConn, err := dialer.DialContext(ctx, network, addr)
if err != nil {
return nil, err
}
hostname := strings.Split(addr, ":")[0]
config := &utls.Config{
ServerName: hostname,
InsecureSkipVerify: query.Insecure,
}
uconn := utls.UClient(rawConn, config, utls.HelloChrome_Auto)
if err := uconn.Handshake(); err != nil {
rawConn.Close()
return nil, err
}
return uconn, nil
}
baseClient := &http.Client{
Transport: transport,
Timeout: time.Second * 10,
}
req, err := http.NewRequest("GET", searchURL, nil)
if err != nil {
return nil, err
@@ -26,7 +75,7 @@ func googleRequest(searchURL string) (*http.Response, error) {
}
func googleResultParser(response *http.Response) ([]core.SearchResult, error) {
doc, err := goquery.NewDocumentFromResponse(response)
doc, err := goquery.NewDocumentFromReader(response.Body)
if err != nil {
return nil, err
}
@@ -34,23 +83,49 @@ func googleResultParser(response *http.Response) ([]core.SearchResult, error) {
results := []core.SearchResult{}
rank := 1
// Get individual results
sel := doc.Find("div.g")
// Use data attributes instead of class names to find results
// Both old and new DOM have data-hveid and data-ved attributes
sel := doc.Find("div[data-hveid][data-ved]")
for i := range sel.Nodes {
item := sel.Eq(i)
// Find URL
linkTag := item.Find("a")
link, _ := linkTag.Attr("href")
// Skip items without an h3 element (which indicates a search result)
if item.Find("h3").Length() == 0 {
continue
}
// Find URL - look for the anchor that contains the h3 title
linkTag := item.Find("h3").Parent()
if linkTag.Is("a") == false {
linkTag = item.Find("h3").Closest("a")
}
link, exists := linkTag.Attr("href")
if !exists || link == "" || link == "#" {
continue
}
link = strings.Trim(link, " ")
// Find title
// Find title - this is inside the h3 element
titleTag := item.Find("h3")
title := titleTag.Text()
// Find description
descTag := item.Find(`div[data-sncf~="1"]`)
// Find description - find div with text content after the heading
// Using attribute selectors that match the description container
descTag := item.Find("div[data-sncf='1']").Find("div").First()
if descTag.Length() == 0 {
// Try another selector approach if the first one fails
descTag = item.Find("div.VwiC3b")
if descTag.Length() == 0 {
// As a last resort, look for any div after the title that might contain description
titleParent := titleTag.Parent()
if titleParent.Is("a") {
titleParent = titleParent.Parent().Parent()
}
descTag = titleParent.NextAll().First().Find("div").First()
}
}
desc := descTag.Text()
if link != "" && link != "#" {
@@ -67,7 +142,7 @@ func googleResultParser(response *http.Response) ([]core.SearchResult, error) {
}
logrus.Tracef("Google search document size: %d", len(doc.Text()))
return results, err
return core.DeduplicateResults(results), err
}
func Search(query core.Query) ([]core.SearchResult, error) {
@@ -77,7 +152,7 @@ func Search(query core.Query) ([]core.SearchResult, error) {
}
logrus.Debugf("Google URL built: %s", googleURL)
res, err := googleRequest(googleURL)
res, err := googleRequest(googleURL, query)
if err != nil {
return nil, err
}

BIN
logo.png

Binary file not shown.

Before

Width:  |  Height:  |  Size: 37 KiB

94
logo.svg Normal file
View File

@@ -0,0 +1,94 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1280 320" role="img" aria-labelledby="title desc">
<title id="title">OpenSERP MultiEngine API Logo</title>
<desc id="desc">Professional banner-like logo with a unified search bar and chips for Google, Bing, DuckDuckGo, Yandex and Baidu, plus an API caption.</desc>
<defs>
<linearGradient id="bar" x1="0" y1="0" x2="1" y2="0">
<stop offset="0%" stop-color="#F5F7FB"/>
<stop offset="100%" stop-color="#FFFFFF"/>
</linearGradient>
<filter id="shadow" x="-10%" y="-30%" width="120%" height="160%">
<feDropShadow dx="0" dy="8" stdDeviation="12" flood-opacity="0.18"/>
</filter>
<filter id="soft" x="-20%" y="-20%" width="140%" height="140%">
<feDropShadow dx="0" dy="2" stdDeviation="3" flood-opacity="0.18"/>
</filter>
<style>
.muted{fill:#5F6B7A}
.chip-label{font:700 14px Inter,Segoe UI,Arial,sans-serif;fill:#1F2937}
.api{font:800 44px "Segoe UI",Inter,Arial,sans-serif;fill:#EA4335}
.title{font:800 46px Inter,Segoe UI,Arial,sans-serif;letter-spacing:0.2px}
</style>
</defs>
<!-- Top brand title similar to the provided mock but neutral -->
<g transform="translate(640,58)" text-anchor="middle">
<text class="title" x="0" y="0">
<tspan fill="#4285F4">O</tspan><tspan fill="#EA4335">p</tspan><tspan fill="#FBBC04">e</tspan><tspan fill="#4285F4">n</tspan><tspan fill="#34A853">S</tspan><tspan fill="#EA4335">E</tspan><tspan fill="#4285F4">R</tspan><tspan fill="#FBBC04">P</tspan>
</text>
</g>
<!-- Main unified search bar -->
<g transform="translate(160,100)">
<rect x="0" y="0" rx="28" ry="28" width="960" height="60" fill="url(#bar)" filter="url(#shadow)"/>
<!-- left magnifier -->
<g class="muted" transform="translate(22,18)">
<circle cx="14" cy="14" r="10" fill="none" stroke="#98A2B3" stroke-width="3"/>
<rect x="22" y="22" width="12" height="4" rx="2" fill="#98A2B3" transform="rotate(45 28 24)"/>
</g>
<!-- placeholder text -->
<text x="70" y="37" font="500 18px Inter,Segoe UI,Arial" fill="#5F6B7A">Search across Google, Bing, DuckDuckGo, Yandex, Baidu…</text>
<!-- right icons -->
<g transform="translate(880,13)" class="muted">
<rect x="48" y="-2" width="1" height="36" fill="#E5E7EB"/>
<circle cx="18" cy="17" r="8" fill="#E5E7EB"/>
<rect x="12" y="8" width="12" height="6" rx="3" fill="#98A2B3"/>
<circle cx="18" cy="17" r="3" fill="#98A2B3"/>
<!-- microphone glyph -->
<rect x="62" y="6" width="8" height="16" rx="4" fill="#98A2B3"/>
<rect x="60" y="24" width="12" height="3" rx="1.5" fill="#98A2B3"/>
</g>
</g>
<!-- Engine chips (brand colored, accessible, non-trademark glyphs) -->
<g transform="translate(640,88)" text-anchor="middle" dominant-baseline="middle">
<!-- layout positions -->
<g transform="translate(-420,-8)">
<rect rx="14" ry="14" width="120" height="28" fill="#FFFFFF" stroke="#E5E7EB" filter="url(#soft)"/>
<circle cx="16" cy="14" r="8" fill="#4285F4"/>
<text class="chip-label" x="78" y="17">Google</text>
</g>
<g transform="translate(-260,-8)">
<rect rx="14" ry="14" width="110" height="28" fill="#FFFFFF" stroke="#E5E7EB" filter="url(#soft)"/>
<circle cx="16" cy="14" r="8" fill="#008373"/>
<text class="chip-label" x="70" y="17">Bing</text>
</g>
<g transform="translate(-100,-8)">
<rect rx="14" ry="14" width="150" height="28" fill="#FFFFFF" stroke="#E5E7EB" filter="url(#soft)"/>
<circle cx="16" cy="14" r="8" fill="#DE5833"/>
<text class="chip-label" x="90" y="17">DuckDuckGo</text>
</g>
<g transform="translate(90,-8)">
<rect rx="14" ry="14" width="120" height="28" fill="#FFFFFF" stroke="#E5E7EB" filter="url(#soft)"/>
<circle cx="16" cy="14" r="8" fill="#FC3F1D"/>
<text class="chip-label" x="78" y="17">Yandex</text>
</g>
<g transform="translate(250,-8)">
<rect rx="14" ry="14" width="115" height="28" fill="#FFFFFF" stroke="#E5E7EB" filter="url(#soft)"/>
<circle cx="16" cy="14" r="8" fill="#2932E1"/>
<text class="chip-label" x="72" y="17">Baidu</text>
</g>
</g>
<!-- API caption -->
<g transform="translate(640,232)" text-anchor="middle">
<text class="api">API</text>
</g>
</svg>

After

Width:  |  Height:  |  Size: 4.3 KiB

View File

@@ -8,7 +8,6 @@ import (
"github.com/go-rod/rod"
"github.com/karust/openserp/core"
"github.com/sirupsen/logrus"
"golang.org/x/time/rate"
)
@@ -38,12 +37,14 @@ type Yandex struct {
core.Browser
core.SearchEngineOptions
pageSleep time.Duration // Sleep between pages
logger *core.EngineLogger
}
func New(browser core.Browser, opts core.SearchEngineOptions) *Yandex {
yand := Yandex{Browser: browser}
opts.Init()
yand.SearchEngineOptions = opts
yand.logger = core.NewEngineLogger("Yandex")
yand.pageSleep = time.Second * 1
return &yand
@@ -80,19 +81,19 @@ func (yand *Yandex) parseResults(results rod.Elements, pageNum int) []core.Searc
}
linkText, err := link.Property("href")
if err != nil {
logrus.Error("No `href` tag found")
yand.logger.Error("Missing href")
}
// Get title
titleTag, err := link.Element("h2")
if err != nil {
logrus.Error("No title `h2` tag found")
yand.logger.Error("Missing h2 title")
continue
}
title, err := titleTag.Text()
if err != nil {
logrus.Error("Cannot extract text from title")
yand.logger.Error("Failed to extract title")
title = "No title"
}
@@ -100,7 +101,7 @@ func (yand *Yandex) parseResults(results rod.Elements, pageNum int) []core.Searc
descTag, err := r.Element(`span.OrganicTextContentSpan`)
desc := ""
if err != nil {
logrus.Trace("No description `span.OrganicTextContentSpan` tag found")
yand.logger.Debug("No description")
} else {
desc = descTag.MustText()
}
@@ -113,7 +114,7 @@ func (yand *Yandex) parseResults(results rod.Elements, pageNum int) []core.Searc
}
func (yand *Yandex) Search(query core.Query) ([]core.SearchResult, error) {
logrus.Tracef("Start Yandex search, query: %+v", query)
yand.logger.Debug("Starting search, query: %+v", query)
allResults := []core.SearchResult{}
searchPage := 0
@@ -124,13 +125,16 @@ func (yand *Yandex) Search(query core.Query) ([]core.SearchResult, error) {
return nil, err
}
page := yand.Navigate(url)
page, err := yand.Navigate(url)
if err != nil {
return nil, err
}
// Get all search results in page
searchRes, err := page.Timeout(yand.Timeout).Search("li.serp-item")
if err != nil {
defer page.Close()
logrus.Errorf("Cannot parse search results: %s", err)
yand.logger.Error("Cannot parse search results: %s", err)
return nil, core.ErrSearchTimeout
}
@@ -139,9 +143,9 @@ func (yand *Yandex) Search(query core.Query) ([]core.SearchResult, error) {
defer page.Close()
if yand.isNoResults(page) {
logrus.Errorf("No results found")
yand.logger.Warn("No results found")
} else if yand.isCaptcha(page) {
logrus.Errorf("Yandex captcha occurred during: %s", url)
yand.logger.Error("Captcha detected: %s", url)
return nil, core.ErrCaptcha
}
break
@@ -149,7 +153,7 @@ func (yand *Yandex) Search(query core.Query) ([]core.SearchResult, error) {
elements, err := searchRes.All()
if err != nil {
logrus.Errorf("Cannot get all elements from search results: %s", err)
yand.logger.Error("Cannot get search elements: %s", err)
break
}
@@ -162,18 +166,19 @@ func (yand *Yandex) Search(query core.Query) ([]core.SearchResult, error) {
// Close tab before opening new one during the cycle
err = page.Close()
if err != nil {
logrus.Error(err)
yand.logger.Debug("Page close error: %v", err)
}
}
time.Sleep(yand.pageSleep)
}
return allResults, nil
yand.logger.Info("Search completed: %d results", len(allResults))
return core.DeduplicateResults(allResults), nil
}
func (yand *Yandex) SearchImage(query core.Query) ([]core.SearchResult, error) {
logrus.Tracef("Start Yandex image search, query: %+v", query)
yand.logger.Debug("Starting image search, query: %+v", query)
searchResults := []core.SearchResult{}
@@ -185,9 +190,12 @@ func (yand *Yandex) SearchImage(query core.Query) ([]core.SearchResult, error) {
}
searchPage += 1
page := yand.Navigate(url)
page, err := yand.Navigate(url)
if err != nil {
return nil, err
}
if !yand.LeavePageOpen {
if !yand.Browser.LeavePageOpen {
defer page.Close()
}
@@ -197,16 +205,16 @@ func (yand *Yandex) SearchImage(query core.Query) ([]core.SearchResult, error) {
results, err := page.Timeout(yand.Timeout).Search("div[role='main'] div[data-state]")
if err != nil {
logrus.Errorf("Cannot find search results: %s", err)
yand.logger.Error("Cannot find search results: %s", err)
}
// Check why no results
if results == nil {
if yand.isCaptcha(page) {
logrus.Errorf("Yandex captcha occurred during: %s", url)
yand.logger.Error("Captcha detected: %s", url)
return searchResults, core.ErrCaptcha
} else if yand.isNoResults(page) {
logrus.Errorf("No results found")
yand.logger.Warn("No results found")
}
return searchResults, core.ErrSearchTimeout
}
@@ -233,7 +241,7 @@ func (yand *Yandex) SearchImage(query core.Query) ([]core.SearchResult, error) {
searchResults = append(searchResults, res)
}
if !yand.LeavePageOpen {
if !yand.Browser.LeavePageOpen {
page.Close()
}
}
@@ -242,5 +250,5 @@ func (yand *Yandex) SearchImage(query core.Query) ([]core.SearchResult, error) {
return searchResults[i].Rank < searchResults[j].Rank
})
return searchResults, nil
return core.DeduplicateResults(searchResults), nil
}

View File

@@ -1,17 +1,66 @@
package yandex
import (
"context"
"crypto/tls"
"net"
"net/http"
"net/url"
"strings"
"time"
"github.com/PuerkitoBio/goquery"
"github.com/corpix/uarand"
"github.com/karust/openserp/core"
"github.com/sirupsen/logrus"
utls "github.com/refraction-networking/utls"
)
func yandexRequest(searchURL string) (*http.Response, error) {
baseClient := &http.Client{}
func yandexRequest(searchURL string, query core.Query) (*http.Response, error) {
// Create HTTP transport with proxy
transport := &http.Transport{}
if query.ProxyURL != "" {
proxyUrl, err := url.Parse(query.ProxyURL)
if err != nil {
return nil, err
}
transport.Proxy = http.ProxyURL(proxyUrl)
}
// Set insecure TLS
if query.Insecure {
transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
}
transport.DialTLSContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
dialer := &net.Dialer{}
rawConn, err := dialer.DialContext(ctx, network, addr)
if err != nil {
return nil, err
}
hostname := strings.Split(addr, ":")[0]
config := &utls.Config{
ServerName: hostname,
InsecureSkipVerify: query.Insecure,
}
uconn := utls.UClient(rawConn, config, utls.HelloChrome_Auto)
if err := uconn.Handshake(); err != nil {
rawConn.Close()
return nil, err
}
return uconn, nil
}
baseClient := &http.Client{
Transport: transport,
Timeout: time.Second * 10,
}
req, err := http.NewRequest("GET", searchURL, nil)
if err != nil {
return nil, err
@@ -26,7 +75,7 @@ func yandexRequest(searchURL string) (*http.Response, error) {
}
func yandexResultParser(response *http.Response) ([]core.SearchResult, error) {
doc, err := goquery.NewDocumentFromResponse(response)
doc, err := goquery.NewDocumentFromReader(response.Body)
if err != nil {
return nil, err
}
@@ -67,7 +116,7 @@ func yandexResultParser(response *http.Response) ([]core.SearchResult, error) {
}
logrus.Tracef("Yandex search document size: %d", len(doc.Text()))
return results, err
return core.DeduplicateResults(results), err
}
func Search(query core.Query) ([]core.SearchResult, error) {
@@ -77,7 +126,7 @@ func Search(query core.Query) ([]core.SearchResult, error) {
}
logrus.Debugf("Yandex URL built: %s", googleURL)
res, err := yandexRequest(googleURL)
res, err := yandexRequest(googleURL, query)
if err != nil {
return nil, err
}