Stealth browser as option. Update default config

Handle failed navigation
Merge pull request #10 from PACHAKUTlQ/main
2026-08-25 17:22:09 +08:00 · 2025-08-18 22:49:45 +03:00 · 2025-07-03 02:29:12 +03:00 · 2025-07-03 02:09:18 +03:00 · 2025-05-27 15:05:53 +08:00 · 2025-04-10 18:07:57 +08:00
29 changed files with 1655 additions and 282 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -21,4 +21,7 @@ raw.go
 dev.md
 .gitignore
 logrus.txt
-logs.txt
+logs.txt
+.release
+core/test/
+.aider*
--- a/1
+++ b/1
@@ -17,6 +17,7 @@ RUN go build -o /app/openserp .
 FROM zenika/alpine-chrome:with-chromedriver

 COPY --from=builder /app/openserp /usr/local/bin/openserp
+ADD config.yaml /usr/src/app

 ENTRYPOINT ["openserp"]

--- a/README.md
+++ b/README.md
@@ -1,82 +1,129 @@
 # OpenSERP (Search Engine Results Page)
+
 ![OpenSERP](/logo.png)

 [![Go Report Card](https://goreportcard.com/badge/github.com/karust/openserp)](https://goreportcard.com/report/github.com/karust/openserp)
 [![Go Reference](https://pkg.go.dev/badge/github.com/karust/openserp.svg)](https://pkg.go.dev/github.com/karust/openserp)
-<!-- ![Docker Image Size (tag)](https://img.shields.io/docker/image-size/karust/openserp/latest)
-![GitHub release (by tag)](https://img.shields.io/github/downloads/karust/openserp/0.1/total) -->
+[![release](https://img.shields.io/github/release-pre/karust/openserp.svg)](https://github.com/karust/openserp/releases)
+
+<!-- ![Docker Image Size (tag)](https://img.shields.io/docker/image-size/karust/openserp/latest) -->

 API access for search engines results if available isn't free.

 Using OpenSERP, you can get search results from **Google**, **Yandex**, **Baidu** via API or CLI!

-See [Docker](#docker), [CLI](#cli) usage examples below.
+See [Docker](#docker) and [CLI](#cli) usage examples below ([search](#search), [images](#images)).
+
+## Docker usage <a name="docker"></a> 🐳
+
+- Run API server:

-## Docker usage  <a name="docker"></a> 🐳
-* Run API server:
 ```bash
 # Use prebuilt image
 docker run -p 127.0.0.1:7000:7000 -it karust/openserp serve -a 0.0.0.0 -p 7000

-# Or
+# Or build one and run using docker-compose.yaml
 docker-compose up --build
 ```

-### *Example request*
+### Request parameters
+
+| Param   | Description                                                            |
+| ------- | ---------------------------------------------------------------------- |
+| text    | Text to search                                                         |
+| lang    | Search pages in selected language (`EN`, `DE`, `RU`...)                |
+| date    | Date in `YYYYMMDD..YYYYMMDD` format (e.g. 20181010..20231010)          |
+| file    | File extension to search (e.g. `PDF`, `DOC`)                           |
+| site    | Search within a specific website                                       |
+| limit   | Limit the number of results                                            |
+| answers | Include google answers as negative rank indexes (e.g. `true`, `false`) |
+
+### **Search**
+
+### _Example request_
+
 Get 20 **Google** results for `hello world`, only in English:
-```JSON
+
+```
 GET http:/127.0.0.1:7000/google/search?lang=EN&limit=20&text=hello world
 ```
+
 You can replace `google` to `yandex` or `baidu` in query to change search engine.
+|

-### Request parameters
-| Param | Description                                                  |
-|-------|--------------------------------------------------------------|
-| text  | Text to search                                               |
-| lang  | Search pages in selected language (`EN`, `DE`, `RU`...)      |
-| date  | Date in `YYYYMMDD..YYYYMMDD` format (e.g. 20181010..20231010) |
-| file  | File extension to search  (e.g. `PDF`, `DOC`)                 |
-| site  | Search only in selected site                                 |
-| limit | Limit the number of results                                  |
+### _Example response_

-### *Example response*
 ```JSON
 [
    {
-        "rank": 0,
+        "rank": 1,
        "url": "https://en.wikipedia.org/wiki/%22Hello,_World!%22_program",
        "title": "\"Hello, World!\" program",
-        "description": "A \"Hello, World!\" program is generally a computer program that ignores any input, and outputs or displays a message similar to \"Hello, World!\"."
+        "description": "A \"Hello, World!\" program is generally a computer program that ignores any input, and outputs or displays a message similar to \"Hello, World!\".",
+        "ad": false
    },
 ]
 ```

+### **Images** **[WIP]**

+### _Example request_
+
+Get 100 **Google** results for `golden puppy`:
+
+```
+GET http://127.0.0.1:7000/google/image?text=golden puppy&limit=100
+```

 ## CLI <a name="cli"></a> ⌨️
-* Use `-h` flag to see commands.
-* You can use `serve` command to serve API:
+
+- Use `-h` flag to see all commands.
+- You can use `serve` command to serve API:
+
 ```bash
-openserp serve 
+openserp serve
 ```
-* Or print results in CLI using `search` command:
+
+- Or print results in CLI using `search` command:
+
 ```bash
-openserp search google "how to get banned in google fast" # Change `google` to `yandex` or `baidu`
+openserp search google "how to get banned from google fast" # Change `google` to `yandex` or `baidu`
 ```
+
 As a result you should get JSON output containting search results:
+
 ```json
 [
- {
-  "rank": 0,
-  "url": "https://www.cyberoptik.net/blog/6-sure-fire-ways-to-get-banned-from-google/",
-  "title": "11 Sure-Fire Ways to Get Banned From Google | CyberOptik",
-  "description": "How To Get Banned From Google · 1. Cloaking: The Art of Deception · 2. Plagiarism: Because Originality is Overrated · 3. Keyword Stuffing: More is Always Better · 4 ..."
- },
+  {
+    "rank": 1,
+    "url": "https://www.cyberoptik.net/blog/6-sure-fire-ways-to-get-banned-from-google/",
+    "title": "11 Sure-Fire Ways to Get Banned From Google | CyberOptik",
+    "description": "How To Get Banned From Google · 1. Cloaking: The Art of Deception · 2. Plagiarism: Because Originality is Overrated · 3. Keyword Stuffing: More is Always Better · 4 ...",
+    "ad": false
+  }
 ]
- ```
+```
+
+### Proxy Support
+
+Both browser mode and raw mode support HTTP and SOCKS5 proxies with authentication:
+
+```bash
+# HTTP proxy with auth
+openserp search google "query" --proxy http://user:pass@127.0.0.1:8080
+
+# Serve with SOCKS5 proxy
+openserp serve --proxy socks5://127.0.0.1:1080
+
+# For HTTPS sites through HTTP proxy, use --insecure to ignore certificate errors
+openserp search google "query" --proxy http://127.0.0.1:8080 --insecure
+openserp search google "query" -x http://127.0.0.1:8080 -k
+```
+
+## License

- ## License
 This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details

 ## Bugs + Questions 👾
-If you have some issues/bugs/questions, feel free to open an issue.
+
+If you have some issues/bugs/questions, feel free to open an issue.
--- a/baidu/all_test.go
+++ b/baidu/all_test.go
@@ -13,7 +13,7 @@ var testQuery = core.Query{Text: "go", Site: "tutorialspoint.com", DateInterval:
 func init() {
 	core.InitLogger(true, true)

-	opts := core.BrowserOpts{IsHeadless: true, IsLeakless: false, Timeout: time.Second * 2, WaitRequests: false}
+	opts := core.BrowserOpts{IsHeadless: false, IsLeakless: false, Timeout: time.Second * 10}
 	browser, _ = core.NewBrowser(opts)
 }

@@ -30,17 +30,8 @@ func TestUrlBuild(t *testing.T) {
 	}
 }

-// func TestSearchRaw(t *testing.T) {
-// 	results, err := Search(testQuery)
-// 	if err != nil {
-// 		t.Fatal(err)
-// 	}
-
-// 	fmt.Println(results)
-// }
-
-func TestSearchBaidu(t *testing.T) {
-	baid := New(*browser)
+func TestSearch(t *testing.T) {
+	baid := New(*browser, core.SearchEngineOptions{})
 	results, err := baid.Search(testQuery)
 	if err != nil {
 		t.Fatal(err)
@@ -50,3 +41,31 @@ func TestSearchBaidu(t *testing.T) {
 		t.Fatal("No results got from Baidu search")
 	}
 }
+
+func TestImageUrlBuild(t *testing.T) {
+	query := core.Query{Text: "金毛猎犬"}
+
+	got, err := BuildImageURL(query, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	want := "https://image.baidu.com/search/acjson?cl=2&fp=result&ie=utf-8&ipn=rj&oe=utf-8&pn=0&rn=30&tn=resultjson_com&word=%E9%87%91%E6%AF%9B%E7%8C%8E%E7%8A%AC"
+	if want != got {
+		t.Fatalf("Want: `%s`, Got `%s`", want, got)
+	}
+}
+
+func TestImageSearch(t *testing.T) {
+	baid := New(*browser, core.SearchEngineOptions{})
+
+	query := core.Query{Text: "each each data", Limit: 60}
+	results, err := baid.SearchImage(query)
+	if err != nil {
+		t.Fatalf("Cannot [ImageBaidu]: %s", err)
+	}
+
+	if len(results) < 60 {
+		t.Fatalf("[ImageBaidu] returned not full result")
+	}
+}
--- a/baidu/search.go
+++ b/baidu/search.go
@@ -1,27 +1,68 @@
 package baidu

 import (
+	"encoding/json"
+	"fmt"
+	"regexp"
 	"strings"
-	"time"

+	"github.com/go-rod/rod"
 	"github.com/karust/openserp/core"
 	"github.com/sirupsen/logrus"
+	"golang.org/x/time/rate"
 )

+type imageDataJson struct {
+	Query        string `json:"queryExt"`
+	TotalResults int    `json:"displayNum"`
+
+	Data []struct {
+		Title       string `json:"fromPageTitle"`
+		PictureDate string `json:"bdImgnewsDate"`
+		ThumbURL    string `json:"thumbURL"`
+		Type        string
+		Height      int
+		Width       int
+		IsCopyright int
+		AdType      string `json:"adType"`
+		URL         []struct {
+			SourcePage string `json:"FromURL"`
+			Original   string `json:"ObjURL"`
+		} `json:"replaceUrl"`
+	}
+}
+
 type Baidu struct {
 	core.Browser
-	checkTimeout time.Duration
+	core.SearchEngineOptions
 }

-func New(browser core.Browser) *Baidu {
+func New(browser core.Browser, opts core.SearchEngineOptions) *Baidu {
 	baid := Baidu{Browser: browser}
-	baid.checkTimeout = time.Second * 2
+	opts.Init()
+	baid.SearchEngineOptions = opts
 	return &baid
 }
+
 func (baid *Baidu) Name() string {
 	return "baidu"
 }

+func (baid *Baidu) GetRateLimiter() *rate.Limiter {
+	ratelimit := rate.Every(baid.GetRatelimit())
+	return rate.NewLimiter(ratelimit, baid.RateBurst)
+}
+
+func (baid *Baidu) isCaptcha(page *rod.Page) bool {
+	_, err := page.Timeout(baid.GetSelectorTimeout()).Search("div.passMod_dialog-body")
+	return err == nil
+}
+
+func (baid *Baidu) isTimeout(page *rod.Page) bool {
+	_, err := page.Timeout(baid.GetSelectorTimeout()).Search("button.timeout-button")
+	return err == nil
+}
+
 func (baid *Baidu) Search(query core.Query) ([]core.SearchResult, error) {
 	logrus.Tracef("Start Baidu search, query: %+v", query)

@@ -33,11 +74,30 @@ func (baid *Baidu) Search(query core.Query) ([]core.SearchResult, error) {
 		return nil, err
 	}

-	page := baid.Navigate(url)
+	page, err := baid.Navigate(url)
+	if err != nil {
+		return nil, err
+	}

 	results, err := page.Timeout(baid.Timeout).Search("div.c-container.new-pmd")
 	if err != nil {
-		return nil, err
+		defer page.Close()
+		logrus.Errorf("Cannot parse search results: %s", err)
+		return nil, core.ErrSearchTimeout
+	}
+
+	// Check why no results, maybe captcha?
+	if results == nil {
+		defer page.Close()
+
+		if baid.isCaptcha(page) {
+			logrus.Errorf("Baidu captcha occurred during: %s", url)
+			return nil, core.ErrCaptcha
+		} else if baid.isTimeout(page) {
+			logrus.Errorf("Baidu timeout occurred during: %s", url)
+			return nil, core.ErrCaptcha
+		}
+		return nil, nil
 	}

 	resultElements, err := results.All()
@@ -81,5 +141,102 @@ func (baid *Baidu) Search(query core.Query) ([]core.SearchResult, error) {
 		}
 	}

-	return searchResults, nil
+	return core.DeduplicateResults(searchResults), nil
+}
+
+func (baid *Baidu) SearchImage(query core.Query) ([]core.SearchResult, error) {
+	logrus.Tracef("Start Baidu Image search, query: %+v", query)
+
+	searchResults := []core.SearchResult{}
+	searchPage := 0
+
+	for len(searchResults) < query.Limit {
+		url, err := BuildImageURL(query, searchPage)
+		if err != nil {
+			return nil, err
+		}
+
+		// Get anti-crawler cookies first, then reload page
+		page, err := baid.Navigate(url)
+		if err != nil {
+			return nil, err
+		}
+
+		if !baid.LeavePageOpen {
+			defer page.Close()
+		}
+		page.Reload()
+		page.WaitLoad()
+
+		result, err := page.Timeout(baid.Timeout).Search("body > pre")
+		if err != nil {
+			defer page.Close()
+			logrus.Errorf("Cannot parse search results: %s", err)
+			return nil, core.ErrSearchTimeout
+		}
+
+		// Check why no results, maybe captcha?
+		if result == nil {
+			defer page.Close()
+
+			if baid.isCaptcha(page) {
+				logrus.Errorf("Baidu captcha occurred during: %s", url)
+				return nil, core.ErrCaptcha
+			} else if baid.isTimeout(page) {
+				logrus.Errorf("Baidu timeout occurred during: %s", url)
+				return nil, core.ErrCaptcha
+			}
+			return nil, nil
+		}
+
+		jsonText, err := result.First.Text()
+		if err != nil {
+			return nil, err
+		}
+
+		var data imageDataJson
+
+		// Fix broken JSON
+		jsonText = strings.ReplaceAll(jsonText, `\'`, "'")
+		matchNewlines := regexp.MustCompile(`[\r\n\t]`)
+		escapeNewlines := func(s string) string {
+			return matchNewlines.ReplaceAllString(s, "\\n")
+		}
+		re := regexp.MustCompile(`"[^"\\]*(?:\\[\s\S][^"\\]*)*"`)
+		fixedJson := re.ReplaceAllStringFunc(jsonText, escapeNewlines)
+
+		err = json.Unmarshal([]byte(fixedJson), &data)
+		if err != nil {
+			logrus.Errorf("Cannot unmarshal JSON: %v\nData: %v", err, jsonText)
+			return nil, err
+		}
+
+		for i, img := range data.Data {
+			if len(img.URL) == 0 {
+				continue
+			}
+			res := core.SearchResult{
+				Rank:        (searchPage * 30) + (i + 1),
+				URL:         img.URL[0].Original,
+				Title:       img.Title,
+				Description: fmt.Sprintf("%v,%v,%vx%x,copyright:%v", img.PictureDate, img.Type, img.Height, img.Width, img.IsCopyright),
+				Ad: func() bool {
+					if img.AdType != "0" {
+						return true
+					} else {
+						return false
+					}
+				}(),
+			}
+			searchResults = append(searchResults, res)
+		}
+
+		searchPage += 1
+
+		if !baid.LeavePageOpen {
+			page.Close()
+		}
+	}
+
+	return core.DeduplicateResults(searchResults), nil
 }
--- a/baidu/search_raw.go
+++ b/baidu/search_raw.go
@@ -1,9 +1,12 @@
 package baidu

 import (
+	"crypto/tls"
 	"fmt"
 	"net/http"
+	"net/url"
 	"strings"
+	"time"

 	"github.com/PuerkitoBio/goquery"
 	"github.com/corpix/uarand"
@@ -11,8 +14,27 @@ import (
 	"github.com/sirupsen/logrus"
 )

-func baiduRequest(searchURL string) (*http.Response, error) {
-	baseClient := &http.Client{}
+func baiduRequest(searchURL string, query core.Query) (*http.Response, error) {
+	// Create HTTP transport with proxy
+	transport := &http.Transport{}
+	if query.ProxyURL != "" {
+		proxyUrl, err := url.Parse(query.ProxyURL)
+		if err != nil {
+			return nil, err
+		}
+		transport.Proxy = http.ProxyURL(proxyUrl)
+	}
+
+	// Set insecure TLS
+	if query.Insecure {
+		transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
+	}
+
+	baseClient := &http.Client{
+		Transport: transport,
+		Timeout:   time.Second * 10,
+	}
+
 	req, err := http.NewRequest("GET", searchURL, nil)
 	if err != nil {
 		return nil, err
@@ -27,7 +49,7 @@ func baiduRequest(searchURL string) (*http.Response, error) {
 }

 func baiduResultParser(response *http.Response) ([]core.SearchResult, error) {
-	doc, err := goquery.NewDocumentFromResponse(response)
+	doc, err := goquery.NewDocumentFromReader(response.Body)
 	if err != nil {
 		return nil, err
 	}
@@ -79,7 +101,7 @@ func Search(query core.Query) ([]core.SearchResult, error) {
 	}
 	logrus.Debugf("Baidu URL built: %s", googleURL)

-	res, err := baiduRequest(googleURL)
+	res, err := baiduRequest(googleURL, query)
 	if err != nil {
 		return nil, err
 	}
@@ -91,5 +113,5 @@ func Search(query core.Query) ([]core.SearchResult, error) {
 	}
 	logrus.Debugf("Baidu Raw results : %v", results)

-	return results, nil
+	return core.DeduplicateResults(results), nil
 }
--- a/baidu/url.go
+++ b/baidu/url.go
@@ -76,3 +76,32 @@ func BuildURL(q core.Query) (string, error) {
 	base.RawQuery = params.Encode()
 	return base.String(), nil
 }
+
+func BuildImageURL(q core.Query, pageNum int) (string, error) {
+	base, _ := url.Parse("https://image.baidu.com/")
+	base.Path += "search/acjson"
+
+	params := url.Values{}
+	params.Add("tn", "resultjson_com")
+	params.Add("cl", "2") // Cl = 2 indicates image search
+
+	if q.Text != "" {
+		params.Add("word", q.Text)
+	}
+
+	if len(params.Get("word")) == 0 {
+		return "", errors.New("Empty query built")
+	}
+
+	if q.Limit != 0 {
+		params.Add("rn", "30")                     // Results per page
+		params.Add("pn", strconv.Itoa(pageNum*30)) // Offset
+	}
+
+	params.Add("fp", "result")
+	params.Add("ipn", "rj")
+	params.Add("ie", "utf-8")
+	params.Add("oe", "utf-8")
+	base.RawQuery = params.Encode()
+	return base.String(), nil
+}
--- a/cmd/root.go
+++ b/cmd/root.go
@@ -2,6 +2,7 @@ package cmd

 import (
 	"fmt"
+	"strings"

 	"github.com/karust/openserp/core"
 	"github.com/sirupsen/logrus"
@@ -11,42 +12,63 @@ import (
 )

 const (
-	version                    = "0.1.1"
-	defaultConfigFilename      = "config"
-	envPrefix                  = "OPENSERP"
-	replaceHyphenWithCamelCase = false
+	version               = "0.4.1"
+	defaultConfigFilename = "config"
+	envPrefix             = "OPENSERP"
 )

-type AppConfig struct {
-	Host          string
-	Port          int
-	Timeout       int
-	ConfigPath    string
-	IsBrowserHead bool `mapstructure:"head"`
-	IsLeaveHead   bool `mapstructure:"leave_head"`
-	IsLeakless    bool `mapstructure:"leakless"`
-	IsDebug       bool `mapstructure:"debug"`
-	IsVerbose     bool `mapstructure:"verbose"`
-	IsRawRequests bool `mapstructure:"raw_requests"`
+type Config struct {
+	App           AppConfig                `mapstructure:"app"`
+	Config2Capcha Config2Captcha           `mapstructure:"2captcha"`
+	GoogleConfig  core.SearchEngineOptions `mapstructure:"google"`
+	YandexConfig  core.SearchEngineOptions `mapstructure:"yandex"`
+	BaiduConfig   core.SearchEngineOptions `mapstructure:"baidu"`
 }

-var appConf = AppConfig{}
+type Config2Captcha struct {
+	ApiKey string `mapstructure:"apikey"`
+}
+
+type AppConfig struct {
+	Host          string `mapstructure:"host"`
+	Port          int    `mapstructure:"port"`
+	Timeout       int    `mapstructure:"timeout"`
+	ConfigPath    string `mapstructure:"config_path"`
+	IsBrowserHead bool   `mapstructure:"head"`
+	IsLeaveHead   bool   `mapstructure:"leave_head"`
+	IsLeakless    bool   `mapstructure:"leakless"`
+	IsDebug       bool   `mapstructure:"debug"`
+	IsVerbose     bool   `mapstructure:"verbose"`
+	IsRawRequests bool   `mapstructure:"raw_requests"`
+	ProxyURL      string `mapstructure:"proxy"`
+	Insecure      bool   `mapstructure:"insecure"`
+	IsStealth     bool   `mapstructure:"stealth"`
+}
+
+var config = Config{}

 var RootCmd = &cobra.Command{
-	Use:     "openserp",
-	Short:   "Open SERP",
-	Long:    `Search via Google, Yandex and Baidu`,
-	Version: version,
+	Use:          "openserp",
+	Short:        "Open SERP",
+	Long:         `Get [Google, Yandex, Baidu] search engine results via API or CLI.`,
+	Version:      version,
+	SilenceUsage: true,
 	PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
-		core.InitLogger(appConf.IsVerbose, appConf.IsDebug)
+		core.InitLogger(config.App.IsVerbose, config.App.IsDebug)
+
 		err := initializeConfig(cmd)
-		logrus.Debugf("Config: %+v", appConf)
-		return err
+		if err != nil {
+			return err
+		}
+
+		logrus.Debugf("Final config: %+v", config)
+		return nil
 	},
+
 	// Run: func(cmd *cobra.Command, args []string) {
 	// 	// Working with OutOrStdout/OutOrStderr allows us to unit test our command easier
 	// 	//out := cmd.OutOrStdout()
-	// 	logrus.Trace("Config:", appConf)
+	// 	logrus.Trace("Config:", config)
 	// },
 }

@@ -54,14 +76,10 @@ var RootCmd = &cobra.Command{
 func bindFlags(cmd *cobra.Command, vpr *viper.Viper) {
 	cmd.Flags().VisitAll(func(flg *pflag.Flag) {
 		configName := "app." + flg.Name
-		//if replaceHyphenWithCamelCase {
-		//	configName = strings.ReplaceAll(f.Name, "-", "")
-		//}

 		// Apply viper config value to the flag if viper has a value
-		if !flg.Changed && vpr.IsSet(configName) {
-			val := vpr.Get(configName)
-			cmd.Flags().Set(flg.Name, fmt.Sprintf("%v", val))
+		if flg.Changed {
+			vpr.Set(configName, flg.Value)
 		}
 	})
 }
@@ -74,34 +92,50 @@ func initializeConfig(cmd *cobra.Command) error {
 	v.SetConfigName(defaultConfigFilename)
 	v.AddConfigPath(".")

-	// Return an error if we cannot parse the config file.
-	if err := v.ReadInConfig(); err != nil {
-		if _, ok := err.(viper.ConfigFileNotFoundError); !ok {
-			return err
+	// 1. Config. Return an error if we cannot parse the config file.
+	err := v.ReadInConfig()
+	if err != nil {
+		err = fmt.Errorf("cannot read config: %v", err)
+		logrus.Warn(err)
+	}
+
+	// 2. Env. Bind environment variables to their equivalent keys with underscores
+	for _, key := range v.AllKeys() {
+		envKey := envPrefix + "_" + strings.ToUpper(strings.ReplaceAll(key, ".", "_"))
+		err := v.BindEnv(key, envKey)
+		if err != nil {
+			logrus.Errorf("Unable to bind ENV valye: %v", err)
 		}
 	}

-	v.SetEnvPrefix(envPrefix)
-
-	// Bind environment variables to their equivalent keys with underscores
-	//v.SetEnvKeyReplacer(strings.NewReplacer("-", "_"))
-	v.AutomaticEnv()
-
-	// Bind the current command's flags to viper
+	// 3. Cmd flags. Bind the current command's flags to viper
 	bindFlags(cmd, v)

+	// Dump Viper values to config struct
+	err = v.Unmarshal(&config)
+	if err != nil {
+		return fmt.Errorf("cannot unmarshall config: %v", err)
+	}
+
+	if config.App.IsDebug {
+		logrus.Debug("Viper config:")
+		v.Debug()
+	}
 	return nil
 }

 func init() {
-	RootCmd.PersistentFlags().IntVarP(&appConf.Port, "port", "p", 7070, "Port number to run server")
-	RootCmd.PersistentFlags().StringVarP(&appConf.Host, "host", "a", "127.0.0.1", "Host address to run server")
-	RootCmd.PersistentFlags().IntVarP(&appConf.Timeout, "timeout", "t", 30, "Timeout to fail request")
-	RootCmd.PersistentFlags().StringVarP(&appConf.ConfigPath, "config", "c", "./config.yaml", "Configuration file path")
-	RootCmd.PersistentFlags().BoolVarP(&appConf.IsVerbose, "verbose", "v", false, "Use verbose output")
-	RootCmd.PersistentFlags().BoolVarP(&appConf.IsDebug, "debug", "d", false, "Use debug output. Disable headless browser")
-	RootCmd.PersistentFlags().BoolVarP(&appConf.IsBrowserHead, "head", "", false, "Enable browser UI")
-	RootCmd.PersistentFlags().BoolVarP(&appConf.IsLeakless, "leakless", "l", false, "Use leakless mode to insure browser instances are closed after search")
-	RootCmd.PersistentFlags().BoolVarP(&appConf.IsRawRequests, "raw", "r", false, "Disable browser usage, use HTTP requests")
-	RootCmd.PersistentFlags().BoolVarP(&appConf.IsLeaveHead, "leave", "", false, "Leave browser and tabs opened after search is made")
+	RootCmd.PersistentFlags().IntVarP(&config.App.Port, "port", "p", 7070, "Port number to run server")
+	RootCmd.PersistentFlags().StringVarP(&config.App.Host, "host", "a", "127.0.0.1", "Host address to run server")
+	RootCmd.PersistentFlags().IntVarP(&config.App.Timeout, "timeout", "t", 30, "Timeout to fail request")
+	RootCmd.PersistentFlags().StringVarP(&config.App.ConfigPath, "config", "c", "", "Configuration file path")
+	RootCmd.PersistentFlags().BoolVarP(&config.App.IsVerbose, "verbose", "v", false, "Use verbose output")
+	RootCmd.PersistentFlags().BoolVarP(&config.App.IsDebug, "debug", "d", false, "Use debug output. Disable headless browser")
+	RootCmd.PersistentFlags().BoolVarP(&config.App.IsBrowserHead, "head", "", false, "Enable browser UI")
+	RootCmd.PersistentFlags().BoolVarP(&config.App.IsLeakless, "leakless", "l", false, "Use leakless mode to insure browser instances are closed after search")
+	RootCmd.PersistentFlags().BoolVarP(&config.App.IsRawRequests, "raw", "r", false, "Disable browser usage, use HTTP requests")
+	RootCmd.PersistentFlags().BoolVarP(&config.App.IsLeaveHead, "leave", "", false, "Leave browser and tabs opened after search is made")
+	RootCmd.PersistentFlags().StringVarP(&config.Config2Capcha.ApiKey, "2captcha_key", "", "", "2 captcha api key")
+	RootCmd.PersistentFlags().StringVarP(&config.App.ProxyURL, "proxy", "x", "", "HTTP or Socks5 proxy URL (e.g. http://user:pass@127.0.0.1:8080)")
+	RootCmd.PersistentFlags().BoolVarP(&config.App.IsStealth, "stealth", "s", false, "Use stealth browser plugin")
 }
--- a/cmd/search.go
+++ b/cmd/search.go
@@ -26,12 +26,14 @@ func search(cmd *cobra.Command, args []string) {
 	var err error
 	engineType := args[0]
 	query := core.Query{
-		Text:  args[1],
-		Limit: 10,
+		Text:     args[1],
+		Limit:    10,
+		ProxyURL: config.App.ProxyURL,
+		Insecure: config.App.Insecure,
 	}
 	results := []core.SearchResult{}

-	if appConf.IsRawRequests {
+	if config.App.IsRawRequests {
 		results, err = searchRaw(engineType, query)
 	} else {
 		results, err = searchBrowser(engineType, query)
@@ -54,13 +56,17 @@ func searchBrowser(engineType string, query core.Query) ([]core.SearchResult, er
 	var engine core.SearchEngine

 	opts := core.BrowserOpts{
-		IsHeadless:    !appConf.IsBrowserHead, // Disable headless if browser head mode is set
-		IsLeakless:    appConf.IsLeakless,
-		Timeout:       time.Second * time.Duration(appConf.Timeout),
-		LeavePageOpen: appConf.IsLeaveHead,
+		IsHeadless:          !config.App.IsBrowserHead, // Disable headless if browser head mode is set
+		IsLeakless:          config.App.IsLeakless,
+		Timeout:             time.Second * time.Duration(config.App.Timeout),
+		LeavePageOpen:       config.App.IsLeaveHead,
+		CaptchaSolverApiKey: config.Config2Capcha.ApiKey,
+		ProxyURL:            config.App.ProxyURL,
+		Insecure:            config.App.Insecure,
+		UseStealth:          config.App.IsStealth,
 	}

-	if appConf.IsDebug {
+	if config.App.IsDebug {
 		opts.IsHeadless = false
 	}

@@ -71,11 +77,11 @@ func searchBrowser(engineType string, query core.Query) ([]core.SearchResult, er

 	switch strings.ToLower(engineType) {
 	case "yandex":
-		engine = yandex.New(*browser)
+		engine = yandex.New(*browser, config.YandexConfig)
 	case "google":
-		engine = google.New(*browser)
+		engine = google.New(*browser, config.GoogleConfig)
 	case "baidu":
-		engine = baidu.New(*browser)
+		engine = baidu.New(*browser, config.BaiduConfig)
 	default:
 		logrus.Infof("No `%s` search engine found", engineType)
 	}
--- a/cmd/serve.go
+++ b/cmd/serve.go
@@ -1,6 +1,7 @@
 package cmd

 import (
+	"fmt"
 	"time"

 	"github.com/karust/openserp/baidu"
@@ -9,8 +10,48 @@ import (
 	"github.com/karust/openserp/yandex"
 	"github.com/sirupsen/logrus"
 	"github.com/spf13/cobra"
+	"golang.org/x/time/rate"
 )

+// rawEngine implements SearchEngine interface for raw HTTP requests
+type rawEngine struct {
+	name string
+}
+
+func (r *rawEngine) Search(q core.Query) ([]core.SearchResult, error) {
+	// Inject proxy settings from config
+	q.ProxyURL = config.App.ProxyURL
+	q.Insecure = config.App.Insecure
+
+	switch r.name {
+	case "google":
+		return google.Search(q)
+	case "yandex":
+		return yandex.Search(q)
+	case "baidu":
+		return baidu.Search(q)
+	default:
+		return nil, fmt.Errorf("unsupported engine: %s", r.name)
+	}
+}
+
+func (r *rawEngine) SearchImage(q core.Query) ([]core.SearchResult, error) {
+	return nil, fmt.Errorf("image search is not supported in raw mode for %s", r.name)
+}
+
+func (r *rawEngine) Name() string {
+	return r.name
+}
+
+func (r *rawEngine) IsInitialized() bool {
+	return true
+}
+
+func (r *rawEngine) GetRateLimiter() *rate.Limiter {
+	// Use default rate limiter for raw requests
+	return rate.NewLimiter(rate.Every(time.Second), 5)
+}
+
 var serveCMD = &cobra.Command{
 	Use:     "serve",
 	Aliases: []string{"listen"},
@@ -20,27 +61,43 @@ var serveCMD = &cobra.Command{
 }

 func serve(cmd *cobra.Command, args []string) {
-	opts := core.BrowserOpts{
-		IsHeadless:    !appConf.IsBrowserHead, // Disable headless if browser head mode is set
-		IsLeakless:    appConf.IsLeakless,
-		Timeout:       time.Second * time.Duration(appConf.Timeout),
-		LeavePageOpen: appConf.IsLeaveHead,
+	if config.App.IsRawRequests {
+		logrus.Warn("Browserless results are very inconsistent or may not even work!")
+		serv := core.NewServer(config.App.Host, config.App.Port,
+			&rawEngine{name: "google"},
+			&rawEngine{name: "yandex"},
+			&rawEngine{name: "baidu"},
+		)
+		serv.Listen()
+		return
 	}

-	if appConf.IsDebug {
+	opts := core.BrowserOpts{
+		IsHeadless:          !config.App.IsBrowserHead, // Disable headless if browser head mode is set
+		IsLeakless:          config.App.IsLeakless,
+		Timeout:             time.Second * time.Duration(config.App.Timeout),
+		LeavePageOpen:       config.App.IsLeaveHead,
+		CaptchaSolverApiKey: config.Config2Capcha.ApiKey,
+		ProxyURL:            config.App.ProxyURL,
+		Insecure:            config.App.Insecure,
+		UseStealth:          config.App.IsBrowserHead,
+	}
+
+	if config.App.IsDebug {
 		opts.IsHeadless = false
 	}

 	browser, err := core.NewBrowser(opts)
 	if err != nil {
 		logrus.Error(err)
+		return
 	}

-	yand := yandex.New(*browser)
-	gogl := google.New(*browser)
-	baidu := baidu.New(*browser)
+	yand := yandex.New(*browser, config.YandexConfig)
+	gogl := google.New(*browser, config.GoogleConfig)
+	baidu := baidu.New(*browser, config.BaiduConfig)

-	serv := core.NewServer(appConf.Host, appConf.Port, gogl, yand, baidu)
+	serv := core.NewServer(config.App.Host, config.App.Port, gogl, yand, baidu)
 	serv.Listen()
 }

--- a/config.yaml
+++ b/config.yaml
@@ -5,4 +5,23 @@ app:
  verbose: true
  timeout: 15
  head: false
-  leakless: true
+  leakless: false
+  leave_head: false
+  stealth: false
+  insecure: false
+
+2captcha:
+  apikey: "123123123123123"
+
+google:
+  rate_requests: 4 # Number of requests per Minute
+  rate_burst: 2 # Number of non-ratelimited requests per Minute
+  captcha: true
+
+yandex:
+  rate_requests: 4
+  rate_burst: 2
+
+baidu:
+  rate_requests: 4
+  rate_burst: 2
--- a/core/browser.go
+++ b/core/browser.go
@@ -1,9 +1,10 @@
 package core

 import (
+	"fmt"
+	"net/url"
 	"time"

-	"github.com/corpix/uarand"
 	"github.com/go-rod/rod"
 	"github.com/go-rod/rod/lib/devices"
 	"github.com/go-rod/rod/lib/launcher"
@@ -12,41 +13,77 @@ import (
 )

 type BrowserOpts struct {
-	IsHeadless    bool          // Use browser interface
-	IsLeakless    bool          // Force to kill browser
-	Timeout       time.Duration // Timeout
-	LanguageCode  string
-	WaitRequests  bool // Wait requests to complete after navigation
-	LeavePageOpen bool // Leave pages and browser open
+	IsHeadless          bool          // Use browser interface
+	IsLeakless          bool          // Force to kill browser
+	Timeout             time.Duration // Timeout
+	LanguageCode        string
+	WaitRequests        bool          // Wait requests to complete after navigation
+	LeavePageOpen       bool          // Leave pages and browser open
+	WaitLoadTime        time.Duration // Time to wait till page loads
+	CaptchaSolverApiKey string        // 2Captcha api key
+	ProxyURL            string        // Proxy URL
+	Insecure            bool          // Allow insecure TLS connections
+	UseStealth          bool          // Use go-rod stealth plugin
+
 }

 // Initialize browser parameters with default values if they are not set
-func (o *BrowserOpts) Init() {
+func (o *BrowserOpts) Check() {
 	if o.Timeout == 0 {
 		o.Timeout = time.Second * 30
 	}

-	if o.LanguageCode == "" {
-		o.LanguageCode = "en"
+	if o.WaitLoadTime == 0 {
+		o.WaitLoadTime = time.Second * 2
 	}
 }

 type Browser struct {
 	BrowserOpts
-	browserAddr string
-	browser     *rod.Browser
+	browserAddr   string
+	browser       *rod.Browser
+	CaptchaSolver *CaptchaSolver
 }

 func NewBrowser(opts BrowserOpts) (*Browser, error) {
-	opts.Init()
+	opts.Check()
 	logrus.Debugf("Browser options: %+v", opts)

 	path, has := launcher.LookPath()
 	logrus.Debug("Browser found: ", has)

+	// Create launcher
+	l := launcher.New().Bin(path).Leakless(opts.IsLeakless).Headless(opts.IsHeadless)
+
+	// Configure proxy if specified
+	if opts.ProxyURL != "" {
+		proxyUrl, err := url.Parse(opts.ProxyURL)
+		if err != nil {
+			return nil, fmt.Errorf("invalid proxy URL: %v", err)
+		}
+
+		// Make sure the proxy URL includes the scheme when passed to launcher
+		// This ensures proper handling of SOCKS5 proxies
+		proxyStr := proxyUrl.String()
+		logrus.Debugf("Setting up proxy: %s", proxyStr)
+		l = l.Proxy(proxyStr)
+
+		// Check if proxy has auth credentials
+		if proxyUrl.User != nil {
+			username := proxyUrl.User.Username()
+			logrus.Debugf("Using proxy authentication: %s:****", username)
+			// We'll handle auth in the Navigate method
+		}
+	}
+
 	var err error
 	b := Browser{BrowserOpts: opts}
-	b.browserAddr, err = launcher.New().Bin(path).Leakless(opts.IsLeakless).Headless(opts.IsHeadless).Launch()
+	b.browserAddr, err = l.Launch()
+
+	if opts.CaptchaSolverApiKey != "" {
+		b.CaptchaSolver = NewSolver(opts.CaptchaSolverApiKey)
+		logrus.Debug("Captcha solver initialized")
+	}

 	return &b, err
 }
@@ -61,31 +98,55 @@ func (b *Browser) IsInitialized() bool {
 }

 // Open URL
-func (b *Browser) Navigate(URL string) *rod.Page {
+func (b *Browser) Navigate(URL string) (*rod.Page, error) {
 	logrus.Debug("Navigate to: ", URL)

 	b.browser = rod.New().ControlURL(b.browserAddr)
 	b.browser.MustConnect()
-	//b.browser.SetCookies(nil)
+	b.browser.SetCookies(nil)
+
+	// Handle proxy authentication before any navigations
+	if b.ProxyURL != "" {
+		proxyUrl, _ := url.Parse(b.ProxyURL)
+
+		// Always ignore certificate errors when using proxies
+		// This fixes the ERR_CERT_AUTHORITY_INVALID error for SOCKS5 proxies
+		b.browser.MustIgnoreCertErrors(true)
+
+		if proxyUrl.User != nil {
+			username := proxyUrl.User.Username()
+			password, _ := proxyUrl.User.Password()
+			// Launch auth handler before any navigation occurs
+			go b.browser.MustHandleAuth(username, password)()
+		}
+	} else if b.Insecure {
+		// Still respect the insecure flag if no proxy is used
+		b.browser.MustIgnoreCertErrors(true)
+	}
+
+	var page *rod.Page
+	if b.UseStealth {
+		page = stealth.MustPage(b.browser)
+	} else {
+		page = b.browser.MustPage(URL)
+	}
+
+	page.MustEmulate(devices.Device{
+		AcceptLanguage: b.LanguageCode,
+	})
+
+	err := page.Navigate(URL)
+	if err != nil {
+		return nil, err
+	}

-	page := stealth.MustPage(b.browser)
 	wait := page.MustWaitRequestIdle()
-	page.Navigate(URL)
-
-	// causes bugs in google
+	// may cause bugs with google
 	if b.WaitRequests {
 		wait()
 	}

-	page.MustEmulate(devices.Device{
-		UserAgent:      uarand.GetRandom(),
-		AcceptLanguage: b.LanguageCode,
-	})
-
-	// Wait till page loads
-	time.Sleep(time.Second * 1)
-
-	return page
+	return page, nil
 }

 func (b *Browser) Close() error {
--- a/core/browser_test.go
+++ b/core/browser_test.go
@@ -27,3 +27,23 @@ func TestCreateBrowser(t *testing.T) {
 // 		t.Fatalf("Error failed initializing leakless browser: %s", err)
 // 	}
 // }
+
+// Manually observe test results for now
+func TestBot(t *testing.T) {
+
+	var err error
+	opts := BrowserOpts{IsHeadless: false, IsLeakless: true, LeavePageOpen: true}
+	browser, err = NewBrowser(opts)
+	if err != nil {
+		t.Fatalf("Error failed initializing browser: %s", err)
+	}
+
+	page, _ := browser.Navigate("https://bot.sannysoft.com")
+	page.MustScreenshotFullPage("./test/screenshot_bot.png")
+
+	page, _ = browser.Navigate("https://www.whatismybrowser.com/")
+	page.MustScreenshotFullPage("./test/screenshot_browser.png")
+
+	page, _ = browser.Navigate("https://abrahamjuliot.github.io/creepjs/")
+	page.MustScreenshotFullPage("./test/screenshot_creep.png")
+}
--- a/core/captcha.go
+++ b/core/captcha.go
@@ -0,0 +1,28 @@
+package core
+
+import (
+	api2captcha "github.com/2captcha/2captcha-go"
+)
+
+type CaptchaSolver struct {
+	client *api2captcha.Client
+}
+
+func NewSolver(apikey string) *CaptchaSolver {
+	cs := CaptchaSolver{}
+	cs.client = api2captcha.NewClient(apikey)
+	return &cs
+}
+
+func (cs *CaptchaSolver) SolveReCaptcha2(sitekey, pageUrl, dataS string) (string, error) {
+	cap := api2captcha.ReCaptcha{
+		SiteKey:   sitekey,
+		Url:       pageUrl,
+		DataS:     dataS,
+		Invisible: false,
+		Action:    "verify",
+	}
+	req := cap.ToRequest()
+	req.SetProxy("HTTPS", "login:password@IP_address:PORT")
+	return cs.client.Solve(req)
+}
--- a/core/captcha_test.go
+++ b/core/captcha_test.go
@@ -0,0 +1,20 @@
+package core
+
+import (
+	"testing"
+)
+
+var (
+	API_KEY = ""
+)
+
+func Test2Captcha(t *testing.T) {
+	solver := NewSolver(API_KEY)
+	sitekey := "6LfwuyUTAAAAAOAmoS0fdqijC2PbbdH4kjq62Y1b"
+	url := "https://www.google.com/sorry/index?continue=https://www.google.de/search%3Fhl%3DDE%26lr%3Dlang_de%26nfpr%3D1%26num%3D500%26pws%3D0%26q%3Dwhere%2Bwhy%2Beach&hl=DE&q=EgRegw55GObHiq4GIjDqmzFKayGXrS2-s9ooWfcskhpK8-6tIjWSaSvhxd3f5eAyUXj7lYq2DYLDXB8ASz0yAXJaAUM"
+	datas := "Ghk0n7ZQNDS0c7ES53eef_YBfSdfeXnyRD0p2OR0R4Dg91CUXKS_hio5Do6TpJ8sHhhOat_NymTASZGe1gqAjP7w9dSvhvRT7QXsrdziO3JPngLDSRzDdjT42GDcSbO0kzInlDPxe1yy2t4yifo9xHpMnlZU7pTVNTQUIXqOMLHAR-iERi6aoSQDQ4d-88-jW3LEinquxEut0OhHG2l2stwG9AnCmNvCsUNJda-H24saFlOh5csK9KNXeeQmpr6at52_skMIMiLXSlY56vYFVCRMkXLQdAM"
+	resp, err := solver.SolveReCaptcha2(sitekey, url, datas)
+	if err != nil || resp == "" {
+		t.Fatalf("Failed to solve recaptchaV2: %s", err)
+	}
+}
--- a/core/common.go
+++ b/core/common.go
@@ -2,16 +2,55 @@ package core

 import (
 	"errors"
+	"sort"
 	"strconv"
+	"time"

 	"github.com/gofiber/fiber/v2"
 )

+var ErrCaptcha = errors.New("captcha detected")
+var ErrSearchTimeout = errors.New("timeout. Cannot find element on page")
+
 type SearchResult struct {
 	Rank        int    `json:"rank"`
 	URL         string `json:"url"`
 	Title       string `json:"title"`
 	Description string `json:"description"`
+	Ad          bool   `json:"ad"`
+}
+
+func DeduplicateResults(results []SearchResult) []SearchResult {
+	unique := make(map[string]bool)
+	var deduped []SearchResult
+
+	for _, result := range results {
+		if result.URL == "" {
+			continue
+		}
+		if !unique[result.URL] {
+			unique[result.URL] = true
+			deduped = append(deduped, result)
+		}
+	}
+
+	sort.Slice(deduped, func(i, j int) bool {
+		return deduped[i].Rank < deduped[j].Rank
+	})
+	return deduped
+}
+
+func ConvertSearchResultsMap(searchResultsMap map[string]SearchResult) *[]SearchResult {
+	searchResults := []SearchResult{}
+
+	for _, v := range searchResultsMap {
+		searchResults = append(searchResults, v)
+	}
+
+	sort.Slice(searchResults, func(i, j int) bool {
+		return searchResults[i].Rank < searchResults[j].Rank
+	})
+	return &searchResults
 }

 type Query struct {
@@ -21,6 +60,9 @@ type Query struct {
 	Filetype     string // File extension to search.
 	Site         string // Search site
 	Limit        int    // Limit the number of results
+	Answers      bool   // Include question and answers from SERP page to results with negative indexes
+	ProxyURL     string // Proxy URL for raw requests
+	Insecure     bool   // Allow insecure TLS connections
 }

 func (q Query) IsEmpty() bool {
@@ -30,22 +72,57 @@ func (q Query) IsEmpty() bool {
 	return false
 }

-func (q *Query) InitFromContext(c *fiber.Ctx) error {
-	q.Text = c.Query("text")
-	q.LangCode = c.Query("lang")
-	q.DateInterval = c.Query("date")
-	q.Filetype = c.Query("file")
-	q.Site = c.Query("site")
+func (searchQuery *Query) InitFromContext(reqCtx *fiber.Ctx) error {
+	searchQuery.Text = reqCtx.Query("text")
+	searchQuery.LangCode = reqCtx.Query("lang")
+	searchQuery.DateInterval = reqCtx.Query("date")
+	searchQuery.Filetype = reqCtx.Query("file")
+	searchQuery.Site = reqCtx.Query("site")

-	limit, err := strconv.Atoi(c.Query("limit", "25"))
+	limit, err := strconv.Atoi(reqCtx.Query("limit", "25"))
 	if err != nil {
 		return err
 	}
-	q.Limit = limit
+	searchQuery.Limit = limit

-	if q.IsEmpty() {
-		return errors.New("Query cannot be empty")
+	searchQuery.Answers, err = strconv.ParseBool(reqCtx.Query("answers", "0"))
+	if err != nil {
+		return err
 	}

+	if searchQuery.IsEmpty() {
+		return errors.New("Query cannot be empty")
+	}
 	return nil
 }
+
+type SearchEngineOptions struct {
+	RateRequests    int   `mapstructure:"rate_requests"`
+	RateTime        int64 `mapstructure:"rate_seconds"`
+	RateBurst       int   `mapstructure:"rate_burst"`
+	SelectorTimeout int64 `mapstructure:"selector_timeout"` // CSS selector timeout in seconds
+	IsSolveCaptcha  bool  `mapstructure:"captcha"`
+}
+
+func (o *SearchEngineOptions) Init() {
+	if o.RateRequests == 0 {
+		o.RateRequests = 6
+	}
+	if o.RateTime == 0 {
+		o.RateTime = 60
+	}
+	if o.RateBurst == 0 {
+		o.RateBurst = 1
+	}
+	if o.SelectorTimeout == 0 {
+		o.SelectorTimeout = 5
+	}
+}
+
+func (o *SearchEngineOptions) GetRatelimit() time.Duration {
+	return (time.Duration(o.RateTime) * time.Second) / time.Duration(o.RateRequests)
+}
+
+func (o *SearchEngineOptions) GetSelectorTimeout() time.Duration {
+	return time.Duration(o.SelectorTimeout) * time.Second
+}
--- a/core/server.go
+++ b/core/server.go
@@ -1,17 +1,21 @@
 package core

 import (
+	"context"
 	"fmt"
 	"strings"

 	"github.com/gofiber/fiber/v2"
 	"github.com/sirupsen/logrus"
+	"golang.org/x/time/rate"
 )

 type SearchEngine interface {
 	Search(Query) ([]SearchResult, error)
+	SearchImage(Query) ([]SearchResult, error)
 	IsInitialized() bool
 	Name() string
+	GetRateLimiter() *rate.Limiter
 }

 type Server struct {
@@ -28,21 +32,69 @@ func NewServer(host string, port int, searchEngines ...SearchEngine) *Server {

 	for _, engine := range searchEngines {
 		locEngine := engine
+		limiter := engine.GetRateLimiter()
+
 		serv.app.Get(fmt.Sprintf("/%s/search", strings.ToLower(locEngine.Name())), func(c *fiber.Ctx) error {
 			q := Query{}
 			err := q.InitFromContext(c)
-
 			if err != nil {
 				logrus.Errorf("Error while setting %s query: %s", locEngine.Name(), err)
 				return err
 			}

+			err = limiter.Wait(context.Background())
+			if err != nil {
+				logrus.Errorf("Ratelimiter error during %s query: %s", locEngine.Name(), err)
+			}
+
 			res, err := locEngine.Search(q)
 			if err != nil {
+				switch err {
+				case ErrCaptcha:
+					err = fmt.Errorf("captcha found, please stop sending requests for a while\n%s", err)
+				case ErrSearchTimeout:
+					err = fmt.Errorf("%s", err)
+				}
+
 				logrus.Errorf("Error during %s search: %s", locEngine.Name(), err)
+				return fiber.NewError(fiber.StatusServiceUnavailable, err.Error())
+			}
+
+			return c.JSON(res)
+		})
+
+		serv.app.Get(fmt.Sprintf("/%s/image", strings.ToLower(locEngine.Name())), func(c *fiber.Ctx) error {
+			q := Query{}
+			err := q.InitFromContext(c)
+			if err != nil {
+				logrus.Errorf("Error while setting %s query: %s", locEngine.Name(), err)
 				return err
 			}

+			err = limiter.Wait(context.Background())
+			if err != nil {
+				logrus.Errorf("Ratelimiter error during %s query: %s", locEngine.Name(), err)
+			}
+
+			res, err := locEngine.SearchImage(q)
+
+			if err != nil && len(res) > 0 {
+				c.Status(503)
+				return c.JSON(res)
+			}
+
+			if err != nil {
+				switch err {
+				case ErrCaptcha:
+					err = fmt.Errorf("captcha found, please stop sending requests for a while: %s", err)
+				case ErrSearchTimeout:
+					err = fmt.Errorf("%s", err)
+				}
+
+				logrus.Errorf("Error during %s search: %s", locEngine.Name(), err)
+				return fiber.NewError(fiber.StatusServiceUnavailable, err.Error())
+			}
+
 			return c.JSON(res)
 		})
 	}
--- a/core/server_test.go
+++ b/core/server_test.go
@@ -4,6 +4,8 @@ import (
 	"fmt"
 	"testing"
 	"time"
+
+	"golang.org/x/time/rate"
 )

 var (
@@ -25,6 +27,12 @@ func (SeMock) IsInitialized() bool {
 func (s SeMock) Search(q Query) (res []SearchResult, err error) {
 	return []SearchResult{{Title: s.EngineName}}, nil
 }
+func (s SeMock) SearchImage(q Query) (res []SearchResult, err error) {
+	return []SearchResult{{Title: s.EngineName}}, nil
+}
+func (s SeMock) GetRateLimiter() *rate.Limiter {
+	return nil
+}

 func TestCreateServer(t *testing.T) {
 	se1 := SeMock{"mock_engine_1"}
--- a/docker-compose.yaml
+++ b/docker-compose.yaml
@@ -3,7 +3,16 @@ version: '3'
 services:
  openserp:
    container_name: serp
-    image: openserp
+    build:
+      context: .
    ports: 
      - 7000:7000
-    command: serve -a 0.0.0.0 -p 7000
+    command: serve -l
+    #volumes:
+    #  - ./config.yaml:/usr/src/app/config.yaml
+    environment:
+      OPENSERP_APP_HOST: "0.0.0.0"
+      OPENSERP_APP_PORT: 7000
+      OPENSERP_BAIDU_RATE_REQUESTS: 6   # Number of requests per Minute
+      OPENSERP_BAIDU_RATE_BURST: 2      # Number of non-ratelimited requests per Minute
+    
--- a/go.mod
+++ b/go.mod
@@ -3,6 +3,7 @@ module github.com/karust/openserp
 go 1.20

 require (
+	github.com/2captcha/2captcha-go v1.1.4
 	github.com/PuerkitoBio/goquery v1.8.1
 	github.com/corpix/uarand v0.2.0
 	github.com/go-rod/rod v0.113.3
@@ -12,6 +13,7 @@ require (
 	github.com/spf13/cobra v1.7.0
 	github.com/spf13/pflag v1.0.5
 	github.com/spf13/viper v1.16.0
+	golang.org/x/time v0.3.0
 )

 require (
--- a/go.sum
+++ b/go.sum
@@ -36,6 +36,8 @@ cloud.google.com/go/storage v1.8.0/go.mod h1:Wv1Oy7z6Yz3DshWRJFhqM/UCfaWIRTdp0RX
 cloud.google.com/go/storage v1.10.0/go.mod h1:FLPqc6j+Ki4BU591ie1oL6qBQGu2Bl/tZ9ullr3+Kg0=
 cloud.google.com/go/storage v1.14.0/go.mod h1:GrKmX003DSIwi9o29oFT7YDnHYwZoctc3fOKtUw0Xmo=
 dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7gZCb22OMCxBHrMx7a5I7Hp++hsVxbQ4BYO7hU=
+github.com/2captcha/2captcha-go v1.1.4 h1:Fm62VPvVhEHYQ8AI+/uquiTg41ml9f8ASjUkVuBvHcE=
+github.com/2captcha/2captcha-go v1.1.4/go.mod h1:hYOq+KVOq/0zAG6OTYW7Y313qDkHv58CcaOyjdBQSco=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
 github.com/PuerkitoBio/goquery v1.8.1 h1:uQxhNlArOIdbrH1tr0UXwdVFgDcZDrZVdcpygAcwmWM=
@@ -421,6 +423,8 @@ golang.org/x/text v0.10.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
+golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
+golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
--- a/google/search.go
+++ b/google/search.go
@@ -2,56 +2,135 @@ package google

 import (
 	"errors"
+	"fmt"
 	"regexp"
 	"strconv"
 	"strings"
 	"time"

 	"github.com/go-rod/rod"
+	"github.com/go-rod/rod/lib/proto"
 	"github.com/karust/openserp/core"
 	"github.com/sirupsen/logrus"
+	"golang.org/x/time/rate"
 )

 type Google struct {
 	core.Browser
-	findNumRgxp  *regexp.Regexp
-	checkTimeout time.Duration
+	core.SearchEngineOptions
+	rgxpGetDigits *regexp.Regexp
 }

-func New(browser core.Browser) *Google {
+func New(browser core.Browser, opts core.SearchEngineOptions) *Google {
 	gogl := Google{Browser: browser}
-	gogl.checkTimeout = time.Second * 5
-	gogl.findNumRgxp = regexp.MustCompile("\\d")
+	opts.Init()
+	gogl.SearchEngineOptions = opts
+	gogl.rgxpGetDigits = regexp.MustCompile("\\d")
 	return &gogl
 }
+
 func (gogl *Google) Name() string {
 	return "google"
 }

-func (gogl *Google) FindTotalResults(page *rod.Page) (int, error) {
-	resultsStats, err := page.Timeout(gogl.checkTimeout).Search("div#result-stats")
+func (gogl *Google) GetRateLimiter() *rate.Limiter {
+	ratelimit := rate.Every(gogl.GetRatelimit())
+	return rate.NewLimiter(ratelimit, gogl.RateBurst)
+}
+
+func (gogl *Google) getTotalResults(page *rod.Page) (int, error) {
+	resultsStats, err := page.Timeout(gogl.GetSelectorTimeout()).Search("div#result-stats")
 	if err != nil {
 		return 0, errors.New("Result stats not found: " + err.Error())
 	}
-	stats, err := resultsStats.First.Text()
+
+	statsText, err := resultsStats.First.Text()
 	if err != nil {
 		return 0, errors.New("Cannot extract result stats text: " + err.Error())
 	}

-	// Escape moment with `seconds` and extract digits
-	allNums := gogl.findNumRgxp.FindAllString(stats[:len(stats)-15], -1)
-	stats = strings.Join(allNums, "")
+	if len(statsText) == 0 {
+		return 0, nil
+	}

-	total, err := strconv.Atoi(stats)
+	// Remove search time seconds info from the end
+	if len(statsText) > 15 {
+		statsText = statsText[:len(statsText)-15]
+	}
+
+	foundDigits := gogl.rgxpGetDigits.FindAllString(statsText, -1)
+	totalNum := strings.Join(foundDigits, "")
+	total, err := strconv.Atoi(totalNum)
 	if err != nil {
 		return 0, err
 	}
 	return total, nil
 }

+func (gogl *Google) solveCaptcha(page *rod.Page, sitekey, datas string) bool {
+	logrus.Debugf("Solve google Captcha: sitekey=%s, datas=%s, url=%s", sitekey, datas, page.MustInfo().URL)
+
+	resp, err := gogl.CaptchaSolver.SolveReCaptcha2(sitekey, page.MustInfo().URL)
+	if err != nil {
+		logrus.Errorf("Error solving google captcha: %s", err)
+		return false
+	}
+
+	logrus.Debug("Resp:", resp)
+	_, err = page.Eval(fmt.Sprintf(`;(() => { document.getElementById("g-recaptcha-response").innerHTML="%s"; submitCallback(); })();`, resp))
+	if err != nil {
+		logrus.Errorf("Error setting captcha response: %s", err)
+		return false
+	}
+
+	return true
+}
+
+func (gogl *Google) checkCaptcha(page *rod.Page) bool {
+	captchaDiv, err := page.Timeout(gogl.GetSelectorTimeout()).Search("div[data-sitekey]")
+	if err != nil {
+		return false
+	}
+
+	sitekey, err := captchaDiv.First.Attribute("data-sitekey")
+	if err != nil {
+		logrus.Errorf("Cannot get Google captcha sitekey: %s", err)
+		return false
+	}
+
+	dataS, err := captchaDiv.First.Attribute("data-s")
+	if err != nil {
+		logrus.Errorf("Cannot get Google captcha datas: %s", err)
+		return false
+	}
+
+	if gogl.IsSolveCaptcha {
+		return !gogl.solveCaptcha(page, *sitekey, *dataS)
+	}
+	return true
+}
+
 func (gogl *Google) preparePage(page *rod.Page) {
 	// Remove "similar queries" lists
-	page.Eval(";(() => { document.querySelectorAll(`div[data-initq]`).forEach( el => el.remove());  })();")
+	_, err := page.Eval(";(() => { document.querySelectorAll(`div[data-initq]`).forEach( el => el.remove());  })();")
+	if err != nil {
+		logrus.Errorf("Error preparing the page: %s", err)
+	}
+}
+
+func (gogl *Google) acceptCookies(page *rod.Page) {
+	diaglogBtns, err := page.Timeout(gogl.Timeout / 10).Search("div[role='dialog'][aria-modal] button")
+	if err != nil {
+		logrus.Errorf("Cannot find cookie consent: %s", err)
+		return
+	}
+	btnElms, err := diaglogBtns.All()
+	if err != nil {
+		logrus.Errorf("Cannot get cookie consent buttons: %s", err)
+		return
+	}
+	btnElms[3].Click(proto.InputMouseButtonLeft, 1)
+
 }

 func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
@@ -64,74 +143,310 @@ func (gogl *Google) Search(query core.Query) ([]core.SearchResult, error) {
 	if err != nil {
 		return nil, err
 	}
+	page, err := gogl.Navigate(url)
+	if err != nil {
+		return nil, err
+	}

-	page := gogl.Navigate(url)
+	defer gogl.close(page)
 	gogl.preparePage(page)

-	totalResults, err := gogl.FindTotalResults(page)
+	// Check first if there captcha
+	if gogl.checkCaptcha(page) {
+		logrus.Errorf("Google captcha occurred during: %s", url)
+		return nil, core.ErrCaptcha
+	}
+
+	// Accept cookie consent to get google answers
+	if query.Answers {
+		gogl.acceptCookies(page)
+	}
+
+	// Find all results using stable attributes
+	results, err := page.Timeout(gogl.Timeout).Search("div[data-hveid][data-ved]")
 	if err != nil {
-		return nil, err
-	}
-	logrus.Tracef("%d total results found", totalResults)
-
-	if totalResults == 0 {
-		return searchResults, nil
+		logrus.Errorf("Cannot parse search results: %s", err)
+		return nil, core.ErrSearchTimeout
 	}

-	results, err := page.Timeout(gogl.Timeout).Search("div[data-hveid][data-ved][lang]")
+	if results == nil {
+		return nil, nil
+	}
+
+	totalResults, err := gogl.getTotalResults(page)
+	if err != nil {
+		logrus.Errorf("Error capturing total results: %v", err)
+	}
+	logrus.Infof("%d SERP results", totalResults)
+
+	searchResultElems, err := results.All()
 	if err != nil {
 		return nil, err
 	}

-	resultElements, err := results.All()
-	if err != nil {
-		return nil, err
-	}
+	rank := 0
+	for _, resEl := range searchResultElems {
+		srchRes := core.SearchResult{}

-	for i, r := range resultElements {
-		// Get URL
-		link, err := r.Element("a")
-		if err != nil {
+		attrs := strings.Join(resEl.MustDescribe().Attributes, " ")
+
+		if strings.Contains(attrs, "data-text-ad") {
+			// 1. Parse ads
+
+			srchRes.Ad = true
+
+			// Get URL
+			link, err := resEl.Element("a")
+			if err != nil {
+				logrus.Debug("No link found")
+				continue
+			}
+			link.MoveMouseOut()
+
+			href, err := link.Property("href")
+			if err != nil {
+				logrus.Debug("No `href` tag found")
+				continue
+			}
+			srchRes.URL = href.String()
+
+			// Get title
+			srchRes.Title = link.MustText()
+
+			// Get description
+			text := resEl.MustText()
+			textSliced := strings.Split(text, "\n")
+			srchRes.Description = strings.Join(textSliced[4:], "\n")
+			rank += 1
+
+		} else if query.Answers && strings.Contains(attrs, "data-ulkwtsb") && !strings.Contains(attrs, "data-ispaa") {
+			// 2. Parse answer boxes
+			answerEls, err := resEl.Page().Search("div[data-hveid][data-ulkwtsb] div[data-q]")
+			if err != nil {
+				logrus.Debugf("Error while parsing answer box 1: %s", err.Error())
+				continue
+			}
+
+			answers, err := answerEls.All()
+			if err != nil {
+				logrus.Debugf("Error while parsing answer box 2: %s", err.Error())
+				continue
+			}
+
+			logrus.Infof("%d answers found", len(answers))
+
+			// Unvail answer contents
+			for _, answ := range answers {
+				answ.Click(proto.InputMouseButtonLeft, 1)
+				answ.Focus()
+				//answ.Page().WaitRepaint()
+			}
+			time.Sleep(time.Millisecond * 2000)
+
+			for i, answ := range answers {
+				answerText := strings.Split(answ.MustText(), "\n")
+				if len(answerText) < 2 {
+					logrus.Debugf("Short answer text: %s", answerText)
+					continue
+				}
+
+				// Get URL
+				link, err := answ.Element("a")
+				if err != nil {
+					logrus.Debug("No answer link found")
+					continue
+				}
+				link.MoveMouseOut()
+
+				href, err := link.Property("href")
+				if err != nil {
+					logrus.Debug("No answer `href` tag found")
+					continue
+				}
+				srchRes.URL = href.String()
+				srchRes.Title = answerText[0]
+				srchRes.Description = strings.Join(answerText[1:len(answerText)-2], "\n")
+				srchRes.Rank = -1 * (i + 1)
+				searchResults = append(searchResults, srchRes)
+			}
 			continue
-		}
-		linkText, err := link.Property("href")
-		if err != nil {
-			logrus.Error("No `href` tag found")
-		}
+		} else if strings.Contains(attrs, "data-ved") {
+			// Parse regular search results
+			// Get title from h3
+			titleTag, err := resEl.Element("h3")
+			if err != nil {
+				continue
+			}
+			srchRes.Title, _ = titleTag.Text()

-		// Get title
-		titleTag, err := link.Element("h3")
-		if err != nil {
-			logrus.Error("No `h3` tag found")
+			// Get URL from parent link of h3
+			link, err := titleTag.Parent()
+			if err == nil && link.MustMatches("a") {
+				href, _ := link.Property("href")
+				srchRes.URL = href.String()
+			}
+
+			// Skip if URL is empty
+			if srchRes.URL == "" {
+				continue
+			}
+
+			// Get description using multiple fallback strategies
+			desc := ""
+			if descTag, err := resEl.Element("div[data-sncf='1'] div"); err == nil {
+				desc = descTag.MustText()
+			} else if descTag, err := resEl.Element("div.VwiC3b"); err == nil {
+				desc = descTag.MustText()
+			} else {
+				// Structural fallback
+				parent, err := titleTag.Parent()
+				if err == nil {
+					parent, err = parent.Parent()
+					if err == nil {
+						parent, err = parent.Parent()
+						if err == nil {
+							if descTag, err := parent.Next(); err == nil {
+								if descDiv, err := descTag.Element("div"); err == nil {
+									desc = descDiv.MustText()
+								}
+							}
+						}
+					}
+				}
+			}
+			srchRes.Description = desc
+
+			rank += 1
+			srchRes.Rank = rank
+			searchResults = append(searchResults, srchRes)
 			continue
-		}

-		title, err := titleTag.Text()
-		if err != nil {
-			logrus.Error("Cannot extract text from title")
-			title = "No title"
-		}
-
-		// Get description
-		// doesn't catch all
-		descTag, err := r.Element(`div[data-sncf~="1"]`)
-		desc := ""
-		if err != nil {
-			logrus.Trace(`No description 'div[data-sncf~="1"]' tag found`)
 		} else {
-			desc = descTag.MustText()
+			//fmt.Println(i, attrs)
+			continue
 		}

-		gR := core.SearchResult{Rank: i + 1, URL: linkText.String(), Title: title, Description: desc}
-		searchResults = append(searchResults, gR)
+		srchRes.Rank = rank
+		searchResults = append(searchResults, srchRes)
 	}

+	return core.DeduplicateResults(searchResults), nil
+}
+
+func (gogl *Google) SearchImage(query core.Query) ([]core.SearchResult, error) {
+	logrus.Tracef("Start Google Image search, query: %+v", query)
+
+	searchResultsMap := map[string]core.SearchResult{}
+	url, err := BuildImageURL(query)
+	if err != nil {
+		return nil, err
+	}
+
+	page, err := gogl.Navigate(url)
+	if err != nil {
+		return nil, err
+	}
+
+	defer gogl.close(page)
+
+	for len(searchResultsMap) < query.Limit {
+		page.WaitLoad()
+		page.Mouse.Scroll(0, 1000000, 1)
+		page.WaitLoad()
+
+		results, err := page.Timeout(gogl.Timeout).Search("div[data-hveid][data-ved][jsaction]")
+		if err != nil {
+			logrus.Errorf("Cannot parse search results: %s", err)
+			return *core.ConvertSearchResultsMap(searchResultsMap), core.ErrSearchTimeout
+		}
+
+		// Check why no results
+		if results == nil {
+			if gogl.checkCaptcha(page) {
+				logrus.Errorf("Google captcha occurred during: %s", url)
+				return *core.ConvertSearchResultsMap(searchResultsMap), core.ErrCaptcha
+			}
+			return *core.ConvertSearchResultsMap(searchResultsMap), err
+		}
+
+		resultElements, err := results.All()
+		if err != nil {
+			return *core.ConvertSearchResultsMap(searchResultsMap), err
+		}
+
+		if len(resultElements) < len(searchResultsMap) {
+			continue
+		}
+
+		for i, r := range resultElements {
+			// TODO: parse AF_initDataCallback to optimize instead of this?
+			err := r.Click(proto.InputMouseButtonRight, 1)
+			if err != nil {
+				logrus.Error("Error clicking")
+				continue
+			}
+
+			dataVed, err := r.Attribute("data-ved")
+			if err != nil {
+				logrus.Error("Cannot find `data-ved` attr")
+				continue
+			}
+
+			// If already have image with this ID
+			if _, ok := searchResultsMap[*dataVed]; ok {
+				continue
+			}
+
+			// Get URLs
+			link, err := r.Element("a:not([ping])")
+			if err != nil {
+				continue
+			}
+
+			linkText, err := link.Property("href")
+			if err != nil {
+				logrus.Error("No `href` tag found")
+			}
+
+			imgSrc, err := parseSourceImageURL(linkText.String())
+			if err != nil {
+				logrus.Errorf("Cannot parse image href: %v", err)
+				continue
+			}
+
+			// Get title
+			titleTag, err := r.Element("h3")
+			if err != nil {
+				logrus.Error("No `h3` tag found")
+				continue
+			}
+
+			title, err := titleTag.Text()
+			if err != nil {
+				logrus.Error("Cannot extract text from title")
+				title = "No title"
+			}
+
+			gR := core.SearchResult{
+				Rank:        i + 1,
+				URL:         imgSrc.OriginalURL,
+				Title:       title,
+				Description: fmt.Sprintf("Height:%v, Width:%v, Source Page: %v", imgSrc.Height, imgSrc.Width, imgSrc.PageURL),
+			}
+			searchResultsMap[*dataVed] = gR
+
+			r.Remove()
+		}
+	}
+
+	return *core.ConvertSearchResultsMap(searchResultsMap), nil
+}
+
+func (gogl *Google) close(page *rod.Page) {
 	if !gogl.Browser.LeavePageOpen {
-		err = page.Close()
+		err := page.Close()
 		if err != nil {
 			logrus.Error(err)
 		}
 	}
-
-	return searchResults, nil
 }
--- a/google/search_raw.go
+++ b/google/search_raw.go
@@ -1,8 +1,11 @@
 package google

 import (
+	"crypto/tls"
 	"net/http"
+	"net/url"
 	"strings"
+	"time"

 	"github.com/PuerkitoBio/goquery"
 	"github.com/corpix/uarand"
@@ -10,8 +13,27 @@ import (
 	"github.com/sirupsen/logrus"
 )

-func googleRequest(searchURL string) (*http.Response, error) {
-	baseClient := &http.Client{}
+func googleRequest(searchURL string, query core.Query) (*http.Response, error) {
+	// Create HTTP transport with proxy
+	transport := &http.Transport{}
+	if query.ProxyURL != "" {
+		proxyUrl, err := url.Parse(query.ProxyURL)
+		if err != nil {
+			return nil, err
+		}
+		transport.Proxy = http.ProxyURL(proxyUrl)
+	}
+
+	// Set insecure TLS
+	if query.Insecure {
+		transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
+	}
+
+	baseClient := &http.Client{
+		Transport: transport,
+		Timeout:   time.Second * 10,
+	}
+
 	req, err := http.NewRequest("GET", searchURL, nil)
 	if err != nil {
 		return nil, err
@@ -26,7 +48,7 @@ func googleRequest(searchURL string) (*http.Response, error) {
 }

 func googleResultParser(response *http.Response) ([]core.SearchResult, error) {
-	doc, err := goquery.NewDocumentFromResponse(response)
+	doc, err := goquery.NewDocumentFromReader(response.Body)
 	if err != nil {
 		return nil, err
 	}
@@ -34,23 +56,49 @@ func googleResultParser(response *http.Response) ([]core.SearchResult, error) {
 	results := []core.SearchResult{}
 	rank := 1

-	// Get individual results
-	sel := doc.Find("div.g")
+	// Use data attributes instead of class names to find results
+	// Both old and new DOM have data-hveid and data-ved attributes
+	sel := doc.Find("div[data-hveid][data-ved]")

 	for i := range sel.Nodes {
 		item := sel.Eq(i)

-		// Find URL
-		linkTag := item.Find("a")
-		link, _ := linkTag.Attr("href")
+		// Skip items without an h3 element (which indicates a search result)
+		if item.Find("h3").Length() == 0 {
+			continue
+		}
+
+		// Find URL - look for the anchor that contains the h3 title
+		linkTag := item.Find("h3").Parent()
+		if linkTag.Is("a") == false {
+			linkTag = item.Find("h3").Closest("a")
+		}
+
+		link, exists := linkTag.Attr("href")
+		if !exists || link == "" || link == "#" {
+			continue
+		}
 		link = strings.Trim(link, " ")

-		// Find title
+		// Find title - this is inside the h3 element
 		titleTag := item.Find("h3")
 		title := titleTag.Text()

-		// Find description
-		descTag := item.Find(`div[data-sncf~="1"]`)
+		// Find description - find div with text content after the heading
+		// Using attribute selectors that match the description container
+		descTag := item.Find("div[data-sncf='1']").Find("div").First()
+		if descTag.Length() == 0 {
+			// Try another selector approach if the first one fails
+			descTag = item.Find("div.VwiC3b")
+			if descTag.Length() == 0 {
+				// As a last resort, look for any div after the title that might contain description
+				titleParent := titleTag.Parent()
+				if titleParent.Is("a") {
+					titleParent = titleParent.Parent().Parent()
+				}
+				descTag = titleParent.NextAll().First().Find("div").First()
+			}
+		}
 		desc := descTag.Text()

 		if link != "" && link != "#" {
@@ -67,7 +115,7 @@ func googleResultParser(response *http.Response) ([]core.SearchResult, error) {
 	}

 	logrus.Tracef("Google search document size: %d", len(doc.Text()))
-	return results, err
+	return core.DeduplicateResults(results), err
 }

 func Search(query core.Query) ([]core.SearchResult, error) {
@@ -77,7 +125,7 @@ func Search(query core.Query) ([]core.SearchResult, error) {
 	}
 	logrus.Debugf("Google URL built: %s", googleURL)

-	res, err := googleRequest(googleURL)
+	res, err := googleRequest(googleURL, query)
 	if err != nil {
 		return nil, err
 	}
--- a/google/search_test.go
+++ b/google/search_test.go
@@ -10,12 +10,12 @@ import (
 var browser *core.Browser

 func init() {
-	opts := core.BrowserOpts{IsHeadless: true, IsLeakless: true, Timeout: time.Second * 2, WaitRequests: false}
+	opts := core.BrowserOpts{IsHeadless: true, IsLeakless: false, Timeout: time.Second * 5, LeavePageOpen: false}
 	browser, _ = core.NewBrowser(opts)
 }

 func TestSearchGoogle(t *testing.T) {
-	gogl := New(*browser)
+	gogl := New(*browser, core.SearchEngineOptions{})

 	query := core.Query{Text: "HEY", Limit: 10}
 	results, err := gogl.Search(query)
@@ -27,3 +27,41 @@ func TestSearchGoogle(t *testing.T) {
 		t.Fatalf("[SearchGoogle] returned empty result")
 	}
 }
+
+func TestParseSourceImageURL(t *testing.T) {
+	//href1 := `/imgres?imgurl=https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2F2%2F26%2FMarmota_marmota_Alpes2.jpg&amp;tbnid=Be_RycOe8xzlpM&amp;vet=12ahUKEwjkh6WzwIeAAxWV_yoKHRzHC9wQMygAegUIARD0AQ..i&amp;imgrefurl=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FAlpine_marmot&amp;docid=7miWbc2QiSw9uM&amp;w=801&amp;h=599&amp;q=alpine%20marmot&amp;ved=2ahUKEwjkh6WzwIeAAxWV_yoKHRzHC9wQMygAegUIARD0AQ`
+	href2 := `/imgres?imgurl=https%3A%2F%2Fstatic.wikia.nocookie.net%2Fnaturerules1%2Fimages%2Ff%2Ff2%2F13d79d934ccf6f7919777fcb6dbb6e6c.jpg%2Frevision%2Flatest%3Fcb%3D20210218225522&tbnid=JxC8NUyBjdNbdM&vet=12ahUKEwiHrJnN1YeAAxXvEBAIHfRADAAQMygCegUIARD4AQ..i&imgrefurl=https%3A%2F%2Fnaturerules1.fandom.com%2Fwiki%2FAlpine_Marmot&docid=XXYeDjL67badNM&w=1600&h=1200&q=alpine%20marmot&ved=2ahUKEwiHrJnN1YeAAxXvEBAIHfRADAAQMygCegUIARD4AQ`
+	want := SourceImage{
+		OriginalURL: "https://static.wikia.nocookie.net/naturerules1/images/f/f2/13d79d934ccf6f7919777fcb6dbb6e6c.jpg/revision/latest?cb=20210218225522",
+		PageURL:     "https://naturerules1.fandom.com/wiki/Alpine_Marmot",
+		Width:       "1600",
+		Height:      "1200",
+	}
+
+	got, err := parseSourceImageURL(href2)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if want != got {
+		t.Fatalf("Want: %v, Got: %v", want, got)
+	}
+}
+
+func TestImageSearch(t *testing.T) {
+	gogl := New(*browser, core.SearchEngineOptions{})
+	query := core.Query{Text: "Ferrari Testarossa", Limit: 77}
+	results, err := gogl.SearchImage(query)
+
+	if err != nil {
+		t.Fatalf("Cannot search images: %s", err)
+	}
+
+	if len(results) < 77 {
+		t.Fatalf("Returned not full result")
+	}
+
+	if results[0].URL == "" {
+		t.Fatalf("First result doesn't contain URL, %v+", results[0])
+	}
+}
--- a/google/url.go
+++ b/google/url.go
@@ -236,17 +236,78 @@ func BuildURL(q core.Query) (string, error) {

 		logrus.Tracef("Query text: %s", text)
 		params.Add("q", text)
+		params.Add("oq", text)
 	}

 	if len(params.Get("q")) == 0 {
-		return "", errors.New("Empty query built")
+		return "", errors.New("empty query built")
 	}

 	// Set search date range
 	if q.DateInterval != "" {
 		intervals := strings.Split(q.DateInterval, "..")
 		if len(intervals) != 2 {
-			return "", errors.New("Incorrect data interval provided")
+			return "", errors.New("incorrect data interval provided")
+		}
+
+		dataParam := fmt.Sprintf("cdr:1,cd_min:%s,cd_max:%s", intervals[0], intervals[1])
+		params.Add("tbs", dataParam)
+	}
+
+	// Limit number of results
+	if q.Limit != 0 {
+		params.Add("num", strconv.Itoa(q.Limit))
+	}
+
+	if q.LangCode != "" {
+		params.Add("hl", q.LangCode)
+		params.Add("lr", "lang_"+strings.ToLower(q.LangCode))
+	}
+
+	params.Add("pws", "0")  // Do not personalize earch results
+	params.Add("nfpr", "1") // Do not auto correct search queries
+	params.Add("sourceid", "chrome")
+	params.Add("ie", "UTF-8")
+
+	base.RawQuery = params.Encode()
+	return base.String(), nil
+}
+
+func BuildImageURL(q core.Query) (string, error) {
+	// TODO: Add new params
+	googleBase := GoogleDomains[strings.ToLower(q.LangCode)]
+	base, err := url.Parse(fmt.Sprintf("https://www.google.%s", googleBase))
+	if err != nil {
+		return "", err
+	}
+
+	base.Path += "search"
+	params := url.Values{}
+	params.Add("tbm", "isch") // Search images
+
+	// Set request text
+	if q.Text != "" || q.Site != "" || q.Filetype != "" {
+		text := q.Text
+		if q.Site != "" {
+			text += " site:" + q.Site
+		}
+		if q.Filetype != "" {
+			text += " filetype:" + q.Filetype
+		}
+
+		params.Add("q", text)
+		params.Add("oq", text)
+	}
+
+	if len(params.Get("q")) == 0 {
+		return "", errors.New("empty query built")
+	}
+
+	// Set search date range
+	if q.DateInterval != "" {
+		intervals := strings.Split(q.DateInterval, "..")
+		if len(intervals) != 2 {
+			return "", errors.New("incorrect data interval provided")
 		}

 		dataParam := fmt.Sprintf("cdr:1,cd_min:%s,cd_max:%s", intervals[0], intervals[1])
@@ -269,3 +330,52 @@ func BuildURL(q core.Query) (string, error) {
 	base.RawQuery = params.Encode()
 	return base.String(), nil
 }
+
+type SourceImage struct {
+	PageURL     string
+	OriginalURL string
+	Width       string
+	Height      string
+}
+
+func parseSourceImageURL(href string) (SourceImage, error) {
+	source := SourceImage{}
+
+	href = strings.ReplaceAll(href, ";", "&")
+	parsed, err := url.QueryUnescape(href)
+	if err != nil {
+		return source, err
+	}
+
+	u, err := url.Parse(parsed)
+	if err != nil {
+		return source, err
+	}
+
+	queryMap, err := url.ParseQuery(u.RawQuery)
+	if err != nil {
+		return source, err
+	}
+
+	val, ok := queryMap["h"]
+	if ok && len(val) > 0 {
+		source.Height = val[0]
+	}
+
+	val, ok = queryMap["w"]
+	if ok && len(val) > 0 {
+		source.Width = val[0]
+	}
+
+	val, ok = queryMap["imgrefurl"]
+	if ok && len(val) > 0 {
+		source.PageURL = val[0]
+	}
+
+	val, ok = queryMap["imgurl"]
+	if ok && len(val) > 0 {
+		source.OriginalURL = val[0]
+	}
+
+	return source, nil
+}
--- a/yandex/search.go
+++ b/yandex/search.go
@@ -1,22 +1,50 @@
 package yandex

 import (
+	"encoding/json"
+	"fmt"
+	"sort"
 	"time"

 	"github.com/go-rod/rod"
 	"github.com/karust/openserp/core"
 	"github.com/sirupsen/logrus"
+	"golang.org/x/time/rate"
 )

+type ImageEntity struct {
+	ID        string `json:"id"`
+	Rank      int    `json:"pos"`
+	Width     int    `json:"origWidth"`
+	Height    int    `json:"origHeight"`
+	Title     string `json:"alt"`
+	OrigURL   string `json:"origUrl"`
+	ThumbURL  string `json:"image"`
+	Freshness string `json:"freshnessCounter"`
+	IsGIF     bool   `json:"gifLabel"`
+}
+
+type ImageData struct {
+	InitalState struct {
+		SerpList struct {
+			Items struct {
+				Entities map[string]ImageEntity `json:"entities"`
+			} `json:"items"`
+		} `json:"serpList"`
+	} `json:"initialState"`
+}
+
 type Yandex struct {
 	core.Browser
-	checkTimeout time.Duration // Timeout for secondary elements check
-	pageSleep    time.Duration // Sleep between pages
+	core.SearchEngineOptions
+	pageSleep time.Duration // Sleep between pages
 }

-func New(browser core.Browser) *Yandex {
+func New(browser core.Browser, opts core.SearchEngineOptions) *Yandex {
 	yand := Yandex{Browser: browser}
-	yand.checkTimeout = time.Second * 2
+	opts.Init()
+	yand.SearchEngineOptions = opts
+
 	yand.pageSleep = time.Second * 1
 	return &yand
 }
@@ -25,29 +53,20 @@ func (yand *Yandex) Name() string {
 	return "yandex"
 }

+func (yand *Yandex) GetRateLimiter() *rate.Limiter {
+	ratelimit := rate.Every(yand.GetRatelimit())
+	return rate.NewLimiter(ratelimit, yand.RateBurst)
+}
+
 func (yand *Yandex) isCaptcha(page *rod.Page) bool {
-	_, err := page.Timeout(yand.checkTimeout).Search("form#checkbox-captcha-form")
-	if err != nil {
-		return false
-	}
-	return true
+	_, err := page.Timeout(yand.GetSelectorTimeout()).Search("form#checkbox-captcha-form")
+	return err == nil
 }

 // Check if nothig is found
 func (yand *Yandex) isNoResults(page *rod.Page) bool {
-	noResFound := false
-
-	_, err := page.Timeout(yand.checkTimeout).Search("div.EmptySearchResults-Title")
-	if err == nil {
-		noResFound = true
-	}
-
-	_, err = page.Timeout(yand.checkTimeout).Search("div>div.RequestMeta-Message")
-	if err == nil {
-		noResFound = true
-	}
-
-	return noResFound
+	_, err := page.Timeout(yand.GetSelectorTimeout()).Search("div.Correction.SearchCorrection")
+	return err == nil
 }

 func (yand *Yandex) parseResults(results rod.Elements, pageNum int) []core.SearchResult {
@@ -105,20 +124,28 @@ func (yand *Yandex) Search(query core.Query) ([]core.SearchResult, error) {
 			return nil, err
 		}

-		page := yand.Navigate(url)
+		page, err := yand.Navigate(url)
+		if err != nil {
+			return nil, err
+		}

 		// Get all search results in page
 		searchRes, err := page.Timeout(yand.Timeout).Search("li.serp-item")
 		if err != nil {
+			defer page.Close()
 			logrus.Errorf("Cannot parse search results: %s", err)
+			return nil, core.ErrSearchTimeout
 		}

 		// Check why no results, maybe captcha?
 		if searchRes == nil {
+			defer page.Close()
+
 			if yand.isNoResults(page) {
 				logrus.Errorf("No results found")
 			} else if yand.isCaptcha(page) {
 				logrus.Errorf("Yandex captcha occurred during: %s", url)
+				return nil, core.ErrCaptcha
 			}
 			break
 		}
@@ -145,5 +172,81 @@ func (yand *Yandex) Search(query core.Query) ([]core.SearchResult, error) {
 		time.Sleep(yand.pageSleep)
 	}

-	return allResults, nil
+	return core.DeduplicateResults(allResults), nil
+}
+
+func (yand *Yandex) SearchImage(query core.Query) ([]core.SearchResult, error) {
+	logrus.Tracef("Start Yandex image search, query: %+v", query)
+
+	searchResults := []core.SearchResult{}
+
+	searchPage := 0
+	for len(searchResults) < query.Limit {
+		url, err := BuildImageURL(query, searchPage)
+		if err != nil {
+			return nil, err
+		}
+		searchPage += 1
+
+		page, err := yand.Navigate(url)
+		if err != nil {
+			return nil, err
+		}
+
+		if !yand.LeavePageOpen {
+			defer page.Close()
+		}
+
+		//page.Keyboard.Press(input.End)
+		//page.WaitLoad()
+		//time.Sleep(time.Duration(time.Second * 2))
+
+		results, err := page.Timeout(yand.Timeout).Search("div[role='main'] div[data-state]")
+		if err != nil {
+			logrus.Errorf("Cannot find search results: %s", err)
+		}
+
+		// Check why no results
+		if results == nil {
+			if yand.isCaptcha(page) {
+				logrus.Errorf("Yandex captcha occurred during: %s", url)
+				return searchResults, core.ErrCaptcha
+			} else if yand.isNoResults(page) {
+				logrus.Errorf("No results found")
+			}
+			return searchResults, core.ErrSearchTimeout
+		}
+
+		data, err := results.First.Attribute("data-state")
+		if err != nil {
+			return nil, err
+		}
+
+		var imgData ImageData
+		if err := json.Unmarshal([]byte(*data), &imgData); err != nil {
+			return nil, err
+		}
+
+		for id := range imgData.InitalState.SerpList.Items.Entities {
+			img := imgData.InitalState.SerpList.Items.Entities[id]
+			res := core.SearchResult{
+				Rank:        img.Rank + 1,
+				URL:         img.OrigURL,
+				Title:       img.Title,
+				Description: fmt.Sprintf("%dx%d, freshness:%s, thumb_url:%s", img.Height, img.Width, img.Freshness, img.ThumbURL),
+			}
+
+			searchResults = append(searchResults, res)
+		}
+
+		if !yand.LeavePageOpen {
+			page.Close()
+		}
+	}
+
+	sort.Slice(searchResults, func(i, j int) bool {
+		return searchResults[i].Rank < searchResults[j].Rank
+	})
+
+	return core.DeduplicateResults(searchResults), nil
 }
--- a/yandex/search_raw.go
+++ b/yandex/search_raw.go
@@ -1,8 +1,11 @@
 package yandex

 import (
+	"crypto/tls"
 	"net/http"
+	"net/url"
 	"strings"
+	"time"

 	"github.com/PuerkitoBio/goquery"
 	"github.com/corpix/uarand"
@@ -10,8 +13,27 @@ import (
 	"github.com/sirupsen/logrus"
 )

-func yandexRequest(searchURL string) (*http.Response, error) {
-	baseClient := &http.Client{}
+func yandexRequest(searchURL string, query core.Query) (*http.Response, error) {
+	// Create HTTP transport with proxy
+	transport := &http.Transport{}
+	if query.ProxyURL != "" {
+		proxyUrl, err := url.Parse(query.ProxyURL)
+		if err != nil {
+			return nil, err
+		}
+		transport.Proxy = http.ProxyURL(proxyUrl)
+	}
+
+	// Set insecure TLS
+	if query.Insecure {
+		transport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
+	}
+
+	baseClient := &http.Client{
+		Transport: transport,
+		Timeout:   time.Second * 10,
+	}
+
 	req, err := http.NewRequest("GET", searchURL, nil)
 	if err != nil {
 		return nil, err
@@ -26,7 +48,7 @@ func yandexRequest(searchURL string) (*http.Response, error) {
 }

 func yandexResultParser(response *http.Response) ([]core.SearchResult, error) {
-	doc, err := goquery.NewDocumentFromResponse(response)
+	doc, err := goquery.NewDocumentFromReader(response.Body)
 	if err != nil {
 		return nil, err
 	}
@@ -67,7 +89,7 @@ func yandexResultParser(response *http.Response) ([]core.SearchResult, error) {
 	}

 	logrus.Tracef("Yandex search document size: %d", len(doc.Text()))
-	return results, err
+	return core.DeduplicateResults(results), err
 }

 func Search(query core.Query) ([]core.SearchResult, error) {
@@ -77,7 +99,7 @@ func Search(query core.Query) ([]core.SearchResult, error) {
 	}
 	logrus.Debugf("Yandex URL built: %s", googleURL)

-	res, err := yandexRequest(googleURL)
+	res, err := yandexRequest(googleURL, query)
 	if err != nil {
 		return nil, err
 	}
--- a/yandex/search_test.go
+++ b/yandex/search_test.go
@@ -10,13 +10,26 @@ import (
 var browser *core.Browser

 func init() {
-	opts := core.BrowserOpts{IsHeadless: true, IsLeakless: true, Timeout: time.Second * 2, WaitRequests: false}
+	opts := core.BrowserOpts{IsHeadless: false, IsLeakless: false, Timeout: time.Second * 15, LeavePageOpen: true}
 	browser, _ = core.NewBrowser(opts)
 }

+// func TestParseImgData(t *testing.T) {
+
+// 	jsonData, _ := os.ReadFile("./testImgData.json")
+// 	var obj ImageData
+// 	if err := json.Unmarshal(jsonData, &obj); err != nil {
+// 		t.Fatal(err)
+// 	}
+
+// 	if (len(obj.InitalState.SerpList.Items.Entities)) != 30 {
+// 		t.Fail()
+// 	}
+// }
+
 func TestSearchYandex(t *testing.T) {

-	yand := New(*browser)
+	yand := New(*browser, core.SearchEngineOptions{})

 	query := core.Query{Text: "HEY", Limit: 10}
 	results, err := yand.Search(query)
@@ -28,3 +41,17 @@ func TestSearchYandex(t *testing.T) {
 		t.Fatalf("[SearchYandex] returned empty result")
 	}
 }
+
+func TestImageYandex(t *testing.T) {
+	yand := New(*browser, core.SearchEngineOptions{})
+
+	query := core.Query{Text: "furry tiger", Limit: 30}
+	results, err := yand.SearchImage(query)
+	if err != nil {
+		t.Fatalf("Cannot [ImageYandex]: %s", err)
+	}
+
+	if len(results) < 30 {
+		t.Fatalf("[ImageYandex] returned empty result")
+	}
+}
--- a/yandex/url.go
+++ b/yandex/url.go
@@ -8,8 +8,10 @@ import (
 	"github.com/karust/openserp/core"
 )

+const baseURL = "https://www.yandex.com"
+
 func BuildURL(q core.Query, page int) (string, error) {
-	base, _ := url.Parse("https://www.yandex.com")
+	base, _ := url.Parse(baseURL)
 	base.Path += "search/"

 	params := url.Values{}
@@ -33,7 +35,40 @@ func BuildURL(q core.Query, page int) (string, error) {
 	}

 	if len(params.Get("text")) == 0 {
-		return "", errors.New("Empty query built")
+		return "", errors.New("empty query built")
+	}
+
+	base.RawQuery = params.Encode()
+	return base.String(), nil
+}
+
+func BuildImageURL(q core.Query, page int) (string, error) {
+	// TODO: Add other parameters
+	base, _ := url.Parse(baseURL)
+	base.Path += "images/search/"
+
+	params := url.Values{}
+	if q.Text != "" {
+		text := q.Text
+
+		if q.DateInterval != "" {
+			text += " date:" + q.DateInterval
+		}
+
+		params.Add("text", text)
+		params.Add("p", fmt.Sprint(page))
+	}
+
+	if len(params.Get("text")) == 0 {
+		return "", errors.New("empty query built")
+	}
+
+	if q.Site != "" {
+		params.Add("site", q.Site)
+	}
+
+	if q.Filetype != "" {
+		params.Add("itype", q.Filetype)
 	}

 	base.RawQuery = params.Encode()
Author	SHA1	Message	Date
Rustem Kamalov	d8c3700252	Stealth browser as option. Update default config	2025-08-18 22:49:45 +03:00
Rustem Kamalov	7f0f32ac4e	Handle failed navigation	2025-07-03 02:29:12 +03:00
Rustem Kamalov	ba94c69dd0	Merge pull request #10 from PACHAKUTlQ/main Support http and socks5 proxy and fix failure due to updates of DOM on Google SERP	2025-07-03 02:09:18 +03:00
Pachakutiq	c387268d20	chore: Exclude .aider* in .gitignore	2025-05-27 15:05:53 +08:00
Pachakutiq	2f8ee2164d	fix: Fix serve mode not supporting raw mode	2025-04-10 18:07:57 +08:00
Pachakutiq	6691ebaadd	fix: Remove duplicated results for all search engines and both modes	2025-04-10 16:59:31 +08:00
Pachakutiq	8099dae3ce	fix: Remove duplicated results in google/search.go	2025-04-10 15:35:54 +08:00
Pachakutiq	2f77dd6fe5	doc: Add proxy guide in README.md	2025-04-09 02:55:59 +08:00
Pachakutiq	90f27c2974	feat: Support short flag `-x` for proxy	2025-04-09 02:54:42 +08:00
Pachakutiq	ec03439b2f	style: Format README.md using prettier	2025-04-09 02:11:22 +08:00
Pachakutiq	5ad89b2ea8	feat: Support proxy for serve	2025-04-09 02:08:01 +08:00
Pachakutiq	b2ffcf8477	feat: Support http and socks5 proxy (support authentication) for chrome driver mode	2025-04-08 23:51:44 +08:00
Pachakutiq	21f12c5996	fix: Google updated DOM of search result page	2025-04-08 18:48:07 +08:00
Pachakutiq	84e2d46c4b	feat: Support http and socks5 proxy with auth for raw search	2025-04-08 16:06:41 +08:00
Pachakutiq	205f722ddc	fix: Fix search crash (slice bounds out of range [4:1])	2024-12-01 23:35:11 +08:00
Rustem Kamalov	aeb0616249	Fix google images selector + baidu images ads tag	2024-05-12 02:57:39 +03:00
Rustem Kamalov	fc49deba67	Fix yandex image search	2024-05-12 01:41:53 +03:00
Rustem Kamalov	0a3c07b149	Collects google ads/answers + captcha solver test	2024-05-06 01:13:07 +03:00
Rustem Kamalov	482f725c10	Remove random UA generator	2024-05-06 00:53:57 +03:00
Rustem Kamalov	923b7c2443	Additional google query params	2024-05-06 00:51:35 +03:00
Rustem Kamalov	e2bdad75dd	Google: Fix total results bug, Fix captcha detection	2024-02-07 01:30:57 +03:00
Rustem Kamalov	b040b24a70	Readme update	2023-08-06 15:09:26 +03:00
Rustem Kamalov	a3455b592d	Add image API	2023-07-23 04:37:01 +03:00
Rustem Kamalov	29a7c2ca14	Yandex images scroll parsing	2023-07-23 04:35:48 +03:00
Rustem Kamalov	38183b804a	Google images scroll parsing	2023-07-23 04:35:28 +03:00
Rustem Kamalov	34074161a5	Baidu. Fix broken JSON parsing	2023-07-23 04:33:53 +03:00
Rustem Kamalov	e12fff8313	Google image search test	2023-07-12 02:17:04 +03:00
Rustem Kamalov	4d6921d007	Baidu [ImageUrlBuild] test fix	2023-07-11 18:53:42 +03:00
Rustem Kamalov	284aebbfb2	Baidu image search test	2023-07-08 01:13:56 +03:00
Rustem Kamalov	6eaab4a059	Yandex image search test	2023-07-05 18:42:14 +03:00
Rustem Kamalov	ffe33d6bc5	Settings via config file and environment variables	2023-06-30 22:49:58 +03:00
Rustem Kamalov	df972585c1	Fix default ratelimiter	2023-06-30 22:04:26 +03:00
Rustem Kamalov	dd5e0a2069	Default ratelimeter	2023-06-28 22:20:19 +03:00
Rustem Kamalov	72bddf7fc9	SearchTimeout error handling and proper tabs close	2023-06-27 23:13:10 +03:00
Rustem Kamalov	c175d0c386	Default config fix, manual bot test	2023-06-26 15:30:24 +03:00
Rustem Kamalov	3f84503ce3	Add captcha check and error	2023-06-25 20:41:47 +03:00
Rustem Kamalov	6c4eb8db1a	Radme and default configs fix	2023-06-25 20:39:14 +03:00