chore(go): classify tests into unit/integration/e2e/manual tiers via build tags (#17586)

## Summary
Classify the Go test suite by dependency level using build tags so the
default
`go test ./...` run stays self-contained, and add local convenience
commands
plus a documented convention.

- Add build tags to 5 real-service tests that were previously un-tagged
and only
soft-isolated via `t.Skip`: `kg_test.go` (integration), `minio_test.go`
  (integration), `template_integration_test.go` (integration),
`stagehand_runtime_integration_test.go` (integration),
`pipeline_e2e_test.go` (e2e).
  The default unit run no longer compiles/attempts these.
- Reclassify the full-pipeline `real_consumer` tests from `integration`
to `e2e`.
- Add `build.sh` shortcuts: `--test-integration`, `--test-e2e`,
`--test-manual`,
`--test-all` (integration + e2e; `manual` is excluded and is local
opt-in only,
  never run in CI).
- Document the tier scheme (unit / integration / e2e / manual +
orthogonal cgo) in
  `AGENTS.md`.

## Tier definitions
| Tier | Build tag | Runs by default? |
|---|---|---|
| Unit | (none) | Yes — in-memory SQLite / miniredis / httptest stubs |
| Integration | `integration` | No (`-tags integration`) — single real
service |
| E2E | `e2e` | No (`-tags e2e`) — full ingest→index→retrieve pipeline |
| Manual | `manual` | No (`-tags manual`) — very slow; never in CI |

## Verification
- `gofmt -l` clean on all changed files; `bash -n build.sh` OK.
- `go list` confirms the default set excludes the tagged files, and
  `-tags integration` / `-tags e2e` include them.
- Full regression: unit / integration / e2e each **97 ok, 0 FAIL**.
- Fixed a regression where `pipeline_knowledge_compiler_test.go` relied
on a
transitive import side-effect from `template_integration_test.go` to
register
the `File`/`Parser`/`TokenChunker` components; it now blank-imports the
  component packages directly.

## Test plan
- [ ] `./build.sh --test` (unit) passes
- [ ] `./build.sh --test-integration` passes (needs real services; skips
otherwise)
- [ ] `./build.sh --test-e2e` passes (needs real services; skips
otherwise)

---------

Co-authored-by: CodeBuddy <noreply@cnb.cool>

AGENTS.md

@@ -51,6 +51,30 @@ Use this file as the local operating guide for the current codebase. Prefer the
 - Remove commented-out Go code instead of leaving recovery notes in place.
 - Keep package comments and doc comments aligned with the current runtime path, not with migration history.
 
+## Go Test Tiers
+Go tests are classified by build tag so the default `go test ./...` run stays self-contained. Tag a test file with `//go:build <tier>` placed before the `package` clause.
+
+| Tier | Build tag | Runs by default? | Needs |
+|---|---|---|---|
+| Unit | (none) | Yes (`go test ./...`) | Native CGO static libs (wired by `build.sh --test`); no external services — uses in-memory SQLite, miniredis, or `httptest` stubs. |
+| Integration | `integration` | No (`-tags integration`) | A real service: MySQL/MinIO/Elasticsearch/Infinity/LLM. Single component, reasonably fast. |
+| E2E | `e2e` | No (`-tags e2e`) | Full cross-component pipeline (ingest → index → retrieve) against real services; heavy/slow. |
+| Manual | `manual` | No (`-tags manual`) | Very slow/expensive (deepdoc render/parity/snapshot/bench). **Local opt-in ONLY — never run in CI.** |
+| Native (orthogonal) | `cgo` / `!cgo` | `cgo` auto-satisfies under CGO_ENABLED=1 | Native static libs (`office_oxide`/`pdfium`/`pdf_oxide`). Combine with tiers, e.g. `//go:build cgo && integration`. |
+
+Run tiers locally via `build.sh`:
+```bash
+bash build.sh --test                      # unit tier (no tags)
+bash build.sh --test-integration ./...    # integration tier
+bash build.sh --test-e2e                  # e2e tier
+bash build.sh --test-manual               # manual tier (very slow)
+bash build.sh --test-all                  # integration + e2e (never includes manual)
+```
+Rules:
+- New tests that touch a real external service MUST carry `integration`/`e2e`/`manual` — do not rely on `t.Skip` + env vars to soft-isolate them in the default unit run. Keep an env guard as a harmless secondary safety net if desired.
+- `manual` is never wired into CI or any automated pipeline.
+- `unit` (no tag) must stay free of external-service dependencies so `go test ./...` passes without MySQL/MinIO/ES/Infinity/LLM. The native CGO static libraries (`office_oxide`/`pdfium`/`pdf_oxide`) are still required at build time and are wired automatically by `build.sh --test`; that is expected, not an external service.
+
 ## Working Rules
 - Before editing, inspect the nearest code path that actually owns the behavior.
 - Keep changes small and local unless the task is explicitly a broader refactor.

build.sh

@@ -432,6 +432,24 @@ run_go_tests() {
         go test -count=1 "$@"
 }
 
+# Run Go tests gated behind a build tag (or space-separated tag list), e.g.
+# `./build.sh --test-integration -run TestFoo ./internal/engine/...`.
+# See "Go Test Tiers" in AGENTS.md for the tier definitions.
+run_go_tests_tagged() {
+    local tags="$1"; shift
+    print_section "Running Go tests (tags: ${tags})"
+
+    cd "$PROJECT_ROOT"
+    setup_cgo_env
+
+    if [ "$#" -eq 0 ]; then
+        set -- ./...
+    fi
+    GOPROXY=${GOPROXY:-https://goproxy.cn,https://proxy.golang.org,direct} CGO_ENABLED=1 \
+        CGO_CFLAGS="$CGO_CFLAGS" CGO_LDFLAGS="$CGO_LDFLAGS" \
+        go test -tags "${tags}" -count=1 "$@"
+}
+
 # Clean build artifacts
 clean() {
     print_section "Cleaning build artifacts"
@@ -487,9 +505,18 @@ OPTIONS:
     --cpp, -c       Build only C++ static library
     --cpp-test      Build C++ test executable (requires --cpp first)
     --go, -g        Build only Go server (requires C++ library to be built)
-    --test, -t      Run Go unit tests (sets up CGO env for office_oxide).
-                    Any extra args are forwarded to `go test`, e.g.
+    --test, -t      Run Go unit tests (no build tag). Sets up the CGO env and
+                    native static libs (office_oxide/pdfium/pdf_oxide) needed to
+                    build (same contract as the Go tier table in AGENTS.md).
+                    Extra args are forwarded to `go test`, e.g.
                     `$0 --test -run TestFoo ./internal/admin/...`
+    --test-integration   Run Go tests tagged 'integration' (need real services,
+                    e.g. MySQL/MinIO/ES/Infinity/LLM). e.g.
+                    `$0 --test-integration ./internal/engine/...`
+    --test-e2e           Run Go tests tagged 'e2e' (full-pipeline, heavy).
+    --test-manual        Run Go tests tagged 'manual' (very slow; local opt-in
+                    ONLY, never run in CI).
+    --test-all           Run 'integration' + 'e2e' tests (excludes 'manual').
     --clean, -C     Clean all build artifacts
     --run, -r       Build and run the server
     --strip, -s     Strip debug symbols from Go binaries (-ldflags="-s -w")
@@ -501,8 +528,12 @@ EXAMPLES:
     $0 --cpp        # Build only C++ library
     $0 --go         # Build only Go server
     $0 --cpp-test   # Build C++ test executable
-    $0 --test       # Run all Go tests
+    $0 --test       # Run all Go tests (unit tier, no build tag)
     $0 --test -run TestFoo ./internal/admin/...      # Targeted Go tests
+    $0 --test-integration ./internal/engine/...      # integration tier
+    $0 --test-e2e                                 # e2e tier
+    $0 --test-manual                             # manual tier (very slow)
+    $0 --test-all                                # integration + e2e (no manual)
     $0 --run        # Build and run
     $0 --clean      # Clean build artifacts
 
@@ -551,6 +582,38 @@ main() {
                 run_go_tests "${args[@]:1}"
             fi
             ;;
+        --test-integration)
+            check_go_deps
+            if [ "${args[1]:-}" = "--" ]; then
+                run_go_tests_tagged integration "${args[@]:2}"
+            else
+                run_go_tests_tagged integration "${args[@]:1}"
+            fi
+            ;;
+        --test-e2e)
+            check_go_deps
+            if [ "${args[1]:-}" = "--" ]; then
+                run_go_tests_tagged e2e "${args[@]:2}"
+            else
+                run_go_tests_tagged e2e "${args[@]:1}"
+            fi
+            ;;
+        --test-manual)
+            check_go_deps
+            if [ "${args[1]:-}" = "--" ]; then
+                run_go_tests_tagged manual "${args[@]:2}"
+            else
+                run_go_tests_tagged manual "${args[@]:1}"
+            fi
+            ;;
+        --test-all)
+            check_go_deps
+            if [ "${args[1]:-}" = "--" ]; then
+                run_go_tests_tagged "integration e2e" "${args[@]:2}"
+            else
+                run_go_tests_tagged "integration e2e" "${args[@]:1}"
+            fi
+            ;;
         --clean|-C)
             clean
             ;;

internal/agent/component/stagehand_runtime_integration_test.go

@@ -38,6 +38,9 @@
 //	export OPENAI_MODEL=...
 //	rtk go test ./internal/agent/component/ -count=1 \
 //	  -run TestStagehandRuntime_Extract -v -timeout 3m
+
+//go:build integration
+
 package component
 
 import (

internal/engine/elasticsearch/kg_test.go

@@ -14,6 +14,8 @@
 //  limitations under the License.
 //
 
+//go:build integration
+
 package elasticsearch
 
 import (

internal/ingestion/pipeline/pipeline_knowledge_compiler_test.go

@@ -24,6 +24,10 @@ import (
 	// init hook fires and the component name referenced by the template is
 	// resolvable in the default runtime factory.
 	"ragflow/internal/agent/runtime"
+	// Blank-import the parent component package so its init registers the
+	// File/Parser/TokenChunker components this test's template references.
+	_ "ragflow/internal/ingestion/component"
+	_ "ragflow/internal/ingestion/component/chunker"
 	_ "ragflow/internal/ingestion/component/knowledge_compiler"
 )
 

internal/ingestion/pipeline/template_integration_test.go

@@ -19,6 +19,8 @@
 // Tokenizer and title weighting runs - the q_<n>_vec first element is
 // 0.1*len(name) + 0.9*len(content), and embedding_token_consumption includes
 // the one title encode (len(name)) plus the per-chunk content encodes.
+//go:build integration
+
 package pipeline
 
 import (

internal/ingestion/service/real_consumer_pipeline_test.go

@@ -1,4 +1,4 @@
-//go:build integration
+//go:build e2e
 
 package service
 

internal/ingestion/task/pipeline_e2e_test.go

@@ -14,6 +14,8 @@
 // limitations under the License.
 //
 
+//go:build e2e
+
 package task
 
 import (

internal/ingestion/task/real_consumer_test.go

@@ -1,5 +1,3 @@
-//go:build integration
-
 //
 //  Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
 //
@@ -14,7 +12,8 @@
 //  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
-//
+
+//go:build e2e
 
 package task
 

internal/storage/minio_test.go

@@ -14,6 +14,8 @@
 //  limitations under the License.
 //
 
+//go:build integration
+
 package storage
 
 import (