## Summary
- `ProcessChunksForPipeline` now sets `kb_id` to a plain string instead
of `[]string{kbID}`, removing an index-physical array shape from the
ingestion domain.
- Stored documents are byte-identical: Elasticsearch overrides `kb_id`
with `datasetID` on write, and Infinity's `transformChunkFields` already
accepts a plain string.
- Infinity is intentionally left unchanged — `service/chunk` paths still
feed `kb_id` as `[]string`, and Infinity handles both forms. The
`dataset` artifact merge (`dataset_artifact_service.go`) is out of scope
for this step.
- Unit assertion updated to expect a string.
## Scope / non-goals
This is the smallest first step (T1) of the index-schema leak cleanup
tracked in #17371. It does **not** move the other leaks (`docnm_kwd`,
`create_timestamp_flt`, position ints) to the engine boundary — those
are later steps behind a read-back golden test.
## Test plan
- `go test ./internal/ingestion/task/indexdoc/...` passes.
- The two `task` "Real" integration tests fail identically on a clean
tree (environment lacks real embedding/parsing); they are pre-existing,
unrelated to this change.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
## Summary
Extract the pipeline-output → search-engine index document mapping
helpers out of the `task` package into a dedicated, dependency-light
leaf package `internal/ingestion/task/indexdoc`.
These functions are pure transforms (they only depend on
`common`/`utility`) and are not task-orchestration concerns:
- `NormalizeChunks`, `DeepCopyChunks` (was unexported `deepCopyChunks`),
`toChunkMaps` → `indexdoc/normalize.go`
- `ProcessChunksForPipeline`, `RenameTextToContentWithWeight`,
`GetEmbeddingTokenConsumption`, `cleanupConsumedChunkFields`,
`mergeChunkMetadata`, `processChunkPositions`,
`AggregateTableDocMetadata`, `resolveTableColumnConfig` →
`indexdoc/process.go`
- `AddPositions` → `indexdoc/position.go`
- `EmbeddingTokenConsumptionKey` constant → `indexdoc/constants.go`
(task/constants.go keeps only `GRAPH_RAPTOR_FAKE_DOC_ID`)
Call sites in `pipeline_executor.go` and `golden_compare.go` now
reference the `indexdoc` package; package-task tests qualify the moved
symbols.
## Why
The `task` package had grown into a "orchestration + pure mapping +
debug" mix. Splitting the pure mapping helpers into a leaf package
sharpens package boundaries, removes a misleading top-level
`ingestion/chunk` candidate (there are already `parser/chunk` and
`service/chunk`), and lets the golden tool / future reuse pull in the
mapping logic without dragging in `task`'s `dao`/`engine`/`service`
dependency graph (Go subpackage import does not pull in the parent).
## Test plan
- `build.sh --test ./internal/ingestion/task/...` — **green** (task
4.7s, indexdoc 0.007s), matching the pre-change baseline.
- `gofmt` clean; `build.sh` builds both `ragflow-cli` and
`ragflow_server` successfully.
- Integration/E2E tiers are delegated to CI (need real MySQL/MinIO/ES
services).
Note: `pipeline_e2e_test.go` has a **pre-existing** compile error
(`server.ElasticsearchConfig` / `server.InfinityConfig` are now defined
under `internal/server/config/`, not re-exported by `internal/server`).
This is unrelated to this change — the diff to that file is only the
added `indexdoc` import and the qualified `EmbeddingTokenConsumptionKey`
reference.
## Summary
Classify the Go test suite by dependency level using build tags so the
default
`go test ./...` run stays self-contained, and add local convenience
commands
plus a documented convention.
- Add build tags to 5 real-service tests that were previously un-tagged
and only
soft-isolated via `t.Skip`: `kg_test.go` (integration), `minio_test.go`
(integration), `template_integration_test.go` (integration),
`stagehand_runtime_integration_test.go` (integration),
`pipeline_e2e_test.go` (e2e).
The default unit run no longer compiles/attempts these.
- Reclassify the full-pipeline `real_consumer` tests from `integration`
to `e2e`.
- Add `build.sh` shortcuts: `--test-integration`, `--test-e2e`,
`--test-manual`,
`--test-all` (integration + e2e; `manual` is excluded and is local
opt-in only,
never run in CI).
- Document the tier scheme (unit / integration / e2e / manual +
orthogonal cgo) in
`AGENTS.md`.
## Tier definitions
| Tier | Build tag | Runs by default? |
|---|---|---|
| Unit | (none) | Yes — in-memory SQLite / miniredis / httptest stubs |
| Integration | `integration` | No (`-tags integration`) — single real
service |
| E2E | `e2e` | No (`-tags e2e`) — full ingest→index→retrieve pipeline |
| Manual | `manual` | No (`-tags manual`) — very slow; never in CI |
## Verification
- `gofmt -l` clean on all changed files; `bash -n build.sh` OK.
- `go list` confirms the default set excludes the tagged files, and
`-tags integration` / `-tags e2e` include them.
- Full regression: unit / integration / e2e each **97 ok, 0 FAIL**.
- Fixed a regression where `pipeline_knowledge_compiler_test.go` relied
on a
transitive import side-effect from `template_integration_test.go` to
register
the `File`/`Parser`/`TokenChunker` components; it now blank-imports the
component packages directly.
## Test plan
- [ ] `./build.sh --test` (unit) passes
- [ ] `./build.sh --test-integration` passes (needs real services; skips
otherwise)
- [ ] `./build.sh --test-e2e` passes (needs real services; skips
otherwise)
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
