3 Commits

Author	SHA1	Message	Date
Jack	8546eb62bf	Feat(ingestion): add canvas pipeline debug (dry-run) mode with View result log (#17538) ... ## Summary Adds a side-effect-free DataFlow canvas **debug (dry-run) mode** plus a **debug run log with a "View result" panel**, so a canvas can be executed synchronously and inspected end-to-end (per-component progress and parsed chunks) without persisting anything. ### Dry-run execution (inline parsed chunks) - `task/debug.go`: `NewDebugTaskContext` builds an in-memory `TaskContext` with **`KB.ID == ""` — the single debug signal used across the ingestion pipeline**. A canvas debug run has no knowledgebase, and production ingestion always supplies one, so `kb_id == ""` occurs ONLY in debug mode. Components gate their own side effects on this signal without any dedicated debug vocabulary (the former `CANVAS_DEBUG_DOC_ID` marker constant is removed). - `task/pipeline_executor.go`: `validateTaskContext` no longer requires a KB when `KB.ID == ""` (debug); debug runs return `collectDebugOutput` (chunks) instead of a no-op; uploaded bytes are delivered as `inputs['binary']` for doc-less runs; `injectDebugPageCap` caps the parser to the first pages for a fast preview via the production `override_params` channel (Parser cpnID + family). - `component/tokenizer.go`: `shouldHaveEmbedding` skips embedding when `kb_id == ""` — the embedder is configured on the knowledgebase, so a debug run has nothing to resolve against and stays side-effect free. - `chunker/register.go`: chunk images are uploaded to MinIO only when a KB is present (persist run). **In debug mode the raw image bytes are intentionally dropped (`delete(ck, "image")`)** — the debug preview does not render chunk images, and dropping the bytes keeps them out of memory and out of the Redis-stored debug log. This is a deliberate trade-off, not an oversight. - `component/file.go`: pass through in-memory binary bytes, skipping `doc_id` -> storage resolution. - `handler/agent.go` + `agent_webhook.go`: detect `dataflow_canvas` and run a sync debug returning chunks inline on the existing chat/completions endpoint; reject DataFlow canvases from webhooks (fixes the previously dead `== "DataFlow"` check; mirrors Python `agent_api.py`). - `parser_dispatch.go`: export `ParserFileFamily` for the executor's page-cap injection. ### Debug run log + "View result" Mirrors Python's debug-log contract so the front-end can replay each component's progress and parsed output: - `task/debug_log_sink.go`: a `DebugLogSink` records every component's lifecycle into a `[{component_id, trace}]` array (each trace entry carries `message`, `progress`, `timestamp`, `elapsed_time`). `Flush` appends a terminal `END` marker whose first trace message is non-empty so the front-end detects completion. On failure the END marker is prefixed `[ERROR]` yet still carries the run, so the failure timeline renders instead of being stuck empty. Timestamps and `elapsed_time` are in seconds (matching the rest of the app). - `task/debug_result_dsl.go`: `BuildDebugResultDSL` builds the `dsl` the END marker carries — the Go analogue of Python's `Graph.__str__` + END-marker `dsl` in `rag/flow/pipeline.py`. It combines the static DSL structure (component_name / downstream / params / graph.nodes) with the run output map (`output["state"][<id>]`) to emit, per component, `obj.params.outputs[<format>].value` (chunks / text / json / html / markdown) — the exact keys the front-end `dataflow-result` page reads to render each step's parsed chunks. Raw embedding vectors (including the dimension-scoped `q_<dim>_vec` keys) are stripped so the stored log stays Python-scale. - `task/pipeline_executor.go`: after the run, attach the built `dsl` to the END marker via the `ResultSink` capability. - `handler/agent.go`: `runCanvasPipelineDebug` generates a stable `message_id` up-front and always flushes the log (success or failure); `respondWithDebugResult` returns `message_id` in **both** the success and the error envelope so the front-end can poll the log. The debug-log endpoint `GET /agents/:id/logs/:message_id` serves the array. - `web/src/pages/agent/hooks/use-run-dataflow.ts`: on a run failure, also surface `message_id` via `setMessageId` so the log sheet renders the failure timeline (the `[ERROR]` END marker is already written). Guarded by `if (msgId)`, so it is a safe no-op when the back-end does not return an id. ## Behavioral notes - Debug parses only the first pages (`debugPageCapPages`) for a fast preview; an explicit `pages` cap already present in the ParserConfig is respected. - Debug mode does not keep chunk images (see above) and does not compute embeddings — it exercises parse + chunk only. ## Test plan - Go: `debug_test.go`, `debug_log_sink_test.go` (trace pairing, END marker, `[ERROR]` prefix, fractional-second timestamp/elapsed_time, size caps, and a real-pipeline test asserting the END-marker `dsl` carries non-empty per-component `params.outputs` with chunks), `debug_result_dsl_test.go` (flat and real nested `output["state"]` shapes, vector stripping, format priority), `debug_pages_integration_test.go`, `pipeline_executor_persist_test.go`, `handler/agent_pipeline_debug_test.go`, `handler/agent_logs_test.go` (incl. `TestRunCanvasPipelineDebug_ErrorStillExposesMessageID` / `TestRespondWithDebugResult_ErrorCarriesMessageID` locking `message_id` on failure), plus updates to `agent_test.go` / `agent_webhook_test.go` / `chunker/image_upload_test.go` / `tokenizer*.go`. - `go build ./...` and `./build.sh --test` for affected packages. 🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy) --------- Co-authored-by: CodeBuddy Code <noreply@tencent.com>	2026-07-31 13:10:08 +08:00
Jin Hai	f53518c110	Go: add context, part5 (#17392) ... Signed-off-by: Jin Hai <haijin.chn@gmail.com>	2026-07-27 10:20:16 +08:00
Jack	2f7c2eb53c	Feat(ingestion): align image to MinIO upload, unify chunk-id computation and add PPT parsing support (#17111) ... ## Summary Align the Go ingestion pipeline with Python's `image` → `img_id` persistence semantics, and unify the chunk-id computation across all paths. ### Changes **1. Image upload at chunker stage ** - Add `ImageUploader` type and `DefaultImageUploader` in `internal/ingestion/component/image_uploader.go` — the write-side counterpart to `FetchBinary`, storing raw image bytes at `(bucket=kbID, key=chunkID)`, no re-encoding. - Add `uploadOneImage` — pure upload primitive (bytes in, `img_id` out), does not touch chunk maps. - Add `uploadChunkImages` / `uploadChunkImage` — caller-side helper: decodes `image` from a chunk, uploads bytes, writes `ck["img_id"]`, `delete(ck,"image")` , bounded by a process-wide semaphore (default 10, env `MAX_CONCURRENT_MINIO`). - Wire via `imageUploadDecorator` in `register.go`: every chunker runs the upload pass at invocation time, writing `ck["id"]` before upload and dropping image bytes right after — peak memory = single chunk image lifetime. **2. Unify chunk-id computation** - Consolidate three separate id-computation paths (`component.ChunkID`, `task.ChunkID`, inline `FormatUint` in API) into one: `common.ChunkID(docID, text string)`, using `%016x` + `xxhash.Sum64String(text+docID)` (matching Python `hexdigest()`). - The chunker decorator writes `ck["id"]` via `common.ChunkID`; the persist stage (`ProcessChunksForPipeline`) falls back to the same function (`if !exists id`). - The API AddChunk path now also calls `common.ChunkID` instead of the divergent `FormatUint(xxhash.Sum64(...))` — fixing a pre-existing inconsistency. - Delete `internal/ingestion/component/chunk_id.go` and `internal/ingestion/task/chunk_builder.go` (both were pure forwarding shells). **3. Preserve `img_id` (never deleted)** - `img_id` is a persistent index field (Infinity, OB) and the only consumer-side reference for image retrieval; it is NEVER removed from the chunk map. Only `image` (raw data URL) is dropped after upload. **4. PPT parser support** Previously PPT parsing failed. Add support to parse. ### Key design decisions | Decision | Choice | |----------|--------| | Upload timing | Chunker stage (not persist), so image bytes are dropped immediately — bounds peak memory to one chunk image | | Upload concurrency | Process-wide semaphore, default 10 (matches Python `minio_limiter`), env `MAX_CONCURRENT_MINIO` | | Image encoding | Store as-is, no JPEG re-encoding (unlike Python) | | `img_id` format | `"<kb_id>-<chunk_id>"` — matches Python task_executor path | | id function | Single `common.ChunkID(docID, text)`, concatenation `text+docID` inside hash (matching Python) | | `removeInternalChunkFields` | Retains `delete(ck,"image")` as defensive fallback for non-chunker paths | ### Files touched | File | Change | |------|--------| | `internal/common/format.go` | Add `ChunkID(docID, text)` | | `internal/common/format_test.go` | Add ChunkID golden-value test | | `internal/ingestion/component/image_uploader.go` | Add `ImageUploader` type + `DefaultImageUploader` | | `internal/ingestion/component/chunker/image_upload.go` | Add `uploadOneImage`, `uploadChunkImages`, `uploadChunkImage`, `decodeChunkImage`, semaphore | | `internal/ingestion/component/chunker/image_upload_test.go` | Tests: upload/drop, skip, no-image, concurrency, missing-id error | | `internal/ingestion/component/chunker/register.go` | Add `imageUploadDecorator` (writes `ck["id"]`, runs upload) | | `internal/ingestion/task/chunk_process.go` | Use `common.ChunkID` for persist fallback | | `internal/service/chunk/chunk.go` | Use `common.ChunkID` instead of `FormatUint` | | `internal/ingestion/component/chunk_id.go` | **Deleted** (moved to `common`) | | `internal/ingestion/task/chunk_builder.go` | **Deleted** (shell, no callers left) | | `internal/ingestion/task/chunk_builder_test.go` | **Deleted** (test migrated to `common/format_test.go`) | ### Verification ``` bash build.sh --test ./internal/service/chunk/... ./internal/common/... ./internal/ingestion/component/... ./internal/ingestion/task/... → ok service/chunk / common / component / chunker / schema / task ```	2026-07-20 19:33:51 +08:00