mirror of
https://github.com/infiniflow/ragflow.git
synced 2026-08-01 21:37:33 +08:00
## Summary
Adds a side-effect-free DataFlow canvas **debug (dry-run) mode** plus a
**debug run log with a "View result" panel**, so a canvas can be
executed synchronously and inspected end-to-end (per-component progress
and parsed chunks) without persisting anything.
### Dry-run execution (inline parsed chunks)
- `task/debug.go`: `NewDebugTaskContext` builds an in-memory
`TaskContext` with **`KB.ID == ""` — the single debug signal used across
the ingestion pipeline**. A canvas debug run has no knowledgebase, and
production ingestion always supplies one, so `kb_id == ""` occurs ONLY
in debug mode. Components gate their own side effects on this signal
without any dedicated debug vocabulary (the former `CANVAS_DEBUG_DOC_ID`
marker constant is removed).
- `task/pipeline_executor.go`: `validateTaskContext` no longer requires
a KB when `KB.ID == ""` (debug); debug runs return `collectDebugOutput`
(chunks) instead of a no-op; uploaded bytes are delivered as
`inputs['binary']` for doc-less runs; `injectDebugPageCap` caps the
parser to the first pages for a fast preview via the production
`override_params` channel (Parser cpnID + family).
- `component/tokenizer.go`: `shouldHaveEmbedding` skips embedding when
`kb_id == ""` — the embedder is configured on the knowledgebase, so a
debug run has nothing to resolve against and stays side-effect free.
- `chunker/register.go`: chunk images are uploaded to MinIO only when a
KB is present (persist run). **In debug mode the raw image bytes are
intentionally dropped (`delete(ck, "image")`)** — the debug preview does
not render chunk images, and dropping the bytes keeps them out of memory
and out of the Redis-stored debug log. This is a deliberate trade-off,
not an oversight.
- `component/file.go`: pass through in-memory binary bytes, skipping
`doc_id` -> storage resolution.
- `handler/agent.go` + `agent_webhook.go`: detect `dataflow_canvas` and
run a sync debug returning chunks inline on the existing
chat/completions endpoint; reject DataFlow canvases from webhooks (fixes
the previously dead `== "DataFlow"` check; mirrors Python
`agent_api.py`).
- `parser_dispatch.go`: export `ParserFileFamily` for the executor's
page-cap injection.
### Debug run log + "View result"
Mirrors Python's debug-log contract so the front-end can replay each
component's progress and parsed output:
- `task/debug_log_sink.go`: a `DebugLogSink` records every component's
lifecycle into a `[{component_id, trace}]` array (each trace entry
carries `message`, `progress`, `timestamp`, `elapsed_time`). `Flush`
appends a terminal `END` marker whose first trace message is non-empty
so the front-end detects completion. On failure the END marker is
prefixed `[ERROR]` yet still carries the run, so the failure timeline
renders instead of being stuck empty. Timestamps and `elapsed_time` are
in seconds (matching the rest of the app).
- `task/debug_result_dsl.go`: `BuildDebugResultDSL` builds the `dsl` the
END marker carries — the Go analogue of Python's `Graph.__str__` +
END-marker `dsl` in `rag/flow/pipeline.py`. It combines the static DSL
structure (component_name / downstream / params / graph.nodes) with the
run output map (`output["state"][<id>]`) to emit, per component,
`obj.params.outputs[<format>].value` (chunks / text / json / html /
markdown) — the exact keys the front-end `dataflow-result` page reads to
render each step's parsed chunks. Raw embedding vectors (including the
dimension-scoped `q_<dim>_vec` keys) are stripped so the stored log
stays Python-scale.
- `task/pipeline_executor.go`: after the run, attach the built `dsl` to
the END marker via the `ResultSink` capability.
- `handler/agent.go`: `runCanvasPipelineDebug` generates a stable
`message_id` up-front and always flushes the log (success or failure);
`respondWithDebugResult` returns `message_id` in **both** the success
and the error envelope so the front-end can poll the log. The debug-log
endpoint `GET /agents/:id/logs/:message_id` serves the array.
- `web/src/pages/agent/hooks/use-run-dataflow.ts`: on a run failure,
also surface `message_id` via `setMessageId` so the log sheet renders
the failure timeline (the `[ERROR]` END marker is already written).
Guarded by `if (msgId)`, so it is a safe no-op when the back-end does
not return an id.
## Behavioral notes
- Debug parses only the first pages (`debugPageCapPages`) for a fast
preview; an explicit `pages` cap already present in the ParserConfig is
respected.
- Debug mode does not keep chunk images (see above) and does not compute
embeddings — it exercises parse + chunk only.
## Test plan
- Go: `debug_test.go`, `debug_log_sink_test.go` (trace pairing, END
marker, `[ERROR]` prefix, fractional-second timestamp/elapsed_time, size
caps, and a real-pipeline test asserting the END-marker `dsl` carries
non-empty per-component `params.outputs` with chunks),
`debug_result_dsl_test.go` (flat and real nested `output["state"]`
shapes, vector stripping, format priority),
`debug_pages_integration_test.go`, `pipeline_executor_persist_test.go`,
`handler/agent_pipeline_debug_test.go`, `handler/agent_logs_test.go`
(incl. `TestRunCanvasPipelineDebug_ErrorStillExposesMessageID` /
`TestRespondWithDebugResult_ErrorCarriesMessageID` locking `message_id`
on failure), plus updates to `agent_test.go` / `agent_webhook_test.go` /
`chunker/image_upload_test.go` / `tokenizer*.go`.
- `go build ./...` and `./build.sh --test` for affected packages.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
---------
Co-authored-by: CodeBuddy Code <noreply@tencent.com>
93 lines
3.3 KiB
Go
93 lines
3.3 KiB
Go
//
|
|
// Copyright 2026 The InfiniFlow Authors. All Rights Reserved.
|
|
//
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
// you may not use this file except in compliance with the License.
|
|
// You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
// See the License for the specific language governing permissions and
|
|
// limitations under the License.
|
|
//
|
|
|
|
package handler
|
|
|
|
import (
|
|
"encoding/json"
|
|
"net/http"
|
|
"testing"
|
|
|
|
"ragflow/internal/common"
|
|
"ragflow/internal/entity"
|
|
"ragflow/internal/ingestion/task"
|
|
)
|
|
|
|
func dataflowCanvas(id, userID string) *entity.UserCanvas {
|
|
cv := makeWebhookCanvas(id, userID, "Webhook", nil)
|
|
cv.CanvasCategory = "dataflow_canvas"
|
|
return cv
|
|
}
|
|
|
|
func decodeSuccess(t *testing.T, body []byte) (int, string, []map[string]any, string) {
|
|
t.Helper()
|
|
var resp struct {
|
|
Code int `json:"code"`
|
|
Data struct {
|
|
MessageID string `json:"message_id"`
|
|
Chunks []map[string]any `json:"chunks"`
|
|
} `json:"data"`
|
|
Message string `json:"message"`
|
|
}
|
|
if err := json.Unmarshal(body, &resp); err != nil {
|
|
t.Fatalf("decode response: %v (body=%s)", err, body)
|
|
}
|
|
return resp.Code, resp.Data.MessageID, resp.Data.Chunks, resp.Message
|
|
}
|
|
|
|
// TestAgentChatCompletions_DataFlowReturnsChunks pins that the existing
|
|
// chat/completions endpoint serves the pipeline debug (dry-run) for DataFlow
|
|
// canvases: it returns the parsed chunks inline, synchronously, without a
|
|
// dedicated debug route. The webhook entry point is intentionally NOT a
|
|
// debug surface — see TestWebhook_DataFlowRejected in agent_webhook_test.go.
|
|
func TestAgentChatCompletions_DataFlowReturnsChunks(t *testing.T) {
|
|
h := &AgentHandler{
|
|
loader: &fakeCanvasLoader{canvas: dataflowCanvas("c1", "u-1")},
|
|
fileService: &fakeAgentFileService{blob: []byte("file-bytes")},
|
|
}
|
|
h.WithNewExecutor(func(_ *task.TaskContext, _ string, _ int) (debugExecutor, error) {
|
|
return &fakeDebugExecutor{result: &task.PipelineResult{Chunks: []map[string]any{{"text": "dbg-chunk"}}}}, nil
|
|
})
|
|
// runCanvasPipelineDebug flushes the debug log via redisStore; inject a no-op
|
|
// store so the (skipped in this test) log write does not nil-panic.
|
|
h.WithRedisStore(&capturedStore{})
|
|
|
|
// The web contract sends the file list wrapped one level:
|
|
// `files: [[{id, name}]]` (see use-run-dataflow.ts:34). The pipeline
|
|
// debug path unwraps both layers and downloads the referenced bytes.
|
|
c, w := webhookCtx("POST", "/api/v1/agents/c1/chat/completions",
|
|
`{"agent_id":"c1","files":[[{"id":"f1","name":"doc.txt"}]]}`, "application/json")
|
|
|
|
h.AgentChatCompletions(c)
|
|
|
|
if w.Code != http.StatusOK {
|
|
t.Fatalf("status = %d, want 200 (body=%s)", w.Code, w.Body.String())
|
|
}
|
|
code, msgID, data, msg := decodeSuccess(t, w.Body.Bytes())
|
|
if code != int(common.CodeSuccess) {
|
|
t.Errorf("code = %d, want %d", code, common.CodeSuccess)
|
|
}
|
|
if msg != "success" {
|
|
t.Errorf("message = %q, want %q", msg, "success")
|
|
}
|
|
if msgID == "" {
|
|
t.Errorf("message_id empty; front-end needs it to poll the debug log")
|
|
}
|
|
if len(data) != 1 || data[0]["text"] != "dbg-chunk" {
|
|
t.Errorf("data.chunks = %#v, want one chunk with text %q", data, "dbg-chunk")
|
|
}
|
|
}
|