### What problem does this PR solve?
`common/misc_utils.once` set `executed=True` before invoking the wrapped
function, so an exception on the first call permanently disabled future
calls and returned the cached `None`.
This change marks `executed=True` only after a successful call, allowing
retries after transient failures while preserving once-only behavior
after success. It also adds regression tests for retry-after-exception
and thread-safe single execution.
### Type of change
- [x] Bug Fix (non-breaking change which fixes an issue)
---------
Co-authored-by: Harsh Kashyap <harshkashyap@Harshs-MacBook-Pro.local>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Jin Hai <haijin.chn@gmail.com>
### What problem does this PR solve?
Fixes#17623 by updating the retrieval API response example to use
`dataset_id`, matching the actual `POST /api/v1/retrieval` response
field and the implementation mapping from internal `kb_id`.
Co-authored-by: Codex <codex@openai.com>
## Summary
Remediates CVE-2026-59939 (HIGH) in `httplib2` by adding
`httplib2>=0.32.0` to `constraint-dependencies` in `pyproject.toml`.
| CVE | Severity | Package | Installed | Fixed in |
|---|---|---|---|---|
| CVE-2026-59939 | HIGH | httplib2 | 0.31.0 | 0.32.0 |
`httplib2` is a transitive dependency pulled in by
`google-api-python-client` and `google-auth-httplib2`. Both declare
`httplib2<1.0.0,>=0.19.0`, allowing 0.32.0 without conflict.
## Summary
Refactor the Go `KnowledgeCompilerComponent` so its parameter is a
**single string template id** instead of a DSL-level `variant` (or
plural group id list). The `variant` is no longer in the DSL — it is now
**derived at runtime from the resolved compilation template's `kind`
field**.
This aligns the Go ingestion port with the frontend Compiler operator,
which emits a singular `compilation_template_group_id` and does not
write `variant` into the generated `compiler.json`.
### Summary
As title, this can not be tested for now
Close#17520
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
## Summary
- Align Go agent logs access with the Python behavior by checking tenant
access to the target agent instead of requiring the token to be bound
to that agent.
- Marshal canvas lifecycle event payloads safely so non-serializable
inputs do not break Thinking expansion in the UI.
## Testing
- `CGO_ENABLED=0 go test -count=1 -v -run
"Handler|Attachment|Logs|Chatbot|Agentbot|BuildWorkflow|NodeLifecycle|VarRef|Resolve|Extract"./internal/handler/
./internal/agent/canvas/...` PASS
### Summary
`TestSearchHandlerUpdateRejectsInvalidSearchID` fails on `main`:
```
--- FAIL: TestSearchHandlerUpdateRejectsInvalidSearchID (0.00s)
search_handler_test.go:98: expected 'no authorization' in message, got No authorization.
```
`UpdateSearch` answers an unauthorized `search_id` with `"No
authorization."` (`internal/handler/search.go:337`), which mirrors the
Python API's message verbatim, but the assertion searched for the
lowercase `"no authorization"`.
The fix lowercases the response before the substring check rather than
changing the handler's message: the capitalized text is the contract the
Python API exposes, so the test — not the handler — was wrong.
`bash build.sh --test -run TestSearchHandler ./internal/handler/`
passes.
## Summary
Adds a side-effect-free DataFlow canvas **debug (dry-run) mode** plus a
**debug run log with a "View result" panel**, so a canvas can be
executed synchronously and inspected end-to-end (per-component progress
and parsed chunks) without persisting anything.
### Dry-run execution (inline parsed chunks)
- `task/debug.go`: `NewDebugTaskContext` builds an in-memory
`TaskContext` with **`KB.ID == ""` — the single debug signal used across
the ingestion pipeline**. A canvas debug run has no knowledgebase, and
production ingestion always supplies one, so `kb_id == ""` occurs ONLY
in debug mode. Components gate their own side effects on this signal
without any dedicated debug vocabulary (the former `CANVAS_DEBUG_DOC_ID`
marker constant is removed).
- `task/pipeline_executor.go`: `validateTaskContext` no longer requires
a KB when `KB.ID == ""` (debug); debug runs return `collectDebugOutput`
(chunks) instead of a no-op; uploaded bytes are delivered as
`inputs['binary']` for doc-less runs; `injectDebugPageCap` caps the
parser to the first pages for a fast preview via the production
`override_params` channel (Parser cpnID + family).
- `component/tokenizer.go`: `shouldHaveEmbedding` skips embedding when
`kb_id == ""` — the embedder is configured on the knowledgebase, so a
debug run has nothing to resolve against and stays side-effect free.
- `chunker/register.go`: chunk images are uploaded to MinIO only when a
KB is present (persist run). **In debug mode the raw image bytes are
intentionally dropped (`delete(ck, "image")`)** — the debug preview does
not render chunk images, and dropping the bytes keeps them out of memory
and out of the Redis-stored debug log. This is a deliberate trade-off,
not an oversight.
- `component/file.go`: pass through in-memory binary bytes, skipping
`doc_id` -> storage resolution.
- `handler/agent.go` + `agent_webhook.go`: detect `dataflow_canvas` and
run a sync debug returning chunks inline on the existing
chat/completions endpoint; reject DataFlow canvases from webhooks (fixes
the previously dead `== "DataFlow"` check; mirrors Python
`agent_api.py`).
- `parser_dispatch.go`: export `ParserFileFamily` for the executor's
page-cap injection.
### Debug run log + "View result"
Mirrors Python's debug-log contract so the front-end can replay each
component's progress and parsed output:
- `task/debug_log_sink.go`: a `DebugLogSink` records every component's
lifecycle into a `[{component_id, trace}]` array (each trace entry
carries `message`, `progress`, `timestamp`, `elapsed_time`). `Flush`
appends a terminal `END` marker whose first trace message is non-empty
so the front-end detects completion. On failure the END marker is
prefixed `[ERROR]` yet still carries the run, so the failure timeline
renders instead of being stuck empty. Timestamps and `elapsed_time` are
in seconds (matching the rest of the app).
- `task/debug_result_dsl.go`: `BuildDebugResultDSL` builds the `dsl` the
END marker carries — the Go analogue of Python's `Graph.__str__` +
END-marker `dsl` in `rag/flow/pipeline.py`. It combines the static DSL
structure (component_name / downstream / params / graph.nodes) with the
run output map (`output["state"][<id>]`) to emit, per component,
`obj.params.outputs[<format>].value` (chunks / text / json / html /
markdown) — the exact keys the front-end `dataflow-result` page reads to
render each step's parsed chunks. Raw embedding vectors (including the
dimension-scoped `q_<dim>_vec` keys) are stripped so the stored log
stays Python-scale.
- `task/pipeline_executor.go`: after the run, attach the built `dsl` to
the END marker via the `ResultSink` capability.
- `handler/agent.go`: `runCanvasPipelineDebug` generates a stable
`message_id` up-front and always flushes the log (success or failure);
`respondWithDebugResult` returns `message_id` in **both** the success
and the error envelope so the front-end can poll the log. The debug-log
endpoint `GET /agents/:id/logs/:message_id` serves the array.
- `web/src/pages/agent/hooks/use-run-dataflow.ts`: on a run failure,
also surface `message_id` via `setMessageId` so the log sheet renders
the failure timeline (the `[ERROR]` END marker is already written).
Guarded by `if (msgId)`, so it is a safe no-op when the back-end does
not return an id.
## Behavioral notes
- Debug parses only the first pages (`debugPageCapPages`) for a fast
preview; an explicit `pages` cap already present in the ParserConfig is
respected.
- Debug mode does not keep chunk images (see above) and does not compute
embeddings — it exercises parse + chunk only.
## Test plan
- Go: `debug_test.go`, `debug_log_sink_test.go` (trace pairing, END
marker, `[ERROR]` prefix, fractional-second timestamp/elapsed_time, size
caps, and a real-pipeline test asserting the END-marker `dsl` carries
non-empty per-component `params.outputs` with chunks),
`debug_result_dsl_test.go` (flat and real nested `output["state"]`
shapes, vector stripping, format priority),
`debug_pages_integration_test.go`, `pipeline_executor_persist_test.go`,
`handler/agent_pipeline_debug_test.go`, `handler/agent_logs_test.go`
(incl. `TestRunCanvasPipelineDebug_ErrorStillExposesMessageID` /
`TestRespondWithDebugResult_ErrorCarriesMessageID` locking `message_id`
on failure), plus updates to `agent_test.go` / `agent_webhook_test.go` /
`chunker/image_upload_test.go` / `tokenizer*.go`.
- `go build ./...` and `./build.sh --test` for affected packages.
🤖 Generated with [CodeBuddy Code](https://cnb.cool/codebuddy)
---------
Co-authored-by: CodeBuddy Code <noreply@tencent.com>
## Summary
- The `eng` flag passed to `removeTOCWord`/`removeContentsTable` was
hardcoded `false`, so English documents used the CJK 3-char TOC prefix
instead of Python's 2-word English prefix. This caused English tables of
contents to be under-deleted (left in indexed text) or over-deleted
(body paragraphs sharing the 3-char prefix dropped).
- `eng` is now derived from the parsed item content via `isEnglishItems`
(a port of Python `is_english`/`_is_english`), mirroring Python's
content heuristic.
## Changes
- `internal/parser/parser/text_toc.go`: add
`isEnglishItems`/`isEnglishTexts` (ASCII-ratio >80% ⇒ English,
fullmatch-anchored regex).
- `internal/parser/parser/docx_parser.go`: pass
`isEnglishItems(sections)` / `isEnglishItems(lineItems)` to
`removeTOCWord` (json + markdown paths).
- `internal/parser/parser/html_parser.go`: pass `isEnglishItems(items)`
to `removeContentsTable`.
- `docs/migration_python_go_diff.md`: close the 1.9/1.10 `eng` residual
and resolve the contradictory Parser 2.11 "Partially fixed" note.
## Notes
- `remove_toc` is a Parser-stage, DSL-configured feature (`remove_toc:
true/false` in the parser family setup). This change only fixes the
internal English/CJK prefix decision; it does not alter the pipeline or
the DSL contract. `remove_header_footer` (precise match) is unaffected.
- `eng` is auto-derived from content rather than exposed as a new config
knob, to stay faithful to Python behavior.
## Test plan
- `CGO_ENABLED=0 go test ./internal/parser/parser/ -run
'TestRemoveContentsTable|TestIsEnglishTexts|TestRemoveTOCWordEnglishDetection'`
- Added `TestIsEnglishTexts` (ASCII-ratio cases) and
`TestRemoveTOCWordEnglishDetection` (English TOC no longer over-deletes
"Chapter 2 Method").
---------
Co-authored-by: CodeBuddy <noreply@codebuddy.ai>
## Summary
- Fixes a compile error in `internal/channels/qqbot.go`: the local
`stringValue(value any)` helper collided with the existing
`stringValue(value *string)` in `feishu.go` within the same package
(`stringValue redeclared in this block`).
- Rename the qqbot variant to `anyToString` (definition + 13 call
sites). The two helpers do different jobs (any->string via `fmt.Sprint`
vs. `*string` deref), so they are not merged.
## Summary
Classify the Go test suite by dependency level using build tags so the
default
`go test ./...` run stays self-contained, and add local convenience
commands
plus a documented convention.
- Add build tags to 5 real-service tests that were previously un-tagged
and only
soft-isolated via `t.Skip`: `kg_test.go` (integration), `minio_test.go`
(integration), `template_integration_test.go` (integration),
`stagehand_runtime_integration_test.go` (integration),
`pipeline_e2e_test.go` (e2e).
The default unit run no longer compiles/attempts these.
- Reclassify the full-pipeline `real_consumer` tests from `integration`
to `e2e`.
- Add `build.sh` shortcuts: `--test-integration`, `--test-e2e`,
`--test-manual`,
`--test-all` (integration + e2e; `manual` is excluded and is local
opt-in only,
never run in CI).
- Document the tier scheme (unit / integration / e2e / manual +
orthogonal cgo) in
`AGENTS.md`.
## Tier definitions
| Tier | Build tag | Runs by default? |
|---|---|---|
| Unit | (none) | Yes — in-memory SQLite / miniredis / httptest stubs |
| Integration | `integration` | No (`-tags integration`) — single real
service |
| E2E | `e2e` | No (`-tags e2e`) — full ingest→index→retrieve pipeline |
| Manual | `manual` | No (`-tags manual`) — very slow; never in CI |
## Verification
- `gofmt -l` clean on all changed files; `bash -n build.sh` OK.
- `go list` confirms the default set excludes the tagged files, and
`-tags integration` / `-tags e2e` include them.
- Full regression: unit / integration / e2e each **97 ok, 0 FAIL**.
- Fixed a regression where `pipeline_knowledge_compiler_test.go` relied
on a
transitive import side-effect from `template_integration_test.go` to
register
the `File`/`Parser`/`TokenChunker` components; it now blank-imports the
component packages directly.
## Test plan
- [ ] `./build.sh --test` (unit) passes
- [ ] `./build.sh --test-integration` passes (needs real services; skips
otherwise)
- [ ] `./build.sh --test-e2e` passes (needs real services; skips
otherwise)
---------
Co-authored-by: CodeBuddy <noreply@cnb.cool>
### Summary
Add token usage reporting for the StepFun model provider. Related
to #17284.
Co-authored-by: Haruko386 <tryeverypossible@163.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
## Summary
- LongCat's token usage did not appear in the server terminal, while
DeepSeek's did — even though both drivers parse and record usage through
the same shared helpers. The gap was a missing debug log in LongCat's
SSE loop, so aggregate usage events were silently processed instead of
printed.
- The official LongCat API docs and live responses from
`api.longcat.chat` return a nested
`usage.completion_tokens_details.reasoning_tokens` breakdown for
thinking mode. The previous flat `LongCatChatResponse.Usage` struct
dropped this field on unmarshal.
- Add test coverage for the nested `reasoning_tokens` field on both the
non-streaming and streaming paths.
Closes#17284
### Summary
Add token usage reporting for the Groq model provider. Related to
#17284.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
